Logging

1. Introduction

The Aviatrix Controller and all of its managed gateways can be configured to forward logs to well known log management systems. The controller and all of the managed gateways will forward the logs directly to the logging server and hence each of them need network connectivity to the logging server. Out of box integration is supported for the following logging service or systems.

  • Remote syslog (recommended to use)
  • Elastic Filebeat
  • Splunk Enterprise/Cloud
  • Sumo Logic
  • Datadog
  • Netflow
  • AWS CloudWatch

Note

We highly recommend user to use remote syslog (rsyslog) as log forwarder which is both efficient and the industry standard. Most log collectors support rsyslog as forwarder. We may only add new features to rsyslog going forward.

In addition to standard information on syslog, Aviatrix also provides capability for user VPN connections, VPN user TCP sessions, security rule violation statistics, Gateway stats and FQDN filter violations.

The Log Management System can be used to sift through the Aviatrix logs and get the meaningful trend charts that helps monitor the network connectivity and user VPN sessions. The following sections provide a list of useful Aviatrix logs which can be parsed on Splunk, Sumo Logic and other log management systems to display relevant analytics of data collected from Aviatrix Controller and gateways.

2. Aviatrix Log Format for Log Management Systems

The following types of Aviatrix log keywords can be identified by the Log Management System for further analysis:

Below are the details of each log keyword.

AviatrixVPNSession:

This log is for gateways that have VPN enabled. To enable VPN, check “VPN Access” when launching a gateway.

Logs with this prefix come from the Controller and contain information such as VPN user name, the VPN gateway IP address and name where the user connects to, client virtual IP address, connection duration, total received bytes, total transmitted bytes, and login and logout time. Two logs will be generated for each VPN connection. One is when the connection is established, the other when it is disconnected.

Example logs:

Connect Log:

Aug 17 22:07:39 ip-172-31-46-24 cloudx_cli: AviatrixVPNSession:
User=Splumo, Status=active, Gateway=splunksumo, GatewayIP=52.52.76.149,
VPNVirtualIP=192.168.0.6, PublicIP=N/A, Login=2016-08-17 22:07:38, Logout=N/A,
Duration=N/A, RXbytes=N/A, TXbytes=N/A

Disconnect log:

Aug 17 22:26:37 ip-172-31-46-24 cloudx_cli: AviatrixVPNSession:
User=Splumo, Status=disconnected, Gateway=splunksumo,
GatewayIP=52.52.76.149, VPNVirtualIP=192.168.0.6, PublicIP=N/A,
Login=2016-08-17 22:07:38, Logout=2016-08-17 22:26:37, Duration=0:0:18:59,
RXbytes=2.1 MB, TXbytes=9.03 MB

AviatrixUser:

This log is for gateways that have VPN enabled. To enable VPN, check “VPN Access” when launching a gateway.

Logs with this prefix come from each VPN gateway managed by the controller. The log contains the information for the TCP session, such as inbound and outbound interface, source IP address, destination IP address, TTL value, protocol name, and packet length. The log record is for each packet that passes through the VPN connection from the client to the destination.

Two example logs:

Aug 17 22:15:47 ip-10-100-0-60 kernel: [14167.983249]
***AviatrixUser***:IN= OUT=eth0 SRC=192.168.0.6 DST=68.67.154.85 LEN=64
TOS=0x00 PREC=0x00 TTL=63 ID=28916 DF PROTO=TCP SPT=50428 DPT=443
WINDOW=65535 RES=0x00 SYN URGP=0

Aug 17 22:15:47 ip-10-100-0-60 kernel: [14167.968275]
***AviatrixUser***:IN= OUT=eth0 SRC=192.168.0.6 DST=10.100.0.2 LEN=66
TOS=0x00 PREC=0x00 TTL=254 ID=13309 PROTO=UDP SPT=64775 DPT=53 LEN=46

AviatrixLicenseVPNUsers:

This log is for gateways that have VPN enabled. To enable VPN, check “VPN Access” when launching a gateway.

Logs with this prefix come from the Controller and can be used to monitor the license usage of active vpn users connected to all vpn gateways.

One example log:

Sep 25 23:40:19 ip-10-40-0-133 cloudxd: AviatrixLicsenseVPNUsers: users=2

Note

There is a typo in some versions (as noted in the above example) that incorrectly shows this entry as AviatrixLicsenseVPNUsers instead of AviatrixLicenseVPNUsers.

AviatrixRule:

You need to configure security policies to see AviatrixRule log.

Logs with this prefix come from each gateway managed by the controller. Any packet that triggers the security policy rule will generate a log record of this type with the first 100 bytes of the packet. It contains the information such as gateway IP address, inbound and outbound interface, MAC address, TTL value, protocol name, source IP address, destination IP address and packet length.

An example for a deny rule event is shown below. The log event prefix is “AvxRl gw1 D:”, where the gateway name is gw1, “D” represents Drop.

2019-04-10T23:33:47.217018+00:00 ip-10-240-0-44 kernel: [ 4976.320353] AvxRl gw1 D:IN=eth0 OUT=eth0 MAC=02:bd:e5:4f:d0:e2:02:d8:14:81:fc:48:08:00 SRC=10.240.1.60 DST=10.230.1.23 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=45312 DF PROTO=ICMP TYPE=8 CODE=0 ID=2833 SEQ=1

Another example for an accept rule event is shown below. The log event prefix is “AvxRl StatefulGW2 A:”, where the gateway name is StatefulGW2, “A” represents Accept.

2019-04-10T23:34:47.602166+00:00 ip-10-240-0-44 kernel: [ 5036.705845] AvxRl StatfulGW2 A:IN=eth0 OUT=eth0 MAC=02:bd:e5:4f:d0:e2:02:d8:14:81:fc:48:08:00 SRC=10.240.1.60 DST=10.230.1.23 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=48453 DF PROTO=ICMP TYPE=8 CODE=0 ID=2834 SEQ=1

AviatrixGwNetStats:

Logs with this prefix come from each gateway managed by the controller. These logs are sampled every minute and give details about gateway network interface.

Two example logs:

2020-06-09T17:29:31.372628+00:00 GW-test-10.23.183.116 perfmon.py: AviatrixGwNetStats:
timestamp=2020-06-09T17:29:31.371791 name=test public_ip=10.23.183.116.fifo private_ip=172.31.78.160
interface=eth0 total_rx_rate=10.06Kb total_tx_rate=12.77Kb total_rx_tx_rate=2.85Kb
total_rx_cum=207.16MB total_tx_cum=1.2MB total_rx_tx_cum=208.36

2020-06-12T08:30:09.297478+00:00 GW-test-10.23.183.116 perfmon.py: AviatrixGwNetStats:
timestamp=2020-06-12T08:30:09.296752 name=test public_ip=10.23.183.116.fifo private_ip=172.31.78.160
interface=eth0 total_rx_rate=8.84Kb total_tx_rate=8.45Kb total_rx_tx_rate=17.29Kb
total_rx_cum=4.63MB total_tx_cum=6.8MB total_rx_tx_cum=11.44MB

AviatrixGwSysStats:

Logs with this prefix come from each gateway managed by the controller. These logs are sampled every minutes and give details about gateway memory, cpu and disk load.

Two example logs:

2020-06-09T17:29:31.372822+00:00 GW-test-10.23.183.116 perfmon.py: AviatrixGwSysStats:
timestamp=2020-06-09T17:29:31.371791 name=test cpu_idle=68
memory_free=414640 memory_available=1222000 memory_total=1871644
disk_total=16197524 disk_free=10982084

2020-06-12T08:22:09.295660+00:00 GW-test-10.23.183.116 perfmon.py: AviatrixGwSysStats:
timestamp=2020-06-12T08:22:09.294333 name=test cpu_idle=99
memory_free=919904 memory_available=1264792 memory_total=1871644
disk_total=16197524 disk_free=11409716

AviatrixFQDNRule

You need to configure FQDN Whitelists in order to see these logs.

Logs with this prefix come from each gateway managed by the controller. Domain name filtering can be configured per gateway via controller. And every time a gateway tries to access a domain name, it will check if the domain name passes the configured filters. If it does, access will be allowed with the state as MATCHED, otherwise it will be discarded with state as NO_MATCH.

Two example logs:

2019-12-12T04:33:46.892381+00:00 ip-172-32-0-6 avx-nfq: AviatrixFQDNRule2[CRIT]nfq_ssl_handle_client_hello() L#281  Gateway=spoke1-fqdn S_IP=172.32.1.144 D_IP=52.218.234.41 hostname=aviatrix-download.s3-us-west-2.amazonaws.com state=MATCHED  Rule=*.amazonaws.com;1

2019-12-12T04:36:53.173210+00:00 ip-172-32-0-6 avx-nfq: AviatrixFQDNRule1[CRIT]nfq_ssl_handle_client_hello() L#281  Gateway=spoke1-fqdn S_IP=172.32.1.144 D_IP=98.137.246.7 hostname=www.yahoo.com state=NO_MATCH drop_reason=NOT_WHITELISTED

AviatrixTunnelStatusChange

Logs with this prefix come from the controller whenever a tunnel status changes. old_state means old state of the tunnel, and new_state is the new changed state of tunnel.

Example log:

2019-11-30T15:44:52.718808+00:00 ip-172-32-0-226 cloudxd: AviatrixTunnelStatusChange: src_gw=oregon-transit(AWS us-west-2) dst_gw=100.20.53.124(NA NA) old_state=Down new_state=Up

AviatrixCMD

Logs with this prefix come from the controller whenever a CLI command is issued. It contains information on the CLI command that was issued, the results of the execution, reason a message if there is a failure and who issued the command.

Example log:

2019-11-19T20:13:44.585942+00:00 ip-172-32-0-226 cloudxd: AviatrixCMD: action=USERCONNECT_UPGRADE_TO_VERSION, argv=['--rtn_file', '/run/shm/rtn957594707', 'userconnect_upgrade_to_version', 'upgrade-status', ''], result=Success, reason=, username=admin

2019-11-19T18:01:59.796230+00:00 ip-172-32-0-226 cloudxd: AviatrixCMD: action=TRANSIT_SPOKE_LIST, argv=['--rtn_file', '/run/shm/rtn2091225061', 'transit_spoke_list', '--spoke_only'], result=Success, reason=, username=admin

AviatrixBGPOverlapCIDR

Log messages with this prefix come from the Controller whenever it detects overlapping CIDRs between on-prem learned and Spoke VPC CIDRs.

Example log:

2018-09-24T20:28:58.330708+00:00 ip-172-31-23-128 cloudxd: AviatrixBGPOverlapCIDR: Time Detected: 2018-09-24 20:28:58.329881

Spoke/Manual CIDRs ['10.0.0.0/8'] have a conflict with BGP Learned CIDRs [u'10.2.0.0/16', u'30.2.0.0/16'] in VPC vpc-782bb21f on connection vgw-bgp-ha.

AviatrixBGPRouteLimitThreshold

Log messages with this prefix come from the Controller whenever it detects that total BGP routes exceed the 80 routes. (AWS VGW has a total 100 route limit.)

Example log:

2018-09-24T20:24:50.600144+00:00 ip-172-31-23-128 cloudxd: AviatrixBGPRouteLimitThreshold: This message is alerting you that the VGW listed below currently has 89 routes, which is approaching the VGW route limits (100). You can reduce the number of routes on VGW both from on-prem side and on Aviatrix Transit gateway by enabling Route Summarization feature.

Time Detected: 2018-09-24 20:24:50.599822

Connection Name: vgw-bgp-ha
VGW Id: vgw-0942b724a5150bc6a

AviatrixGuardDuty

Log messages with this prefix come from the Controller whenever it receives an alert message from AWS GuardDuty.

Example log:

2018-09-23T00:00:50.369963-07:00 ip-172-31-89-197 cloudxd: AviatrixGuardDuty: Account [aws], Region [us-east-1], Instance ID [i-0a675b03fafedd3f2], at 2018-09-23T02:05:35Z, 163.172.7.97 is performing SSH brute force attacks against i-0a675b03fafedd3f2.  Please tighten instance security group to avoid UnauthorizedAccess:EC2/SSHBruteForce threat.

2018-09-23T00:00:50.332066-07:00 ip-172-31-89-197 cloudxd: AviatrixGuardDuty: Account [aws], Region [us-east-1], Instance ID [i-0a675b03fafedd3f2], at 2018-09-23T06:35:40Z, Unprotected port on EC2 instance i-0a675b03fafedd3f2 is being probed. Please tighten instance security group to avoid Recon:EC2/PortProbeUnprotectedPort threat.

AviatrixFireNet

Log messages with this prefix come from the Controller whenever a firewall instance state changes.

Example log:

2019-11-19T09:38:40.070080-08:00 ip-172-31-93-101 cloudxd: AviatrixFireNet: Firewall i-021f23187b8ac81c9~~tran-fw-1 in FireNet VPC vpc-0f943cd05455358ac~~cal-transit-vpc-1 state has been changed to down.

2019-11-19T09:39:03.066869-08:00 ip-172-31-93-101 cloudxd: AviatrixFireNet: Firewall i-021f23187b8ac81c9~~tran-fw-1 in FireNet VPC vpc-0f943cd05455358ac~~cal-transit-vpc-1 state has been changed to unaccessible.

2019-11-19T09:40:12.878075-08:00 ip-172-31-93-101 cloudxd: AviatrixFireNet: Firewall i-021f23187b8ac81c9~~tran-fw-1 in FireNet VPC vpc-0f943cd05455358ac~~cal-transit-vpc-1 state has been changed to up.

AviatrixVPNVersion

Log messages with this prefix come from the Controller whenever it rejects an Aviatrix VPN client connection.

Example log:

2020-02-07T11:38:48.276150-08:00 Controller-52.204.188.212 cloudxd: AviatrixVPNVersion:  The VPN connection was rejected as it did not satisfy the minimum version requirements. Current version: AVPNC-2.4.10 Required minimum version: AVPNC-2.5.7 . The rejected VPN user name is tf-aws-52-tcplb-user1

AviatrixGatewayStatusChanged

These log messages will be seen from the Controller’s syslogs when a gateway’s status changes

Example log:

2020-03-29T00:09:13.201669+00:00 ip-10-88-1-63 cloudxd: AviatrixGatewayStatusChanged: status=down gwname=EMEA-ENG-VPNGateway

3. Logging Configuration at Aviatrix Controller

To enable logging at the Aviatrix Controller, go to Settings->Logging page. Once logging is enabled, both the Controller and all gateways will forward logs directly to the logging server.

Two examples for Remote Syslog and Logstash Forwarder follow below.

3.1 Remote Syslog

On the Aviatrix Controller:
  1. Server: FQDN or IP address of the remote syslog server
  2. Port: Listening port of the remote syslog server (6514 by default)
  3. CA Certificate: Certificate Authority (CA) certificate
  4. Server Public Certificate: Public certificate of the controller signed by the same CA
  5. Server Private Key: Private key of the controller that pairs with the public certificate
  6. Protocol: TCP or UDP (TCP by default)
  7. Optional Custom Template: Useful when forwarding to 3rd party servers like Datadog or Sumo (Details bellow)
On the Remote syslog server:
  1. Install rsyslog and rsyslog-gnutls packages
  2. Create a new config file in /etc/rsyslog.d with the similar content as in the following box depends on your rsyslog version to enable tls connection. Please make sure key paths are readable by the syslog user
  3. Make sure the output directory /var/log is writable by rsyslog user/daemon
  4. Restart rsyslog service and check port is listening and no error in /var/log/syslog
  5. Confirm the port is allowed in the security group / fireware for incoming traffic

(version <8)

$ModLoad imtcp
$InputTCPServerRun 514

$DefaultNetstreamDriver gtls

#Certificate location
$DefaultNetstreamDriverCAFile /etc/cert/rsyslog-ca.pem
$DefaultNetstreamDriverCertFile /etc/cert/rsyslog-crt.pem
$DefaultNetstreamDriverKeyFile /etc/cert/rsyslog-key.pem

$InputTCPServerStreamDriverAuthMode x509/certvalid
$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode

# Re-direct logs to host specific directories
$template TmplMsg, "/var/log/aviatrix/%HOSTNAME%/%PROGRAMNAME%"
*.info,mail.none,authpriv.*,cron.none ?TmplMsg
& ~

(version >=8)

global(
    DefaultNetstreamDriver="gtls"
    DefaultNetstreamDriverCAFile="/etc/cert/rsyslog-ca.pem"
    DefaultNetstreamDriverCertFile="/etc/cert/rsyslog-crt.pem"
    DefaultNetstreamDriverKeyFile="/etc/cert/rsyslog-key.pem"
)
template(name="TmplMsg" type="list") {
    constant(value="/var/log/aviatrix/")
    property(name="hostname")
    constant(value="/")
    property(name="programname" SecurePath="replace")
    constant(value="")
    }
ruleset(name="remote"){
    *.info;mail.none;authpriv.*;cron.none action(type="omfile" DynaFile="TmplMsg")
}
module(
    load="imtcp"
    StreamDriver.Name="gtls"
    StreamDriver.Mode="1"
    StreamDriver.Authmode="anon"
)
input(type="imtcp" port="514" ruleset="remote")
Then
  1. Go to /var/log/aviatrix directory
  2. Find the directory of desired controller or gateway
    1. Controller’s directory name is in a format of Controller-public_IP_of_controller
    2. Gateway’s directory name is in a format of GW-gateway_name-public_IP_of_gateway
  3. Each controller/gateway directory should have
    1. auth.log
    2. syslog

3.1.a Using Rsyslog to send logs to Sumo

Since Sumo agents on the controller and gateways tend to consume a lot of cpu/memory resources, we strongly suggest that rsyslog is used instead to send logs to Sumo. This is documented by Sumo. Follow the following instructions:

  1. Follow the directions in Sumo document to create a cloud syslog source on your collections. Save the token, host and tcp tls port.
  2. Go to Controller/Settings/Logging/Remote Syslog and enable the service
  3. Enter the Server ip/fqdn that you received from the first step
  4. Provide the port - obtained from the first step
  5. Upload the CA cert from Sumo pointed by their documentation
  6. Keep the Protocol set to TCP
  7. For Optional Custom Template, copy the following string and replace the string ADD_YOUR_SUMO_TOKEN_HERE with the token you received in the first step. Please do keep the square brackets around the token.

Note

<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [YOUR_TOKEN] %msg%n

rsyslog_template

3.1.b Using Rsyslog to send logs to Datadog

  1. Go to Controller/Settings/Logging/Remote Syslog and enable the service
  2. Server: intake.logs.datadoghq.com
  3. Port: 10516
  4. Protocol: TCP
  5. For Optional Custom Template, copy the following string and replace the string DATADOG_API_KEY with your own key. Please do keep the square brackets around the token.

Note

<DATADOG_API_KEY> <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% - - - %msg%n

3.1.c Using Rsyslog to send logs to Splunk

  1. Follow the directions in Splunk Monitornetworkports to create a listener in Splunk.
  2. Go to Controller/Settings/Logging/Remote Syslog and enable the service
  3. Server: your Splunk server fqdn or ip
  4. Port: your Splunk listener port
  5. Protocol: TCP
  6. Optional Custom Template: (leave blank)

3.1.d Using Rsyslog to send logs to Logstash (ElasticSearch/Kibana/ELK stack)

  1. Follow the directions in Logstash TCP input to create a tcp listener in Logstash.
  2. Go to Controller/Settings/Logging/Remote Syslog and enable the service
  3. Server: your Logstash server fqdn or ip
  4. Port: your Logstash listener port
  5. Protocol: TCP
  6. Optional Custom Template: (leave blank)

A sample config of Logstash to work with Rsyslog in ELK stack v7 is

input {
    syslog {
        port => 6514
    }
}

output {
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
    }
}

3.2 Filebeat Forwarder

On the Aviatrix Controller:
  1. Server: FQDN or IP address of logstash server
  2. Port: Listening port of logstash server (5000 by default)
  3. Optional Configuration File: (Deprecated)

A sample config of Logstash to work with Filebeat in ELK stack v7 is

input {
    beats {
        port => 5000
    }
}

filter {
  mutate {
    rename => {
      "[host][name]" => "[host]"
    }
  }
}

output {
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
    }
}

3.3 Splunk Logging

On the Aviatrix Controller:
  1. How to configure: Manual Input or Import File
  2. Splunk Server: FQDN or IP address of Splunk Enterprise Server
  3. Splunk Server Listening Port: Listening port of Splunk Enterprise Server
  4. Splunk inputs.conf stanza: (Deprecated)

Note: If “Import File” is selected for “How to configure”, please provide the Splunk configuration file.

3.4 Sumo Logic

On the Aviatrix Controller:
  1. Access ID : ID of SumoLogic server
  2. Access Key: Access key of SumoLogic server
  3. Source Category: The category string of the source
  4. Additional Configurations: (Deprecated)

Steps to upgrade Sumologic Collectors(eg: Controllers/Gateways) from SumoLogic servers.

Please note that Sumo collector is memory intensive and needs instances with at least 2GB of memory - for AWS, t3.small, or higher depending on features deployed.

3.5 DataDog Agent

You may refer to this link, DatadogIntegration to set up. However, based on the past year experience, the vendor has changed the client root certificates for a few times.
  1. You may disable DataDog Agent and re-enable it to fetch the current new root certificate.
  2. Or, we highly recommend to follow above 3.1.b steps to use Remote Syslog as client to forward to any servers and will not encounter any of these cert issues.

Before 5.3 release, DataDog agent woulld only upload metrics from the Aviatrix Controller and Gateways - from release 5.3, we also upload syslogs to bring it on par with Sumo and Splunk agent behavior.

3.6 Cloudwatch

Please follow this link AWS CloudWatch Integration for instruction.

4. Log management system Apps

The Aviatrix controller can be configured to forward logs to various log management systems. Aviatrix also provides apps with prebuilt dashboards for popular log management systems like Splunk and Sumo Logic.

Splunk App for Aviatrix

Splunk app for Aviatrix can be downloaded from Splunkbase.

Click SplunkforAviatrix to check instructions on GitHub.

Sample

splunk_sample

Sumo Logic App for Aviatrix

The Sumo Logic app installation guide is also available on GitHub.

Sample

sumo_sample

5. Loggly integration via Syslog

To configure Loggly integration through an intermediary syslog server relay:

  1. Build an rsyslog server relay using a Linux distribution of your choice
  2. Configure Aviatrix to send rsyslog traffic to the relay (section 3.1 above)
  3. Follow this document to configure the relay to send to Loggly

6. Netflow

Aviatrix gateways support Netflow protocol v5 and v9.

Please follow this link Netflow Integration to enable it.