Aviatrix Controller and Gateway Logging

In the CoPilot Settings > Configuration > Logging Services page, you can configure the forwarding of logs from the Aviatrix platform to the log servers of well known log management systems. Each service indicates if it is enabled or disabled.

Overview of Controller and Gateway Logging

The Aviatrix Controller and all of its managed gateways can be configured to forward their logs to well known log management systems. The Controller and all the managed gateways will forward the logs directly to the logging server. As such, the Controller and each managed gateway needs network connectivity to the logging server.

Aviatrix supports using Remote Syslog (rsyslog) for forwarding log messages. Remote Syslog as the log forwarder is both efficient and the industry standard. Most log collectors support rsyslog as a log forwarder.

Log data collected from Aviatrix Controller and all the managed gateways can be forwarded by Remote Syslog to your log server, such as:

Datadog
NetFlow
AWS CloudWatch

In addition to standard information on syslog, Aviatrix logs also provide informational insights into UserVPN connections, VPN user TCP sessions, security rule violation statistics, gateway stats and FQDN filter violations.

The chosen log management system can sift through the Aviatrix logs to get meaningful trend charts that can help monitor the network connectivity and UserVPN sessions. See Aviatrix Log Formats for a list of useful Aviatrix logs which can be parsed on the log management system to display relevant analytics of data collected from Aviatrix Controller and gateways.

The process the Gateways and Controller use for exporting their log files is as follows:

The process the Gateways and Controller use for exporting their log files is different in Controller software versions earlier than 7.0.1726. In addition, the log file prefix previously included the log filename and log timestamp. For details, see Field Notice 42.

Aviatrix gateways and Controller stream the log lines being written to the syslog and auth.log files.
When you use the default rsyslog server configuration, the logs streamed from the Controller and gateways have multiple files.
Each file is named with the application that generated the log.

For example, all logs generated by the avx-gw-state-sync application are re-directed to a file named avx-gw-state-sync" on the log server.

Log Format

The log format is shown below. Your syslog collectors and any related automation for ingesting logs must accept the log format.

Format: [Timestamp] GW-[Name of Gateway]-[Public IP of Gateway] [Name of Application generating log][Application Process ID]: [Log message]

Example of log format:

Mar 23 19:17:50 GW-UdpGateway-50.17.41.173 avx-gw-state-sync[11249]: warn#011gateway_launcher/gateway_launcher.go:212#011daemon exited

Prefix of log: [Timestamp] GW-[Name of Gateway]-[Public IP of Gateway]

Example prefix: Mar 23 19:17:50 GW-gg-aws-usw2-s127-35.162.124.66

Syslog Profiles

A total of 10 profiles from index 0 to 9 are supported for remote syslog, while index 9 is reserved for CoPilot Remote Syslog Profile.

Newly deployed gateways are added to a profile if it is the only profile enabled in the index range of 0 to 8.

If more than one profile is enabled in the range of 0 to 8, the newly deployed gateway will not be added to any profile in the range of 0 to 8. You can use the advanced options in the logging "Edit Options" window to edit the exclude and include list.

Newly deployed gateways will always be added to profile 9 which is reserved for CoPilot to monitor.

Configuring the Aviatrix Remote Syslog Forwarder

If you want to forward syslog data collected on Aviatrix gateways, the Aviatrix platform offers a remote syslog forwarder for sending syslog data from Aviatrix gateways to your designated remote syslog server.

Before configuring the Aviatrix Remote Syslog forwarder, collect the following information:

FQDN or IP address of the remote syslog server
Port number of the listening port of the remote syslog server
Certificate Authority (CA) certificate
Public certificate signed by the same CA
Private key of the Controller that pairs with the public certificate

You can configure the Aviatrix Remote Syslog forwarder in Settings > Configuration > Logging Services > Remote Syslog.

Remote Syslog Configuration in CoPilot

You can configure one or more syslog profiles.

If you use Aviatrix CoPilot, you must configure the Aviatrix CoPilot Remote Syslog Profile to forward syslog data to CoPilot. Several CoPilot features, including the Administration > Audit feature, rely on syslog data.

The Aviatrix Platform expects certificates in PEM format. Convert any certificates downloaded from your external logging server’s documentation into PEM format. Attempting to upload the wrong format may return an Exception Error.

In Aviatrix CoPilot, go to Settings > Configuration > Logging Services.
Under Remote Syslog, click Edit Profile.

Configure the following:

Field	Description
Profile	Select an existing profile or create a new one. For syslog forwarding to CoPilot, select Aviatrix CoPilot Remote Syslog Profile.
Profile Name	The name given to the profile
Server	FQDN or IP address of the remote syslog server
Protocol	TCP or UDP
Port	Listening port of the remote syslog server
Server CA Certificate	Certificate Authority (CA) certificate for the syslog server. in PEM format
Client Certificate	Public certificate for Aviatrix platform signed by the same CA
Client Private Key	Private encryption key that pairs with the public certificate
Custom Template	Define a custom format for syslog messages, which can be useful when forwarding to third party servers like Datadog
Gateways included	Select the gateways that will forward their logs to Remote Syslog

Field

Description

Profile

Select an existing profile or create a new one.

For syslog forwarding to CoPilot, select Aviatrix CoPilot Remote Syslog Profile.

Profile Name

The name given to the profile

Server

FQDN or IP address of the remote syslog server

Protocol

TCP or UDP

Port

Listening port of the remote syslog server

Server CA Certificate

Certificate Authority (CA) certificate for the syslog server. in PEM format

Client Certificate

Public certificate for Aviatrix platform signed by the same CA

Client Private Key

Private encryption key that pairs with the public certificate

Custom Template

Define a custom format for syslog messages, which can be useful when forwarding to third party servers like Datadog

Gateways included

Select the gateways that will forward their logs to Remote Syslog

Click Save.

The Aviatrix Datadog Agent

If you use Datadog, the Aviatrix platform offers a Datadog agent for sending system metrics from Aviatrix gateways to your configured Datadog instance.

You can configure the Aviatrix Datadog agent in Settings > Configuration > Logging Services > Datadog Agent.

Before configuring the Aviatrix Datadog agent, collect the following information:

Datadog account
Datadog API key

Configuring Rsyslog with Datadog

If you use Datadog, you can configure the Aviatrix Remote Syslog forwarder to send syslog data from Aviatrix gateways to your configured Datadog instance.

In CoPilot, go to Settings > Configuration > Logging Services > Datadog Agent and click Enable.
Configure the following:
- API Key: copy the following string and replace the string DATADOG_API_KEY with your own key.
  DATADOG_API_KEY <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% - - - %msg%\\n
- Site: select a Datadog site (datadoghq.com is default)
- Export: Syslog & Metrics or Only Metrics
- Gateways Included: select the gateways that will forward their logs to Remote Syslog.
Click Save.

The Aviatrix CloudWatch Agent

If you use CloudWatch, the Aviatrix platform offers a CloudWatch agent for sending syslog from Aviatrix Controller and Aviatrix gateways to your configured AWS CloudWatch instance.

You can configure the Aviatrix CloudWatch agent in Settings > Configuration > Logging Services > CloudWatch Agent.

Before configuring the Aviatrix CloudWatch agent, collect the following information:

CloudWatch role ARN
AWS Cloud Type
AWS Region
Log group name (optional)
Gateways included: select the gateways that will forward logs to CloudWatch.

Configuring the Aviatrix CloudWatch Agent

If you use AWS CloudWatch, you can configure the Aviatrix Remote Syslog forwarder to send syslog data from Aviatrix gateways to your configured CloudWatch instance.

In Settings > Logging Services > CloudWatch Agent, click Enable.
Configure the following:
- CloudWatch role ARN
- AWS Cloud Type: standard is the only option
- AWS Region
- Log group name, if using CloudWatch groups (optional advanced setting)
- VPCs/VNets Included: select the VPCs that should forward logs to CloudWatch (optional advanced setting)
Click Save.

The agent details display on the Logging Services page.

The Aviatrix NetFlow Agent

The Aviatrix platform offers a NetFlow Agent for sending NetFlow data from Aviatrix gateways to your designated service point.

You can configure the Aviatrix NetFlow Agent in CoPilot at Settings > Configuration > Logging Services > NetFlow Agent.

If you want to use any of the following features in Aviatrix CoPilot, you must enable the Aviatrix NetFlow Agent to forward NetFlow data to CoPilot:

FlowIQ
CostIQ
Anomaly Detection
Traffic monitoring
ThreatIQ (if you configured the feature prior to Controller version 7.2.4820)

Requires that the NetFlow Sampling Rate is set to 100%.
Geoblocking (if you configured the feature prior to Controller version 7.2.4820)

Requires that the NetFlow Sampling Rate is set to 100%.

The sampling rate is based on 1 out of N packets. So, if you set a sampling rate of 25%, then 1 out of 4 packets is used to create the sample.

The sampling rate can be increased or decreased to adjust the amount of data flow for FlowIQ, CostIQ, Anomaly Detection, and Traffic monitoring. It must remain at 100% for ThreatIQ and Geoblocking.

Have the following information available to configure the Aviatrix NetFlow Agent:

IP address of the destination NetFlow service (server)
Port number and protocol type of the destination NetFlow service
NetFlow version
Traffic mode (IPT or L7)
Sampling rate (0 to 100%)
Names of any gateways to be excluded

Configuring the NetFlow Agent

In CoPilot, go to Settings > Configuration > Logging Services > NetFlow Agent and click Enable.

Configure the following:

Field	Description
Server	IP address of the destination NetFlow service
Port	Port number of the destination NetFlow service
Version	NetFlow version being used; versions 9 and 5 are supported
Protocol	Supported protocols are UDP or TCP
Mode	Supported traffic modes are IPT or L7
Sampling Rate	The rate can be 0 to 100%
Gateways Excluded	Names of any gateways to be excluded

Field

Description

Server

IP address of the destination NetFlow service

Port

Port number of the destination NetFlow service

Version

NetFlow version being used; versions 9 and 5 are supported

Protocol

Supported protocols are UDP or TCP

Mode

Supported traffic modes are IPT or L7

Sampling Rate

The rate can be 0 to 100%

Gateways Excluded

Names of any gateways to be excluded

Click Save.

The agent details display on the Logging Services page.

Remote Syslog Configuration on the Remote Syslog Server

On the Remote syslog server:

Install rsyslog and rsyslog-gnutls packages.
Create a new config file in /etc/rsyslog.d with the similar content as shown in the below example, depending on your rsyslog version, to enable the tls connection. Please make sure key paths are readable by the syslog user.
Make sure the output directory /var/log is writable by rsyslog user/daemon.
Restart rsyslog service and check port is listening and no error in /var/log/syslog.

Confirm the port is allowed in the security group / firewall for incoming traffic.

Remote Syslog version 8 or lower:

$ModLoad imtcp
$InputTCPServerRun 514

$DefaultNetstreamDriver gtls

#Certificate location
$DefaultNetstreamDriverCAFile /etc/cert/rsyslog-ca.pem
$DefaultNetstreamDriverCertFile /etc/cert/rsyslog-crt.pem
$DefaultNetstreamDriverKeyFile /etc/cert/rsyslog-key.pem

$InputTCPServerStreamDriverAuthMode x509/certvalid
$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode

# Re-direct logs to host specific directories
$template TmplMsg, "/var/log/aviatrix/%HOSTNAME%/%PROGRAMNAME%"
*.info,mail.none,authpriv.*,cron.none ?TmplMsg
& ~

Remote Syslog version 8 or greater:

global(
    DefaultNetstreamDriver="gtls"
    DefaultNetstreamDriverCAFile="/etc/cert/rsyslog-ca.pem"
    DefaultNetstreamDriverCertFile="/etc/cert/rsyslog-crt.pem"
    DefaultNetstreamDriverKeyFile="/etc/cert/rsyslog-key.pem"
)
template(name="TmplMsg" type="list") {
    constant(value="/var/log/aviatrix/")
    property(name="hostname")
    constant(value="/")
    property(name="programname" SecurePath="replace")
    constant(value="")
    }
ruleset(name="remote"){
    *.info;mail.none;authpriv.*;cron.none action(type="omfile" DynaFile="TmplMsg")
}
module(
    load="imtcp"
    StreamDriver.Name="gtls"
    StreamDriver.Mode="1"
    StreamDriver.Authmode="anon"
)
input(type="imtcp" port="514" ruleset="remote")

Go to /var/log/aviatrix directory.
Find the directory of the desired Controller or gateway.
- Controller’s directory name is in a format of Controller-public_IP_of_controller
- Gateway’s directory name is in a format of GW-gateway_name-public_IP_of_gateway
Each controller/gateway directory should have:
- auth.log
- syslog