Aviatrix Controller and Gateway Logging

In the CoPilot > Settings > Configuration > Logging Services page, you can configure the forwarding of logs from the Aviatrix platform to the log servers of well known log management systems. Each service indicates if it is enabled or disabled.

Overview of Controller and Gateway Logging

The Aviatrix Controller and all of its managed gateways can be configured to forward their logs to well known log management systems. The Controller and all the managed gateways will forward the logs directly to the logging server. As such, the Controller and each managed gateway needs network connectivity to the logging server.

Aviatrix supports using Remote Syslog (rsyslog) for forwarding log messages. Remote Syslog as the log forwarder is both efficient and the industry standard. Most log collectors support rsyslog as a log forwarder.

Log data collected from Aviatrix Controller and all the managed gateways can be forwarded by Remote Syslog to your log server, such as:

Datadog
NetFlow
AWS CloudWatch

In addition to standard information on syslog, Aviatrix logs also provide informational insights into UserVPN connections, VPN user TCP sessions, security rule violation statistics, gateway stats and FQDN filter violations.

The chosen log management system can sift through the Aviatrix logs to get meaningful trend charts that can help monitor the network connectivity and UserVPN sessions. See Aviatrix Log Formats for a list of useful Aviatrix logs which can be parsed on the log management system to display relevant analytics of data collected from Aviatrix Controller and gateways.

The process the Gateways and Controller use for exporting their log files is as follows:

The process the Gateways and Controller use for exporting their log files is different in Controller software versions earlier than 7.0.1726. In addition, the log file prefix previously included the log filename and log timestamp. For details, see Field Notice 42.

Aviatrix gateways and Controller stream the log lines being written to the syslog and auth.log files. When you use the default rsyslog server configuration shown in Remote Syslog Configuration on Controller, the logs streamed from the Controller and gateways have multiple files. Each file is named with the application that generated the log. For example, all logs generated by the avx-gw-state-sync application are re-directed to a file named avx-gw-state-sync" on the log server.

The log format is shown below. Your syslog collectors and any related automation for ingesting logs must accept the log format.

Format: [Timestamp] GW-[Name of Gateway]-[Public IP of Gateway] [Name of Application generating log][Application Process ID]: [Log message]

Example of log format:

Mar 23 19:17:50 GW-UdpGateway-50.17.41.173 avx-gw-state-sync[11249]: warn#011gateway_launcher/gateway_launcher.go:212#011daemon exited

Prefix of log: [Timestamp] GW-[Name of Gateway]-[Public IP of Gateway]

Example prefix: Mar 23 19:17:50 GW-gg-aws-usw2-s127-35.162.124.66

Enabling Aviatrix Controller Logging

To enable logging at the Aviatrix Controller, you can use the Controller UI (Settings > Logging) or the CoPilot UI (Settings > Configuration > Logging Services).

After setting the logging configuration in this page, you must click Enable for the configuration to be saved. After logging is enabled, both the Controller and all gateways will forward logs directly to the logging server.

Configuring the Aviatrix Datadog Agent

If you use Datadog, the Aviatrix platform offers a Datadog agent for sending system metrics from Aviatrix gateways to your configured Datadog instance.

Before configuring the Aviatrix Datadog agent, collect the following information:

Datadog account
Datadog API key

You can configure the Aviatrix Datadog agent in CoPilot > Settings > Configuration > Logging Services > Datadog Agent.

Configuring the Aviatrix NetFlow Agent

If you want to forward NetFlow data collected on Aviatrix gateways, the Aviatrix platform offers a NetFlow Agent for sending NetFlow data from Aviatrix gateways to your designated service point.

Before configuring the Aviatrix NetFlow agent, collect the following information:

IP address of the destination NetFlow service
Port number of the destination NetFlow service

You can configure the Aviatrix NetFlow Agent in CoPilot at Settings > Configuration > Logging Services > NetFlow Agent.

If you want to use the FlowIQ feature in Aviatrix CoPilot, you must configure the Aviatrix NetFlow Agent to forward NetFlow data to CoPilot.

Configuring the Aviatrix CloudWatch Agent

If you use CloudWatch, the Aviatrix platform offers a CloudWatch agent for sending syslog from Aviatrix Controller and Aviatrix gateways to your configured AWS CloudWatch instance.

Before configuring the Aviatrix CloudWatch agent, collect the following information:

CloudWatch role ARN
AWS Cloud Type
AWS Region
Log group name (optional)
Gateways included: select the gateways that will forward logs to CloudWatch.

You can configure the Aviatrix CloudWatch agent in CoPilot > Settings > Configuration > Logging Services > CloudWatch Agent.

Configuring the Aviatrix Remote Syslog Forwarder

If you want to forward syslog data collected on Aviatrix gateways, the Aviatrix platform offers a remote syslog forwarder for sending syslog data from Aviatrix gateways to your designated remote syslog server.

Before configuring the Aviatrix Remote Syslog forwarder, collect the following information:

FQDN or IP address of the remote syslog server
Port number of the listening port of the remote syslog server
Certificate Authority (CA) certificate
Public certificate signed by the same CA
Private key of the Controller that pairs with the public certificate

You can configure the Aviatrix Remote Syslog forwarder in CoPilot > Settings > Configuration > Logging Services > Remote Syslog.

If using the Controller UI, see Remote Syslog Configuration on Controller (Controller UI) below.

If you use Aviatrix CoPilot, you must configure the Aviatrix Remote Syslog forwarder to forward syslog data to CoPilot. Several CoPilot features, including the CoPilot > Administration > Audit feature, rely on syslog data.

Remote Syslog Configuration in CoPilot

In Aviatrix CoPilot, go to Settings > Configuration > Logging Services.
Under Remote Syslog, click Edit Profile.
Configure the following:

Field	Description
Profile	Select an existing profile or create a new one
Profile Name	The name of the profile
Server	FQDN or IP address of the remote syslog server
Protocol	TCP or UDP (TCP by default)
Port	Listening port of the remote syslog server (6514 by default)
Server CA Certificate	Certificate Authority (CA) certificate
Client Certificate	Public certificate of the Controller signed by the same CA
Client Private Key	Private key of the Controller that pairs with the public certificate
Custom Template	Useful when forwarding to third party servers like Datadog.
Gateways included	Select the gateways that will forward their logs to Remote Syslog

Field

Description

Profile

Select an existing profile or create a new one

Profile Name

The name of the profile

Server

FQDN or IP address of the remote syslog server

Protocol

TCP or UDP (TCP by default)

Port

Listening port of the remote syslog server (6514 by default)

Server CA Certificate

Certificate Authority (CA) certificate

Client Certificate

Public certificate of the Controller signed by the same CA

Client Private Key

Private key of the Controller that pairs with the public certificate

Custom Template

Useful when forwarding to third party servers like Datadog.

Gateways included

Select the gateways that will forward their logs to Remote Syslog

Click Save.

Remote Syslog Configuration on Controller (Controller UI)

On the Aviatrix Controller, go to Controller > Settings > Logging > Remote Syslog tile:

In Profile Index, select a profile to edit and then click the Status button to expand the configuration pane.
- Server: FQDN or IP address of the remote syslog server
- Port: Listening port of the remote syslog server (6514 by default)
- CA Certificate: Certificate Authority (CA) certificate
- Server Public Certificate: Public certificate of the Controller signed by the same CA
- Server Private Key: Private key of the Controller that pairs with the public certificate
- Protocol: TCP or UDP (TCP by default)
- Optional Custom Template: Useful when forwarding to third party servers like Datadog. See topic Using Rsyslog with Datadog below for an example.

A total of 10 profiles from index 0 to 9 are supported for remote syslog, while index 9 is reserved for CoPilot.

Newly deployed gateways will be added to a profile if it is the only profile enabled in the index range of 0 to 8.

If more than one profile is enabled in the range of 0 to 8, the newly deployed gateway will not be added to any profile in the range of 0 to 8. You can use the advanced options in the logging "Edit Options" window to edit the exclude and include list.

Newly deployed gateways will always be added to profile 9 which is reserved for CoPilot to monitor.

Using Rsyslog with Datadog

If you use Datadog, you can configure the Aviatrix Remote Syslog forwarder to send syslog data from Aviatrix gateways to your configured Datadog instance.

In CoPilot > Settings > Configuration > Logging Services > Remote Syslog, enable the service by clicking Edit Profile and then clicking Disable in the Manage Remote Syslog dialog.
You are prompted if you indeed want to disable syslog forwarding. Click Disable.
Configure the following:
- API Key: copy the following string and replace the string DATADOG_API_KEY with your own key.
  DATADOG_API_KEY <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% - - - %msg%\\n
- Site: the Datadog site (datadoghq.com)
- Export: Syslog & Metrics or Only Metrics
- Gateways included: select the gateways that will forward their logs to Remote Syslog.
- Server: intake.logs.datadoghq.com
Click Save.

Remote Syslog Configuration on the Remote syslog Server

On the Remote syslog server:

Install rsyslog and rsyslog-gnutls packages.
Create a new config file in /etc/rsyslog.d with the similar content as shown in the below example, depending on your rsyslog version, to enable the tls connection. Please make sure key paths are readable by the syslog user.
Make sure the output directory /var/log is writable by rsyslog user/daemon.
Restart rsyslog service and check port is listening and no error in /var/log/syslog.

Confirm the port is allowed in the security group / fireware for incoming traffic.

Remote Syslog version 8 or lower:

$ModLoad imtcp
$InputTCPServerRun 514

$DefaultNetstreamDriver gtls

#Certificate location
$DefaultNetstreamDriverCAFile /etc/cert/rsyslog-ca.pem
$DefaultNetstreamDriverCertFile /etc/cert/rsyslog-crt.pem
$DefaultNetstreamDriverKeyFile /etc/cert/rsyslog-key.pem

$InputTCPServerStreamDriverAuthMode x509/certvalid
$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode

# Re-direct logs to host specific directories
$template TmplMsg, "/var/log/aviatrix/%HOSTNAME%/%PROGRAMNAME%"
*.info,mail.none,authpriv.*,cron.none ?TmplMsg
& ~

Remote Syslog version 8 or higher:

global(
    DefaultNetstreamDriver="gtls"
    DefaultNetstreamDriverCAFile="/etc/cert/rsyslog-ca.pem"
    DefaultNetstreamDriverCertFile="/etc/cert/rsyslog-crt.pem"
    DefaultNetstreamDriverKeyFile="/etc/cert/rsyslog-key.pem"
)
template(name="TmplMsg" type="list") {
    constant(value="/var/log/aviatrix/")
    property(name="hostname")
    constant(value="/")
    property(name="programname" SecurePath="replace")
    constant(value="")
    }
ruleset(name="remote"){
    *.info;mail.none;authpriv.*;cron.none action(type="omfile" DynaFile="TmplMsg")
}
module(
    load="imtcp"
    StreamDriver.Name="gtls"
    StreamDriver.Mode="1"
    StreamDriver.Authmode="anon"
)
input(type="imtcp" port="514" ruleset="remote")

Go to /var/log/aviatrix directory.
Find the directory of the desired Controller or gateway.
- Controller’s directory name is in a format of Controller-public_IP_of_controller
- Gateway’s directory name is in a format of GW-gateway_name-public_IP_of_gateway
Each controller/gateway directory should have:
- auth.log
- syslog

Format for CA Certificates from External Logging Servers

The Aviatrix Controller expects certificates in PEM format. Convert any certificates downloaded from your external logging server’s documentation into PEM format. Attempting to upload the wrong format may return an Exception Error.