Gateway Logging

In the Settings > Logging Services page, you can configure the forwarding of logs from the Aviatrix Platform to remote syslog (rsyslog).

Overview of Gateway Logging

The Aviatrix managed gateways can be configured to forward their logs to a syslog log management system. All managed gateways will forward the logs directly to the logging server. Therefore, each managed gateway needs network connectivity to the logging server.

Aviatrix supports using remote syslog (rsyslog) for forwarding log messages. Remote syslog as the log forwarder is both efficient and the industry standard. Most log collectors support rsyslog as a log forwarder.

In addition to standard information on syslog, Aviatrix logs also provide informational insights into security rule violation statistics, gateway stats, and FQDN filter violations.

The chosen log management system can sift through the Aviatrix logs to get meaningful trend charts that can help monitor the network connectivity. See Aviatrix Log Formats for a list of useful Aviatrix logs that can be parsed on the log management system to display relevant analytics of data collected from Aviatrix gateways.

The process the Aviatrix gateways use for exporting their log files is as follows:

  • Aviatrix gateways stream the log lines being written to the syslog and auth.log files.

  • When you use the default rsyslog server configuration, the logs streamed from the gateways have multiple files.

  • Each file is named with the application that generated the log.

    For example, all logs generated by the avx-gw-state-sync application are re-directed to a file named avx-gw-state-sync" on the log server.

Log Format

The log format is shown below. Your syslog collectors and any related automation for ingesting logs must accept the log format.

Format: [Timestamp] GW-[Name of Gateway]-[Public IP of Gateway] [Name of Application generating log][Application Process ID]: [Log message]

Example of log format:

Mar 23 19:17:50 GW-UdpGateway-50.17.41.173 avx-gw-state-sync[11249]: warn#011gateway_launcher/gateway_launcher.go:212#011daemon exited

Prefix of log: [Timestamp] GW-[Name of Gateway]-[Public IP of Gateway]

Example prefix: Mar 23 19:17:50 GW-gg-aws-usw2-s127-35.162.124.66

The Aviatrix Remote Syslog Forwarder

If you want to forward syslog data collected on Aviatrix gateways, the Aviatrix Platform offers a remote syslog forwarder for sending syslog data from Aviatrix gateways to your designated remote syslog server.

You can configure the Aviatrix Remote Syslog forwarder in Settings > Logging Services > Remote Syslog.

Before configuring the Aviatrix Remote Syslog forwarder, collect the following information:

  • FQDN or IP address of the remote syslog server

  • Port number and protocol type of the listening port of the remote syslog server

  • Certificate Authority (CA) certificate for the server

  • Public client certificate signed by the same CA

  • Private client key that pairs with the public certificate

Aviatrix PaaS expects certificates in PEM format. Convert any certificates downloaded from your external logging server’s documentation into PEM format. Attempting to upload the wrong format could return an Exception Error.

Syslog Profiles

You configure profiles for remote syslog to define the remote server where logs are sent, the protocol and port for access, certificates (optional), and from which VPCs or VNets to collect logs. A total of 9 profiles are supported for remote syslog.

Newly deployed gateways are added to a profile if it is the only profile enabled.

If more than one profile is enabled, the newly deployed gateway will not be added to any profile. You can use the advanced options in the logging "Edit Options" window to edit the exclude and include list.

Configure Remote Syslog

  1. Go to Settings > Logging Services.

  2. Under Remote Syslog, click + Profile.

  3. Configure the following:

    Field Description

    Profile Name

    Enter a unique descriptive name for the profile

    Server

    FQDN or IP address of the remote syslog server

    Protocol

    TCP or UDP

    Port

    Listening port of the remote syslog server

    Server CA Certificate

    Certificate Authority (CA) certificate for the syslog server

    Client Certificate

    Public certificate for Aviatrix platform, signed by the same CA as the server

    Client Private Key

    Private encryption key that pairs with the public certificate

    Custom Template

    (Optional) Define a custom format for syslog messages, which can be useful when forwarding to third party servers like Datadog

    VPCs/VNets included

    Select the VPCs/VNets that should forward logs to remote syslog

  1. Click Enable.

    The agent details display on the Logging Services page.

Configure Remote Syslog on the Remote Syslog Server

Configure the remote server to receive data from Aviatrix PaaS.

On the remote syslog (rsyslog) server:

  1. Install rsyslog and rsyslog-gnutls packages.

  2. Create a new configuration file in /etc/rsyslog.d to enable the TLS connection.

    The file should contain content similar to what is shown in the example below, depending on your rsyslog version. Please make sure key paths are readable by the syslog user.

    Remote Syslog version 8 or greater:

    global(
    DefaultNetstreamDriver="gtls"
    DefaultNetstreamDriverCAFile="/etc/cert/rsyslog-ca.pem"
    DefaultNetstreamDriverCertFile="/etc/cert/rsyslog-crt.pem"
    DefaultNetstreamDriverKeyFile="/etc/cert/rsyslog-key.pem"
)
template(name="TmplMsg" type="list") {
    constant(value="/var/log/aviatrix/")
    property(name="hostname")
    constant(value="/")
    property(name="programname" SecurePath="replace")
    constant(value="")
    }
ruleset(name="remote"){
    *.info;mail.none;authpriv.*;cron.none action(type="omfile" DynaFile="TmplMsg")
}
module(
    load="imtcp"
    StreamDriver.Name="gtls"
    StreamDriver.Mode="1"
    StreamDriver.Authmode="anon"
)
input(type="imtcp" port="514" ruleset="remote")

    Remote Syslog version 7 or lower:

    $ModLoad imtcp
$InputTCPServerRun 514

$DefaultNetstreamDriver gtls

#Certificate location
$DefaultNetstreamDriverCAFile /etc/cert/rsyslog-ca.pem
$DefaultNetstreamDriverCertFile /etc/cert/rsyslog-crt.pem
$DefaultNetstreamDriverKeyFile /etc/cert/rsyslog-key.pem

$InputTCPServerStreamDriverAuthMode x509/certvalid
$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode

# Re-direct logs to host specific directories
$template TmplMsg, "/var/log/aviatrix/%HOSTNAME%/%PROGRAMNAME%"
*.info,mail.none,authpriv.*,cron.none ?TmplMsg
& ~

  1. Make sure the output directory /var/log is writable by rsyslog user/daemon.

  2. Restart rsyslog service and verify that the port is listening and there are no errors in /var/log/syslog.

  3. Confirm the port is allowed in the security group/firewall for incoming traffic.

  4. Go to the /var/log/aviatrix directory.

  5. Find the directory of the desired gateway.

    • The gateway directory name is in a format of GW-gateway_name-public_IP_of_gateway

  6. Each gateway directory should have:

    • auth.log

    • syslog

Edit or Disable Remote Syslog

You can edit an existing profile. You can also disable a profile. When a profile is disabled, the configuration is deleted and no longer available.

  1. Go to Settings > Logging Services.

  2. Under Remote Syslog, click Edit Profile.

  3. If there is more than one profile, select the profile you want to edit.

  4. Do one of the following:

    • Make the desired updates and click Save.

    • Click Disable and confirm disabling the profile.

      The profile is deleted and no longer available.