Overview

​

Overview of Gateway Logging

• Datadog Agent
• AWS CloudWatch Agent

• Aviatrix gateways stream the log lines being written to the syslog and auth.log files.
• When you use the default rsyslog server configuration, the logs streamed from the gateways have multiple files.
• Each file is named with the application that generated the log. For example, all logs generated by the avx-gw-state-sync application are re-directed to a file named avx-gw-state-sync” on the log server.

​

Log Format

Format: [Timestamp] GW-[Name of Gateway]-[Public IP of Gateway] [Name of Application generating log][Application Process ID]: [Log message]

Mar 23 19:17:50 GW-UdpGateway-50.17.41.173 avx-gw-state-sync[11249]: warn#011gateway_launcher/gateway_launcher.go:212#011daemon exited

​

The Aviatrix Remote Syslog Forwarder

• FQDN or IP address of the remote syslog server
• Port number and protocol type of the listening port of the remote syslog server
• Certificate Authority (CA) certificate for the server
• Public client certificate signed by the same CA
• Private client key that pairs with the public certificate

​

Syslog Profiles

​

Configure Remote Syslog

1. Go to Settings > Logging Services.
2. Under Remote Syslog, click + Profile.
3. Configure the following:

   Field	Description
   Profile Name	Enter a unique descriptive name for the profile
   Server	FQDN or IP address of the remote syslog server
   Protocol	TCP or UDP
   Port	Listening port of the remote syslog server
   Server CA Certificate	Certificate Authority (CA) certificate for the syslog server
   Client Certificate	Public certificate for Aviatrix Platform-as-a-Service (PaaS), signed by the same CA as the server
   Client Private Key	Private encryption key that pairs with the public certificate
   Custom Template	(Optional) Define a custom format for syslog messages, which can be useful when forwarding to third party servers like Datadog
   VPC/VNets included	Select the VPC/VNets that should forward logs to remote syslog

4. Click Enable. The agent details display on the Logging Services page.

​

Configure Remote Syslog on the Remote Syslog Server

1. Install rsyslog and rsyslog-gnutls packages.
2. Create a new configuration file in /etc/rsyslog.d to enable the TLS connection. The file should contain content similar to what is shown in the example below, depending on your rsyslog version. Make sure key paths are readable by the syslog user. Remote Syslog version 8 or later:
   global(
       DefaultNetstreamDriver="gtls"
       DefaultNetstreamDriverCAFile="/etc/cert/rsyslog-ca.pem"
       DefaultNetstreamDriverCertFile="/etc/cert/rsyslog-crt.pem"
       DefaultNetstreamDriverKeyFile="/etc/cert/rsyslog-key.pem"
   )
   template(name="TmplMsg" type="list") {
       constant(value="/var/log/aviatrix/")
       property(name="hostname")
       constant(value="/")
       property(name="programname" SecurePath="replace")
       constant(value="")
       }
   ruleset(name="remote"){
       *.info;mail.none;authpriv.*;cron.none action(type="omfile" DynaFile="TmplMsg")
   }
   module(
       load="imtcp"
       StreamDriver.Name="gtls"
       StreamDriver.Mode="1"
       StreamDriver.Authmode="anon"
   )
   input(type="imtcp" port="514" ruleset="remote")
   
   Remote Syslog version 7 or earlier:
   $ModLoad imtcp
   $InputTCPServerRun 514
   
   $DefaultNetstreamDriver gtls
   
   #Certificate location
   $DefaultNetstreamDriverCAFile /etc/cert/rsyslog-ca.pem
   $DefaultNetstreamDriverCertFile /etc/cert/rsyslog-crt.pem
   $DefaultNetstreamDriverKeyFile /etc/cert/rsyslog-key.pem
   
   $InputTCPServerStreamDriverAuthMode x509/certvalid
   $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
   
   # Re-direct logs to host specific directories
   $template TmplMsg, "/var/log/aviatrix/%HOSTNAME%/%PROGRAMNAME%"
   *.info,mail.none,authpriv.*,cron.none ?TmplMsg
   & ~
   
3. Make sure the output directory /var/log is writable by rsyslog user/daemon.
4. Restart rsyslog service and verify that the port is listening and there are no errors in /var/log/syslog.
5. Confirm the port is allowed in the security group/firewall for incoming traffic.
6. Go to the /var/log/aviatrix directory.
7. Find the directory of the desired gateway.
   • The gateway directory name is in a format of GW-gateway_name-public_IP_of_gateway
8. Each gateway directory should have:
   • auth.log
   • syslog

​

Related Links for Log Collectors

• Sumo document about using rsyslog
• Splunk document about creating a listener
• Logstash document about creating a TCP listener

​

Edit or Disable Remote Syslog

1. Go to Settings > Logging Services.
2. Under Remote Syslog, click Edit Profile.
3. If there is more than one profile, select the profile you want to edit.
4. Do one of the following:
   • Make the desired updates and click Save.
   • Click Disable and confirm disabling the profile. The profile is deleted and no longer available.

​

The Aviatrix Datadog Agent

• Datadog API key
• Datadog site you want to send to
• If only specific VPCs or VNets should forward logs to Datadog, have their names available

​

Configure the Aviatrix Datadog Agent

1. In Settings > Logging Services > Datadog Agent, click Enable.
2. Configure the following:

   Field	Description
   API Key	Enter your Datadog API key.
   Site	Select the Datadog site from the list.
   Export	Select Syslog & Metrics or Only Metrics.
   VPC/VNets Included	(Optional advanced setting) Select All or select specific VPC/VNets to forward logs to Datadog.

3. Click Enable. The agent details display on the Logging Services page.

​

Edit or Disable the Datadog Agent

1. Go to Settings > Logging Services.
2. Under Datadog Agent, click Edit.
3. Do one of the following:
   • Make the desired updates and click Save.
   • Click Disable and confirm disabling the agent. The agent configuration is deleted.
4. Click Save. The agent details display on the Logging Services page.

​

The Aviatrix CloudWatch Agent

1. Assign the required roles and policies in AWS
2. Enable the CloudWatch agent in Aviatrix at Settings > Logging Services > CloudWatch Agent

​

Assign Roles and Policies to CloudWatch

1. Log in to the collector’s AWS account, go to Identity and Access Management (IAM) > Roles and click Create role.
2. Under Trusted entity type, select AWS account.
3. Under An AWS account, select your current account or a new account and click Next.
4. Under Permissions policies, find and select the policy CloudWatchAgentServerPolicy and click Next.
   Click the checkbox, not the policy name.
5. Under Role details, enter the Role name aviatrix-role-cloudwatch. Do not change the role name.
6. Under Select trusted entities, click Edit and add trust relationships for all Aviatrix gateway AWS accounts.
7. Click Create role.
8. On the Roles page, click the name of the new role aviatrix-role-cloudwatch and copy the ARN value under Summary. Keep this ARN value as you will need it to configure another policy and to enable CloudWatch in Aviatrix Platform-as-a-Service (PaaS) console.
9. Return to the Roles page and click the role name aviatrix-role-ec2-<long-id>. Example: aviatrix-role-ec2-a-bb1cc222dd3e44f5g6hh7i88j99k0mm111
10. Under Permissions policies, click Add permissions > Attach policies.
11. On the Roles page > Permissions tab, click Add permissions > Create inline policy.
12. On Specify permissions, select the JSON editor.
13. In the Policy editor, enter (or select in the right pane) the following:
    • Sid: VisualEditor0
    • Effect: Allow
    • Action: sts:AssumeRole
    • Resource: arn:aws:iam:<your-account-number>:role/aviatrix-role-cloudwatch
    This is the ARN you previously copied. You will also need it when enabling CloudWatch in Aviatrix Platform-as-a-Service (PaaS) console. Example:
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "VisualEditor0",
          "Effect": "Allow",
          "Action": ["sts:AssumeRole"],
          "Resource": [
            "arn:aws:iam:<your-account-number>:role/aviatrix-role-cloudwatch"
          ]
        }
      ]
    }
    
    
14. Click Next.
15. Enter a policy name, review the policy, and click Create policy. The new permission policy displays on the aviatrix-role-ec2-<your-id> page.

​

Configure the CloudWatch Agent in Aviatrix

• CloudWatch role ARN
• AWS Cloud Type
• AWS Region
• Log group name (optional)
• VPC/VNets to forward logs (all, or specific names)

1. On Settings > Logging Services > CloudWatch Agent, click Enable.
2. Configure the following:

   Field	Description
   CloudWatch role ARN	This is the ARN for the AWS IAM role aviatrix-role-cloudwatch. It can be copied from the IAM > Roles > Summary page for that role.
   AWS Cloud Type	”Standard” is the only option.
   AWS Region	The region in which CloudWatch is configured.
   Log Group Name	(Optional) If you use log groups, enter the unique identifier for the CloudWatch log group in the selected AWS region.
   VPC/VNets Included	(Optional advanced setting) Select All or select specific VPC/VNets to forward logs to CloudWatch.

3. Click Enable. The agent details display on the Logging Services page.

​

Edit or Disable the CloudWatch Agent

1. In Aviatrix Platform-as-a-Service (PaaS) console, go to Settings > Logging Services.
2. Under CloudWatch Agent, click Edit.
3. Do one of the following:
   • Make the desired updates and click Save.
   • Click Disable and confirm disabling the agent. The agent configuration is deleted.