> ## Documentation Index > Fetch the complete documentation index at: https://docs.aviatrix.com/llms.txt > Use this file to discover all available pages before exploring further. # Overview > Aviatrix Gateway Logging Aviatrix Platform-as-a-Service (PaaS) collects log data from Aviatrix managed gateways. You can forward the logs by remote syslog to a log server. In the *Settings > Logging Services* page, you can configure the forwarding of logs from Aviatrix Platform-as-a-Service (PaaS) to remote syslog (rsyslog) and the log servers of well-known log management systems. ## Overview of Gateway Logging The Aviatrix managed gateways can be configured to forward their logs to a syslog log management system. All managed gateways will forward the logs directly to the logging server. Therefore, each managed gateway needs network connectivity to the logging server. Aviatrix supports using remote syslog (rsyslog) for forwarding log messages. Remote syslog as the log forwarder is both efficient and the industry standard. Most log collectors support rsyslog as a log forwarder. In the Aviatrix Platform-as-a-Service (PaaS) console, you can enable log data collection from Aviatrix gateways to the following log server: * [Datadog Agent](#the-aviatrix-datadog-agent) * [AWS CloudWatch Agent](#the-aviatrix-cloudwatch-agent) In addition to standard information on syslog, Aviatrix logs also provide informational insights into security rule violation statistics, gateway stats, and FQDN filter violations. Your log management system can sift through the Aviatrix logs to get meaningful trend charts that can help monitor aspects of your network. See [Aviatrix Log Formats](/docs/cloud/platform-administration/manage/log-formats) for a list of useful Aviatrix logs that can be parsed on the log management system to display relevant analytics of data collected from Aviatrix gateways. The process the Aviatrix gateways use for exporting their log files is as follows: * Aviatrix gateways stream the log lines being written to the *syslog* and *auth.log* files. * When you use the default rsyslog server configuration, the logs streamed from the gateways have multiple files. * Each file is named with the application that generated the log. For example, all logs generated by the *avx-gw-state-sync* application are re-directed to a file named *avx-gw-state-sync*" on the log server. ### Log Format The log format is shown below. Your syslog collectors and any related automation for ingesting logs must accept the log format. ``` Format: [Timestamp] GW-[Name of Gateway]-[Public IP of Gateway] [Name of Application generating log][Application Process ID]: [Log message] ``` *Example of log format:* ``` Mar 23 19:17:50 GW-UdpGateway-50.17.41.173 avx-gw-state-sync[11249]: warn#011gateway_launcher/gateway_launcher.go:212#011daemon exited ``` Prefix of log: \[Timestamp] GW-\[Name of Gateway]-\[Public IP of Gateway] Example prefix: `Mar 23 19:17:50 GW-gg-aws-usw2-s127-35.162.124.66` ## The Aviatrix Remote Syslog Forwarder If you want to forward syslog data collected on Aviatrix gateways, the Aviatrix Platform-as-a-Service (PaaS) console offers a remote syslog forwarder for sending syslog data from Aviatrix gateways to your designated remote syslog server. You can configure the Aviatrix Remote Syslog forwarder in *Settings > Logging Services > Remote Syslog*. Before configuring the Aviatrix Remote Syslog forwarder, collect the following information: * FQDN or IP address of the remote syslog server * Port number and protocol type of the listening port of the remote syslog server * Certificate Authority (CA) certificate for the server * Public client certificate signed by the same CA * Private client key that pairs with the public certificate Aviatrix Platform-as-a-Service (PaaS) expects certificates in PEM format. Convert any certificates downloaded from your external logging server's documentation into PEM format. Attempting to upload the wrong format could return an Exception Error. ### Syslog Profiles You configure profiles for remote syslog to define the remote server where logs are sent, the protocol and port for access, certificates (optional), and from which VPCs or VNets to collect logs. A total of 9 profiles are supported for remote syslog. Newly deployed gateways are added to a profile if it is the only profile enabled. If more than one profile is enabled, the newly deployed gateway will not be added to any profile. You can use the advanced options in the logging "Edit Options" window to edit the exclude and include list. Configure the following: | Field | Description | | - | - | | Profile Name | Enter a unique descriptive name for the profile | | Server | FQDN or IP address of the remote syslog server | | Protocol | TCP or UDP | | Port | Listening port of the remote syslog server | | Server CA Certificate | Certificate Authority (CA) certificate for the syslog server | | Client Certificate | Public certificate for Aviatrix Platform-as-a-Service (PaaS), signed by the same CA as the server | | Client Private Key | Private encryption key that pairs with the public certificate | | Custom Template | (Optional) Define a custom format for syslog messages, which can be useful when forwarding to third party servers like Datadog | | VPC/VNets included | Select the VPC/VNets that should forward logs to remote syslog | ## Configure Remote Syslog 1. Go to *Settings > Logging Services*. 2. Under Remote Syslog, click *+ Profile*. 3. Configure the following: | Field | Description | | - | - | | Profile Name | Enter a unique descriptive name for the profile | | Server | FQDN or IP address of the remote syslog server | | Protocol | TCP or UDP | | Port | Listening port of the remote syslog server | | Server CA Certificate | Certificate Authority (CA) certificate for the syslog server | | Client Certificate | Public certificate for Aviatrix Platform-as-a-Service (PaaS), signed by the same CA as the server | | Client Private Key | Private encryption key that pairs with the public certificate | | Custom Template | (Optional) Define a custom format for syslog messages, which can be useful when forwarding to third party servers like Datadog | | VPC/VNets included | Select the VPC/VNets that should forward logs to remote syslog | 4. Click *Enable*. The agent details display on the Logging Services page. ### Configure Remote Syslog on the Remote Syslog Server Configure the remote server to receive data from Aviatrix Platform-as-a-Service (PaaS). On the remote syslog (rsyslog) server: 1. Install rsyslog and rsyslog-gnutls packages. 2. Create a new configuration file in */etc/rsyslog.d* to enable the TLS connection. The file should contain content similar to what is shown in the example below, depending on your rsyslog version. Make sure key paths are readable by the syslog user. Remote Syslog version 8 or later: ```none theme={null} global( DefaultNetstreamDriver="gtls" DefaultNetstreamDriverCAFile="/etc/cert/rsyslog-ca.pem" DefaultNetstreamDriverCertFile="/etc/cert/rsyslog-crt.pem" DefaultNetstreamDriverKeyFile="/etc/cert/rsyslog-key.pem" ) template(name="TmplMsg" type="list") { constant(value="/var/log/aviatrix/") property(name="hostname") constant(value="/") property(name="programname" SecurePath="replace") constant(value="") } ruleset(name="remote"){ *.info;mail.none;authpriv.*;cron.none action(type="omfile" DynaFile="TmplMsg") } module( load="imtcp" StreamDriver.Name="gtls" StreamDriver.Mode="1" StreamDriver.Authmode="anon" ) input(type="imtcp" port="514" ruleset="remote") ``` Remote Syslog version 7 or earlier: ```none theme={null} $ModLoad imtcp $InputTCPServerRun 514 $DefaultNetstreamDriver gtls #Certificate location $DefaultNetstreamDriverCAFile /etc/cert/rsyslog-ca.pem $DefaultNetstreamDriverCertFile /etc/cert/rsyslog-crt.pem $DefaultNetstreamDriverKeyFile /etc/cert/rsyslog-key.pem $InputTCPServerStreamDriverAuthMode x509/certvalid $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode # Re-direct logs to host specific directories $template TmplMsg, "/var/log/aviatrix/%HOSTNAME%/%PROGRAMNAME%" *.info,mail.none,authpriv.*,cron.none ?TmplMsg & ~ ``` 3. Make sure the output directory */var/log* is writable by rsyslog user/daemon. 4. Restart rsyslog service and verify that the port is listening and there are no errors in */var/log/syslog*. 5. Confirm the port is allowed in the security group/firewall for incoming traffic. 6. Go to the */var/log/aviatrix* directory. 7. Find the directory of the desired gateway. * The gateway directory name is in a format of GW-gateway\_name-public\_IP\_of\_gateway 8. Each gateway directory should have: * auth.log * syslog ## Related Links for Log Collectors Aviatrix supports using Remote Syslog (rsyslog) for forwarding log messages. Most log collectors support rsyslog as a log forwarder. The following are links related to log collectors and rsyslog: * [Sumo document about using rsyslog](https://help.sumologic.com/03Send-Data/Sources/02Sources-for-Hosted-Collectors/Cloud-Syslog-Source) * [Splunk document about creating a listener](https://docs.splunk.com/Documentation/Splunk/latest/Data/Monitornetworkports) * [Logstash document about creating a TCP listener](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-tcp.html) ## Edit or Disable Remote Syslog You can edit an existing profile. You can also disable a profile. When a profile is disabled, the configuration is deleted and no longer available. 1. Go to *Settings > Logging Services*. 2. Under Remote Syslog, click *Edit Profile*. 3. If there is more than one profile, select the profile you want to edit. 4. Do one of the following: * Make the desired updates and click *Save*. * Click *Disable* and confirm disabling the profile. The profile is deleted and no longer available. ## The Aviatrix Datadog Agent If you use Datadog, the Aviatrix Platform-as-a-Service (PaaS) console offers a Datadog agent for sending system metrics from Aviatrix gateways to your configured Datadog instance. You can configure the Aviatrix Datadog agent in *Settings > Logging Services > Datadog Agent*. Before configuring the Aviatrix Datadog agent, collect the following information: * Datadog API key * Datadog site you want to send to * If only specific VPCs or VNets should forward logs to Datadog, have their names available ### Configure the Aviatrix Datadog Agent If you use Datadog, you can configure the Aviatrix Remote Syslog forwarder to send syslog data from Aviatrix gateways to your configured Datadog instance. 1. In *Settings > Logging Services > Datadog Agent*, click *Enable*. 2. Configure the following: | Field | Description | | - | - | | API Key | Enter your Datadog API key. | | Site | Select the Datadog site from the list. | | Export | Select Syslog & Metrics or Only Metrics. | | VPC/VNets Included | (Optional advanced setting) Select All or select specific VPC/VNets to forward logs to Datadog. | 3. Click *Enable*. The agent details display on the Logging Services page. *Related Topics* [Datadog documentation: Using Rsyslog to send logs to Datadog](https://www.datadoghq.com/architecture/using-rsyslog-to-send-logs-to-datadog/) ## Edit or Disable the Datadog Agent You can edit an existing agent configuration. You can also disable a logging agent. When an agent is disabled, the configuration is deleted and no longer available. 1. Go to *Settings > Logging Services*. 2. Under Datadog Agent, click *Edit*. 3. Do one of the following: * Make the desired updates and click *Save*. * Click *Disable* and confirm disabling the agent. The agent configuration is deleted. 4. Click *Save*. The agent details display on the Logging Services page. ## The Aviatrix CloudWatch Agent If you use CloudWatch, the Aviatrix Platform-as-a-Service (PaaS) console offers a CloudWatch agent for sending syslog from Aviatrix gateways to your configured AWS CloudWatch instance. Only gateways on AWS cloud accounts onboarded using CloudFormation are supported. AWS gateways created from an access account with an AWS secret key and access key are not supported. Configuring the CloudWatch agent is a two-step process. You must do the following: 1. [Assign the required roles and policies in AWS](#assign-roles-and-policies-to-cloudwatch) 2. [Enable the CloudWatch agent in Aviatrix](#configure-the-cloudwatch-agent-in-aviatrix) at *Settings > Logging Services > CloudWatch Agent* ### Assign Roles and Policies to CloudWatch You can configure Aviatrix gateways in different AWS accounts to send and update logs to the collector's AWS account. Follow the instructions below to set up an IAM role and policies *on the collector's AWS account*. Logs from CloudWatch can be exported to S3 buckets. See the [AWS Documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3Export.html). 1. Log in to the collector's AWS account, go to *Identity and Access Management (IAM) > Roles* and click *Create role*. create role in aws 2. Under *Trusted entity type*, select *AWS account*. 3. Under *An AWS account*, select your current account or a new account and click *Next*. 4. Under *Permissions policies*, find and select the policy *CloudWatchAgentServerPolicy* and click *Next*. Click the checkbox, not the policy name. 5. Under *Role details*, enter the *Role name* `aviatrix-role-cloudwatch`. Do not change the role name. 6. Under *Select trusted entities*, click *Edit* and add trust relationships for all Aviatrix gateway AWS accounts. trust relationship syntax 7. Click *Create role*. 8. On the Roles page, click the name of the new role *aviatrix-role-cloudwatch* and copy the ARN value under Summary. Keep this ARN value as you will need it to configure another policy and to enable CloudWatch in Aviatrix Platform-as-a-Service (PaaS) console. 9. Return to the *Roles* page and click the role name `aviatrix-role-ec2-`. Example: aviatrix-role-ec2-a-bb1cc222dd3e44f5g6hh7i88j99k0mm111 10. Under *Permissions policies*, click *Add permissions > Attach policies*. 11. On the Roles page > Permissions tab, click *Add permissions > Create inline policy*. 12. On *Specify permissions*, select the *JSON* editor. 13. In the *Policy editor*, enter (or select in the right pane) the following: * Sid: VisualEditor0 * Effect: Allow * Action: sts:AssumeRole * Resource: arn:aws:iam:\:role/aviatrix-role-cloudwatch This is the ARN you previously copied. You will also need it when enabling CloudWatch in Aviatrix Platform-as-a-Service (PaaS) console. Example: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": ["sts:AssumeRole"], "Resource": [ "arn:aws:iam::role/aviatrix-role-cloudwatch" ] } ] } ``` policy editor with required content added 14. Click *Next*. 15. Enter a policy name, review the policy, and click *Create policy*. The new permission policy displays on the *aviatrix-role-ec2-\* page. *Related Topics* [AWS documentation: Application-level CloudWatch configuration](https://docs.aws.amazon.com/prescriptive-guidance/latest/implementing-logging-monitoring-cloudwatch/application-level-configuration.html) ### Configure the CloudWatch Agent in Aviatrix If you use AWS CloudWatch, you can configure the Aviatrix Remote Syslog forwarder to send syslog data from Aviatrix gateways to AWS CloudWatch. *Prerequisites* You must have configured the CloudWatch agent in AWS, including [assigning roles and policies required by Aviatrix Platform-as-a-Service (PaaS)](#assign-roles-and-policies-to-cloudwatch). Before configuring the Aviatrix CloudWatch agent, collect the following information: * CloudWatch role ARN * AWS Cloud Type * AWS Region * Log group name (optional) * VPC/VNets to forward logs (all, or specific names) Perform the following steps in Aviatrix Platform-as-a-Service (PaaS) console: 1. On *Settings > Logging Services > CloudWatch Agent*, click *Enable*. 2. Configure the following: | Field | Description | | - | - | | CloudWatch role ARN | This is the ARN for the AWS IAM role *aviatrix-role-cloudwatch*. It can be copied from the IAM > Roles > Summary page for that role. | | AWS Cloud Type | "Standard" is the only option. | | AWS Region | The region in which CloudWatch is configured. | | Log Group Name | (Optional) If you use log groups, enter the unique identifier for the CloudWatch log group in the selected AWS region. | | VPC/VNets Included | (Optional advanced setting) Select All or select specific VPC/VNets to forward logs to CloudWatch. | 3. Click *Enable*. The agent details display on the Logging Services page. ## Edit or Disable the CloudWatch Agent You can edit or disable an existing agent configuration in Aviatrix Platform-as-a-Service (PaaS). When an agent is disabled, the configuration is deleted and no longer available. 1. In Aviatrix Platform-as-a-Service (PaaS) console, go to *Settings > Logging Services*. 2. Under *CloudWatch Agent*, click *Edit*. 3. Do one of the following: * Make the desired updates and click *Save*. * Click *Disable* and confirm disabling the agent. The agent configuration is deleted. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.