Troubleshooting Your Multicloud Network

There are several ways to troubleshoot and debug errors within Aviatrix. All of these are performed in the Aviatrix Controller from the Troubleshoot menu.

Upload tracelog

UnderTroubleshoot, click Logs and select a gateway at Upload Tracelog. The Controller and gateway tracelog will be uploaded to Aviatrix. The Aviatrix support team will be alerted. If no gateway is selected, only the Controller log is uploaded.

Run diagnostics on a gateway

  1. Under Troubleshoot, click Diagnostics > Gateway. In the Diagnostics area, select a gateway to run diagnostics.

  2. Click Run.

  3. After that operation finishes, click Show to display on the console. The diagnostics test if the gateway is reachable and its services are up and running.

Please refer to the Service Description of Diagnostic Result.

If you could not determine the root cause based on the diagnostics, click Submit in the Gateway tab > Diagnostics area to send the diagnostics result to Aviatrix support team.

Debug peering tunnel status

Click Peering on the console. click Diag on each peer pair and run various tests.

Debug Site2Cloud tunnel status

In the Aviatrix Controller, click Site2Cloud > Diagnostics. Fill out the necessary fields and click OK.

Debug gateway connectivity

To test if a gateway can reach a certain IP or host:

  1. Click Troubleshoot > Diagnostics > Network.

  2. At the Network Connectivity Utility panel, select a gateway.

  3. Specify the remote host name, port number. The TCP protocol test is reliable. Currently UDP test is not reliable.

Network Traceroute

You can run a traceroute function from a selected Aviatrix gateway to test reachability from this gateway to any destination.

  1. Go to Troubleshoot > Diagnostics > Network.

  2. Scroll down to Gateway Utility.

  3. Enter a destination IP or host name and select a gateway and click Trace Route. The Trace Route results should be displayed when the execution finishes.

You can launch an Aviatrix gateway in a specific VPC and public subnet and use it as an EC2 instance to test connectivity to a destination host or IP address. For example, launch an Aviatrix gateway in a Spoke VPC (where the Spoke VPC gateway is launched from the Transit Network Workflow. When you select this test gateway for Trace Route testing, you are effectively testing connectivity going from an EC2 > Spoke VPC GW > Transit GW > VGW > on-prem network.

Packet capture

  1. Click Troubleshoot > Diagnostics > Network.

  2. At the Packet Capture panel, select a gateway where you wish to do packet capture. You can further filter on Host and Port number.

  3. Click Start to start the capture.

  4. Click Stop to stop the capture, then click Download to download the pcap file. You can also specify capture time. The pcap file can be viewed in Wireshark.

DNS Error

If you see a DNS related error on the controller console, check your VPC/VNet DNS setting. It is possible that the controller or gateway does not have connectivity to the DNS server.

If your DNS server is located on-prem, make sure the VPC/VNet where controller is launched has connection to reach the private DNS server.

Checking the Spire Agent Certificate

  1. Ensure that the gateway instance where the certificate is located is running.

  2. Log in to the Aviatrix Controller.

  3. Navigate to Troubleshoot > Diagnostics > Gateway > Service Actions.

  4. In the Gateway field, select a gateway.

  5. In the Services field, select PKI.

  6. In the Actions field, select restart.

  7. Click OK to confirm.

  8. Wait for 30 seconds.

  9. In the Actions field, select status and click OK.

  10. In the Show Results area below, make sure the service is active and stably running (look for the line "Active: active (running) since").

  11. If the result indicates the service is active (running) for more than 30 seconds, the problem is solved.

If you have any problems following these steps, please contact https://support.aviatrix.com.

Recovering an Expired or Unattested Agent

  1. Log in to the Aviatrix Controller.

  2. Navigate to Settings > Maintenance > Upgrade > Selective Gateway Upgrade.

  3. Select all gateways.

  4. Click Image Upgrade (note that a maintenance window might be needed because this operation may affect the data-plane).

  5. After the gateway image upgrade is completed, navigate to Troubleshoot > Diagnostics > Gateway > Service Actions.

  6. In the Gateway field, select a gateway.

  7. In the Services field, select PKI.

  8. In the Actions field, select status and click OK.

  9. In the Show Results area below, make sure the service is active and stably running (look for the line "Active: active (running) since").

  10. If the result indicates the service is active (running) for more than 30 seconds, the gateway is fixed.

If you have any problems following these steps, please contact https://support.aviatrix.com.