Deploying Aviatrix Secure Edge on Customer Managed VMware ESXi and KVM

This document provides instructions for deploying Aviatrix Secure Edge on VMware ESXi or on open-source Kernel-based Virtual Machine (KVM).

Aviatrix Secure Edge Network Connectivity

The following diagram shows an example of network connectivity for Aviatrix Edge Gateway to Transit Gateway in AWS.

650

Prerequisites

Aviatrix Secure Edge requires the following:

  • Aviatrix Controller 6.8 or later.

  • VMware ESXi

  • KVM

  • Aviatrix Transit Gateway BGP ASN configured. High-Performance Encryption (HPE) is optional for Edge Gateway attachments.

  • Access to Aviatrix Controller using the Internet or private network with DNS resolution from the Edge Gateway Management interface

  • BGP-enabled router to peer with Edge Gateway LAN interface via BGP over LAN

Downloading the Aviatrix Secure Edge Image File

Before you begin the deployment of the Edge Gateway, download the Aviatrix Secure Edge image file from the Aviatrix Support Portal. You will use the image file to deploy the Aviatrix Secure Edge virtual machine.

  1. Log in to the Aviatrix Support Portal: <https://support.aviatrix.com>_.

  2. From the top navigation menu, click on Downloads.

  3. Answer the questions that are presented, then click Download next to the image that you want.

    The Aviatrix Secure Edge image file downloads to your Downloads folder.

Aviatrix Secure Edge Installation Requirements

The following sections describe the virtual machine instance configuration, network interfaces, ports and protocols, and access requirements for the Aviatrix Edge Gateway to communicate with the Aviatrix Controller and the Aviatrix Transit Gateway.

Virtual Machine CPU and Memory Configurations

The following table provides CPU and memory configurations of the virtual machine instance supported for the Aviatrix Edge Gateway deployment.

Deployment Type Hardware Profile Storage Requirements Note

Small

2 vCPU - 4GB

64 GB

PoC / Test only

Medium

4 vCPU - 8GB

64 GB

<5Gbps throughput

Large

8 vCPU - 16GB

64 GB

~10Gbps throughput

X-Large

16 vCPU - 32GB

64 GB

~10Gbps throughput

Aviatrix recommends that you not change the Aviatrix Secure Edge VM resource allocation after deploying it. Aviatrix support may not be able to assist with any issue that occurs on a system with customized resource allocation.

Oversubscription of host resources can lead to a reduction of performance and your instance could become unstable. We recommend that you follow the guidelines and the best practices for your host hypervisor.

Aviatrix Secure Edge Networking and Ports and Protocols

650

Aviatrix Edge Gateway Network Interfaces

An Aviatrix Edge Gateway has three interfaces: one WAN interface on eth0, one LAN interface on eth1, and one Management interface on eth2.

Interface Description

WAN on eth0

Interface to connect to the Aviatrix Transit Gateway. Requires a default gateway and Layer 3 reachability to Transit Gateway Private or Public IP.

LAN on eth1

Interface to connect to the LAN network. Requires a BGP session with LAN Router.

Management on eth2

Interface to connect to the Aviatrix Controller. Requires a default gateway, DNS access and Internet access to Aviatrix Controller, Aviatrix software download, and tracelog upload.

Aviatrix Edge Gateway Ports and Protocols

The Aviatrix Edge Gateway requires outbound access to communicate with the Aviatrix Controller. You must allow access on these ports on your firewall.

  • MGMT: TCP 443 access to the Aviatrix Controller’s public IP address

  • MGMT: TCP 443 access to the Aviatrix Controller’s private IP address (only permit this access if you selected Management over Private Network for management IP connectivity)

  • WAN: UDP 500/4500

Additional required outbound ports are described in the table below.
Source Destination Port Purpose

WAN eth0

Aviatrix Transit Gateway eth0 private or public IP address.

UDP 500

IPsec

WAN eth0

Aviatrix Transit Gateway eth0 private or public IP address.

UDP 4500

IPsec

Mgmt eth2

DNS server

UDP 53

DNS lookup

Mgmt eth2

Aviatrix Controller FQDN or private or public IP address.

TCP 443

Edge to Controller

Mgmt eth2

Aviatrix CoPilot FQDN or private or public IP address.

UDP 5000

Syslog

Mgmt eth2

Aviatrix CoPilot FQDN or private or public IP address.

UDP 31283

Netflow

For additional Edge Gateway requirements, see Aviatrix Products: Required Access for External Sites.

Aviatrix Secure Edge Deployment Workflow

The diagram below provides a high-level view of the four-step process for deploying Aviatrix Secure Edge in Aviatrix Controller. You have the option to use either VMware ESXi or an open-source Kernel-based Virtual Machine (KVM) to deploy the Aviatrix Secure Edge VM and attach the ZTP .iso file.

600

  1. Create the Aviatrix Edge Gateway ZTP file.

  2. Deploy the Aviatrix Secure Edge virtual machine and attach the ZTP .iso file.

  3. Attach Aviatrix Edge Gateway to Transit Gateway.

  4. Connect Aviatrix Edge Gateway to external device.

Creating the Edge Gateway ZTP ISO Image File

You must have port 443 open to the IP address of the Aviatrix Controller. For the required access for Edge Gateway deployment, refer to Aviatrix Edge Gateway Ports and Protocols.

To create the Edge Gateway ISO image file, follow these steps.

  1. Log in to Aviatrix Controller 6.8 or later.

  2. Go to MULTI-CLOUD TRANSIT > Setup

  3. In the Launch an Aviatrix Spoke Gateway page, enter the following values:

    1. Cloud Type: Is always set to Aviatrix.

    2. ZTP File Type: Select iso.

      The ISO file is the equivalent of the Zero-Touch Provisioning (ZTP) token. ZTP allows network engineers to remotely deploy and provision network devices at remote locations. For KVM deployments, cloud-init is also supported.

      The following cloud-init modules have been verified for Edge Gateway deployments: users, write_files, and chpasswd.

    3. Gateway Name: Enter a name for the new Edge Gateway.

    4. Site ID: Select an existing Site ID or create a new Site ID by entering a name (such as, edge-01) and click Add item.

      For guidance on whether to select an existing Site ID or create a new one, see Edge Site ID Guidelines.

    5. Management Connection Type: Select Static.

    6. Management Connection Type: Select DHCP or Static, depending on your environment.

      Steps (f-n) are applicable only for static IP configuration on the management interface. For IP and DNS settings, enter using the applicable format. For example, if the Edge Gateway’s WAN IP is 10.1.1.151, enter 10.1.1.151/24 or what your netmask is.

    7. Management Interface IP/Mask: Enter the management interface IP/mask for the Aviatrix Secure Edge VM.

    8. Default Gateway IP: Enter the IP address of the Default Gateway for the Management Subnet.

    9. Primary DNS Server: Enter the DNS server IP address.

    10. Secondary DNS server: Enter the DNS server IP address, this field is optional.

    11. WAN Interface IP/Mask: Enter the interface IP/mask for the Aviatrix Secure Edge VM.

    12. WAN Default Gateway: Enter the IP address of the Edge Gateway WAN interface.

    13. Management Over Private Network: Check the box if the Edge Gateway management connection to the Aviatrix Controller is over a private network. Leave it unchecked if the connection is over the public internet.

    14. Management Egress IP CIDR: Enter the IP address of the Aviatrix Secure Edge VM visible to the Aviatrix Controller (IP address to be allowed in the Controller Security Group. This IP is optional and can be added later).

      This field adds a security bypass filter rule for the incoming traffic on TCP/443 to your Controller.

    15. LAN Interface IP/Mask: Enter the interface IP/mask for the Aviatrix Secure Edge VM.

    16. Active-Standby: Check the box for active-standby mode (see Active Standby Modes. Leave unchecked for Active-Active mode.

      The Active-Active and Active-Standby modes are configured when you create the first Edge ZTP for a particular Site ID. If you need to change a configuration from Active-Active to Active-Standby, delete all the Edge Gateway for that Site ID and recreate the Edge Gateway with the new setting.

  4. To create the ISO image file, click Create.

    edge-launch-spoke-gateway

    Aviatrix Controller prompts you to download the ISO file.

    Controller downloads the ZTP .iso file to your downloads folder.

  5. Log in to your ESXi or KVM host and upload the .iso file to a datastore or storage device.

    Controller displays a message that confirms when you have successfully downloaded the .iso file you created for the Edge Gateway. The .iso file will expire 24 hours after you create it, so you must mount the .iso file to an Aviatrix Secure Edge VM to complete the Edge Gateway registration within that timeframe, as you cannot download it again and will have to repeat the above steps.

Next, deploy the Aviatrix Secure Edge virtual machine and attach the ZTP .iso file.

Deploying the Aviatrix Secure Edge Virtual Machine and Attaching ZTP ISO File

Deploying the Aviatrix Secure Edge Virtual Machine in VMware ESXi

To deploy the Aviatrix Secure Edge virtual machine in VMware ESXi, follow these steps.

  1. Download the ESXi OVA file by using the link provided to you by Aviatrix Support. See Downloading the Aviatrix Secure Edge Image File.

  2. Log in to VMware vSphere Web client to access the ESXi host.

    You can use vSphere Web client to manage ESXi host, launch a VM, mount ISO files, and start and stop the Aviatrix Edge Gateway.

  3. To load the OVA file into the ESXi using vSphere, go to: ESXi > Virtual Machines > Create/Register VM.

  4. Select Deploy a virtual machine from an OVF or OVA file. Click Next.

  5. Enter a name for the Aviatrix Secure Edge VM and drag the OVA file into the blue pane. Click Next.

    edge_ova_load_file

  6. In the Select storage page, select the storage device for the instance you created (the OVA is installed in this instance). Click Next.

  7. In the Deployment options window, enter the network interface mappings and select the Deployment type. (Refer to the pull-down menu or see Virtual Machine CPU and Memory Configurations.)

    If necessary, you can change the network interface mappings after deployment.
    edge_ova_deploy_options

  8. Click Next.

  9. In the Ready to complete page, click Finish.

Next, attach the ZTP .iso and the Edge will auto-mount the media which contains the configuration file to be provisioned to the Edge.

Attaching the ISO Image to the Aviatrix Secure Edge Virtual Machine in VMware ESXi

  • The ZTP ISO file can only be used for a single Aviatrix Secure Edge VM instance, and only one time for that instance.

  • The ZTP token expires after 24 hours. If you wait too long to boot up the VM with the attached ISO image, it will not work. In that case, delete the Edge Gateway in the Controller UI and create a new Edge Gateway to receive a new ISO file.

  1. Upload the ISO file you downloaded from Aviatrix Controller to your VMware datastore.

  2. In vSphere, select the Aviatrix Secure Edge VM you created and click Edit settings.

  3. Select the Virtual Hardware tab.

  4. Next to CD/DVD Drive 1, click the down arrow and select Datastore ISO file from the pull-down menu.

  5. To load the ISO to the virtual CD drive, next to Status, check Connect at power on.

  6. Next to the CD/DVD Media field, click Browse. Select the ISO file you downloaded.

    edge_edit_settings

    Connect at power on (step 4) is required when you attach the ISO image to the VM for the first time. If the VM is powered on at the time you attach the ISO image, select the Datastore ISO file and save the configuration to make the ISO available to ZTP.

  7. Click Save.

Next, verify the Edge Gateway in Controller.

Deploying the Aviatrix Secure Edge Virtual Machine in KVM

Before you begin, on the KVM Linux host ensure the LAN, WAN, and MGMT network bridges are associated with the physical ethernet interfaces on the KVM sever. Refer to the KVM product documentation.

  1. Download the KVM QCOW2 file by using the link provided to you by Aviatrix Support. See Downloading the Aviatrix Secure Edge Image File.

  2. Launch Virtual Machine Manager UI to access the KVM host.

  3. Create a new virtual machine from an existing disk image.

    1. From File menu, select New virtual machine.

    2. Select the option Import existing disk image.

    3. Click Forward.

      edge-kvm-new-vm

  4. Provide the path to the KVM QCOW2 file and specify the operating system type and version.

    1. Enter the path or use the Browse button to locate the KVM QCOW2 file you previously downloaded.

    2. For OS type, select Linux.

    3. For Version, select Ubuntu 18.04 LTS.

    4. Click Forward.

      edge-kvm-new-vm-2

  5. Enter the memory and CPU settings for the Aviatrix Secure Edge VM and click Forward.

    edge-kvm-new-vm-3

  6. Enter a name for the Aviatrix Secure Edge VM and check the Customize configuration before install checkbox, then click Finish.

    edge-kvm-new-vm-4

  7. Add the LAN and MGMT virtual bridge interfaces.

    1. Click Add Hardware.

      edge-kvm-new-vm-5

    2. In Add New Virtual Hardware, select Network from the left pane and add two additional network interfaces for the LAN and MGMT virtual bridges. The virtual bridge for the WAN interface is automatically added as part of the VM image creation.

    3. For Network source, select the name of the virtual bridge for the LAN interface.

    4. For Device model, select virtio.

    5. Repeat steps a and b and add the virtual bridge for the MGMT interface.

      edge-kvm-new-vm-6

  8. Choose the storage device and attach the iso file to the VM.

    1. In Add New Virtual Hardware, select Storage from the left pane.

    2. Select the option Select or create custom storage.

    3. Click Manage.

    4. Locate and select the KVM iso file which you previously uploaded.

    5. Click Choose Volume.

    6. Click Finish.

      edge-kvm-new-vm-7

  9. Click Begin Installation to create the Aviatrix Secure Edge VM instance on the KVM host.

After you attach the ZTP .iso, the KVM hypervisor will auto-mount the media which contains the configuration file to provision the Edge Gateway.

For more information about deploying virtual machines and attaching .iso file in KVM, refer to KVM product documentation.

For network performance, see also Enabling Multiqueue virtio-net on KVM.

Next, verify the Edge Gateway in Controller.

KVM Hypervisor does not support configuration of RX/TX queue size during runtime. RX/TX queue size should be configured during Aviatrix Secure Edge VM bootup.

Verifying the Edge Gateway in Controller

To verify the Edge Gateway is up, wait for 5 minutes after you have attached the ZTP .iso file then do the following:

  1. In Aviatrix Controller, go to Multi-Cloud Transit > List > Spoke.

  2. In the State column, verify that the Edge Gateway you created is in the up state.

    Click the refresh button to update the registration status.

    edge-verify

    If the Edge Gateway status is not up, you can troubleshoot edge connectivity using CLI commands on the Edge Gateway console. See Troubleshooting Edge Gateway Connectivity.

Next, attach the Edge Gateway to the Transit Gateway.

Attaching the Edge Gateway to the Transit Gateway

For Edge Gateway attachment over a public network, you must update the WAN Public IP on the Edge Gateway and configure BGP ASN on the Edge Gateway before you attach Edge Gateway.

Update WAN Public IP

To update the WAN Public IP, follow these steps.

  1. In Aviatrix Controller, go to Gateway > Select a Spoke Gateway.

  2. Select the Edge Gateway you want to attach and click Edit.

  3. In IP Configurations, click Discover Public IP.

  4. Verify the WAN Public IP and click Update.

If you have multiple Edge Gateways, make sure each Edge Gateway has a unique WAN Public IP.

edge-ip-config

Configuring BGP ASN on the Edge Gateway

To configure BGP AS Number (ASN) on the Edge Gateway, follow these steps.

  1. In Aviatrix Controller, go to MULTI-CLOUD TRANSIT > Advanced Config > Edit Spoke.

  2. In the BGP Spoke Gateway pull-down menu, select the Edge Gateway you created and enter the Local AS Number for the Edge Gateway.

  3. Click CHANGE.

Next, attach the Edge Gateway to the Transit Gateway.

Attaching Edge Gateway to Transit Gateway

After you have configured the BGP ASNs on both the Transit and Edge Gateway, follow these steps to attach the Edge Gateway to the Transit Gateway.

To create an High Performance Encryption Mode attachment, make sure the Transit Gateway is created with High Performance Encryption Mode enabled.

If you want Jumbo Frame enabled on the Edge Gateway, make sure to enable Jumbo Frame on the Edge Gateway before you attach it to the Transit Gateway. See Jumbo Frame.

  1. In Aviatrix Controller, go to MULTI-CLOUD TRANSIT > List > Spoke. Confirm that the Edge Gateway you created is up.

  2. Navigate to MULTI-CLOUD TRANSIT > Setup > Attach / Detach > 1a Attach Spoke Gateway to Transit Network.

    edge-attach-spoke-to-transit

  3. In the Spoke Gateway/Source Gateway pull-down menu, select the Edge Gateway you created.

  4. In the Transit Gateway/NextHop Gateway pull-down menu, select your Transit Gateway.

  5. To connect over a private network, check Over Private Network box. Leave unchecked to connect using a public network.

  6. To configure Jumbo Frame on Edge Gateway, check Jumbo Frame box.

  7. To build High-Performance Encryption (HPE), check High Performance Encryption (HPE) Mode box. Leave unchecked if you do not require HPE.

    For High Performance Mode Tunnel Number, enter the number of HPE tunnels to create for High Performance Encryption Mode over the Internet or private network.

  8. Click ATTACH.

  9. Verify the Edge Gateway attachment in the following ways:

    • From Controller: Navigate to Multi-Cloud Transit > List > Spoke

    • From CoPilot: Navigate to Topology > Network Graph > Network.

Next, connect the Edge Gateway to the external device.

Connecting the Edge Gateway to an External Device (BGP over LAN)

To connect the Edge Gateway to LAN Routing using BGP over LAN, follow these steps.

  1. Go to MULTI-CLOUD TRANSIT > Setup > External Connection.

  2. In Connect to VGW/External Device/Azure VNG, enter the following values:

    1. Select these options: External Device, BGP, and LAN.

    2. VPC Name/Site ID: Select an existing Edge Site ID from the drop-down list.

    3. Connection Name: Enter a unique name to identify the connection to the LAN router.

    4. Aviatrix Gateway BGP ASN: Enter the BGP AS number the Edge Gateway will use to exchange routes with the LAN router.

    5. Primary Aviatrix Gateway: Select the Edge Gateway you created.

    6. Remote BGP AS Number: Enter the BGP AS number configured on the LAN router.

    7. Remote LAN IP: Enter the LAN router IP address for BGP peering.

    8. Local LAN IP: Enter the Edge LAN interface IP address for BGP peering.

      edge-connect-external-device

  3. Click CONNECT.

Enabling Multiqueue virtio-net on KVM

Multiqueue virtio-net allows network performance to scale with the number of vCPUs, by allowing packet processing (packet sending and receiving) through multiple TX and RX queues.

To enable Multiqueue virtio-net support on KVM, when launching the Edge Gateway VM using virt-install, add the driver_queues parameter to the network interface details.

--network bridge=<bridge-name>, model=virtio,driver_queues=N

where, N is the number of vCPUs.

Selective Gateway Upgrade for Aviatrix Secure Edge

The Aviatrix Secure Edge base OS is not upgradeable. To update the base OS to a newer version, you need to deploy the latest version of the Aviatrix Secure Edge image to a new VM.

As Aviatrix Secure Edge base OS is not field upgradeable, Aviatrix Secure Edge does not support selective gateway image update and software rollback.

Troubleshooting Edge Gateway Connectivity

You can use the Clish commands below to troubleshoot the Edge Gateway.

Once the Edge Gateway registers with the Aviatrix Controller, the password changes to the WAN IP address of the Edge Gateway.
Command Description

change_console_password

Changes the password for the CLI login.

check_conduit

Check conduit state.

check_network [dns][reachability]

Troubleshoot network connectivity.

diagnostics

Show gateway diagnostics from /home/ubuntu/cloudx-aws/avx_edge_status.json, which is written by register process or reset_config process.

logout

Log out of the console.

ping [-c count] [dest]

Ping destination, optional parameter ping packet count. The default is 5.

reboot

Reboot the system.

set_controller_ip [controller_ip]

Set the Controller IP address, usually performed after Controller migration when the Controller IP address is changed.

show_interfaces

Show output from the command “ifconfig -a

more”.

show_routes