Deploying Aviatrix Edge 2.0¶
This document provides instructions for deploying Aviatrix Edge 2.0 on VMware ESXi or on open-source Kernel-based Virtual Machine (KVM).
For additional information about Aviatrix Edge 2.0, see Aviatrix Edge FAQ.
For examples of Edge 2.0 design patterns, see Aviatrix Edge Design Patterns.
For the Aviatrix Edge 1.0 for ESXi workflow, see Deploying Aviatrix Edge 1.0 for VMware ESXi.
Aviatrix Edge Network Connectivity¶
The following diagram shows an example of network connectivity for Edge Gateway to Transit Gateway in AWS.
Aviatrix Edge 2.0 requires the following:
- Aviatrix Controller 6.8
- VMware ESXi
OVA image for VMware ESXi (see Requesting Aviatrix Edge Gateway Image File).
VMware ESXi 6.7 or 7.0.1
Sufficient VMware ESXi resources to run Edge Gateway (see Aviatrix Edge 2.0 Installation Requirements).
(Optional) VMware vCenter Server
For information about installing VMware products, refer to the VMware product documentation.
QCOW2 image for KVM (see Requesting Aviatrix Edge Gateway Image File).
KVM server running in Linux Bare Metal Server
QEMU Version 1.5.3, Release 160.el7_6.3
Sufficient KVM resources to run Edge Gateway (see Aviatrix Edge 2.0 Installation Requirements).
For information about installing KVM products, refer to KVM product documentation.
- Aviatrix Transit Gateway BGP ASN configured. High-Performance Encryption (HPE) is optional for Edge 2.0 attachments.
- Access to Aviatrix Controller using the Internet or private network with DNS resolution from the Edge Gateway Management interface
- BGP-enabled router to peer with Edge Gateway LAN interface via BGP over LAN
Requesting an Aviatrix Edge Gateway Image File¶
Before you begin the deployment of the Edge Gateway, submit a request to Aviatrix Support for a link to the Edge Gateway image file. You will use the image file to deploy the Edge virtual machine.
Log in to the Aviatrix Support Portal: https://support.aviatrix.com.
Select Submit a request.
For Subject, enter Requesting access to Edge image.
For Edge Location, enter the physical address of the location where you will install the Edge VM(s), such as a data center, headend, co-location site, or office. If you are installing Edge VMs at more than one location, provide the following information for each physical location:
- Physical Address (Do not enter a P.O.Box.)
- State or Locality
- Zip Code or Postal Code
For Type of VM, enter OVA for VMware ESXi or QCOW2 for KVM.
Click Submit. Aviatrix Support will respond with a link you can use to download the Edge Gateway image file.
Aviatrix Edge 2.0 Installation Requirements¶
The following sections describe the virtual machine instance configuration, network interfaces, ports and protocols, and access requirements for the Edge Gateway to communicate with the Aviatrix Controller and the Aviatrix Transit Gateway.
Virtual Machine CPU and Memory Configurations¶
The following table provides CPU and memory configurations of the virtual machine instance supported for the Aviatrix Edge Gateway deployment.
|Deployment Type||Hardware Profile||Storage Requirements||Note|
|Small||2 vCPU - 4GB||64 GB||PoC / Test only|
|Medium||4 vCPU - 8GB||64 GB||<5Gbps throughput|
|Large||8 vCPU - 16GB||64 GB||~10Gbps throughput|
|X-Large||16 vCPU - 32GB||64 GB||~10Gbps throughput|
We recommend that you not change the Edge VM resource allocation after deploying it. Aviatrix support may not be able to assist with any issue that occurs on a system with customized resource allocation.
Oversubscription of host resources can lead to a reduction of performance and your instance could become unstable. We recommend that you follow the guidelines and the best practices for your host hypervisor.
Aviatrix Edge Networking and Ports and Protocols¶
The following sections describe the Edge network interfaces, port, and protocols.
Aviatrix Edge Network Interfaces¶
|WAN eth0||Interface to connect to the Aviatrix Transit Gateway. Requires a default gateway and Layer 3 reachability to Transit Gateway Private or Public IP.|
|LAN eth1||Interface to connect to the LAN network. Requires a BGP session with LAN Router.|
|Management eth2||Interface to connect to the Aviatrix Controller. Requires a default gateway, DNS access and Internet access to Aviatrix Controller, Aviatrix software download, and tracelog upload.|
Aviatrix Edge Ports and Protocols¶
The Aviatrix Controller requires access to the following ports for Edge Gateway deployment. You must allow access on these ports on your firewall.
- MGMT: TCP 443 access to the Aviatrix Controller’s public IP address
- MGMT: TCP 443 access to the Aviatrix Controller’s private IP address (only permit this access if you selected Management over Private Network for management IP connectivity)
- WAN: UDP 500/4500
|WAN eth0||Aviatrix Transit Gateway eth0 private or public IP address.||UDP||500||IPsec|
|WAN eth0||Aviatrix Transit Gateway eth0 private or public IP address.||UDP||4500||IPsec|
|Mgmt eth2||DNS server||UDP||53||DNS lookup|
|Mgmt eth2||Aviatrix Controller FQDN or public IP address. controller.aviatrixnetwork.com spire-server.aviatrixnetwork.com time-server.aviatrixnetwork.com||TCP||443||Edge to Controller|
Aviatrix Edge 2.0 Deployment Workflow¶
The diagram below provides a high-level view of the four-step process for deploying Aviatrix Edge 2.0 in Aviatrix Controller. You have the option to use either VMware ESXi or an open-source Kernel-based Virtual Machine (KVM) to deploy the Edge VM and attach the ZTP .iso file.
1. Create Edge Gateway ZTP ISO Image File¶
You must have port 443 open to the IP address of the Aviatrix Controller. For the required access for Edge Gateway deployment, refer to Aviatrix Edge Ports and Protocols.
To create the Edge Gateway ISO image file, follow these steps.
Log in to Aviatrix Controller 6.8.
Go to MULTI-CLOUD TRANSIT > Setup
In the Launch an Aviatrix Spoke Gateway page, enter the following values:
Cloud Type: Is always set to Aviatrix.
ZTP File Type: Select iso.
The ISO file is the equivalent of the Zero-Touch Provisioning (ZTP) token. ZTP allows network engineers to remotely deploy and provision network devices at remote locations. For KVM deployments, cloud-init is also supported.
Gateway Name: Enter a name for the new Edge Gateway.
Site ID: Select an existing Site ID or create a new Site ID by entering a name (such as, edge-01) and click Add item.
For guidance on whether to select an existing Site ID or create a new one, see Edge Site ID Guidelines.
- Management Connection Type: Select DHCP or Static, depending on your environment.
Steps (f-n) are applicable only for static IP configuration on the management interface. For IP and DNS settings, enter using the applicable format. For example, if the Edge Gateway’s WAN IP is 10.1.1.151, enter 10.1.1.151/24 or what your netmask is.
- Management Interface IP/Mask: Enter the management interface IP/mask for the Edge VM.
- Default Gateway IP: Enter the IP address of the Default Gateway for the Management Subnet.
- Primary DNS Server: Enter the DNS server IP address.
- Secondary DNS server: Enter the DNS server IP address, this field is optional.
- WAN Interface IP/Mask: Enter the interface IP/mask for the Edge VM.
WAN Default Gateway: Enter the IP address of the Edge WAN interface.
Management Over Private Network: Check the box if the Edge management connection to the Aviatrix Controller is over a private network. Leave it unchecked if the connection is over the public internet.
Management Egress IP CIDR: Enter the IP address of the Edge VM visible to the Aviatrix Controller (IP address to be allowed in the Controller Security Group. This IP is optional and can be added later).
This field adds a security bypass filter rule for the incoming traffic on TCP/443 to your Controller.
LAN Interface IP/Mask: Enter the interface IP/mask for the Edge VM.
- Active-Standby: Check the box for active-standby mode (see Active-Standby Edge.) Leave unchecked for Active-Active mode.
The Active-Active and Active-Standby modes are configured when you create the first Edge ZTP for a particular Site ID. If you need to change a configuration from Active-Active to Active-Standby, delete all the Edge Gateway for that Site ID and recreate the Edge Gateway with the new setting.
To create the ISO image file, click Create. Aviatrix Controller prompts you to download the ISO file.
Controller downloads the ZTP .iso file to your downloads folder.
Log in to your ESXi or KVM host and upload the .iso file to a datastore or storage device.
Controller displays a message that confirms when you have successfully downloaded the .iso file you created for the Edge gateway. The .iso file will expire 24 hours after you create it, so you must mount the .iso file to an Edge VM to complete the Edge gateway registration within that timeframe, as you cannot download it again and will have to repeat the above steps.
Next, deploy the Edge virtual machine and attach the ZTP .iso file in the VMware or KVM environment. See `Deploy Edge Virtual Machine and Attach ZTP ISO File`_.
Edge Site ID Guidelines¶
Aviatrix Edge 2.0 uses Site ID to identify an Edge location and Edge Gateway pair. This allows to group multiple Edge Gateways at the same Edge location using the same Site ID. Multiple Edge Gateways can be grouped and deployed in Active-Active or Active-Standby mode.
Follow these guidelines to decide whether to use an existing Site ID or create a new one.
- Use an existing Site ID if:
- You want to have Active-Standby on 2 Edge Gateways (assign the same Site ID).
- You want to have ECMP on multiple Edge Gateways (assign the same Site ID).
- Edge Gateways with the same Site ID:
- Can only join the same domain.
- Can have the same or different local ASN.
- Need to have FireNet traffic inspection configured per site.
- If you want to configure FireNet management on the Edge Gateway, you need to configure it per site.
- When multiple Edge Gateways are attached to a common transit, the transit will propagate routes from Edge Gateways with the same Site ID to other Edge Gateways with a different Site ID but will not propagate routes from the Edge Gateways to other Edge Gateways with the same Site ID.
2. Deploy Edge Virtual Machine and Attach ZTP ISO File¶
To deploy the Edge virtual machine on KVM, skip to step 2c. Deploying the Edge Virtual Machine in KVM.
2a. Deploying the Edge Virtual Machine in VMware ESXi¶
To deploy the Edge virtual machine in VMware ESXi, follow these steps.
Download the ESXi OVA file by using the link provided to you by Aviatrix Support. See Requesting an Aviatrix Edge Gateway Image File.
Log in to VMware vSphere Web client to access the ESXi host.
You can use vSphere Web client to manage ESXi host, launch a VM, mount ISO files, and start and stop the Aviatrix Edge Gateway.
To load the OVA file into the ESXi using vSphere, go to: ESXi > Virtual Machines > Create/Register VM.
Select Deploy a virtual machine from an OVF or OVA file. Click Next.
Enter a name for the Edge VM and drag the OVA file into the blue pane. Click Next.
In the Select storage page, select the storage device for the instance you created (the OVA is installed in this instance). Click Next.
In the Deployment options window, enter the network interface mappings and select the Deployment type. (Refer to the pull-down menu or see Virtual Machine CPU and Memory Configurations.)
If necessary, you can change the network interface mappings after deployment.
In the Ready to complete page, click Finish.
Next, attach the ZTP .iso and the Edge will auto-mount the media which contains the configuration file to be provisioned to the Edge.
2b. Attaching the ISO Image to the Edge Virtual Machine in VMware ESXi¶
- The ZTP ISO file can only be used for a single Edge VM instance, and only one time for that instance.
- The ZTP token expires after 24 hours. If you wait too long to boot up the VM with the attached ISO image, it will not work. In that case, delete the Edge Gateway in the Controller UI and create a new Edge Gateway to receive a new ISO file.
Upload the ISO file you downloaded from Aviatrix Controller to your VMware datastore.
In vSphere, select the Edge VM you created and click Edit settings.
Select the Virtual Hardware tab.
Next to CD/DVD Drive 1, click the down arrow and select Datastore ISO file from the pull-down menu.
To load the ISO to the virtual CD drive, next to Status, check Connect at power on.
Next to the CD/DVD Media field, click Browse. Select the ISO file you downloaded.
Connect at power on (step 4) is required when you attach the ISO image to the VM for the first time. If the VM is powered on at the time you attach the ISO image, select the Datastore ISO file and save the configuration to make the ISO available to ZTP.
Next, verify Edge in Controller. See `Verifying Edge in Controller`_.
2c. Deploying the Edge Virtual Machine in KVM¶
Before you begin, on the KVM Linux host ensure the LAN, WAN, and MGMT network bridges are associated with the physical ethernet interfaces on the KVM sever. Refer to the KVM product documentation.
Download the KVM QCOW2 file by using the link provided to you by Aviatrix Support. See Requesting an Aviatrix Edge Gateway Image File.
Launch Virtual Machine Manager UI to access the KVM host.
Create a new virtual machine from an existing disk image.
- From File menu, select New virtual machine.
- Select the option Import existing disk image.
- Click Forward.
Provide the path to the KVM QCOW2 file and specify the operating system type and version.
- Enter the path or use the Browse button to locate the KVM QCOW2 file you previously downloaded.
- For OS type, select Linux.
- For Version, select Ubuntu 18.04 LTS.
- Click Forward.
Enter the memory and CPU settings for the Edge Gateway VM and click Forward.
Enter a name for the Edge Gateway VM and check the Customize configuration before install checkbox, then click Finish.
Add the LAN and MGMT virtual bridge interfaces.
Click Add Hardware.
In Add New Virtual Hardware, select Network from the left pane and add two additional network interfaces for the LAN and MGMT virtual bridges. The virtual bridge for the WAN interface is automatically added as part of the VM image creation.
Choose the storage device and attach the iso file to the VM.
Click Begin Installation to create the Edge Gateway VM instance on the KVM host.
After you attach the ZTP .iso, the KVM hypervisor will auto-mount the media which contains the configuration file to provision the Edge Gateway.
For more information about deploying virtual machines and attaching .iso file in KVM, refer to KVM product documentation.
Next, verify Edge in Controller. See `Verifying Edge in Controller`_.
2d. Enabling Multiqueue virtio-net on KVM¶
Multiqueue virtio-net allows network performance to scale with the number of vCPUs, by allowing packet processing (packet sending and receiving) through multiple TX and RX queues.
To enable Multiqueue virtio-net support on KVM, when launching the Edge Gateway VM using virt-install, add the driver_queues parameter to the network interface details.
–network bridge=<bridge-name>, model=virtio,driver_queues=*N*
where, N is the number of vCPUs.
KVM Hypervisor does not support configuration of RX/TX queue size during runtime. RX/TX queue size should be configured during Edge VM bootup.
2e. Verifying Edge in Controller¶
To verify the Edge Gateway is up, wait for 5 minutes after you have attached the ZTP .iso file then do the following:
In Aviatrix Controller, go to Multi-Cloud Transit > List > Spoke.
In the State column, verify that the Edge Gateway you created is in the up state.
Click the refresh button to update the registration status.
If the Edge Gateway status is not up, you can troubleshoot Edge connectivity using CLI commands on the Edge Gateway console. See Troubleshooting Edge Gateway Connectivity.
Next, attach the Edge Gateway to the Transit Gateway. See `Attach Edge Gateway to Transit Gateway`_.
3. Attach Edge Gateway to Transit Gateway¶
For Edge Gateway attachment over a public network, you must update the WAN Public IP on the Edge Gateway and configure BGP ASN on the Edge Gateway before you attach Edge Gateway.
3a. Update WAN Public IP¶
To update the WAN Public IP, follow these steps.
- In Aviatrix Controller, go to Gateway > Select a Spoke Gateway.
- Select the Edge Gateway you want to attach and click Edit.
- In IP Configurations, click Discover Public IP.
- Verify the WAN Public IP and click Update.
3b. Configure BGP ASN on the Edge Gateway¶
To configure BGP AS Number (ASN) on the Edge Gateway, follow these steps.
- In Aviatrix Controller, go to MULTI-CLOUD TRANSIT > Advanced Config > Edit Spoke.
- In the BGP Spoke Gateway pull-down menu, select the Edge Gateway you created and enter the Local AS Number for the Edge Gateway.
- Click CHANGE.
3c. Attach Edge Gateway to Transit Gateway¶
After you have updated the WAN Public IP on the Edge Gateway and configured the BGP ASNs on both the Transit and Edge Gateway, follow these steps to attach the Edge Gateway to the Transit Gateway.
To create an Insane Mode attachment, make sure the Transit Gateway is created with Insane Mode enabled.
If you want Jumbo Frame enabled on the Edge Gateway, make sure to enable Jumbo Frame on the Edge Gateway before you attach it to the Transit Gateway. See Jumbo Frame.
In Aviatrix Controller, go to MULTI-CLOUD TRANSIT > List > Spoke. Confirm that the Edge Gateway you created is up.
Navigate to MULTI-CLOUD TRANSIT > Setup > Attach / Detach > 1a Attach Spoke Gateway to Transit Network.
In the Spoke Gateway/Source Gateway pull-down menu, select the Edge Gateway you created.
In the Transit Gateway/NextHop Gateway pull-down menu, select your Transit Gateway.
To connect over a private network, check Over Private Network box. Leave unchecked to connect using a public network.
To configure Jumbo Frame on Edge Gateway, check Jumbo Frame box.
To build High-Performance Encryption (HPE), check Insane Mode box. Leave unchecked if you do not require HPE.
For Insane Mode Tunnel Number, enter the number of HPE tunnels to create for Insane Mode over the Internet or private network.
Verify the Edge Gateway attachment in the following ways:
- From Controller: Navigate to Multi-Cloud Transit > List > Spoke
- From CoPilot: Navigate to Topology > Network Graph > Network.
4. Connect Edge Gateway to External Device (BGP over LAN)¶
To connect the Edge Gateway to LAN Routing using BGP over LAN, follow these steps.
Go to MULTI-CLOUD TRANSIT > Setup > External Connection.
In Connect to VGW/External Device/Azure VNG, enter the following values:
- Select these options: External Device, BGP, and LAN.
- VPC Name/Site ID: Select an existing Edge Site ID from the drop-down list.
- Connection Name: Enter a unique name to identify the connection to the LAN router.
- Aviatrix Gateway BGP ASN: Enter the BGP AS number the Edge Gateway will use to exchange routes with the LAN router.
- Primary Aviatrix Gateway: Select the Edge Gateway you created.
- Remote BGP AS Number: Enter the BGP AS number configured on the LAN router.
- Remote LAN IP: Enter the LAN router IP address for BGP peering.
- Local LAN IP: Enter the Edge LAN interface IP address for BGP peering.
- Click CONNECT.
Active-Active Edge and Active-Standby Edge Modes¶
When deploying multiple Edge Gateways, you have the option to use Active-Active mode or Active-Standby mode for connectivity between Edge Gateways and Transit Gateways.
In Active-Active mode, all Edge-to-Transit connections perform load sharing and transit the traffic.
Active-Active mode can support more than 2 Edge Gateways. While there is no maximum number of Edge Gateways, Aviatrix recommends a maximum of 4.
Active-Standby mode provides the flexibility on Aviatrix Transit Gateways and Aviatrix BGP Spoke Gateways to connect to on-prem with only one active peering and one backup/standby peering.
- The Active-Standby Preemptive setting is per site or location and is decided when you create the first Edge Gateway for that site. You cannot choose a different setting when you add more Edge Gateways to that site. For more information about preemptive and non-preemptive active-standby modes, see `Active-Standby`_.
The Transitive Routing feature allows an Edge Gateway to forward routes between multiple Transit Gateways that are connected to it. In Edge 2.0, you have the option to enable or disable Transitive Routing for an Edge Gateway; it is disabled by default.
Configuring Transitive Routing¶
To configure Transitive Routing, follow these steps.
- Attach the Edge Gateway to the first Transit Gateway. Follow the steps in `3b. Attach Edge Gateway to Transit Gateway`_.
- Repeat and attach the Edge Gateway to the second Transit Gateway.
- Navigate to MULTI-CLOUD TRANSIT > Advanced Config > Transitive Routing.
- Click the toggle to enable Transitive Routing.
- Verify routes on each Aviatrix Transit Gateway.
Transit Peering over Public Network for Backup Path¶
If you have a multi-cloud environment across Cloud Service Providers, for example, AWS and Azure, you can create Transit Gateway Peering over public network and use the Transit Gateway Peering as a secondary or backup path while the Edge Gateway with Transitive Routing enabled is used as the primary path for forwarding traffic.
Configuring Transit Peering over Public Network¶
To create Transit Peering over public network to use as backup path, follow these steps.
In the Aviatrix Controller, go to MULTI-CLOUD TRANSIT > Transit Peering.
Create a Transit Gateway Peering by following the Multi-Cloud Transit Gateway Peering over Public Network Workflow.
Go to MULTI-CLOUD TRANSIT > Advanced Config. Select the first Transit Gateway and take note the Local AS Number.
Scroll down to the Connection AS Path Prepend section. Select the Transit Peering connection name.
In the Prepend AS Path field, input the same Local AS Number three times separated by space.
Repeat steps 3, 4, and 5 for the second Transit Gateway.
Interactions with NAT¶
In Aviatrix Edge 2.0, the following NAT scenarios are supported:
- Customized SNAT on Edge Gateway - For traffic initiated from Edge location towards Transit Gateway or CSP.
- DNAT on Edge Gateway - For traffic initiated from CSP towards Edge location.
ActiveMesh connections are available in the NAT connection for non-HPE connections.
Default RBAC Access Account for Edge¶
In Aviatrix Edge 2.0, you have the option to create a default RBAC group and assign users to this group with permissions to create, delete, and manage Edge Gateways.
Creating the Default RBAC Access Account for Edge¶
To create an RBAC group with permissions to create, delete, and manage Edge gateways, follow these steps.
Log in to Aviatrix Controller 6.8.
Go to ACCOUNTS > Permission Groups > ADD NEW.
In the Group Name field, enter a name for the group, and then click OK.
In Permission Groups, select new group name, and then click MANAGE PERMISSION.
In Permissions for group “Group Name”, click ADD NEW.
In Add permissions to group “Group Name”, select Gateway – All read/write for Gateway.
Click OK, and then click Close.
In Permission Groups, select the new group name, and then click MANAGE ACCESS ACCOUNTS.
In Access accounts for group “Group Name”, click ADD NEW.
In Add access accounts to group “Group Name”, select edge_admin.
Click OK, and then click Close.
You can now create or assign a user account with the newly created RBAC group.
Selective Gateway Upgrade for Edge 2.0¶
The Aviatrix Edge 2.0 base OS is not upgradeable. To update the base OS to a newer version, you need to deploy the latest version of the Aviatrix Edge image to a new VM.
As Edge 2.0 base OS is not field upgradeable, Edge 2.0 does not support selective gateway image update and software rollback.
Troubleshooting Edge Gateway Connectivity¶
You can use the Clish commands below to troubleshoot the Edge Gateway.
To run Clish on the Edge Gateway, log in with the username admin.
|change_console_password||Changes the password for the CLI login.|
|check_conduit||Check conduit state.|
|check_network [dns][reachability]||Troubleshoot network connectivity.|
|diagnostics||Show gateway diagnostics from /home/ubuntu/cloudx-aws/avx_edge_status.json, which is written by register process or reset_config process.|
|logout||Log out of the console.|
|ping [-c count] [dest]||Ping destination, optional parameter ping packet count. The default is 5.|
|reboot||Reboot the system.|
|set_controller_ip [controller_ip]||Set the Controller IP address, usually performed after Controller migration when the Controller IP address is changed.|
|show_interfaces||Show output from the command “ifconfig -a | more”.|
|show_routes||Show output from the command “ip route show table all”.|
About BGP and Routing over Public and Private Networks¶
If the connectivity to the Cloud Service Provider (CSP) is over a private network:
- The edge (WAN) router runs a BGP session to VGW (AWS) where the edge router advertises an Edge Gateway WAN subnet network, and the VGW advertises the Transit VPC CIDR.
- The Edge Gateway LAN interface runs a BGP session to the edge router where the edge router advertises the on-prem network address range to Edge Gateway LAN interface.
- The Edge Gateway WAN interface runs a BGP session to the Transit Gateway in the Transit VPC where Transit Gateway advertises all Spoke VPC CIDRs to the Edge Gateway, and the Edge Gateway advertises on-prem network to the Transit Gateway.
If the connectivity to the CSP is over a public network:
- The Edge Gateway LAN and WAN interfaces do not use public IP addresses. The interfaces rely on the edge router or Firewall NAT function and Internet connectivity.
- The Edge Gateway LAN interface runs a BGP session to the edge router where the edge router advertises the on-prem network address range to the Edge Gateway LAN interface.