Controller High Availability in AWS

​

Overview

Aviatrix Controller HA in AWS leverages an auto-scaling group and a Lambda function to perform monitoring the health of the current Controller, launching a new Controller and restoring the configuration when the active Controller instance becomes unreachable. When a new Controller is launched, the existing Controller is terminated, its EIP is associated to the newly launched Controller, and the private IP is created in the new Controller subnet. Existing configuration is restored, resulting in a seamless experience when failover happens.

​

Prerequisites

• Existing AVX Controller. If you have not yet launched an AVX Controller, please see AWS Getting Started Guide.
  • Enable Controller backups.
  • AMI aviatrix_cloud_services_gateway_043018_YYYY-xxxxxx or later. If you are on an older AMI, see Migrating Your Aviatrix Controller to migrate to the latest Controller AMI first (Controller > Settings > Maintenance > Backup & Restore).

Aviatrix recommends contacting Aviatrix Support before migrating your Controller.

• The Controller’s VPC should have one or more public subnets, preferably in different AZs for HA across multiple AZ.
• To use Controller HA with an ELB, see Load Balancing.
• Controller has enabled backup function.

​

Controller HA Details

Aviatrix Controller HA operates by relying on an AWS Auto Scaling Group. This ASG has a desired capacity of 1 (and minimum capacity = 0 and maximum capacity = 1). If the Controller EC2 instance is stopped or terminated, it will be automatically re-deployed by the ASG. An AWS Lambda script is notified via SNS when new instances are launched by the Auto Scaling Group. This script handles configuration using a recent Controller backup file. The Aviatrix Controller manages these backups once enabled. Restoring the Aviatrix Controller from a newly built instance requires access to the S3 bucket to retrieve the latest backup file. In order to do this, the newly built EC2 Controller instance must be granted permission to read files in the bucket. The simplest method of doing this is via an IAM user with programmatic access to the S3 bucket. The lambda script also requires access to the S3 bucket. It is recommended that the backup bucket is used in the same account that was used to launch the controller.

​

Enabling AWS Controller High Availability

1. Log in to the AWS console and switch to the region where your existing AVX Controller is installed.
2. Launch a CloudFormation template stack:

• For Controller version 7.2 or later, use this CloudFormation template.
• For Controller earlier than version 7.2, use this CloudFormation template.

3. Click Next on the Create stack page to accept the CloudFormation stack defaults:

• Prerequisite - Prepare template: Template is ready
• Specify template: Amazon S3 URL

4. On the Specify Stack details page, populate the fields as follows:

5. Click Next.
6. Populate any additional CloudFormation Options.
7. Click Next.
8. Check “I acknowledge that AWS CloudFormation might create IAM resources with custom names.”
9. Click Create.
10. Refresh the Stacks page and wait for the status of this stack to change to CREATE_COMPLETE

​

Disabling AWS Controller High Availability

You can disable Controller HA by deleting the Controller HA CloudFormation stack.

• Please take a backup from the Controller first: Go to Controller > Settings > Maintenance > Backup & Restore > Backup Now. Verify that the S3 bucket now contains these backup files.
• Check the ASG capacity first. It should be minimum capacity=0, maximum capacity=1, desired capacity=1. If these are changed, deleting the Controller HA Cloudformation stack could have an impact on your current Controller.
• Log in to AWS Console, go to CloudFormation Service, identify the CloudFormation stack you used to enable Controller HA and delete the stack.

​

AWS High Availability FAQ

• How can I know which version of the HA script I am running?

versions.py file found in the AWS Lambda function with the name <controller_name>-ha would show the information. You can also see the version in the cloudwatch logs. Only versions from 1.5 and above are visible.

• How can I get notifications for H/A events?

Enter an email address to receive notifications for autoscaling group events while launching the CFT. You would receive an email to subscribe to SNS. Click on the link from the email to accept SNS event notifications

• My H/A event failed. What can I do?

You can manually restore the saved backup to a newly launched Controller. Please ensure controller H/A is disabled and re-enabled by deleting and re-creating the CFT stack to ensure that lambda is pointing to the right backup

• How do I ensure that lambda is pointing to the right backup?

In the AWS Lambda, verify if the INST_ID environment variable is updated correctly to the current controller instance ID and the PRIV_IP environment variable is updated to the current controller private IP.

• Where do I find logs related to Controller H/A ?

All logs related to H/A can be found in AWS Cloudwatch under the log group <controller_name>-ha

• How do I make lambda talk to the Controller privately within the VPC?

Launch CFT with Private access set to True. Attach lambda to the VPC from the AWS console. Ensure that the VPC that you have attached the lambda to has internet access via NAT gateway or VPC endpoints. You can also ensure that lambda has internet access by attaching an EIP(Elastic IP) to the lambda ENI(Network Interface). Please ensure that everything is reverted before you destroy the stack. Otherwise the lambda will not have internet access to respond to the CFT (CFT may get stuck on destroy).

• Can two Controllers in two different regions be linked such that they can detect if one or the other is down? Is this possible?

Our Controller HA script leverages EC2 auto-scaling. EC2 auto-scaling does not support cross regions but it does support cross AZs. The script will automatically bring up a new Controller in case the existing Controller enters an unhealthy state.

• Could a Controller in a different region be used to restore a saved configuration in case of disaster recovery? Will the change in the Controller’s IP cause any issues?

A Controller can be manually launched from a different region and the backed up configuration can be restored on it. The Controller’s new EIP should not cause any issues unless SAML VPN authentication is being used. (All peering tunnels will still work). In that case, SAML VPN client will need to reach the Controller IP address. If the FQDN hostname is used for the Controller for SAML, then it should work after changing the Route 53 to resolve to the correct EIP in the different region.

• How do I manage the Controller HA stack if the controller instance’s disk is encrypted?

If EBS Encryption using Customer managed key is enabled, the Autoscaling Group created may not have permissions to launch the instance. You will need to allow the service-linked role created for the Autoscaling group to have permissions to use this key for the cryptographic operation. To do so, go to AWS KMS→Customer managed keys→select the key and add the “AWSServiceRoleForAutoScaling” role to the list of Key Users.

• What do I need to do after I change the Controller name?

Please delete the CFT stack and then create a new CFT stack using the new Controller name.

• How do I update from CloudFormation template version 3 to version 4?

If you are currently running a version 7.2 or later Controller AMI with the v3 CloudFormation stack, you can update to the v4 template to benefit from improved security controls and access restrictions. To update your stack:

• Direct stack update (recommended): Click Update Stack in the CloudFormation console and apply the v4 template. This is faster than deleting and recreating the stack.
• Full replacement: If there are changes to the controller configuration, delete the current stack and create a new one with the v4 template.