Troubleshooting Controller and Gateway Upgrades

This section provides some solutions for resolving some commonly encountered issues during the upgrade process for your Controller and Gateways.

If the Controller upgrade fails, your Controller is reverted to its previous state.

Troubleshooting Errors During the Controller and Gateway Upgrade Dry Run

Check out the following solutions for the errors you encountered during the Controller and Gateway Upgrade Dry Run.

"Check Backup" Error

Symptom:

During dry run, you encounter a Check Backup blocker issue.

Solution:

  1. Go to CoPilot > Settings > Maintenance > Backup & Restore > External Storage, make sure an external storage is configured. If not, configure a new external storage.

  2. Go to CoPilot > Settings > Maintenance > Backup & Restore > Back Up Controller, click Backup Now. Then follow the backup wizard to back up the Controller to an available S3 bucket.

"Check All Gateway Versions" Error

Symptom:

During the dry run, you encounter a "Check All Gateway Versions" block issue.

Solution:

Go to CoPilot > Cloud Fabric > Gateways > Gateway Management, select the required gateways. Then click Actions > Upgrade software to upgrade gateways to match the Controller’s version.

"Check All Gateway State" Error

Symptom:

During the dry run, you encounter a "check all gateway states" issue.

Solution:

Reboot gateways and make sure all gateway states are up and all gateways are all reachable from the Controller.

"Invalid max_duration value" Error

Symptom:

During the dry run, you encounter an "invalid max_duration value" error.

Solution:

Enter a valid time limit between 40 to 1440 minutes.

"Check Private Mode" Error

Symptom:

During the dry run, you encounter a "check private mode" issue.

Solution:

From CoPilot UI > Settings > Configuration > Private Mode, toggle off the private mode. If this error persists, contact Aviatrix Support.

"Check Disk Usage" Error

Symptom:

During the dry run, you encounter a "check disk usage" issue.

Solution:

Reduce disk usage on Controller and make sure the disk usage is under at least 95%. See details in Disk Usage.

"Only members of the group 'admin' may modify Controller setting" Error

Symptom:

During the dry run, you encounter an "Only members of the group 'admin' may modify Controller setting" error.

Solution:

  1. Go to CoPilot UI > Cloud Resources > Cloud Account, click the account that you want to perform Controller upgrade. Then edit the permission.

  2. Assign this account with admin permission.

"Check Controller Migration" Error

Symptom:

During the dry run, you encounter a "Check Controller Migration" issue.

Solution:

Wait for the current migration to complete. Ensure that no other processes interfere with the migration, and monitor the system’s status dashboard or relevant logs for any updates.

"Controller Upgrade/Restore in Progress" Error

Symptoms:

During the dry run, you receive a "Controller Upgrade/Restore in Progress" Error.

Solution:

Wait for the current upgrade to complete. Ensure that no other processes interfere with the migration, and monitor the system’s status dashboard or relevant logs for any updates.

"The available memory should be no less than 536870912 bytes " Error

Symptom:

During the dry run, you encounter an "Available memory should be no less than 536870912 bytes" issue.

Solution:

This issue might be temporary, so please try again. If it continues, try resizing your Controller or contact Aviatrix Support for assistance.

"No identity-based policy allows the ec2:AllocateAddress Action" Error

Symptom:

During the dry run, you encounter a "No identity-based policy allows the ec2:AllocateAddress Action" Error. Or you receive a "Check AWS Allocate EIP" blocker issue.

Solution:

Modify the AWS account permission on EC2 role to allow EIP allocation.

"Check AWS Associate EIP" Error

Symptom:

During the dry run, you encounter a "Check AWS Allocate EIP" issue.

Solution:

Modify the AWS account permission on EC2 role to allow EIP allocation.

"Check AWS Disassociate EIP" Error

Symptom:

During the dry run, you encounter a "Check AWS Disassociate EIP" issue.

Solution:

Modify the AWS account permission on EC2 role to allow EIP disassociation.

"Check AWS Add SG Ingress Rule" Error

Symptom:

During the dry run, you receive a "Check AWS Add SG Ingress Rule" blocker issue.

Solution:

Modify the AWS account permission to add ingress rule in to ec2 role.

"Check AWS Remove SG Ingress Rule" Error

Symptom:

During the dry run, you encounter a "Check AWS Remove SG Ingress Rule" issue.

Solution:

Modify the AWS account permission to remove ingress rule in to EC2 role.

"Check AWS Run Instance" Error

Symptom:

During the dry run, you encounter a "Check AWS Run Instance" issue.

Solution:

Modify the AWS account permission in the EC2 role to give this account proper permission to run EC2 instances.

"Check AWS Stop Instance" Error

Symptom:

During the dry run, you encounter a "Check AWS Stop Instance" issue.

Solution:

Modify the AWS account permission in the EC2 role to give this account proper permission to stop EC2 instances.

"Check AWS Terminate Instance" Error

Symptom:

During the dry run, you receive a "Check AWS Terminate Instance" issue.

Solution:

Modify the AWS account permission in the EC2 role to give this account proper permission to terminate EC2 instances.

"Check AWS Disable Stop Protection" Error

Symptom:

During the dry run, you encounter a "Check AWS Disable Stop Protection" blocker issue.

Solution:

Modify the AWS account permission in the EC2 role to enable stop protection on EC2 instances.

"Check AWS Describe Volume" Error

Symptom:

During the dry run, you encounter a "Check AWS Describe Volume" blocker issue.

Solution:

Modify the AWS account permission to allow this account describe Controller volume.

"Check AWS Quota EIP" Error

Symptom:

During the dry run, you encounter a "Check AWS Quota EIP" issue.

Solution:

In an AWS account, you need to increase the quota for Elastic IP (EIP) addresses.

"Check Controller HA" Error

Symptom:

During the dry run, you encounter a "Check Controller HA" issue.

Solution:

  1. Delete the CloudFormation Stack before the Controller Upgrade.

  2. Redeploy the Cloud Formation after the Controller Upgrade completes. See Controller High Availability in AWS for details.

"Check AWS Describe Instance" Error

Symptom:

During the dry run, you encounter a "Check AWS Describe Instance" issue.

Solution:

Modify the account permissions to allow the account to describe Controller instance.

"Check AWS Controller EIP" Error

Symptom:

During the dry run, you encounter a "Check AWS Controller EIP" issue.

Solution:

  1. Go to CoPilot UI > Settings > Configuration > General, check the stored EIP, which as associated with Aviatrix Controller.

  2. Modify the EIP in AWS to match the EIP stored in Controller.

"Check Universal License Type" Error

Symptom:

During the dry run, you encounter a "Check Universal License Type" issue.

Solution: Subscribe to a universal license before you perform Controller upgrade.

"Check Registered CloudN Instance" Error

Symptom:

During the dry run, you receive a "Check Registered CloudN Instance" issue.

Solution:

Contact Aviatrix Support for the details about upgrading Controller with CloudN.

"Dry Run of upgrade to software userconnect version latest failed"

Symptom:

During the dry run of the upgrade, you encounter the error message: "Dry Run of upgrade to software UserConnect version latest failed" with the error code "AVXERR-UPGRADE-CTRL-0099".

Solution:

From Controller UI, go to Troubleshoot > Diagnostics > mTLS > Spire Nodes. Choose controller, then click RE-ATTEST.

"Check Azure Quota Compute Blocking" Error

Symptom:

During the dry run, you encounter a "Check Azure Quota Compute Blocking" issue.

Solution:

In an Azure account, release the unused resources or increase the resource quota limit.

"Check Azure Export Resource Group" Error

Symptom:

During the dry run, you encounter a "Check Azure Export Resource Group" issue.

Solution:

Update your Azure account permission to allow export of resource group.

"UPGRADE_IMAGE threads.tasks execute" Error

Symptom: During the upgrade dry run, you encounter a "UPGRADE_IMAGE threads.tasks execute" issue.

Solution:

  • Check the IAM policy to ensure that the correct IAM policy is applied.

  • Verify the resource availability to make sure that resources are available in the specified region for deployment.

"Check Azure Get VM" Error

Symptom:

During the dry run, you encounter a "Check Azure Get VM" issue.

Solution:

Update your Azure account permission to allow the account to get Controller VM details.

"Azure Controller Public IP" Error

Symptom:

During the dry run, you encounter an "Azure Controller Public IP" issue.

Solution:

Update your Azure Controller EIP to match it with the stored IPv4 address. Alternatively, you can also contact Aviatrix Support to modify the stored IPv4 address in Controller.

Troubleshooting Errors During Controller and Gateway Upgrade

Check out the following solutions for the errors you encountered during the Controller and Gateway Upgrade.

"Customer ID is not set up, please enter one." Error

Symptom:

When you are prompted to log back into CoPilot after upgrade, a "Customer ID is not set up, please enter one." error occurs.

Solution:

Wait for some time to allow upgrade completed from the backend. Then enter your username and password to login.

"EIP Switch Azure Failure" Error

Symptom:

During upgrade, you see an "EIP move failed" error.

Solution:

Change your Azure EIP from basic to standard EIP.

"Migration Timeout - x gateways did not register to new Controller" Error

Symptom:

During upgrade, you see a "Migration Timeout - x gateways did not register to new Controller" error.

Solution:

Contact Aviatrix Support to register all your gateways to your new Controller.