Troubleshooting Controller and Gateway Upgrades

Symptom:

During dry run, you encounter a Check Backup blocker issue.

Solution:

Symptom:

During the dry run, you encounter a "Check All Gateway Versions" block issue.

Solution:

Go to CoPilot > Cloud Fabric > Gateways > Gateway Management, select the required gateways. Then click Actions > Upgrade software to upgrade gateways to match the Controller’s version.

Symptom:

During the dry run, you encounter a "check all gateway states" issue.

Solution:

Reboot gateways and make sure all gateway states are up and all gateways are all reachable from the Controller.

Symptom:

During the dry run, you encounter an "invalid max_duration value" error.

Solution:

Enter a valid time limit between 40 to 1440 minutes.

Symptom:

During the dry run, you encounter a "check private mode" issue.

Solution:

From CoPilot UI > Settings > Configuration > Private Mode, toggle off the private mode. If this error persists, contact Aviatrix Support.

Symptom:

During the dry run, you encounter a "check disk usage" issue.

Solution:

Reduce disk usage on Controller and make sure the disk usage is under at least 95%. See details in Disk Usage.

Symptom:

During the dry run, you encounter an "Only members of the group 'admin' may modify Controller setting" error.

Solution:

Symptom:

During the dry run, you encounter a "Check Controller Migration" issue.

Solution:

Wait for the current migration to complete. Ensure that no other processes interfere with the migration, and monitor the system’s status dashboard or relevant logs for any updates.

Symptoms:

During the dry run, you receive a "Controller Upgrade/Restore in Progress" Error.

Solution:

Wait for the current upgrade to complete. Ensure that no other processes interfere with the migration, and monitor the system’s status dashboard or relevant logs for any updates.

Symptom:

During the dry run, you encounter an "Available memory should be no less than 536870912 bytes" issue.

Solution:

This issue might be temporary, so please try again. If it continues, try resizing your Controller or contact Aviatrix Support for assistance.

Symptom:

During the dry run, you encounter a "No identity-based policy allows the ec2:AllocateAddress Action" Error. Or you receive a "Check AWS Allocate EIP" blocker issue.

Solution:

Modify the AWS account permission on EC2 role to allow EIP allocation.

Symptom:

During the dry run, you encounter a "Check AWS Allocate EIP" issue.

Solution:

Modify the AWS account permission on EC2 role to allow EIP allocation.

Symptom:

During the dry run, you encounter a "Check AWS Disassociate EIP" issue.

Solution:

Modify the AWS account permission on EC2 role to allow EIP disassociation.

Symptom:

During the dry run, you receive a "Check AWS Add SG Ingress Rule" blocker issue.

Solution:

Modify the AWS account permission to add ingress rule in to ec2 role.

Symptom:

During the dry run, you encounter a "Check AWS Remove SG Ingress Rule" issue.

Solution:

Modify the AWS account permission to remove ingress rule in to EC2 role.

Symptom:

During the dry run, you encounter a "Check AWS Run Instance" issue.

Solution:

Modify the AWS account permission in the EC2 role to give this account proper permission to run EC2 instances.

Symptom:

During the dry run, you encounter a "Check AWS Stop Instance" issue.

Solution:

Modify the AWS account permission in the EC2 role to give this account proper permission to stop EC2 instances.

Symptom:

During the dry run, you receive a "Check AWS Terminate Instance" issue.

Solution:

Modify the AWS account permission in the EC2 role to give this account proper permission to terminate EC2 instances.

Symptom:

During the dry run, you encounter a "Check AWS Disable Stop Protection" blocker issue.

Solution:

Modify the AWS account permission in the EC2 role to enable stop protection on EC2 instances.

Symptom:

During the dry run, you encounter a "Check AWS Describe Volume" blocker issue.

Solution:

Modify the AWS account permission to allow this account describe Controller volume.

Symptom:

During the dry run, you encounter a "Check AWS Quota EIP" issue.

Solution:

In an AWS account, you need to increase the quota for Elastic IP (EIP) addresses.

Symptom:

During the dry run, you encounter a "Check Controller HA" issue.

Solution:

Symptom:

During the dry run, you encounter a "Check AWS Describe Instance" issue.

Solution:

Modify the account permissions to allow the account to describe Controller instance.

Symptom:

During the dry run, you encounter a "Check AWS Controller EIP" issue.

Solution:

Symptom:

During the dry run, you encounter a "Check Universal License Type" issue.

Solution: Subscribe to a universal license before you perform Controller upgrade.

Symptom:

During the dry run, you receive a "Check Registered CloudN Instance" issue.

Solution:

Contact Aviatrix Support for the details about upgrading Controller with CloudN.

Symptom:

During the dry run of the upgrade, you encounter the error message: "Dry Run of upgrade to software UserConnect version latest failed" with the error code "AVXERR-UPGRADE-CTRL-0099".

Solution:

From Controller UI, go to Troubleshoot > Diagnostics > mTLS > Spire Nodes. Choose controller, then click RE-ATTEST.

Symptom:

During the dry run, you encounter a "Check Azure Quota Compute Blocking" issue.

Solution:

In an Azure account, release the unused resources or increase the resource quota limit.

Symptom:

During the dry run, you encounter a "Check Azure Export Resource Group" issue.

Solution:

Update your Azure account permission to allow export of resource group.

Symptom: During the upgrade dry run, you encounter a "UPGRADE_IMAGE threads.tasks execute" issue.

Solution:

Symptom:

During the dry run, you encounter a "Check Azure Get VM" issue.

Solution:

Update your Azure account permission to allow the account to get Controller VM details.

Symptom:

During the dry run, you encounter an "Azure Controller Public IP" issue.

Solution:

Update your Azure Controller EIP to match it with the stored IPv4 address. Alternatively, you can also contact Aviatrix Support to modify the stored IPv4 address in Controller.

Symptom:

When you are prompted to log back into CoPilot after upgrade, a "Customer ID is not set up, please enter one." error occurs.

Solution:

Wait for some time to allow upgrade completed from the backend. Then enter your username and password to login.

Symptom:

During upgrade, you see an "EIP move failed" error.

Solution:

Change your Azure EIP from basic to standard EIP.

Symptom:

During upgrade, you see a "Migration Timeout - x gateways did not register to new Controller" error.

Solution:

Contact Aviatrix Support to register all your gateways to your new Controller.