> ## Documentation Index
> Fetch the complete documentation index at: https://docs.aviatrix.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting Controller and Gateway Upgrades

> This section provides some solutions for resolving some commonly encountered issues during the upgrade process for your Controller and Gateways.

<a id="start-controller-gateway-upgrade-troubleshooting" />

This section provides some solutions for resolving some commonly encountered
issues during the upgrade process for your Controller and Gateways.

<Note>
  If the Controller upgrade fails, your Controller is reverted to its previous
  state.
</Note>

## Troubleshooting Errors During the Controller and Gateway Upgrade Dry Run

Check out the following solutions for the errors you encountered during the
Controller and Gateway Upgrade Dry Run.

### "Check Backup" Error

*Symptom:*

During dry run, you encounter a Check Backup blocker issue.

*Solution:*

1. Go to *CoPilot > Settings > Maintenance > Backup & Restore > External
   Storage*, make sure an external storage is configured. If not, configure a
   new external storage.
2. Go to *CoPilot > Settings > Maintenance > Backup & Restore > Back Up
   Controller*, click *Backup Now*. Then follow the backup wizard to back up the
   Controller to an available S3 bucket.

### "Check All Gateway Versions" Error

*Symptom:*

During the dry run, you encounter a "Check All Gateway Versions" block issue.

*Solution:*

Go to *CoPilot > Cloud Fabric > Gateways > Gateway Management*, select the
required gateways. Then click *Actions > Upgrade* software to upgrade gateways
to match the Controller's version.

### "Check All Gateway State" Error

*Symptom:*

During the dry run, you encounter a "check all gateway states" issue.

*Solution:*

Reboot gateways and make sure all gateway states are up and all gateways are all
reachable from the Controller.

### "Invalid max\_duration value" Error

*Symptom:*

During the dry run, you encounter an "invalid max\_duration value" error.

*Solution:*

Enter a valid time limit between 40 to 1440 minutes.

### "Check Private Mode" Error

*Symptom:*

During the dry run, you encounter a "check private mode" issue.

*Solution:*

Private Mode has been removed as of Controller 10.1.0, so this setting cannot be
toggled off from CoPilot. Contact
[Aviatrix Support](https://support.aviatrix.com) for help resolving this issue.

### "Check Disk Usage" Error

*Symptom:*

During the dry run, you encounter a "check disk usage" issue.

*Solution:*

Reduce disk usage on Controller and make sure the disk usage is under at least
95%. See details in <a href={"/docs/enterprise/" + "10.1" +
"/guides/troubleshooting/troubleshoot-diagnostic-result#disk-usage"}>Disk
Usage</a>.

### "Only members of the group 'admin' may modify Controller setting" Error

*Symptom:*

During the dry run, you encounter an "Only members of the group 'admin' may
modify Controller setting" error.

*Solution:*

1. Go to *CoPilot UI > Cloud Resources > Cloud Account*, click the account that
   you want to perform Controller upgrade. Then edit the permission.
2. Assign this account with admin permission.

### "Check Controller Migration" Error

*Symptom:*

During the dry run, you encounter a "Check Controller Migration" issue.

*Solution:*

Wait for the current migration to complete. Ensure that no other processes
interfere with the migration, and monitor the system's status dashboard or
relevant logs for any updates.

### "Controller Upgrade/Restore in Progress" Error

*Symptoms:*

During the dry run, you receive a "Controller Upgrade/Restore in Progress"
Error.

*Solution:*

Wait for the current upgrade to complete. Ensure that no other processes
interfere with the migration, and monitor the system's status dashboard or
relevant logs for any updates.

### "The available memory should be no less than 536870912 bytes " Error

*Symptom:*

During the dry run, you encounter an "Available memory should be no less than
536870912 bytes" issue.

*Solution:*

This issue might be temporary, so try again. If it continues, try resizing your
Controller or contact [Aviatrix Support](https://support.aviatrix.com) for
assistance.

### "No identity-based policy allows the ec2:AllocateAddress Action" Error

*Symptom:*

During the dry run, you encounter a "No identity-based policy allows the
ec2:AllocateAddress Action" Error. Or you receive a "Check AWS Allocate EIP"
blocker issue.

*Solution:*

Modify the AWS account permission on EC2 role to allow EIP allocation.

### "Check AWS Associate EIP" Error

*Symptom:*

During the dry run, you encounter a "Check AWS Allocate EIP" issue.

*Solution:*

Modify the AWS account permission on EC2 role to allow EIP allocation.

### "Check AWS Disassociate EIP" Error

*Symptom:*

During the dry run, you encounter a "Check AWS Disassociate EIP" issue.

*Solution:*

Modify the AWS account permission on EC2 role to allow EIP disassociation.

### "Check AWS Add SG Ingress Rule" Error

*Symptom:*

During the dry run, you receive a "Check AWS Add SG Ingress Rule" blocker issue.

*Solution:*

Modify the AWS account permission to add an ingress rule to the EC2 role.

### "Check AWS Remove SG Ingress Rule" Error

*Symptom:*

During the dry run, you encounter a "Check AWS Remove SG Ingress Rule" issue.

*Solution:*

Modify the AWS account permission to remove an ingress rule from the EC2 role.

### "Check AWS Run Instance" Error

*Symptom:*

During the dry run, you encounter a "Check AWS Run Instance" issue.

*Solution:*

Modify the AWS account permission in the EC2 role to give this account proper
permission to run EC2 instances.

### "Check AWS Stop Instance" Error

*Symptom:*

During the dry run, you encounter a "Check AWS Stop Instance" issue.

*Solution:*

Modify the AWS account permission in the EC2 role to give this account proper
permission to stop EC2 instances.

### "Check AWS Terminate Instance" Error

*Symptom:*

During the dry run, you receive a "Check AWS Terminate Instance" issue.

*Solution:*

Modify the AWS account permission in the EC2 role to give this account proper
permission to terminate EC2 instances.

### "Check AWS Disable Stop Protection" Error

*Symptom:*

During the dry run, you encounter a "Check AWS Disable Stop Protection" blocker
issue.

*Solution:*

Modify the AWS account permission in the EC2 role to enable stop protection on
EC2 instances.

### "Check AWS Describe Volume" Error

*Symptom:*

During the dry run, you encounter a "Check AWS Describe Volume" blocker issue.

*Solution:*

Modify the AWS account permission to allow this account to describe Controller
volume.

### "Check AWS Quota EIP" Error

*Symptom:*

During the dry run, you encounter a "Check AWS Quota EIP" issue.

*Solution:*

In an AWS account, you need to increase the quota for Elastic IP (EIP)
addresses.

### "Check Controller HA" Error

*Symptom:*

During the dry run, you encounter a "Check Controller HA" issue.

*Solution:*

1. Delete the CloudFormation Stack before the Controller Upgrade.
2. Redeploy the Cloud Formation after the Controller Upgrade completes. See <a href={"/docs/enterprise/" + "10.1" +
   "/guides/controlplane/controller-ha-aws"}>Controller High Availability in
   AWS</a> for details.

### "Check AWS Describe Instance" Error

*Symptom:*

During the dry run, you encounter a "Check AWS Describe Instance" issue.

*Solution:*

Modify the account permissions to allow the account to describe Controller
instance.

### "Check AWS Controller EIP" Error

*Symptom:*

During the dry run, you encounter a "Check AWS Controller EIP" issue.

*Solution:*

1. Go to *CoPilot UI > Settings > Configuration > General*, check the stored
   EIP, which is associated with Aviatrix Controller.
2. Modify the EIP in AWS to match the EIP stored in Controller.

### "Check Universal License Type" Error

*Symptom:*

During the dry run, you encounter a "Check Universal License Type" issue.

*Solution:*

Subscribe to a universal license before you perform Controller upgrade.

### "Check Registered CloudN Instance" Error

*Symptom:*

During the dry run, you receive a "Check Registered CloudN Instance" issue.

*Solution:*

Contact [Aviatrix Support](https://support.aviatrix.com) for the details about
upgrading Controller with CloudN.

### "Dry Run of upgrade to software userconnect version latest failed"

*Symptom:*

During the dry run of the upgrade, you encounter the error message: "Dry Run of
upgrade to software UserConnect version latest failed" with the error code
"AVXERR-UPGRADE-CTRL-0099".

*Solution:*

From Controller UI, go to *Troubleshoot > Diagnostics > mTLS > Spire Nodes*.
Choose *controller*, then click *RE-ATTEST*.

### "Check Azure Quota Compute Blocking" Error

*Symptom:*

During the dry run, you encounter a "Check Azure Quota Compute Blocking" issue.

*Solution:*

In an Azure account, release the unused resources or increase the resource quota
limit.

### "Check Azure Export Resource Group" Error

*Symptom:*

During the dry run, you encounter a "Check Azure Export Resource Group" issue.

*Solution:*

Update your Azure account permission to allow export of resource group.

### "UPGRADE\_IMAGE threads.tasks execute" Error

*Symptom:*

During the upgrade dry run, you encounter a "UPGRADE\_IMAGE threads.tasks
execute" issue.

*Solution:*

* Check the IAM policy to ensure that the correct IAM policy is applied.

* Verify the resource availability to make sure that resources are available in
  the specified region for deployment.

### "Check Azure Get VM" Error

*Symptom:*

During the dry run, you encounter a "Check Azure Get VM" issue.

*Solution:*

Update your Azure account permission to allow the account to get Controller VM
details.

### "Azure Controller Public IP" Error

*Symptom:*

During the dry run, you encounter an "Azure Controller Public IP" issue.

*Solution:*

Update your Azure Controller EIP to match it with the stored IPv4 address.
Alternatively, you can also contact
[Aviatrix Support](https://support.aviatrix.com) to modify the stored IPv4
address in Controller.

## Troubleshooting Errors During Controller and Gateway Upgrade

Check out the following solutions for the errors you encountered during the
Controller and Gateway Upgrade.

### "Customer ID is not set up, enter one." Error

*Symptom:*

When you are prompted to log back into CoPilot after upgrade, a "Customer ID is
not set up, enter one." error occurs.

*Solution:*

Wait for some time to allow the upgrade to complete on the backend. Then enter
your username and password to login.

### "EIP Switch Azure Failure" Error

*Symptom:*

During upgrade, you see an "EIP move failed" error.

*Solution:*

Change your Azure EIP from `basic` to `standard` EIP.

### "Migration Timeout - x gateways did not register to new Controller" Error

*Symptom:*

During upgrade, you see a "Migration Timeout - x gateways did not register to
new Controller" error.

*Solution:*

Contact [Aviatrix Support](https://support.aviatrix.com) to register all your
gateways to your new Controller.

<a id="end-controller-gateway-upgrade-troubleshooting" />
