Corrected Issues in Aviatrix Release 8.2.20
| Issue | Description |
|---|---|
| AVX-65016 | Fixed an issue where the firewall state did not recover from the Unaccessible state after the first vendor integration failure. |
| AVX-74719 | Fixed an issue where performing a Controller backup restore could cause a temporary traffic outage of approximately 40 seconds due to all routes being deleted and re-added during route reconvergence. Routes are now preserved during backup restore to prevent traffic disruption. |
| AVX-74739 | Fixed an issue where the database migration timeout during Controller upgrade was hard-coded at 15 minutes, causing upgrades to fail and roll back in large-scale deployments with thousands of gateways and tunnels. The migration timeout is now user-configurable. |
| AVX-75117 | Fixed a memory leak in the TrafficServer (ATS) process on gateways with DCF intrusion analysis and decryption enabled under high-concurrency traffic conditions that could cause the ATS process to crash and enter a restart loop. |
| AVX-75135 | Fixed an issue where tunnel status report processing on the Controller was slower after upgrading from version 8.0 to 8.1 due to increased database query overhead. The database queries have been optimized to restore processing performance. |
| AVX-75256 | Fixed an issue where FQDN gateway data was not correctly displayed after upgrading from version 7.2.x to 8.0 or later, causing the Egress FQDN Gateway View to appear empty. Gateways with FQDN tags now display correctly in the UI and are returned properly by the list_fqdn_gateways API. |
| AVX-75414 | Fixed an issue where a background service responsible for collecting network topology data experienced unbounded memory growth, which could eventually cause the process to be terminated due to excessive memory consumption. |
| AVX-75496 | Fixed an issue where a Controller upgrade could fail during a database migration step if a corrupted VPC record was present in the Controller database, causing the upgrade to fail and trigger a rollback. |
| AVX-75582 | Fixed an issue where the Aviatrix Controller retried Azure RequestDisallowedByPolicy errors unnecessarily, causing event handler congestion and delaying gateway deployments. |
| AVX-76296 | Fixed an issue in GCP global VPC environments where the Controller removed all gateway routes during routine gateway operations such as resize or image upgrade, causing extended traffic blackholing. |
| AVX-76719 | Fixed an issue where Controller restore operations could stall for approximately 30 minutes during the scheduler shutdown phase due to connectivity issues with AWS API endpoints or invalid/expired IAM credentials. |
| AVX-76919 | Fixed an issue where the gateway state synchronization service could crash if the gateway’s DNS configuration was not yet fully initialized, temporarily interrupting state synchronization between the Controller and gateways. |
| AVX-77088 | Fixed an issue where editing legacy FQDN domain name filters on a gateway could cause all FQDN filtering processes to stop simultaneously, resulting in a brief filtering outage until the processes were automatically restarted. |
| AVX-77419 | Fixed an issue where importing a system-defined Distributed Cloud Firewall (DCF) ruleset, such as the V1 Policy List or the Kubernetes Policy List, into Terraform state as an aviatrix_dcf_ruleset resource and then destroying the imported resource could delete the underlying system-defined ruleset on the Controller. Deletion of these system-defined rulesets is now prevented at the Controller. |
| AVX-77767 | Fixed an issue where Azure transit gateways with very large numbers of IPsec tunnels (3000+) could lose all spoke peerings and enter a high-CPU charon state under sustained high tunnel activity. |
Known Issues in Aviatrix Release 8.2.20
| Issue | Description |
|---|---|
| AVX-62003 | Azure gateway image upgrades may fail when the Controller does not have the required Azure image subscription access. During the upgrade, the system deletes the existing gateway before validating subscription availability, which can result in gateway deletion without a replacement being created. This leaves dangling gateways in the Controller and can cause potential service outages. Impact:
None. To avoid outages, ensure the Controller subscription includes access to the required Azure image before attempting upgrades. |
| AVX-62299 | When upgrading from Controller version 7.1 to 7.2 or 8.0, Spoke Gateways with routing through a Public Subnet Filtering (PSF) Gateway may fail to upgrade and become unreachable if the PSF Gateway has not been upgraded first. This issue affects AWS environments where Spoke Gateway route tables are configured to point to a PSF Gateway. To avoid this issue, follow the correct upgrade sequence:1. Upgrade the PSF Gateway first. 2. Wait for the PSF Gateway upgrade to complete successfully. 3. Then upgrade the dependent Spoke Gateways. |
| AVX-64868 | In some scenarios involving rapid VRRP state transitions, the keepalived VRRP state may not be reported accurately to the Controller. This can result in temporary discrepancies between the actual VRRP status and what is displayed in the Controller UI, leading to confusion and difficulties during troubleshooting. Impact:
|
| AVX-66631 | Transit gateways with large-scale tunnel deployments (1300+ tunnels) may experience extended traffic loss during image upgrades. Although the image upgrade completes successfully, traffic may remain down for several minutes afterward due to delayed tunnel reconfiguration. Workaround:
|
| AVX-67126 | Dry-run validation may fail when upgrading the Controller from version 8.0.10 to 8.1.0 due to a gateway version mismatch error. This occurs when the upgrade path starts from 8.0.0, progresses to 8.0.10 successfully, but encounters a dry-run failure when proceeding to 8.1.0. |
| AVX-67180 | In some environments running Aviatrix Controller version 8.1.x, the Controller UI may become inaccessible after the Controller is restarted. When this issue occurs, API requests fail with RequestRefused errors and the UI cannot be accessed, although SSH access to the Controller remains available.Impact: Users are unable to log in to the Controller UI or perform API operations. Workaround: Restart the VM where the Aviatrix Controller is running. |
| AVX-68887 | When attaching VPN users to profiles using the attach_vpn_user_to_profile API, the CoPilot or Controller UI may continue to display the user profile as N/A even though the attachment operation completes successfully.In some cases, users later reappear as active but still show no profile association in the UI. This results in a display inconsistency between the UI and the backend state. Impact: VPN user profile assignments may appear unsuccessful in the UI, which can cause confusion during profile management. There is no functional impact: the VPN profile is correctly assigned in the backend, and users can connect to the VPN as expected. Affected Scenario: OpenVPN profile management operations that use API-based user-to-profile attachment. Workaround: None. |
| AVX-69649 | The migration dry-run EIP accounting does not include public IPs that are not part of the Elastic IP quota, potentially producing inaccurate dry-run results. Impact:
Manually verify EIP allocation and quotas before performing the migration. Contact Aviatrix Support for assistance. |
| AVX-70543 | When HA-enabled spoke gateways have DPI/IDS or Layer7 policies configured with “Destination: Anywhere” and the destination smart group contains private CIDRs, the policies become invalid and cause traffic drops. Affected Scenario: Spoke gateways with HA enabled using DPI/IDS or Layer7 policies where destination smart groups include private CIDR ranges. Impact:
|
| AVX-70864 | In Controller version 8.2.0, gateways may remain in a Configuration Not Up-to-Date state when applying Distributed Cloud Firewall (DCF) policies under certain conditions. This issue can occur when DCF policies with Web Groups and IDS or IPS enabled are pushed to gateways. The configuration update does not complete within the expected time, which may leave the gateway out of sync. Impact:
Gateways running Controller 8.2.0 with DCF policies that include IDS or IPS features, particularly on smaller gateway instance sizes. Workaround: Restart the affected gateway to clear the condition and allow the configuration to be applied successfully. If the issue persists, contact Aviatrix Support for assistance. |
| AVX-70958 | When clients use HTTP/2 connections, TrafficServer incorrectly reuses origin connections, potentially causing connection handling issues in MITM SNI verification scenarios. Impact:
Use both IP address and SNI instead of IP alone to ensure proper connection isolation. |
| AVX-71245 | Additional Distributed Cloud Firewall (DCF) log support records end-session events for IDS and IPS signature matches. These logs include a reason field indicating the match type (IPS_POLICY_DENY or IDS_POLICY_ALERT) along with the Signature ID (SID) of the matched rule. Due to a bug, the end-session log is omitted when Decryption is not enabled for Intrusion Analysis. Impact:
Workaround: Enable Decryption under Intrusion Analysis to ensure end-session logs are generated. |
| AVX-71441 | When upgrading gateways from version 8.1.20 to 8.2.0, in very rare cases, the gateway can enter an infinite retry loop attempting to download a non-existent configuration file from the Controller, causing the upgrade process to fail completely Impact:
Workaround: Retry the gateway upgrade operation from the Controller UI or Copilot UI. If the issue persists, perform an image upgrade of the impacted gateway. |
| AVX-71559 | When performing a batch gateway image upgrade in Azure, some gateways may fail during the upgrade process. Impact:
Retry the gateway image upgrade for the failed gateways individually rather than as a batch operation. Contact Aviatrix Support for assistance. |
| AVX-71820 | When deploying a load balancer–enabled VPN gateway with an overlapping VPN CIDR on Controller versions 8.0, 8.1, or 8.2, the gateway creation fails. Impact:
Workaround: Ensure that the VPN CIDR does not overlap with existing gateways behind the load balancer before deployment. Contact Aviatrix Support for assistance. |
| AVX-72940 | Creating a new gateway with the same name as an existing gateway may cause local files of the existing gateway to be deleted when the creation fails. The existing gateway name disappears from the Controller CLI when this issue occurs. This can break SSH access (sshgw) for the existing gateway. Impact:
|
| AVX-73436 | When using the update_spoke_vpc_route_table API to onboard an Azure route table, the default route (0.0.0.0/0) is not programmed in the spoke VNET route table if the Spoke Gateway has learned the default route from any of the following sources:
Impact:
Manually add the default route to the Azure route table. Contact Aviatrix Support for assistance. |
| AVX-74577 | Users are unable to modify tags on third-party firewall instances when those tags contain values with multiple colons (for example, team:iac:module.version:v1.5.3). Attempts to update tags after deployment fail with a too many values to unpack error. Initial deployment is unaffected because tags are passed via a different code path during creation.Impact:
Avoid using multiple colons in tag values when modifying tags after deployment. Use alternative delimiters such as hyphens or underscores. |
| AVX-75586 | When using Terraform to create gateways and immediately attach them to transit gateways, a race condition can occur where the Terraform provider attempts to create the spoke-to-transit attachment before the gateway has fully transitioned to the “Up” state. The gateway creation API returns before the gateway is operationally ready, causing subsequent attachment operations to fail. Impact:
Add a delay or polling mechanism in Terraform configurations between gateway creation and spoke-to-transit attachment resources. Use depends_on with a time_sleep resource to allow the gateway to reach the “Up” state before attempting attachment. |
| AVX-75607 | Gateway launch may fail with a tls: bad certificate error when pulling container images. The Controller’s registry TTL eviction (garbage collection) may fire while images are being downloaded to the gateway, corrupting in-flight blob transfers. The gateway’s container initialization cannot complete, and a supporting certificate-management process on the gateway exits with code 125.Impact:
Restart the Controller’s application service to reset the registry state, then retry gateway creation. Contact Aviatrix Support for assistance. |
| AVX-76132 | Unable to configure more than one OpenVPN gateway behind a UDP Load Balancer. Only the first gateway is retained while additional gateways are incorrectly excluded. This issue affects versions 6.9, 7.x, 8.x, and 9.0.0. Impact:
Contact Aviatrix Support for assistance in applying the workaround. |
| AVX-76413 | **Description: **When the Controller’s state synchronization service starts up and finds gateways stored as “Down” in its internal state store, the controller reprograms the network as if those gateways are down, without allowing time for the gateways to connect and prove they are up. This can occur when the controller has previously lost connectivity to gateways (for example, during a transient network issue) and the service then restarts while the gateways themselves remain healthy and continue forwarding traffic. **Impact: **A temporary data plane disruption occurs while the controller has the gateways marked as down. The controller automatically detects the gateways are up and reprograms the network correctly immediately afterward. In a large-scale user deployment (~1,400 gateways), full recovery completed in approximately 6 minutes. The underlying behavior has existed for 4+ years and has been observed in a user environment only once. **Workaround: **Not applicable. The system recovers automatically, no user action required. Recovery typically completes within minutes (~6 min observed in a ~1,400-gateway deployment; smaller deployments recover faster). |
| AVX-77135 | When the Use SSL to connect option is enabled for LDAP configuration, VPN gateway configuration updates may not apply correctly. Impact:
Contact Aviatrix Support for assistance. |
| AVX-77618 | Running Controller diagnostics may report NTP-related errors that do not reflect the actual NTP synchronization status of the Controller. Impact:
Verify Controller time synchronization through other means before acting on NTP-related diagnostic errors. Contact Aviatrix Support for assistance. |
| AVX-77958 | When performing a gateway software rollback from version 8.2.20 to 8.1.40 on a Controller deployed in GCP, AWS and AWS GovCloud spoke gateways may fail to roll back successfully. The rollback may partially succeed, leaving affected gateways in an inconsistent state. Impact:
Retry the rollback, as the issue is timing-related. If the issue persists, contact Aviatrix Support for further assistance. |
| AVX-78616 | When a single gateway image-upgrade or delete request includes both members of an ActiveMesh-enabled HA pair (a primary gateway and its HA peer), the operation can partially fail. Impact: Gateways in the request are processed in parallel, so whichever gateway in the pair is drained second fails with an error similar to: Cannot drain <gateway-name> as it is the last undrained gateway in the group.Workaround: Submit gateway image-upgrade or delete requests with primary gateways and HA gateways grouped into separate requests, rather than including both members of an ActiveMesh HA pair together. Contact Aviatrix Support for assistance. |
| AVX-78859 | IP rules for a Site-to-Cloud mapped connection could be erased when the gateway started up. Impact: Traffic loss occurs for the affected Site-to-Cloud connection. The condition can persist across gateway reboots and does not resolve on its own; the gateway configuration must be reapplied. Workaround: Reconfigure the affected Site-to-Cloud mapped connection to restore the IP rules. Contact Aviatrix Support for assistance. |
| AVX-79173 | During a Controller-initiated image upgrade, an Aviatrix gateway can intermittently be left in a partially configured state because of a timing-related issue in the gateway’s post-upgrade configuration sequence. Impact: The affected gateway’s image upgrade is reported as only partially successful. The gateway remains in a partially configured state until an administrator intervenes to restore it to the UP state. Workaround: Retry the image upgrade on the affected gateway. Because the issue is timing-related, a subsequent retry typically succeeds without further action. |
| AVX-79905 | Automatic Controller Access Security Group Management enforcement could unintentionally disable Security Group Management when a transient error occurred during an enforcement check. Impact: When Security Group Management became disabled, all gateways experienced keep-alive failures and new gateways could not come up. Workaround: Manually re-enable Security Group Management if it becomes disabled. Contact Aviatrix Support for assistance. |
| AVX-80806 | A Controller upgrade from 8.1.x to 8.2.20 could fail during data migration with an IP-pool allocation error. Impact: The Controller upgrade aborts and cannot complete until remediated. Workaround: Contact Aviatrix Support for assistance. A fix is planned for a future release. |
| AVX-79866 | NAT rules could fail to install on gateways with legacy tunnel peerings that predated recent tunnel-peering enhancements. Impact:
Re-establish the affected legacy tunnel peering so it picks up the current tunnel-peering fields. Contact Aviatrix Support for assistance identifying affected peerings. |
| AVX-80983 | When Azure Security Group Management reads the current state of a security group rule before adding a new gateway’s IP address to it during gateway launch, an Azure API throttling response on that read can be misinterpreted as the rule not existing. In that case, the existing rule is deleted and a new rule is created instead of the existing rule being updated. Impact:
Contact Aviatrix Support if you observe unexpected security group rule changes on gateway launch in an Azure environment. |
| AVX-79860 | New gateway deployment, or a software or image upgrade for an existing gateway, could fail if a container responsible for gateway-to-Controller secure communication did not start within the default 15-minute initialization window. Impact:
Retry the gateway deployment or upgrade. Contact Aviatrix Support for assistance. |
| AVX-80977 | A VPN user’s profile assignment could appear as N/A immediately after the user was attached to a profile through the API, until the user’s next reconnection. Impact:
Have the affected user reconnect to update the displayed profile assignment. Contact Aviatrix Support for assistance. |
| AVX-81892 | On gateways enforcing Distributed Cloud Firewall (DCF) policies that include SmartGroups with FQDN selectors, an internal policy-update path can deadlock in rare cases, causing the gateway data-plane process to stop forwarding traffic while remaining alive. The deadlock is more likely when the FQDN selector resolves to a set of addresses that rotate frequently (short DNS TTL), because each resolution change triggers a policy reload that races with an internal hit-counter update. Impact:
|