Corrected Issues in Aviatrix Release 9.0.0
| Issue | Description |
|---|---|
| AVX-62003 | Fixed an issue where Azure gateway image upgrades could fail when the Controller did not have the required Azure image subscription access, resulting in gateway deletion without a replacement being created. The upgrade process now validates subscription access before proceeding. |
| AVX-62506 | Fixed an issue where during a gateway software upgrade, traffic matching DCF WebGroup rules may be briefly dropped during the upgrade. This impacts both Layer 7 (HTTP/HTTPS) and Layer 4 traffic and occurs across all supported cloud providers (AWS, Azure, and GCP). The disruption typically lasts a few seconds but may vary depending on gateway load and policy complexity. |
| AVX-67180 | Fixed an issue where users could receive a RequestRefused error when attempting to log on to the Controller UI under certain conditions. |
| AVX-67571 | Fixed an issue where OpenVPN clients could not connect to VPN gateways configured with DUO multi-factor authentication (MFA) in Oracle Cloud Infrastructure (OCI) environments due to ECONNREFUSED errors during tunnel establishment. |
| AVX-68561 | Fixed an issue where enabling Distributed Cloud Firewall Site-to-Cloud (DCF S2C) in large-scale deployments with 1300+ gateways caused gateway configurations to become out of sync with the Controller and elevated Controller CPU load. |
| AVX-68887 | Fixed an issue where VPN user profile assignments appeared as N/A in the CoPilot or Controller UI after attaching users to profiles via the attach_vpn_user_to_profile API, even though the assignment completed successfully in the backend. |
| AVX-69649 | Fixed an issue where the migration dry-run EIP accounting did not include public IPs that are not part of the Elastic IP quota, potentially causing inaccurate dry-run results. |
| AVX-71122 | Fixed an issue where the Controller failed to fetch and update new SAML signing certificates after Identity Provider (IdP) certificate rotation, causing SAML single sign-on (SSO) authentication failures. |
| AVX-71245 | Resolved an issue where DCF (Distributed Cloud Firewall) end session logs for IDS and IPS signature matches were omitted when Decryption was not enabled. The reason field in the logs showed only POLICY instead of the specific values IPS_POLICY_DENY or IDS_POLICY_ALERT. Decryption needed to be enabled for the reason field to show complete values. There was no impact to DCF policy actions and logs under Intrusion Analysis. |
| AVX-71441 | Fixed an issue where gateways upgrading from version 8.1.20 to 8.2.0 could enter an infinite retry loop attempting to download a non-existent configuration file from the Controller, causing the upgrade process to fail. |
| AVX-71630 | Resolved an issue where incorrect eBPF filters could be applied to the eth1 interface on Azure gateways with Accelerated Networking enabled during upgrades from versions earlier than 7.2.2994. The upgrade process now properly handles interface filter configuration to prevent unintended traffic drops. |
| AVX-71672 | Fixed an issue where upgrading the Controller to version 8.1 could fail during database migration if the tunnel rtt_avg field contained None values. The migration logic now correctly handles these values, preventing conversion errors and allowing the upgrade to complete successfully. |
| AVX-71719 | Fixed an issue where ICMP traffic passing through Suricata inspection on gateways only triggered alert rules once until the Suricata process restarted. Alert rules now fire consistently for all matching traffic. |
| AVX-71826 | Fixed an issue where in Aviatrix software versions 8.1.x and 8.2.0, the VRRP state file /etc/localgateway/vrrp_state.json, may be empty on AEP and self-managed Edge-as-Spoke gateways configured in active-active HA pairs. This prevents VRRP state updates from being sent from the edge gateways to the Aviatrix Controller, and Aviatrix CoPilot will not display the updated VRRP states. This is a cosmetic issue and there will be no disruption to traffic. |
| AVX-72207 | Fixed an issue where upgrading OpenVPN gateways to Controller version 8.1 with profiles containing FQDN-based policies resulted in service disruption due to a DNS resolution limitation. Users can now access whitelisted FQDNs in OpenVPN Profiles after upgrading. |
| AVX-72369 | Fixed an issue where gateways included in custom syslog profiles were removed from those profiles after a gateway image upgrade, causing syslog forwarding configurations to be lost. |
| AVX-72835 | Fixed an issue where the database migration during Controller upgrade to 8.1 could fail if a High Availability Gateway (HAGW) entry appeared before its corresponding primary gateway in the vpc_info database collection. The migration logic now correctly handles HAGW records regardless of document ordering in the database. |
| AVX-72847 | Fixed an issue where the avx-gw-state-sync service leaked D-Bus connections over time, which could lead to resource exhaustion and degraded gateway state synchronization. |
| AVX-72871 | Fixed an issue where upgrading the Controller from version 8.0.x to 8.1.x could fail with an error indicating that the page must be reloaded to complete the upgrade, due to a database migration issue with incorrectly typed field values. The migration logic now includes proper type checking to handle these records. |
| AVX-72881 | Fixed an issue where during upgrades from older Aviatrix Controller versions to versions 8.1.20 and 8.2.0, the NetflowMode migration may fail due to incomplete NetflowMode records in the database. Older Controller versions allowed NetflowMode records with missing or empty fields, such as an empty port value. Newer releases enforce stricter validation and fail when encountering these incomplete records. |
| AVX-73061 | Fixed a memory leak in the Cloud Asset Inventory (CAI) service where the L1 cache did not remove network interfaces when cloud instances were deleted, causing memory consumption to grow over time. |
| AVX-73377 | Fixed an issue where avx-nfq processes on FQDN-enabled gateways were killed and restarted during gateway software upgrade, causing a traffic outage of approximately 10 minutes after the upgrade completed. The upgrade process now correctly handles the nfq service transition without extended traffic interruption. |
| AVX-73629 | Fixed an issue where the VPC name field could be overwritten with incorrect data during the AM4.0 migration, causing affected VPC records to become unfindable via index lookups. The migration logic now correctly preserves the VPC name field. |
| AVX-73742 | Fixed two issues with single IP HA tunnel failover in Site-to-Cloud deployments. The Controller no longer sends stale IPsec session teardown messages to gateways that temporarily lost connectivity, and now stops failover of pending tunnels to the HA gateway if the active gateway reconnects quickly, reducing unnecessary tunnel downtime. |
| AVX-74055 | Fixed an issue where duplicate iptables mangle table MARK rules could remain on gateways during mapped Site-to-Cloud tunnel failover, gateway image upgrade, or rollback scenarios. |
| AVX-74226 | Fixed an issue where CoPilot deployments and migrations could fail with “Unsupported instance size” errors for valid instance types. Instance type validation no longer incorrectly blocks supported sizes. |
| AVX-74418 | Fixed an issue where the Controller did not properly handle BGP route updates during transit gateway failover, causing stale routes to persist in the routing table. The Controller now correctly clears and repropagates routes after failover completes. |
| AVX-74465 | Fixed an issue where Aviatrix HPE gateway (including HA gateway) creation failed in OCI VCNs with DNS disabled. Gateways can now be created regardless of VCN DNS configuration. |
| AVX-74577 | Fixed an issue where modifying tags on third-party firewall instances failed with a too many values to unpack error when tag values contained multiple colons (for example, team:iac:module.version:v1.5.3). The tag parsing logic now correctly handles multi-colon values by using a bounded split on the first colon delimiter. |
| AVX-74719 | Fixed an issue where performing a Controller backup restore could cause a temporary traffic outage of approximately 40 seconds due to all routes being deleted and re-added during route reconvergence. Routes are now preserved during backup restore to prevent traffic disruption. |
| AVX-74739 | Fixed an issue where the database migration timeout during Controller upgrade was hard-coded at 15 minutes, causing upgrades to fail and roll back in large-scale deployments with thousands of gateways and tunnels. The migration timeout is now user-configurable. |
| AVX-74990 | Fixed an issue where controller software upgrade from version 8.0.40/8.0.50 to 8.1.20 may cause Controller CPU load to spike due to a schema migration being skipped during the upgrade. This can result in sluggish Controller UI performance. |
| AVX-75117 | Fixed a memory leak in the TrafficServer (ATS) process on gateways with DCF intrusion analysis and decryption enabled under high-concurrency traffic conditions that could cause the ATS process to crash and enter a restart loop. |
| AVX-75135 | Fixed an issue where tunnel status report processing on the Controller took longer after upgrading from version 8.0 to 8.1 due to increased database query overhead, with average processing time increasing from approximately 150ms to 230ms per report. |
| AVX-75136 | Fixed an issue where enabling DCF intrusion analysis on gateways that do not meet the minimum memory requirement for Suricata may cause unexpected traffic drops. The Controller displays a notification that certain gateways do not meet memory requirements, but existing traffic through those gateways may still be disrupted. |
| AVX-75496 | Fixed an issue where a Controller upgrade could fail during a database migration step if a corrupted VPC record was present in the Controller database, causing the upgrade to fail and trigger a rollback. |
| AVX-75607 | Fixed an issue where gateway launch could fail with a tls: bad certificate error when pulling container images. The Controller’s registry TTL eviction could fire while images were being downloaded, corrupting in-flight blob transfers and causing the gateway’s container initialization to fail. |
| AVX-75944 | Fixed an issue where migrating an existing Controller to a new Controller VM could temporarily disrupt datapath on gateways with NAT configured due to configuration changes being applied too early during database migration. |
| AVX-77767 | Fixed an issue where Azure transit gateways with very large numbers of IPsec tunnels (3000+) could lose all spoke peerings and enter a high-CPU charon state under sustained high tunnel activity. |
Known Issues in Aviatrix Release 9.0.0
| Issue | Description |
|---|---|
| AVX-62299 | When upgrading from Controller version 7.1 to 7.2 or 8.0, Spoke Gateways with routing through a Public Subnet Filtering (PSF) Gateway may fail to upgrade and become unreachable if the PSF Gateway has not been upgraded first. This issue affects AWS environments where Spoke Gateway route tables are configured to point to a PSF Gateway. To avoid this issue, follow the correct upgrade sequence:1. Upgrade the PSF Gateway first. 2. Wait for the PSF Gateway upgrade to complete successfully. 3. Then upgrade the dependent Spoke Gateways. |
| AVX-64868 | In some scenarios involving rapid VRRP state transitions, the keepalived VRRP state may not be reported accurately to the Controller. This can result in temporary discrepancies between the actual VRRP status and what is displayed in the Controller UI, leading to confusion and difficulties during troubleshooting. Impact:
|
| AVX-65016 | In some environments, the Firewall state may not recover from Unaccessible after the first vendor integration failure. This issue has been observed when integrating with third-party firewall vendors, leaving the gateway firewall state stuck even after the environment stabilizes. Impact: Firewall integration appears stuck in Unaccessible state Recovery does not occur automatically after initial failure May require manual intervention to restore proper firewall state reporting Workaround: Contact Aviatrix Support for manual correction. |
| AVX-66631 | Transit gateways with large-scale tunnel deployments (1300+ tunnels) may experience extended traffic loss during image upgrades. Although the image upgrade completes successfully, traffic may remain down for several minutes afterward due to delayed tunnel reconfiguration. Impact:
|
| AVX-67126 | Dry-run validation may fail when upgrading the Controller from version 8.0.10 to 8.1.0 due to a gateway version mismatch error. This occurs when the upgrade path starts from 8.0.0, progresses to 8.0.10 successfully, but encounters a dry-run failure when proceeding to 8.1.0. |
| AVX-70543 | When DPI/IDS or Layer7 policies are configured with “Destination: Anywhere” on HA-enabled spoke gateways where the destination smart group contains private CIDRs, the policies become invalid and cause traffic drops. Impact: All egress traffic matching the policy rules gets dropped Network connectivity loss for affected traffic flows Workaround: Modify the policy destination from “Anywhere” to specific target destinations that exclude conflicting private CIDR ranges, or disable HA on the affected spoke gateway if operationally acceptable. |
| AVX-70864 | DCF smart group push configuration may fail with a context deadline exceeded error, causing DFW configuration to not be applied correctly to gateways.Impact:
Contact Aviatrix Support for assistance. |
| AVX-70958 | When clients use HTTP/2 connections, TrafficServer incorrectly reuses origin connections, potentially causing connection handling issues in MITM SNI verification scenarios. Impact:
Use both IP address and SNI instead of IP alone to ensure proper connection isolation. |
| AVX-71320 | Source IP translation may not work correctly for policy-based Site-to-Cloud connections with SNAT configuration when traffic is sent from cloud to site. Impact:
No workaround is available for this issue. Upgrade to Controller 10.1.0 or later, where this issue is fixed. |
| AVX-71559 | When performing a batch gateway image upgrade in Azure, some gateways may fail during the upgrade process. Impact:
Retry the gateway image upgrade for the failed gateways individually rather than as a batch operation. Contact Aviatrix Support for assistance. |
| AVX-71820 | When deploying a load balancer–enabled VPN gateway with an overlapping VPN CIDR on Controller versions 8.0, 8.1, or 8.2, the gateway creation fails. Impact:
Ensure that the VPN CIDR does not overlap with existing gateways behind the load balancer before deployment. Contact Aviatrix Support for assistance. |
| AVX-72553 | The SAML Endpoint field is not displayed when creating a VPN user for GeoVPN configurations with a SAML-enabled gateway. Impact:
Contact Aviatrix Support for assistance. |
| AVX-72940 | Creating a new gateway with the same name as an existing gateway may cause local files of the existing gateway to be deleted when the creation fails. The existing gateway name disappears from the Controller CLI when this issue occurs. Impact:
Do not reuse gateway names. Contact Aviatrix Support if recovery is required. |
| AVX-73334 | After upgrading a VPN gateway image to Controller 9.0 with FIPS 140-3 enabled, the Aviatrix VPN Client (AVPNC 2.17.7, which uses OpenVPN 2.4.9) cannot connect to the gateway and returns AUTH_FAILED — client incompatible with this server. FIPS 140-3 on the gateway side requires OpenVPN 2.6.0 or later (RFC 5705 TLS Keying Material Exporters), which AVPNC 2.17.7 does not support.Impact:
|
| AVX-73433 | After changing rx_queue_size on a gateway interface, creating a new interface does not apply the updated rx_queue_size value.Impact:
Contact Aviatrix Support for assistance. |
| AVX-73436 | When using the update_spoke_vpc_route_table API to onboard an Azure route table, the default route (0.0.0.0/0) is not programmed in the spoke VNET route table if the Spoke Gateway has learned the default route from any of the following sources:
Impact:
Manually add the default route to the Azure route table. Contact Aviatrix Support for assistance. |
| AVX-75051 | When upgrading FIPS-enabled gateways to version 9.0.0, the dry-run validation does not detect that an image upgrade is required for FIPS 140-3. The software upgrade correctly blocks FIPS-enabled gateways with an error requiring an image upgrade, but because the dry-run passes, customers may not discover the requirement until the upgrade phase, leading to partial upgrade failures. Impact:
Before upgrading to 9.0.0, perform image upgrades on all FIPS-enabled gateways to enable FIPS 140-3 mode. Do not rely on dry-run results to validate FIPS gateway upgrade readiness. Contact Aviatrix Support for assistance. |
| AVX-75452 | In Azure environments, when Distributed Cloud Firewall (DCF) Security Group Orchestration attaches a Network Security Group (NSG) to a subnet, the subnet name is changed to all lowercase. Although Azure resource names are generally case-insensitive, this modification causes issues with Infrastructure as Code tools such as Terraform, which treat resource names as case-sensitive. Terraform may flag affected subnets for replacement, potentially disrupting existing deployments. Impact:
In Terraform, add a lifecycle block with ignore_changes for subnet_id (as well as id in the azurerm_subnet resource) to prevent forced resource replacement. Note that this workaround does not restore the original subnet name casing. |
| AVX-75582 | In Azure environments, when a custom IAM policy blocks resource creation with a RequestDisallowedByPolicy error, the Aviatrix Controller unnecessarily retries the operation instead of failing immediately. Since this error requires manual policy changes to resolve, the repeated retries congest the Controller’s event handler, causing new gateway deployments to be delayed in reaching an operational state and spoke-to-transit attachments to fail.Impact:
Update the Azure IAM policy to allow the required permissions for Aviatrix Controller resource operations before deploying new gateways. |
| AVX-75586 | When using Terraform to create gateways and immediately attach them to transit gateways, a race condition can occur where the Terraform provider attempts to create the spoke-to-transit attachment before the gateway has fully transitioned to the “Up” state. The gateway creation API returns before the gateway is operationally ready, causing subsequent attachment operations to fail. Impact:
Add a delay or polling mechanism in Terraform configurations between gateway creation and spoke-to-transit attachment resources. Use depends_on with a time_sleep resource to allow the gateway to reach the “Up” state before attempting attachment. |
| AVX-75599 | The TrafficServer (ATS) process on gateways may crash with a signal 6 (Aborted) when a custom Suricata rule feed containing a large number of rules (20,000+) is applied to a DCF intrusion analysis profile. The crash occurs due to an assertion failure in the IDS inspection queue handling, causing traffic interruption on affected gateways. Impact:
Reduce the number of custom rules to under 1,000, or remove the custom rule feed from the intrusion analysis profile and use the system default IPS profile until a fix is available in a future release. Contact Aviatrix Support for assistance. |
| AVX-75805 | When upgrading the Controller from version 8.2.10 to 9.0.0, the default Decryption CA used by Distributed Cloud Firewall (DCF) is changed. Customers using DCF decryption features must reestablish trust with the new CA after the upgrade, which may cause temporary disruption to decrypted traffic flows. Impact:
After upgrading to 9.0.0, redistribute the new Decryption CA certificate to all endpoints that require trust. Plan for a maintenance window to account for the CA change during the upgrade. |
| AVX-75869 | In OCI environments, security list rules may not be restored when a spoke gateway re-joins a transit gateway after a leave operation. Impact:
Manually verify and restore security list rules after re-joining. Contact Aviatrix Support for assistance. |
| AVX-75872 | During a Controller upgrade, a locking race condition between the initial setup process and post-upgrade actions can cause post-upgrade configuration steps to fail or require manual intervention. Impact:
Contact Aviatrix Support to manually complete the post-upgrade configuration. |
| AVX-76132 | Unable to configure more than one OpenVPN gateway behind a UDP Load Balancer. Only the first gateway is retained while additional gateways are incorrectly excluded. This issue affects versions 6.9, 7.x, 8.x, and 9.0.0. Impact:
Contact Aviatrix Support for assistance in applying the workaround. |
| AVX-76296 | During routine gateway operations (such as resize or image upgrade) in GCP global VPC environments, traffic may be blackholed due to incorrect route handling. Impact:
Schedule gateway operations during maintenance windows. Contact Aviatrix Support for assistance. |
| AVX-76413 | When the avx-ctrl-state-sync service starts up and finds gateways stored as “Down” in etcd, the controller reprograms the network as if those gateways are down, without allowing time for the gateways to connect and prove they are up. This can occur when the controller has previously lost connectivity to gateways (for example, during a transient network issue) and the service then restarts while the gateways themselves remain healthy and continue forwarding traffic.Impact: A temporary data plane disruption occurs while the controller has the gateways marked as down. The controller automatically detects the gateways are up and reprograms the network correctly immediately afterward. In a large-scale user deployment (~1,400 gateways), full recovery completed in approximately 6 minutes. The underlying behavior has existed for 4+ years and has been observed in a user environment only once. Workaround: Not applicable. The system recovers automatically, no user action required. Recovery typically completes within minutes (~6 min observed in a ~1,400-gateway deployment; smaller deployments recover faster). |
| AVX-77011 | Existing user tags on Aviatrix gateway VM instances cannot be deleted from the Controller UI or CoPilot UI. Attempts to delete tags complete without an error message, but the tags remain on the underlying cloud provider VM instance. Adding new tags is not affected. The issue reproduces on both AWS and Azure gateways. Impact:
Remove the tags directly from the cloud provider console (AWS or Azure) on the gateway VM instance. Contact Aviatrix Support for assistance. |
| AVX-78616 | When a single gateway image-upgrade or delete request includes both members of an ActiveMesh-enabled HA pair (a primary gateway and its HA peer), the operation can partially fail. Impact: Gateways in the request are processed in parallel, so whichever gateway in the pair is drained second fails with an error similar to: Cannot drain <gateway-name> as it is the last undrained gateway in the group.Workaround: Submit gateway image-upgrade or delete requests with primary gateways and HA gateways grouped into separate requests, rather than including both members of an ActiveMesh HA pair together. Contact Aviatrix Support for assistance. |
| AVX-78671 | Enabling Logging on the Default Action Rule and then disabling it can cause the Monitor tab to continue showing log entries for that rule for up to one hour after Logging is turned off. Impact: * After you disable Logging on the Default Action Rule (following a period when Logging was enabled and traffic was flowing), the Monitor tab continues to display log entries for that rule for up to one hour. * Excess log volume during that hour can make it harder to review logs for other Distributed Cloud Firewall (DCF) rules. Workaround: No action is required. Logging for the Default Action Rule stops automatically once the one-hour period elapses. Until then, you can filter Monitor tab entries by rule name to distinguish residual Default Action Rule logs from current rule activity. |
| AVX-79173 | During a Controller-initiated image upgrade, an Aviatrix gateway can intermittently be left in a partially configured state because of a timing-related issue in the gateway’s post-upgrade configuration sequence. Impact: The affected gateway’s image upgrade is reported as only partially successful. The gateway remains in a partially configured state until an administrator intervenes to restore it to the UP state. Workaround: Retry the image upgrade on the affected gateway. Because the issue is timing-related, a subsequent retry typically succeeds without further action. |
| AVX-79521 | On Controller versions prior to 10.1.0, it is possible to onboard multiple Aviatrix access accounts that map to the same underlying cloud service provider (CSP) account. Because several Controller features assume a one-to-one relationship between an Aviatrix access account and a CSP account, duplicate access accounts can lead to unpredictable configuration and operational behavior. Duplicate is defined per CSP as follows:
Impact:
Onboard only one Aviatrix access account per underlying CSP account. If duplicate access accounts have already been onboarded and are causing issues, contact Aviatrix Support for assistance with identifying and removing the duplicates. |
| AVX-79855 | When Distributed Cloud Firewall is disabled on a transit gateway with Site2Cloud (S2C) or other external connections landing on it, and no policy change is queued at the time, existing DCF filtering rules on the interface are not removed. The disable path is skipped when there is no policy difference to push. Impact:
Restart the Aviatrix gateway to remove the stale DCF filter rules. Alternatively, contact Aviatrix Support to have the stale DCF filter rules manually removed. |
| AVX-80983 | When Azure Security Group Management reads the current state of a security group rule before adding a new gateway’s IP address to it during gateway launch, an Azure API throttling response on that read can be misinterpreted as the rule not existing. In that case, the existing rule is deleted and a new rule is created instead of the existing rule being updated. Impact:
Contact Aviatrix Support if you observe unexpected security group rule changes on gateway launch in an Azure environment. |
| AVX-79860 | New gateway deployment, or a software or image upgrade for an existing gateway, could fail if a container responsible for gateway-to-Controller secure communication did not start within the default 15-minute initialization window. Impact:
Retry the gateway deployment or upgrade. Contact Aviatrix Support for assistance. |
| AVX-80977 | A VPN user’s profile assignment could appear as N/A immediately after the user was attached to a profile through the API, until the user’s next reconnection. Impact:
Have the affected user reconnect to update the displayed profile assignment. Contact Aviatrix Support for assistance. |