Corrected Issues in Aviatrix Release 8.1.30
| Issue | Description |
|---|---|
| AVX-67571 | Fixed an issue where OpenVPN clients could not connect to VPN gateways configured with DUO multi-factor authentication (MFA) in Oracle Cloud Infrastructure (OCI) environments due to ECONNREFUSED errors during tunnel establishment. |
| AVX-68013 | Fixed an issue where spoke-to-transit gateway attachment could fail with a check_task_status decode error, preventing successful attachment completion. |
| AVX-68726 | Fixed an issue where Azure Controller Security Group management operations could fail, preventing proper network security group rule updates. |
| AVX-69342 | Fixed an issue where duplicate resource ID entries could be created in the database when a Controller experienced out-of-memory conditions followed by upsizing and restart, preventing the Controller from starting properly. The database now handles resource IDs correctly to avoid duplicates. |
| AVX-70253 | Fixed an issue where FireNet deployment with bootstrap configuration failed in Google Cloud due to a change in how the credential JSON file is read. |
| AVX-71087 | Fixed an issue where the default access control rules did not properly allow ICMP traffic used for debugging. The updated rules ensure ICMP-based troubleshooting continues to work after upgrades. |
| AVX-71122 | Fixed an issue in some environments where the Aviatrix Controller failed to fetch and update a rotated SAML signing certificate from the configured Identity Provider (IdP) metadata URL, which caused SAML single sign-on (SSO) authentication failures. The Controller now correctly retrieves and applies updated SAML certificates after IdP rotation. |
| AVX-71135 | Resolved an issue where upgrading to Controller 8.1 failed during database migration if VPC tunnel records contained non-numeric values in the timestamp field. The migration logic now correctly handles timestamp values, preventing conversion errors and allowing the upgrade to complete successfully. |
| AVX-71217 | Fixed an issue where the VRRP state file became empty on AEP edge gateways configured in active-active HA pairs after upgrading gateway software from version 7.2 to 8.0.30. The VRRP state file now correctly retains primary/backup state information after upgrades. |
| AVX-71630 | Resolved an issue where incorrect eBPF filters could be applied to the eth1 interface on Azure gateways with Accelerated Networking enabled during upgrades from versions earlier than 7.2.2994. The upgrade process now properly handles interface filter configuration to prevent unintended traffic drops. |
| AVX-71672 | Fixed an issue where upgrading the Controller to version 8.1 could fail during database migration when the tunnel rtt_avg field contained None values. The migration logic now correctly handles None values, allowing the upgrade to complete successfully. |
| AVX-71807 | Fixed an issue where the packet mark eBPF program was not loaded on some gateways, potentially causing incorrect traffic classification. |
| AVX-71826 | Fixed an issue where in Aviatrix software versions 8.1.x and 8.2.0, the VRRP state file may be empty on AEP and self-managed Edge-as-Spoke gateways configured in active-active HA pairs. This prevents VRRP state updates from being sent from the edge gateways to the Aviatrix Controller, and Aviatrix CoPilot will not display the updated VRRP states. This is a cosmetic issue and there will be no disruption to traffic. |
| AVX-72207 | Fixed an issue where OpenVPN gateway profiles containing FQDN-based policies could cause service disruption after upgrading to Controller 8.1, due to a DNS resolution limitation. VPN users can now access whitelisted FQDNs in OpenVPN profiles after upgrading. |
| AVX-72369 | Fixed an issue where gateways included in custom syslog profiles were removed from those profiles after a gateway image upgrade, causing syslog forwarding configurations to be lost. |
| AVX-72835 | Fixed an issue where the database migration during Controller upgrade to 8.1 could fail if a High Availability Gateway (HAGW) entry appeared before its corresponding primary gateway in the vpc_info database collection. The migration logic now correctly handles HAGW records regardless of document ordering in the database. |
| AVX-72847 | Fixed an issue where the gateway state synchronization service leaked D-Bus connections over time, which could lead to resource exhaustion and degraded gateway state synchronization. |
| AVX-72871 | Fixed an issue where upgrading the Controller from version 8.0.x to 8.1.x could fail with an error indicating that the page must be reloaded to complete the upgrade, due to a database migration issue with incorrectly typed field values. The migration logic now includes proper type checking to handle these records. |
| AVX-73001 | Fixed an issue where upgrading Spoke Gateways to version 8.1.20 in environments using Transit FireNet with Egress through Firewall and customized SNAT policies could result in loss of the default route, causing traffic disruption. |
| AVX-73061 | Fixed a memory leak in the Cloud Asset Inventory (CAI) service where the L1 cache did not remove network interfaces when cloud instances were deleted, causing memory consumption to grow over time. |
| AVX-73136 | Fixed an issue where agent certificate renewal failed for 8.0.x gateways managed by 8.1.x Controllers due to SPIRE version differences. Certificate renewal now works correctly in mixed-version deployments. |
| AVX-73377 | Fixed an issue where the traffic-processing service on FQDN-enabled gateways was interrupted and restarted during gateway software upgrade, causing a traffic outage of approximately 10 minutes after the upgrade completed. The upgrade process now correctly handles this service transition without extended traffic interruption. |
| AVX-73589 | Fixed an issue in some high-traffic environments using FQDN filtering where the NFQ process could stall due to a deadlock caused by a signal interrupting a thread already executing a non-reentrant function. |
| AVX-73629 | Fixed an issue where the VPC name field could be overwritten with incorrect data during the AM4.0 migration, causing affected VPC records to become unfindable via index lookups. The migration logic now correctly preserves the VPC name field. |
| AVX-73742 | Fixed two issues with single IP HA tunnel failover in Site-to-Cloud deployments. The Controller no longer sends stale IPsec session teardown messages to gateways that temporarily lost connectivity, and now stops failover of pending tunnels to the HA gateway if the active gateway reconnects quickly, reducing unnecessary tunnel downtime. |
| AVX-74055 | Fixed an issue where duplicate iptables mangle table MARK rules could remain on gateways during mapped Site-to-Cloud tunnel failover, gateway image upgrade, or rollback scenarios. |
| AVX-74146 | Fixed an issue where prolonged traffic loss could occur during a Controller software upgrade from version 8.1.20 to 8.1.30. |
| AVX-74251 | Fixed an issue where some Azure gateways displayed the Image Version without the Image Build timestamp, making it difficult to identify the exact gateway image. |
| AVX-74418 | Fixed an issue where the Controller did not properly handle BGP route updates during transit gateway failover, causing stale routes to persist in the routing table. The Controller now correctly clears and repropagates routes after failover completes. |
| AVX-74465 | Fixed an issue where Aviatrix HPE gateway (including HA gateway) creation failed in OCI VCNs with DNS disabled. Gateways can now be created regardless of VCN DNS configuration. |
| AVX-74719 | Fixed an issue where performing a Controller backup restore could cause a temporary traffic outage of approximately 40 seconds due to all routes being deleted and re-added during route reconvergence. Routes are now preserved during backup restore to prevent traffic disruption. |
| AVX-74739 | Fixed an issue where the database migration timeout during Controller upgrade was hard-coded at 15 minutes, causing upgrades to fail and roll back in large-scale deployments with thousands of gateways and tunnels. The migration timeout is now user-configurable. |
| AVX-74988 | Fixed an issue where Edge-as-a-Transit (EaT) gateways with HPE peering using many-to-one IP addressing could fail to report tunnel status to the Controller due to duplicate tunnel ping IP pairs in the monitoring job. |
| AVX-74990 | Fixed an issue where controller software upgrade from version 8.0.40/8.0.50 to 8.1.20 may cause Controller CPU load to spike due to a schema migration being skipped during the upgrade. This can result in sluggish Controller UI performance. |
| AVX-75135 | Fixed an issue where tunnel status report processing on the Controller took longer after upgrading from version 8.0 to 8.1 due to increased database query overhead, with average processing time increasing from approximately 150ms to 230ms per report. |
| AVX-75256 | Fixed an issue where FQDN gateway data was not correctly displayed after upgrading from version 7.2.x to 8.0 or later, causing the Egress FQDN Gateway View to appear empty. Gateways with FQDN tags now display correctly in the UI and are returned properly by the list_fqdn_gateways API. |
| AVX-75496 | Fixed an issue where a Controller upgrade could fail during a database migration step if a corrupted VPC record was present in the Controller database, causing the upgrade to fail and trigger a rollback. |
| AVX-75944 | Fixed an issue where migrating an existing Controller to a new Controller VM could temporarily disrupt datapath on gateways with NAT configured due to configuration changes being applied too early during database migration. |
Known Issues in Aviatrix Release 8.1.30
| Issue | Description |
|---|---|
| AVX-62003 | Azure gateway image upgrades may fail when the Controller does not have the required Azure image subscription access. During the upgrade, the system deletes the existing gateway before validating subscription availability, which can result in gateway deletion without a replacement being created. This leaves dangling gateways in the Controller and can cause potential service outages. Impact: Existing gateways may be deleted during image upgrade Replacement gateway creation fails due to missing subscription Customers may experience connectivity loss and dangling gateway entries in the Controller Manual intervention required, leading to support escalations Workaround: None. To avoid outages, ensure the Controller subscription includes access to the required Azure image before attempting upgrades. |
| AVX-62299 | When upgrading from Controller version 7.1 to 7.2 or 8.0, Spoke Gateways with routing through a Public Subnet Filtering (PSF) Gateway may fail to upgrade and become unreachable if the PSF Gateway has not been upgraded first. This issue affects AWS environments where Spoke Gateway route tables are configured to point to a PSF Gateway. To avoid this issue, follow the correct upgrade sequence:1. Upgrade the PSF Gateway first. 2. Wait for the PSF Gateway upgrade to complete successfully. 3. Then upgrade the dependent Spoke Gateways. |
| AVX-62506 | During a gateway software upgrade, traffic matching DCF WebGroup rules may be briefly dropped during the upgrade. This impacts both Layer 7 (HTTP/HTTPS) and Layer 4 traffic and occurs across all supported cloud providers (AWS, Azure, and GCP). The disruption typically lasts a few seconds but may vary depending on gateway load and policy complexity. Workaround: None Recommendations: Schedule gateway upgrades during maintenance windows or low-traffic periods. Use HA deployments and upgrade gateways one at a time in HA pairs. Monitor logs for “Failed to load policy” messages to confirm when policies are reloaded. |
| AVX-64868 | In some scenarios involving rapid VRRP state transitions, the keepalived VRRP state may not be reported accurately to the Controller. This can result in temporary discrepancies between the actual VRRP status and what is displayed in the Controller UI, leading to confusion and difficulties during troubleshooting. Impact: Controller UI may show incorrect VRRP status such as both peers reporting Primary or Initializing No impact on actual VRRP traffic handling or failover behavior. Workaround: Use diagnostic logs to verify actual VRRP state. |
| AVX-65016 | In some environments, the Firewall state may not recover from Unaccessible after the first vendor integration failure. This issue has been observed when integrating with third-party firewall vendors, leaving the gateway firewall state stuck even after the environment stabilizes. Impact: Firewall integration appears stuck in Unaccessible state Recovery does not occur automatically after initial failure May require manual intervention to restore proper firewall state reporting Workaround: Contact Aviatrix Support for manual correction. |
| AVX-66631 | Transit gateways with large-scale tunnel deployments (1300+ tunnels) may experience extended traffic loss during image upgrades. Although the image upgrade completes successfully, traffic may remain down for several minutes afterward due to delayed tunnel reconfiguration. Impact: Traffic loss may persist after image upgrade completes Route service startup is blocked until all tunnels are sequentially reconfigured Configuration push may time out with Context cancelled during Phase 1 Create errorWorkaround: Schedule maintenance windows to account for potential traffic loss beyond upgrade completion. Consider staggering upgrades across transit gateways to reduce impact. Monitor tunnel and route service status post-upgrade through the CoPilot UI. |
| AVX-67126 | Dry-run validation may fail when upgrading the Controller from version 8.0.10 to 8.1.0 due to a gateway version mismatch error. This occurs when the upgrade path starts from 8.0.0, progresses to 8.0.10 successfully, but encounters a dry-run failure when proceeding to 8.1.0. |
| AVX-67180 | In some environments running Aviatrix Controller version 8.1.x, the Controller UI may become inaccessible after the Controller is restarted. When this issue occurs, API requests fail with RequestRefused errors and the UI cannot be accessed, although SSH access to the Controller remains available.Impact: Users are unable to log in to the Controller UI or perform API operations. Workaround: Restart the VM where the Aviatrix Controller is running. |
| AVX-68108 | When upgrading the Controller from version 8.0.30 to 8.1.10, the UI may display a misleading “Service temporarily unavailable” error message immediately after the upgrade begins. This message can persist for 5–10 minutes but does not indicate upgrade failure. The upgrade continues normally in the background and the Controller becomes accessible again once the upgrade finishes. Impact: Users may believe the upgrade has failed. Error message persists for 5–10 minutes, especially in larger deployments (50+ gateways). No effect on upgrade success or Controller functionality. Workaround: Ignore the message during upgrade. Wait 10–15 minutes for the process to complete. Refresh the browser and verify the new Controller version after reconnection. |
| AVX-68561 | In large-scale deployments with 1300+ gateways, enabling Distributed Cloud Firewall Site-to-Cloud (DCF S2C) can cause gateway configurations to become out of sync with the Controller. Even after disabling DCF S2C, the issue may persist and lead to elevated Controller resource usage. Impact: Gateway configurations may show as out of sync in the Controller UI Controller CPU load (conduit process) increases significantly Performance degradation may occur during DCF S2C operations Issue may persist after disabling DCF S2C Workaround: Monitor Controller CPU usage before enabling DCF S2C in large-scale environments. Consider enabling DCF S2C during scheduled maintenance windows. For deployments with 1300+ gateways, evaluate the necessity of DCF S2C functionality. |
| AVX-68887 | When attaching VPN users to profiles using the attach_vpn_user_to_profile API, the CoPilot or Controller UI may continue to display the user profile as N/A even though the attachment operation completes successfully.Impact: VPN user profile assignments may appear unsuccessful in the UI, which can cause confusion during profile management. There is no functional impact: the VPN profile is correctly assigned in the backend, and users can connect to the VPN as expected. Affected Scenario: OpenVPN profile management operations that use API-based user-to-profile attachment. Workaround: None. |
| AVX-69649 | The migration dry-run EIP accounting does not include public IPs that are not part of the Elastic IP quota, potentially producing inaccurate dry-run results. Impact:
Manually verify EIP allocation and quotas before performing the migration. Contact Aviatrix Support for assistance. |
| AVX-71820 | When deploying a load balancer–enabled VPN gateway with an overlapping VPN CIDR on Controller versions 8.0, 8.1, or 8.2, the gateway creation fails. Impact: VPN gateway deployment fails Error message does not clearly indicate the root cause Workaround: Ensure that the VPN CIDR does not overlap with existing gateways behind the load balancer before deployment. Contact Aviatrix Support for assistance. |
| AVX-72553 | The SAML Endpoint field is not displayed when creating a VPN user for GeoVPN configurations with a SAML-enabled gateway. Impact:
Contact Aviatrix Support for assistance. |
| AVX-72940 | Creating a new gateway with the same name as an existing gateway may cause local files of the existing gateway to be deleted when the creation fails. The existing gateway name disappears from the Controller CLI when this issue occurs. Impact: SSH access to the existing gateway may stop working Gateway recovery may require manual intervention Workaround: Do not reuse gateway names. Contact Aviatrix Support if recovery is required. |
| AVX-73433 | After changing rx_queue_size on a gateway interface, creating a new interface does not apply the updated rx_queue_size value.Impact:
Contact Aviatrix Support for assistance. |
| AVX-73436 | When using the update_spoke_vpc_route_table API to onboard an Azure route table, the default route (0.0.0.0/0) is not programmed in the spoke VNET route table if the Spoke Gateway has learned the default route from any of the following sources:
Impact:
Manually add the default route to the Azure route table. Contact Aviatrix Support for assistance. |
| AVX-74226 | CoPilot deployments and migrations may fail with “Unsupported instance size” errors when selecting valid instance types. The instance type validation incorrectly blocks supported sizes during CoPilot deployment or migration operations. Impact: CoPilot deployment or migration may fail when selecting certain valid instance types Error message “Unsupported instance size” is displayed even for supported sizes Workaround: Contact Aviatrix Support for assistance with CoPilot deployment using the affected instance types. |
| AVX-74577 | Users are unable to modify tags on third-party firewall instances when those tags contain values with multiple colons (for example, team:iac:module.version:v1.5.3). Attempts to update tags after deployment fail with a too many values to unpack error. Initial deployment is unaffected because tags are passed via a different code path during creation.Impact: Third-party firewall instance tag updates fail when tag values contain multiple colons Initial deployment with multi-colon tags is not affected Workaround: Avoid using multiple colons in tag values when modifying tags after deployment. Use alternative delimiters such as hyphens or underscores. |
| AVX-75452 | In Azure environments, when Distributed Cloud Firewall (DCF) Security Group Orchestration attaches a Network Security Group (NSG) to a subnet, the subnet name is changed to all lowercase. Although Azure resource names are generally case-insensitive, this modification causes issues with Infrastructure as Code tools such as Terraform, which treat resource names as case-sensitive. Terraform may flag affected subnets for replacement, potentially disrupting existing deployments. Impact: Subnet names in Azure are modified to lowercase when Security Group Orchestration attaches NSGs Terraform plans may show unexpected resource replacements for affected subnets Customer naming conventions in the cloud may be altered without consent Workaround: In Terraform, add a lifecycle block with ignore_changes for subnet_id (as well as id in the azurerm_subnet resource) to prevent forced resource replacement. Note that this workaround does not restore the original subnet name casing. |
| AVX-75607 | Gateway launch may fail with a tls: bad certificate error when pulling container images. The Controller’s registry TTL eviction (garbage collection) may fire while images are being downloaded to the gateway, corrupting in-flight blob transfers. The gateway’s container initialization cannot complete, and a supporting certificate-management process on the gateway exits with code 125.Impact:
Restart the Controller’s application service to reset the registry state, then retry gateway creation. Contact Aviatrix Support for assistance. |
| AVX-75869 | In OCI environments, security list rules may not be restored when a spoke gateway re-joins a transit gateway after a leave operation. Impact:
Manually verify and restore security list rules after re-joining. Contact Aviatrix Support for assistance. |
| AVX-75872 | During a Controller upgrade, a locking race condition between the initial setup process and post-upgrade actions can cause post-upgrade configuration steps to fail or require manual intervention. Impact:
Contact Aviatrix Support to manually complete the post-upgrade configuration. |
| AVX-76132 | Unable to configure more than one OpenVPN gateway behind a UDP Load Balancer. Only the first gateway is retained while additional gateways are incorrectly excluded. This issue affects versions 6.9, 7.x, 8.x, and 9.0.0. Impact:
Contact Aviatrix Support for assistance in applying the workaround. |
| AVX-76296 | During routine gateway operations (such as resize or image upgrade) in GCP global VPC environments, traffic may be blackholed due to incorrect route handling. Impact:
Schedule gateway operations during maintenance windows. Contact Aviatrix Support for assistance. |
| AVX-76413 | When the Controller’s state synchronization service starts up and finds gateways stored as “Down” in its internal state store, the controller reprograms the network as if those gateways are down, without allowing time for the gateways to connect and prove they are up. This can occur when the controller has previously lost connectivity to gateways (for example, during a transient network issue) and the service then restarts while the gateways themselves remain healthy and continue forwarding traffic. Impact: A temporary data plane disruption occurs while the controller has the gateways marked as down. The controller automatically detects the gateways are up and reprograms the network correctly immediately afterward. In a large-scale user deployment (~1,400 gateways), full recovery completed in approximately 6 minutes. The underlying behavior has existed for 4+ years and has been observed in a user environment only once. Workaround: Not applicable. The system recovers automatically, no user action required. Recovery typically completes within minutes (~6 min observed in a ~1,400-gateway deployment; smaller deployments recover faster). |
| AVX-76719 | During a Controller restore operation, the restore process may stall for approximately 30 minutes during the scheduler shutdown phase due to connectivity issues with AWS API endpoints or invalid/expired IAM credentials. The restore completes successfully after the stall. Impact:
Wait for the restore to complete. The stall is temporary and does not affect the final result. Contact Aviatrix Support for assistance if the restore appears stuck for more than 60 minutes. |
| AVX-77088 | On Controller and gateway running 8.1.x or 8.2.x, editing legacy FQDN domain name filters can cause all FQDN filtering processes on the gateway to stop simultaneously. Gateway monitoring restarts the processes automatically, but a brief filtering outage may occur during the restart. Impact:
Schedule edits to legacy FQDN filters during a maintenance window. Contact Aviatrix Support for assistance. |
| AVX-79173 | During a Controller-initiated image upgrade, an Aviatrix gateway can intermittently be left in a partially configured state because of a timing-related issue in the gateway’s post-upgrade configuration sequence. Impact: The affected gateway’s image upgrade is reported as only partially successful. The gateway remains in a partially configured state until an administrator intervenes to restore it to the UP state. Workaround: Retry the image upgrade on the affected gateway. Because the issue is timing-related, a subsequent retry typically succeeds without further action. |
| AVX-80983 | When Azure Security Group Management reads the current state of a security group rule before adding a new gateway’s IP address to it during gateway launch, an Azure API throttling response on that read can be misinterpreted as the rule not existing. In that case, the existing rule is deleted and a new rule is created instead of the existing rule being updated. Impact: Azure security group rules can be unexpectedly deleted and recreated during gateway launch when the Azure API throttles the read call. The delete-and-recreate sequence can result in a brief gap in security group coverage for the affected rule, or in duplicate rules if the sequence does not fully complete. Workaround: Contact Aviatrix Support if you observe unexpected security group rule changes on gateway launch in an Azure environment. |
| AVX-79860 | New gateway deployment, or a software or image upgrade for an existing gateway, could fail if a container responsible for gateway-to-Controller secure communication did not start within the default 15-minute initialization window. Impact: Gateway deployment or upgrade fails and the gateway does not become operational. Retrying the deployment or upgrade before the initialization window expires may succeed. Workaround: Retry the gateway deployment or upgrade. Contact Aviatrix Support for assistance. |
| AVX-80977 | A VPN user’s profile assignment could appear as N/A immediately after the user was attached to a profile through the API, until the user’s next reconnection. Impact: The user’s profile displays as N/A until the user’s next reconnection, even though the profile attachment succeeded. This can cause confusion when auditing or verifying profile-based access policies shortly after attachment. Workaround: Have the affected user reconnect to update the displayed profile assignment. Contact Aviatrix Support for assistance. |