Performance Improvement with Gateway Scaling

About Aviatrix Gateway Scaling

The gateway scaling feature helps ensure that appropriate sizing is applied to VPCs/VNets on AWS and Azure Spoke gateways. This can reduce CSP costs, improve performance and mitigate packet loss. You can apply manual, automatic, or scheduled scaling to your selected VPCs/VNets.

Manual scaling occurs when the metrics in your trigger conditions are met for a configured time period. A recommendation to scale up (increase the gateway size or add gateway instances) or scale down (decrease the gateway size or remove gateway instances) is then generated that you can choose to apply. With manual scaling you can scale a VPC/VNet up and down one time.

When the instance size changes, it changes for all gateways in that VPC/VNet.

Automatic scaling occurs when the (combination of) metrics are outside the ideal operating range for a set time period, and a gateway scaling event has not occurred in the duration specified by the cool-down period. Gateways in the selected VPCs/VNets are added or removed automatically.

Scheduled scaling (adding and removing a gateway instance) occurs during a configured time period. You should only select this option if you can regularly predict traffic and resource consumption within regularly scheduled hours.

VPCs/VNets cannot have both automatic and scheduled scaling applied.

Each VPC/VNet can only scale one gateway at a time. Multiple VPC/VNets can scale gateways at the same time.

You can only scale down gateways created with auto or scheduled scaling. Gateways created using different methods, such as Terraform or CoPilot, cannot be removed via this scaling feature.

Before You Begin

  • For gateway scaling features, the user account you have designated as your CoPilot service account must have a minimum of all_write permissions.

  • (optional) Configure at least one email address or webhook in CoPilot under Monitor > Notifications > Recipients. This email address or webhook is where recommendations are sent when a manual scaling policy is triggered.

  • The Aviatrix Controller must be upgraded to version 7.0 to support the creation of new instances using gateway scaling.

  • Any AWS/Azure Spoke gateways that you want to use in gateway scaling cannot have HPE, BGP, NAT, DNAT, SNAT, FQDN gateways, or Site2Cloud enabled.

  • Remember that not all regions support all gateway (instance) sizes. Before adding or removing gateway instances (manual scaling), you should determine what increased or decreased instance size best suits the requirements of your network infrastructure.

  • (automatic scaling) Make note of your base (starting) number of gateways. Gateways will not be scaled down if the base (starting) number of gateways has already been reached.

To set up gateway scaling, see Performance Improvement with Gateway Scaling.

Creating a Policy

In CoPilot under AirSpace > Scaling > Policies, you create policies that monitor VPCs/VNets to determine when to generate a recommendation or when gateway scaling should occur.

scaling policy tab

If network architecture is changed after a policy is created (for example, High Performance Encryption is enabled on a VPC that is included in a policy), the scaling event related to the policy may fail. If an error occurs during policy creation or execution you should first check for any network architecture changes.

On this tab you can:

  • Create policies

  • Edit policies

  • Filter policies

  • Download the contents of the tab as a CSV

  • Delete policies: click the trash can icon (you are asked to confirm the deletion)

  • View events for selected automatic policies

  • View events or show recommendations for selected manual policies

Policies are never invalidated.

Viewing Policy Details

When you click a policy name on the Scaling > Policies tab details are displayed for that policy:

  • VPCs/VNets in the policy

  • If it is a manual, automatic, or scheduled scaling policy

  • The metrics in the trigger conditions or the ideal operating range

  • Evaluation Period - The duration of time (in minutes) where CoPilot evaluates the average of the metric(s) specified to trigger the action.

  • Cool Down Period - (Automatic scaling only) The number of minutes that must elapse between consecutive gateway scaling events. Even if the monitored VPCs/VNets fall outside the ideal operating range for the specified Evaluation Period, gateway scaling will not occur again until after the Cool Down Period has elapsed.

  • Who receives alerts/emails related to the execution of the policy (if configured)

  • The time period, start and end date, and performance metrics for the selected time period. Metrics for manual scaling polices are shown individually. You can select which VPC/VNet to view metrics for.

  • Events related to the executed policy

Creating a Manual Scaling Policy

  1. On the Cloud Fabric > Scaling > Policies tab, click the Policy button and then click +Policy. The Create New Scaling Policy dialog displays.

  2. Enter a name for the scaling policy that clearly describes what the policy will do.

  3. Select Manual Scaling.

  4. Select Spoke VPC/VNets, Transit VPC/VNets or SNAT Enabled VPC/VNets.

    400
  1. In the Monitor VPC/VNets field, enter the name of the VPC/VNet, which displays the matching VPC/VNets that can be assigned to the policy.

    SNAT enabled VPC/VNets that also have BGP or egress enabled will not be available in the Monitor VPC/VNets field.
  2. Click Select All to select all of the matching VPCs/VNets and assign them to the policy. Or, select the checkbox next to specific VPC/VNets to include them.

  1. Under Trigger Conditions, select ‘Matches all conditions (AND)’ or ‘Matches any condition (OR)’ to determine if all conditions must be met, or if only one of the conditions must be met. The metric chosen must have an average greater than or less than the chosen threshold limit, relative to the Evaluation Period. In either case these conditions must be sustained for the configured Evaluation Period to generate manual scaling recommendations.

    scaling manual condition

    When a scale recommendation is triggered, the recommendation will stay active as long as the metrics are above or below the threshold. If the metrics fall out of the threshold, the recommendation will no longer be active.

    Available system and network trigger conditions are:

    • System:

      • CPU Used (%)

      • Memory Used

      • Memory Used (%)

    • Network:

      • Transmitted Rate

      • Received Rate

      • Total Rate

      • Packets Transmitted Rate

      • Packet Received Rate

      • Total Rate (in packets)

      • Bandwidth Ingress Limit Exceeded Rate

      • Bandwidth Egress Limit Exceeded Rate

      • PPS Limit Exceeded Rate

      • Conntrack Limit Exceeded Rate

      • Packet Drop Rate

      • Packet Drop Rate (%)

  1. Continue adding trigger conditions as needed. You can enter a maximum of five trigger conditions, or five groups that each contain a maximum of five trigger conditions.

  2. In the Evaluation Period field, enter the duration of time (in minutes) where CoPilot evaluates the average of the metric(s) specified to trigger the action of recommending gateway scaling.

  3. In the Recommendation Operation field, select the scaling operation to perform, which is dictated by the VPC/VNet type (Spoke or Transit).

    In Spoke VPC/VNets you can increase/decrease instance sizes (scale up/down) or add/remove gateway instances (scale out/in).

    In Transit VPC/VNets and SNAT Enabled VPC/VNets you can only increase/decrease instance sizes (scale up/down).

    400

    • When scaling up/out, the current gateway size is increased for as long as the trigger conditions are met, or a gateway instance is added. If you are adding a gateway instance, an instance is added to an empty subnet.

    • When scaling down/in, the current gateway size is decreased for as long as the trigger conditions are met, or a gateway instance is removed.

      Prior to version 7.0 gateways in a single VPC/VNet are scaled to use the same instance size.

    Once the VPC/VNet recommendation has been created, and the scaling action has completed, the VPC/VNet will be removed from the policy.

Once a recommendation has been accepted, the gateway will not scale (up/down or in/out) until another manual policy has been configured, and another recommendation has been generated.
  1. In the Send Notifications To field, select the email address where the notification is sent when the trigger conditions are met. If this field is left blank, you will not receive alerts.

  2. Click Save to save the policy and begin monitoring the selected VPCs/VNets.

  3. After you receive your notification email, go to the Scaling > Recommendations tab to execute the recommendation.

Implementing Recommendations

After receiving a notification email indicating that trigger conditions have been met and sustained for the configured Evaluation Period (if you entered this information when creating the policy), you can choose to implement the manual scaling recommendation. A recommendation is generated per VPC/VNet rather than per policy.

On the Cloud Fabric > Scaling > Recommendations tab, you can choose to scale up or down (Transit VPCs), or up/down in/out (Spoke VPCs) depending on the Operation you selected when creating the related policy. After the recommendation has executed it is removed from the Recommendations list and is displayed as an event on the Scaling > Events tab.

scaling recommendation

Clicking a recommendation in the Time field for a policy on the Recommendation tab shows the policy details and the performance of the metrics that have been triggered over the selected time period. After viewing this historical detail you can decide if you want to apply the recommendation.

A recommendation is not valid when it falls outside of the triggered conditions.

A recommendation can become valid again if it once again falls in the triggered conditional range, and there is at least one VPC remaining in the manual policy. If the VPC is removed from the policy it will not trigger again.

To implement a recommendation:

  1. Click Scale Up, Scale Down, or Scale In next to a recommendation in the list.

  2. If you scale up or down by increasing or decreasing current gateway size:

    1. Select the new instance size.

    2. Acknowledge that this operation could cause a brief network downtime.

    3. Click Begin Scaling.

      scaling manual instance size
  3. If you scale in or out by adding or removing instances:

    1. Scale Up:

      1. Click +Instance.

      scaling instance

    1. Select an available subnet.

    2. Select an existing IP, or allocate a new IP to the instance.

    3. Click Begin Scaling.

  1. Scale Down:

    1. Click the Delete icon next to the instance to delete (at least one row must remain).

    2. Click Begin Scaling.

You cannot add an instance if no subnet is available in that VPC/VNet.

After scaling has occurred:

  • a message displays on the Scaling > Recommendations tab indicating this, and that the VPC/VNet that triggered the recommendation has been removed from the policy.

  • The related recommendation is removed from the Recommendations list.

  • An event is generated and added to the Events page.

Scaling operations run one at a time per policy. Multiple policies can run at the same time.

Creating an Automatic Scaling Policy

Automatic Scaling is in Preview mode.

When you apply automatic scaling, gateways are added and removed to handle system load.

If system load is within the ideal operating range, the number of gateways will not change. If system load is lower than the ideal operating range, scaling down will only commence if the number of existing gateways is greater than your base (starting) number of gateways. If your base (starting) number of gateways has already been reached, no action is taken.

Aviatrix recommends applying one automatic scaling policy per VPC/VNet.

VPCs/VNets cannot have both automatic and scheduled scaling applied.

To create an automatic scaling policy:

  1. On the Cloud Fabric > Scaling > Policies tab, click +Policy. The Create New Scaling Policy dialog displays.

    640

  1. Enter a name for the policy that clearly describes what the policy will do.

  2. Select the VPCs/VNets to monitor.

  3. Select Automatic Scaling.

  4. Under Ideal Operating Range, select Matches all conditions (AND) or Matches any condition (OR) to determine if all conditions must be met, or if only one of the conditions must be met.

  5. You can enter a maximum of one condition per metric. Available metrics are CPU Used (%) and Memory Used (%).

    For example, if you have just the CPU Used (%) metric in your policy and the percentage range is from 20-80%, automatic scaling is not triggered until conditions fall outside of this ideal operating range (for example, 90%). Automatic scaling metrics are always percentages.

  6. In the Evaluation Period field, enter the number of minutes during which CoPilot evaluates the average of the metric(s) specified to trigger automatic scaling. When the average metric conditions are outside the ideal operating range, automatic scaling occurs (addition or removal of gateways).

    Each metric and its operation (AND/OR) is taken into account when determining when automatic scaling will occur.

  7. In the Send Notifications To field, select the email address/webhook where the notification is sent when gateway scaling has occurred. If this field is left blank you will not receive alerts.

  8. In the Cool Down Period field, enter the number of minutes that must elapse between consecutive gateway scaling events. Even if the monitored VPCs/VNets fall outside the ideal operating range for the specified Evaluation Period, gateway scaling will not occur again until after the Cool Down Period has elapsed.

  9. Click Save.

After automatic scaling is performed on a VPC/VNet that VPC/VNet is removed from the policy.

  • Only a gateway that has been added using automatic scaling can be removed using automatic scaling. User-created gateways are not removed by automatic scaling.

  • If two metrics configured in your automatic scaling policy give contradictory recommendations, the policy defaults to scaling Up or Out - in other words, increasing gateway number or size. For example, if the CPU Used metric states that the scaling event should scale out, but the Memory Use metric stated that the policy should scale in, the policy defaults to scaling out.

Creating a Scheduled Scaling Policy

You can create a scheduled scaling policy to add a gateway instance during a scheduled time period, and remove it when the scheduled time period ends. The gateway instance is created in the first available empty subnet. Gateways can only be removed if they were previously added using gateway scaling.

VPCs/VNets cannot have both automatic and scheduled scaling applied.

Scheduled scaling occurs in the local time zone in which the browser is deployed. For example, if the scheduled scaling event is set to begin at 9 am, it begins at 9 am according to the local browser time zone.

  1. On the Cloud Fabric > Scaling > Policies tab, click +Policy. The Create New Scaling Policy dialog displays.

  2. Select the Scheduled Scaling option.

500
  1. Enter a name for the policy that clearly describes what the policy will do.

  2. In the Scheduled VPC/VNets field, select the VPCs/VNets on which to apply the policy.

    You can type a string in this field to display the matching VPCs/VNets. You can then click Select All to select the matching VPCs/VNets.

  3. Select Scheduled Scaling.

  4. Under Scaling Schedule, select a time to begin scaling out, and a time to begin scaling in.

  5. Select the days when this scheduled scaling will be performed.

  6. In the Send Notifications To field, select the email address/webhook which should receive the notification when the scaling begins and ends.

    If this field is left blank you will not receive alerts.

Deleting a Policy

To delete a policy, navigate to the Cloud Fabric > Scaling > Policies tab and click the delete icon next to the policy. You see a warning when attempting to delete a policy.

When you delete a policy:

  • Existing recommendations for that policy become invalid (manual scaling).

  • If a scaling event is occurring related to that policy, it will complete.

  • Gateways added with manual scaling can be deleted like a regular gateway.

  • Any gateways created with auto scaling will scale down automatically. A gateway can only be scaled down automatically if it was created with auto scaling.

Viewing the Events Tab

The Events tab lists when auto or manual scaling events occurred, and if the action related to the event succeeded or failed.

gateway scaling events

The Events tab shows the following information:

  • Time of the event

  • Policy name

  • VPC/VNet name

  • Trigger Conditions (Evaluation Period)

  • Gateway Instances

  • Event name

Clicking an event time in the list shows Trigger Condition details and if the policy type is Manual or Automatic.

gateway scaling events details