By default, the Orchestrator is actively involved in learning the dynamic routes. VMware SD-WAN Edges and Gateways rely on the Orchestrator to calculate initial route preferences and return them to the Edge and Gateway. The Distributed Cost Calculation feature enables you to distribute the route cost calculation to the Edges and Gateways. Only an Operator user can configure Customer settings, including Distributed Cost Calculation.

Note:

Anybody experiencing an issue with Orchestrator based route calculation needs Distributed Cost Calculation enabled.

This default method of involving the Orchestrator in both dynamic route calculation and the distribution of those routes to Edges and Gateways has the following drawbacks:

  • If the Orchestrator is under a high load, the route convergence time is significantly high (for example, as much as 40 seconds for 2000+ routes), as the Orchestrator takes that time to calculate the preference for all the synchronized routes and returns those preferences to the Edges and Gateways.
  • Using the Orchestrator for route calculation means that new dynamic routes learned while the Orchestrator was unreachable are not advertised until the Orchestrator becomes reachable again.

When a customer enterprise uses Distributed Cost Calculation, the Orchestrator is no longer actively involved in the route preference calculation and instead routes are properly inserted in order by the Edge and Gateway instantly upon learning them and then convey these preferences to the Orchestrator.

When you choose to enable Distributed Cost Calculation for the Edges and Gateways, the feature provides the following benefits:

  • Minimizes the impact on route learning when an Orchestrator is unreachable.
  • Route convergence time is reduced from minutes to seconds in large networks with thousands of dynamic routes.
  • Network delays are significantly reduced.
  • Provides instantaneous Data Plane convergence.
  • Supports enhanced re-ordering and pinning of routes on the Overlay Flow Control.
  • Provides an option to refresh routes in the Overlay Flow Control page. Whenever there is a change in the Overlay Flow Control policy, the Refresh Routes option applies the changes to the existing routes immediately, without the need to restart the Edge or Gateway.

Enabling Distributed Cost Calculation has the following impacts on the Customer Enterprise network:

  • All the local dynamic routes are refreshed, and the preference and advertise action of these routes are updated. This updated information is advertised to the Gateway, Orchestrator, and eventually across the Enterprise. The customer's network needs to completely rebuild the route table, which for most customer deployments will take less than 5 seconds. A large scale customer deployment (like 100,000+ routes) may take up to 2 minutes. During the time the route table is being rebuilt, customer traffic for all sites is impacted.
  • Any existing flow using these routes can potentially be affected due to the change in the routing entries.
Note: It is recommended to enable Distributed Cost Calculation in a maintenance window to minimize the impact on the Customer Enterprise.

To configure Distributed Cost Calculation for a customer:

Prerequisites

Ensure the following before you enable the Distributed Cost Calculation feature.

  • All the Edges and Gateways must use software version 3.4.0 or later.
  • The software image associated with the Operator Profile must use version 3.4.0 or later.

Procedure

  1. In the Operator portal, navigate to Manage Customers.
  2. Select a customer and either click Edit Customer System Settings or click the link to the customer.
  3. In the Enterprise portal, go to Global Settings > Customer Configuration.

    1. In the Customer Configuration page, navigate to the Additional Configuration > SD-WAN Settings > OFC Cost Calculation section and configure the following:
      • Select the Distributed Cost Calculation checkbox to delegate the cost calculation of routes to Edges and Gateways.
      • Select the Use NSD Policy checkbox to use the Non SD-WAN Destination policy for route cost calculation of Edges and Gateways. This option is available only for Edges and Gateways that are running Software version 4.3.0 or later.
    2. Click Save Changes.
      Note: After enabling Distributed Cost Calculation, it is recommended to refresh the routes in the Overlay Flow Control page in the SD-WAN service of the Enterprise portal.
      Note: When an Enterprise has Distributed Cost Calculation activated and a user tries to deactivate the software update in the Operator Profile page, then the user must ensure that, in future, no Edges in the Enterprise are downgraded to software image versions lower than 3.4.0. If one or more Edges in the Enterprise is using software image version below 3.4.0, the Enterprise traffic may take a sub-optimal path. The sub-optimal path will be corrected only when the Edge is upgraded to 3.4.0 or later versions.
      The following are some of the scenarios in which the software versions can change and the user must make sure the Edges are using the software image version 3.4.0 or later:
      • Factory Reset - When an Edge is reset to factory settings, it restores the software version of the Edge to factory image version which can be below 3.4.0.
      • Edge Activation - When an Edge is activated, it may come up with software versions below 3.4.0.

    Results

    Once Distributed Cost Calculation is activated, all the dynamic routes are assigned with new preferences and advertise action based on the Distributed Cost Calculation and the new information is propagated across the Enterprise Network.

    The Orchestrator is no longer actively involved in the route preference calculation and instead the routes are properly inserted in order by the Edge and Gateway instantly upon learning them and then these preferences are conveyed to the Orchestrator.

    The Overlay Flow Control policy is sent to Edges and Gateways in Control Plane Configuration updates. Edges and Gateways send the routes with computed cost and advertise action to the Orchestrator. Edges and Gateways handle the order of the routes based on the cost and route attributes.

    To view a summary of all the routes in your network, click Configure > Overlay Flow Control in the SD-WAN service of the Enterprise portal. You can view the routes and advertise action in the Overlay Flow Control page. For more information, see the topic Overlay Flow Control in the VMware Administration Guide.