VMware SASE 5.1.0 | 22 July 2024

  • VMware SASE™ Orchestrator Version R51010-20231215-GA

  • VMware SD-WAN™ Gateway Version R5103-20230621-GA

  • VMware SD-WAN™ Edge Version  R5102-20240718-GA

Check for additions and updates to these release notes.

What Is in The Release Notes

The release notes cover the following topics:

This release is recommended for all customers who require the features and functionality first made available in Release 5.1.0.

Compatibility

Release 5.1.0 Orchestrators, Gateways, and Hub Edges support all previous VMware SD-WAN Edge versions greater than or equal to Release 3.4.0.

The following SD-WAN interoperability combinations were explicitly tested:

Orchestrator

Gateway

Edge

Hub

Branch/Spoke

5.1.0 

5.1.0 

3.4.5 

3.4.5 

5.1.0 

5.1.0 

3.4.6 

3.4.6 

5.1.0 

5.1.0 

5.1.0 

3.4.5 

5.1.0 

5.1.0 

3.4.6 

5.1.0 

5.1.0 

4.2.2 

4.2.2 

4.2.2 

5.1.0 

5.1.0 

4.2.2 

4.2.2 

5.1.0 

5.1.0 

5.1.0 

4.2.2 

5.1.0 

5.1.0 

4.2.2 

5.1.0 

5.1.0 

4.3.0 

4.3.0 

4.3.0 

5.1.0 

5.1.0 

4.3.0 

4.3.0 

5.1.0 

5.1.0 

5.1.0 

4.3.0 

5.1.0 

5.1.0 

4.3.0 

5.1.0 

5.1.0 

4.3.1 

4.3.1 

4.3.1 

5.1.0 

5.1.0 

4.3.1 

4.3.1 

5.1.0 

5.1.0 

5.1.0 

4.3.1 

5.1.0 

5.1.0 

4.3.1 

5.1.0 

5.1.0 

4.5.0 

4.5.0 

4.5.0 

5.1.0 

5.1.0 

4.5.0 

4.5.0 

5.1.0 

5.1.0 

5.1.0 

4.5.0 

5.1.0 

5.1.0 

4.5.0 

5.1.0 

5.1.0 

4.5.1 

4.5.1 

4.5.1 

5.1.0 

5.1.0 

4.5.1 

4.5.1 

5.1.0 

5.1.0 

5.1.0 

4.5.1 

5.1.0 

5.1.0 

4.5.1 

5.1.0 

5.1.0 

5.0.1 

5.0.1 

5.0.1 

5.1.0 

5.1.0 

5.0.1 

5.0.1 

5.1.0 

5.1.0 

5.1.0 

5.0.1 

5.1.0 

5.1.0 

5.0.1 

5.1.0 
Note:

The above table is fully valid for customers using SD-WAN services only. Customers requiring access to VMware Cloud Web Security or VMware Secure Access need their Edges upgraded to Release 4.5.0 or later.

Caution:

VMware SD-WAN Releases 3.2.x, 3.3.x, and 3.4.x have reached the End of Support.

  • Releases 3.2.x and 3.3.x reached End of General Support (EOGS) on December 15, 2021, and End of Technical Guidance (EOTG) March 15, 2022.

  • Release 3.4.x for the Orchestrator and Gateway reached End of General Support (EOGS) on March 30, 2022, and End of Technical Guidance (EOTG) on September 30, 2022.

  • Release 3.4.x for the Edge reached End of Support (EOGS) on December 31, 2022, and End of Technical Guidance (EOTG) on March 31, 2023.

  • For more information please consult the Knowledge Base article: Announcement: End of Support Life for VMware SD-WAN Release 3.x (84151)

Important:

VMware SD-WAN Release 4.0.x has reached End of Support; Releases 4.2.x, 4.3.x, and 4.5.x have reached End of Support for Gateways and Orchestrators.

  • Release 4.0.x reached End of General Support (EOGS) on September 30, 2022, and End of Technical Guidance (EOTG) December 31, 2022. 

  • Release 4.2.x Orchestrators and Gateways reached End of General Support (EOGS) on December 30, 2022, and End of Technical Guidance on (EOTG) March 30, 2023.   

  • Release 4.2.x Edges reached End of General Support (EOGS) on June 30, 2023, and will reach End of Technical Guidance (EOTG) September 30, 2025.

  • Release 4.3.x Orchestrators and Gateways reached End of General Support (EOGS) on June 30, 2023, and End of Technical Guidance (EOTG) September 30, 2023.

  • Release 4.3.x Edges reached End of General Support (EOGS) on June 30, 2023, and will reach End of Technical Guidance (EOTG) September 30, 2025.

  • Release 4.5.x Orchestrators and Gateways reached End of General Support (EOGS) on September 30, 2023, and End of Technical Guidance on (EOTG) December 31, 2023.

  • For more information please consult the Knowledge Base article: Announcement: End of Support Life for VMware SD-WAN Release 4.x (88319).

Note:

Release 3.x did not properly support AES-256-GCM, which meant that customers using AES-256 were always using their Edges with GCM deactivated (AES-256-CBC). If a customer is using AES-256, they must explicitly deactivate GCM from the Orchestrator prior to upgrading their Edges to a 4.x or 5.x Release. Once all their Edges are running a 4.x/5.x release, the customer may choose between AES-256-GCM and AES-256-CBC.

Upgrade Paths for Orchestrator, Gateway, and Edge

The following lists the paths for customers wishing to upgrade their Orchestrator, Gateway, or Edge from an older release to Release 5.1.0.

Orchestrator

Orchestrators using Release 4.0.0 or later can be upgraded to Release 5.1.0. 

Gateway

Upgrading a Gateway using Release 4.0.0 or later to Release 5.1.0 is fully supported for all Gateway types.

Important:

When deploying a new Gateway using 5.1.0 the VMware ESXi instance must be at least version 6.7, Update 3 up to version 7.0. Using an earlier ESXi instance will result in the Gateway's Dataplane Service failing when trying to run Release 5.1.0 or later.

Important:

Prior to upgrading a Gateway to 5.1.0, the ESXi instance must be upgraded to at least version 6.7, Update 3 up to version 7.0. Using an earlier ESXi instance will result in the Gateway's Dataplane Service failing when trying to run Release 5.1.0 or later.

Edge

An Edge can be upgraded directly to Release 5.1.0 from any Release 3.x or later.

New SD-WAN Features

Hub or Cluster Interconnect

For the first time, multiple Hub Edges or Hub Clusters can be interconnected over the overlay instead of the underlay and increase the range of Spoke Edges that can communicate with each other. Hubs can now share routes with one another, allowing Spoke Edges connected to one Hub Edge or Hub Cluster to communicate through the overlay with the Spoke Edges connected to another Hub Edge or Hub Cluster. Spoke Edges in a multi-Hub deployment can now leverage Dynamic Multi-Path Optimization (DMPO) for improved traffic quality while providing full end-to-end visibility of all traffic in the network. Hub or Cluster Interconnect provides a customer with increased scalability, reliability, and availability in their multi-Hub Edge or Hub Cluster network.

Important:

Enabling Hub or Cluster Interconnect introduces a fundamental change to the VMware SD-WAN routing protocol where it allows packets to traverse more than one hop in the network. While this change has been tested in representative topologies, it is not possible to test for all routing scenarios that may be encountered when making such a change of allowing distant routes to be distributed. As a result, VMware is releasing this feature as early access and will be closely monitoring deployments where it is enabled for unexpected routing behavior.

New Orchestrator User Interface

Orchestrator Release 5.1.0 includes the complete implementation of our New User Interface which was first introduced in Release 4.0.0. The New UI brings improved usability and a consistent look and feel across all VMware SASE services. In addition, the New UI adds integrated In Product Help to point users to relevant documentation and other materials which can assist in using the SD-WAN service.

The New UI is the default interface on the Release 5.1.0 Orchestrator and the user retains the option to switch to the Classic Orchestrator UI when using SD-WAN.

Note:

The Classic User Interface will be deprecated in the next minor release of the VMware SASE Orchestrator. Customers are strongly encouraged to use and familiarize themselves with the New Orchestrator UI.

Flow Visibility

In previous releases, the Orchestrator UI only displays aggregated flow information and statistics individually from the lens of Application, Source, or Destination and does not combine all this information on one screen to provide a single, end-to-end view. As a result, monitoring, troubleshooting, and reporting are hindered by the lack of detailed visibility of each flow.

With Release 5.1.0, the New Orchestrator UI includes a “Flows” tab that displays the consolidated data for each traffic flow. The Orchestrator UI displays each flow’s key parameters in a single view.  In addition, the Flow Visibility feature allows customers to view historical flow data, filter data based on matched parameters, and download end-to-end flow statistics.

Local DNS Entries

Release 5.1.0 supports local DNS entries on the Edge to point traffic to specific domains.  When Local DNS is configured, the Edge looks to the local host file first before trying to resolve a domain with a DNS server.

Power-On Self-Test (POST) for Orchestrator, Gateway, and Edge

Release 5.1.0 adds improved device hardening and visibility through a power-on self-test (POST). The POST is a process performed by a software routine invoked automatically immediately after a device is powered on or rebooted. The POST process includes:

  • Software integrity verification.

  • Cryptographic module algorithms known answer test verification.

  • Test of entropy (noise) source.

  • Display of the POST results: Pass/Fail.  The system will continue to bring up the rest of the applications only if POST is passed. If POST fails error messages display where the test failed, and system boot-up sequence stops.

For Orchestrators and Gateways this feature is only available in a greenfield deployment. Edges do not have this feature activated by default and a user would need to activate it through the Orchestrator UI.

New SD-WAN Enhancements

High Availability (HA) Local Route Synchronization and BGP Graceful Restart

For a site deployed in a High Availability topology where BGP or OSPF is also used, an HA failover can be both slow and disruptive to customer traffic due to high packet loss as the Standby Edge synchronizes the routes. To ensure faster and less disruptive HA failovers, VMware introduces two enhancements: HA Local Route Synchronization and BGP Graceful Restart.

HA Local Route Synchronization automatically synchronizes routes between the Active and Standby Edges and uses these routes for forwarding on the Active Edge while also ensuring that the route table is immediately available after an HA failover.

BGP Graceful Restart ensures faster Edge restarts and HA failovers by having the neighboring BGP devices participate in the restart to ensure that no route changes occur in the network for the duration of the restart. Without BGP Graceful Restart, the peer Edge deletes all routes once the TCP session terminates between BGP peers and these routes need to be rebuilt post Edge restart or HA failover. BGP Graceful Restart changes this behavior by ensuring that peer Edges do not delete routes if a new session is established within a configurable restart timer.

Important:

For best peformance, Dynamic Cost Calculation (DCC) should also be activated in the customer enterprise. With DCC activated, preference and advertisement decisions are local to the Edge and the Edge synchronizes from Active to Standby as soon as it learns the routes from the routing process. For more information on DCC see VMware SD-WAN Routing Overview and Configure Distributed Cost Calculation.

Note:

HA Local Route Synchronization is only available for enterprises using BGP. HA Local Route Synchronization where OSPF is used will be available in a future release.

RADIUS Authentication on a Switched Interface

Customers can use RADIUS authentication using the 802.1x protocol on switched ports, where previously they were limited to routed ports. Switched port RADIUS authentication is configured through a VLAN associated with that port. This enhancement benefits customer sites where no other routers are available to expand local access, but which require secure device authentication through 802.1x.

MAC Address Bypass (MAB)

On routed interfaces customers can now check MAC addresses against a list on a RADIUS server to bypass 802.1x for LAN devices that do not support 802.1x authentication. MAB simplifies IT operations, saves time, and enhances scalability by no longer requiring customers to manually configure every MAC address that may need authentication.

Important:

RADIUS-based MAB is not supported for VLANs and thus cannot be used for switched ports. RADIUS-based MAB is supported for routed interfaces only.

Configuration Changes that Cause an Edge Restart

Several Edge configuration changes that previously triggered an Edge Service restart no longer do so in Release 5.1.0. In particular, frequently used Edge interface configuration changes like modifying an Edge LAN IP address on a switched interface or modifying a CIDR IP, CIDR prefix, or fixed IP no longer cause a disruptive Edge restart. To see a complete list, consult the KB article VMware SD-WAN Edge Configuration Changes That Can Trigger an Edge Service Restart (60247).

SNMP

Release 5.1.0 adds the following enhancements to SNMP:

  • SNMPv2, support for additional community strings and 64-bit counters.

  • SNMPv3, support for SHA2, additional username and passwords, and separate authentication and private keys.

  • The MIB adds the following telemetry:

    • Link UUID to interface name mapping.

    • Link bandwidth/capacity.

    • Link throughput.

Edge 2000, 3800, and 3810 Flow Capacity Increase

Edge models 2000, 3800, and 3810 each increase their flow capacity maximum from 1.9M flows to 3.8M flows when using Release 5.1.0 Edge software.

Note:

The Edge model 3400 is not impacted by this change and this model's flow capacity maximum remains 1.9M flows.

Packet Capture (PCAP) on Gateways

A user can initiate a PCAP on a Gateway through the Orchestrator UI. Up to 120 seconds of Gateway traffic can be captured with the option to define simple or complex filters to ensure the user is only capturing what they need. This feature is accessible to users as follows:

  • Partner Administrators can initiate a PCAP on their own Partner Gateways only.

  • Operators can initiate a PCAP on both Partner Gateways and Hosted Gateways.

External Certificate Authority (CA)

Two new API-ready modes are added to the External CA feature:

  • Manual Mode provides support for any certificate authority and provides flexibility and control with the user manually performing each step in the certificate process.

  • Asynchronous Mode provides support for any certificate authority with the ability to script the manual steps and automating the recurring tasks.

Non SD-WAN Destination (NSD) and Cloud Security Service (CSS) Tunnels

In previous SD-WAN versions, the tunnels for an NSD or CSS formed only when traffic passed through it and persisted as long as there was traffic passing through it. In the absence of traffic for a period of time, the tunnel was torn down and had to be rebuilt the next time traffic was sent in either direction, causing latency for that traffic while the tunnel was being reestablished. Beginning with Release 5.1.0, all NSD and CSS tunnels will be triggered and established at the time either service is initially configured and will persist whether traffic is passing or not for any period of time, improving the user experience for either service.

Important Notes

VMware Security Advisory 2024-0008

BGP Extended Community String Appended to the BGP Prefixes

Intra-Cluster BGP routes are automatically tagged with an internal BGP community that gets appended to the existing BGP communities by each Edge. This additional community string combines 1 byte for the hop count and 3 bytes derived from the Edge Logical ID. As a result, customers using BGP peering on the Spoke/Hub LAN side should not filter the BGP prefixes advertised by the Edges with the new BGP communities string.

Dead Peer Timeout (DPD) for Non SD-WAN Destinations

Release 5.1.0 brings major changes to Dead Peer Timeout (DPD) for Non SD-WAN Destinations. In previous releases, the default value of the DPD was 20 seconds and a user could deactivate DPD by configuring the DPD timeout timer to 0 seconds. With VMware SD-WAN moving to the QuickSec IPsec toolkit, DPD changes as follows:

  • Probe Interval: Exponential (0.5 second, 1 second, 2 seconds, 4 seconds, 8 seconds, 16 seconds).

  • Default Minimum DPD Interval: 47.5 seconds (QuickSec waits for 16 seconds after the last retry. Therefore, 0.5+1+2+4+8+16+16 = 47.5).

  • Default Minimum DPD interval + DPD Timeout (seconds): 67.5 seconds.

As a result of the above changes, a user cannot deactivate DPD by configuring the DPD timeout timer to 0 seconds. The DPD timeout value in seconds will get added onto the default minimum value of 47.5 seconds. So even if a user configured DPD for 0 seconds, in reality the DPD would be 47.5.

Features That Must be Configured on the Classic Orchestrator

With Release 5.1.0, VMware makes the New User Interface the default interface for the Orchestrator with the understanding that a user can perform all monitoring and configuration tasks on it. However, a few features are not fully integrated into the New UI:

  • Secure Access - Edge and Profile Settings

  • Zscaler - Edge and Profile Settings

  • TACACS - Edge Settings and Network Services page

  • Partner Settings - Partner page

To configure the above features, a customer can select the Open Classic Orchestrator option at the top of the Orchestrator screen, which will open a new browser tab with the Classic UI.


These features will be fully integrated into the New User Interface in a later Orchestrator software release.

Mixing Wi-Fi Capable and Non-Wi-Fi Capable Edges in High Availability Is Not Supported 

Beginning in 2021, VMware SD-WAN introduced Edge models which do not include a Wi-Fi module: the Edge models 510N, 610N, 620N, 640N, and 680N. While these models appear identical to their Wi-Fi capable counterparts except for Wi-Fi, deploying a Wi-Fi capable Edge and a Non-Wi-Fi capable Edge of the same model (for example, an Edge 640 and an Edge 640N) as a High-Availability pair is not supported. Customers should ensure that the Edges deployed as a High Availability pair are of the same type: both Wi-Fi capable, or both Non-Wi-Fi capable.

BGPv4 Filter Configuration Delimiter Change for AS-PATH Prepending

Through Release 3.x, the VMware SD-WAN BGPv4 filter configuration for AS-PATH prepending supported both comma and space based delimiters. However, beginning in Release 4.0.0 and forward, VMware SD-WAN only supports a space based delimiter in an AS-Path prepending configuration. Customers upgrading from 3.x to 4.x or 5.x need to edit their AS-PATH prepending configurations to "replace commas with spaces" prior to upgrade to avoid incorrect BGP best route selection.

Extended Upgrade Time for Edge 3x00 Models

Upgrades to this version will take longer than normal (3-5 minutes) on Edge 3x00 models (i.e., 3400, 3800 and 3810) if the Edge is upgrading directly from Release 4.0.0, 4.0.1, or 4.2.0. This is due to a firmware upgrade which resolves issue 53676. If an Edge 3400 or 3800 is upgraded to Release 5.1.0 using any other Edge release, then the Edge would upgrade as expected. For more information, please consult Fixed Issue 53676 in the respective release notes.

Limitation with BGP over IPsec on Edge and Gateway, and Azure Virtual WAN Automation

The BGP over IPsec on Edge and Gateway feature is not compatible with Azure Virtual WAN Automation from Edge or Gateway. Only static routes are supported when automating connectivity from an Edge or Gateway to an Azure vWAN.

Limitation When Deactivating Autonegotiation on VMware SD-WAN Edge Models 520, 540, 620, 640, 680, 3400, 3800, and 3810

When a user deactivates autonegotiation to hardcode speed and duplex on ports GE1 - GE4 on a VMware SD-WAN Edge model 620, 640 or 680; on ports GE3 or GE4 on an Edge 3400, 3800, or 3810; or on an Edge 520/540 when an SFP with a copper interface is used on ports SFP1 or SFP2, the user may find that even after a reboot the link does not come up.

This is caused by each of the listed Edge models using the Intel Ethernet Controller i350, which has a limitation that when autonegotiation is not used on both sides of the link, it is not able to dynamically detect the appropriate wires to transmit and receive on (auto-MDIX). If both sides of the connection are transmitting and receiving on the same wires, the link will not be detected. If the peer side also does not support auto-MDIX without autonegotiation, and the link does not come up with a straight cable, then a crossover Ethernet cable will be needed to bring the link up.

For more information please see the KB article Limitation When Deactivating Autonegotiation on VMware SD-WAN Edge Models 520, 540, 620, 640, 680, 3400, 3800, and 3810 (87208).

Available Languages

The VMware SASE Orchestrator using version 5.1.0 is localized into the following languages: Czech, English, European Portuguese, French, German, Greek, Italian, Spanish, Japanese, Korean, Simplified Chinese, and Traditional Chinese.

Orchestrator API Changes

Orchestrator API Changes since 5.0.0

Changes to the VMware SASE Orchestrator Portal API ("API v1")

The complete API Changelog is available for download at developer.vmware.com (see "VMware SD-WAN Orchestrator API v1").

We anticipate that the following changes may require action from developers:

  • Issue #66795: This fix introduces a mechanism by means of which an API Token will only be valid and accepted by the Orchestrator for Non-Native users if they are in enabled status and if the SSO user is active in their respective Identity Provider. If a non-native user becomes inactive (In other words, deleted in the IdP or does not have a valid refresh token), all API tokens for this user will be revoked via a backend job.

Note:

  • Tokens issued on behalf of an SSO-enabled user before this fix are subject to legacy behavior where the Orchestrator will continue to honor them until they expire.

  • SSO users from Identity providers which do not support Refresh Tokens or Introspection endpoint will not be allowed to use Token Based Authentication feature. 

  • Issue #87878: vcoInventory/getPendingInventory changed response payload. Removed fields token, vcoInstanceLogicalId, vcoUrl, edgeMappingId, enterpriseId, enterpriseProxyId, uuid, mac, imei, owner. They are not used on the frontend, and it does not affect the UI because these fields were not used on the UI. This API is intended to be used to get a list of ZTP-available Edges and these fields are not required for Edge assignment (only serialNumber is required). So, if a customer uses this API for ZTP, and they have a somewhere strict payload validation which may impact them (depends on their implementation). In general - returned information is sufficient for successful ZTP flow.

  • Issue #84303: Added a validation for BGP neighbor maxHop attribute to not allow configuring a maxHop value of less than 2 when a neighbor localIP is present. Previously, configuring a maxHop value as 1 was allowed irrespective of whether the neighbor localIP is present or not. And now as per the change whenever a neighbor localIP is present, the minimum max hop value allowed is 2 and if a user tries to configure a maxHop value less than 2, then they will get an error saying Invalid MaxHop for Neighbor. MaxHop value ranges from 2 to 255 when localIp is present. A patch is being written to take care of the existing configuration.

  • Issue #84114: The migration of client devices from MySQL to ClickHouse eliminated the clientDeviceId field. Because we did not identify an external client by using the clientDeviceId field, the impact should be negligible. The only client using the client devices endpoint with clientDeviceId appears to be the Classic Orchestrator UI. The UI has been enhanced to update or fetch records in ClickHouse using the logicalId or ipAddress and macAddress combination. External clients should follow suit when using the client device endpoint to update the hostname or when fetching specific devices by ID.

Changes to the VMware SASE Orchestrator API v2

  • Issue #98750: The lastContact field in the Edge record returned by Edge-related APIs is deprecated and should no longer be used. Instead, the edgeState field in the response should be used to determine the Edge's actual state as the single source of truth. If any client code ever uses lastContact and cannot replace it with edgeState API v1 still provides accurate lastContact in the response, which can be used as a compromise but not recommended.

  • Issue #30901: With the Flow Visibility feature in place in Release 5.1.0, the mandatory groupBy clause is no longer required for flowstats. If the groupBy clause is not mentioned, by default we consider the API end point is querying or calling for the Flow Visibility API endpoint which in turn resolves for all the resolvers like application, client devices, etc. However, this is applicable only for the flow stats metric API call and the flow stats series API still remains the same without any modifications.

  • Issue #95089: The APIv2 rate-limiting module has not been enforcing the same default policy that the Portal API rate-limiter does, which was always our intention. A change in this release effectuates that policy for APIv2. We are advising users to review best practices for avoiding triggering the rate-limiter and handling responses where rate limiting has been triggered.

Note on Developer Documentation

Historically, VMware SASE/SD-WAN API documentation was hosted on VMware {code} at code.vmware.com. VMware {code} was recently migrated to a new domain, developer.vmware.com. As a result of the migration, some perma-links to specific pages that were previously hosted code.vmware.com may cease to work as expected.

In conjunction with the migration, VMware will continue to use the Developer Documentation Portal at https://developer.vmware.com/apis, where all VMware SASE/SD-WAN API documentation now resides.

Document Revision History

July 22nd, 2024. Twenty-Seventh Edition.

April 2nd, 2024. Twenty-Sixth Edition.

  • Added an Important Note regarding CVE-2024-22247, which details a missing authentication and protection mechanism vulnerability that impacts an SD-WAN Edge. VMware's response to this vulnerability is documented in VMSA-2024-0008. More information on mitigating this vulnerability is found in the KB article: VMware Response to CVE-2024-22247 (VMSA-2024-0008) (97391).

March 4th, 2024. Twenty-Fifth Edition.

  • Changed Edge Open Issue #115136 to a Fixed Issue and moved it to the Edge/Gateway Resolved Issues section for the second Edge/Gateway rollup build R5102-20230310-GA. The fix for the issue is found on the Edge component of R5102-20230310-GA and was omitted from the tenth edition of the 5.1.0 Release Notes.

  • Added Fixed Issue #97055 to the Orchestrator Resolved Issues section for the original GA Orchestrator build R5100-20221202-GA. This issue was omitted from the First Edition of the Release Notes.

December 18th, 2023. Twenty-Fourth Edition.

  • Added a new Orchestrator rollup build R51010-20231215-GA to the Orchestrator Resolved section. This is the tenth Orchestrator rollup build and is the new default Orchestrator GA build for Release 5.1.0.

  • Orchestrator build R51010-20231215-GA includes the fixes for issues #117941, #125006, #128310, #129239, and #131789, each of which is each documented in this section.

October 9th, 2023. Twenty-Third Edition.

  • Added a new Orchestrator rollup build R5109-20231003-GA to the Orchestrator Resolved section. This is the ninth Orchestrator rollup build and is the new default Orchestrator GA build for Release 5.1.0.

  • Orchestrator build R5109-20231003-GA includes the fixes for issues #119938 and #128310, each of which is each documented in this section.

  • Added Open Issue #105933 to the Edge/Gateway Known Issues section.

September 20th, 2023. Twenty-Second Edition.

  • Added a new Orchestrator rollup build R5108-20230916-GA to the Orchestrator Resolved section. This is the eighth Orchestrator rollup build and is the new default Orchestrator GA build for Release 5.1.0.

  • Orchestrator build R5108-20230916-GA includes the fixes for issues #94610, #104775, #105580, #106191, #115981, #116531, #117822, #118728, #121085, #121441, #121469, and #124778, each of which is each documented in this section.

  • Moved Open Issue #62701 from Edge Gateway Known Issues to the Edge/Gateway Resolved Issues section for original GA build R5100-20221204-GA. This action should have been taken in the 1st Edition of these Release Notes.

  • Added Open Issue #115136 and #117037 to the Edge and Gateway Known Issues section.

  • Document Revision History reorganized to read from newest entries to oldest for an improved user experience.

August 22nd, 2023. Twenty-First Edition.

August 3rd, 2023. Twentieth Edition.

July 26th, 2023. Nineteenth Edition.

  • Added Fixed Issue #103708 to the Edge/Gateway Resolved Issues section for the second Edge rollup build R5102-20230310-GA. This issue was omitted from the tenth edition of the 5.1.0 Release Notes.

July 23rd, 2023. Eighteenth Edition.

  • Added a new Orchestrator rollup build R5107-20230722-GA to the Orchestrator Resolved section. This is the seventh Orchestrator rollup build and is the new default Orchestrator GA build for Release 5.1.0.

  • Orchestrator build R5107-20230722-GA includes the fix for issue #122271, which is documented in this section.

  • Removed Open Issue #53359 from the Edge/Gateway Known Issues section as this was fixed in 4.3.0.

  • Added Open Issue and #103708 and #117775 to the Edge/Gateway Known Issues section.

July 15th, 2023. Seventeenth Edition.

July 6th, 2023. Sixteenth Edition.

  • Added a new Orchestrator rollup build R5106-20230705-GA to the Orchestrator Resolved section. This is the sixth Orchestrator rollup build and is the new default Orchestrator GA build for Release 5.1.0.

  • Orchestrator build R5106-20230705-GA includes the fixes for issues #84772, #115411, #115433, #116633, #117772, #117988, #117993, #118074, #118544, #118733, #119733, and #120606, each of which is each documented in this section.

  • Added Fixed Issue #95565 to the Edge/Gateway Resolved Issues section for the original build R5100-20221204-GA. This issue was omitted in error from the original 5.1.0 Release Notes.

  • Added Open Issue #107994 to the Edge/Gateway Known Issues section.

  • Added Open Issue #112826 to the Orchestrator Known Issues section.

June 23rd, 2023. Fifteenth Edition.

  • Added a new Gateway rollup build R5103-20230621-GA to the Edge/Gateway Resolved section. This is the third Gateway rollup build and is the new Gateway GA build for Release 5.1.0.

  • Gateway build  R5103-20230621-GA includes the fixes for issues #82808, #100172, #101536, #104619, #107309, #108473, #111646, #111888, #111924, #112016, #112017, #112019, #112020, #112800, #114052, #114084, #114282, #114932, #115604, #115692, and #116182, each of which is documented in this section.

  • Added Open Issues #115148 and #119033 to the Edge/Gateway Known Issues section.

June 13th, 2023. Fourteenth Edition.

  • Added a new Orchestrator rollup build R5105-20230611-GA to the Orchestrator Resolved section. This is the fifth Orchestrator rollup build and is the new Orchestrator GA build for Release 5.1.0.

  • Orchestrator build R5105-20230611-GA includes the fixes for issues #87089, #105861, #106295, #107180, #107766, #110826, #111957, #112044, #112333, #112451, #112500, #112605, #112809, #112906, #112912, #112992, #113209, #113254, #113366, #113375, #113963, #114240, #114291, #114564, #114602, #114912, #115307, #115439, #115624, #115653, #115719, #116141, #116523, #116770, #116790, #116976, #117527, #117800, and #118071.

  • Added the following Open Issues to the Edge/Gateway Known Issues section: #82808, #107309, #111924, #112016, #112017, #112019, #114084, #114282, #115692, and #116182. All of these issues impact the VMware SD-WAN Gateway.

May 11th, 2023. Thirteenth Edition.

  • Added the following Open Issues to the Edge/Gateway Known Issues section: #108473, #111646, #111888, #112020, #112800, #114052, and #114932. All of these issues impact the VMware SD-WAN Gateway.

April 27th, 2023. Twelfth Edition.

  • Added a new Orchestrator rollup build R5104-20230426-GA to the Orchestrator Resolved Issues section. This is the fourth Orchestrator rollup build for Release 5.1.0.

  • Orchestrator build R5104-20230426-GA includes the fixes for issues #95631, #104785, #106327, #106929, #107071, #107349, #107980, #108072, #108363, #109284, #109300, #109532, #109533, #109715, #109788, #109836, #109911, #110094, #110330, #110946, #111104, #111407, #111444, #111665, #111934, #111944, #111946, #112094, #112201, #112224, #112437, #112458, #112885, and #114475. Each issue is documented in this section.

  • Two of the tickets fixed, #106907 and #106929 were originally marked as fixed in Orchestrator Release 5.1.0.2. However, the fixes were incomplete in 5.1.0.2 and are only fully resolved in Orchestrator version 5.1.0.4. As a result these tickets were removed from the Orchestrator version 5.1.0.2 Resolved section and moved into 5.1.0.4.

  • Added #94612 to the Edge/Gateway Resolved Issues section for the original build R5100-20221204-GA. This issue was omitted in error from the original 5.1.0 Release Notes.

  • Updated the Compatibility section to mark all 3.x releases as having reached their End of Service Life (EOSL). Also updated the 4.x section to mark 4.2.x Orchestrators and Gateways as End of Support Life (EOSL).

  • Added an Important Notes titled BGP Extended Community String Appended to the BGP Prefixes, please see the note for more details.

March 15th, 2023. Eleventh Edition.

  • Added a new Orchestrator rollup build R5103-20230315-GA to the Orchestrator Resolved Issues section. This is the third Orchestrator rollup build for Release 5.1.0.

  • Orchestrator build R5103-20230315-GA includes the fixes for issues #107587, #107725, #108533, #108833, and #109064. Each issue is documented in this section. R5104-202304xx-GA

March 14th, 2023. Tenth Edition.

  • Added a new Edge and Gateway rollup build R5102-20230310-GA to the Edge/Gateway Resolved section. This is the second Edge/Gateway rollup build and is the new Edge and Gateway GA build for Release 5.1.0.

  • Edge and Gateway build  R5102-20230310-GA includes the fixes for issues #98782, #104141, #105744, and #106587, each of which is documented in this section.

March 6th, 2023. Ninth Edition.

  • In the New SD-WAN Enhancements section, revised the entry Edge 3x00 Flow Capacity Increase. The orignal entry excluded the 2000 and erroneously included the Eddge 3400. The revised entry now reads:

    • Edge 2000, 3800, and 3810 Flow Capacity Increase

    • Edge models 2000, 3800, and 3810 each increase their flow capacity maximum from 1.9M flows to 3.8M flows when using Release 5.1.0 Edge software.

    • A Note is added to make clear that the Edge model 3400 is not impacted by this change and this model's flow capacity maximum remains 1.9M flows.

February 28th, 2023. Eighth Edition.

  • Replaced Orchestrator rollup build R5102-20230216-GA with R5102-20230222-GA. The new Orchestrator build corrects an upgrade issue seen by the VMware Operations team when upgrading an Orchestrator to build R5102-20230216-GA. The upgrade issue was caused by a version mismatch in the upgrade package manifest.

  • The new build also includes fixes for: #106907, #108074, and #108309.

February 17th, 2023. Seventh Edition.

  • Added a new Orchestrator rollup build R5102-20230216-GA to the Orchestrator Resolved Issues section. This is the second Orchestrator rollup build for Release 5.1.0.

  • Orchestrator build R5102-20230216-GA includes the fixes for issues #40584, #105610, #106159, #106242, #106592, #106806, #106929, #107410, #107637, and #107885. Each issue is documented in this section.

  • Added #89725 to the Edge/Gateway Resolved Issues section for the original build R5100-20221204-GA. This issue was omitted in error from the original 5.1.0 Release Notes.

  • Removed Issue #39659 from the Edge/Gateway Known Issues section as this is a duplicate of another ticket, #39501 which was resolved in Release 4.3.0.

January 29th, 2023. Sixth Edition.

  • In the Compatibility section, revised the Import Note regarding End of Support for 4.2.x and added Release 4.3.x to reflect newly revised dates for the SD-WAN Edge software.

  • In the New SD-WAN Enhancements section, added the Non SD-WAN Destination (NSD) and Cloud Security Service (CSS) Tunnels enhancement. This was omitted in the first edtion of the Release Notes in error.

  • In the Important Notes section, added a note on Dead Peer Timeout (DPD) for Non SD-WAN Destinations. This note covers the changes in behavior for DPD as a result of VMware SD-WAN software changing to the Qucksec IPsec toolkit. This material was omitted in the first edtion of the Release Notes in error.

January 20, 2023. Fifth Edition.

  • Added a new Gateway rollup build R5101-20230112-GA to the Edge/Gateway Resolved section. This is the first Gateway rollup build and is the new Gateway GA build for Release 5.1.0.

  • Gateway build R5101-20230112-GA includes the fixes for issues #97272 and #104487, each of which is documented in this section.

  • Amended the language for Enhanced Feature MAC Address Bypass (MAB) to make it clear this feature is not supported for VLANs and thus cannot be used for switched ports, which are dependent on a VLAN for 802.1x authentication. Thus MAB is only supported on routed interfaces as of this version of Release 5.1.0.

January 12, 2023. Fourth Edition.

  • Added wording about two 5.1.0 enhancements: HA Local Route Synchronization and BGP Graceful Restart.

January 05, 2023. Third Edition.

  • Added a new Orchestrator rollup build R5101-20221220-GA to the Orchestrator Resolved Issues section. This is the first Orchestrator rollup build for Release 5.1.0.

  • Orchestrator build R5101-20221220-GA includes the fixes for issues #100133, #101835, #102806, and #103622, each of which is documented in this section.

December 15, 2022. Second Edition.

  • Removed Open Issue #39134 from Edge/Gateway Known Issues as Engineering concluded it was fixed, and this ticket was in error already added to Edge/Gateway Resolved Issues in the first edition of the 5.1.0 Release Notes.

December 08, 2022. First Edition.

Edge/Gateway Resolved Issues

Edge Version R5102-20240718-GA

Edge version R5102-20240718-GA was released on 07-22-2024 and is a hotfix build for Release 5.1.0.

This Edge hotfix build includes a remediation for CVE-2024-6387, a critical vulnerability in OpenSSH. For more information on this OpenSSH vulnerability, please consult the article Broadcom Software Defined Edge Division response to CVE-2024-6387.

Edge build R5102-20240718-GA adds no additional software fixes to the 2nd Edge rollup build, version R5102-20230310-GA.

Resolved in Gateway Version R5103-20230621-GA

Gateway build R5103-20230621-GA was released on 06-23-2023 and is the 3rd Gateway rollup for Release 5.1.0.

This Gateway rollup build addresses the below critical issues since the 2nd Gateway rollup build, R5102-20230310-GA.

  • Fixed Issue 82808: For a VMware SD-WAN Edge that is using a Cloud Security Service (CSS) and has turned on L7 Health Check, the customer may observe traffic failing using these CSS tunnels even though the VMware SASE Orchestrator continues to mark the tunnels as UP.

    Even though the L7 probe fails with a 4XX HTTP error, the VMware SD-WAN Gateway does not acknowledge the failure and does not inform the Orchestrator to mark the CSS tunnels as DOWN.

  • Fixed Issue 100172: If a user attempts to SSH to an Edge via a VMware SD-WAN Gateway, the Gateway may experience a Dataplane Service failure and generate a core followed by a restart to recover.

    The Gateway can encounter this issue when a user is attempting to SSH to an Edge via the Gateway and that SSH session generates a FRAG_NEEDED ICMP error message.

  • Fixed Issue 101536: A VMware SD-WAN Gateway may experience a Dataplane Service failure, generate a core, and restart to recover.

    When looking in the core file, the user would observe logging related to the Gateway's mutex monitor and can be experienced when Hub and Cluster Interconnect is being used by a customer enterprise connected to that Gateway.

  • Fixed Issue 104619: When two or more enterprises share the same Partner Gateway and have all have a partner handoff configuration where the enterprises are using IPv4, if a partner handoff in one enterprise is removed, security association (SA) establishment will fail for the other enterprises connected to that Partner Gateway.

    For example, if there are two enterprises called Enterprise-1 and Enterprise-2 using a Partner Gateway, and a partner handoff is configured on both enterprises, the SD-WAN service sets a single IP address in the Gateway's virtual network interface. If a user disables the partner handoff in Enterprise-2 then the SD-WAN service goes to the next flow and deletes this IP address from the virtual network interface even though the same IP address is being used for the Enterprise-1 handoff. As a result, Enterprise-1 will not be able to establish IPsec tunnels with their Edges.

    If a Gateway experiences this issue without a fix, rebooting the Gateway will remediate the issue.

  • Fixed Issue 107309: When a customer configures the L7 Health Check for a Non SD-WAN Destination via Edge on a 4.x Orchestrator and the Orchestrator is upgraded to Release 5.x, if the customer attempts to modify the L7 probe retry value, the Edge does not apply the new value.

    For example, if the L7 Health Check probe retry value is 3 (the tunnel is marked as down on 3 failed probes) and the customer changes this value to 1, the L7 Health Check continues to use the original value of 3 retries before the tunnel is marked down.

  • Fixed Issue 108473: A VMware SD-WAN Gateway may experience a Dataplane Service failure, generate a core, and restart to recover service.

    The Gateway can get into a situation where there is a sequence number overflow, and this triggers a deletion of all SAs (IPsec security associations). When trying to delete all SAs, the Gateway process tries to find a tunnel based on a tunnel ID, but the tunnel does not exist, and this causes a Gateway service failure.

  • Fixed Issue 111646: A VMware SD-WAN Gateway under a high CPU load may experience a Dataplane Service failure and restart to recover.

    A user looking in the Gateway generated core would see mutex monitor exception and the message Program terminated with signal SIGXCPU, CPU time limit exceeded message. The issue is related to a Gateway process releasing a lower priority thread lock.

  • Fixed Issue 111888: A VMware SD-WAN Gateway deployed with 4 cores and greater than 2000 tunnels connected to it may experience high CPU usage and tunnels connected to the Gateway may be unstable.

    One of the Gateway threads is utilizing too much CPU capacity in a 4 core Gateway which is hindering the Gateway's ability to maintain stable tunnels.

  • Fixed Issue 111924: A customer may observe that across all their sites Multi-Path traffic (in other words, traffic that traverses the VMware SD-WAN Gateway) is being dropped even though their VMware SD-WAN Edge's tunnels to the Gateway are up and stable.

    There is no limit on the maximum number of times a Gateway can retransmit a VCMP packet (SD-WAN's management protocol), and such retransmits can overwhelm low bandwidth links. These retransmits will also cause packet build-up on the scheduler when the Edge has a low bandwidth link since the retransmits cannot be drained fast enough. Eventually the scheduler queues become full and lead to the scheduler dropping packets from all Edges. Direct traffic that does not use the Gateway would not be affected by this issue.

    When this issue encountered and a Gateway without a fix for this issue is being issued, the only way to remediate is for an Operator user to identify the Edges which are causing the packet build-up on the scheduler using the debug.py --qos_dump_net command and block them in the affected Gateway.

  • Fixed Issue 112016: A VMware SD-WAN Gateway may experience multiple Dataplane Service failures with generated cores after a Gateway restart is initiated.

    When examining the cores an Operator would observe that each failure is triggered by a mutex monitor issue. There is a noticeable time increase in VCMP (SD-WAN's management protocol) processing done for the thread managing it. During a Gateway start, this leads to the VCMP thread continuously running at 100% for long durations (more than 60 seconds) leading to multiple mutex monitor related Gateway service failures.

  • Fixed Issue 112017: An Operator may observe that a VMware SD-WAN Gateway deployed with 4 cores experiences high load which leads to one or more a Dataplane Service failures.

    The Gateway core logs would point to the mutex monitor triggering the service failure. There are several tickets that address the above symptom, and in this case the cause stems from VCMP (management protocol) threads maxing out a 4 core Gateway's CPU processes, which triggers the mutex monitor. This ticket adds the ability to allow an Operator user to configure a VCMP half-open connection limit of 20. This can be done either through the Gateway's command line interface (CLI) using debug.py, or by a static configuration file.

  • Fixed Issue 112019: A VMware SD-WAN Gateway may experience a Dataplane Service failure and restart to recover under high CPU load.

    As with other Gateway service failure tickets in the 5.1.0.3 rollup build, the Operator or Partner would observe a mutex monitor trigger in the core file. With this ticket, the remediation is to move the NAT debug logs outside of the NAT table lock scope to prevent one of the causes for this issue.

  • Fixed Issue 112020: A VMware SD-WAN Gateway deployed with 4 cores which is under a high CPU load may experience a Dataplane Service failure and restart as a result.

    Looking in the Gateway core file a user would observe a mutex monitor failure caused by a Gateway process not being able to run because the CPU is running at maximum capacity because of a high tunnel count.

  • Fixed Issue 112800: Customers using a VMware SD-WAN Gateway may experience poor performance including tunnels and routes taking far longer to converge.

    When looking at the monitoring for a Gateway, a user would observe the dataplane cores (dp-cores) running at 100%, which results from the Gateway failing to flush stale flow dispatcher flows.

  • Fixed Issue 114052: A VMware SD-WAN Gateway may experience a Dataplane Service failure, generate a core, and restart as a result.

    The issue is caused by a thread in the Gateway's IPsec process timing out and triggering a Gateway service failure.

  • Fixed Issue 114084: For a customer who has configured a Zscaler-type Cloud Security Service (CSS) with L7 Health Check for a VMware SD-WAN Edge, when updating the Zscaler Cloud Server on the VMware SASE Orchestrator, the updated details are not applied to the Edge.

    Despite the Orchestrator showing the new Zscaler Cloud Server configuration, the Edge and Gateway do not send either traffic or the L7 probes through this new server but the old Zscaler server.

  • Fixed Issue 114282: When a VMware SD-WAN Gateway deployed with 4 cores is restarted, it may take up to twenty minutes to converge ~3000 tunnels for the connected customer enterprises.

    The expectation is for a Gateway to converge ~3000 tunnels in ~5 minutes versus the twenty minutes observed in this issue. The slower rate would cause customer traffic disruption until the tunnels were fully restored. The cause of the slower convergence is traced to a configuration in the Gateway's IPsec process which manages security associations and keys that is corrected with this ticket.

  • Fixed Issue 114932: VMware SD-WAN Edge client users may experience degraded traffic performance for traffic traversing the Edge's primary VMware SD-WAN Gateway.

    Operator Users can observe high CPU Utilization for the Gateway even though the tunnel count is within supported limits. The high CPU utilization arises from a stale IKE SA (security association) staying for a longer time in the IKE table and results in tunnels taking a longer time to converge and this causes increased traffic drops and path instability for customer traffic traversing the Gateway.

  • Fixed Issue 115604: A VMware SD-WAN Edge or Gateway may experience a Dataplane Service failure and generate a core with an Assert in the logging.

    When an Edge or Gateway processes a corrupted packet, the software can hit an assert where actual user packet length is more than internal packet buffer. The Gateway is expected to drop this kind of packet and prevent it from being sent to the Edge, but instead processes it and this results in the service failure and restart.

  • Fixed Issue 115692: An Operator may observe escalating memory usage in a VMware SD-WAN Gateway which can lead to memory exhaustion and a defensive service restart to clear the memory.

    In this instance the Gateway is experiencing an IKE memory leak from the Gateway renewing certificates with peer sites.

    Without a fix for this issue, the Operator can only monitor memory usage on the Gateway and proactively restart the Gateway in a maintenance window that ensures the least disruption to the customer sites using the Gateway.

  • Fixed Issue 116182: A VMware SD-WAN Gateway may experience a Dataplane Service failure and generate a core and restart as a result.

    This issue is observed on Gateways where the connected SD-WAN Edges are configured with an internet backhaul policy using either IPv6 or IPv4/IPv6 Mixed-Mode to a Non SD-WAN Destination (NSD) via Gateway. In this scenario, when the Gateway receives IPv6 packets destined for an NSD using IPv4, this triggers the Gateway service to fail.

Resolved in Edge and Gateway Version R5102-20230310-GA

Edge and Gateway build R5102-20230310-GA was released on 03-14-2023 and is the 2nd Edge and Gateway rollup for Release 5.1.0.

This Gateway rollup build addresses the below critical issues since the first Edge and Gateway rollup, R5101-20230112-GA.

  • Fixed Issue 98782: A VMware SD-WAN Gateway may experience a Dataplane Service failure during IPsec tunnel establishment, generating a core and restarting as a result.

    When a Gateway experiences this issue, the restart can result in a brief disruption of customer traffic for both Edges connected to that Gateway and Non SD-WAN Destinations using the Gateway for IPsec tunnels. The is caused by a race condition when the Gateway is establishing an IPsec tunnel triggers the Dataplane Service failure.

  • Fixed Issue 103708: When new rules are added in a BGP filter configuration, there may be unexpected BGP routes received and sent by the VMware SD-WAN Edge.

    When new rules are added to the BGP filters from the Orchestrator, the prefix lists are added in the Edge's routing configuration without removing the old entries. This behavior results in stale route prefix lists and unexpected filtering behavior.

  • Fixed Issue 104141: Users behind a VMware SD-WAN Edge or customers connected to a VMware SD-WAN Gateway may experience significant issues for any traffic that is using that Edge, or traversing that Gateway to the point that no traffic may be forwarded.

    When the issue is encountered, the Edge or Gateway has an unbounded number of memory buffers (mbufs) being consumed by the jitter buffer queue due to increasing management tunnel time stamps received from a peer. This triggers integer underflow in the jitter calculation, causing packets to be buffered effectively indefinitely. At first this only affects buffered flows, but over a long enough period the number of mbufs consumed for the jitter buffer queue approaches the total of mbufs available and the SD-WAN device (Edge or Gateway) can become unable to forward all traffic entirely. If this affects a Gateway it would only affect multi-path traffic that traverses the Gateway and customer traffic going direct would not be affected.

    Another ticket, #105744 also addresses the symptoms found here but fixes a separate cause. The difference between the two tickets: the fix included in #104141 addresses the memory buffers being consumed by the jitter buffer queue due to the increasing management time stamps received by the peer. The fix included in #105744 restricts the jitter buffer count to 25% of the total memory buffers no matter what else happens to ensure that this issue cannot recur.

    Without a fix for this issue for either the Edge or Gateway, a user can monitor the memory buffer (mbuf) usage on the Orchestrator and look for increased mbuf usage due to packets being queued in the jitter buffer. If the user does observe the issue they can flush flows for the Edge (through Remote Diagnostics) or Gateway to temporarily alleviate the issue but the issue would eventually recur until the fix was applied.

  • Fixed Issue 105744: Users behind a VMware SD-WAN Edge or customers connected to a VMware SD-WAN Gateway may experience significant issues for any traffic that is using that Edge, or traversing that Gateway to the point that no traffic may be forwarded.

    This ticket and Issue #104141 are directly related and have the same symptoms and cause which will be repeated here: when the issue is encountered, the Edge or Gateway has an unbounded number of memory buffers (mbufs) being consumed by the jitter buffer queue due to increasing management tunnel time stamps received from a peer. This triggers integer underflow in the jitter calculation, causing packets to be buffered effectively indefinitely. At first this only affects buffered flows, but over a long enough period the number of mbufs consumed for the jitter buffer queue approaches the total of mbufs available and the SD-WAN device (Edge or Gateway) can become unable to forward all traffic entirely. If this affects a Gateway it would only affect multi-path traffic that traverses the Gateway and customer traffic going direct would not be affected.

    The difference between the two tickets: the fix included in #104141 addresses the memory buffers being consumed by the jitter buffer queue due to the increasing management time stamps recieved by the peer. The fix included in #105744 restricts the jitter buffer count to 25% of the total memory buffers to ensure that this issue could not recur.

    Without a fix for this issue for either the Edge or Gateway can be monitored on the Orchestrator where a user would look for increased mbuf usage due to packets being queued in the jitter buffer and the user can flush flows for the Edge or Gateway to temporarily alleviate the issue but the issue would eventually recur until the fix was applied.

  • Fixed Issue 106587: A customer may observe all traffic randomly being dropped and the state persisting.

    The problem is related to IPsec Security Assocation (SA) installation on the responder side. When the Edge initiates a new SA or does a rekey, there is a chance that the old SA Security Parameter Index (SPI) packet arrives before the new SA (SPI). In such a case, the VMware SD-WAN Gateway deletes the newly created SA (SPI). The added fix will prevent deleting the new created SA (SPI).

    Without a fix for this issue, a Partner or Operatur user would need to reboot the Gateway to restart all the affected IPSec tunnels.

  • Fixed Issue 115136: A customer may observe a gradual memory usage increase on a VMware SD-WAN Edge in a customer enterprise that uses BGP for routing.

    The Edge's BGP daemon is causing a gradual memory leak on the Edge over several days and can do this even when BGP is not configured for that Edge. If the memory leak continues for a sufficient period to bring the Edge's memory usage beyond the critical threshold of 60% of available RAM for more than 90 seconds, the Edge will defensively restart its service to clear the leak which can result in customer traffic disruption for 10-15 seconds.

    The only remediation without an Edge fix is to restart the BGP process by terminating it, or preemptively perform an HA failover/Edge service restart in a suitable service window.

Resolved in Gateway Version R5101-20230112-GA

Gateway build R5101-20230112-GA was released on 01-19-2023 and is the 1st Gateway rollup for Release 5.1.0.

This Gateway rollup build addresses the below critical issues since the original Gateway build, R5100-20221204-GA.

  • Fixed Issue 97272: On a site with a High Availability topology where OSPF is used, when the site experiences a split brain condition (both SD-WAN Edges are Active), the default route to the core router is removed and the HA site cannot reach peer sites in the network.

    The core router has its link-state advertisement (LSA) age synchronized with the Active Edge. When an HA split brain condition is experienced, the Standby Edge moves to active and sends a new LSA age to the Core Router. Since both the Active and Standby Edge have the same Router ID, a different LSA age is sent by the new Active Edge. This mismatch causes the LSA age to be set to a maximum value of 3600 in the Core router which also removes the core route to the HA site, resulting in a complete outage at the site.

  • Fixed Issue 104487: Due to database service slowness, VMware SASE Orchestrator users may experience overall slowness and some API failures. Other side effects include SD-WAN Gateways/Edges appearing offline in the UI, configuration changes made through the Orchestrator UI not being pushed to the target SD-WAN Edges, and a loss of reporting capabilities.

    The issues are all caused by the Orchestrator's database service failing with the error: too many open files. This error is observed by the VMware SASE Orchestrator operator via logs. An end user accessing VMware SASE Orchestrator via the UI would experience slowness and intermittent API failures, causing error messages on the UI.

Resolved in Edge and Gateway Version R5100-20221204-GA

Edge and Gateway build R5100-20221204-GA was released on 12-08-2022 and resolves the following issues since Edge build R5012-20221107-GA and Gateway build R5011-20221007-GA.

Note:

Release 5.1.0 contains all Edge and Gateway fixes that are listed in 5.0.0 Release Notes, and all Edge and Gateway fixes in 5.0.1 Release Notes up to the above listed builds.

  • Fixed Issue 26085: A customer using a Hub/Spoke topology and Partner Gateways may observe traffic being dropped at a VMware SD-WAN Spoke Edge if one of the Gateways is unconfigured from a Hub Edge.

    The traffic dropped is using a stale route for a Gateway that is no longer assigned.  When a Gateway is unconfigured from a Hub Edge, the Gateway itself does not know this has occurred  and treats the event like a simple tunnel down event.  As a result the Gateway continues to provide the Spoke Edge with its route and the Spoke Edge does not remove the remote route (reachable via Hub Edge) because the Hub Edge is still reachable to the Spoke Edge.

    When this issue is present in the absent of a fixed build, the only way to remediate it is to slap the Spoke Edge to Gateway link.

  • Fixed Issue 29929: For a site deployed with a High-Availability topology, when there is an HA failover, a user may not be able to log into the Local UI for the HA Edges.

    When the local credentials are modified for the HA Edges on the Orchestrator, the correct Active Edge applies the change, but the new credentials are not synchronized to the Standby Edge. As a result, when there is an HA failover and the Standby Edge is promoted to Active, the Edge uses the default user/password credentials and a user trying to log into the Local UI is not successful if using the newer credentials.

  • Fixed Issue 32413: Temperature is not included in the Health Stats or the MIB.

    The fix adds CPU Temperature to the MIB used for SNMP and the metrics measured in Monitor > Edge > System, which is also known as "Edge Health Stats".

  • Fixed Issue 32654: Users at a VMware SD-WAN Edge site where a WAN interface is down may observe traffic dropping.

    The traffic drops as a result of Connected Routes remain advertised to a VMware SD-WAN Gateway though the reachability is False on the VMware SD-WAN Edge when the connected interface is down.

  • Fixed Issue 39134: When looking at the Monitor > Edge > System page, CPU Percentage is not accurate.

    Also known as the "Edge Health Stats", the VMware SASE Orchestrator does not receive accurate information about the Edge CPU usage and outputs this to graphs on the System page that are inaccurate and misleading to customers trying to troubleshoot an Edge issue.

  • Fixed Issue 45453: A customer cannot configure a VMware SD-WAN Edge to have 2 WAN interfaces uses the same VLAN.

    The issue arises to a scenario where a site connects multiple Edge WAN ports to the same L2 switch, on the same VLAN. The problem with this configuration is that the Edge can sometimes use the wrong Source MAC address when sending management traffic.

  • Fixed Issue 50920: VMware SD-WAN Edge does not send a warning when the number of connected tunnels reaches 60% of the hardware defined limit for that Edge model.

    The Edge sends a warning when the number of connected tunnels reaches its hardware limit that reads “Established tunnel count exceeds the device capacity”. Once this limit is reached, the Edge will not allow additional dynamic tunnels until existing tunnels are torn down. However, there is no intermediate warning sent to warn the customer that this tunnel limit has the potential to be reached, leaving the customer without proper lead time to manage their network.

  • Fixed Issue 53337: Packet drops may be observed with an AWS instance of a VMware SD-WAN Gateway when the throughput is above 3200 Mbps.

    When traffic exceeds a throughput above 3200 Mbps and a packet size of 1300 bytes, packets drops are observed at RX and at IPv4 BH handoff.

  • Fixed Issue 54846: VMware SD-WAN SNMP MIBs use counters for Jitter, Latency, and Packet Loss.

    In VMware SNMP MIBs, Latency, Jitter, and Packet Loss are defined as Counter64 which is not appropriate for these types. Counters should be used for data types that are ever increasing values and which never reset in SNMP like bytes Tx/Rx. In contrast, latency, jitter, and packet loss do not have ever increasing values but dynamically adjusted values and should not use counters.

  • Fixed Issue 55327: The SSH connection from a VMware SD-WAN Gateway to a VMware SD-WAN Edge may not work if the tunnel from the Edge to the Gateway continuously flaps.

    If the tunnel from Edge to Gateway flaps continuously, the route entry installed in the Edge for allowing the SSH connection from the Gateway may get deleted and cause the SSH connection to fail.

  • Fixed Issue 56153: For a customer enterprise where a Non SD-WAN Destination via Gateway is deployed and where BGP over IPsec is being used, if an inbound BGP filter is unassigned by the customer, the filter is not removed on the VMware SD-WAN Gateway and the route map is applied with it.

    This issue can cause unexpected routing for the customer since they are expecting the inbound BGP filter to be inactive when it is still being used by the Gateway and Edge.

  • Fixed Issue 60844: A Business Policy that is designed to drop all traffic that matches the policy criteria by configuring a Rate Limit of 0% does not work.

    While the configuration for a 0% Rate Limit is allowed, the Edge does not observe it as 0% but 100%, completely defeating the purpose of the Business Policy.

    On an Edge without the fix, the workaround is to use a Firewall rule to match and drop traffic versus a Business Policy.

  • Fixed Issue 61804: A customer enterprise using BGP may observe that when a VMware SD-WAN Edge learns routes from a peer that these routes are advertised back to the peer itself.

    The Edge should not be advertising peer-learned routes to the peer itself and this causes adverse routing behavior and traffic being dropped.

  • Fixed Issue 62701: For a VMware SD-WAN Edge deployed as part of an Edge Hub Cluster, If Cloud VPN is not activated under the Global Segment but is activated under a Non-Global Segment, a control plane update sent by the Orchestrator may cause all the WAN links to flap on the Hub Edge.

    The Hub Edge's WAN links going down, then up in rapid succession (flap) will impact real time traffic like voice calls. This issue was observed on a customer deployment where Cloud VPN was not activated on the Hub Edge's Global segment, but the Cluster configuration was configured as on, which means this Hub Edge was part of a Cluster (and a Cluster configuration is applicable to all segments). When a configuration change is pushed to the Hub Edge, the Hub Edge's dataplane will start parsing data and will start with the Global Segment where it will see Cloud VPN not activated and the Hub Edge erroneously thinks clustering has been deactivated on this Global Segment. As a result, the Hub Edge will tear down all tunnels from the Hub's WAN link(s) which will cause link flaps on all that Edge's WAN links. For any such incident the WAN links only go down and recover a single time per control pane update.

    On an enterprise where the Edges do not use a version with a fix for this issue, the workaround is to activate Cloud VPN on all segments, meaning the Global Segment and all Non-Global Segments.

  • Fixed Issue 64526: When a user changes the GE2 interface on a VMware SD-WAN Edge from switched to routed and then configures a subinterface on this interface and attempts to save the changes, the Orchestrator throws an error.

    The is triggered only when the Edge interface configuration is changed at the Profile level (versus at the Edge level). The error the user would see is "Unknown addressing type DHCP_STATELESS for subinterface GE2 - ignored" and this is seen under the Orchestrator's Events page for that Customer.

  • Fixed Issue 65530: A customer deploying Metanoia SFP's on a VMware SD-WAN Edge may observe issues with the module.

    The issues may arise from not being on a newer level of firmware that is updated with the 5.1.0 Release. The changes to the CSP version and SFP UPG Firmware version are shown in the table below:

    Edge version

    CSP version

    SFP UPG Firmware version

    5.1.0

    3.2.9.13

    1_62_8559

    4.0.0 - 5.0.x

    3.2.8.11

    1_62_8431

    These updates ensure greater stability for the Metanoia SFPs on the Edge.

  • Fixed Issue 65919: For a customer who has configured a Zscaler Cloud Security Service (CSS), if the Edge service restarts or DNS is deleted the Zscaler tunnel may fail.

    Even though DNS queries are successful, DNS does not show the Zscaler FQDN and as a result the tunnel does not come up. When checking Edge logs for DNS there will be no Zscaler entries in the DNS cache.

    To remediate the issue, a user would need to perform a nslookup or a ping, after which the DNS entry is created and the Zscaler tunnel comes up. 

  • Fixed Issue 67900: If a WAN link is configured as auto-detected on the VMware SASE Orchestrator, the Orchestrator may mark it as a private link even though it should be set as a public link.

    The Orchestrator requests that the link be set as private even though the customer has set the configuration for that link as public. This can cause significant connectivity issues with a Gateway as the link will be trying to connect to a private IP and fail in the process.

  • Fixed Issue 68335: For a customer enterprise using a Hub/Spoke topology where the Hubs are Clusters, VMware SD-WAN Spoke Edges which are unable to connect with a Hub Cluster may consume high amounts of bandwidth that is classified as SD-WAN control traffic.

    When a Spoke Edge fails to establish overlays with its datacenter Hub Clusters, the Edge requests controllers to reassign a different cluster member leading to controllers sending continuous control events and consuming link bandwidth.

    On an Edge without the fix, the workaround for this issue is to create and assign a temporary Profile without assigning the unable-to-reach Hub Cluster in its profile.

  • Fixed Issue 70248: When a CRL is issued on the Edge side, the tunnel goes down and reestablishes again.

    When the Orchestrator revokes any certificate, for example a Gateway certificate, the Edge brings down the tunnel but reestablishes again.

    The issue is related to CRL validity time. If CRL has an update time ahead of Edge time the tunnel will be reestablished again.

    In the absence of a fix, the only workaround is to ensure that the Edge and the Orchestrator have a date and time match.

  • Fixed Issue 70311: A VMware SD-WAN Edge may experience a Dataplane Service Failure and restart as a result.

    During the Edge service restart, customer traffic would be disrupted for ~15-30 seconds. This issue occurs inconsistently, but when it does occur the Edge is tearing down an IKE security association (SA). This typically only occurs when: the SA timer (as configured on the VMware SD-WAN Orchestrator) expires; or the user modifies the IPSec configuration on the Orchestrator.

  • Fixed Issue 71302: For a customer enterprise using Non SD-WAN Destination via Edge, the port used is 500 instead of 4500.

    This is not the expected behavior and can cause issues with traffic using that NSD via Edge if there is some other device blocking port 500.

  • Fixed Issue 71719: PPTP Connection is not Established along Edge to Cloud path.

    Connection to the PPTP server behind the VMware SD-WAN Edge does not get established.

  • Fixed Issue 75553: A customer deploying a Non SD-WAN Destination (NSD) via Gateway that uses a Policy Based type is not able to configure redundant VMware SD-WAN Gateways.

    In previous builds VMware would always mark a Policy Based NSD route as "Reachable" regardless of status with the effect being that the NSD's Primary Gateway route would never be marked as unreachable and trigger a failover to a redundant Gateway.

    In Release 5.1.0 and later, the Gateway path is marked as reachable unless it fails the IKE/IPsec negotiation, at which point it would be marked as "Unreachable" and NSD traffic would traverse via the redundant Gateway as it does with a Route Based NSD.

  • Fixed Issue 75668: The DSCP tag is reset for LAN side traffic when it is routed to an internal LAN destination.

    For the routed/direct user traffic, the Edge resets the DSCP tag to 0 and traffic that ingresses and egresses on the same Edge (in other words, stays local to the Edge) has the DSCP tag modified to a CSP=0DSCP marking and is reset to CS0 for underlay traffic when it traverses the Edge.

  • Fixed Issue 76881: A VMware SD-WAN Edge may experience critical memory usage levels when more than 10,000 DHCPv6 leases are used and potentially trigger an Edge Service restart.

    When a large number of DHCPv6 clients are connected to the DHCPv6 server, each client would be provided with a lease and the memory of the DHCPv6 server would continue to grow. The Edge does not limit the number of DHCPv6 leases which can be allocated in contrast to the hard limit of 5K IPv4 addresses. As a result there is the potential for the Edge to allocate enough leases to reach a critical level of Edge memory state and trigger a service restart.

    The fix for the issue restricts the maximum number of clients to 10,000.

  • Fixed Issue 77066: A VMware SD-WAN Gateway may experience a Dataplane Service failure and trigger a core and restart the service to recover.

    The issue is triggered by a memory corruption of the Gateway caused by two Gateway processes that respectively handle transmission and reception packets simultaneously trying to access the same node in a search tree.

  • Fixed Issue 77457: If a user tries to generate a packet capture (PCAP) for an interface on a standby VMware SD-WAN Edge, the VMware SASE Orchestrator reports that the PCAP has failed.

    When a user tries to generate a PCAP for the Standby Edge in an Enhanced High Availability deployment, the Orchestrator UI records the Request Status as Failed and the explanation "Failed to upload diagnostic bundle: 'unicode' does not have the buffer interface."


  • Fixed Issue 77611: For a customer site using a High-Availability topology, when the HA Edge is migrated to a difference configuration Profile, both Edges may enter into an Active-Active state (split brain) and restart at the same time to recover.

    During this Active-Active state the client users would observe significant traffic quality issues from packet loss.

    The issue is caused by a failure of the HA Edges to follow a process when moving to a new profile where the Standby Edge should restart first to apply the changes and remain as the Standby. Then the Active would apply the configuration changes and restart and only then promote the Standby to Active while it demoted itself to Standby. Instead, the Standby Edge immediately promotes itself to Active after its restart, causing the Active-Active state.

  • Fixed Issue 78037: A VMware SD-WAN Edge may experience a spike in memory usage followed by a Dataplane Service failure there a DHCPv6 server is configured with more than 1K addresses.

    Issue occurs for both route and switched interfaces. Issue can occur when +1K addresses are configured for Clients on a DHCPv6 server. When over 1K clients are getting addresses the quantity of DHCPv6 solicit packets gneerated leads to Edge memory exhaustion and the failure of the Edge service.

  • Fixed Issue 78050: A VMware SD-WAN Edge may experience a Dataplane Service failure when a PPTP server is present on the LAN side.

    When a PPTP server is present in the LAN side, and a PPTP client from the internet connects to it via an inbound firewall rule, the Edge service can fail due to a PPTP control channel lookup failure. This control channel lookup is needed to ensure the GRE data channel is sent out via the same link back to a PPTP client.

    On an Edge using a build without a fix for this issue, a customer's only alternative is to not use PPTP sessions.

  • Fixed Issue 78435: A VMware SD-WAN Edge that is activated with a URL through the Local UI may throw an error that the Edge activation failed, when it actually succeeded.

    URL activation of Edge with local UI throws error "edge activation could not be completed"


    The issue occurs because the Edge refers to an older activation request with incorrect parameters when responding to the request for activation status. Meanwhile, the current activation request with the correct parameters is actually processing. As a resultthe Local UI throws an error even though the Edge activation is processing correctly.

  • Fixed Issue 79437: A VMware SD-WAN Gateway deployed on a server using an Intel X710 NIC with SR-IOV enabled for interfaces may fail to deploy.

    If this failure occurs, the Operator observes that the X710 SR-IOV interfaces are removed from DPDK and not seen when running debug.py --dpdk_ports_d. In addition, /opt/vc/etc/dpdk.json does not display the SR-IOV interfaces. Deploying a Gateway to the 5.1.0 Release ensures this issue would not be encountered.

  • Fixed Issue 79338: When a large number of DHCPv6 clients try to fetch a new IPv6 address, a few clients may not be able to get one.

    Where there are more than 1024 clients trying to acquire an IPv6 address at the same time, some of the clients may not get a valid IPv6 address. This issue occurs because the neighbor cache is fixed and the neighbor entries of a few clients would not get created. Since the neighbor entries are not present, the renew messages would fail.

    For a customer not using a build with the fix, they can retry acquiring IPv6 address from the clients after a few minutes.

  • Fixed Issue 79550: A VMware SD-WAN Gateway or SD-WAN Edge may experience a Dataplane Service failure and restart.

    This issue can occur if a repeated managment traffic (VCMP) fragment is received, causing the process to write beyond the allocated buffer for that packet and triggering the failure.

  • Fixed Issue 81881: A Configure > Device configuration change made on the VMware SASE Orchestrator may not be applied to the targeted VMware SD-WAN Edge even though the Edge is connected to the Orchestrator.

    And Edge can experience this issue under a heavy load where there are frequent, rapid configuration changes. In that scenario the Edge's management thread responsible for configuration application becomes stuck and does not apply additional changes.

  • Fixed Issue 81353: A VMware SD-WAN Gateway deployed using an Azure platform may be observed to drop packets at Rx of interfaces.

    The packets are dropped due to a low buffer. The ring buffer setting was not part of the Non-DPDK managed interfaces used by Azure platforms and the NIC Rx ring buffer queues are set as low numbers.

  • Fixed Issue 81355: VMware SD-WAN Gateways deployed using the Azure platform may experience issues with packets a size greater that 1500.

    The packets greater thant 1500 bytes are dropped with error message: pkt_too_big_drop. Packets much larger than 1500 bytes are dropped with error message: sock_too_big_dropp.

    The issue is the result of the Azure platform not using DPDK bounded interfaces which keeps the Gateway's DPDK.json list empty and the DPDK network configurations do not inititialize the Linux interface's TSO/GSO settings.

  • Fixed Issue 81377: When the Secure Access subnets are updated, the old subnets are not revoked from BGP.

    When a Secure Access subnet configuration is modified, SD-WAN only deletes the old subnets from the Forwarding Information Base (FIB) and other local route tables. The issue is that SD-WAN is not revoking the route from the BGP table used for redistribution leading to these obsolete routes remaining in BGP.

  • Fixed Issue 81483: For a customer with a Hub-Spoke topology, if a user increases the IKE and/or IPsec lifetimes via the VMware SASE Orchestrator, the customer may notice that some tunnels retain the old lifetimes. Because of this, some tunnels will perform rekeys more frequently than expected because they retained the previous lifetimes.

    If the customer decreases the lifetimes they will technically hit this bug (one end of the Hub-Spoke technology may retain the older, larger lifetime) but will not observe any difference in functionality because the Edge that correctly received the lower lifetime will perform the rekeys first, hence masking the issue.

    The only workaround for this issue is to change the lifetime values until all the tunnels reflect the correct the state and then reboot the Edge.

  • Fixed Issue 82104: In rare cases, VMware SD-WAN Edges activated in a High Availability topology may be unable to communicate with a VMware SASE Orchestrator which will mark the site as down and preclude any intervention through the Orchestrator to the site.

    This issue occurs only when an unusual and invalid configuration is applied to the HA Edges. The configuration specifies that the HA port is configured as "trunk" (which should not be allowed), with zero VLANs (also should not be allowed), but where "all VLANs" are set. Instead of throwing an error at this configuration and preventing a user from activating HA for the Edges, the Orchestrator allows it. This allowed configuration triggers a management plane failure on the HA Edges which no longer send a heartbeat to the Orchestrator. The fix here allows this invalid configuration to work on an HA Edge pair without breaking the management traffic to the Orchestrator.

  • Fixed Issue 82188: For VMware SD-WAN LTE models (Edge 510-LTE, 610-LTE), when the IPv6 setting is toggled off, the tunnels via CELL interface may fail.

    When the IPv6 checkbox in Device Settings is turned off, the internal state of the CELL interface goes to UNKNOWN. This is not updated even when the IPv6 setting is toggled to on for the CELL interface. Due to this the tunnels via the CELL interface would fail. If the LTE Edge is only using CELL interfaces for traffic this result in the Edge going offline and would drop all Edge traffic which would be very disruptive to the customer.

    Without the fix, a user would need to restart the Edge Service after toggling the IPv6 setting off. If the Edge is offline because it only uses the CELL interface for bandwidth, a local user would need to power cycle the Edge to recover it.

  • Fixed Issue 83040: A customer enterprise with a Hub/Spoke topology that uses both Partner Gateways and Non SD-WAN Destination (NSD) may observe traffic that should use an NSD instead uses a Hub.

    The Spoke Edge would have a business policy which backhauls traffic to the NSD and if a Partner Gateway handoff is also configured for it, the Spoke sends traffic that should use a NSD instead to the Hub Edge. The Hub in turn sends the traffic direct to the internet. If the Partner Gateway handoff is disabled, then this NSD traffic is routed properly.

  • Fixed Issue 83227: For a VMware SD-WAN Edge running Release 5.0.0 which is configured with 128 Segments, the Edge's dnsmasq process will stop and exit.

    When IPv6 is activated on 128 segments and DCPv6 servers are configured in the LAN of each segment, the dnsmasq process will stop as the total open file descriptors is exceeded. The dnsmasq process will continue for ~30 minutes before exiting at which point the Edge's DHCP assignment of IP addresses will fail.

    Rebooting the Edge restores the dnsmasq process for ~30 minutes but it will fail again. The only real workaround is to reduce the number of segments to less than 128.

  • Fixed Issue 83821: For customers using NetFlow, Tx and Rx packets/bytes for SD-WAN control traffic are not update properly in NetFlow records.

    The SDWAN control data (Tx/Rx packets/bytes) that are exported to a NetFlow collector are always zero. Since the entries are not populated in the flow container's chat_stats, the NetFlow also does not export the data.

  • Fixed Issue 84000: A VMware SD-WAN Gateway connected to Edges deployed in a dual stack (IPv4/IPv6) configuration where is a high frequency of tunnel teardowns and creations with the Edges may experience a memory leak which if sufficiently high would trigger a service restart.

    When a VCMP (Encrypted Management) tunnel is created and deleted multiple times for an Edge with a dual stack configuration, in the Gateway for that Edge there can be the appearance of a pi leak if the Gateway is operating at high scale. There is no real pi leak, but pi deletion is happening slowly and this slow delete rate can cause shared memory issues which may ultimately become critical.

    On a Gateway without the fix, a service restart will temporarily clear the memory.

  • Fixed Issue 84136: Customer may observe high CPU utilization and poor traffic performance on a VMware SD-WAN Edge upgraded to some Edge releases.

    This issue occurs on an Edge where there are more than 400 IP rules configured under the Configure > Firewall > Edge Access section (Support Access or SNMP Access allowed IP addresses). When the Edge tries to send the firewall configuration in that scenario the management process maxes out the CPU and it times out and then this process repeats.

    On customer sites that are also using a High-Availability topology, the symptoms would include "HA unknown events" because the Active Edge is not sending heartbeats within the expected time window.

  • Fixed Issue 84225: When a connected VMware SD-WAN Edge interface goes down, the configured subnet is still redistrubted to OSPF or BGP.

    The Edge attracts traffic from the peer for the subnet when it is down and may cause traffic outages due to the peer preferring this path over other alternates to the subnet.

    On an Edge without a fix for this issue, the user needs to reboot the Edge in a planned service window to clear the issue.

  • Fixed Issue 84313: On a customer enterprise with a Hub/Spoke topology, an IPv6 overlay link may be advertised to the underlay peers of VMware SD-WAN Spoke Edges.

    On configuring an IPv6 address on the overlay and enabling advertise, the same address is getting advertised via the underlay as well.

  • Fixed Issue 84741: A user observe inaccurate throughput statistics on the Monitor > Transport screens.

    RX (incoming) packets and link statistics are not incremented to the Orchestrator for traffic which is sent direct on an interface where Reverse Path Forwarding (RPF) is deactivated on the WAN Overlay.

  • Fixed Issue 84790: When a VMware SD-WAN Edge with any model type other than 510/510-LTE is rebooted, the Edge may erroneously report the critical event Unable to launch service wifihang to the VMware SASE Orchestrator.

    The wifihang event message is designed for use only with the Edge 510/510-LTE models and alerts a customer to a problem with that Edge model's Wi-Fi process. When this event message is observed on any other Edge model, whether that model uses Wi-Fi or not (for example: the Edge 3400), the event message is spurious and the event can be safely ignored.

  • Fixed Issue 85154: When a VMware SD-WAN Virtual Edge on AWS with instance type C4.xlarge is upgraded from an older Edge release to a newer release (including Release 4.3.1) and is then downgraded back to an older Edge release, the Edge goes into a deactivated state where the Edge does not form management tunnels with the Gateway and Orchestrator.

    The cause of the issue is the Orchestrator erroneously deactivating the Edge because of what the Orchestrator detects as a serial number mismatch.

    There is no workaround for this issue beyond NOT downgrading from the newer Edge release once the AWS Edge is on this release.

  • Fixed Issue 85156: For a site deployed with a High-Availability topology, the customer may observe multiple reboots of the VMware SD-WAN Standby Edge with a potential disruption to customer traffic.

    The HA control data synchronization processing logic on the Standby Edge for data received via TCP can lead to the data getting only partially read. This can cause multiple such short messages to be processed on the Standby which can slow down the Standby node. In low-end Edge platforms (for example, Edge models 510, 520, 610, 620), this slow down can significantly impact heartbeat processing between the Active and Standby which leads to the Standby Edge incorrectly being promoted to Active. In an Active-Active state the tie-break goes to the Active Edge and the Standby Edge is rebooted to demote it back to its proper Standby status.

    When this issue is encountered on a conventional HA topology the customer impact would be minimal as the Standby Edge does not pass customer traffic. On an Enhanced HA deployment, where the Standby Edge is also passing traffic, the reboot(s) would disrupt some customer traffic. 

    The fix for this issue adds enhancements in the Edge TCP message processing logic to improve the performance on the Standby Edge and prevent a system slowdown.

  • Fixed Issue 85269: For customer enterprises using a Hub/Spoke topology where Multicast is configured, a multicast receiver may not receive the traffic source behind a Hub Edge where the source IP address is manually set either at the Hub Edge or a spoke Edge but not both.

    Pimd logs and debug commands would provide users with the confirmation if PIM join send fails due to a mismatch between a PIM neighbor IP address and a next-hop IP address, preventing LHR from reaching the RP and registering (*,G).

  • Fixed Issue 86098: For a site using an Enhanced High-Availability topology where a PPPoE WAN link is used on the Standby Edge, a user may observe that the default proxy route is not installed in the Active Edge and traffic using that link fails.

    When an Enhanced HA Edge pair come up, the PPPoE link synchronizes with the Standby Edge and provides a default route with a next hop of 0.0.0.0.  As a result this route is not installed on the Active and traffic using this link is dropped.

  • Fixed Issue 86994: On a customer enterprise where Dynamic Branch-to-Branch is activated, when attempting to troubleshoot a VMware SD-WAN Edge in this enterprise the dispcnt debugging command does not work.

    The dispcnt debug command does not provide all the counter values and fails with Domain (null) does not exist. This also fails when referring to the relevant logs in an Edge diagnostic bundle. This significantly hinders troubleshooting a customer network issue.

    This issue arises in enterprises where Dynamic Branch-to-Branch is activated due to the large quantity of tunnels that are created and torn down towards each peer. The counters to store various metrics of the peers are stored in a shared memory and over time, these shared memory segments get into a bad state due to a collision and the counters are not fetched by the dispcnt command.

    This issue can only be cleared by performing a service restart of the Edge.

  • Fixed Issue 87205: For a customer deploying a VMware SD-WAN Edge with a Partner Gateway, when an Edge learns new routes from the Partner Gateway, customer traffic may be disrupted.

    This issue is caused by traffic matching the wrong Business Policy. For example, DHCP traffic destined for the Partner Gateway could instead be matched to the Internet Backhaul rule with a resulting disruption in customer traffic.

    Without the fix, the issue is remediated by flushing the Edge's flows using the Remote Diagnostic > Flush Flows. This remediation does not prevent future potential occurrences when new routes are learned by the Edge to the Partner Gateway.

  • Fixed Issue 87552: When a VMware SD-WAN Edge is configured to use either a Non SD-WAN Destination (NSD) via Edge or a Cloud Security Service (CSS), the VMware SD-WAN Edge may periodically experience a Dataplane Service failure and restart when NSD or CSS tunnels are unstable.

    When the Edge tears down a tunnel to either an NSD or a CSS (IPsec or GRE), the incorrect release of a previously chosen tunnel is performed that triggers an exception in the Edge Dataplane Service and an Edge Service restart to restore the service. Restarting the Edge service will result in a 10-15 second disruption of customer traffic.

    On an Edge without a fix for this issue, associating an NSD via Edge or CSS to one WAN link will decrease the likelihood of this issue occurring. In other words, instead of configuriing an NSD or CSS on multiple WAN links, choose one WAN link only. This will lose redundancy but will mitigate the impact of this issue.

  • Fixed Issue 88055: On VMware SD-WAN Edge models 3x00, a customer may observe that when the throughput is sustained at 10 Gbps or greater, the WAN path latency may become stuck and degrade the stability and throughput of the Edge.

    In 10G environments with rapid clock drift between VCMP endpoints, WAN path latency measurements can get stuck which impairs the effectiveness of Dynamic Mutlipath Omptization (DMPO) and this leads to incorrect path selection and throughput degradation.

  • Fixed Issue 88152: SNMP requests to a VMware SD-WAN Edge's subinterface do not work.

    This is a day one behavior and any SNMP requests to a subinterface of the Edge will timeout. The fix for this issue adds support for these SNMP requests to an Edge's subinterface.

  • Fixed Issue 88317: On a VMware SD-WAN Edge which uses both public and private links and has SD-WAN Reachable configured, when a public link goes down, direct traffic does not use the private link as expected.

    When a business policy is set to prefer the pubic link, the flow does not use the SD-WAN reachable private link while the preferred public link is down. The fix adds the logic to allow SD-WAN reachable links as well when direct link selection tries to find out private links as a last resort.

  • Fixed Issue 88550: For customers using Edge Network Intelligence, a VMware SD-WAN Edge is not able to communicate with the Edge Network Intelligence service when a DNS is not explicitly configured.

    When DNS is not configured explicitly, the Edge Network Intelligence service uses Google DNS by default. If DNS chooses a Loopback interface as a source interface, then reachability to the service is broken due to DNS lookup failure.

    For a customer enterprise not using an Edge build with the fix, the workaround is to configure DNS explicitly on the Orchestrator and choose a real interface as the source interface versus a Virtual Loopback interface.

  • Fixed Issue 89364: For a site using an Enhanced High-Availability topology, if a user runs Remote Diagnostics > Interface Status, the link speed of the Standby Edge interface shows as 0 Mbps / Half-Duplex.

    Speed and auto-negotiation details are not fetched from the Standby Edge where the interface is up, and the details are not displayed correctly.

  • Fixed Issue 89596: A VMware SD-WAN Edge may experience a Dataplane Service failure and restart as a result, disrupting customer traffic.

    This issue can occur when a customer has configured NAT. When a new flow using NAT is established there is a very rare race condition which may trigger an exception in the Edge service that causes a failure and a restart.

    Without a fix for this issue, the only way to prevent the issue is to disable NAT.

  • Fixed Issue 89725: For VMware SD-WAN Edges using Edge software releases prior to 5.1.0, the Remote Diagnostic utilities WAN Link Bandwidth Test and Traceroute may not work.

    The WAN Link Bandwidth Test and Traceroute utilities require additional input for the interface (for BW Test) or IP Address (for Traceroute). When this issue is encountered, the user cannot configure those inputs as the dropdown menu option is not shown and thus the test in either instance results in an error.

  • Fixed Issue 90044: When a VMware SD-WAN Gateway is configured with an ICMP probe and the Gateway is restarted, the ICMP probe does not recover and remains down.

    The ICMP Probe state in debug.py --icmp reads as DOWN after a Gateway restart.

    On a Gateway without a fix for this issue, the workaround is to deactivate the ICMP probe and then reactivate it.

  • Fixed Issue 90098: For a customer enterprise where Branch-to-Branch VPN is configured, in some scenarios a tunnel can be tried endlessly even though it cannot ever come up due to a configuration change.

    The scenario involves an Edge trying to create a tunnel with a peer Edge that is either offline or had an IP address changed. The Edge does not realize the peer is unreachable and endlessly tries to create a tunnel to the non-existent destination which impairs overall performance and cannot be stopped by the customer.

    The issue is caused by the lack of expiration time limit for non-working Branch-to-Branch tunnels. In addition the issue is difficult to troubleshoot because there is no message generated about where the Edge is getting the Branch-to-Branch message reply and there is no debug command on the connected Gateway to display the valide Branch-to-Branch information for a peer.

  • Fixed Issue 90216: Traceroute might not show the correct IP address of a VMware SD-WAN Hub Edge when the traffic flow is from Client > Spoke Edge > Hub > Server.

    If a Spoke Edge has a configured Business Policy to backhaul its traffic to a Hub Edge with Transport Group configured to use Private Wired and Mandatory, when the traceroute packet reaches the Hub Edge, the Hub Edge responds with the incorrect IP address (in this case, the public IP address, instead of the private IP address) to the traceroute.

  • Fixed Issue 90884: For a customer enterprise using a Hub Cluster/Spoke topology, when a Hub Edge in the Cluster is reassigned to one or more Spoke Edges, the users at those Spoke Edge locations may experience traffic failure.

    Hub Edges in a Cluster can be reassigned as part of a Cluster rebalance when an enterprise upgrades their Edge software, so this issue may be observed post-upgrade. When this issue is encountered, the VMware SD-WAN Gateway does not send the new Spoke Edge routes to the Hub Edge because the Gateway is expecting all Hub Edges to have all Spoke Edge routes, and thus these routes are not in the Hub Edge's routing table. As a result traffic between the Spoke Edges and the Hub Edge in the Cluster is impacted because the forwarding path is down.

    If the issue is encountered in an enterprise not using Gateways with a fix for this issue, it can be temporarily resolved by performing a route reinit on the Hub Edge. However, the issue may recur on a new Hub Edge reassignment.

  • Fixed Issue 91164: On a customer enterprise deployed with a Hub/Spoke topology where the VMware SD-WAN Hub Edge is configured for High-Availability, the HA Hub Edge may not forward internet backhaul traffic after an HA failover.

    The issue is confined to a scenario where the Standby Edge does not set the destination route for internet backhaul flows when the backhaul flow is configured to route via a static route using a non-WAN overlay interface. When the Standby Edge is then promoted to Active in an HA failover these factors cause the internet backhaul traffic to fail.

  • Fixed Issue 91188: An IPv6 ping to a host connected to a VLAN on a switched interface fails if done when using Remote Diagnostics > Ping and selecting the VLAN interface as the source interface.

    The source interface name 'VLAN-x' is known only to the VMware SD-WAN Edge and the Edge OS needs the source interface in the form 'br-networkx' as the VLAN interface is created with that name in the Edge's OS.

  • Fixed Issue 91203: For a customer enterprise configured with a Hub/Spoke topology where the VMware SD-WAN Spoke Edge is configured to backhaul traffic through a Hub Edge, a user may observe poor traffic performance for backhauled flows.

    The backhaul leg on the Hub Edge is determined by the Source and Destination route types (in other words, Source = enterprise, Destination = cloud) but this approach may lead to inconsistent behavior as it depends on incidents based on route changes and can result in dropped packets for backhauled flows. The fix for this issue is to make the backhaul leg determination based on the Spoke Edge's messaging.

  • Fixed Issue 92454: The Remote Diagnostic > Traceroute does not work when a domain name that only resolves to an IPv4 address is entered in the Destination field.

    If a domain name resolves to an IPv4 address only, the Traceroute command executed through Remote Diagnostics does not work. This is because the VMware SD-WAN Edge always tries to resolve the domain name for the IPv6 record and fails to find the IPv4 address.

    On an Edge without this fix, the workaround is to use the IPv4 address corresponding to the domain name directly in the Traceroute command. The IPv4 address can be obtained by supplying the domain name to the Remote Diagnostic > DNS Test.

  • Fixed Issue 92758: A site with a High-Availability topology may experience several different issues on the VMware SD-WAN HA Edges including in incorrect LED status or an HA failure.

    The incorrect LED status on the Active Edge shows as Yellow instead of Green even though the Edge is up and the WAN links are up and stable.

    This issue is traced to a shared memory corruption on the Edge which manifests in several forms. This can be confirmed by fetching the counters with getcntr tool for a specific domain such as vcedge.com. The output of the tool shows “Domain does not exist” and the counter name is not found.

    VMware SD-WAN relies on the ftok() system call to derive keys of SYSV shared memory. ftok() uses the last 16 bits of inode for calculating the key. This can cause a key collision when inode numbers differ by at least 64K . When such collision occurs, the dynamic tunnel shared memory counters can corrupt global shared memory variables resulting in several possible Edge issues including an incorrect LED status, inability of counters, or HA failure.

  • Fixed Issue 93062: When a user runs the Remote Diagnostic "Interface Status" on the VMware Orchestrator, the Orchestrator either returns an error for that test and does not complete or the test does not return results for routed interfaces.

    The error message seen is "error reading data for test". If the test does complete, the results for routed interfaces are empty with no information about speed or duplex. Either way the Interface Status is broken. The issue is related to the debug command that underlies Interface Status omitting DPKD activated ports.

    On an Edge without this fix, the user would need to generate a diagnostic bundle for the Edge to see the status for routed interface

  • Fixed Issue 93853: A VMware SD-WAN Gateway under a heavy load may experience a Dataplane Service failure with a SIGXCPU code and restart the service to recover.

    Under heavy load, several Gateway threads performing various activities such as routing and logging are starved of CPU resource and are not able to complete the task within the stipulated time frame. The Gateway service interprets these lagging threads as deadlocked and raises the SIGXCPU signal with a subsequent Gateway Dataplane process termination.

  • Fixed Issue 94204: A user may observe that attempts to generate a diagnostic bundle for a VNF capable VMware SD-WAN Edge fail.

    A diagnostic bundle fails to complete on a VNF capable Edge because the Edge runs out of disk space. This can happen if the Edge has generated one or more cores and is caused by the Edge sending these cores to the /vnf/tmp folder. Each core is unpacked in the /vnf/tmp folder and due to a core's unpacked size quickly fills this folder which causes the diagnostic bundle to fail. 

    VNF (Virtual Network Function) capable Edges include the following models: 520v, 620, 640, 680, and 840.

  • Fixed Issue 94401: On a VMware SD-WAN Edge where the Stateful Firewall is enabled, a TCP Established flow may time out too quickly and get flushed.

    The TCP Established flow, is treated as a TCP Non-Established flow and is subject to a shorter timeout. When there is a TCP Reset (RST) seen in a TCP flow, followed by a TCP 3-way handshake, even though the TCP state shows as Established, the flow gets flushed after being subjected to a Non-Established TCP flow timeout.

  • Fixed Issue 94612: Traffic to the BGP network prefix may not be reachable.

    When this issue is encountered, the BGP network prefix is not configured under BGP and is not advertised to peer nodes.

    On an Edge without a fix for this issue, the only workaround is to delete the BGP networks and reconfigure them.

  • Fixed Issue 94775: On a customer enterprise using a Hub/Spoke topology where the VMware SD-WAN Spoke Edge backhauls their traffic through a Hub Edge, client users may observe traffic performance issues.

    The is caused by the wrong flag being set for backhauled traffic, the backhauled packets are handled on the Spoke Edge as if they were on a Hub Edge. This leads to route lookup issues on the Hub and the backhaul packets get dropped.

  • Fixed Issue 94980: For a site deployed with a High Availability topology, the VMware SD-WAN Standby Edge may experience a Dataplane Service error and restart after a PPPoE WAN link is configured for the HA Edges.

    When examining the core generated by the Standby Edge, a user would see the message vc_is_use_cloud_gateway_set after the PPPoE link is configured.

    On an HA site without a fix for this issue the customer would need to ensure that a PPPoE link was only configured in a maintenance window to manage the risk of this issue.

  • Fixed Issue 95047: When a security port scanning utility scans a VMware SD-WAN Edge where Edge Network Intelligence (Analytics) is not activated, the scan will report that Syslog Port 514 is closed, which means it could be accessible.

    Edge Network Intelligence listens on Port 514 (Syslog). If Analytics are not activated, the Port 514 is still accessible, but it will not respond to requests. Therefore, a port scanner reports the port as "closed" (in other words, the port is accessible but there is no application listening on it).

  • Fixed Issue 95121: When a "locked SIM" (a SIM that is password locked) is used in a VMware SD-WAN Edge model 510-LTE or 610-LTE, the customer will experience failures establishing connection in the network.

    Users encounter failure in path establishment when using locked LTE SIM cards with the SIM slots of Edge 510-LTE and 610-LTE models because SIM unlock is not working from the Orchestrator and this is due to lack of support for locked SIMs in the Edge's ModemManager scripts.

  • Fixed Issue 95501: For a customer enterprise that uses a Hub/Spoke topology and BGP for routing, client users at VMware SD-WAN Spoke Edges may observe poor traffic performance.

    An administrator would observe that the Spoke Edge prefers routes marked with uplink community from a Hub not included in its profile over the Hub Edge configured to be used for that Spoke Edge. This is because the Spoke Edge traffic is taking a Dynamic Branch to Branch path for uplink prefixes.

    The issue is caused by SD-WAN resetting the uplink flag for routing messages received from a Hub Edge. As a result, when a Dynamic Branch to Branch tunnel is formed, direct routes are installed for these uplink prefixes leading to suboptimal routing and degraded traffic performance.

  • Fixed Issue 95565: On a site using a High Availability topology, the VMware SD-WAN Active Edge may experience a Dataplane Service failure with a core generated and triggering a High Availability failover.

    The issue is triggered by the Active Edge's WAN links flapping one or more times (going down and then come up rapidly) while also using SNMP where there are frequent SNMP queries. There is a timing issue where the interface coming back up and the SNMP query together can trigger a deadlock which causes the Dataplane Service to fail and generate a core. While only a single WAN link flap can cause this issue, the greater the frequency of WAN link flaps, the greater the potential for this issue to occur.

    On an HA Edge pair that experiences this issue and does not have the fix, the workaround is to disable SNMP as this is a timing issue and this reduces, but does not eliminate the risk.

  • Fixed Issue 96626: When a VMware SD-WAN Edge interface has a secondary IP address assigned to it, connections through the secondary IP address fail.

    Any request coming from another branch to an IP in the secondary network will generate an ARP from the primary IP address rather than the secondary IP address. As a result, the ARP would remain unresolved, leading to failure in the traffic going through the secondary IP address.

  • Fixed Issue 96739: When a user looks at the Monitor > Application tab for a VMware SD-WAN Edge on a VMware SASE Orchestrator, the screen may show Destination FQDNs with the wrong domain names.

    This issue can occur when the Edge's statistics reach its limit (known as an overflow condition), and instead of displaying these statistics as Overflow, the Orchestrator displays random domain names on the Application tab's Destination FQDN.

  • Fixed Issue 96994: When doing an SNMP walk on an VMware SD-WAN Edge, some of the interfaces may not be visible.

    Missing interfaces can be valid interfaces which were meant to be visible on snmpwalk. But due to the appearance of an invalid interface in the Edge's hardware list the valid interfaces appearing after the invalid one in the list will not be visible or returned by snmpwalk. An interface here is invalid if it appears in the hardware list but does not appear when running the ifconfig command on the Edge.

    While this issue can potentially affect any Edge, it has a greater chance of being encountered on a Virtual Edge deployed using Azure. This is due to the Azure Edge's tendency to list a higher number of interfaces on its hardware list versus the number of interfaces identified in the Edge itself using ifconfig.

  • Fixed Issue 97152: When a customer enterprise has a Business Policy configured with a Service Group as anything wired and Link Mode as "Available", traffic is not steered over to a wireless link when the wired link(s) goes down and client users at the site would observe their traffic that matches that rule is failing.

    When a Business Policy rule has explicit Service Group of wired WAN links with a Link Mode of Available, and there are wireless links available at the site, the expectation is that traffic using that rule would fail over to the wired WAN link(s) if the wired links in the service group went down (in other words, became unavailable) to ensure the seamless flow of traffic matching that rule. In this issue, steering the traffic to the wireless links is not occurring.

  • Fixed Issue 97321: From the time a user activates Edge Network Intelligence Analytics on a VMware SD-WAN Edge, the Edge can potentially trigger an Edge Service restart, each instance of which causes 10-15 seconds of customer traffic disruption.

    When Analytics is enabled on the Edge, the Edge can experience an out of memory condition followed by a "double free" memory state. The Edge restarts its service to restore memory.

    The symptoms for this issue can happen multiple times while Analytics are activated.

  • Fixed Issue 99718: BGP neighbor does not get established when the secondary IP address on a SVI (Switch Virtual Interface) is used.

    When the Edge processes ingress packets, it verifies if the ingress packet's destination address matches with the ingress interface's IP address. Since only primary IP addresses are compared, packets with a destination IP address as a secondary IP address are dropped. As a result, the BGP session is not formed on this secondary IP.

  • Fixed Issue 100363: A VMware SD-WAN Gateway may experience a Dataplane Service failure and trigger a restart of the service, which results in a 1-5 second disruption of traffic.

    This issue occurred during stress testing with the failure occurring at futex_abstimed_wait and the result of a deadlocked thread that triggers the service failure and restart.

Orchestrator Resolved Issues

Resolved in Orchestrator Version R51010-20231215-GA

Orchestrator build R51010-20231215-GA was released on 12-18-2023 and is the 10th Orchestrator rollup for Release 5.1.0.

This Orchestrator rollup build addresses the below important issue since the 9th Orchestrator rollup build, R5109-20231003-GA.

  • Fixed Issue 117941: The VLAN Advertise checkbox always displays as unchecked on the Orchestrator UI.

    Even when a user selects the VLAN Advertise checkbox and Saves Changes, the VLAN Advertise checkbox on the Orchestrator's UI reverts to being unchecked.

  • Fixed Issue 125006: For a VMware SASE Orchestrator configured with a Disaster Recovery (DR) topology, the Orchestrator's database may fail and this leads to the Standby Orchestrator going into an error state and in rare cases this can lead to Edges and Gateways showing as offline on the Orchestrator UI and triggering events and alerts.

    The database state is expected to autorecover within a few minutes and the DR Orchestrators would resynchronize. However, sometimes this state can exceed the tolerance period, and both Edges and Gateways would begin to send their heartbeats to the Standby Orchestrator instead of the Active Orchestrator. As a result, the Active Orchestrator marks both Edges and Gateways not heart beating to it as down and triggers events and alerts. This issue is purely on the management side and does not affect network traffic.

    On an Orchestrator without a fix, the way to avoid this issue is to increase the Orchestrator System Property that governs tolerance for a failed synchronization: vco.disasterRecovery.transientErrorToleranceSecs by a suitable amount to provide a longer autorecovery window.

  • Fixed Issue 128310: VMware SASE Orchestrator users may experience overall slowness and some API failures due to issues with the Orchestrator's database service. Other side effects include SD-WAN Gateways/Edges appearing offline in the UI, configuration changes made through the Orchestrator UI not being pushed to the target SD-WAN Edges, and a loss of reporting capabilities.

    These issues are all caused by the Orchestrator's database service failing with the error: too many open files. This error can be observed by an Operator user on the VMware SASE Orchestrator via logs. An enterprise or partner user accessing VMware SASE Orchestrator via the UI would experience slowness and intermittent API failures, causing error messages on the UI.

  • Fixed Issue 129239: On the Configure > Profile page of the Orchestrator UI, when a user edits any setting at the Profile level and attempts to Save, they may observe that the API call times out and is not successful.

    When this issue is encountered, the Orchestrator UI displays a red error banner on the browser page that reads "configuration/updateConfigurationModule time out". The issue is the result of other Orchestrator database queries taking too long to complete and causing the attempt to save the new Profile settings to time out.

  • Fixed Issue 131789: When configuring Single Sign On (SSO) for an organization, even though the role information is present in the Identity Provider (IdP) response, the users cannot login to the Orchestrator.

    The Orchestrator cannot match the role of a user logging in via Single Sign On (SSO), if the IdP sends role information in a nested JSON structure. Starting in version 5.0.1.7, the Orchestrator can reference and match the role of a SSO user even if it is present in a nested JSON structure.If experiencing this issue on an Orchestrator without a fix for this issue, the workaround is to configure the IdP to send the role detail in the immediate level, and not in the nested structure.

Resolved in Orchestrator Version R5109-20231003-GA

Orchestrator build R5109-20231003-GA was released on 10-05-2023 and is the 9th Orchestrator rollup for Release 5.1.0.

This Orchestrator rollup build addresses the below important issue since the 8th Orchestrator rollup build, R5108-20230916-GA.

  • Fixed Issue 119938: For a customer who uses automation for Zscalar tunnels, it may take a long time to create an automated IPsec tunnel from a VMware SD-WAN Edge to Zscaler.

    When customers configure Zscaler sub-locations at Edge > Device Settings, it can take a long time for these configurations to sync with the Zscaler cloud. This is because the Zscaler cloud needs to update its records for each sub-location, which can be a time-consuming process.

    This issue is caused by the automation framework on the Orchestrator, which enqueues IPsec tunnel creation and sub-location creation actions in the automation queue. It also enqueues update actions in the automation queue when the Edge WAN IP changes. However, the wait time for items in the queue increases due to the large number of update actions. In some customer deployment environments, the Edge WAN IP can change up to 4000 times in one day (for example, mobile WAN links).

  • Fixed Issue 128310: VMware SASE Orchestrator users may experience overall slowness and some API failures due to issues with the Orchestrator's database service. Other side effects include SD-WAN Gateways/Edges appearing offline in the UI, configuration changes made through the Orchestrator UI not being pushed to the target SD-WAN Edges, and a loss of reporting capabilities.

    The issues are all caused by the Orchestrator's database service failing with the error: too many open files. This error can be observed by an Operator user on the VMware SASE Orchestrator via logs. An enterprise or partner user accessing VMware SASE Orchestrator via the UI would experience slowness and intermittent API failures, causing error messages on the UI.

Resolved in Orchestrator Version R5108-20230916-GA

Orchestrator build R5108-20230916-GA was released on 09-20-2023 and is the 8th Orchestrator rollup for Release 5.1.0.

This Orchestrator rollup build addresses the below important issue since the 7th Orchestrator rollup build, R5107-20230722-GA.

  • Fixed Issue 94610: When a user initiates a forced HA Failover through Remote Actions > Force HA Failover, the VMware SASE Orchestrator may not generate and send an Alert for the HA failover.

    Since the HA failover is forced by the Orchestrator, both the Active and Standby Edge anticipate the failover, and this can cause both the HA_GOING_ACTIVE and HA_READY messages to be sent in the same heartbeat by the HA Edge. If the ‘HA State’ sent in the heartbeat shows as ‘Ready,’ this fools the Orchestrator into not generating an Alert for the failover because it only sees this 'Ready' message and does not see the 'Going Active' one.

  • Fixed Issue 104775: When a user configures a previously active WAN link on a VMware SD-WAN Edge to be a backup, the VMware SASE Orchestrator UI does not display the status correctly on Monitor > Edges > Overview.

    The status should display as Standby Idle with a gray colored status dot, but instead does not display the link type or backup status. This is a cosmetic defect as the WAN link is performing its role as a backup.

  • Fixed Issue 105580: For a VMware SASE Orchestrator where FIPS mode is configured, an attempt to set up Disaster Recovery (DR) for the Orchestrator may fail.

    SSL_CTX_new failed when trying to connect. The DR setup with FIPS configured includes an Orchestrator build with MySQL version 8.0.28 or higher and would fail during the DR COPYING_DP phase with error message: SSL_CTX_new failed when trying to connect.

  • Fixed Issue 106191: A user cannot make changes to a VMware SD-WAN Edge configuration if the Edge is using a profile where any interface is configured with a static IP address.

    If an Edge configuration change is attempted, the VMware SASE Orchestrator UI will display the error message: 'invalid probe interval for interface' and prevent saving the changes.

    On an Orchestrator without a fix for this issue, the only workaround is to create a new profile without static interface assignments and apply that to the Edge.

  • Fixed Issue 115981: For a customer enterprise using VMware SASE Orchestrator's APIv2, when running the API for getting Enterprise Events, the Orchestrator returns only a limited set.

    The specific call is https://\{api_host}//api/sdwan/v2/enterprises/{enterpriseLogicalId}/events. When invoked it returns only the top level hierarchy and does include details like the enterpriseName, EdgeName, segmentName, or edgeID. In addition, APIv2 does not support graph traversal.

    On an Orchestrator without a fix for this issue the only workaround is to use APIv1, which forces a customer to maintain two sets of API families. In addition APIv1 does not support Cloud Web Security.

  • Fixed Issue 116531: If a user attempts to generate a report on the VMware SASE Orchestrator where at least one Edge description includes a comma (,) the report may not format properly.

    When an Edge description includes a comma (as shown in the screenshot below) the Orchestrator's report service confuses this and breaks out the text after each comma into the report's next column, versus containing the entire text string in the Edge Description column as expected.

    So instead of Bytes Transmitted having the values associated with it, it shows the text after the first comma, Bytes Received would have the text after the second comma (if there was one), and so forth. The report would still include the data for Bytes Transmitted and Bytes Received, it would just be pushed farther to the right and not be aligned to the correct columns.

    On an Orchestrator without a fix for this issue, the user would need to ensure the Edge description used no commas.

  • Fixed Issue 117822: When a customer looks at Monitor > Edges > QoE, they may observe gaps in the QoE graphs that are not explained by any issues with the Edge's WAN links.

    The gaps are the result of an issue with the Orchestrator database where the internal job queue for link data is missed and the link data is not backfilled.

  • Fixed Issue 118728: On a partner portal or a customer enterprise, some users may not be allowed to login to the VMware SASE Orchestrator.

    The user may see the error 'user does not have privilege [READ:PROXY] required to access [enterpriseProxy/getEnterpriseProxy]' even though the user has the correct privileges to log in. This is true of native authentication and two factor authentication. This error actually reflects an expired password even though the Orchestrator is not letting the user know that is the real reason they cannot login and the user cannot reset their password since they cannot login.

    On an Orchestrator without a fix for this issue, a partner or customer administrator with a suitable role can send the password reset email to an affected user to reset their password.

  • Fixed Issue 121085: If a Partner Administrator is on the Global Settings > User Management > Service Permissions page, the option "Reset to System Default" does not work as expected.

    The expected behavior when Reset to System Default is selected is for all custom role permission packages to revert to an Unpublished state as the default settings for user role permissions are restored. However, the Partner user would observe that one or more custom packages remain in a Published state and those that remain Published cannot be modified or deleted.

  • Fixed Issue 121441: When a customer activates VNF Insertion on a VMware SD-WAN Edge's VLAN, the Edge deletes and then redeploys the VNF, which makes it unusable for that Edge.

    After VNF insertion is activated on the VLAN, the customer would observe VNF deletion and redeployment events on the Monitor > Events page and also receive an email with the message "(VNF_VM_DELETED) / Unknown vendor VNF was deleted on the edge <Edge Name>". The issue is traced to the Orchestrator sending a new UUID (Universally Unique IDentifier) after VNF insertion is activated on a VLAN. The Edge interprets this UUID change as a trigger to delete the VNF and redeploy it, which makes it unusable for that Edge.

    This issue occurs on the New UI only. If experiencing this issue on a 5.1.x Orchestrator without a fix for this issue, switch to the Classic UI to configure the VNF.

  • Fixed Issue 121469: When a user navigates to the Global Settings > User Management page, they may observe that all user accounts show as locked according to a banner on the UI, even though most or possibly all of the accounts are not actually locked.

    The error message banner for any user account would read 'This account has been locked due to too many failed login attempts', even though when looking at the user list page their status shows as Unlocked, and Local UI login remains possible.

  • Fixed Issue 124778: When using the New UI on the VMware SASE Orchestrator, if a customer navigates to Global Settings > Customer Configuration, they do not see the option to configure their Security Policy.

    Security Policy section is where a customer can configure their Edge IPsec Proposal including Encryption, DH Group, and so forth. This option is present on the Classic UI but missing on the New UI.

Resolved in Orchestrator Version R5107-20230722-GA

Orchestrator build R5107-20230722-GA was released on 07-22-2023 and is the 7th Orchestrator rollup for Release 5.1.0.

This Orchestrator rollup build addresses the below important issue since the 6th Orchestrator rollup build, R5106-20230705-GA.

  • Fixed Issue 122271: When a customer adds additional LAN-side NAT rules to a Profile using the VMware SASE Orchestrator's New UI, they may observe that all traffic matching these rules fails.

    The New UI incorrectly calculates the LAN-side NAT outside mask from the inside address prefix. When the rules are not written such that the inside and outside prefix are the same (in other words: 1:1) the behavior of the rules changes and can become nonfunctional if a user modifies any LAN Side NAT rule from the New UI.

    On an Orchestrator without a fix for this issue, the customer should only use the Orchestrator's Classic UI for configuring LAN-side NAT rules.

Resolved in Orchestrator Version R5106-20230705-GA

Orchestrator build R5106-20230705-GA was released on 07-06-2023 and is the 6th Orchestrator rollup for Release 5.1.0.

This Orchestrator rollup build addresses the below important issues since the 5th Orchestrator rollup build, R5105-20230611-GA.

  • Fixed Issue 84772: When a VLAN's IPv6 DHCP Server is configured as Relay type at the Profile level, if a user then changes the type to Activated at the Edge level using Edge Override, and then later toggles off the Edge Override, the Edge will continue to use the Activated type versus reverting to the Relay type provided by the Profile.

    The VMware SASE Orchestrator is failing to impose the Profile configuration once Edge Override has been toggled off, resulting in the wrong IPv6 Server settings for the affected Edge.

  • Fixed Issue 115411: For a VMware SASE Orchestrator using version 5.1.0 or later and deployed with a Disaster Recovery topology, the synchronization may fail due to a database issue.

    The specific process that fails is dr_utils.js and is a result of this process being deprecated on the latest version of the Orchestrator's database software found in Release 5.1.0 and later.

  • Fixed Issue 115433: A user with the role "Enterprise Support" cannot see DHCP configuration details when looking at the VMware SASE Orchestrator's New UI.

    The same Enterprise Support user can see the DHCP configuration details if they use the Classic UI.

  • Fixed Issue 116633: When logging into the VMware SASE Orchestrator using either Safari or Firefox, if a user configures an invalid VLAN at the profile level (for example, ""), and then configures a valid VLAN value at the VMware SD-WAN Edge, the call still goes through but shows an error.

    The interface value where no value is set should be null but is instead vlanID = "". If a user logs into the Orchestrator with a Chromium based browser, this issue is not observed.

  • Fixed Issue 117772: When looking at Monitor > Network Overview screen for a VMware SASE Orchestrator for an Enterprise Customer, WAN links that have a status of Degraded or Down may not be included in the Link status screen if more than 10 WAN links are being monitored.

    This issue is exclusive to the New Orchestrator UI due to a frontend issue that does not account for Degraded or Down WAN links. Monitoring works as expected on the Classic UI.

  • Fixed Issue 117988: The "Inbound Route Learning" with the "Exact Match" checkbox configured for OSPF under a VMware SD-WAN Interface does not match what is configured on the Edge when comparing the values on the Classic UI and the New UI of the VMware SASE Orchestrator.

    An Exact Match option does not display the correct value even if though it is correctly stored in the Edge's database when looking at the New UI. The correct values are represented on the Classic UI.

  • Fixed Issue 117993: When a Partner User managing customer enterprises which use native authentication (in other words, username/password) or an Enterprise User attempts to reset a password for an Enterprise User, the attempt fails.

    The user would observe the error: user does not have privileges required to access [enterpriseUser/sendEnterpriseUserPasswordResetEMail]. This issue is only experienced on the New UI and is the result of missing request parameters. If encountering this issue, the user can switch to the Classic UI and the password reset works as expected.

  • Fixed Issue 118074: A user may not be able to open some device settings on the Configure > Edge > Device page of the VMware SASE Orchestrator's New UI.

    Settings that may not be accessible include Interfaces, IPv6, Cloud VPN, Non SD-WAN Destination (NSD), and Cloud Security Service (CSS). The issue is traced to the WAN Settings requiring a Public IP Address and if this address is absent, an error is thrown on the New UI and blocks access to those settings. Device settings works as expected on the Classic UI.

  • Fixed Issue 118544: A user may observe that an Operator Profile does not load and is inaccessible and thus cannot be assigned to a customer enterprise.

    There is an issue with the Orchestrator database where the Operator Profile is present but an incorrect logical ID is added to a configuration module if a customer enterprise is deleted, and this prevents it from loading.

  • Fixed Issue 118733: When using the VMware SASE Orchestrator's New UI and either a Business Policy or Firewall Rule is configured at the profile level and then overriden at the Edge level, when looking at the Configure > Edges > List Edges screen, the icons for the overridden Edge are not correctly represented for Device, Business, and Firewall.

    The icons where Edge Override is checked should show as a solid color and are instead shown as empty. The Classic Orchestrator correctly displays the icons as solid when an Edge Override has been configured for a particular category.

  • Fixed Issue 119733: The VMware SASE Orchestrator's database may experience a failure and stop working.

    The issue is the result of the database not being on the latest stable version of MySQL and Orchestrator Release 5.1.0.6 corrects that deficiency with an updated MySQL version.

  • Fixed Issue 120606: When attempting to create a new role on the Cutomer > Global Settings > User Management > New Role, the would observe an error and privileges are not loaded.

    When encountering this issue the user sees "method error" on the UI page while creating a new role and which also prevents privileges from being loaded.

     

Resolved in Orchestrator Version R5105-20230611-GA

Orchestrator build R5105-20230611-GA was released on 06-13-2023 and is the 5th Orchestrator rollup for Release 5.1.0.

This Orchestrator rollup build addresses the below important issues since the 4th Orchestrator rollup build, R5104-20230426-GA.

  • Issues occurring only on the New Orchestrator UI, and not on the Classic Orchestrator UI

    Orchestrator Release 5.1.0.5 includes a large number of fixes related to issues a customer can experience only when using the New UI and which are not experienced when using the Classic UI. The below table lists all such fixes with descriptions of the symptoms for each issue.

    New UI-Only Resolved Issues

    Ticket

    Symptom/Description

    Fixed Issue #87089

    User has no option to edit the client address on the Monitor > Edges > Sources page. The fix adds a blue pencil icon that can be clicked to edit an entry.

    Fixed Issue #112044

    For a site where a Non SD-WAN Destination via Edge of the Zscaler type is deployed, when looking at Monitor > Network Services, the New UI does not display the IaaS Subscriptions column under the Deployment Status page.

    Fixed Issue #112451

    When a user is editing a configuration profile and under Device > Interfaces > Your Edge Models the user selects the Edge 3810, the web browser page stops responding and cannot be saved, and the user must refresh the page to recover it. The customer cannot use the Edge 3810 for that profile.

    Fixed Issue #112500

    For a site where a Cloud Security Service (CSS) with Zscaler type is deployed, when looking at the Monitor > Edges and hovering over the Edge tunnels for an Edge, the UI does not show the Zscaler CSS City and State.

    Fixed Issue #112809

    For a Customer Administrator with an Enterprise Read Only role, they can view Monitoring > Routing page on the UI.

    Fixed Issue #112906

    A user may observe that UI pages load with odd formatting where the page has large sections of white space.

    Fixed Issue #112912

    For a Customer Administrator with an Enterprise Support role, they do not have the option to select Remote Actions > Reboot for an Edge.

    Fixed Issue #113254

    A Partner Administrator with a Superuser or Standard role cannot change the default operator profile for a customer under their management.

    Fixed Issue #113366

    A user cannot configure a static route where a VLAN secondary IP is the next hop.

    Fixed Issue #113963

    If a user creates a Firewall rule and sets an Application for that rule, and then later edits the same rule and changes the Application, the change is not applied and the rule retains the previous Application.

    Fixed Issue #114291

    If a Cloud Security Service (CSS) is configured on a profile, the user cannot switch between segments with no opportunity to save Device Settings after a CSS is changed on several different segments.

    Fixed Issue #114564

    User cannot configure the 802.1P Setting under the optional configuration for an Edge Interface.

    Fixed Issue #114602

    User has no option to configure Operator Alerts or Enterprise Alerts under Service Settings > Alerts & Notifications.

    Fixed Issue #114912

    The Partner Overview page does not include settings for the User Agreement.

    Fixed Issue #115307

    When a user session remains idle long enough to trigger a timeout, the UI does not display a logout message and instead goes into sleep mode and when the user accesses the UI, the Orchestrator "wakes up" and allows the user to access the Orchestrator instead of redirecting them to the login page.

    Fixed Issue #115439

    When a user navigates to Configure > Profile > Device Settings > BGP they would observe BGP as present, but if they sort by Segment Aware, the option to configure BGP is now missing.

    Fixed Issue #115653

    When a Partner Administrator is viewing their Manage Customers page, the UI does not communicate whether the Customer is using Gateways that are in a Quiesced service state. This prevents the Partner from taking timely action to ensure the Customer is using only active Gateways.

    Fixed Issue #115719

    A Backhaul rule disappears from the Configure > Business Policy page if any change is made to the Device Settings at a Profile level.

    Fixed Issue #116523

    When a user is configuring a VNF insertion configuration and is attempting to configure VNF High Availability in the Edge, the Orchestrator does not save the VNF change once HA configured.

    Fixed Issue #117527

    When a user is configuring BGP at the Profile level, the browser may become unresponsive if a large number of rules are configured.

  • Fixed Issue 105861: When a WAN link goes down for several minutes and comes back, the Monitor > QoE graph does not reflect the actual link state.

     The QoE should show red while the link is down and then resume the color (green if good quality) when the link is restored, and this is not happening and causes confusion for users. The issue is caused by the VMware SASE Orchestrator's databased not recording the link down event correctly.

  • Fixed Issue 106295: For a Non SD-WAN Destination with an AWS type, when primary and secondary Gateways are setup with BGP configured on each, the VMware SASE Orchestrator may show the redundant secondary tunnels as down even though they are up.

    The AWS side will report both primary and secondary tunnels as up, but on the Orchestrator's Monitor > Network Services page the secondary shows down. The is strictly a cosmetic issue.

  • Fixed Issue 107180: When a user is configuring a VNF with a Fortinet type, they cannot see Fortinet Image Version 6.49 or 7.2.0 in the dropdown menu.

    These images are supported in Release 5.1.0 but are not accessible by a customer.

  • Fixed Issue 107766: When a customer configures either a Non SD-WAN Destination via Edge or a Cloud Security Service (CSS) and also configures the Level 7 (L7) Health Check option, the customer may observe that the tunnels are unexpectedly marked as down or up compared with what should occur based on their L7 Health Check configuration value.

    The issue is that the Orchestrator pushes to the VMware SD-WAN Edge the default L7 Health Check parameters regardless of what the customer has configured. As a result, even if the conditions of the tunnel match what the customer configured the tunnel status can remain unchanged because it is adhering to the default L7 Health Check value.

  • Fixed Issue 110826: When a VMware SD-WAN Edge is assigned to a customer enterprise, the VMware SASE Orchestrator does not automatically move the Edge to the "Assigned Inventory" tab on the Maestro inventory management application.

    When unassigned Edges are requested from Maestro and the Orchestrator looks for already assigned serial numbers, the Edge model number is not compared causing issues with Edges being assigned twice.

  • Fixed Issue 111957: After an Operator upgrades a VMware SASE Orchestrator, users may observe errors related to failed long running schema updates. For example, newly learned routes (BGP or OSPF) may be missing from the OFC page. Errors will also be observed in the upload logs on an Orchestrator related to learned routes.

    A foreign key on VELOCLOUD_LEARNED_PARTNER_ROUTE_ASSOC is dropped and re-added on an Orchestrator upgrade from Release 4.2.x to Release 4.3.x or later as a long running schema update. This foreign key does not already exist on Orchestrators which had their upgrade path through some a 2.1.x build with an upgrade defect AND a Gateway that was learning BGP routes was deleted from the Orchestrator. In such Orchestrators, the foreign key addition on 4.2.x > 4.3.x+ upgrade fails if there is a foreign key violating records already present in the table.

    The fix for this issue corrects the root cause by deleting the foreign key that is violating the records and then re-adding the foreign key.

    If upgrading to an Orchestrator without a fix for this issue, the workaround is to run the following query in VeloCloud schema of MySQL: DELETE FROM VELOCLOUD_LEARNED_PARTNER_ROUTE_ASSOC WHERE gatewayId not in (select id from VELOCLOUD_GATEWAY) and then retriggering the long running schema updates.

  • Fixed Issue 112333: A VMware SASE Orchestrator handling ~4K VMware SD-WAN Edges with ~6K tunnels and continuous traffic flow may become unstable over a period of ~4 days and start to randomly log users out.

    The stress causes the Orchestrator's database to fail and trigger the error "connect ECONNREFUSED" followed by the Orchestrator's IP address. This issue has only been observed in a scale test environment and not in a field deployment.

  • Fixed Issue 112605: A customer may observe that when attempting to assign a Hub Cluster through the Configure > Device > Cloud VPN > Edge to SD-WAN Sites, the profile is unresponsive, and the configuration cannot be saved.

    The Orchestrator creates duplicate configuration associations where there are multiple backhaul Business Policies, and the duplicate references trigger the failure to configure and assign Hub Clusters.

  • Fixed Issue 112992: When a customer enterprise is migrated from one VMware SASE Orchestrator to a different one, the Orchestrator adds a UUID records.

    The UUID record is added to the VELOCLOUD_ENTERPRISE_OBJECT records when it should not be.

  • Fixed Issue 113209: A user cannot delete a VMware SD-WAN Edge from the SASE Orchestrator if the Edge is in a 'Degraded' state.

    The Orchestrator throws an error message that reads "Degraded edges and connected edges cannot be deleted". While a user should not be able to delete a connected Edge, they should be able to delete a degraded Edge. 

  • Fixed Issue 113375: For a VMware SASE Orchestrator deployed with a Disaster Recovery (DR) topology, when the Orchestrator is upgraded from 4.5.x to 5.1.x the upgrade may fail.

    The upgrade script and the iptables both fail with return code 1. The issue can be triggered during the period the Active and Standby Orchestrators are running different software versions and the iptables are not properly loaded when the Standby Orchestrator attempts to upgrade (which is expected and temporary), and the error fails the entire upgrade. The fix simply turns the iptables error into a warning and allows the upgrade to continue.

  • Fixed Issue 114240: Under Edge IPsec Proposal, when a user changes the Encryption to either AES-128-GCM or AES-256-GCM and saves the configuration, the Orchestrator UI throws an error.

    The error reads: "instance.ipsec.encryption is not one of enum values: AES_128_CBC, AES_256_CBC, DES_CBC". The issue is the result of the two GCM type encryptions being mistakenly added to the Encryption menu when they are not valid options. These options are removed on the updated Orchestrator.

  • Fixed Issue 115624: A VMware SASE Orchestrator may experience high CPU utilization and instability along with slow loading Device Settings and Network Settings UI pages when the portal service is restarted.

    The issue was found on an Orchestrator with ~2K connected Edges while also using Cloud Security Services (CSS) and could occur with this number of connected Edges or greater. The issue is caused by several Orchestrator processes related to Edge, Profile, and Network configurations being uploaded from the Edges to the Orchestrator and taking much longer to complete than expected (~60 seconds or more).

  • Fixed Issue 116141: When a user makes changes to the Device Settings on a configuration profile, the VMware SASE Orchestrator takes an excessive amount of time to validate the changes.

    Each change can take up to a minute to validate and apply when it should take just seconds. The issue is traced to an Orchestrator process that is not only retrieving a count of all Edge configuration records associated with that profile, but is also decoding and parsing each record ,which is both unnecessary and taxing on the Orchestrator CPU. The fix ensures the process only retrieves the record count.

  • Fixed Issue 116770: When an Operator user creates an External Certificate Authority (CA) with a chain length of 2 or more in root of trust, the VMware SASE Orchestrator does not allow a pathLengthConstraint value of 0 in a subordinate.

    A subordinate not being permitted to sign a subordinate below it is a valid configuration and should be allowed by the Orchestrator but a process blocks the configuration from being saved.

  • Fixed Issue 116790: When a VMware SASE Orchestrator is upgraded to Release 5.1.x or later, Customer VMware SD-WAN Edges may be inadvertently downgraded to an older Edge version than the Edge is configured to use.

    The issue is triggered when a customer enterprise is deleted from the Orchestrator where the deleted enterprise was associated with an Operator Profile by its logical ID in the Orchestrator database. When the enterprise is deleted, the Operator Profile is also deleted. Customers who have Edge Image Management configured and multiple Operator Profiles available and who had this deleted Operator Profile assigned as their default are then assigned an Operator Profile that remains available in their Edge Image Management menu. As a result the customer could be assigned an Operator Profile containing a much older Edge Release, and the Edges assigned this Operator Profile can have their software changed and possibly downgraded with potentially disruptive effects including network failure if the Edge is now running an older release that does not support features the customer is using.

  • Fixed Issue 116976: When a Partner Administrator logs into a VMware SASE Orchestrator, the Partner may observe that the Orchestrator is not displaying the correct status WAN links that are down for greater than 24 hours on the VMware SD-WAN Edges they manage.

    The Orchestrator API only returns recent links for Partners/MSPs while regular customer enterprise users get the correct link status information, which impacts a Partner's ability to properly support their customers on link issues.

  • Fixed Issue 117800: When a VMware SASE Orchestrator is upgraded from Release 4.x to 5.1.x or later, the Operator may observe that after the backend process restarts, the same upgradeSchema.sql file is created, even though it was successfully executed.

    This issue occurs with the post-upgrade schema update and once the post-schema script is executed, the upgradeSchema.sql file should not be generated again.

  • Fixed Issue 118071: User attempt to change the VPN settings in Configure > Profile > Device may fail with an error if there are multiple VMware SD-WAN Gateways assigned to the customer all of which have ~2K+ Edges assigned.

    The Orchestrator error reads "Error validating changes". The VMware SASE Orchestrator is failing to update the VPN settings because the API is waiting for a databased query which returns an excess of 10K rows, something that can happen with an Orchestrator operating at a large scale. The issue is caused by the Orchestrator getting all Gateways regardless of type (Primary, and Secondary) when it should only have the Edge report its Primary Gateway. This vastly increases the number of records returned and can overflow a query at scale.

Resolved in Orchestrator Version R5104-20230426-GA

Orchestrator build R5104-20230426-GA was released on 04-27-2023 and is the 4th Orchestrator rollup for Release 5.1.0.

This Orchestrator rollup build addresses the below important issues since the 3rd Orchestrator rollup build, R5103-20230315-GA.

  • Issues occurring only on the New Orchestrator UI, and not on the Classic Orchestrator UI

    Orchestrator Release 5.1.0.4 includes a large number of fixes related to issues a customer may experience only when using the New UI and which are not experienced when using the Classic UI. The below table lists all such fixes with descriptions of the symptoms for each issue.

    New UI-Only Resolved Issues

    Ticket

    Symptom/Description

    Fixed Issue #104785

    When a user is on the Monitor > Events page and clicks the Events refresh button, the Events table timestamp is not updated and the most recent events will not appear in the table unless the web browser page is reloaded.

    Fixed Issue #106327

    When a user is on the Monitor > Events page and is configuring a filter for Events, some filter options are missing and limit how Edge Events may be filtered.

    Fixed Issue #107071

    When a user is on the Monitor > Events page and selects a large enough time window that > 5M Events are returned, the page times out and fails to load.

    Fixed Issue #107980

    When a user is on the Configure > Edge > Overview page, they cannot change the name of an Edge unless they also fill out Contact & Location data if the local contact data is blank.

    Fixed Issue #108072

    When a user attempts to configure a loopback interface with the same IP address on various segments, the Orchestrator prevents this and displays a warning message of "Must be unique".

    Fixed Issue #109284

    When a user adds a tunnel for either an Azure or Zscaler type Non SD-WAN Destination (NSD) via Edge, the fields Local Identification Type, Local Identification, and PSK can be edited.

    Fixed Issue #109300

    When a user looks at any area of the Orchestrator where a license is to be configured, the user can only see the license selected and no other licenses. The Orchestrator filters out all other license options when one is selected and the fix for this issue includes a new Clear button to the right of the license drop-down menu to allow the selection of a different license option.

    Fixed Issue #109532

    When looking at the Configure > Network Services page, the DNS IP addresses differ on the New UI versus the Classic UI because the New UI is showing different fields.

    Fixed Issue #109533

    On the Configure > Edge > Device > Connectivity > VLAN page, DHCP server shows as Enabled when a more accurate status of Relay should be shown.

    Fixed Issue #109715

    A down WAN link disappears from the Monitor > Network Overview page after 24 hours even though the Edge Down Link Limit is set to a value > 24 hours.

    Fixed Issue #109788

    When configuring any Non SD-WAN Destination, a user cannot configure a Site Subnet with a 0.0.0.0/0 value and the Orchestrator declares it an "Invalid CIDR Prefix".

    Fixed Issue #109836

    The Partner Gateway static routes are not distributed to the Edges when configuring these routes on the New UI. These routes should be distributed to the connected Edges as Cloud Routes (Type = Cloud) and shown in a route table dump, but are not.

    Fixed Issue #109911

    When configuring a Business Policy, in the Link Steering > Interfaces section if the WAN overlay type is 'Disabled', that interface cannot be selected. The New UI allows interfaces which are related to WAN overlay that are auto discovered or user defined only.

    Fixed Issue #110094

    When editing an Edge interface and adding a lot of filters at once to OSPF, the filters grid disappears. When enough OSPF filters are added such that pagination appears in the OSPF filters grid, the grid height is set to 0 and a user can no longer edit or see the filter grid.

    Fixed Issue #110330

    Both Partner and Customer administrators may not be able to see the Global Settings > Service Permissions tab even though their role permits them to do so. Service Permissions is where user role customization is documented.

    Fixed Issue #111104

    When looking at the Service Settings > Edge Management > Software & Firmware Images page, if a user expands a row at the bottom of the table they will be unable to see all the details.

    Fixed Issue #111407

    When a user is on the Configure > Edge > Device > Connectivity > Interfaces page, they are not permitted to configure a user-defined interface without also adding a WAN link.

    Fixed Issue #111444

    When configuring a Cloud Security Service (CSS) with a Zscaler type, the user is not permitted to configure the Zscaler Login URL with a URL and gets the message FQDN is invalid, nor is the login URL value is sent to a server and saved.

    Fixed Issue #111934

    When updating any field of a Non SD-WAN Destination via Edge with Zscaler type, the Orchestrator removes the DH Group field.

    Fixed Issue #111944

    An Operator Superuser cannot modify or delete a System Property.

    Fixed Issue #112094

    When a user changes the credentials for a Cloud Security Service (CSS) on a Non-Global Segment and in the same session switches to a Global Segment, the CSS credentials are removed.

    Fixed Issue #112201

    If a user configures a Cloud Security Service (CSS) through an API and sets the CSS to None (Empty), under the VMware Edge's CSS configuration the Orchestrator will not show a configuration, but the Edge's database and API responses have the CSS up and working. A CSS in this state cannot be used as part of a Business Policy.

    Fixed Issue #112224

    When a customer administrator with a Superuser, Standard, or Support role navigates to Diagnostics, the Diagnostic Bundles page link is not present even though their role permits access to this section. As a result a Customer Administrator cannot generate (but not download) a diagnostic bundle and cannot generate and download a packet capture.

    Fixed Issue #112437

    If a user opens multiple Remote Diagnostics sessions for the same Edge, the page may fail to load after enough sessions are opened.

  • Fixed Issue 95631: When a user navigates to Monitor > Network Services > Cloud Security Service Sites and tries to sort the Events by the State Change Time, the sorting sequence is incorrect.

    When a user tries to sort by the State Change Time column of the CSS table, the sorting is wrong because dates are not sorted by their timestamps but by the parsed date. This issue is seen on both the Classic UI and the New UI.

  • Fixed Issue 106907: When a customer has configured a Non SD-WAN Destination (NSD) via Edge which is associated with a Business Policy, if a user later uses Edge Override to deactivate the NSD via Edge, the Business Policy will not be removed on a VMware SASE Orchestrator using Release 5.1.0.

    The Orchestrator should remove the Business Policy once the NSD via Edge is deactivated because if the rule persists, traffic that matches that Business Policy will be steered to a non-existent destination resulting in that traffic being dropped.

  • Fixed Issue 106929: An Operator may be unable to assign a Software Version to an Operator Profile on the VMware SASE Orchestrator using Release 5.1.0.

    The Orchestrator throws an error similar to the following:

    Error message.

    The issue is the result of the Orchestrator timing out the attempt to add the Software Image because of competing database API queries. This issue is more likely to occur on an Orchestrator hosting a large number of Edges as is the case on VMware Hosted Orchestrators and can occur whether using the New UI or the Classic UI.

    On an Orchestrator without a fix for this issue the only workaround is to increase the timeout value for database connections.

  • Fixed Issue 107349: When looking at Monitor > Edge > Sources on the VMware SASE Orchestrator, some or even all sources may not resolve and show 'Unable to resolve'.

    This issue is inconsistent and some sites in a customer enterprise may display the IPs and MAC addresses for sources as expected. The issue is caused by an Orchestrator database table for Edge names not being updated.

    If a customer is experiencing this issue on an Orchestrator without a fix for this issue they can turn off Cloud VPN in a maintenance window and then turn it on again to force the Orchestrator to retreive the correct Edge names.

  • Fixed Issue 108363: After a VMware SASE Orchestrator is upgraded to a 5.x Release, VMware SD-WAN Edges which deploy Cloud Security Services (CSS) like Zscaler and have a Level 7 (L7) Health Check also configured may experience a loss of traffic using that CSS for about 30 seconds.

    After the Orchestrator is upgraded, it triggers a configuration update to all Edges and this can cause some CSS sites with L7 Health Check configured to go down until the configuration is corrected.The issue is linked to Fixed Issue #107302 which addresses the issue on the Edge. The fix here prevents the Orchestrator from triggering configuration updates to Edges on an upgrade and thus protects Edges which do not have the fix for #107302.

  • Fixed Issue 110946: A VMware SD-WAN Orchestrator using release 4.2.x or earlier may fail when upgraded to become a SASE Orchestrator using release 4.3.x or later.

    A 4.2.x or earlier Orchestrator does not clean up the apt cache prior to running the apt update service routine when the Orchestrator is upgraded to 4.3.x or later, and as a result the MySQL database restarts during the upgrade and the upgrade fails.

    If upgrading to an Orchestrator version without a fix for this issue, the Operator can run the command rm -rf /var/lib/apt/lists/ prior to the upgrade.

  • Fixed Issue 111665: When updating A VMware SASE Orchestrator from an earlier Orchestrator release to Release 5.1.0, the client device migration may not complete.

    The issue impacts the migration of client device data from an Orchestrator using MySQL to one using ClickHouse. After a certain number of records have been migrated, the migration process ends prematurely.

  • Fixed Issue 111946: A user cannot see the paths on the Edge > Monitor > Paths tab on a VMware SASE Orchestrator when the peer list is greater than 100.

    When a user navigates to the Edge > Monitor > Paths tab, the Orchestrator's backend returns all records even if there are more than 100. This is because the backend omits the limit constraint, which is the maximum number of records that should be returned. The records that are returned after the limit count are unnormalized, meaning that they are not formatted in a way that is compatible with the UI. This causes an error in the UI. The Orchestrator should only return the records that are within the submitted limit.

  • Fixed Issue 112458: When a new VMware SD-WAN Gateway is added to a Gateway pool and a Gateway Rebalance is initiated, the new Gateway is not redistributed to VMware SD-WAN Edges even though the existing Gateways are overloaded.

    The VMware SASE Orchestrator should reassign multiple Edges to the least loaded Gateway in the same geographic region when a Gateway Rebalance is performed, and with this issue the Edges retain their existing Gateway assignment despite that assigned Gateway being overloaded.

  • Fixed Issue 112885: A Gateway Rebalance may be triggered for workflows not directly related to triggering one through the VMware SASE Orchestrator UI.

    The fix here addresses the issue that Gateway rebalances may occur for workflows outside of what is intended for a dedicated enterprise. A Gateway Rebalance should only be triggered through the Orchestrator UI at the Operator level and through no other means.

  • Fixed Issue 114475: When an Operator attempts to upgrae a VMware SASE Orchestrator from Release 4.2.0 to 5.1.0, the Orchestrator may report that the upgrade failed.

    In the logs the Operator would observe this entry: Error while initializing CWS Server service Error: Too many connections. This issue is triggered by MySQL restarting before vco-db-schema is installed which is caused by MySQL not setting the maximum number of connections. In addition, while the Orchestrator reports that the installation failed, in reality the installation did finish and they can restart the Orchestrator and all services will work as expected.

Resolved in Orchestrator Version R5103-20230315-GA

Orchestrator build R5103-20230315-GA was released on 03-15-2023 and is the 3rd Orchestrator rollup for Release 5.1.0.

This Orchestrator rollup build addresses the below critical issues since the 2nd Orchestrator rollup build, R5102-20230222-GA.

  • Fixed Issue 107587: An Operator cannot edit the user details for other Operators or for Customer Administrators on the VMware SASE Orchestrator.

    Operators are expected to have the privilege to edit user details under Administration > User Management for another Operator if their role allows it and to edit a Customer Administrator if management delegation is activated. With this issue the Operator would observe user details as 'read only' with no power to edit.

  • Fixed Issue 107725: When a user using the New UI of the VMware SASE Orchestrator navigates to Monitor > Edges > Map Distribution, the map only shows a maximum of 20 VMware SD-WAN Edges at any one time and not all the Edges a customer has deployed.

    This issue impacts customers deploying > 20 Edges as the Map Distribution for the New UI will only shows 20 Edges at a time on the customer's Edge list. When the user clicks to the next page of the list, the map would display only those Edges on that page. There is no option to list all the Edges for a customer enterprise.

  • Fixed Issue 108533: When a user navigates to Service Settings > Edge Management > Configure Updates, the explanatory text for the settings is not clear on the New UI of the VMware SASE Orchestrator.

    The setting name Disable Edge Configuration Updates contradicts the setting's actual function, and with this fix is changed to Enable Edge Configuration Updates to align with the function and explanatory text.

  • Fixed Issue 108833: If a user changes a BGP filter on a non-global segment on the New UI of the VMware SASE Orchestrator, the Orchestrator overwrites this BGP filter change on the global segment with this new setting.

     For a customer using BGP on multiple segments with unique settings on each, this issue has the potential to cause a network outage for the users using the global segment where the BGP settings are overwritten. The issue is not observed when using the Classic UI.

  • Fixed Issue 109064: A user can configure the same SSID (Service Set Identifier) on two different Wi-Fi interfaces for a VMware SD-WAN Edge if the user configures the MAC filter on one and not on the other Wi-Fi interface.

    Configuring the same SSID to WLAN1 and WLAN2, one with MAC filter on and other with MAC filter off, or with a different MAC allowed list on each interface should not be allowed by the VMware SASE Orchestrator as this allows unapproved MAC addresses to get connected.

Resolved in Orchestrator Version R5102-20230222-GA

Orchestrator build R5102-20230222-GA was released on 02-24-2023 and is the 2nd Orchestrator rollup for Release 5.1.0.

This Orchestrator rollup build addresses the below critical issues since the 1st Orchestrator rollup build, R5101-20221220-GA.

  • Fixed Issue 40584: When a user looks at Monitor > Edge > Sources on the VMware SASE Orchestrator, they may observe that even though new client devices have been added to a VMware SD-WAN Edge, the Orchestrator does not display the latest client device when IP Visibility mode is used.

    This issue can occur when the Edge's Visibility Mode is configured as Visibility by IP Address. The issue is caused by the Orchestrator not processing the latest client information for the Edge when using the IP Address mode correctly and thus only showing the older client and their IP address.

  • Fixed Issue 105610: When a user attempts to create a new IPv4 Object Group or attempts to update an existing IPv4 Object Group which includes a wild card mask that starts with '255' and ends with '0' (for example, 255.0.1.0), the VMware SASE Orchestrator does not allow this wild card mask and throws an error even though this is a valid wild card expression and it should be permitted.

    Beginning with 5.0.x and later, Orchestrators lack the validation for wild card masks in Object Groups, and as a result throws an error when a user configures a wild card mask for one.

  • Fixed Issue 106159: A VMware SASE Orchestrator running Release 5.1.0.0 and 5.1.0.1 does not allow native users to create API tokens when authentication is configured as Single Sign-On (SSO) and the corresponding Identity Provider (IdP) does not support introspection endpoint.

    Release 5.1.0.0 introduces a check of IdP introspection endpoint during API token creation validation. This check does not distinguish native users from non-native users and as long as SSO is configured for authentication, the Orchestrator will validate whether IdP supports introspection endpoint. As a result, if IdP does not support introspection endpoint, the validation will prevent both native and non-native users from creating API tokens. This condition is true for both Operator Users and Partner Administrators.

  • Fixed Issue 106242: A user accessing the Diagnostics > Remote Diagnostics page on the VMware SASE Orchestrator may experience being unexpectedly logged out of the Remote Diagnostics page while performing any Edge diagnostic.

    When a user encounters this issue, it is because the Orchestrator has reached a limit to the number of connections possible, and the Orchestrator signs out Remote Diagnostics users to ensure normal functioning. The issue is caused by the Orchestrator erroneously not releasing database connections once they are no longer needed, causing the Orchestrator to trigger connection limit behaviors.

  • Fixed Issue 106592: On a VMware SASE Orchestrator running Release 5.1.0.0 and 5.1.0.1, a customer using API's may observe the following symptoms: a) Active API tokens are revoked by the Orchestrator and, b) Services like APIv2 no longer work.

    Release 5.1.0.0 introduces a new backend job called batchRevokeApiTokenForInactiveSsoUsers that will run every 6 hours to revoke previously issued API tokens for inactive Single-Sign On (SSO) users. Defects in the backend job cause it to erroneously revoke all API tokens on the Orchestrator regardless of whom the tokens are created for.

    With the fix, a customer administrator with a super user or a partner administrator should manually delete inactive Identify Provider (IdP) users from the Orchestrator to prevent unauthorized access via API tokens.

  • Fixed Issue 106806: When a VMware SASE Orchestrator is upgraded to Release 5.1.0, customers connected to the Orchestrator may observe disruptions in customer traffic.

    The Orchestrator creates a new device settings module version as part of its upgrade to 5.1.0. The Orchestrator then pushes this new device settings version to all connected Edges, and this has the potential to be disruptive because certain device setting changes can trigger an Edge service restart which results in 10-15 seconds of customer traffic disruption. The fix for this issue ensures the Orchestrator does not push an updated device settings version to all connected Edges when it is upgraded to Release 5.1.0.

  • Fixed Issue 107410: For a Partner Administrator using the New UI on the VMware SASE Orchestrator, when attempting to assign a software image to one of the Partner's Customers the user would observe that they cannot scroll on the dropdown menu that lists the software images.

    The impact is that unless the Partner sees the software image they want on the initial display of images when the dropdown menu is opened, they will not be able to scroll down to find the image they want if it is farther down. Operator users have no issue doing this and Partner Administrator can still scroll on the Classic UI when assigning a software image to a Customer.

  • Fixed Issue 107637: A customer with a large of number of VMware Edges may find when navigating to Configure > Edges on the VMware SASE Orchestrator's Classic UI that the page does not load.

    This issue occurs on the Classic UI and the page would time out with an "An unexpected error has occurred" message. A user would not experience this issue when using the New UI.

    Error Screen.

    When troubleshooting the issue, an Operator would find in the Orchestrator logs that the method enterprise/getEnterpriseEdgeList is failing with the following error:

    2023-02-06T17:21:05.412Z - error: [[email protected]] [24775] API Invocation error for enterprise/getEnterpriseEdgeList: { code: undefined, message: 'Internal error' }

    This issue is caused by an Orchestrator API which retrieves the Edge list for a customer in scenarios like Configure > Edges. On the Classic UI this API is used in a way that can cause it to time out, especially when a large number of Edges needs to be retrieved (for example, five hundred Edges or greater).

  • Fixed Issue 107885: For any user on the New UI of the VMware SASE Orchestrator, the Configure > Overview page may not load for some VMware SD-WAN Edges.

    This issue is inconsistent and the Configure > Overview page may load for some Edges. The issue is triggered when the Edge's QoS module is not populated with segment information.

    On an Orchestrator without a fix for this issue, a user can configure a test Business Policy that does not impact user traffic and save it, at which point the Overview page will successfully load.

  • Fixed Issue 108074: When a user pulls down the Edge information box on the Monitor > Edge screen, the Description is missing when looking at this information on the New UI of a VMware SASE Orchestrator.

    On the Classic UI screen for Monitor > Edge, the dropdown Edge information screen shows the configured Description:


    On the New UI screen for Monitor > Edge, the dropdown Edge information screen does not show the configured Description:


    A user can configure the Edge's Description on the Configure > Edge > Overview page.

  • Fixed Issue 108309: On a VMware SASE Orchestrator's New UI using Release 5.1.0 , when a user attempts to associate a VMware SD-WAN Edge with a Non SD-WAN Destination (NSD) by using Edge Override, the tunnel configuration option is not displayed.

    The tunnel configuration option should display as the last column on the screen as Action with a + symbol in this issue that + symbol is missing.


    This issue is limited to Edge specific actions and if a customer adds an NSD through a Profile that the Edge is using (versus using Edge Override), the Action + option shows correctly.


Resolved in Orchestrator Version R5101-20221220-GA

Orchestrator build R5101-20221220-GA was released on 12-20-2022 and is the 1st Orchestrator rollup for Release 5.1.0.

This Orchestrator rollup build addresses the below critical issues since Orchestrator build, R5100-20221202-GA.

  • Fixed Issue 100133: A VMware SASE Orchestrator may experience performance issues.

    When a customer configures alarge number of Business Policy rules by associating then with an Edge Hub Cluster, the Orchestrator experiences performance issues each time it has to push these configurations to the Edges in that enteprise.

  • Fixed Issue 101835: The Cloud VPN section is not available in the New Orchestrator UI if the user selects a Non-Global Segment where Cloud VPN is configured.

    In the New Orchestrator UI, on the Configure > Edge > Device settings page, the Cloud VPN section is not available if the user selects a Non-Global Segment where Cloud VPN is configured.

  • Fixed Issue 102806: Customer cannot edit the Partner Gateway Handoff configuration at a per Gateway level.

    This issue occurs when a customer configures the Partner Gateway Handoff during an upgrade.

  • Fixed Issue 103622: Operator user may observe the error message: “You don’t have permission to access this data” when attempting to access some pages in the VMware SASE Orchestrator.

    This issue is seen in the New Orchestrator UI when an Operator user attempts to access the Operator or Partner User Management page, after visiting the Global Settings, Cloud Web Security, or Secure Access pages under a Customer.

Resolved in Orchestrator Version R5100-20221202-GA

Orchestrator Version R5100-20221202-GA was released on 12-08-2022 and resolves the following issues since Orchestrator Version R5011-20221129-GA.

Note:

Release 5.1.0 contains all Orchestrator fixes that are listed in either the 5.0.0 or 5.0.1 Release Notes.

  • Fixed Issue 12132: When configuring a static route on the VMware SASE Orchestrator, the UI warns of an Edge Service Restart that does not actually occur.

    When changing the configuration on any static route and saving changes, the Orchestrator UI warns there will be an Edge restart and a disruption in service. This message is invalid and there is no Edge service restart associated with changing a static route configuration. The fix removes the spurious warning.

  • Fixed Issue 36058: A WAN link configured as a Backup link may show as gray on the Monitor > Overview page of the VMware SASE Orchestrator UI when the link is actually down.

    Screen would look like this:


    When looking at the Monitor > Edge > Overview page, the Backup link status displays an accurate status.

  • Fixed Issue 52952: The VMware SASE Orchestrator UI allows a user to configure invalid inputs for AS Path Prepend.

    The Orchestrator UI does not inspect an invalid AS Path Prepend value. A user can enter the AS PATH Prepend with values which include a comma, and the UI accepts this even though this is an invalid configuration for the Edge routing process. The invalid configuration is not applied and the user is given no feedback, for example a warning, an error message, or a tip regarding the invalid configuration.

  • Fixed Issue 53740: When configuring a Firewall rule on the VMware SASE Orchestrator UI, a user cannot configure a protocol value for a protocol it wants to match.

    The Orchestrator only allows TCP/UDP/ICMP/GRE in the protocol match criteria and does not allow protocol values from 0-255. The change allows a user to configure a matching Firewall rule and under Destination the user selects Other from the Protocol menu and then enter a value from 0-255.

    Note:

    While a user can enter a value from 0-255, protocol values 144-255 are considered reserved or unassigned per the IANA Assigned Internet Protocol Numbers documentation.

  • Fixed Issue 68347: The VMware SASE Orchestrator erroneously shows a Zscaler IPsec for GRE tunnel event as an Edge IPsec tunnel event on the Alerts page.

    Alerts are not expected to be generated for GRE tunnel events.

  • Fixed Issue 70987: A user may be unable to delete a VMware SD-WAN Edge from the VMware SASE Orchestrator.

    The Edges are offline but not being updated to disconnected but instead stay in a degraded status and thus not eligible for deletion.

  • Fixed Issue 72444: When a user configures IPv4 and IPv6 BGP neighbors for a VMware SD-WAN Edge and then attempts to turn off the BGP settings using the On/Off sliding button, the configuration is not saved and the VMware SASE Orchestrator UI says "No Changes".

    In this rare case where a user is both configuring BGP setting but also turning off BGP, the user's actions are not being saved for BGP settings as they should.

  • Fixed Issue 73481: When two or more VMware SD-WAN Gateways are deployed at the same location, an Operator may observe that one Gateway is being used by most of the VMware SD-WAN Edges while the other Gateway(s) are underutilized which may lead to performance issues on the fully utilized Gateway.

    With this issue one Gateway will be 90% utilized while another at the same location will be sitting at 10% utilization with resources to spare. Primary and Secondary Gateway assignments for Edges must take into account the existing load on the Gateways. If not, a single Gateway gets overwhelmed as the Primary Gateway while there is plenty of capacity left unused on a Secondary Gateway.

  • Fixed Issue 75175: When a Customer Administrator attempts to access the Gateway Information page on the VMware SASE Orchestrator, the page fails with an error.

    After a Gateway migration is done, Monitor > Edge > View (under Gateway), gets the error message: Error Loading Gateway info


    Customer Administrators are not expected to be able to access the Gateway Information page, however when they try to access it due to their privilege level the sections errors out.

  • Fixed Issue 76091: A user may observe that while editing a subnet on the Configure > Device screen, the screen freezes.

    If either the Reset button or the move Edge to "Eligible VPN Exits" is clicked and then the user clicks Save for a subnet that does not have learned routes, the UI freezes with the loading icon spinning forever.


     

    The issue is caused by the learnedRoute array missing from those subnets, which triggers a UI failure.

  • Fixed Issue 76182: When selecting certain time intervals on the Monitor > Edge page of the VMware SASE Orchestrator UI, the Orchestrator may return incomplete data.

    When a user query with intervals which are multiples of 5 i.e. 12:00:00 - 13:00:00 backend sends only 11 samples of 5 minutes, instead of the correct 12 samples due to a in issue with the Edge Link Statistics API.

  • Fixed Issue 77538: When a customer enterprise is migrated from one VMware SASE Orchestrator to a different Orchestrator, the customer may observe duplicate Operator Profiles and Edge software images.

    Not only are the duplicate Operator Profiles confusing for the customer, they also still point to the old Orchestrator and thus an Edge using one would heartbeat to the wrong Orchestrator, causing the Edge to be marked as down and not getting configuration updates.

  • Fixed Issue 79383: When making a configuration change in Configure > Device, a user may see an error message but not the segment name sage where the error occurred.

    One example is while switching and Edge interface from route to switched at the profile level, a user would see an string 'object.segment.name' instead of Segment name when there is some error is exists in device settings.


    If the customer enterprise is using multiple segments, troubleshooting becomes very difficult when it is not clear which segment has the error.

  • Fixed Issue 80445: When configuring OSPF on the VMware SASE Orchestrator, a user can observe that the OSPF Area ID allows duplicate entries between IP and Number, and the OSPF Area ID value '0' is allowed as a valid ID.

    The Orchestrator does not check for duplicate entries against both format type IPs and Number for Area IDs. In addition, it allows the value '0' as a valid OSPF Area ID. As a result user can mistakenly configure and publish an invalid OSPF configuration with negative effects following from that.

  • Fixed Issue 81364: When using the New Orchestrator User Interface to configure VMware SD-WAN Edge interfaces, speed and duplex settings are not getting updated for routed interfaces.

    On an Orchestrator without a fix for this issue, the user would need to use the Classic Orchestrator to configure speed and duplex settings for an Edge's routed interface.

  • Fixed Issue 81366: When using the New Orchestrator User Interface to configure a customer site using High-Availablity, the changes are not saved for the APR probe interval under the LOS values.

    The revised APR probe interval under LOS Detection does not display the configured values for the HA Edge. On an Orchestrator without a fix for this issue, APR probe values need to be configured on the Classic Orchestrator.

  • Fixed Issue 83342: When attempting to create a report with too many Edges, the VMware SASE Orchestrator delivers a vague error message that does not explain why the report failed.

    Generating a report which errors out does not show the error details in the UI and prevents the user from correcting what is causing the failure.


    The service log data will have the missing details if needed on an Orchestrator without the fix.

    N/A

  • Fixed Issue 83345: If the attempt to create a report fails on the VMware SASE Orchestrator due to timing out, the UI does not display a validation error.

    This ticket is related to 83342 and in both cases the logs do not proved enought detail to debug the issue. The difference here is the cause of the issue can be generating a report with no Edges can also causes a report time out with no error explanation.

  • Fixed Issue 83694: When a user logs into a VMware SD-WAN Edge's Local UI, the VMware SASE Orchestrator does not record and display this action in Monitor > Events.

    The customer administrators would not be aware of any local user logins to an Edge's local user interface.

  • Fixed Issue 84064: For a customer who is a deploying Microsoft Azure Virtual Hub, they have the option of configuring BGP on the VMware SASE Orchestrator.

    BGP is not officially supported, on "Microsoft Azure Virtual Hub", which is Automated. But users are allowed to configure BGP on the Orchestrator and should a user configure Azure automation with BGP, tunnels will go down to the Gateway and the Azure site will not pass traffic.


  • Fixed Issue 88811: On a VMware SASE Orchestrator using Release 5.0.x, an Operator Superuser cannot create API tokens for customer users even though they have delegated priviliges.

    This issue is caused due to the Operator Superuse not having the CREATE ENTERPRISE_API_TOKEN privilege. As a result the Operator cannot create, read, or revoke API tokens for other users.

  • Fixed Issue 88845: When a user tries remove interfaces from a Corporate VLAN list and saves changes on a VMware SASE Orchestrator using the Classic UI, the Orchestrator does not remove the interfaces.

    During the saving changes of this action, the Orchestrator ignores the configuration change because the json data format does not match the Classic UI's data format.

  • Fixed Issue 88910: On a VMware SASE Orchestrator running Release 4.5.1 or 5.0.x, a user is unable to run a WAN Link speed test for a VMware SD-WAN Edge using Remote Diagnostics > WAN Link Speed Test.

    When trying to use the WAN Link Speed Test diagnostic the user will not be able to choose an interface for the speed test as there is no drop down menu for the interfaces present.

    On an Orchestrator without a fix for this issue, if a user wants to force a bandwidth test for a WAN link, as a workaround the user can change the bandwidth measurement method for that WAN link, which triggers a retest automatically.  This can be done as follows:

    1. On the Orchestrator UI for a particular Edge, go to Configure > Device and scroll down to WAN Settings.

    2. For the WAN link to be retested, select Edit for the link to be remeasured and then select Advanced to reveal additional settings.

    3. Under the Bandwidth Measurement menu change the Bandwidth Measurement method from the current to a different method (for example if the link uses Burst Mode, change to Slow Start, or vice versa) and click Update Link and then Save Changes at the top of the Device page.  

    4. Once a measurement has been performed on the WAN link, change the measurement method back to the original method to ensure the most accurate measurement for the respective WAN link. Doing this triggers an additional bandwidth test.

  • Fixed Issue 88998: A user cannot clone a Business Policy or Firewall Rule on a VMware SASE Orchestrator running Version 5.0.x using the Classic UI.

    In the 5.0.x Release for the Orchestrator's Classic UI, IPv6 and Mixed (IPv4 and IPv6) options were disabled for the Edge Business Policy rule. But users were able to use them in case of cloning already existing rules but the result is an error message.

    This issue does not occur on the Orchestrator's New UI and a user can work around this issue using the New UI.

  • Fixed Issue 91231: For a VMware SASE Orchestrator deployed with a Disaster Recovery topology, the initial setup of DR replication may fail if a large table truncation job is triggered during the large table import stage of the DR process.

    If the background import of statistics lasts more than 24 hours, it collides with automated truncating jobs that run daily. The job trucates the old partitions from large tables which get replicated on the standby MySQL server as well.

    However between the process, MySQL might have data import going on for the same table and that causes the replication to fail and as a result of this overall DR process also fails.

    Should this issue be encountered on an Orchestrator without the fix, the DR process can be started after the UTC day change hour. However this does not guarantee the prevention when the Orchestrator is large and where DR can take over 24 hours to finish.

  • Fixed Issue 91393: For a customer using collecting data from the Edge using syslong or telnet, the user may observe two names for the same application in NetFlow data from the VMware SD-WAN Edge.

    This issue is encountered when there are applications for which the displayName in the language file (appids_en_us.json) is different from the displayName in the applications file (applications.json). Since displayName of the applications.json file is used on the Edge side, this should not change.

    The issue is seen because the displayName of the applications.json file was getting changed to the displayName from the appids_en_us.json file as part of an API response and whenever application map data is updated, it is updating every application map displayName even when the user has not changed it and the same is getting pushed to the Edge.

  • Fixed Issue 91407: When a user defined WAN overlay is configured as a Backup link, the VMware SASE Orchestrator running Release 5.0.x shows the link as up on Monitor > Edge when the link is actually in Backup mode.

    This is a display issue only and Backup WAN link feature works properly. The issue is caused by the Orchestrator not storing the Link Mode values for the Edge properly and results in the wrong status being displayed.

  • Fixed Issue 93846: For Partners and Operators managing Edge inventory using ZTP, when a user tries to reassign a VMware SD-WAN Edge to a different customer that was previously assigned to one customer and then deleted, the VMware SASE Orchestrator returns the error "Edge is not found."

    The Orchestrator determines the Edge does not exist because it does not see the logical Edge after it is deleted from a customer enterprise and a user is unable to reassign it to another customer.

  • Fixed Issue 97055: For a user logged onto the VMware SASE Orchestrator as either a Partner or an Enterprise Administrator, if that user goes to the Configure > Edges page of the New UI, they would observe that they do not have the Assign Software/Firmware Image option in the dropdown menu when one or more Edges is selected.

    The result is that a user cannot assign software or firmware to an individual or group of Edges when using the New UI. The issue is caused by the incorrect privileges being assigned for Partner and Enterprise users on the New UI only, as these privileges are correct on the Classic UI.

    On a 5.0.1 Orchestrator, the user can optionally switch to the Classic UI and perform the task. This issue is fixed on any Orchestrator using version 5.1.0 or later.

Known Issues

Edge/Gateway Known Issues

Open Issues in Release 5.1.0.

  • Issue 105933: A user cannot SSH to VMware SD-WAN Edge models 610/610-LTE or 520/540 via a routed interface.

    There is no drop rule for duplicate SSH packets which originate via an af-pkt driver used by the affected Edge's OS. Because of this the Edge kernel receives 2 SSH packets: one via the vce1 interface, and another direct SSH packet because of the nature of the driver. This causes the Edge kernel to reply for 2 SSH requests, confusing the SSH client and results in the SSH failure.

    Workaround: For an Edge without a fix for this issue, the user can add an IP table rule to drop the SSH packets received from interfaces other than vce1.

  • Issue 117037: For a customer using a Hub/Spoke topology where multiple WAN links are used to send and receive traffic from the Spoke Edge to the Hub Edge, customers may observe lower than expected performance for traffic that is steered by Business Policies because the WAN links are not aggregating the WAN link's bandwidth.

    SD-WAN uses a counter for accounting the number of packets buffered in a resequencing queue. This counter is managed per peer and used to make sure only 4K packets are buffered per peer. Under some conditions, this counter can become negative. Prior to Release 4.2.x, when this counter became negative, the respective counter was immediately reset back to 0 after flushing the packets in the resequencing queue. However, starting in Release 4.3.x, this counter is updated automatically to ensure that the counter stays within expected bounds.

    The result of this change in behavior can cause cases where the counter accounting is incorrect and the resequencing queue can stay at a very high number to which SD-WAN reacts by flushing every single packet. This action not only prevents bandwidth aggregation but can reduce the effectiveness of flows that would otherwise be on a single link.

    Workaround: On Edges without a fix for this issue, the workaround is to configure business policies that steer matching traffic to a single mandatory link.

Orchestrator Known Issues

check-circle-line exclamation-circle-line close-line
Scroll to top icon