Updated 25 SEPTEMBER, 2023 VMware SASE™ Orchestrator Version R5002-20220517-GA VMware SD-WAN™ Gateway Version R5002-20220506-GA VMware SD-WAN™ Edge Version R5002-20220506-GA VMware Cloud Web Security™ using Orchestrator Version R5002-20220517-GA VMware Secure Access™ using Orchestrator Version R5002-20220517-GA VMware Edge Network Intelligence™ using Orchestrator Version R5002-20220517-GA Check regularly for additions and updates to these release notes.

What's in the Release Notes

The release notes cover the following topics:

• Recommended Use
• Compatibility
• Upgrade Paths for Orchestrator, Gateway, and Edge
• Important Notes
• New SD-WAN Features
• SD-WAN Feature Enhancements
• New SASE Features
• New Edge Network Intelligence Features
• Feature Enhancements
• Orchestrator API Changes
• Revision History
• Resolved Issues
• Known Issues

Recommended Use

This release is recommended for all customers who require the features and functionality first made available in Release 5.0.0.

Compatibility

Release 5.0.0 Orchestrators, Gateways, and Hub Edges support all previous VMware SD-WAN Edge versions greater than or equal to Release 3.2.2 
Note: this means releases prior to 3.2.2 are not supported.

The following SD-WAN interoperability combinations were explicitly tested:

Orchestrator	Gateway	Edge
		Hub	Branch/Spoke
5.0.0	4.2.0	4.2.1 & 4.2.2	4.2.1 & 4.2.2
5.0.0	5.0.0	4.2.1 & 4.2.2	4.2.1 & 4.2.2
5.0.0	5.0.0	5.0.0	4.2.1 & 4.2.2
5.0.0	5.0.0	4.2.1 & 4.2.2	5.0.0
5.0.0	R3.4.5	3.4.5 & 3.4.6	3.4.5 & 3.4.6
5.0.0	5.0.0	3.4.5 & 3.4.6	3.4.5 & 3.4.6
5.0.0	5.0.0	5.0.0	3.4.5 & 3.4.6
5.0.0	5.0.0	3.4.5 & 3.4.6	5.0.0
5.0.0	4.3.0	4.3.0	4.3.0
5.0.0	5.0.0	4.3.0	4.3.0
5.0.0	5.0.0	5.0.0	4.3.0
5.0.0	5.0.0	4.3.0	5.0.0
5.0.0	4.5.0	4.5.0	4.5.0
5.0.0	5.0.0	4.5.0	4.5.0
5.0.0	5.0.0	5.0.0	4.5.0
5.0.0	5.0.0	4.5.0	5.0.0
5.0.0	4.5.0	5.0.0	4.5.0
5.0.0	3.2.2	3.2.2	3.2.2
5.0.0	5.0.0	3.2.2	3.2.2
5.0.0	5.0.0	5.0.0	3.2.2
5.0.0	3.3.2 P2	3.3.2 P2	3.3.2 P2
5.0.0	5.0.0	3.3.2 P2	3.3.2 P2
5.0.0	5.0.0	3.3.2 P2	5.0.0

Note: The above table is only valid for customers using SD-WAN services. Customers requiring access to VMware Cloud Web Security or VMware Secure Access need their Edges upgraded to Release 4.5.0 or later.

Warning: VMware SD-WAN Releases 3.2.x and 3.3.x have reached the End of Support.

• Releases 3.2.x and 3.3.x reached End of General Support (EOGS) on December 15, 2021, and End of Technical Guidance (EOTG) March 15, 2022.

Warning: VMware SD-WAN Releases 3.4.x is approaching End of Support for the Orchestrator and Gateway.

• Release 3.4.x for the Orchestrator and Gateway reached End of General Support (EOGS) on March 30, 2022, and will reach End of Technical Guidance (EOTG) on September 30, 2022.
• Note: This is for the Orchestrator and Gateway only. 3.4.x for the Edge is scheduled to enter its End of Support window beginning on December 31, 2022.
• For more information please consult the Knowledge Base article: Announcement: End of Support Life for VMware SD-WAN Release 3.x (84151)

Warning: VMware SD-WAN Releases 4.0.x and 4.2.x are approaching End of Support. 

• Release 4.0.x will reach End of General Support (EOGS) on September 30, 2022, and End of Technical Guidance (EOTG) December 31, 2022. 
• Release 4.2.x Orchestrators and Gateways will reach End of General Support (EOGS) on December 30, 2022, and End of Technical Guidance (EOTG) March 30, 2023.   
• Release 4.2.x Edges will reach End of General Support (EOGS) on June 30, 2023, and End of Technical Guidance (EOTG) September 30, 2023.
• For more information please consult the Knowledge Base article: Announcement: End of Support Life for VMware SD-WAN Release 4.x (88319)

Note: Release 3.x did not properly support AES-256-GCM, which meant that customers using AES-256 were always using their Edges with GCM deactivated (AES-256-CBC). If a customer is using AES-256, they must explicitly deactivate GCM from the Orchestrator prior to upgrading their Edges to a 4.x Release. Once all their Edges are running a 4.x release, the customer may choose between AES-256-GCM and AES-256-CBC.

Upgrade Paths for Orchestrator, Gateway, and Edge

The following lists the paths for customers wishing to upgrade their Orchestrator, Gateway, or Edge from an older release to Release 5.0.0.

Orchestrator

Due to infrastructure changes in the Orchestrator beginning in Release 4.0.0, any Orchestrator using a 3.x Release needs to be first upgraded to 4.0.0 prior to being upgraded to 5.0.0. Orchestrators using Release 4.0.0 or later can be upgraded to Release 5.0.0.  Thus, the upgrade paths for the Orchestrator are as follows:

Orchestrator using Release 3.x → 4.0.0 → 5.0.0.

Orchestrator using Release 4.x → 5.0.0.

Gateway

Gateway upgrades from 3.x to 5.0.0 are not supported. In place of upgrading, a 3.x Gateway needs to be freshly deployed with the same VM attributes, and the old instance is then deprecated.

Upgrading a Gateway using Release 4.0.0 or later is fully supported for all Gateway types.

Note: When deploying a new Gateway using 5.0.0 the VMware ESXi instance must be at least version 6.7, Update 3 up to version 7.0. Using an earlier ESXi instance will result in the Gateway's Dataplane Service failing when trying to run Release 5.0.0 or later.

Note: Prior to upgrading a Gateway to 5.0.0, the ESXi instance must be upgraded to at least version 6.7, Update 3 up to version 7.0. Using an earlier ESXi instance will result in the Gateway's Dataplane Service failing when trying to run Release 5.0.0 or later.

Edge

An Edge can be upgraded directly to Release 5.0.0 from any Release 3.x or later.

Important Notes

LAN-Side NAT Behavioral Change

When a LAN-side NAT is configured for many-to-one translations using Port Address Translation (PAT), traffic initiated from the opposite direction can allow unexpected access to fixed addresses based on the outside mask and original IP address. This new behavior applies to Destination NAT (DNAT), Source NAT (SNAT), and Source and Destination NAT (S+D NAT) rules.

For example, a SNAT rule with an inside network of 192.168.1.0/24 and an outside address of 10.1.1.100/32 permits outside-to-inside translation to 192.168.1.100.

To address this new behavior, SD-WAN now blocks traffic when a connection is initiated in the reverse PAT direction.

To restore the original behavior, a user needs to configure two rules of the same type as the original rule (SNAT, DNAT, S+D NAT) in a particular order. For example, using the earlier SNAT scenario a user needs to configure the following:

1. SNAT rule with an inside network of 192.168.1.100/32 and an outside address of 10.1.1.100/32
2. SNAT rule with an inside network of 192.168.1.0/24 and an outside address of 10.1.1.100/32

If the original rule is a DNAT or S+D NAT, then the user would need two DNAT or S+D NAT rules with the same structure and order.

In Release 4.5.0 and forward, a user can determine if flows are dropped for this type of traffic in the dispcnt logs of a diagnostic bundle by searching for the counter lan_side_nat_reverse_pat_drop.

Edge Diagnostic Bundle Triggered on a Reboot

Starting with Release 5.0.0, the Orchestrator will force the Edge to generate a diagnostic bundle before rebooting it through Remote Action > Reboot. This is the default behavior and cannot be changed or disabled. Users will see a new Edge Event, MGD_REBOOT_DIAG_BUNDLE, which indicates that a diagnostic bundle was generated by the Edge in response to an Orchestrator-initiated reboot. This new behavior assists users and technical support engineers in diagnosing and resolving issues that existed prior to the Edge reboot.

Expanded NAT Range for SD-WAN Gateways

Some SD-WAN Gateways may face the limitations of the original NAT range as they grow and handle more traffic. For example, the original NAT range may not be enough for Microsoft 365 (formerly Office 365) and some other services. The original NAT range only used ports 20001-64000, which allowed for 44K translations to a specific destination. To solve this problem, Gateway Release 4.5.1 and 5.0.0 change the Gateway NAT range to ports 5000-65534. This allows for ~60K translations to a specific destination.

Higher Baseline CPU Usage for Gateways

Gateways using Release 5.0.0 and forward will have increased CPU usage when Gateway CPU cores are idle and users should expect the baseline CPU usage to significantly increase over what they observe when the Gateway is using a 4.x software version. The increase is the result of the Release 5.x software using updated fast path code which includes run-to-completion scheduling and increases the number of times the Gateway process checks for work. This polling occurs even when the CPU is idle and a user should not interpret the higher CPU idle baseline as a diminished capacity to handle data and packet processing.

Potential Issue With Sites Using a High-Availability Topology

A site where a pair of Edges are deployed in a High-Availability topology may encounter an issue where the Standby Edge reboots one or more times to resolve an Active-Active state. The Standby Edge reboot(s) can cause a disruption of customer traffic with the impact greater on sites using an Enhanced HA topology as the Standby Edge also passes customer traffic. The issue is being tracked by Issue #85369 under the Edge/Gateway Known Issues section of these Release Notes.

AWS Edge Upgrade Not Supported for Release 5.0.0

 A VMware SD-WAN Virtual Edge which uses an AWS C4 instance cannot be upgraded to Release 5.0.0 due to Edge Open Issue #82264. For this issue, when an AWS C4 Virtual Edge is upgraded to Release 5.0.0, the Edge does not recover and there is no way to remediate the Edge's state beyond reactivating the Edge to a non-5.0.0 version.  No other AWS instances (for example, C5) are affected by this issue.

However, due to the critical nature of this defect, an AWS Edge Upgrade software package is not available for Release 5.0.0. Issue #82264 is resolved in all Release 5.0.1 builds and includes an AWS Edge Upgrade package. 

Grafana No Longer Available on Orchestrator

Release 5.0.0 Orchestrator does not include the Grafana application due to license restrictions. Grafana is primarily used by customers and partners who run the Orchestrator on-premises to monitor the Orchestrator's performance. Going forward for such needs, a customer or partner would need to host their own Grafana application outside the Orchestrator and configure Telegraf on the Orchestrator to point to it.

VMware SASE Builds Include a Fourth Digit

Beginning with Release 5.0.0 and going forward, the release build will now include a fourth digit.

For software releases, VMware SASE follows an a.b.c numbering scheme where:  

• a = Major (for example, 5.0.0) → A release with multiple large features and potentially significant architectural changes.
• b = Minor (for example, 5.2.0) → A release with a handful of small features or a couple of large features and no significant architectural changes
• c = Maintenance (for example, 5.2.1) → A release with potentially a large number of fixes for field found issues and internally found issue fixes with no features except potentially new hardware platform support.

With Release 5.0.0 a fourth digit is added to Edge, Gateway, and Orchestrator builds, so the numbering is a.b.c.d where

• d = Rollup Build (for example, 5.2.1.1) → A rollup is a cumulative aggregate of known customer found defect fixes or critical internal found defects.

Rollup Builds for 4.x and earlier are distinguished by the image name's GA date, which is not an optimal way of communicating the build version to a customer. Adding a fourth digit for 5.0.0 builds and later allows customers to more clearly see what software version is being used for a particular component.

This build numbering convention is true only for Release 5.0.0 and later and 4.x and earlier releases will continue with three digits with rollup builds identified in the existing manner by date.

Mixing Wi-Fi Capable and Non-Wi-Fi Capable Edges in High Availability Is Not Supported 

Beginning in 2021, VMware SD-WAN introduced Edge models which do not include a Wi-Fi module: the Edge models 510N, 610N, 620N, 640N, and 680N. While these models appear identical to their Wi-Fi capable counterparts except for Wi-Fi, deploying a Wi-Fi capable Edge and a Non-Wi-Fi capable Edge of the same model (for example, an Edge 640 and an Edge 640N) as a High-Availability pair is not supported. Customers should ensure that the Edges deployed as a High Availability pair are of the same type: both Wi-Fi capable, or both Non-Wi-Fi capable.

Accessing Cloud Web Security and Secure Access

A customer wishing to access VMware Cloud Web Security or VMware Secure Access must upgrade their Edges to Release 4.5.0 or later.  These services are inaccessible on Edges using a release earlier than 4.5.0.

BGPv4 Filter Configuration Delimiter Change for AS-PATH Prepending

Through Release 3.x, the VMware SD-WAN BGPv4 filter configuration for AS-PATH prepending supported both comma and space based delimiters. However, beginning in Release 4.0.0 and forward, VMware SD-WAN only supports a space based delimiter in an AS-Path prepending configuration.
Customers upgrading from 3.x to 4.x or 5.x need to edit their AS-PATH prepending configurations to "replace commas with spaces" prior to upgrade to avoid incorrect BGP best route selection.

Extended Upgrade Time for Edge 3x00 Models

Upgrades to this version may take longer than normal (3-5 minutes) on Edge 3x00 models (i.e., 3400, 3800 and 3810). This is due to a firmware upgrade which resolves issue 53676. If an Edge 3400 or 3800 had previously upgraded its firmware when on Release 3.4.5/3.4.6, 4.0.2, 4.2.1, or 4.3.0 then the Edge would upgrade as expected. For more information, please consult Fixed Issue 53676 in the respective release notes.

Limitation with BGP over IPsec on Edge and Gateway, and Azure Virtual WAN Automation

The BGP over IPsec on Edge and Gateway feature is not compatible with Azure Virtual WAN Automation from Edge or Gateway. Only static routes are supported when automating connectivity from an Edge or Gateway to an Azure vWAN.

Limitation When Deactivating Autonegotiation on VMware SD-WAN Edge Models 520, 540, 620, 640, 680, 3400, 3800, and 3810

When a user deactivates autonegotiation to hardcode speed and duplex on ports GE1 - GE4 on a VMware SD-WAN Edge model 620, 640 or 680; on ports GE3 or GE4 on an Edge 3400, 3800, or 3810; or on an Edge 520/540 when an SFP with a copper interface is used on ports SFP1 or SFP2, the user may find that even after a reboot the link does not come up.

This is caused by each of the listed Edge models using the Intel Ethernet Controller i350, which has a limitation that when autonegotiation is not used on both sides of the link, it is not able to dynamically detect the appropriate wires to transmit and receive on (auto-MDIX). If both sides of the connection are transmitting and receiving on the same wires, the link will not be detected. If the peer side also does not support auto-MDIX without autonegotiation, and the link does not come up with a straight cable, then a crossover Ethernet cable will be needed to bring the link up.

For more information please see the KB article Limitation When Deactivating Autonegotiation on VMware SD-WAN Edge Models 520, 540, 620, 640, 680, 3400, 3800, and 3810 (87208).

New SD-WAN Features

IPv6

Release 5.0.0 provides support for the following IPv6 features:

• IPv6 Overlay​
  • Customer has the option to configure the User Defined Overlay for IPv4 Only, IPv6 Only or IPv4 & IPv6 (Dual-Stack)
  • Dual-Stack mode prevents link oversubscription by accounting for both IPv4 and IPv6 traffic over the same link.
• IPv6 Edge Activation Email
  • ​The Edge Activation Email can now be sent with a IPv4 and/or IPv6 activation link.
  • IPv6 activation links are for Greenfield deployments only.
• IPv6 Overlay Flow Control
  • The Edge maintains separate IPv4 and IPv6 tables with separate route lookups.
  • Dynamic Cost Calculation (DCC) is activated by default with IPv6
  • Includes an option for IPv6 Stale Route Refresh
• IPv6 Business Policy
  • Business policies now have an IP Version configuration parameter for IPv4, IPv6, or IPv4 & IPv6
  • IPv4 and IPv6 options allow for matches by VLAN, Interface, Ports, and Object Groups.
  • Default Business Policies are automatically updated for IPv4 and IPv6 with Release 5.0.0
  • Note: some applications have no IPv6 support including Skype, Zoom, GoToMeeting, Sharepoint, and VMware Horizon, amongst others
• IPv6 Stateful Firewall:
  • Stateful Firewall Rules can be created for IPv4, IPv6, or IPv4 & IPv6
  • Configuration page includes an enhanced IP Address field which supports IPv6 addresses
• IPv6 NAT
  • IPv6-to-IPv6 Network Prefix Translation (NAT66) by default on SD-WAN Gateways
  • 1:1 NAT IPv6 maps a specific public IPv6 address to a specific LAN IPv6 address
  • 1:1 NAT IPv6 can translate outside IPv6 addresses in different subnets from the WAN interface if the ISP routes traffic for the subnet towards the SD-WAN Edge
  • Port Forwarding with a IPv6 address
• Partner Gateway Handoff with BGPv6 and BFDv6
  • ​Supports Inbound & Outbound BGPv6 filters
  • Partner has both IPv4 and IPv6 configuration options
  • IPv6 routes learned via BGP are synchronized with the Overlay Flow Control
  • This feature can be configured on the Classic UI only with Release 5.0.0

Edge Factory Software and Firmware Updates on VMware SD-WAN Edges

Starting with Release 5.0.0, SD-WAN Edge platform components can be updated outside of the Edge image software. A 5.0.0 Edge connected to a 5.0.0 Orchestrator allows the partner or customer administrator to manage platform firmware, and update the Edge's factory default image.

Secure Edge Access + CLI

Secure Edge Access + CLI delivers a secure method to support CLI access to Edges using key pairs generated per user and sends a logged-in user into an Edge CLI shell that only exposes SD-WAN troubleshooting commands and meets CSO requirements.

Both the Edge and the Orchestrator must be using Release 5.0.0 or later for this feature to be available.

SD-WAN Feature Enhancements

Firewall Source Match

In Release 4.x and earlier, when configuring a Firewall rule, the user had to select between an interface or an IP address in the Source match criteria, but not both. In Release 5.0.0 a user can now configure a rule for both the interface and the IP address so that user can match a specific IP address on the incoming interface.

Greek Localization

Release 5.0.0 adds Greek language localization for the Orchestrator and for SD-WAN and SASE documentation.

New SASE Features

Data Loss Prevention (DLP)

Data Loss Prevention (DLP) protects a Cloud Web Security customer against loss from accidental or intentional sharing of restricted data. The DLP feature is available for Cloud Web Security customers with an Advanced Package. Cloud Web Security DLP includes the following capabilities:

• Prevents data loss to ensure compliance with HIPAA, PCI, GDPR, and other data privacy laws
• Over 350 out-of-the-box dictionaries for inspecting traffic
• Custom dictionaries may be configured using regular expression or strings
• Non-compliant activity is reported to designated administrators

Note: The DLP feature is also available in Release 4.5.0/4.5.1 beginning with VMware SASE Orchestrator build R450-20220315-GA. 

New Edge Network Intelligence Features

Self-Healing Networks

Self-Healing Networks leverages both Edge Network Intelligence and SD-WAN to remediate network issues in real time. The Self-Healing Networks feature analyzes global conditions, detects degradation in real time, and provides the customer with a recommended corrective action that the customer has the option to follow.

Network incidents where application performance degrades are automatically detected when Self-Healing is used. The feature then provides a report on the Incident that provides key insights including: affected Edges, root causes, and other impacted applications. The Incident Report also provides a Remediation recommendation. In the initial implementation of Self-Healing, the user would manually elect to act upon the Remediation recommendation.

Any remediation actions taken through Edge Network Intelligence are automatically annotated on the Network History page and include the number of Edges updated and the application(s) impacted.

Activating Analytics and Self-Healing

Previously Edge Network Intelligence Analytics had to be activated one Edge at a time. In the New UI a user can select multiple Edges and configure the Analytics Settings to activate either Analytics alone, or Analytics + Self-Healing.

Orchestrator API Changes Since Release 4.5.0

Changes to the VMware SD-WAN Orchestrator Portal API ("API v1")

The complete API Changelog is available for download at developer.vmware.com (see: "VMware SD-WAN Orchestrator API v1").

The following changes may require action from developers:

• The deviceSettings, QOS, and firewall configuration modules have been augmented with new configuration options in order to support IPv6. 
• Some of these settings are introduced by way of a system patch on upgrade of the Orchestrator to the 5.0.0 release, and are treated as required by the server following the upgrade. Users that rely on "template" profiles will need to update those profiles to include the new required settings.
• The revised schema for each of these modules is documented in the API reference (see "segmentBasedDeviceSettings", "firewall", and "QOS" in the modules section of the reference at the bottom of the page).
• With the introduction of support for the configuration of Edge platform component firmware independently of software (see "Edge Factory Software and Firmware Updates on VMware SD-WAN Edges"), the API now requires the following properties to be present in imageUpdate configuration module "data": "devicefamily", "modemFirmware", "platformFirmware", and "factoryFirmware". This module is exclusive to Operator Profiles, and the change should not impact client applications concerned with Customer Profiles or Edge-Specific settings.
• 76036: Adds validation logic to all /metrics/* methods which causes the API to respond with an error in cases where the query interval start time precedes the start of the system-global retention window (2 weeks for Edge "health stats," 40 weeks for all other metric types, by default). Previously the server honored these queries, but returned incomplete data.
• 74050: Removes a redundant "vlanId" property (which was unused by the Edge) from IPv6-specific addressing options (the "v6detail" object) for routed interfaces contained within Profile and Edge deviceSettings configuration modules.
• 69028: Introduces validation that prevents Customer Administrators from invoking enterprise/reassignDataCenterGateway (which changes the Gateway designated for tunneling to a target Non-SD-WAN Site) unless the target gateway is in "quiesced" mode.
• 64145: Causes a minor change in API behavior when handling updates to device settings configuration modules in cases where partner handoff gateways are configured, whereby the server will now always honor the gateway assignments specified in "segments[] > handOffGateways > gatewaysList" over those specified in "segments[] > handOffGateways > gateways", whenever the former property is present. Previously we had issued guidance to some users that the "segments[] > handOffGateways > gateways" needed to be deleted in some cases to trigger this behavior, but that is no longer necessary.

Changes to the VMware SD-WAN Orchestrator SD-WAN API v2

This release adds support for configuration of profile and Edge device settings such as HA, Wi-Fi, cloud security, routing protocols, and more. It introduces the following new API operations:

• For configuration of Customer Profiles:
  • GET /enterprises/{enterpriseLogicalId}/profiles/{profileLogicalId}/deviceSettings
  • PATCH /enterprises/{enterpriseLogicalId}/profiles/{profileLogicalId}/deviceSettings
  • PUT /enterprises/{enterpriseLogicalId}/profiles/{profileLogicalId}/deviceSettings
• For configuration of specific Edges:
  • GET /enterprises/{enterpriseLogicalId}/edges/{edgeLogicalId}/profiles/{profileLogicalId}/deviceSettings
  • PATCH /enterprises/{enterpriseLogicalId}/edges/{edgeLogicalId}/profiles/{profileLogicalId}/deviceSettings
  • PUT /enterprises/{enterpriseLogicalId}/edges/{edgeLogicalId}/profiles/{profileLogicalId}/deviceSettings

With these API operations, VMware SASE has introduced some key API capabilities and design patterns (for example, support for partial resource updates via HTTP patch, consistent handling and reporting of syntax errors, and asynchronous-by-default execution) that will guide how users perform many configuration management tasks as additional functionality is introduced in future releases.

There are two other changes that may require action on the part of API users:

• In order to standardize URL structures across the broader suite of VMware SASE platform API services (including Cloud Web Security and Secure Access), the canonical base path for SD-WAN "APIv2" is changed in this release from "/sdwan" to "/api/sdwan/v2". To ensure backward-compatibility with earlier Orchestrator software releases, the Orchestrator continues to honor requests to the "/sdwan" base path so long as clients are configured to follow HTTP 308 redirects. Based on an analysis of API traffic, most users should not observe any disruption as a result of this change. Nevertheless, it is recommended that all APIv2 users begin using the new URL base path (/api/sdwan/v2) upon upgrade to the 5.0.0 Orchestrator release (or else users should ensure that client applications and scripts are configured to follow HTTP redirects). Additional changes of this kind are not expected going forward.
• Due to an operational error, the APIv2 rate-limiting module has not been enforcing the same default policy that the Portal API rate-limiter does, which was always the intention. A change in this release effectuates that policy for APIv2. Users should review best practices for avoiding triggering the rate-limiter and handling responses where rate limiting is triggered.

Changes to Developer Documentation

Historically, VMware SASE/SD-WAN API documentation was hosted on VMware {code} at code.vmware.com. VMware {code} was recently migrated to a new domain, developer.vmware.com. As a result of the migration, some permalinks to specific pages that were previously hosted on code.vmware.com may no longer work as expected.

In conjunction with the migration, VMware also recently launched a new Developer Documentation Portal at https://developer.vmware.com/apis, where all VMware SASE/SD-WAN API documentation now reside.

Document Revision History

February 28th 2022. First Edition.

March 4th, 2022. Second Edition.

• Under Orchestrator API Changes Since Release 4.5.0, added material to the Changes to the VMware SD-WAN Orchestrator Portal API ("API v1") section regarding the ability to upgrade Edge firmware. Added text begins: "With the introduction of support for the configuration of Edge platform component firmware independently of software..."

March 17th, 2022. Third Edition.

• Added Fixed Issue #71745 to the Edge/Gateway Resolved Issue for the GA build R5000-20220225-GA. It was previously omitted as it was not previously found in the field.
• Under the Compatibility section, added a new Warning that Release 3.4.x software is approaching End of Support for the Orchestrator and Gateway with End of General Support (EOGS) on March 30, 2022, and End of Technical Guidance (EOTG) June 30, 2022. This is for the Orchestrator and Gateway only. The 3.4.x Edge software is scheduled to enter its End of Support window beginning on December 31, 2022.
• Added a new Important Note regarding the Limitation with BGP over IPsec on Edge and Gateway, and Azure Virtual WAN Automation. The note reads: "The BGP over IPsec on Edge and Gateway feature is not compatible with Azure Virtual WAN Automation from Edge or Gateway. Only static routes are supported when automating connectivity from an Edge or Gateway to an Azure vWAN."

March 21st, 2022. Fourth Edition.

• Added Greek Localization for SASE Documentation to the Feature Enhancements section. This enhancement adds downloadable PDF versions of the primary SASE (Release Notes, SD-WAN, Cloud Web Security, and Secure Access) documentation which can be accessed at this site.  

March 23rd, 2022. Fifth Edition.

• Added a new Orchestrator build R5001-20220317-GA to the Orchestrator Resolved section. This build is the new Orchestrator GA build for Release 5.0.0. As a reminder, the fourth digit represents a rollup build and this is the first Orchestrator rollup build for 5.0.0.
• Added Fixed Issues #69573, #76994, #78688, #83538, #83550, #83582, #83902, #83940, #84083, #84286, #84297, #84299, and #84300 to the Orchestrator Resolved Issues for R5001-20220317-GA.
• The fixed Orchestrators issues impact VMware SASE services as follows:
  • VMware SD-WAN: #78688, #83582, #83902, and #84083.
  • VMware Cloud Web Security: #76994, #83550, #83940, #84286, #84297, #84299, and #84300.
  • VMware Secure Access: #69573, #83538.
• Added Issue #84825, to the Edge/Gateway Known Issues section.

May 3rd, 2022. Sixth Edition.

• Added a new warning in the Compatibility section regarding Release 4.0.x approaching End of Support.
• Removed Issues #46254 and #49738 from the Edge/Gateway Known Issues section as they had been resolved with the 4.3.0 Release and are thus also resolved in 5.0.0.

May 24th, 2022. Seventh Edition.

• Added a new Edge/Gateway rollup build R5002-20220506-GA to the Edge/Gateway Resolved section. This is the new Edge/Gateway GA build for Release 5.0.0.
• Edge/Gateway build R5002-20220506-GA includes the fixes for issues #74316, #83209, and #87956 which are each documented in this section.
• Added a new Orchestrator build R5002-20220517-GA to the Orchestrator Resolved section. This build is the new Orchestrator GA build for Release 5.0.0.
• Added Fixed Issues #81960, #84600, #84969, #85291, #85338, #85412, #86083, #86573, #88796, and #89005 to the Orchestrator Resolved Issues for R5002-20220517-GA.
• The fixed Orchestrators issues impact VMware SASE services as follows:
  • VMware SD-WAN: #84969, #85291, #85338, #86573, and #88796.
  • VMware Cloud Web Security: #81960, #84600, #85412, #86083, and #89005.
• Added Open Issue #62701 to the Edge/Gateway Known Issues section as this issue remains unresolved on all releases at this time.
• Added Open Issue #88796 to the Edge/Gateway Known Issues section. This issue affects both Orchestrators and Gateway OVA's but the fix for the Orchestrator is included with the R5002-20220517-GA build.  As of this revision, there is no Gateway OVA with the fix for #88796.
• Amended the Edge/Gateway Fixed Issues section to add Fixed Issue #77525 for R5000-20220225-GA, as this was omitted in error.

June 1st, 2022, Eighth Edition

• Added Fixed Issue #81835 to Orchestrator Resolved Issues for the original GA build. This ticket was omitted in error from the First Edition of the 5.0.0 Release Notes.
• Added Issues #85369 and #85461 to the Edge/Gateway Known Issues section.

June 16th, 2022, Ninth Edition

• Added a new Important Note: "Potential Issue With Sites Using a High-Availability Topology" regarding ongoing issues with customer sites using a High-Availability topology for a pair of Edges. This issue continues to be tracked by Issue #85369 located in Edge/Gateway Known Issues.
• Under Compatibility, amended the End of Life dates for Release 4.2.x Edge software. The Edge software is broken out as a separate item and now reads: "Release 4.2.x Edges will reach End of General Support (EOGS) on June 30, 2023, and End of Technical Guidance (EOTG) September 30, 2023."  The separate Orchestrator and Gateway entry retains the same End of Life dates as before.
• Added Fixed Issue #53951 in the Edge/Gateway Resolved Issues section. This ticket was omitted from the original 5.0.0 Release Notes in error.
• Added Open Issue #75348 to the Orchestrator Known Issues section.

July 1st, 2022. Nineteenth Edition

• Added Open Issues #88604 and #91746 to the Edge/Gateway Known Issues section.

July 18th, 2022. Twentieth Edition

• Added Open Issues #91365, #92481, and #92676 to the Edge/Gateway Known Issues section.

July 28th, 2022. Twenty-First Edition

• Revised the Release Notes to eliminate all 5.0.0.0 references that are not strictly related to a Edge, Gateway, or Orchestrator build. The Release Notes are properly titled the VMware SASE 5.0.0 Release Notes as that is the software version covered and the fourth digit is reserved for specific builds for that software.

August 15th, 2022. Twenty-Second Edition.

• Added Fixed Issue #89217 to the Edge/Gateway Resolved Issues section. This ticket is added to the 5.0.0.0 build section as the fix is a Platform Firmware fix that can be applied on an Edge using any Release 5.0.0 build.

November 9th, 2022. Twenty-Third Edition.

• Under the SD-WAN Feature Enhancements section, the Greek Localization section is updated. The wording: "While the Orchestrator localizes into Greek automatically, there is a limitation for the documentation because the Greek language is not currently supported by the VMware Docs platform. As an interim solution, VMware SD-WAN, Cloud Web Security, and Secure Access documentation in Greek is published here in a PDF format." is removed. As of this edition, the Greek language is now supported on the VMware Docs platform and a user can view Greek versions of SD-WAN and SASE documentation at docs.vmware.com.
• Added #82104 to the Edge/Gateway Known Issues section.

November 14th, 2022. Twenty-Fourth Edition.

• Added Open Issue #100377 to the Edge/Gateway Known Issues section.
• Note: All customers are strongly recommended to upgrade their Edges to the 5.0.1.2 build to ensure their users do not encounter Issue #100377.

November 29th, 2022. Twenty-Fifth Edition.

• Added Fixed Issue #76837 to Edge/Gateway Resolved Issues for the original GA build. This ticket was omitted in error from the First Edition of the 5.0.0 Release Notes.

January 30h, 2023. Twenty-Sixth Edition.

• Previously Fixed Issue #89217 is moved to the Edge/Gateway Known Issues section and changed to an Open Issue because the Edge build needed is now a 5.0.1 version, not a 5.0.0 version. The ticket is also revised to reflect the need for a specific Edge version (R5012-20230123-GA-103475) and new Platform Firmware version (R131-20221216-GA) needed to resolve the issue. The ticket also adds a link to the KB Article that covers #89217 and which includes step-by-step instructions for upgrading a 6x0 Edge.
• In the Compatibility section, revised the Import Note regarding End of Support for 4.2.x and added Release 4.3.x to reflect newly revised dates for the SD-WAN Edge software.

February 17th, 2023. Twenty-Sixth Edition.

• Removed Issue #39659 from the Edge/Gateway Known Issues section as this is a duplicate of another ticket, #39501 which was resolved in Release 4.3.0.

March 16th, 2023. Twenty-Seventh Edition.

• Added Fixed Issue #75593 to Edge/Gateway Resolved Issues for the original GA build. This ticket was omitted in error from the First Edition of the 5.0.0 Release Notes.
• Added a new Important Note, Higher Baseline CPU Usage for Gateways. This note explains why a user would observe a higher baseline CPU usage in a Gateway using Release 5.0.0 or later as a result of updated fast path code, but which does not impact the capacity of the Gateway to handle data and process handing.

July 26th, 2023. Twenty-Eighth Edition.

• Added Fixed Issue #77608 to Edge/Gateway Resolved Issues for the original GA build. This ticket was omitted in error from the First Edition of the 5.0.0 Release Notes.
• Added Open Issue #103708 to the Edge/Gateway Known Issues section.

September 6th, 2023. Twenty-Ninth Edition.

• Added a new Important Note: Expanded NAT Range for SD-WAN Gateways. This note should have been added in the first edition of these notes and was omitted in error.

September 25th, 2023. Thirtieth Edition.

• Added two new Important Notes:
  • Edge Diagnostic Bundle Triggered on a Reboot. This note covers a new behavior introduced in this release.
  • LAN-Side NAT Behavioral Change. This note covers a new behavior for LAN-Side NAT when 1:Many rules are configured and what a user needs to do with regards to this change.

Resolved Issues

The resolved issues are grouped as follows.

• Edge/Gateway Resolved Issues
• Orchestrator Resolved Issues
• Secure Access Resolved Issues
• Cloud Web Security Resolved Issues

Edge/Gateway Resolved Issues

Resolved in Edge/Gateway Version R5002-20220506-GA

Edge/Gateway rollup version R5002-20220506-GA was released on 05-12-2022.

This Edge/Gateway rollup build addresses the below critical issues since Edge/Gateway version R5000-20220225-GA.

• Fixed Issue 74316: A VMware SD-WAN Spoke Edge may not connect to any or all of the assigned Hub Edge Clusters, even if the Edge has a service restart or a full reboot.

  There is an issue with the cluster reassignment logic which creates cluster assignment mapping without the cluster member’s endpoint information in a specific Cluster-member-to-Super-Gateway overlay flap scenario. As a result, Spoke Edges assigned to the Hub Cluster member subsequently fails to receive the endpoint information of the Hub Cluster member leading to no overlays between Spoke Edges and Hub Clusters.

  Without the fix the only way to temporarily remediate the condition is for someone with Gateway access to trigger a cluster reassignment manually on the Super Gateways.

• Fixed Issue 83209: For customers using OSPF in their enterprise, OSPF routing may not work as expected.

  The Issue occurs when there is a change in the OSPF router-id and the Edge service is restarted. Only loopback interfaces and Interfaces with 'Advertise' flag configured are considered for router-id selection. When there is a new loopback interface configured with a higher IP address, upon restarting the Edge service, the new loopback IP address is selected as the router-id and if the Edge is elected as the DR (Designated Router) the issue is seen.

  Without the fix, the only workaround is to force the use of the old Router ID. To bring back the old Router ID, configure Advertise Flag on the respective interface (an Edge service restart will be required).

• Fixed Issue 87956: For a VMware SD-WAN Edge using a single WAN interface onto which two or more user-defined WAN overlays with the same next hop are configured, if a WAN link connected to the single interface goes down and up, only one of the user-defined overlay tunnels is reestablished.

  For example, if there are user-defined overlays with different source IP addresses but the same next hop which steers traffic to a Hub Edge and a Gateway respectively, on a WAN link flap, only the tunnel to the Gateway is reestablished and the tunnel to the Hub Edge is not, resulting in an impact to customer traffic intended for the Hub Edge.

  By design the Edge does not support having multiple user-defined WAN overlays with the same next hop but the Edge does not perform a check on the configuration and instead treats it as valid. Only when the WAN link flaps does the Edge do a check and enforce a single user-defined WAN overlay to the exclusion of other WAN overlays. This is why the configuration "works" for the Edge when it is applied or when the Edge is restarted and the configuration is reapplied. The fix for this issue allows multiple user-defined WAN overlays for the same interface and thus all the tunnels will be reestablished after a WAN link flap or any other circumstance that could tear down the overlay tunnels.

  Without the fix, the only way to restore all the tunnels is to restart the Edge service, which can be done on the Orchestrator using Test & Troubleshoot > Remote Actions > Restart Service.

Edge/Gateway Resolved Issues

Resolved in Edge/Gateway Version R5000-20220225-GA

The below issues have been resolved since Edge version R450-20220203-GA, and Gateway version R450-20220203-GA.

• Fixed Issue 34234: A WAN link on a VMware SD-WAN Edge may show an incorrect bandwidth capacity value on the Orchestrator UI and the customer may observe the WAN link not being utilized to its actual bandwidth capacity.

  With this issue the WAN link is not being remeasured for the most current bandwidth capacity value with the specified frequency. This is caused by the service resetting the date on the bandwidth value cache every time there is a tunnel tear down/bring up (i.e. flap) on the WAN link. Because the cached bandwidth value is reset the SD-WAN service is fooled into thinking this is a new value and no additional bandwidth testing is needed. In a WAN link where there are frequent tunnel flaps, this WAN link bandwidth value could be quite old and not at all representative of the current WAN link bandwidth values and this will cause customer traffic issues because the SD-WAN service will not use the WAN link to its real capacity.

• Fixed Issue 35807: A DPDK routed interface will be deactivated completely if the interface is first deactivated and later reactivated from the VMware SASE Orchestrator.

  Deactivating a routed interface unbinds that network device from the DPDK controlled hardware and rebinds it back to the Linux driver and the Edge service restarted. The /opt/vc/etc/dpdk.json file is updated to remove this interface, so on reactivation, the file is not updated to unbind from Linux back to the DPDK controlled driver.

  Without the fix, the only way to remediate the issue is to reboot the Edge to clear the state and back to a default boot state to re-add devices to the edge DPDK layer

• Fixed Issue 42488: On a VMware SD-WAN Edge where VRRP is activated for either a switched or routed port, if the cable is disconnected from the port and the Edge Service is restarted, the LAN connected routes are advertised.

  If the link on a port is removed and the interface is not deactivated, the Edge does not revoke the route from the Gateway causing other Edges to forward the traffic to the Edge with no link connected. The customer impact is that traffic might blackhole for the connected route for interfaces which do not have a link connected.

  Without the fix the only workaround is to deactivate the interface if no link is connected

• Fixed Issue 53378: On a VMware SD-WAN Edge where a WAN link bandwidth is manually configured under WAN Settings, the bandwidth settings are honored on traffic using the Global Segment, but not honored on traffic using a Non-Global Segment.

  On a WAN link with a higher capacity than the manually configured capacity in WAN Settings, the Global Segment would enforce the lower configured value but a non-Global Segment would use the actual capacity of the link.  This occurs despite Underlay Accounting being configured on the Edge interface that the WAN link is using.

• Fixed Issue 53951: A VMware SD-WAN Edge may experience either a failure of traffic sent direct to the internet or a loss of connectivity to the VMware SD-WAN Orchestrator and the Edge is marked as down.

  This issue can affect an Edge in one of two scenarios:

  1. For an Edge which uses public WAN links, when there is a flap (link goes down and then comes up) on a WAN link, the impact to the customer in this scenario is that traffic that is steered to the affected link and is also classified as Direct is dropped. This issue is especially impactful for a site where Business Policy rules are configured to force certain traffic to use one WAN link only while also being sent Direct.
  2. When turning on HA on an Edge using PPPoE WAN links, there is a change in the PPPoE interface IP and the old self route is deleted but with the new PPPoE IP address the new self route is not getting added. As a result the communication between the Orchestrator and the Edge no longer works.

  Without the fix, the way to temporarily correct the issue is to either restart the Edge service to ensure Direct traffic is sent on the affected public WAN link, or reboot the Edge (where PPPoE links are used) which recovers the route to the Orchestrator.

• Fixed Issue 63629: In a network topology where the VMware SD-WAN Hub Edge and Spoke Edge have different IP family preferences (in other words, IPv4/IPv6 dual stack), the customer can see more bandwidth allocated to the peer than expected.

  If both IPv4 and IPv6 families are configured, the Edge internally creates two different link objects. The bandwidth values are added for both of them when it should be added only for one.

  Without the fix, the workaround for this issue is to not have different tunnel preferences if Hub/Spoke topology have dual stack configured.

• Fixed Issue 64627: A VMware SD-WAN Gateway may experience a Dataplane Service restart with a brief disruption in traffic.

  When there are a large number of subpaths configured on WAN links for a VMware SD-WAN Edge or there are frequent flaps of of the Edge's management tunnels with its connected Gateway, this may lead to exhaustion of the Gateway's memory counters which triggers a restart of the Gateway to recover.

• Fixed Issue 65964: When looking at the details for an Alert, the time section is incorrect on the VMware SASE Orchestrator.

  The details of Alerts are not sent out within the notification delay are similar to the details of an Alert that is sent out. The notification time is whatever the current time is at the time the details are viewed. On the New UI, the notification time is listed as "Pending".

• Fixed Issue 68044: A VMWare SD-WAN Edge using a VNF may experience a Dataplane Service failure and restart.

  To monitor whether the traffic passes via the VNF, the Edge process sends periodic Gratuitous ARPs (GARP). There is the potential for memory corruption when the timer for GARPs runs at the same time the Edge updates the VNF configuration.

• Fixed Issue 68482: VMware SD-WAN Hardware Edge running Release 4.5.0 may not successfully update its WAN link bandwidth values after a service restart.

  When a VMware SD-WAN Edge's service restarts the Edge is supposed to test bandwidth and compare against the cached values. With this issue the test results are not applied and the cached value continues to be used.

• Fixed Issue 68923: On a customer enterprise using BGP, a default route may be redistributed to a BGP peer though the reachable status for the installed default route is set to 'False'.

  If a static route is configured on an Edge pointing to any Edge interface and that BGP peer learns the default route from the Edge and that interface is later deactivated (which changes the reachable flag for that route to False), the route continues to be advertised. It is equally true that a route that is not being redistributed because the interface was down, but then when the interface comes up and marks the route status as 'True', the route would continue to not be redistributed. The cause in both instances is the Edge not readvertising the route on an interface status change that reflects the new route status.

• Fixed Issue 70118: When looking at the route_events_msg_dump log from a diagnostic bundle, the user logs do not include the netmask. 

  For some customer issues it is useful to include the netmask when examining routing events but is missing in this instance.

• Fixed Issue 71745: A VMware SD-WAN Gateway or SD-WAN Edge may experience a Dataplane Service failure and restart as a result.

  Both Edges and Gateways have an an internal library for managing UUIDs (universally unique identifiers). A very rare race condition in this library can cause a "use-after-free" issue that triggers a segmentation fault and a service failure for the respective Edge or Gateway. A factor that increases the risk of an Edge or Gateway experiencing this issue are frequent tunnel flaps (tunnels being torn down and rebuilt).

• Fixed Issue 71785: SNMP walk may not work properly on an Edge running release 4.3.0 or later.

  When an Edge is upgraded to Release 4.3.0 some IP's that SNMP uses are blocked and the Firewall settings have to be changed to allow traffic on Port 161.

• Fixed Issue 71788: For a customer deployed with a High-Availability topology, there is a potential for the site to experience an unexpected HA failover that includes a disruption of customer of traffic.

  This is an inconsistent issue that is not readily recreated but where encountered, the HA Edge's periodic handler thread runs more than 180ms during flow synchronization between the Active and Standby Edge and this causes a deadlock and triggers a SIGKILL message on the Active Edge's mutex mon thread resulting in a core and an Edge reboot.

• Fixed Issue 72270: For Edges deployed in a High-Availability, software upgrades that also include a firmware upgrade may cause the HA Edges to be both unexpectedly upgrade and reboot together and trigger customer traffic disruption and OSPF or BFD route learning.

  This is true of Edge 3x00 models that upgrade to a build that includes a CPLD firmware upgrade. By design the Standby Edge upgrades first and then fails over to Active so that Active can then upgrade, but the Active is either waiting for the failover or in the absence of a failover observes a five minutes delay before upgrading also. The Standby Edge is also upgrading it's firmware including the OS kernel this may take longer than five minutes and would create a scenario where both the Standby and Active Edge are upgrading and rebooting simultaneously and causing disruption to customer traffic and forcing OSPF or BFD routes to be relearned which itself is disruptive. The fix here simply extends the Standby's timer to ensure that only one Edge is upgrading at a time.

• Fixed Issue 73320: When a VMware SD-WAN Edge interface is configured with a DHCP address type and when the IP address assigned to the interface is updated, some of the direct flows can fail due to NAT failure.

  When the DHCP lease expires, the IP address is removed for the interface. Before a new IP is assigned to the interface, NAT entries for any new flows during this transient time will be created with "0.0.0.0" as the source NAT IP and the packets will be dropped. Once a valid IP address is assigned to the interface, the existing NAT entry is not deleted and a new NAT entry is not created with a valid IP causing the traffic to be dropped.

• Fixed Issue 73916: For a customer enterprise using a public DNS (for example, Google.com), DNS lookups of unqualified or partially qualified host names incorrectly resolve to the address of "app.nyansa.com".

  Under the DHCP settings of a VLAN or routed interface, if an explicit DNS Search Path (option 119) has not been defined for an Edge, when the Edge or one of its clients tries to resolve an unqualified (for example: hostname) or partially qualified (for example: hostname.corp) DNS name, the Edge incorrectly tries to look up that name under .nyansa.com, which will always return the address app.nyansa.com, which is incorrect for this case. The nyansa hostname is associated with VMware SASE's Edge Network Intelligence application.

  On an Edge without a fix for this issue, a user needs to define an explicit DNS Search Path (option 119) for each enabled DNS service (on a VLAN or a routed interface) on the Edge.

• Fixed Issue 73933: A VMware SD-WAN Edge model 500 with a Release 1.8.x factory image cannot be activated to Release 4.5.0 or later.

  An Edge 500 that has been hard reset to its factory image (where that image is Release 1.8.x) cannot be activated in a scenario where it is assigned a 4.5.x release. 4.5.0+ builds changed the format of the upgrade bundle to use sha256sums instead of sha1sums. As older builds do not have these libraries, the upgrade to 4.5.x fails.

  Without the fix the workaround is to activate the Edge 500 to a more recent release like 4.2.x or 4.3.x and then proceed with an extra step to upgrade the Edge to the 4.5.x build once the Edge is using the more recent build.  The Edge 500 is also EOL so the other workaround is to RMA the Edge for the latest equivalent Edge model.

• Fixed Issue 74149: For a customer using a Zscaler type Cloud Security Service where the L7 Health Check is configured, if a VMware SD-WAN Edge is rebooted while a WAN link is also down, the L7 Health Check process may not send probes to the Zscaler service even after both the Edge and the WAN link(s) are fully restored.

  This issue is not consistent and happens rarely even when the listed conditions are met. When the Edge is being rebooted, and L7 Health check is configured, and if the Edge WAN interface undergoes a state transition Up/Down, during restart and initialization time, the Edge may miss sending L7 Probes.

  Without the fix, the only way to get the Edge to resume sending L7 Probes is to turn off and then turn back on L7 Health Check.

• Fixed Issue 74225: A VMware SD-WAN Gateway or SD-WAN Edge may experience traffic issues and observe packets with zero MAC addresses in logs.

  Even if a Gateway or Edge runs a build that includes the fix for issue #62552, which also addressed a Gateway or Edge sending out packets with a 00:00:00:00 MAC address, there were locations within the Gateway/Edge dataplane process that could still send packets with a source and/or destination MAC address of zero. In these locations the ARP state machine was not validating the source and destination MAC Address. 

  The only way to reliably know this issue is hitting is to check logs for zero MAC addresses, specifically ones using tcpdump.

• Fixed Issue 74306: A user making a Skype call behind a VMware SD-WAN Edge may experience poorer than expected call quality.

  By default, the VMware SD-WAN service classifies all Skype and MS Teams packets, including call packets, as APP_CLASS_INTERNET_INSTANT_MESSAGING (10) class. Instant Messaging class has a lower priority and thus call quality could be reduced by being deprioritized for bandwidth and link quality.  The fix changes packets matching Skype and MS Teams to APP_CLASS_BUSINESS_COLLABORATION (5) which means calls would receive the expected High Priority/Real Time quality level treatment.

  Without this fix, the workaround is to customize the application map so that Skype and MS Teams were classified as Business Collaboration.

• Fixed Issue 74632: On a customer enterprise with a Hub/Spoke topology, IPv6 tunnels from a VMware SD-WAN Spoke Edge to a Hub Edge may not come up on Edge interfaces configured as IPv6 DHCP stateful.

  When an interface is configured with DHCP stateful, tunnels may not come up immediately if the router advertisement is received late.

• Fixed Issue 75121: A customer may observe an unreachable route for a backhaul flow which leads to packet drops.

  There was an issue in the backhaul and interface flow route lookup code which was picking the first route even if it was unreachable. Since this unreachable route is not good to carry the packet, the packets were getting dropped. The resolution mandates a check on route reachability for all types of flows.

• Fixed Issue 75593: Customer deployments using BGP may experience issues with degraded performance because of suboptimal routing due to unexpected route preferences for uplink BGP routes.

  This issue is caused by a customer enterprise's BGP prefix advertise and preference values not being updated properly when the route prefix is updated from a non-uplink to an uplink or vice-versa, which results in asymmetric routing.

  A customer without a fix for this issue can disable and enable BGP uplink community to restore correct routing.

• Fixed Issue 75772: For a customer using Edge Network Intelligence where an Edge has Analytics active, the Edge may experience a memory leak that results in the Edge restarting to clear the memory leak.

  Where Analytics is configured and DHCP is also configured on an Edge interface, the client connectivity events cause the memory usage to increase. Over a sufficient period of time the memory usage may reach a critical threshold and the Edge would defensively restart the Edge service to restore normal memory levels.  As with any memory leak issue, the smaller the initial memory footprint (for example, an Edge 510, 520, or 610) the more vulnerable the Edge is to having an Edge restart occur.

• Fixed Issue 75827: A VMware SD-WAN Gateway may experience a Dataplane Service failure and restart as a result.

  In the logs, a user would observe a Gateway process failure with de2e_info_reply_msg_recieved. The root cause is a NULL pointer exception that is not handled in the Gateway process which triggers an exception and causes the service failure and restart.

• Fixed Issue 75882: For a customer site configured with a High Availability topology, the Standby Edge may remain stuck in an Initializing state and not be available for failover.

  SD-WAN sends WAN side heartbeats when HA is enabled. The Standby Edge can experience a Dataplane Service failure when SD-WAN sends the heartbeat on the Standby Edge's interface before the interface's internal variables are initialized. The fix delays the sending of WAN side heartbeats until the interface is properly initialized.

• Fixed Issue 76315: An Operator User may observe a VMware SD-WAN Gateway dropping the first few packets of every flow the Gateway sends out.

  The drop reason will be listed as gwrouting_vpn_unreachable. The issue is the result of a processing delay of the management protocol's QoS (Quality of Service) synchronization messages which causes the first few packets of every flow to be incorrectly classified and dropped.

• Fixed Issue 76837: A customer using BGP may observe that a peer router is not sending traffic to a VMware SD-WAN Edge within its network.

  Troubleshooting the issue would reveal that the default route via default-originate is not being advertised by the Edge. The issue is caused by a route map string associated with the default route being truncated and so the Edge does not match the default route with anything in its route map, and this results in the peer router either dropping traffic or sending it using an invalid route where the traffic is blackholed.

  On an Edge without a fix for this issue, a user would need to configure a static route on the peer router for the default route until it is possible to upgrade to an Edge version that includes the fix.

• Fixed Issue 76966: When creating very large network configurations with more than 100 segments or VLANs, the DNS and DHCP services on the VMware SD-WAN Edge stop working.

  When such large configurations are sent to the Edge, the scripts that run on the Edge to configure and start the "dnsmasq" service (for DNS and DHCP) fail because of an overly long command line that is synthesized during the restart.

  There is no workaround to this issue beyond reducing the number of segments or VLANs.

• Fixed Issue 77224: If an invalid static route (for example, 2.2.2.2/0) is configured on a VMware SASE Orchestrator for a VMware SD-WAN Edge, several connected Edges may have an invalid or stale route.

  When there is genuine default route (0.0.0.0/0) and also an invalid route with prefix length 0 (e.g. 2.2.2.2/0). The invalid route causes the default route to not get deleted from remote Edges though it was deleted from the originator Edge.

  This is connected to Orchestrator ticket #77159 which allows invalid routes with 0 length prefix's to be configured.

• Fixed Issue 77357: A VMware SD-WAN Edge 3400 or 3800 that is deployed in Japan may lock up and reboot spontaneously.

  The Edge 3400 and 3800 have an incorrect voltage warning threshold (100V) set in the system's Baseboard Management Controller (BMC), which happens to match the 100V electrical supply in Japan. The result for an Edge 3400 or 3800 in this region is a continuous series of power supply alarms and if the volume of alarms is sufficiently frequent, the Edge will lock up and reboot.

  Note: This is a follow up to Edge issue 51291 which while the fix was successful in resolving the issue, the fix was not lasting because the manually set voltage thresholds were getting cleared out by a CPLD upgrade.  Otherwise the symptom and description are the same but the fix here ensures the voltage thresholds persist over any subsequent Edge upgrade.

• Fixed Issue 77525: For a site using a High-Availability topology, when the VMware SD-WAN HA Edges are upgraded to a new software image, the Standby may fail to upgrade and the VMware SD-WAN Orchestrator UI incorrectly lists the Standby Edge's status as 'Active' even though it is not.

  When the Active Edge detects the Standby Edge it tries to fetch the Standby Edge's software version and if the version is greater than 3.4.x then the Active Edge copies the network configuration file to the Standby Edge. While fetching the Standby Edge software version, there may be an exception which is not handled by the Edge's HA code and this leads to an HA worker thread getting struck and further communication with the Standby Edge fails. At this point the management process between the Active and Standby Edge is broken and anything have to do with the management plane, including software management, Standby Edge status, and configuration changes, will not be synchronized between the Active and Standby Edge. This results in the Standby Edge being falsely detected as Active and which appears as an Active/Active "split brain" state on the Orchestrator but is not as the Standby Edge is still performing its proper role.

  If there is an HA failover and the Standby Edge is promoted to Active, the Edge would be with a mismatched set of configurations and software. The Orchestrator would detect the configuration mismatch and push the updated configuration to this Edge while also completing the software upgrade the Standby Edge previously missed. And since an Edge software upgrade requires a reboot, the customer would observe another failover while the newly Active Edge was rebooted and then demoted back to Standby status. 

  This issue is not consistently encountered when an HA site upgrades the Edges' software. In addition, this issue can also happen when bringing up a new HA site, or when a standalone site is brought up to High-Availability, anytime the Standby Edge has to upgrade its software. But these secondary scenarios are more rare when compared with HA Edges undergo a software upgrade

  Without the fix, a customer observing this issue would need to restart the Edge service or trigger an HA failover to clear the issue.

• Fixed Issue 77608: For a customer enterprise with a Hub/Spoke topology, a VMware SD-WAN Edge may run out of memory and defensively restart to clear the state.

  When a huge amount of IPv6 traffic (~1M flows) originating over a /64 subnet combined with neighbor discovery failures rabidly consumes the Edge's memory and triggers the restart.

• Fixed Issue 77625: On a site deployed with a High Availability topology, a user may observe the VMware SD-WAN Standby Edge rebooting multiple times.

  The site goes into a Active/Active (Split-Brain) state due to HA threads being starved while processing the HA heartbeat packet. In an Active-Active state the tie-break goes to the Active Edge and the Standby Edge is rebooted to demote it back to its proper Standby status. In this case though, the Active/Active event is detected multiple times with Standby Edge reboots each time to recover the site. 

  Field issues have involved Edge 6x0 (610, 620, 640, 680) models but the issue is platform agnostic and could occur on other Edge models.

• Fixed Issue 77629:On a site deployed with a High Availability topology, if the user turns HA off, both the Standby and Active Edges may become deactivated.

  This issue can have a serious impact on a customer site since both Edges are deactivated and no customer traffic could pass until one of the Edges was reactivated. The expected behavior when HA is turned off for a site is for the Standby Edge to deactivate and the Active Edge to become the standalone Edge for that site. In some cases when HA is turned off, while the Standby Edge is applying the 'HA off' configuration and prior to deactivating, it continues to send heartbeats to the Orchestrator and the Orchestrator erroneously updates the HA Edge serial number. During the next cycle the Active sends the Orchestrator its heartbeat which the Orchestrator compares to the newly updated serial number and detects a serial number mismatch and initiates a deactivation of the Active Edge as well.

  This is a timing issue and is not consistent but without the fix, the best practice is to turn off HA only in a maintenance window to be safe.

• Fixed Issue 77732: On a site deployed with an Enhanced High Availability topology with a dual transport layer (Internet + MPLS) the public IP address is not shown for the link connected on the subinterface of the Standby Edge.

  For the IPv4 management tunnel the source IP is set to 0.0.0.0 which is caused by the wrong HA interface FSM (Finite State Machine) action. Specifically, when the HA wait_for_peer timer expires, it tries to apply interface synchronization information to determine if there are any IP Refresh events coming from the Standby Edge and it refreshes the IP address even if there is no change in the IP/NextHop and this causes the issue. 

• Fixed Issue 77755: On a VMware SD-WAN Edge running Release 4.0.0 and later, if a customer deploys a VNF (Virtual Network Functions) image where the checksum is configured with capital letters, the VNF deployment fails due to a checksum mismatch and the image will be removed from the Edge.

  This issue is caused by 4.0.0 and later Edge software performing a case sensitive comparison of the Operator-configured checksum with the Edge calculated checksum. A configured checksum containing capital letters causes a checksum mismatch, even if the calculated checksum values match.

  Release 5.0.0 and later uses a case insensitive comparison to verify an Operator-configured checksum matches the calculated checksum on the Edge.

• Fixed Issue 78003: For a customer using a Hub/Spoke topology, static tunnels from the VMware SD-WAN Spoke Edge to a Hub Edge might not form.

  Typically if there are lots of Dynamic Edge-to-Edge tunnels already established on the Spoke Edge, the maximum tunnel number check is hit on the Spoke for the static tunnel. This check prevents static tunnel formation from the Spoke to the Hub.

• Fixed Issue 78300: If a VMware SD-WAN Edge is using a WAN link configured to be a backup, a user may observe logs or Orchestrator Events which suggest that tunnels are coming up or going down for this link.

  By design, tunnels do not get established for backup links. But any tunnel request from a remote end (typically a dynamic Edge-to-Edge tunnel) might change the link status as it goes through the stack. In this fix, care have been taken so that no logs indicate that any tunnel formation or tear down is going on for the back up link.

• Fixed Issue 78568: For a customer using BGP and connected to VMware SD-WAN Partner Gateways, a Partner Gateway may continue to advertise a VMware SD-WAN Edge's VLAN subnet after that subnet's advertise flag is set to False.

  The routes continue to be advertised because when the Edge breaks the BGP neighbor adjacency state with an L3 BGP neighbor, one of the connected Partner Gateways maintains ownership of the Edge VLAN subnet. Stale routes on a Partner Gateway negatively affect customer traffic and can lead to entire customer flows getting dropped because traffic is routed to now non-existent routes.

  Without a fix, the only way to remediate the issue and clear the stale BGP routes is for a Partner or Operator to restart the Partner Gateway service in a suitable maintenance window.

• Fixed Issue 79132: For a VMware SD-WAN Edge configured as a Spoke in a Hub/Spoke topology, a public link displays the wrong download bandwidth capacity when looking at Monitor > Edge on the VMware SASE Orchestrator UI.

  For the Spoke Edge's public links, the link statistics are loaded with the value measured against the Spoke's connected Hub Edge instead of the Primary Gateway, which is then uploaded to the Orchestrator and displayed for the download bandwidth.

  This issue is triggered when the Spoke Edge's Primary Gateway is restarted after The Spoke Edge establishes Spoke/Hub tunnels. So the only way to avoid this issue without the fix is to ensure the Gateway is not restarted post-tunnel establishment with the Spoke and Hub Edges.

• Fixed Issue 79335: IPv6 traffic may not pass through a VMware SD-WAN Edge.

  The interface hop limit value is set to 0 when a RA (router advertisement) is received with a 0 hop limit, and this can cause IPv6 drops.

  On an Edge without a fix for this, the user can advertise RA with a non-zero hop limit.

• Fixed Issue 80551: On VMware SD-WAN Edge models which include an internal LTE modem (Edge 510-LTE and Edge 610-LTE), LTE Tunnels via an IPV6 link become UNSTABLE when the IPv6 address on the CELL interface changes.

  Whenever the IPv6 address on a CELL interface changes (for example, due to a DHCP lease expiration), the IPv6 tunnels become UNSTABLE. This is because the tunnels continue to use the old IPv6 address rather than the new one.
  
  Without the fix the only way to remediate the issue is to restart the Edge's service.

• Fixed Issue 80654: For a site configured with an Enhanced High Availability topology, a user may observe intermittent traffic drops on the VMware SD-WAN Standby Edge's WAN links.

  When there are frequent path flaps (paths being added and deleted frequently), under certain timing scenarios the TCP connection between the Active and Standby Edges is reset, leading to packet drops for traffic traversing a WAN link on the Standby Edge.

• Fixed Issue 81196: User cannot login to the Local UI of a deactivated VMware SD-WAN Edge with the factory default password.

  A user can "Deactivate" an Edge either through the Local UI using Reset Setting > Reset Configuration, or on the VMware SASE Orchestrator by selecting Remote Actions > Deactivate for the Edge.  After the user "Deactivates" the Edge, all configurations should be cleared and the login credentials of the local UI should revert to the default values, but they do not. The credentials remain the same as before the deactivation and if a user had changed the Local UI default credentials to some other value, that new value persists past the Edge Deactivation. If the local UI credentials were never updated, then the user would not any issue as the credentials before and after deactivation would remain as the defaults.

• Fixed Issue 81224: On a site deployed with a High Availability topology, when the site experiences an HA failover, the OSPF route tags may not propagate post-HA failover.

  On an HA failover, OSPF external LSA's (link state advertisements) do not have a route tag, which leads to improper routing with an adverse affect on customer traffic.

• Fixed Issue 81517: On a site deployed with an Enhanced High Availability topology which uses VMware SD-WAN Edge models 6x0's, the HA link state is not updating properly.

  The HA link is the link that connects the Enhanced HA Edge pair and if this link does not update properly the site may have issues.

Orchestrator Resolved Issues

Resolved in Orchestrator Version R5002-20220517-GA

Orchestrator rollup version R5002-20220517-GA was released on 05-17-2022.

This Orchestrator rollup build addresses the below critical issues since Orchestrator rollup version R5001-20220317-GA.

• Fixed Issue 81960: For a customer using the VMware Cloud Web Security service, when configuring the DLP Rules or the Web Rules, the VMware SASE Orchestrator UI shows the number of items selected, but does not show whether all items in a category are selected at a single glance.

  The UI should display not only the total number of rules selected, but should also display "All Types Selected" where all types can be DLP Rule Applications, DLP Rule Document Types, DLP Rule File Types, DLP Rule Categories, DLP Dictionaries, Web Rule Categories, and Web Rule Threats. This way the user can know at a glance whether all rules of a specified type are selected.

• Fixed Issue 84600: When a VMware SASE Orchestrator using a Disaster Recovery (DR) topology is upgraded to Release 4.5.0 or 5.0.0.1, the Operator may observe that the Standby Orchestrator's Cloud Web Security and Secure Access services are constantly restarting.

  This has no customer impact as it only occurs on the Standby Orchestrator and does not impact any services, but an Operator would notice logging for the Standby Orchestrator regarding the Cloud Web Security and Secure Access (listed as 'ztnad') services restarting every few seconds. The issue is caused by each service making an API call to the Standby Orchestrator that fails because that Orchestrator cannot serve the requests.

• Fixed Issue 84969: When a VMware SD-WAN Edge running a 4.2.x Release which is also configured with an overridden non-default Management IP is upgraded to Release 4.3.x or higher on a VMware SD-WAN Orchestrator running 4.3.x or higher, the Edge may lose the configured overridden Management IP.

  An Orchestrator running 4.3.x or higher is not automatically creating the loopback interface while also retaining the overridden non-default Management IP for an Edge, when that Edge is upgraded from 4.2.x to a 4.3.x or later build.

• Fixed Issue 85291: When a VMware SASE Orchestrator using a Disaster Recovery (DR) topology is upgraded to Orchestrator Release build 5.0.0.0 or 5.0.0.1, DR may fail.

  In Operator Events the Operator would observe events with name "DR Configuration Failure" and a message "Warning: active unable to get standby DR status...accessing the cert file is failed to initiate a secure API call..." The DR failure is caused by an access rights issue while reading the root of trust for a self-signed certificate.

• Fixed Issue 85338: When a VMware SASE Orchestrator using a Disaster Recovery (DR) topology is upgraded to Orchestrator Release build 5.0.0.0 or 5.0.0.1, Management Settings are removed from Operator Profiles.

  The Management Settings removed include the Orchestrator Address in all three forms: FQDN Address; Orchestrator IPv4 Address, and/or Orchestrator IPv6 Address, where applicable. Without the fix an Operator needs repopulate the appropriate Orchestrator Address fields or the Operator Profile will not be valid.

• Fixed Issue 85412: When a VMware SASE Orchestrator using a Disaster Recovery (DR) topology is upgraded to Orchestrator Release build 5.0.0.0 or 5.0.0.1, DR may fail with an error at the Copy DB stage.

  As part of the replication process the Standby Orchestrator database is being repurposed and due to an incorrect MySQL statement, certain tables fail to be created on the Standby's database, preventing the DR process from proceeding any further.

• Fixed Issue 86083: For a customer using the VMware Cloud Web Security service, the Cloud Web Security Event Log is missing important details on the VMware SASE Orchestrator UI.

  The Event log does not include the following details:

  • For configuration changes, the username is included in the event but not what change was made.
  • When a Cloud Web Security rule is deleted, the Orchestrator does not include the deleted data with the event.
  • Activating or deactivating Authentication does not include a timestamp for the event.
  • For CASB events, only the application ID is included, not the application name.
• Fixed Issue 86573: When a user runs a Remote Diagnostic for a VMware SD-WAN Edge using a 3.4.x Release on a VMware SASE Orchestrator using Release 5.0.0.1, the diagnostic times out and the Orchestrator UI does not display a result.

  When the 3.4.x Edge uses the live mode of communication and does not support RealtimeConnection (which is introduced in Edge versions 4.x), and the Orchestrator version is 5.x, the Remote Diagnostics UI page stalls and does not return a result.

• Fixed Issue 88796: When deploying either a VMware SASE Orchestrator or a VMware SD-WAN Gateway and using an OVA on vSphere, the OVF properties set as part of the deployment (password, network information, etc.) are not applied to the image and the system cannot be accessed after deployment.

  This only affects new system deployed from OVA using OVF/vApp properties (versus using ISO files). This issue is caused by upstream changes to cloud-init in recent updates.

  Without the fix, the workaround is for the Operator to deploy the system using a cloud-init user-data ISO file.

  Note: While this issue affects both Gateway and Orchestrator builds, the fix here applies only to the Orchestrator build. The issue that affects Gateway builds is tracked as a Known Issue ticket of the same # (88796).

• Fixed Issue 89005: For customers using the VMware Cloud Web Security service, a user cannot activate or deactivate an existing SAML authentication on the VMware SASE Orchestrator.

  When a user navigates to Cloud Web Security > Configure > Authentication > Enable SAML Authentication and enters valid information, after clicking Save Changes, the configuration fails and the error message reads "adminSaml" is not allowed. 

___________________________________________________________________

Resolved in Orchestrator Version R5001-20220317-GA

Orchestrator version R5001-20220317-GA was released on 03/18/2022. This Orchestrator build addresses several critical issues since Orchestrator version R5000-20220225-GA.

• Fixed Issue 69573: For customers using VMware Secure Access, when creating a Remote Access service, if validation fails on the Enterprise & Network Settings Screen, the Next button is grayed out as expected but the error messages are not displayed.

  When creating a Remote Access service if the user enters invalid data in the Customer Subnet or Subnet bits fields, no error message is displayed below these fields to help the user understand why the configuration failed.

• Fixed Issue 76994: For a customer using the VMware Cloud Web Security service, when a customer using the Cloud Web Security has that service removed on the VMware SASE Orchestrator by an Operator and then later restores the service to the customer, users will observe that the UI for Cloud Web Security is unusable and observe multiple errors.

  A user logging into the Cloud Web Security section of the Orchestrator would see multiple error when trying to configure a security policy, while the service itself would work, the customer would not be able to modify existing policies.

• Fixed Issue 78688: A VMware SASE Orchestrator which hosts customers using a Zscaler Cloud Security Service (CSS) may experience a CPU usage spike which leads to high latency in process requests and the Orchestrator not updating Edge health statistics.  

  The Orchestrator Edge Health Statistics processing has a database lookup that is not optimized and this increases the CPU utilization by the Orchestrator.

• Fixed Issue 83538: For customers using the Secure Access service, when creating a Remote Access service, the Enterprise & Network Settings Screen shows internal error message keys.

  When creating a Remote Access service, if the user enters invalid data in the customer subnet or subnet bits fields, an untranslated error message is displayed below these fields. This error message is of no use to the user and does not point to resolving the actual issue regarding invalid data in either field.

• Fixed Issue 83550: For customers using the Cloud Web Security service who have configured security policies with the Data Loss Prevention (DLP) feature, when monitoring DLP activity on the VMware SASE Orchestrator UI, the user will be unable to see the 'Block Count by Date' screen.

  The Cloud Web Security > Monitor > DLP screen will also show a message banner "error while getting dlp top block count by date - Backend Error". Both Threat Origins and Block Count by User screens will display properly on this page.

• Fixed Issue 83582: When upgrading a VMware SASE Orchestrator from Release 4.5.0 to Release 5.0.0, the process takes much longer than expected and until the process completes all Orchestrator services are unavailable.

  The schema update can take more than 15 minutes for the Edge Statistics table to update during the upgrade when the LRQ schema should be updated instead and this is causing a major delay in the Orchestrator update completing.

• Fixed Issue 83902: For a VMware SASE Orchestrator deployed with a Disaster Recovery (DR) topology, when the Orchestrator is upgraded to Release 5.0.0, DR synchronization between the Active and Standby Orchestrators may fail.

  An Operator user would observe a "Failure Copying DB" error message during the Copy DB phase of the DR synchronization on the Orchestrator UI.  In the Orchestrator logs, an Operator would see this entry "Error running mysql schema copy: ERROR 1049 (42000) at line 4116: Unknown database 'velocloud_ztnad'".

• Fixed Issue 83940: For customers using VMware Cloud Web Security, a customer with a Standard License is able to see pages for both Data Loss Prevention (DLP) and CASB Application Control pages on the VMware SASE Orchestrator UI.

  A customer with a Standard Cloud Web Security license does not have access to either DLP or CASB Application Control features and the UI should not display pages for those features. This issue is caused by a missing route configurations in the Cloud Web Security UI.

• Fixed Issue 84286: For customers using the VMware Cloud Web Security service, a user with read-only privileges does not see logs on the Cloud Web Security > Events page until the user selects a time interval.

  The Cloud Web Security > Events page for a read-only user shows as blank, implying there are no logs to display. But if the user selects a new time interval, then the correct logs for that time interval will display.

• Fixed Issue 84297: For customer using VMware Cloud Web Security, read-only users are able to edit Cloud Web Security configurations for the Content Inspection Engine and Authentication pages.

  The VMware SASE Orchestrator is not properly enforcing the read-only user role for the selected Cloud Web Security configuration pages. 

• Fixed Issue 84299: For customers using VMware Cloud Web Security, customer administrators with either a Security Admin or Security Read role are unable to open the Cloud Web Security > Monitor page on the VMware SASE Orchestrator UI.

  The customer administrators with Security roles (Admin and Read) are not provided with the necessary privileges to view the Cloud Web Security's Monitor page by the Orchestrator.

• Fixed Issue 84300: For customers using the VMware Cloud Web Security service, a customer administrator with any read-only role can remove auditor email addresses from the Configure > DLP page.

  Customer administrator roles with read-only status for Cloud Web Security are: Customer Support, Security Read, and Network Admin. Any of these administrators can go to Cloud Web Security > Configure > DLP Settings and under the Auditors tab could delete the configured email addresses of the auditors who are supposed to get alert emails for DLP.

___________________________________________________________________

Resolved in Version R5000-20220225-GA

The below issues have been resolved since Orchestrator version R450-20220203-GA. 

• Fixed Issue 52301: The filter for the "Gateway Type" column in the Customer Usage grid for an activated Gateway does not function correctly on the VMware SASE Orchestrator.

  Located on the Overview page of an activated Gateway, the user is unable to filter effectively by Gateway Type.

• Fixed Issue 54015: User is able to configure an ICMP probe name with special characters and scripts on the VMware SASE Orchestrator.

  On configuring an ICMP Probe with script '<script>alert('test');</script>' and 'test<script>alert('test');</script>' as input for ICMP probe name it shows error as “An unsafe character “<” and “>” was found in a field while saving".  What should be returned is: {"error":{"code":-32603,"message":"script injection error"}}.  

• Fixed Issue 61182: On the VMware SASE Orchestrator New UI, when BGP state is changed to "Removed" for either the Edge or Gateway, the Orchestrator is not displaying the neighbor state.

  This is a display issue only but is confusing for users who are not getting a correct status for a BGP state change to "Removed".

• Fixed Issue 62456: On the VMware SASE Orchestrator's New UI, a user can activate a Cloud Security Service (CSS) without selecting a service.

  The user can configure a CSS without a service and save without an error on the New UI.  The Classic UI works as expected and throws an error. The issue can occur for a VMware SD-WAN Edge where CSS configuration is activated but no service is selected and then later the Edge will automatically deactivate the CSS configuration with no user notification or Event.

• Fixed Issue 62459: On the VMware SASE Orchestrator's New UI, when configuring a Cloud Security Service (CSS) with Zscaler type, the L7 Health Check option is missing.

  The L7 Health Check option shows on the Classic UI, but is missing if using the New UI. This is a significant omission as this feature is very important for many customers using a Zscaler CSS.

• Fixed Issue 63605: On the VMware SASE Orchestrator's New UI, a user can configure an MD5 hash option for a Cloud Security Service.

  MD5 is a deprecated option beginning with Release 4.5.0 and should not display on a 4.5.0 or later Orchestrator as an option for a CSS.

• Fixed Issue 66226: On the VMware SASE Orchestrator's New UI, when a user has configured some invalid fields inside a collapsed "accordion" section of the browser UI, the user does not have any validity indicator on that accordion and is not able to understand why they are not allowed to save changes.

  When a user enters invalid data in a field and then collapses that section to look at other sections on that UI page, the user cannot save changes but the collapsed accordion section is not marked as invalid. The only action a user can take is to expand the accordion section to check what fields are invalid.

• Fixed Issue 67179: For a customer who has configured a Cloud Security Service, if the CSS provider's Tunneling Protocol is changed from GRE to IPsec, when the user looks at the Configure > Device > Cloud Security Service section for an Edge using that CSS, the user would observe empty FQDN rows in the Credentials section.

  If the CSS provider is changed from GRE to IPsec the previously populated FQDN rows for CSS Credentials are now empty. It is a display issue but is nonetheless a hindrance to the customer user trying to confirm credentials.

• Fixed Issue 67784: On the VMware SASE Orchestrator's Classic UI, for an Edge using six or more WAN links, when consulting the Monitor > QoE tab, the user would observe that the text labels for each link are misaligned for the respective QoE graph they are supposed to match.

  Display issue that hinders a user's ability to see at a glance the QoE status of each of the six or more WAN links.

• Fixed Issue 69240: When a Partner Administrator with a Business Specialist role logs into a VMware SASE Orchestrator running Release 4.5.0, the administrator can observe customer details.

  A Partner Administrator with a Business Specialist role is designed to create and modify customer accounts the Partner is supporting. This role should not be able to view customer configurations.

• Fixed Issue 69981: The tunnel overall status shows differently when looking at the Monitor > Edges page versus the Monitor > Network Services page on the VMware SASE Orchestrator.

  When a user consults the Monitor > Edges page and notes a tunnel's overall status and then compares this status with what is seen on the Monitor > Network Services page, the user may observe the tunnel's overall status in the Cloud Security Service (CSS) section will not match the status seen on the Monitor > Edges page.

• Fixed Issue 71778: When a VMware SASE Orchestrator deployed in a Disaster Recovery (DR) topology is upgraded to Release 4.5.0, an Operator is not able to manually switch Gateways for a Non-SD-WAN Destination (NSD) object.

  The API call to switch the Gateway started enforcing request validation from 4.5.0, however the Orchestrator UI client did not provide a necessary parameter for the API.

• Fixed Issue 71898: When configuring a RADIUS Service, if a configuration field is left empty and the user attempts to save the configuration, the VMware SASE Orchestrator UI throws a general error.

  The error message "An unexpected error has occurred" gives no insight to the user as to what needs to be corrected. In addition the configuration field that needs to be corrected is not highlighted.

• Fixed Issue 72039: When working on the Configure > Device page of the VMware SASE Orchestrator, the user is unable to sort features by those that are segment-aware and those that are segment-agnostic.

  Some device settings are applied across segments versus those that must be configured for each segment, and there is no way to just sort the UI to view those settings for ease of use.

• Fixed Issue 74251: The VMware SD-WAN Edge does not honor the port number in a LAN-side NAT rule that was configured through the Orchestrator API.

  This issue does NOT impact users who configure LAN-side NAT rules through the Orchestrator UI. The port configured as part of the LAN side NAT rule is not pushed properly by the VMware SASE Orchestrator to the Edge. In cases where a LAN-side NAT rule is configured via the updateConfigurationModule API method and a string value was passed for "insidePort" or "outsidePort", the Orchestrator API previously allowed the request. However, the Edge does not honor those string values (it expects integers instead). The Orchestrator API validation logic has been modified to reject string values (and now behaves in a manner consistent with what was already documented in the API reference).

• Fixed Issue 74592: Link statistic queries for a longer period of time (for example, one year) take a long time to return a result on the VMware SASE Orchestrator.

  The queries take a long time because of the absence of enterpriseLogicalID and the Orchestrator lacking sufficient checks in the code to ensure the timestamp format.

• Fixed Issue 75117: A user may observe when consulting diagnostic bundle logs a large number of 'DNS Cache Max Limit Reached. Failed to add cache entry' entries that crowds out other log entries of greater importance.

  On a VMware SD-WAN Hardware Edge, if DNS queries exceeds 32K, this will result in continuous DNS Cache logging that crowds out other log entries. This hinders a user's ability to troubleshoot another issue by consulting the logs in a diagnostic bundle.

• Fixed Issue 75431: When examining a Non SD-WAN Destination (NSD) configuration on the VMware SASE Orchestrator's New UI, the 'Local Auth ID' (FQDN information) is missing.

  A user navigating to Monitor > Network Services > Non SD-WAN Destinations via Gateway and then selecting Details would not see the 'Local Auth ID' field which contains the FQDN information.  This information displays properly on the Classic UI.

• Fixed Issue 75895: An Edge Cloud Security Service (CSS) tunnel alert generation may be skipped for some CSS tunnel events.

  When a customer has an Edge configured with a Cloud Security Service, and there are tunnel Up/Down events on the Edge at the same time, the VMware SASE Orchestrator may fail to generate alerts for all events.

• Fixed Issue 76008: When cloning an enterprise on a VMware SASE Orchestrator, if the chosen Operator Profile is not originally assigned to the parent enterprise, the clone operation fails.

  If the new enterprise is assigned a software image that is NOT assigned to the enterprise being cloned, then the API call will fail with an error.

  Without the fix, the workaround for this issue is to initially assign the new enterprise the same software image as the cloned enterprise and then change to the desired operator profile afterwards.

• Fixed Issue 76036: Attempting to access either 'Partner Overview' page and/or a 'Configure > Customer' page for that Partner on a VMware SASE Orchestrator fails to load with an "An unexpected error has occurred" message.

  The Partner Overview page and/or a Configure > Customer page for a customer supported by that partner may fail to load because the `enterpriseProxy /getEnterpriseProxyGatewayPools` API times out.  The trigger for these pages not loading is if they include a large number of Gateway pools and Gateways which may lead to the enterpriseProxy /getEnterpriseProxyGatewayPools API used on the page timing out, and causing the page loading issue for each UI page.

• Fixed Issue 76078: Some role privileges are removed if a VMware SASE Orchestrator is upgraded to 4.5.0. If the role customization package contains that list of privileges then the package will not be applied in the Orchestrator.

  After an Orchestrator upgrade from a lower release to 4.5.0, the Orchestrator does not apply existing role customization packages when the list of removed privileges in 4.5.0 is available in that package.

• Fixed Issue 77136: When a customer enterprise is migrated from one VMware SASE Orchestrator to another, the Single Sign-On (SSO) configuration is not successfully migrated.

  This is a major annoyance for a customer who would need to manually reconfigure their SSO details after their enterprise is migrated to a different Orchestrator.

• Fixed Issue 77159: A user is able to configure a static route with a bad network prefix with a 0 netmask specification resulting in potentially significant customer traffic disruption.

  The Orchestrator does not do a validation for a bad network prefix with a 0 netmask (for example, 2.2.2.2/0) and instead installs this route on the FIB. Because the static route prefix is 0, it is likely to pick up a lot of traffic destined for the cloud and that traffic may be dropped. 

• Fixed Issue 77515: When a Partner Administrator with a Superuser role assigns a software image to a customer on the Configure > Customer page, an error is thrown when attempting to save the change and the action fails.

  For a partner user while logging the event "Modified the assigned software image list" to update 'Assigned Software/Firmware Images', if no Software/Firmware Images were removed for that event under "removedSoftwareImages", it was processing and listing images from all the imageUpdate modules present in VELOCLOUD_CONFIGURATION_MODULE for the particular operator. In effect the logic is broken for this process when a partner user performs the action and this causes the error and the failure to change customer's default software image.

• Fixed Issue 78684: For a VMware Non SD-WAN Destination via Gateway with a Microsoft Azure Virtual Hub type, when a user performs a Resync Configuration for this Azure NSD, the VMware SASE Orchestrator does not propagate the NSD static routes.

  The routes are not propagated to the Overly Flow Control (table) or the Edge's Device Settings due to an issue with the API responsible for these actions whenever Resync Configuration is selected for the Azure NSD.

• Fixed Issue 80593: A user's deny permission for the privileges used on the "Remote Diagnostics" page doesn't have any effect if the VMware SASE Orchestrator localization changes to a Non-English language.

  Role Customization Package permissions for the privileges used in "Remote Diagnostics" page are not applied when the Orchestrator locale changes to a language other than English. The reason for this issue is that the Orchestrator's Role permission check is done in a translated value, but is compared against an untranslated value which is failing the string match condition.

• Fixed Issue 80930: For a customer who is using a Non SD-WAN Destination via Edge, when a Business Policy rule is created for that Edge, the VMware SASE Orchestrator may also create a duplicate IPsec tunnel configuration for each IPsec tunnel the NSD via Edge had configured and once these duplicates are created, the customer user is unable to make any further configuration changes to the Edge without throwing multiple errors.

  The error the user would see on the Orchestrator UI reads: Edge Segment "Segment Name" Branch to Non SD-WAN Destination via Edge service "Service Name" cannot add duplicated tunnels with same WAN link. This issue has been observed in the field once and has not been successfully recreated by Engineering, which suggests that this issue is rare.  It is caused by the Orchestrator making an extra API call when a business policy is created that also adds duplicates over every IPsec tunnel the Edge already has. 

  The only way to resolve the issue is to delete the duplicate entries and then reenter the configuration details.

• Fixed Issue 80963: When changing the time interval range on the Edge > Monitor > QoE tab, a sample reflected at one timestamp may change and shift over to a new timestamp depending on the selected time interval range.

  A rounding issue in sample sizes causes timestamps to shift earlier for longer time ranges queried on the QoE tab. For example, an event that occurred at 10:02 A.M. will show properly for small time ranges (for example, one hour), but when querying for a larger time range (for example, six hours), the sample might instead show up earlier at say 10:00 A.M., which results in confusion for the user.

• Fixed Issue 81396: On the VMware SASE Orchestrator New UI, a user is not able to update a security VNF configuration for a VMware SD-WAN Edge.

  The specific issue is the UUID of the security VNF. When changing the values like VM-1 IP, VM-1 Hostname, the Orchestrator does not provide a new UUID for the security VNF. Updating a security VNF works as expected on the Orchestrator's Classic UI.

• Fixed Issue 81835: The Monitor > Edge > QoE page of the VMware SASE Orchestrator UI may not accurately represent a WAN link's status (whether it is online, offline, or degraded) or accurately represent link metrics for the time period selected. 

  Different time intervals can lead to the QoE graph showing different results for a WAN link status. And for a link's metrics, the QoE graph may present a particular QoE value (latency, loss, or jitter) that does not reflect the real metric value at that exact time.

  This issue is caused by multiple WAN links belonging to different enterprises being assigned the same link logical ID which leads to a malfunction in the Orchestrator's link data backfill process. The Orchestrator erroneously assumes the WAN link logical Id to be unique because it is not tied to a customer's enterprise ID. This allows for duplicate link logical IDs and the possibility of incorrect link metrics and status.

  The fix for this issue stores the link keys in the Orchestrator's database as a combination of the customer enterprise logical ID and the WAN link's logical ID, ensuring each WAN link is unique.

Secure Access Resolved Issues

Resolved in Version R5000-20220227-GA

The below issues have been resolved since version R450-20210922-GA.

• Issue 70493: When a customer edits a VMware Secure Access service configuration and removes the association of a Cloud Web Security policy, saving the configuration fails.

  Editing a Secure Access service configuration where a Cloud Web Security policy is being removed fails with an "Invalid CWS Policy" error.

Cloud Web Security Resolved Issues

Resolved in Version R5000-20220227-GA

The below issues have been resolved since version R450-20210922-GA.

• Fixed Issue 79324: On the Cloud Web Security service, for specific Applications, some CASB Application Controls are effectively linked (in other words, setting one Application to block may also block another Application), while the CASB Exception Rule Wizard presents them individually.

  The CASB Exception Rule Wizard now alerts users when CASB Application Controls for the selected Application are linked, and changing the affected Control will also change the others. Additionally, only Applications without linked Controls can be grouped together in an Exception Rule, while an application with linked Controls needs its own Exception Rule.

Known Issues

Open Issues in Release 5.0.0.

The known issues are grouped as follows.

• Edge/Gateway Known Issues
• Orchestrator Known Issues
• Cloud Web Security Known Issues
• Secure Access Known Issues

Edge/Gateway Known Issues

• Issue 14655:

  Plugging or unplugging an SFP adapter may cause the device to stop responding on the Edge 540, Edge 840, and Edge 1000 and require a physical reboot.

  Workaround: The Edge must be physically rebooted.  This may be done either on the Orchestrator using Remote Actions > Reboot Edge, or by power-cycling the Edge.

• Issue 25504:

  Static route costs greater than 255 may result in unpredictable route ordering.  

  Workaround: Use a route cost between 0 and 255

• Issue 25595:

  A restart may be required for changes to static SLA on a WAN overlay to work properly.  

  Workaround: Restart Edge after adding and removing Static SLA from WAN overlay

• Issue 25742:

  Underlay accounted traffic is capped at a maximum of the capacity towards the VMware SD-WAN Gateway, even if that is less than the capacity of a private WAN link which is not connected to the Gateway.  

• Issue 25758:

  USB WAN links may not update properly when switched from one USB port to another until the VMware SD-WAN Edge is rebooted.  

  Workaround: Reboot the Edge after moving USB WAN links from one port to another.

• Issue 25855:

  A large configuration update on the Partner Gateway (e.g. 200 BGP-configured VRFs) may cause latency to increase for approximately 2-3 seconds for some traffic via the VMware SD-WAN Gateway.

  Workaround: No workaround available.

• Issue 25921:

  VMware SD-WAN Hub High Availability failover takes longer than expected (up to 15 seconds) when there are three thousand branch Edges connected to the Hub.  

• Issue 25997:

  The VMware SD-WAN Edge may require a reboot to properly pass traffic on a routed interface that has been converted to a switched port.  

  Workaround: Reboot the Edge after making the configuration change.

• Issue 26421:

  The primary Partner Gateway for any branch site must also be assigned to a VMware SD-WAN Hub cluster for tunnels to the cluster to be established.  

• Issue 28175:

  Business Policy NAT fails when the NAT IP overlaps with the VMware SD-WAN Gateway interface IP.  

• Issue 31210:

  VRRP: ARP is not resolved in the LAN client for the VRRP virtual IP address when the VMware SD-WAN Edge is primary with a non-global CDE segment running on the LAN interface. 

• Issue 32731:

  Conditional default routes advertised via OSPF may not be withdrawn properly when the route is deactivated. Reactivating the route, followed by deactivating it again will retract it successfully. 

• Issue 32960:

  Interface “Autonegotiation” and “Speed” status might be displayed incorrectly on the Local Web UI for activated VMware SD-WAN Edges.

• Issue 32981:

  Hard-coding speed and duplex on a DPDK-configured port may require a VMware SD-WAN Edge reboot for the configurations to take effect as it requires deactivating DPDK.

• Issue 34254:

  When a Zscaler CSS is created and the Global Segment has FQDN/PSK settings configured, these settings are copied to Non-Global Segments to form IPsec tunnels to a Zscaler CSS.

• Issue 35778:

  When there are multiple user-defined WAN links on a single interface, only one of those WAN links can have a GRE tunnel to Zscaler. 

  Workaround: Use a different interface for each WAN link that needs to build GRE tunnels to Zscaler.

• Issue 36923:

  Cluster name may not be updated properly in the NetFlow interface description for a VMware SD-WAN Edge which is connected to that Cluster as its Hub.

• Issue 38682:

  A VMware SD-WAN Edge acting as a DHCP server on a DPDK-configured interface may not properly generate “New Client Device" events for all connected clients.

• Issue 38767:

  When a WAN overlay that has GRE tunnels to Zscaler configured is changed from auto-detect to user-defined, stale tunnels may remain until the next restart.

  Workaround: Restart the Edge to clear the stale tunnel.

• Issue 39134:

  The System health statistic “CPU Percentage” may not be reported correctly on Monitor > Edge > System for the VMware SD-WAN Edge, and on Monitor > Gateways for the VMware SD-WAN Gateway.

  Workaround: Users should use handoff queue drops for monitoring Edge capacity not CPU percentage.

• Issue 39374:

  Changing the order of VMware SD-WAN Partner Gateways assigned to a VMware SD-WAN Edge may not properly set Gateway 1 as the local Gateway to be used for bandwidth testing.

• Issue 39608:

  The output of the Remote Diagnostic “Ping Test” may display invalid content briefly before showing the correct results.

• Issue 39624:

  Ping through a subinterface may fail when the parent interface is configured with PPPoE.

• Issue 39753:

  Deactivating Dynamic Branch-to-Branch VPN may cause existing flows currently being sent using Dynamic Branch-to-Branch to stall.

• Issue 40096:

  If an activated VMware SD-WAN Edge 840 is rebooted, there is a chance an SFP module plugged into the Edge will stop passing traffic even though the link lights and the VMware SD-WAN Orchestrator will show the port as 'UP'. 

  Workaround: Unplug the SFP module and then replug it back into the port.

• Issue 40421:

  Traceroute is not showing the path when passing through a VMware SD-WAN Edge with an interface configured as a switched port.

• Issue 42278:

  For a specific type of peer misconfiguration, the VMware SD-WAN Gateway may continuously send IKE init messages to a Non-SD-WAN peer. This issue does not disrupt user traffic to the Gateway; however, the Gateway logs will be filled with IKE errors and this may obscure useful log entries.

• Issue 42388:

  On a VMware SD-WAN Edge 540, an SFP port is not detected after deactivating and then reactivating the interface from the VMware SD-WAN Orchestrator.

• Issue 42872:

  Activating Profile Isolation on a Hub profile where a Hub cluster is associated does not revoke the Hub routes from the routing information base (RIB).

• Issue 43373:

  When the same BGP route is learnt from multiple VMware SD-WAN Edges, if this route is moved from preferred to eligible exit in the Overlay Flow Control, the Edge is not removed from the advertising list and continues to be advertised.

  Workaround: Activate Distributed Cost Calculation (DCC) on the VMware SD-WAN Orchestrator.

• Issue 44995:

  OSPF routes are not revoked from VMware SD-WAN Gateways and VMware SD-WAN Spoke Edges when the routes are withdrawn from the Hub Cluster.

• Issue 45189:

  With source LAN side NAT is configured, the traffic from a VMware SD-WAN Spoke Edge to a Hub Edge is allowed even without the static route configuration for the NAT subnet.

• Issue 45302:

  In a VMware SD-WAN Hub Cluster, if one Hub loses connectivity for more than 5 minutes to all of the VMware SD-WAN Gateways common between itself and its assigned Spoke Edges, the Spokes may in rare conditions be unable to retain the hub routes after 5 minutes. The issue resolves itself when the Hub regains contact with the Gateways.

• Issue 46053:

  BGP preference does not get auto-corrected for overlay routes when its neighbor is changed to an uplink neighbor.

  Workaround: An Edge Service Restart will correct this issue.

• Issue 46137:

  A VMware SD-WAN Edge running 3.4.x software does not initiate a tunnel with AES-GCM encryption even if the Edge is configured for GCM.

• Issue 46216:

  On a Non SD-WAN Destinations via Gateway or Edge where the peer is an AWS instance, when the peer initiates Phase-2 re-key, the Phase-1 IKE is also deleted and forces a re-key.  This means the tunnel is torn down and rebuilt, causing packet loss during the tunnel rebuild.

  Workaround: To avoid tunnel destruction, configure the Non SD-WAN Destinations via Gateway/Edge or CSS IPsec rekey timer to less than 60 minutes.  This prevents AWS from initiating the re-key.

• Issue 46391:

  For a VMware SD-WAN Edge 3800, the SFP1 and SFP2 interfaces each have issues with Multi-Rate SFPs (i.e. 1/10G) and should not be used in those ports.

  Workaround: Please use single rate SFP's per the KB article VMware SD-WAN Supported SFP Module List (79270).  Multi-Rate SFPs may be used with SFP3 and SFP4.

• Issue 46918:

  A VMware SD-WAN Spoke Edge using the 3.4.2 Release does not update the private network id of a Cluster Hub node properly.

• Issue 47084:

  A VMware SD-WAN Hub Edge cannot establish more than 750 PIM (Protocol-Independent Multicast) neighbors when it has 4000 Spoke Edges attached.

• Issue 47664:

  In a Hub and Spoke configuration where Branch-to-Branch via Hub VPN is deactivated, trying to U-turn Branch-to-Branch traffic using a summary route on an L3 switch/router will cause routing loops.

  Workaround: Configure Cloud VPN to activate Branch-to-Branch VPN and select “Use Hubs for VPN”.

• Issue 47681:

  When a host on the LAN side of a VMware SD-WAN Edge uses the same IP as that Edge’s WAN interface, the connection from the LAN host to the WAN does not work.

• Issue 48166:

  A VMware SD-WAN Virtual Edge on KVM is not supported when using a Ciena virtualization OS and the Edge will experience recurring Dataplane Service Failures.

• Issue 48175:

  A VMware SD-WAN Edge running Release 3.4.2 will form an OSPF adjacency on a non-global segment if the non-global segment has an interface configured in the same IP range as an interface configured on the global segment

• Issue 48502:

  In some scenarios, a VMware SD-WAN Hub Edge being used to backhaul internet traffic may experience a Dataplane Service Failure due the improper handling of backhaul return packets.

• Issue 48530:

  VMware SD-WAN Edge 6x0 models do not perform autonegotiation for triple speed (10/100/1000 Mbps) copper SFP's.

  Workaround: Edge 520/540 supports triple speed copper SFPs but this model has been marked for End-of-Sale by Q1 2021.

• Issue 48597: Multihop BGP neighborship does not stay up if one of the two paths to the peer goes down

  If there is a Multihop BGP neighborship with a peer to which there are multiple paths and one of them goes down, user will notice that the BGP neighborship goes down and does not come up using the other available path(s). This includes the Local IP-loopback neighborship case too.

  Workaround: There is no workaround for this issue.

• Issue 48666:

  IPsec-fronted Gateway Path MTU calculation does not account for 61 Byte IPsec overhead, resulting in higher MTU advertisement to LAN client and subsequent IPsec packet fragmentation.

  Workaround: There is no workaround for this issue.

• Issue 49738:

  In some cases, when a VMware SD-WAN Spoke Edge is configured to use multiple Hub Edges, the Spoke Edge may not form tunnels to one of the Hubs configured in the Hub list.

• Issue 50518:

  On a VMware SD-WAN Gateway where PKI is configured, if >6000 PKI tunnels attempt to connect to the Gateway, the tunnels may not all come up because inbound SAs do not get deleted.

  Note: Tunnels using pre-shared key (PSK) authentication do not have this issue.

• Issue 51436: For a site using an Enhanced High-Availability topology while deploying a VMware SD-WAN Edge using an LTE modem, if the site gets into a "split-brain" state, the HA failover takes ~5-6 minutes.

  As part of the recovery from a split-brain state, the LAN ports are brought down on the Active Edge and this impacts LAN traffic during the time the ports are down and until the site can recover.

  Workaround: There is no workaround for this issue

• Issue 52955: DHCP decline is not sent from Edge and DHCP rebinding is not restarted after DAD failure in Stateful DHCP.

  If DHCPv6 server allocates an address which is detected as duplicate by the kernel during a DAD check then the DHCPv6 client does not send a decline. This will lead to traffic dropping as the interface address will be marked as DAD check failed and will not be used. This will not lead to any traffic looping in the network but traffic blackholing will be seen.

  Workaround: There is no workaround for this Issue.

• Issue 53219: After a VMware SD-WAN Hub Cluster rebalances, a few Spoke Edges may not have their RPF interface/IIF set properly.

  On the affected Spoke Edges, multicast traffic will be impacted. What happens is that after a cluster rebalance, some of the Spoke Edge fail to send a PIM join.

  Workaround: This issue will persist until the affected Spoke Edge has an Edge Service restart.

• Issue 53337: Packet drops may be observed with an AWS instance of a VMware SD-WAN Gateway when the throughput is above 3200 Mbps.

  When traffic exceeds a throughput above 3200 Mbps and a packet size of 1300 bytes, packets drops are observed at RX and at IPv4 BH handoff.

  Workaround: There is no workaround for this issue.

• Issue 53359: BGP/BFD session may fail during some DDoS attack scenarios.

  If traffic is flooded from the client connected to the routed interface to the LAN client, the BGP/BFD session can fail. Also when real-time high priority traffic is flooded to the overlay destination, the BGP/BFD session can fail.

  Workaround: There is no workaround for this issue.

• Issue 53934: In an enterprise where a VMware SD-WAN Hub Cluster is configured, if the primary Hub has Multihop BGP neighborships on the LAN side, the customer may experience traffic drops on a Spoke Edge when there is a LAN side failure or when BGP is not configured on all segments.

  In a Hub cluster, the primary Hub has Multihop BGP neighborship with a peer device to learn routes. If the physical interface on the Hub by which BGP neighborship is established, goes down, then BGP LAN routes may not become zero despite BGP view being empty. This may cause Hub Cluster rebalancing to not happen. The issue may also be observed when BGP is not configured for all segments and when there are one or more Multihop BGP neighborships.

  Workaround: Restart the Hub which had the LAN-side failure (or BGP not activated).

• Issue 57210: Even when a VMware SD-WAN Edge is working normally and is able to reach the internet, the LED in the Local UI's Overview page shows as "Red".

  The Edge's Local UI determines the Edge's connectivity by whether it can resolve a well-known name via Google's DNS resolver (8.8.8.8). If it cannot do so for any reason, then it thinks it is offline and shows the LED as red.

  Workaround: There is no workaround for this issue, except to ensure that DNS traffic to 8.8.8.8 can reach the destination and be resolved successfully.

• Issue 61543: If more than one 1:1 NAT rule is configured on different interfaces with the same Inside IP, the inbound traffic can be received on one interface and the outbound packets of the same flow can be routed via different interface.

  For the NAT flows from Outside to Inside, the 1:1 NAT rules will be matched against the Outside IP and the interface where the packets are received. For the outbound packets of the same flow, the VMWare SD-WAN Edge will try to match the NAT rules again comparing the Inside IP and the outbound traffic can be routed via the interface configured in the first matching rule with "Outbound Traffic" configured.

  Workaround: There is no workaround for this issue outside of ensuring no more than one 1:1 NAT rule is configured with a particular Inside IP address.

• Issue 62701: For a VMware SD-WAN Edge deployed as part of an Edge Hub Cluster, If Cloud VPN is not activated under the Global Segment but is activated under a Non-Global Segment, a control plane update sent by the Orchestrator may cause all the WAN links to flap on the Hub Edge.

  The Hub Edge's WAN links going down, then up in rapid succession (flap) will impact real time traffic like voice calls. This issue was observed on a customer deployment where Cloud VPN was not activated on the Hub Edge's Global segment, but the Cluster configuration was configured as on, which means this Hub Edge was part of a Cluster (and a Cluster configuration is applicable to all segments). When a configuration change is pushed to the Hub Edge, the Hub Edge's dataplane will start parsing data and will start with the Global Segment where it will see Cloud VPN not activated and the Hub Edge erroneously thinks clustering has been deactivated on this Global Segment. As a result, the Hub Edge will tear down all tunnels from the Hub's WAN link(s) which will cause link flaps on all that Edge's WAN links. For any such incident the WAN links only go down and recover a single time per control pane update.

  Workaround: The workaround is to activate Cloud VPN on all Segments, this means the Global Segment along with all Non-Global Segments.

• Issue 63629: In a network topology where the VMware SD-WAN Hub Edge and Spoke Edge have different IP family preferences (in other words, IPv4/IPv6 dual stack), the customer can see more bandwidth allocated to the peer than expected.

  If both IPv4 and IPv6 families are configured, the Edge internally creates two different link objects. The bandwidth values are added for both of them when it should be added only for one.

  Workaround: The workaround for this issue is to not have different tunnel preferences if the Hub/Spoke topology has dual stack configured.

• Issue 65560: Traffic from a customer to PE (Provider Edge) device fails.

  BGP neighborship between a Partner Gateway and Provider Edge does not get established when tag-type is selected as "none" on the handoff configuration. This is because ctag, stag values get picked from /etc/config/gatewayd instead of the handoff configuration on the Orchestrator when tag-type is "none".

  Workaround: Update the ctag, stag values to 0 each under vrf_vlan->tag_info in /etc/config/gatewayd. Do a vc_procmon restart.

• Issue 67458: When a VMware SD-WAN Hub Edge with a large number of Spoke Edges is upgraded to Release 4.2.1 or later, some tunnels to other Spoke Edges will not come up for the Hub Edge.

  A large number of Spoke Edges is understood at ~1000 or more. This issue is not consistent, but generally ~1/3rd of the VeloCloud Management Protocol (VCMP) tunnels are not established between the Hub Edge and the connected Spoke Edges. This is caused by the Hub Edge ignoring the 
  MP_INITs as the number of half open TDs exceeds the Hub Edge's upper limit.

  Workaround: Restarting the Edge Service will restore full tunnel connectivity.

• Issue 67879: A Cloud Security Service (CSS) tunnel is deleted after a user changes a WAN Overlay setting from auto-detect to user-defined on a WAN interface setting.

  After saving the changes, the CSS tunnels do not come back up until the customer takes down and then puts back up the tunnel. Changing the WAN configuration will bring down the CSS tunnel and parse the CSS setup again. However, in some corner cases, the nvs_config->num_gre_links is 0 and the CSS tunnel fails to come up.

  Workaround: Deactivate the CSS setup, and then reactivate it and this will bring the CSS tunnel up.

• Issue 68057: DHCPv6 release packet is not sent from the VMware SD-WAN Edge on the changing of a WAN interface address mode from DHCP stateful to static IPv6 address and the lease remains active till reaching its valid time.

  The DHCPv6 client possesses a lease which it does not release when the configuration change is done. The lease remains valid till its lifetime expires in the DHCPv6 server and is deleted.

  Without the fix, there is no way of remediating this issue as the lease would remain active till valid lifetime.

• Issue 68851: If a VMware SD-WAN Edge and VMware SD-WAN Gateway each have the same TCP syslog server configured, the TCP connection is not established from the Edge to the syslog server.

  If the Edge and Gateway each have the same TCP server and if the syslog packets from the Edge are routed via the Gateway, the syslog server sends a TCP reset to the Edge.

  Workaround: Send the syslog packets direct from the Edge instead of routing via a Gateway or configure a different syslog server for the Edge and Gateway..

• Issue 69284: For a site using a High-Availability topology where the Edges deploy VNF's in an HA configuration and are using Release 4.x, if these HA Edges are downgraded to a 3.4.x Release where HA VNF's are not supported, and then upgraded to 4.5.0, when the HA VNF's are reactivated, the Standby Edge VNF will not come up. 

  The VNF state on the Standby Edge is communicated as down via SNMP. If the HA VNF pair is downgraded from a version supporting VNF-HA (release 4.0+) to a release which does not support it with VNF configured on the Orchestrator. This issue will be seen when the Edge is upgraded back to a version supporting VNF-HA and it is configured on the Orchestrator again.

  Workaround: VNF should first be deactivated in the case of an HA if the Edge is being downgraded to a version which does not support it.

• Issue 70311: A VMware SD-WAN Edge may experience a Dataplane Service Failure and restart as a result.

  During the Edge service restart, customer traffic would be disrupted for ~15-30 seconds. This issue occurs inconsistently, but when it does occur the Edge is tearing down an IKE security association (SA). This typically only occurs when: the SA timer (as configured on the VMware SD-WAN Orchestrator) expires; or the user modifies the IPSec configuration on the Orchestrator.

  Workaround: There is no workaround for this issue.

• Issue 71719: PPTP Connection is not Established along Edge to Cloud path.

  Connection to the PPTP server behind the VMware SD-WAN Edge does not get established.

  Workaround: There is no workaround for this issue, not even an Edge restart or reboot.

• Issue 72358: If the IP address of a VMware SD-WAN Orchestrator DNS name changes, the VMware SD-WAN Gateway's management plane process fails to resolve it properly and the Gateway will be unable to connect to the Orchestrator.

  The Gateway's management process periodically checks the DNS resolution of the Orchestrator's DNS name, to see if it has changed recently so that the Gateway can connect to the right host. The DNS resolution code has an issue in it so that all of these resolution checks fail, and the Gateway will keep using the old address and thus no longer be able to connect to the Orchestrator.

  Workaround: Until this issue is resolved, an Operator User should not change the IP address of the Orchestrator. If the Orchestrator's IP address must be changed, all Gateways connecting to that Orchestrator will have to be reactivated.

• Issue 72925: For a customer who uses SNMP polling for monitoring their enterprise and deploys lower model VMware SD-WAN Edges (for example, Edge models 510, 520, or 610) which are running a 4.x software release, SNMP polling takes exceptionally long to process and can even timeout.

  This issue significantly reduces the effectiveness of SNMP polling for network monitoring when using Edges in the 510, 5x0, and 6x0 series. This issue is caused by the Release 4.x SNMPagent taking an unnecessarily long amount of time in traversing the debug command list, which is not actually required for the SNMP process.

  Workaround: There is no workaround for this issue.

• Issue 77541: When a USB modem that supports IPv6 is unplugged and then replugged into a VMware SD-WAN Edge USB interface, an IPv6 address may not provisioned to the USB interface.

  This affects USB modems that are IP-based, versus being managed by the ModemManager application. Most Inseego modems are IP-based and this is important because Inseego is the modem manufacturer VMware SASE recommends. USB modems supporting IPv6 which use ModemManager versus being IP-based will be fine in a plug out and plug in scenario.

  Workaround: The Edge needs to be rebooted (or power-cycled) after the USB modem is replugged into the Edge's USB port. Post reboot, the Edge will retrieve the IPv6 address for the modem.

• Issue 81852: For a VMware SD-WAN Edge that is using a Zscaler type Cloud Security Service (CSS) which uses GRE tunnels that has turned on L7 Health Check, when that Edge is upgraded to Release 5.0.0, in some instances the customer may observe L7 Health Check errors.

  This is typically seen during software upgrade or during startup time. When L7 Health check for a CSS using GRE tunnels is turned on, error messages related to socket getaddress error may be seen. The observed error is intermittently seen, and not consistent. Because of this, L7 Health Check probe messages are not sent out.

  Workaround: Without the fix, to remediate the issue, a user needs to turn off and then turn back on the L7 Health Check configuration, and this feature would then work as expected.

• Issue 81859: When activating a VMware SD-WAN Edge 610-LTE, the CELL interface may not come up after the Edge completes its activation.

  This issue is not consistent but when it occurs it can have a major impact if the Edge 610-LTE's only public link is the mobile CELL link as the Edge would be effectively down and intervention for this Edge would need to be local in the form of someone power cycling the Edge to recover it.

  Workaround: If encountering this issue and the 610-LTE has other wired public WAN links, the user would need to either restart the Edge service through the Orchestrator using Remote Actions > Service Restart in a suitable maintenance window, or restart the Edge's modem to restore the CELL interface.

  If the 610-LTE only uses a CELL interface for internet, someone local to the Edge would have to power cycle the Edge as it would be inaccessible through the Orchestrator.

  If the 610-LTE Edge being activated only uses CELL for internet, the Edge should be activated with someone present to potentially power cycle it should it go down after completing activation.

• Issue 82104: In rare cases, VMware SD-WAN Edges activated in a High Availability topology may be unable to communicate with a VMware SASE Orchestrator which will mark the site as down and preclude any intervention through the Orchestrator to the site.

  This issue occurs only when an unusual and invalid configuration is applied to the HA Edges. The configuration specifies that the HA port is configured as "trunk" (which should not be allowed), with zero VLANs (also should not be allowed), but where "all VLANs" are set. Instead of throwing an error at this configuration and preventing a user from activating HA for the Edges, the Orchestrator allows it and this configuration triggers a management plane failure on the HA Edges which no longer send a heartbeat to the Orchestrator and the Orchestrator marks the site as down.

  Workaround: Avoid using the configuration outlined above.

• Issue 82182: For a VMware SD-WAN Edge Model 510 or 510-LTE which is running Edge Release 5.0.0, when a user attempts a service restart of the Edge, the Edge may also reboot.

  An Edge reboot would disrupt customer traffic for 2-3 minutes while the Edge was going through the reboot process. On an Edge 510/510-LTE, there is a Wi-Fi device hang monitoring script which may fail to stop during the Edge service restart and this triggers the reboot.

  Workaround: There is no workaround, but Edge service restarts for these models should only be done in a maintenance window or with the understanding this issue may arise.

• Issue 82184: On a VMware SD-WAN Edge which is running Edge Release 5.0.0, when a traceroute or traceroute6 is run to the Edge's br-network IPv4/IPv6 address, the traceroute will not properly terminate when a UDP probe used.

  Traceroute or traceroute6 to the Edge's br-network IPv4/IPv6 address will not properly when Default Mode (in other words, UDP probe) is used.

  Workaround: Use -I option in traceroute and traceroute6 to use ICMP probe and then traceroute to br-network IPv4/IPv6 address will work as expected.

• Issue 82264: A VMware SD-WAN Virtual Edge which uses an AWS C4 instance cannot be upgraded to Release 5.0.0.

  An AWS C4 Virtual Edge upgraded to Release 5.0.0 does not recover and the only remediation is for the user to reactivate the Edge to a non-5.0.0 version.  No other AWS instances (for example, C5) are affected by this issue, but due to the critical nature of this defect an AWS Edge Upgrade software package is not available for Release 5.0.0.

  Workaround: Customer can upgrade their AWS C4 Edge instances to Release 5.0.1, which includes a fix for this issue.

• Issue 82415: A VMware SD-WAN Gateway deployed a KVM image with  Intel® Ethernet Server Adapter X710; SR-IOV; and Bond0 does not respond if activated to Release 4.5.0 or 5.0.0.

  For a Gateway so deployed paths do not come up and the debug.py commands do not work.

  Workaround: There is no workaround. Avoid using these builds for this specific Gateway deployment until new builds with this issue fixed are rolled out.

• Issue 83166: When a VMware SD-WAN Gateway is freshly deployed with a AWS c5.4xlarge instance type from the AWS Portal with IPv6 option selected, neither IPv6 or the default routes are configured.

  As a result of IPv6 and default routes not being configured, the AWS Gateway IPv6 management tunnels are not forming and the Gateway will not work.

  Workaround: There is no workaround for this issue, avoid deploying a Gateway with the properties mentioned above.

• Issue 83227: For a VMware SD-WAN Edge running Release 5.0.0 which is configured with 128 Segments, the Edge's dnsmasq process will stop and exit.

  When IPv6 is configured on all 128 segments and DCPv6 servers are configured in the LAN of each segment, the dnsmasq process will stop as the total open file descriptors is exceeded. The dnsmasq process will continue for ~30 minutes before exiting at which point the Edge's DHCP assignment of IP addresses will fail. 

  Workaround: Rebooting the Edge restores the dnsmasq process for ~30 minutes but it will fail again. The only real workaround is to reduce the number of segments to less than 128.

• Issue 84825: For a site deployed with a High-Availability topology where BGP is configured, if the site has greater than 512 BGPv4 match/set rules configured, the customer may observe the HA Edge pair continuously failing over without ever recovering.

  Greater than 512 BGPv4 match and set rules is understood as a customer configuring more than 256 such rules on the inbound filter and 256 rules on the outbound filter. This issue would be disruptive to the customer as the repeated failover would cause flows for real time traffic like voice calls to be continuously dropped and then recreated. When HA Edges experience this issue, the process that synchronizes Edge CPU threads fails causing the Edge to reboot to recover, but the promoted Edge also experiences the same issue and reboots in turn with no recovery reached at the site.

  Workaround: Without a fix for this issue, the customer must ensure that no more than 512 BGPv4 match and set rules are configured for an HA site.

  If a site is experiencing this issue and has more than 512 BGP/v4 match and set rules configured, the customer must immediately reduce the number of rules to 512 or less to recover the site.

  Alternatively, if the customer must have more than 512 BGPv4 match and set rules, they can downgrade the HA Edges to Release 3.4.6 where this issue is not encountered, but at the cost of Edge features found in later releases. This can only be done if their Edge model is supported on 3.4.6 and the customer should confirm that is so before downgrading.

• Issue 85369: For a site deployed with a High-Availability topology, the customer may observe customer traffic disruptions and possibly multiple reboots of the VMware SD-WAN Standby Edge.

  A condition triggered by load and system events causes the Active Edge to experience delays in the timely delivery of HA heartbeats to the Standby Edge. The delay causes the Standby Edge to miss heartbeats and incorrectly assume the Active role causing an Active-Active state. To recover from the Active-Active state the Standby Edge reboots, possibly multiple times. 

  If the site does become Active-Active, a conventional HA setup would experience minimal traffic disruption since the Standby Edge does not pass traffic in this topology, but on an Enhanced HA deployment, where the Standby Edge is also passing traffic, the reboot(s) would disrupt some customer traffic.

  Workaround: There is no workaround for this issue.

• Issue 85461: If a VMware SD-WAN Edge is used to forward DNS, and LAN devices connected to the Edge are using the Edge for DNS forwarding, all DNS traffic may fail.

  All DNS forwarding traffic is affected, not just Conditional DNS. Depending on the Edge software, this issue can be encountered on an Edge as follows:

  • If the Edge is using Release 4.2.2, the Edge can encounter this issue if the Edge is using routed LAN ports with no Gateway IP address specified. Switched LAN ports + VLANs are not affected in 4.2.2. 
  • If the Edge is using either Release 4.3.0/4.3.1, 4.5.0/4.5.1, or 5.0.0.x, the Edge can encounter the issue if the Edge is using switched LAN ports and VLANs, or the Edge is using routed LAN ports with no Gateway IP address specified.

  For switched interfaces, the cause of the issue stems from the deprecation and removal of the Management IP interface in favor of a loopback interface in Releases 4.3.x, 4.5.x, and 5.0.0.x and later. Because DNS uses segment NAT, the DNS packet has no matching entry for the destination IP when the Edge does segment NAT table lookup and the Edge drops the packet.

  For routed interfaces, the lack of a Gateway IP means the DNS packet is routed to the Edge as the next hop and the Edge does not forward the DNS packet further.

  Workaround: The workaround for this issue is to either not use the Edge to forward DNS, or...

  • When using Edge Release 4.2.2: use either switched LAN ports or routed LAN ports that include a Gateway IP address.
  • When using Release 4.3.x or 4.5.x, use only routed LAN ports with a Gateway IP address specified. 
• Issue 88604: For a site using a High-Availability topology, if a WAN interface goes down and then comes back up on a VMware SD-WAN Standby Edge, the event is not recorded on the VMware SASE Orchestrator.

  A user does not have visibility on Standby Edge interface events, which is especially impactful on Enhanced HA deployments where the Standby Edge is also passing traffic.

  Workaround: There is no workaround for this issue.

• Issue 88796: When deploying either a VMware SASE Orchestrator or a VMware SD-WAN Gateway and using an OVA on vSphere, the OVF properties set as part of the deployment (password, network information, etc.) are not applied to the image and the system cannot be accessed after deployment.

  This only affects new system deployed from OVA using OVF/vApp properties (versus using ISO files). This issue is caused by upstream changes to cloud-init in recent updates.

  Note: This open ticket applies only to Gateway builds.  The Orchestrator builds are fixed with Release R5002-20220517-GA.

  Workaround: Without the fix, the workaround is for the Operator to deploy the system using a cloud-init user-data ISO file.

• Issue 89217: A VMware SD-WAN Edge in the 6x0 model line (610, 610N, 610-LTE, 620, 620N, 640, 640N, 680, 680N) may suddenly power off for no reason.

  The 6x0 Edge would have all lights off, both the front status LED and the rear Ethernet port lights, and can only be recovered by manually power cycling the Edge.

  The cause of the issue is traced to a PIC microcontroller exclusive to the Edge 6x0 line which uses a PIC firmware version of v20M or earlier (v20L, v20K, v20J). This issue can only occur when the 6x0 Edge uses a PIC version of v20M or earlier, but even with this version the odds of experiencing the power off issue are rare (approximately 1/1,000). The issue cannot occur on a 6x0 Edge with a PIC firmware version of v20N or later.

  Note: A 6x0 Edge's Firmware including PIC version can be determined on an Orchestrator using 5.x by going to the Monitor > Edge > Overview page for that Edge and clicking the dropdown information box next to the Edge name which includes the Edge Information, Device Version, and the Device Firmware.

  The issue is resolved by upgrading the 6x0 Edge to Platform Firmware 1.3.1 (R131-20221216-GA), which includes PIC version v20N. To do this the 6x0 Edge must be connected to a VMware SASE Orchestrator using Release 5.x (5.0.0 or later), and the 6x0 Edge must first be upgraded to Edge hoftix build R5012-20230123-GA-103475. Once the 6x0 Edge is upgraded to R5012-20230123-GA-103475, the user would then update the 6x0 Edge Platform Firmware to version R131-20221216-GA in the same way that an Edge's software version is modified.

  For more information and a step-by-step guide to upgrading a 6x0 Edge to Platform Firmware 1.3.1, see the KB Article: VMware SD-WAN 6X0 model Edges may power off with no LEDs and require a power cycle to come back to a working state (88970). This KB article was updated on January 27th, 2023 to reflect the new Edge and Platform Software needed to resolve the issue.

  For information on uploading a Platform Firmware bundle to an Orchestrator, consult the Platform Firmware and Factory Images with New Orchestrator UI section of the VMware SD-WAN Operator Guide.

  For information on updating a 6x0 Edge’s Platform Firmware, consult the View or Modify Edge Information section of the VMware SD-WAN Administration Guide.

  Note: If the user prefers to keep the Edge on a lower software release (for example, Release 4.3.1, or 4.5.1), the customer can temporarily upgrade the Edge to a 5.x build, perform the Platform Firmware upgrade to version 1.3.0 (R130-20220328-GA) so that the PIC version is v20N, and then downgrade the Edge’s software back to their preferred version. Downgrading the 6x0 Edge's software to an earlier version does not also downgrade the Edge's Platform Firmware and the Edge would continue to use Platform Firmware version 1.3.0 (R130-20220328-GA). In this use case the customer Edges would need to be on an Orchestrator using Release 5.x.

  Note: If the 6x0 Edge is on an Orchestrator that does not use version 5.x and has experienced this issue and requires an update of its PIC firmware, the customer may reach out to VMware SD-WAN Support and they will manually update the Edge’s PIC version.

  Workaround: To recover the Edge from the problem state:​

  1. Disconnect the Edge from the power source.
  2. Wait 20 seconds.
  3. Reconnect the Edge to the power source.

  If you do not wish to upgrade the 6x0 Edge's platform firmware, the user can ensure the power to the Edge is consistent and does not flap rapidly or consistently. A good way to ensure a reliable power source is to connect the 6x0 Edge to a Uninterruptible Power Supply (UPS).

  If the user prefers to keep the Edge on a lower software release (for example, Release 4.3.1, or 4.5.1), the customer can temporarily upgrade the Edge to R5012-20230123-GA-103475, perform the Platform Firmware upgrade to version 1.3.1 (R131-20221216-GA) so that the PIC version is v20N, and then downgrade the Edge’s software back to their preferred version. Downgrading the 6x0 Edge's software to an earlier version does not also downgrade the Edge's Platform Firmware and the Edge would continue to use Platform Firmware version 1.3.1. In this use case the customer Edges would need to be on an Orchestrator using Release 5.x.

  If the 6x0 Edge is on an Orchestrator that does not use version 5.x and has experienced this issue and requires an update of its PIC firmware, the customer may reach out to VMware SD-WAN Support and they will manually update the Edge’s PIC version.

• Issue 91365: For a customer using Edge Network Intelligence, an VMware SD-WAN Edge where Analytics is configured experiences a memory leak that will result in the Edge triggering an Edge Service restart to clear the memory.

  When the Analytics function is activated on an Edge, the Edge's Dataplane service begins leaking memory at a steady rate that will result in the Edge needing to trigger an unscheduled Service Restart to clear the memory leak when it reaches a critical level (60% memory utilization for longer than 90 seconds). An Edge Service restart causes a 10-15 second disruption in customer traffic. In the field the time it takes to trigger an Edge Service restart has been ~3 to 4 days, and once the memory is cleared the memory leak will resume with the same general time window for the next Edge Service restart. The period when the Edge would reach a critical memory usage level depends on the Edge model and the amount of information the Analytics feature is recording for that Edge.

  Workaround: The customer has two options, a) temporarily deactivate Analytics for the Edge until a fixed Edge build is delivered; or b) monitor the Edge's memory. When memory utilization reaches 40% and the Orchestrator records a Memory Warning Event, schedule a manual Edge Service Restart in a maintenance window to clear the memory and ensure minimal customer impact.

• Issue 91746: A VMware SD-WAN Edge using either wired or wireless 802.1x authentication (for example, RADIUS, Cisco ISE) may experience certificate authentication failures with all traffic requiring this authentication dropping at the Edge.

  The issue is caused by the Edge improperly altering the L4 headers of IP fragmented packets which results in the packets becoming corrupted before exiting the Edge. This primarily impacts UDP packets and as these packets are used for 802.1x certificate authentication has the potential to cause 802.1x wired or wireless clients to fail.

  Workaround: On an Edge where this issue is encountered, the workarounds are to either, a) deactivate 802.1x authentication, or b) roll the Edge back to a prior Edge software build where 802.1x authentication worked properly because this issue is not present.

• Issue 92481: If a WAN interface on a VMware SD-WAN Edge is deactivated on the VMware SASE Orchestrator, the interface will still be reported as 'UP' by SNMP.

  The key debug process for interfaces output does not include the physical port details for Edge WAN interfaces (for example, GE3 or GE4 on an Edge 6x0 or 3x00 model). As a result when SNMP polls those interfaces it always returns a result of UP regardless of how these interfaces are configured.

  Workaround: There is no workaround for this issue.

• Issue 92676: For a customer deployment where a Non VMware SD-WAN Destination (NSD) via Gateway is configured to use redundant tunnels and redundant Gateways and is also using BGP over IPsec, if the Primary and Secondary Gateways advertise a prefix with an equal AS path to the Primary and Secondary NSD tunnels, the Primary NSD tunnel will prefer a redundant Gateway path over the Primary Gateway.

  The impact of the Primary NSD over Gateway tunnel preferring the redundant Gateway path over the Primary Gateway is experienced only for return traffic to the Gateway from the NSD.

  Workaround: Configure a higher (3 or more) metric on the redundant Gateway for the interested prefix as this will help the NSD's primary tunnel choose the Primary Gateway for return traffic.

• Issue 100377: When a VMware SD-WAN Edge is upgraded to Release 5.0.x, LAN-side client users may observe that all customer traffic drops and they are unable to connect to other sites and the internet.

  Description: This issue occurs randomly and only affects LAN-side traffic. The Edge remains connected to both the Gateway and Orchestrator. On the Orchestrator when looking at Monitor > Edge > Health a user would observe an escalating level of handoff queue drops. The issue is caused by a change in behavior introduced in the 5.x Edge build that can lead to a packet leak, a particular packet connected with the change is no longer released and over time the packet buffer becomes overloaded and begins dropping all packets.

  Workaround: VMware strongly recommends upgrading all customer Edges to the 5.0.1.2 build. On an Edge without a fix for this issue, a restart of the Edge service will clear the packet buffer, but only temporarily.

• Issue 103708: When new rules are added in a BGP filter configuration, there may be unexpected BGP routes received and sent by the VMware SD-WAN Edge.

  When new rules are added to the BGP filters from the Orchestrator, the prefix lists are added in the Edge's routing configuration without removing the old entries. This behavior results in stale route prefix lists and unexpected filtering behavior.

  Workaround: There is no workaround for this issue beyond upgrading the Edges to fixed build R5014-20230713-GA in the 5.0.x train, or R5102-20230310-GA and later.

Orchestrator Known Issues

• Issue 19566:

  After High Availability failover, the serial number of the standby VMware SD-WAN Edge may be shown as the active serial number in the Orchestrator.

• Issue 21342:

  When assigning Partner Gateways per-segment, the proper list of Gateway Assignments may not show under the Operator option "View" Gateways on the VMware SD-WAN Edge monitoring list.

• Issue 24269:

  Monitor > Transport > Loss not graphing observed WAN link loss while QoE graphs do reflect this loss. 

• Issue 25932:

  The VMware SD-WAN Orchestrator allows VMware SD-WAN Gateways to be removed from the Gateway Pool even when they are in use.

• Issue 32335:

  The ‘End User Service Agreement’ (EUSA) page throws an error when a user is trying to accept the agreement.

  Workaround: Ensure no leading or trailing spaces are found in Enterprise Name.

• Issue 32435:

  A VMware SD-WAN Edge override for a policy-based NAT configuration is permitted for tuples which are already configured at the profile level and vice versa.

• Issue 32856:

  Though a business policy is configured to use the Hub cluster to backhaul internet traffic, the user can unselect the Hub cluster from a profile on a VMware SD-WAN Orchestrator that has been upgraded from Release 3.2.1 to Release 3.3.x.

• Issue 32913:

  After turning on High Availability, Multicast details for the VMware SD-WAN Edge are not displayed on the Monitoring Page. A failover resolves the issue.

• Issue 33026:

  The ‘End User Service Agreement’ (EUSA) page does not reload properly after deleting the agreement.

• Issue 34828:

  Traffic cannot pass between a VMware SD-WAN Spoke Edge using release 2.x and a Hub Edge using release 3.3.1.

• Issue 35658:

  When a VMware SD-WAN Edge is moved from one profile to another which has a different CSS setting (e.g. IPsec in profile1 to GRE in profile2), the Edge level CSS settings will continue to use the previous CSS settings (e.g. IPsec versus GRE). 

  Workaround: At the Edge level, deactivate GRE and then reactivate GRE to resolve the issue.

• Issue 35667:

  When a VMware SD-WAN Edge is moved from one profile to another profile which has the same CSS setting but a different GRE CSS name (the same endpoints), some GRE tunnels will not show in monitoring.

  Workaround: At the Edge level, deactivate GRE and then reactivate GRE to resolve the issue.

• Issue 36665:

  If the VMware SD-WAN Orchestrator cannot reach the internet, user interface pages that require accessing the Google Maps API may fail to load entirely.

• Issue 38056:

  The Edge-Licensing export.csv file not show region data.

• Issue 38843:

  When pushing an application map, there is no Operator event, and the Edge event is of limited utility.

• Issue 39633:

  The Super Gateway hyper link does not work after a user assigns the Alternate Gateway as the Super Gateway.

• Issue 39790:

  The VMware SD-WAN Orchestrator allows a user to configure a VMware SD-WAN Edge’s routed interface to have greater than the supported 32 subinterfaces, creating the risk that a user can configure 33 or more subinterfaces on an interface which would cause a Dataplane Service Failure for the Edge.

• Issue 40341:

  Though the Skype application is properly categorized on the backend as Real Time traffic, when editing the Skype Business Policy on the VMware SD-WAN Orchestrator, the Service Class may erroneously display “Transactional”.

• Issue 41691:

  User cannot change the 'Number of addresses' field although the DHCP pool is not exhausted on the Configure > Edge > Device page.

• Issue 43276:

  User cannot change the Segment type when a VMware SD-WAN Edge or Profile has a partner gateway configured.

• Issue 47269:

  The VMware SD-WAN 510-LTE interface may appear for Edge models that do not support an LTE interface.

• Issue 47713:

  If a Business Policy Rule is configured while Cloud VPN is turned off, the NAT configuration must be reconfigured upon turning on Cloud VPN.

• Issue 47820:

  If a VLAN is configured with DHCP turned off at the Profile level, while also having an Edge Override for this VLAN on that Edge with DHCP configured, and there is an entry for the DNS server field set to none (no IP configured), the user will be unable to make any changed on the Configure > Edge > Device page and will get an error message of ‘invalid IP address []’ that does not explain or point to the actual problem.

• Issue 48085:

  The VMware SD-WAN Orchestrator allows a user to delete a VLAN which is associated with an interface.

• Issue 48737:

  On a VMware SD-WAN Orchestrator which is using the Release 4.0.0 new user interface, If a user is on a Monitor page and changes the Start & End time interval and then navigates between tabs, the Orchestrator does not update Start & End interval time to the new values.

• Issue 49225:

  VMware SD-WAN Orchestrator does not enforce a limit of 32 total VLANs.

• Issue 49790:

  When a VMware SD-WAN Edge is activated to Release 4.0.0, the activation is posted twice in Events.

  Workaround: Ignore the duplicate event.

• Issue 50531:

  When two Operators of differing privileges use the same browser window when accessing the New UI on a 4.0.0 Release version of the VMware SD-WAN Orchestrator, and the Operator with lesser privileges tries to login after the Operator with higher privileges, that lesser privileged Operator will observe multiple errors stating that the "user does not have privilege".

  Note: There is no escalation in privileges for the Operator with lower privileges, only the display of error messages.

  Workaround: The next operator may refresh that page prior to logging in to prevent seeing the errors, or each Operator may use different browser windows to avoid this display issue.

• Issue 51722: On the Release 4.0.0 VMware SD-WAN Orchestrator, the time range selector is no greater than two weeks for any statistic in the Monitor > Edge tabs.

  The time range selector does not show options greater than "Past 2 Weeks" in Monitor > Edge tabs even if the retention period for a set of statistics is much longer than 2 weeks.  For example, flow and link statistics are retained for 365 days by default (which is configurable), while path statistics are retained only for 2 weeks by default (also configurable).  This issue is making all monitor tabs conform to the lowest retained type of statistic versus allowing a user to select a time period that is consistent with the retention period for that statistic.

  Workaround: A user may use the "Custom" option in the time range selector to see data for more than 2 weeks.

• Issue 60522: On the VMware SD-WAN Orchestrator UI, the user observes a large number of error messages when they try to remove a segment.

  The issue can be observed when adding a segment to a profile and the associating the segment with multiple VMware SD-WAN Edges. When the user attempts to remove the added segment from the profile, they will see a large number of error messages.

  Workaround: There is no workaround for this issue.

• Issue 60039: RMA Reactivation does not work when the VMware SD-WAN Edge model is changed.

  When performing an RMA Reactivation for a site where the Edge model is also being changed, the VMware SD-WAN Orchestrator does not save the model change making the reactivation link ineffective. This only affects RMA Reactivations where the Edge model is changed, an RMA Reactivation where the Edge model remains the same will work as expected.

  Workaround: If using a different Edge model for a site, the user would need to create a new Edge and manually apply all Edge-specific settings.

• Issue 62624: User sees the Customer name when attempting to uncheck the Partner Gateway checkbox while the Partner Gateway is in use.

  When a user unchecks the Partner Gateway checkbox for a particular Gateway on the VMware SD-WAN Orchestrator UI while the Gateway is used by one or more customers and a customer profile as well, the Orchestrator shows only the name of the Profile and Edge not the name of the customer names using the Gateway.

  Workaround: There is no workaround for this issue.

• Issue 68463: When using the New UI on the VMware SD-WAN Orchestrator and looking at the QoE section, the wrong graph values are shown.  

  When going on QoE in the old UI "Latency Fair" is displayed on the graph, whereby when visiting the new UI (for the same Edge and time) "Jitter Fair" is displayed. This is caused by QoE being incorrectly mapped on the New UI.

  Workaround: There is no work around for this issue beyond using the Old UI to confirm correct QoE values.

• Issue 75348: A customer who has configured a Non SD-WAN Destination (NSD) via Gateway where "Deactivate Site Subnets" is checked may observe a 0.0.0.0/32 route under Configure > Edge > Device > Static Route Settings > NSD Routes despite that route not being configured by the customer.

  When the Deactivate Site Subnets box is checked on the NSD configuration UI, a default 0.0.0.0/32 subnet is created from the UI itself and while it is not shown on the NSD configuration screen this route shows up on the Configure > Edge > Device UI screen. The issue is purely cosmetic and this route is not disruptive to customer traffic. 

  Workaround: The route can be cleared on the Orchestrator UI when a user checks the box for Redundant VeloCloud Cloud VPN and then unchecks it on the screen where the NSD is configured. Once the 0.0.0.0/32 route is removed, it will not show up in the NSD Routes section again.

• Issue 82680: For customer using MT-GRE Tunnel Automation, when a user turns off the Cloud-to-Cloud Interconnect (CCI) flag on a VMware SD-WAN Gateway which is configured to use CCI, the Zscaler MT-GRE entries may not get deleted from the Zscaler portal consistently.

  After a CCI site has been deleted from the Gateway, the entries for this site should also be removed. This issue has only been seen during test automation and has not been reproduced manually, but remains a risk. 

  Workaround: Manually delete the resource from Zscaler before retrying.

• Issue 82681: For customer using MT-GRE Tunnel Automation, when a user turns off the Cloud-to-Cloud Interconnect (CCI) flag on a VMware SD-WAN Gateway which is configured to use CCI, and the user deactivates the CCI flag from a VMware SD-WAN Edge with CCI configured which is using a Zscaler Cloud Security Service, the Zscaler MT-GRE entries may not get deleted from the Edge or from the Zscaler portal.

  After a CCI site has been deleted from the Gateway, the entries for this site should also be removed. This issue has only been seen during test automation and has not been reproduced manually, but remains a risk.

  Workaround: Manually delete the resource from Zscaler before retrying.

• Issue 82725: A VMware SASE Orchestrator may not generate the password reset link correctly.

  This issue occurs when the URL for the Orchestrator is not exactly https://domain/ or https://domain/operator/.  However, if for example the URL is https://domain/test/ the password reset link does not work and directs you back to the login page.

  Workaround: If the Orchestrator URL cannot be corrected to a URL as shown above, the only option is for a Superuser or Operator to manually enter a new password for the user and then share that with the affected user so that they could in turn reconfigure a different password for themselves once they were successfully logged back in.

• Issue 82775: On a VMware SASE Orchestrator using Release 5.0.0, when a Zscaler type Cloud Security Service (CSS) is configured for a customer and associated with a VMware SD-WAN Edge, and then a Business Policy is configured with a CSS backhaul rule, a user is unable to change the CSS hash or encryption parameters for that CSS.

  The Orchestrator locks the user out from modifying the Zscaler CSS configuration once it is associated with an CSS Backhaul Business Policy.

  Workaround: The user needs to delete the CSS Backhaul Business Policy to modify the Zscaler CSS configuration and then recreate the same Business Policy.

• Issue 82864: On a VMware SASE Orchestrator using Release 5.0.0, when a user is on the Configure > Profiles page and selects 'Modify', the user is redirected to the Profile > Overview page instead of the Profile > Device Settings page.

  The Configure > Profiles 'Modify' button is not mapping to the correct page.

  Workaround: There is no workaround.

Cloud Web Security Known Issues

• Issue 62934: For an enterprise using VMware Cloud Web Security, if a client user opens a Chrome browser in Incognito and attempts to download a file, the download may occasionally not be successful.

  Incognito requires 3rd party cookies. Turn on 3rd party cookies and retry the operation. On an unsuccessful download the user would observe a screen which reads either "Error occurred contact your administrator" or for files from a custom web server: "This page is not working". Occasionally some web servers or Files may have a variance in File signature, that the Cloud Web Security Service may not be able to recognize, and hence this issue.

  Workaround: Turn on allowing 3rd party Cookies and retry. No known workaround for this issue if using an Incognito window.

• Issue 63149: When a customer deployment has overlapping subnets in a profile and configures a subnet for a VMware Cloud Web Security policy, and associates the Cloud Web Security policy to the profile and segment, Edge clients on that subnet will not be able to connect to the internet.

  If there are overlapping subnets configured for the LAN segments behind VMware SD-WAN Edges within the same segment, then the resources behind the Edges cannot have Cloud Web Security policies applied for the internet-bound traffic. This is because there is no way to uniquely identify the destination Edge for the return traffic from the internet to Cloud Web Security.

  Workaround: Turn on LAN side NAT on the Edge and have a unique subnet represent the traffic originated from resources behind the Edge.

• Issue 65001: For a customer using VMware Cloud Web Security, a user cannot configure the Inspection Engine to turn on/off File Hash checks when using the VMware Orchestrator to do so.

  When a user is using the Orchestrator to configure the Cloud Web Security Inspection engine's File Hash check parameter for either "Action for Unknown File Download" and "Action for unknown document Download", these changes are not sent to the Inspection Engine and are not applied.

  Workaround: There is no workaround for this issue.

Secure Access Known Issues

• Issue 64541: For a customer using VMware Secure Access, when using the option in Workspace ONE UEM Configuration to configure Tunnel Hostname within the Organization Group, if a user selects 'Yes', the hostname will be created in the UEM console automatically instead of being configured manually.

  The user should have the option to configure the hostname manually and not just have it automatically created. 

  Without the fix, the workaround is to manually set it in the UEM console.