This site will be decommissioned on January 30th 2025. After that date content will be available at techdocs.broadcom.com.

 

VMware SASE 4.5.1 | 01 May 2024

  • VMware SASE™ Orchestrator Version R451-20220831-GA

  • VMware SD-WAN™ Gateway Version R451-20220701-GA

  • VMware SD-WAN™ Edge Version R451-20230112-GA-87923

Check for additions and updates to these release notes.

What's in the Release Notes

The release notes cover the following topics:

This release is recommended for all customers who require the features and functionality first made available in Release 4.5.0, as well as those customers impacted by the issues listed below which have been resolved since Release 4.5.0.

Compatibility

Release 4.5.1 Orchestrators, Gateways, and Hub Edges support all previous VMware SD-WAN Edge versions greater than or equal to Release 3.2.0

Note: this means releases prior to 3.2.0 are not supported.

The following SD-WAN interoperability combinations were explicitly tested:

Orchestrator

Gateway

Edge

Hub

Branch/Spoke

4.5.0

4.5.1

4.5.0

4.5.0

4.5.0

4.5.1

4.5.1

4.5.1

4.5.0

4.5.0

4.5.1

4.5.0

4.5.1

3.4.6

3.4.6

3.4.6 

4.5.1

4.5.1

3.4.6

3.4.6

4.5.1

4.5.1

4.5.1

3.4.6

4.5.1

4.5.1

3.4.6

4.5.1

4.5.1

4.2.2

4.2.2

4.2.2

4.5.1

4.5.1

4.2.2

4.2.2

4.5.1

4.5.1

4.5.1

4.2.2

4.5.1

4.5.1

4.2.2

4.5.1

4.5.1

4.3.1

4.3.1

4.3.1

4.5.1

4.5.1

4.3.1

4.3.1

4.5.1

4.5.1

4.5.1

4.3.1

4.5.1

4.5.1

4.3.1

4.5.1

4.5.1

4.5.0

4.5.0

4.5.0

4.5.1

4.5.1

4.5.0

4.5.0

4.5.1

4.5.1

4.5.1

4.5.0

4.5.1

4.5.1

4.5.0

4.5.1

5.0.0

4.5.1

4.5.0

4.5.1

5.0.0

5.0.0

4.5.0

4.5.1

Note: The above table is only valid for customers using SD-WAN services. Customers requiring access to VMware Cloud Web Security or VMware Secure Access need their Edges upgraded to either Release 4.5.0 or 4.5.1.

Caution:

VMware SD-WAN Releases 3.2.x, 3.3.x, and 3.4.x have reached the End of Support.

  • Releases 3.2.x and 3.3.x reached End of General Support (EOGS) on December 15, 2021, and End of Technical Guidance (EOTG) March 15, 2022.

  • Release 3.4.x for the Orchestrator and Gateway reached End of General Support (EOGS) on March 30, 2022, and End of Technical Guidance (EOTG) on September 30, 2022.

  • Release 3.4.x for the Edge reached End of Support (EOGS) on December 31, 2022, and End of Technical Guidance (EOTG) on March 31, 2023.

  • For more information please consult the Knowledge Base article: Announcement: End of Support Life for VMware SD-WAN Release 3.x (84151)

Important:

VMware SD-WAN Release 4.0.x has reached End of Support; Releases 4.2.x, 4.3.x, and 4.5.x have reached End of Support for Gateways and Orchestrators.

  • Release 4.0.x reached End of General Support (EOGS) on September 30, 2022, and End of Technical Guidance (EOTG) December 31, 2022. 

  • Release 4.2.x Orchestrators and Gateways reached End of General Support (EOGS) on December 30, 2022, and End of Technical Guidance on (EOTG) March 30, 2023.   

  • Release 4.2.x Edges reached End of General Support (EOGS) on June 30, 2023, and will reach End of Technical Guidance (EOTG) September 30, 2025.

  • Release 4.3.x Orchestrators and Gateways reached End of General Support (EOGS) on June 30, 2023, and End of Technical Guidance (EOTG) September 30, 2023.

  • Release 4.3.x Edges reached End of General Support (EOGS) on June 30, 2023, and will reach End of Technical Guidance (EOTG) September 30, 2025.

  • Release 4.5.x Orchestrators and Gateways reached End of General Support (EOGS) on September 30, 2023, and End of Technical Guidance on (EOTG) December 31, 2023.

  • For more information please consult the Knowledge Base article: Announcement: End of Support Life for VMware SD-WAN Release 4.x (88319).

Note:

Note: Release 3.x did not properly support AES-256-GCM, which meant that customers using AES-256 were always using their Edges with GCM deactivated (AES-256-CBC). If a customer is using AES-256, they must explicitly deactivate GCM from the Orchestrator prior to upgrading their Edges to a 4.x Release. Once all their Edges are running a 4.x release, the customer may choose between AES-256-GCM and AES-256-CBC.

Upgrade Paths for Orchestrator, Gateway, and Edge

Orchestrator

Due to infrastructure changes in the Orchestrator beginning in Release 4.0.0, any Orchestrator using a 3.x Release needs to be first upgraded to 4.0.0 prior to being upgraded to 4.5.1. Orchestrators using Release 4.0.0 or later can be upgraded to Release 4.5.1.  Thus, the upgrade paths for the Orchestrator are as follows:

Orchestrator using Release 3.x → 4.0.0 → 4.5.1.

Orchestrator using Release 4.x → 4.5.1.

Gateway

Gateway upgrades from 3.x to 4.5.1 are not supported. In place of upgrading, a 3.x Gateway needs to be freshly deployed with the same VM attributes, and the old instance is then deprecated.

Upgrading a Gateway using Release 4.0.0 or later is fully supported for all Gateway types.

Edge

An Edge can be upgraded directly to Release 4.5.1 from any Release 3.x or later.

Important Notes

VMware Security Advisory 2024-0008

Expanded NAT Range for SD-WAN Gateways

Some SD-WAN Gateways may face the limitations of the original NAT range as they grow and handle more traffic. For example, the original NAT range may not be enough for Microsoft 365 (formerly Office 365) and some other services. The original NAT range only used ports 20001-64000, which allowed for 44K translations to a specific destination. To solve this problem, Gateway Release 4.5.1 and 5.0.0 change the Gateway NAT range to ports 5000-65534. This allows for ~60K translations to a specific destination.

Potential Issue With Sites Using a High-Availability Topology

A site where a pair of Edges are deployed in a High-Availability topology may encounter an issue where the Standby Edge reboots one or more times to resolve an Active-Active state. The Standby Edge reboot(s) can cause a disruption of customer traffic with the impact greater on sites using an Enhanced HA topology as the Standby Edge also passes customer traffic. The issue is being tracked by Issue #85369, which is fixed in the 1st rollup build for Release 4.5.1: R451-20220701-GA. The issue is tracked under the Edge/Gateway Resolved Issues section for R451-20220701-GA in these Release Notes and it is strongly recommended that customers with HA sites upgrade their Edges to R451-20220701-GA.

Accessing Cloud Web Security and Secure Access

A customer wishing to access VMware Cloud Web Security or VMware Secure Access must upgrade their Edges to Release 4.5.0 or later.  These services are inaccessible on Edges using a release earlier than 4.5.0.

BGPv4 Filter Configuration Delimiter Change for AS-PATH Prepending

Through Release 3.x, the VMware SD-WAN BGPv4 filter configuration for AS-PATH prepending supported both comma and space based delimiters. However, beginning in Release 4.0.0 and forward, VMware SD-WAN only supports a space based delimiter in an AS-Path prepending configuration.Customers upgrading from 3.x to 4.x need to edit their AS-PATH prepending configurations to "replace commas with spaces" prior to upgrade to avoid incorrect BGP best route selection.

Extended Upgrade Time for Edge 3x00 Models

Upgrades to this version may take longer than normal (3-5 minutes) on Edge 3x00 models (i.e., 3400, 3800 and 3810). This is due to a firmware upgrade which resolves issue 53676. If an Edge 3400 or 3800 had previously upgraded its firmware when on Release 3.4.5/3.4.6, 4.0.2, 4.2.1, or 4.3.0 then the Edge would upgrade as expected. For more information, please consult Fixed Issue 53676 in the respective release notes.

Limitation with BGP over IPsec on Edge and Gateway, and Azure Virtual WAN Automation

The BGP over IPsec on Edge and Gateway feature is not compatible with Azure Virtual WAN Automation from Edge or Gateway. Only static routes are supported when automating connectivity from an Edge or Gateway to an Azure vWAN.

Limitation When Deactivating Autonegotiation on VMware SD-WAN Edge Models 520, 540, 620, 640, 680, 3400, 3800, and 3810

When a user deactivates autonegotiation to hardcode speed and duplex on ports GE1 - GE4 on a VMware SD-WAN Edge model 620, 640 or 680; on ports GE3 or GE4 on an Edge 3400, 3800, or 3810; or on an Edge 520/540 when an SFP with a copper interface is used on ports SFP1 or SFP2, the user may find that even after a reboot the link does not come up.

This is caused by each of the listed Edge models using the Intel Ethernet Controller i350, which has a limitation that when autonegotiation is not used on both sides of the link, it is not able to dynamically detect the appropriate wires to transmit and receive on (auto-MDIX). If both sides of the connection are transmitting and receiving on the same wires, the link will not be detected. If the peer side also does not support auto-MDIX without autonegotiation, and the link does not come up with a straight cable, then a crossover Ethernet cable will be needed to bring the link up.

For more information please see the KB article Limitation When Deactivating Autonegotiation on VMware SD-WAN Edge Models 520, 540, 620, 640, 680, 3400, 3800, and 3810 (87208).

Mixing Wi-Fi Capable and Non-Wi-Fi Capable Edges in High Availability Is Not Supported 

Beginning in 2021, VMware SD-WAN introduced Edge models which do not include a Wi-Fi module: the Edge models 510N, 610N, 620N, 640N, and 680N. While these models appear identical to their Wi-Fi capable counterparts except for Wi-Fi, deploying a Wi-Fi capable Edge and a Non-Wi-Fi capable Edge of the same model (for example, an Edge 640 and an Edge 640N) as a High-Availability pair is not supported. Customers should ensure that the Edges deployed as a High Availability pair are of the same type: both Wi-Fi capable, or both Non-Wi-Fi capable.

Document Revision History

April 12th, 2024. Thirty-Third Edition.

  • Added Open Issue #142366 to the Edge/Gateway Known Issues section.

April 2nd, 2024. Thirty-Second Edition.

  • Added an Important Note regarding CVE-2024-22247, which details a missing authentication and protection mechanism vulnerability that impacts an SD-WAN Edge. VMware's response to this vulnerability is documented in VMSA-2024-0008. More information on mitigating this vulnerability is found in the KB article: VMware Response to CVE-2024-22247 (VMSA-2024-0008) (97391).

March 26th, 2024. Thirty-First Edition.

March 5th, 2024. Thirtieth Edition.

November 17th, 2023. Twenty-Ninth Edition.

  • Corrected errors where the symbol > was not presented correctly.

September 6th, 2023. Twenty-Eighth Edition.

  • Added a new Important Note: Expanded NAT Range for SD-WAN Gateways. This note should have been added in the first edition of these notes and was omitted in error.

  • Document Revision History reorganized to read from newest entries to oldest for an improved user experience.

July 19th, 2023. Twenty-Seventh Edition.

  • Added Open Issue #94430 to the Edge/Gateway Known Issues section.

  • Removed Open Issue #53359 from the Edge/Gateway Known Issues section as this was fixed in 4.3.0.

  • Updated the Compatibility section to indicated which 4.x versions had reached End of General Support.

July 10th, 2023. Twenty-Sixth Edition.

  • Added Fixed Issue #97743 to the Edge/Gateway Resolved Issues for the R451-20220916-GA build which was the 2nd Edge rollup. This ticket was omitted in error from the Twenty-Seventh Edition Edition of the 4.5.1 Release Notes.

  • Added Issues #103662 and #112131 to the Edge/Gateway Known Issues section.

June 14th, 2023. Twenty-Fifth Edition.

  • Added Issues #90884 and #113291 to the Edge/Gateway Known Issues section.

April 25th, 2023. Twenty-Fourth Edition.

  • Updated the Compatibility section to mark all 3.x releases as having reached their End of Service Life (EOSL). Also updated the 4.x section to mark 4.2.x Orchestrators and Gateways as End of Service Life (EOSL).

March 14th, 2023. Twenty-Third Edition.

  • Added Issue #75593 to the Edge/Gateway Known Issues section.

February 17th, 2023. Twenty-Second Edition.

  • Removed Issue #39659 from the Edge/Gateway Known Issues section as this is a duplicate of another ticket, #39501 which was resolved in Release 4.3.0.

February 2nd, 2023. Twenty-First Edition.

  • Added #78003 to the Edge/Gateway Resolved Issues section for the original build R451-20220513-GA. This issue was omitted in error from the original 4.5.1 Release Notes.

  • Added #98979 to the Edge/Gateway Known Issues section.

January 30th, 2023. Twentieth Edition.

  • Revised Known Issue #89217 to reflect a revised Edge version (R5012-20230123-GA-103475) and Platform Firmware version (R131-20221216-GA) needed to resolve the issue. The ticket also adds a link to the KB Article that covers #89217 and which includes step-by-step instructions for upgrading a 6x0 Edge.

  • In the Compatibility section, revised the Import Note regarding End of Support for 4.2.x and added Release 4.3.x to reflect newly revised dates for the SD-WAN Edge software.

January 19th, 2022. Nineteenth Edition.

  • Added a new Edge Hotfix build R451-20230112-GA-87923 to the Edge/Gateway Resolved section and is the new Edge GA build for Release 4.5.1. Edge Hotfix build R451-20230112-GA-87923 includes the fix for issue #87923, which is documented in this section. 

  • The Gateway build remains unchanged.

December 19th, 2022. Eighteenth Edition.

  • Added a new Edge rollup build R451-20221213-GA to the Edge/Gateway Resolved section. This is the third Edge rollup build and is the new Edge GA build for Release 4.5.1. The Gateway build remains unchanged.

  • Edge build R451-20221213-GA includes the fixes for issues #68335, #77342, #87552, #91365, #94612, #96994, #97483, #97559, #100089, and #101049, which are each documented in this section. 

  • Issue #91365 was removed from the Important Notes section and from the Known Issues section.

  • Added #78050, #81353, #84501, #89235, #95399, #97997, and #100005 to the Edge/Gateway Known Issues section.

December 7th, 2022. Thirtieth Edition.

  • Added Fixed Issue #76508 to the Orchestrator Resolved Issues section for the original build R451-20220517-GA and was omitted from the first edition of the Release Notes in error.

November 29th, 2022. Seventeenth Edition.

  • Added #76837, #82104,#97321, and #97559 to the Edge/Gateway Known Issues section.

October 5th, 2022. Sixteenth Edition.

  • Added #98136 to the Edge/Gateway Known Issues section.

  • Added #75117 to the Edge/Gateway Resolved Issues section for the original build R451-20220513-GA. This issue was omitted in error from the original 4.5.1 Release Notes.

September 23rd, 2022. Fifteenth Edition.

  • Added a new Edge rollup build R451-20220916-GA to the Edge/Gateway Resolved section. This is the second Edge rollup build and is the new Edge GA build for Release 4.5.1. The Gateway build remains unchanged.

  • Edge build R451-20220916-GA includes the fixes for issues #86098, #93383, #94204, #94395, #95565, #96441, and #96888, which are each documented in this section.

  • Added #87982 to the Edge/Gateway Known Issues section.

September 14th, 2022. Fourteenth Edition.

  • Added Issue #91875to the Edge/Gateway Known Issues section.

September 9th, 2022. Thirteenth Edition.

  • Added an updated Orchestrator 3rd rollup build R451-20220831-GA to the Orchestrator Resolved section. Build R451-20220831-GA replaces the original 3rd Orchestrator rollup build R451-20220810-GA, and is the new Orchestrator GA build for Release 4.5.1. This updated build does not add any new fixed issues but does contain troubleshooting changes required by VMware Engineering.

  • Added Issue #87552 and #93383 to the Edge/Gateway Known Issues section.

August 26th, 2022. Twelfth Edition.

  • Moved Issue #86808 from the Edge/Gateway Known Issues to Edge/Gateway Resolved Issues under the original GA build R451-20220513-GA.  This issue was misclassified as not fixed for Release 4.5.1 when the fix for it had been included in the original Release 4.5.1 GA Edge build R451-20220513-GA.

  • Removed Open Issue #49712 from Edge/Gateway Known Issues as Engineering concluded it was caused by a configuration error versus a defect in the code.

August 16th, 2022. Eleventh Edition.

  • Added Open Issue #89217 to the Edge/Gateway Known Issues section.

August 11th, 2022. Tenth Edition.

  • Added a new Orchestrator rollup build R451-20220810-GA to the Orchestrator Resolved section. This is the third Orchestrator rollup build and is the new Orchestrator GA build for Release 4.5.1.

  • Orchestrator rollup build R451-20220810-GA includes fixed issues #80735, #83507, #88120, #90067, #91179, and #92082, which are all documented in this section.

August 2nd, 2022. Ninth Edition.

  • Added a new Important Note: "Issue with Edges Using Edge Network Intelligence" regarding Known Issue #91365 where customers using the Edge Network Intelligence service would experience where any Edge using Analytics will restart every 3-4 days due to a memory leak. Issue #91365 is fixed in a new hotfix build R451-20220720-GA-91365 and continues to be tracked in Edge/Gateway Known Issues.

  • Added Fixed Issues #84214 and #86546 to the Orchestrator Resolved Issues section for Orchestrator rollup build R451-20220520-GA. Both tickets were omitted in error from the initial publication of this first Orchestrator rollup build.

July 14th, 2022. Eighth Edition.

  • Added a new Edge/Gateway rollup build R451-20220701-GA to the Edge/Gateway Resolved section. This is the first Edge/Gateway rollup build and is the new Edge/Gateway GA build for Release 4.5.1.

  • Edge/Gateway build R451-20220701-GA includes the fixes for issues #64196, #78568, #83083, #85369, #85461, #87304, #89627, #89725, #89861, #89873, #90151, #90280, #91746, and #92216, which are each documented in this section.

  • Added Open Issues #81859, #91365, #92481,and #92676 to the Edge/Gateway Known Issues section.

  • Revised the Important Note: "Potential Issue With Sites Using a High-Availability Topology" to now indicate that issue #85369 is fixed in the new Edge build R451-20220701-GA and that customers with HA sites are recommended to upgrade their Edges to this new build as soon as possible.

July 1st, 2022. Seventh Edition.

  • Added Open Issues #88604 and #91746 to the Edge/Gateway Known Issues section.

June 17th, 2022. Sixth Edition.

  • Added a new Orchestrator rollup build R451-20220612-GA to the Orchestrator Resolved section. This is the second Orchestrator rollup build and is the new Orchestrator GA build for Release 4.5.1.

  • Orchestrator rollup build R451-20220612-GA includes fixed issues #86848, #89800, which is documented in this section.

  • Added three new tested and verified interoperability combinations to the Compatibility table.  All three combinations involve 4.5.0 Orchestrator and Gateway software and are listed at the top of the table.

June 8th, 2022. Fifth Edition.

  • Added a new Important Note: "Potential Issue With Sites Using a High-Availability Topology" regarding ongoing issues with customer sites using a High-Availability topology for a pair of Edges. This issue continues to be tracked by Issue #85369 located in Edge/Gateway Known Issues.

  • Under Compatibility, amended the End of Life dates for Release 4.2.x Edge software. The Edge software is broken out as a separate item and now reads: "Release 4.2.x Edges will reach End of General Support (EOGS) on June 30, 2023, and End of Technical Guidance (EOTG) September 30, 2023."  The separate Orchestrator and Gateway entry retains the same End of Life dates as before.

June 1st, 2022, Fourth Edition.

  • Added Fixed Issue #81835 to the Orchestrator Resolved Issues for the initial GA build. This ticket was omitted in error from the First Edition of the 4.5.1 Release Notes.

  • Added Issues #85369 and #85461 to the Edge/Gateway Known Issues section.

May 24th, 2022. Third Edition.

  • Added a new Orchestrator rollup build R451-20220520-GA to the Orchestrator Resolved section. This is the first Orchestrator rollup build and is the new Orchestrator GA build for Release 4.5.1.

  • Orchestrator rollup build R451-20220520-GA includes a fix for issue #89349, which is documented in this section.

May 20th, 2022. Second Edition.

  • Added Open Issue #89349 to the Orchestrator Known Issues section. This is a cosmetic issue where a user deploying a 4.5.1 Orchestrator using the R451-20220517-GA build would see the Beta license agreement warning on the login page. R451-20220517-GA is not a Beta build and this messaging was mistakenly left in for this final GA build. This issue will be corrected with the upcoming first Orchestrator rollup build.

May 18th, 2022. First Edition.

Edge and Gateway Resolved Issues

Resolved in Edge Version R451-20230112-GA-87923

Edge version R451-20230112-GA-87923 was released on 01-19-2023 and is a Hotfix build for Release 4.5.1.

This Edge Hotfix build addresses the below critical issue since the 3rd Edge rollup build, version R451-20221213-GA.

  • Fixed Issue 87923: When a malformed ICMP packet is sent to a VMware SD-WAN Edge, the Edge may experience a Dataplane Service failure and restart as a result.

    The Edge does not validate the IP packet length (for example, an ICMP packet with a IP packet length of 24) and this can lead to a memory corruption of the Edge which triggers the Dataplane Service failure and restart.

Resolved in Edge Version R451-20221213-GA

Edge version R451-20221213-GA was released on 12-19-2022 and is the 3rd Edge rollup for Release 4.5.1.

This Edge rollup build addresses the below critical issues since the 2nd Edge rollup build, version R451-20220916-GA.

  • Fixed Issue 68335: SD-WAN control traffic consumes high bandwidth when the Edge fails to connect to datacenter clusters.

    When an Edge fails to establish overlays with its datacenter clusters, the Edge requests controllers to reassign a different cluster member, leading to controllers sending continuous control events and consuming link bandwidth.

  • Fixed Issue 77342: In the case of VRRP, when the Edge is primary, the cloud traffic from the client is dropped.

    The issue is due to a faulty cloud route in the Edge route table. As a result, the ARP for the cloud IP fails because there is no next hop, and the packet is dropped with an ipv4-send-encap-error.

  • Fixed Issue 87552: When a VMware SD-WAN Edge is configured to use either a Non SD-WAN Destination (NSD) via Edge or a Cloud Security Service (CSS), the VMware SD-WAN Edge may periodically experience a Dataplane Service failure and restart when NSD or CSS tunnels are unstable.

    When the Edge tears down a tunnel to either an NSD or a CSS (IPsec or GRE), the incorrect release of a previously chosen tunnel is performed that triggers an exception in the Edge Dataplane Service and an Edge Service restart to restore the service. Restarting the Edge service will result in a 10-15 second disruption of customer traffic.

    On an Edge without a fix for this issue, associating an NSD via Edge or CSS to one WAN link will decrease the likelihood of this issue occurring. In other words, instead of configuriing an NSD or CSS on multiple WAN links, choose one WAN link only. This will lose redundancy but will mitigate the impact of this issue.

  • Fixed Issue 91365: For a customer using Edge Network Intelligence, an VMware SD-WAN Edge where Analytics is configured experiences a memory leak that will result in the Edge triggering an Edge Service restart to clear the memory.

    When the Analytics function is enabled on an Edge, the Edge’s Dataplane service begins leaking memory at a steady rate that will result in the Edge needing to trigger an unscheduled Service Restart to clear the memory leak when it reaches a critical level (60% memory utilization for longer than 90 seconds). An Edge Service restart causes a 10-15 second disruption in customer traffic. In the field, the time it takes to trigger an Edge Service restart has been ~3 to 4 days, and once the memory is cleared the memory leak will resume with the same general time window for the next Edge Service restart. The period when the Edge would reach a critical memory usage level depends on the Edge model and the amount of information the Analytics feature is recording for that Edge.

  • Fixed Issue 94612: Traffic to the BGP network prefix is not reachable.

    The BGP network prefix failed to be configured under BGP and is not advertised to peer nodes.

  • Fixed Issue 96994: When doing an SNMP walk on a VMware SD-WAN Edge, some of the interfaces might not be visible.

    Missing interfaces can be valid interfaces which were meant to be visible on snmpwalk. However, due to the appearance of an invalid interface in the Edge's hardware list the valid interfaces appearing after the invalid one in the list will not be visible or returned by snmpwalk. An interface here is invalid if it appears in the hardware list but does not appear when running the ifconfig command on the Edge.

    While this issue can potentially affect any Edge, it has a greater chance of being encountered on a Virtual Edge deployed using Azure. This is due to the Azure Edge's tendency to list a higher number of interfaces on its hardware list versus the number of interfaces identified in the Edge itself using ifconfig.

  • Fixed Issue 97483: The throughput of 2 core Virtual Edges does not cross 500Mbps (in tx direction) despite having adequate CPU power.

    The throughput of 2 core Virtual Edges was soft capped to 500Mbps to prevent it from being overloaded. However, enhancements in the Edge software have now made it possible for 2 core Virtual Edges to handle more traffic without being overloaded, which causes the cap to be too limiting. The cap has now been raised to 1000M.

  • Fixed Issue 97559: On a customer site deployed with an Enhanced High-Availability topology, a WAN link connected to the VMware SD-WAN Edge in a Standby role may show as down on the VMware SASE Orchestrator and not pass customer traffic even though the Edge's WAN interface where the WAN link is connected is up.

    A user looking at a tcpdump or diagnostic bundle logging would observe ARP requests coming in and the Standby Edge not responding as a result of its port being blocked.

    In Enhanced HA, when an Edge assumes the role of Standby, the following events should occur in sequence:

    1. The Standby Edge blocks all ports.

    2. The Standby Edge then detects that it is deployed in Enhanced HA and unblocks its WAN ports to pass traffic.

    When this issue occurs, Event 1, the initial port blocking takes an unexpectedly long time to complete and the follow-up Event 2, the unblocking of all WAN ports is completed prior to the completion of Event 1. And then Event 1 completes and thus the final state is all WAN ports are blocked on the Standby Edge.

    On an HA Edge without a fix for this issue, the workaround is to force an HA failover that promotes the Standby Edge to Active brings up the HA Edge's WAN link(s).

  • Fixed Issue 100089: Unexpected routes are observed in VMware SD-WAN Edge OSPF/BGP neighbors.

    The internal management routes corresponding to vce1/gwd1(169.254.129.x) are redistributed into BGP/OSPF by the SD-WAN Edge and are received by the respective OSPF/BGP neighbors connected to the SD-WAN Edges.

  • Fixed Issue 101049: If a customer enterprise uses both secure and non-secure routes, a high path loss might be observed.

    An example of using both secure and non-secure routes is when a Partner Gateway is used and the SD-WAN Edge learns subnets from a BGP neighbor (non-secure), and the SD-WAN Edge learns those same subnets from another SD-WAN Edge in the network (secure). The secure route is preferred, but if the secure route is revoked, the traffic does not switch to the non-secure route. This issue was caused when the Edge service did not sequence the management tunnels responsible for proper routing.

Resolved in Edge Version R451-20220916-GA

Edge version R451-20220916-GA was released on 09-22-2022 and is the 2nd Edge rollup for Release 4.5.1.

This Edge rollup build addresses the below critical issues since the 1st Edge rollup build, version R451-20220701-GA.

  • Fixed Issue 86098: For a site using an Enhanced High-Availability topology where a PPPoE WAN link is used on the Standby Edge, a user may observe that the default proxy route is not installed in the Active Edge and traffic using that link fails.

    When an Enhanced HA Edge pair come up, the PPPoE link synchronizes with the Standby Edge and provides a default route with a next hop of 0.0.0.0. As a result this route is not installed on the Active and traffic using this link is dropped.

  • Fixed Issue 93383: A VMware SD-WAN Edge may suffer one or more Dataplane Service failures with a disruption in customer traffic.

    The issue is caused by a rare instance of a mismatch of the number of interfaces stored in the Edge in two different data structures which triggers an exception and results in the Edge service failing one or more times. The Edge service needs to restart to recover which, in a non-HA deployment, would cause 10-15 seconds of customer traffic disruption. However, if the Edge service fails three consecutive times, the Edge will require a reboot or power cycle to recover.

  • Fixed Issue 94204: A user may observe that attempts to generate a diagnostic bundle for a VMware SD-WAN Edge fail.

    The Edge diagnostic bundles fail to complete because the Edge runs out of disk space. This can happen if the Edge has generated one or more cores and is caused by the Edge sending these cores to the /vnf/tmp folder. Each core is unpacked in the /vnf/tmp folder and due to a core's unpacked size quickly fills this folder which causes the diagnostic bundle to fail.

  • Fixed Issue 94395: On a site deployed with a High Availability topology, HA failover may fail as the Standby Edge is not moved to an Active state after the Active Edge has failed, resulting in a disruption of customer traffic.

    This issue can be encountered when more than one pair of HA Edges are connected to the same upstream WAN switches or broadcast network. In this scenario, the HA Edges may process non-peer HA WAN heartbeats and this affects the local HA state and leads to indeterministic HA behavior, including the Standby Edge not being promoted to Active.

    On an HA pair which does not have a fix for this issue, the workaround for this issue is to avoid sharing the same broadcast network between two different HA pairs.

  • Fixed Issue 95565: On a site using a High Availability topology, the VMware SD-WAN Active Edge may experience a Dataplane Service failure with a core generated and triggering a High Availability failover.

    The issue is triggered by the Active Edge's WAN links flapping one or more times (going down and then come up rapidly) while also using SNMP where there are frequent SNMP queries. There is a timing issue where the interface coming back up and the SNMP query together can trigger a deadlock which causes the Dataplane Service to fail and generate a core. While only a single WAN link flap can cause this issue, the greater the frequency of WAN link flaps, the greater the potential for this issue to occur.

    On an HA Edge pair that experiences this issue and does not have the fix, the workaround is to disable SNMP as this is a timing issue and this reduces the risk.

  • Fixed Issue 96441: On a site using a High Availability Topology, the customer may observe frequent HA failovers.

    The issue is triggered by the HA interface being marked by the Edge as down and then coming back up within 500-1000ms which can trigger an HA failover. However, these interface down events are spurious and caused by a DPDK-enabled interface using polling with an interval of 500ms to determine interface status. Using this method, the underlying device driver can sometimes report a spurious interface down event and each event causes the Edge to mark the interface as down until the next poll of the interface status (in 500ms) reports that the interface is up.

  • Fixed Issue 96888: In certain load conditions, the routing protocols for either BGP or OSPF may randomly restart, leading to route re-convergence and traffic disruption.

    Under higher load conditions the BGP and OSPF routing protocol processes are made to wait longer than expected by the Edge CPU to get scheduled and this leads to a stall and restart of the routing protocol. The routing protocol delay is caused by insufficient CPU bandwidth allocation and can occur on any Edge model.

  • Fixed Issue 97743: On a customer enterprise deployed with a High-Availability topology, when either a VMware SD-WAN HA Edge's LAN or WAN interface flaps the Active Edge may become unresponsive.

    When the Active Edge detects that its Standby Edge peer has a greater number of LAN/WAN interfaces, it moves to a Standby state and becomes unresponsive or is deadlocked with a 100% CPU utilization. The only way to recover the affected HA Edge is to reboot it.

Resolved in Edge/Gateway Version R451-20220701-GA

Edge/Gateway version R451-20220701-GA was released on 07-08-2022 and is the 1st Edge/Gateway rollup for Release 4.5.1.

This Edge/Gateway rollup build addresses the below critical issues since the original 4.5.1 Edge/Gateway GA build, version R451-20220513-GA.

  • Fixed Issue 64196: For a VMware SD-WAN Edge where one or more BGP policies with outbound filters are configured, if a configuration change is made to an outbound filter, the routes will flap and the outbound filters may be applied inconsistently with customer traffic disrupted.

    A configuration change can be either modifying an existing filter or adding a new outbound filter. When encountering this issue the customer may observe a BGP neighbor employing a BGP policy with an outbound filter which is denying all routes when it is configured to allow all routes to be advertised, resulting in significant customer traffic issues. This issue is caused by the BGP neighbor associating with a stale route map, versus the updated map resulting from the configuration change.

    On an Edge without the fix for this issue, if the Edge Service is restarted through the Orchestrator using Remote Actions > Service Restart, all routes are restored post-restart.

  • Fixed Issue 78568: For a customer using BGP and connected to VMware SD-WAN Partner Gateways, a Partner Gateway may continue to advertise a VMware SD-WAN Edge's VLAN subnet after that subnet's advertise flag is set to False.

    The routes continue to be advertised because when the Edge breaks the BGP neighbor adjacency state with an L3 BGP neighbor, one of the connected Partner Gateways maintains ownership of the Edge VLAN subnet. Stale routes on a Partner Gateway negatively affect customer traffic and can lead to entire customer flows getting dropped because traffic is routed to now non-existent routes.

    Without a fix, the only way to remediate the issue and clear the stale BGP routes is for a Partner or Operator to restart the Partner Gateway service in a suitable maintenance window.

  • Fixed Issue 83083: A VMware SD-WAN Gateway upgraded to Release 4.3.1 or later may experience a slow memory leak which can lead to the Gateway's service restarting to clear the memory.

    Gateway restarts can be disruptive to customer traffic for the 30-45 seconds it takes the for the Gateway service to restart. Each time an Operator user runs the debug.py --flow_dump all all all command on the Gateway, the Gateway will leak some of its memory. Running this debug command a sufficient number of times will cause the Gateway's memory usage to reach a critical level and trigger a Gateway service restart to clear the memory.

    For a Gateway without the fix, an Operator must avoid running the debug.py --flow_dump all all all command on the Gateway. If using this debug command is unavoidable, monitor the memory usage and schedule maintenance windows to preemptively restart the service to clear the memory prior to an unscheduled restart.

  • Fixed Issue 85369: For a site deployed with a High-Availability topology, the customer may observe traffic disruptions and possibly multiple reboots of the VMware SD-WAN Standby Edge.

    A condition triggered by load and system events causes the Active Edge to experience delays in the timely delivery of HA heartbeats to the Standby Edge. The delay causes the Standby Edge to miss heartbeats and incorrectly assume the Active role causing an Active-Active state. To recover from the Active-Active state the Standby Edge reboots, possibly multiple times.

    If the site does become Active-Active, a conventional HA setup would experience minimal traffic disruption since the Standby Edge does not pass traffic in this topology, but on an Enhanced HA deployment, where the Standby Edge is also passing traffic, the reboot(s) would disrupt some customer traffic.

  • Fixed Issue 85461: If a VMware SD-WAN Edge is used to forward DNS, and LAN devices connected to the Edge are using the Edge for DNS forwarding, all DNS traffic may fail.

    All DNS forwarding traffic is affected, not just Conditional DNS. Depending on the Edge software, this issue can be encountered on an Edge as follows:

    • If the Edge is using Release 4.2.2, the Edge can encounter this issue if the Edge is using routed LAN ports with no Gateway IP address specified. Switched LAN ports + VLANs are not affected in 4.2.2.

    • If the Edge is using either Release 4.3.0/4.3.1, 4.5.0/4.5.1, or 5.0.0.x, the Edge can encounter the issue if the Edge is using switched LAN ports and VLANs, or the Edge is using routed LAN ports with no Gateway IP address specified.

    For switched interfaces, the cause of the issue stems from the deprecation and removal of the Management IP interface in favor of a loopback interface in Releases 4.3.x, 4.5.x, and 5.0.0.x and later. Because DNS uses segment NAT, the DNS packet has no matching entry for the destination IP when the Edge does segment NAT table lookup and the Edge drops the packet.

    For routed interfaces, the lack of a Gateway IP means the DNS packet is routed to the Edge as the next hop and the Edge does not forward the DNS packet further.

    Without a fix for this issue, the workaround for this issue is to either not use the Edge to forward DNS, or...

    • When using Edge Release 4.2.2: use either switched LAN ports or routed LAN ports that include a Gateway IP address.

    • When using Release 4.3.x or 4.5.x, use only routed LAN ports with a Gateway IP address specified.

  • Fixed Issue 87304: If a LAN interface on a VMware SD-WAN Edge is deactivated on the VMware SASE Orchestrator, the interface will still be reported as 'UP' by SNMP.

    Description: The key debug process for interfaces output does not include the physical port details for Edge LAN interfaces (for example, GE1 or GE2). As a result when SNMP polls those interfaces it always returns a result of UP regardless of how these interfaces are configured.

  • Fixed Issue 89627: On a VMware SD-WAN Gateway where the Telegraf service is used, a user may observe an escalating level of memory usage which would eventually lead to a Gateway Service restart to clear the memory.

    When Telegraf is used, a set of metrics are fetched from the Gateway service and exported. During the the data fetch operation, a small amount of memory is leaked and the memory used for storing the metrics is not freed.

    Without the fix, the workaround is either shutting off the Telegraf service to prevent the memory leak, or to carefully monitor memory usage on the Gateway. When memory utilization reaches ~60%, schedule a preemptive Gateway Service Restart to clear the memory. The second workaround buys time until updating the Gateway to a build with a fix for the issue while still using Telegraf.

  • Fixed Issue 89725: For VMware SD-WAN Edges using Edge software release 4.5.1 GA, the Remote Diagnostic utilities WAN Link Bandwidth Test and Traceroute do not work.

    The WAN Link Bandwidth Test and Traceroute utilities require additional input for the interface (for BW Test) or IP Address (for Traceroute). With this issue, the user cannot configure those inputs as the dropdown menu option is not shown and thus the test in either instance results in an error.

  • Fixed Issue 89861: A user may observe a steady increase in memory utilization on a VMware SD-WAN Edge where there is an extensive use of Object Groups with Business Policies.

    When using Object Groups with Business Policies, on every update done to the Object Group on the Edge, the Edge's memory objects pertaining to the address group are leaked. And if these Object Group updates are done regularly (for example, using a script), the memory utilization will steadily increase until reaching a critical level (60% memory utilization for longer than 90 seconds) and triggering an Edge Service restart to recover the memory. An unplanned Edge Service restart to clear the memory can cause a brief disruption in customer traffic. This impact is more commonly observed on entry level Edge models (for example, the 510, 610, or 620) which have smaller amounts of Edge memory, but over a long enough period every Edge model could reach a critical memory level and restart.

    Without a fix for this issue, the workaround would be to limit the frequency of Object Group updates or, if that is not possible, regularly monitor the memory and when memory utilization reaches 40% and you receive a Memory Warning Event on the Orchestrator, schedule a Edge Service Restart in a maintenance window to clear the memory and ensure minimal customer impact.

  • Fixed Issue 89873: A user may observe an increase in memory utilization on a VMware SD-WAN Edge resulting in a Memory Usage Warning Event on the Orchestrator and potentially an unscheduled Edge Service restart to recover the Edge's memory.

    This issue occurs when UDP flows with unique IP address and ports are processed at a high rate on the Edge. Flow creation is handled asynchronously on the Edge and when multiple packets of a same flow are enqueued to the flow creation service, the flow objects are leaked and result in an Edge memory leak. The impact is more commonly observed on entry level Edge models (for example, the 510, 610, or 620) which have smaller amounts of Edge memory, but over a long enough period every Edge model could reach a critical memory level (60% memory utilization for longer than 90 seconds) and restart. An unplanned Edge Service restart to clear the memory can cause a brief disruption in customer traffic.

    Without a fix for this issue, the only way to prevent this issue impacting a customer site is to monitor the memory. When memory utilization reaches 40% and the Orchestrator records a Memory Warning Event, schedule a Edge Service Restart in a maintenance window to clear the memory and ensure minimal customer impact.

  • Fixed Issue 90151: For BGP over IPsec on Gateway, applying different BGP filters to primary and secondary neighbors may not work as expected.

    When different filters are applied to a Non SD-WAN Destination (NSD)-BGP on the VMware SD-WAN Gateway's primary and secondary neighbors, only one gets applied to both BGP neighbors.

    The cause of this issue is that for Partner Gateway (PG)-BGP, the SD-WAN service identifies BGP filters using a combination of enterprise_logical_id and segment_id and using enterprise_logical_id was sufficient for Partner Gateway-BGP, because for a given enterprise-segment combination, we could only have 1 PG-BGP neighbor.

    However, this method was inherited for NSD-BGP on the Gateway where there can be up to 2 BGP neighbors (Primary and Secondary) for the same enterprise-segment combination. As a result, the enterprise_logical_id and segment_id combination does not suffice for differentiating between filters of 2 different NSD-BGP neighbors.

  • Fixed Issue 90280: On a site deployed with a High Availability topology and configured to use Dynamic Edge-to-Edge, the VMware SD-WAN HA Edge may fail over unexpectedly.

    This issue can be encountered at a site that has a high rate of creation and destruction of dynamic tunnels between other Edges. In such a scenario the Edge may incorrectly account for the number of interfaces that are up which can lead the Edge service to conclude that all the links are down and trigger an HA failover.

  • Fixed Issue 91746: A VMware SD-WAN Edge using either wired or wireless 802.1x authentication (for example, RADIUS, Cisco ISE) may experience certificate authentication failures with all traffic requiring this authentication dropping at the Edge.

    The issue is caused by the Edge improperly altering the L4 headers of IP fragmented packets which results in the packets becoming corrupted before exiting the Edge. This primarily impacts UDP packets and as these packets are used for 802.1x certificate authentication has the potential to cause 802.1x wired or wireless clients to fail.

    Workaround: On an Edge where this issue is encountered, the workarounds are to either, a) turn off 802.1x authentication, or b) roll the Edge back to a prior Edge software build where 802.1x authentication worked properly because this issue is not present.

  • Fixed Issue 92216: A user may observe an alert indicating that a VMware SD-WAN Edge is using 60% of its specified tunnel limit.

    This "Soft Cap Limit Alarm" for Edge Tunnels is confusing to the customer as there is no actual cap at 60% utilization. The only meaningful cap is the maximum specified tunnel limit for an Edge model. Edge tunnel limits for all Edge models can be found in the VMware SD-WAN Edge Platform Specifications datasheet, a download link to the most current datasheet is located at the bottom of the VMware SD-WAN Hardware Guide page.

Resolved in Edge/Gateway Version R451-20220513-GA

The below issues have been resolved since Edge Version R450-20220413-GA, and Gateway Version R450-20211123-GA-74198-70416-74482-30022.

Note:

Release 4.5.1 contains all Edge and Gateway fixes that are listed in the 4.5.0 Release Notes.

  • Fixed Issue 35807: A DPDK routed interface will be deactivated completely if the interface is toggled off and then on from the VMware SD-WAN Orchestrator.

    Deactivating a routed port unbinds that network device from the DPDK controlled hardware and rebinds it back to the Linux driver and the Edge service restarted. The /opt/vc/etc/dpdk.json file is updated to remove this interface, so on reactivation, the file is not updated to unbind from Linux back to the DPDK controlled driver.

    Without the fix, the only way to remediate the issue is to reboot the Edge to clear the state and back to a default boot state to re-add devices to the edge DPDK layer.

  • Fixed Issue 48017: OSPF and BGP routes may take a longer than expected time to converge on the Overlay Flow Control (OFC).

    Under high load, a situation may arise where some or all the routes learned on a VMware SD-WAN Edge may not show up on the OFC or get the necessary advertise and preference value assigned (this is with Dynamic Cost Calculation (DCC) not used). This can lead to the Edge constantly retrying the syncing of such routes to the VMware SD-WAN Orchestrator that will further increase the load on the Orchestrator.

  • Fixed Issue 49787: A user navigating to the Remote Diagnostics page for a VMware SD-WAN Edge on the VMware SD-WAN Orchestrator UI may observe the UI processing the request but the Edge's diagnostics page does not load.

    The issue can occur for an Edge where certificates are not used and is the result of the VMware SD-WAN Edge's connection being continually reset because the Edge is repeatedly renewing its certificate.

  • Fixed Issue 51036: ifStats reports the wrong value when polling a VMware SD-WAN Edge via SNMP if the Edge interfaces are configured to use DPDK.

    This is an expected behavior for DPDK configured ports on both hardware and virtual Edges. The only way to get speed values for DPDK configured ports is by using the command "debug.py --dpdk_ports". But the Edge's SNMP module does not rely on this command to extract speed values for DPDK configured ports. SNMP only queries via the Edge's Linux kernel interface, which unfortunately does not populate speed values for dpdk_ports. On an Edge build with a fix for this issue, SNMP does rely on the command "debug.py --dpdk_ports" to gather statistics for DPDK configured ports.

  • Fixed Issue 53243: Traffic may drop to a Non SD-WAN Destination (NSD) that uses a Check Point type because the Gateway does not delete the Phase 2 Security Association (SA).

    Peers would be out of sync in terms of the SA and the link would be down until the SA expires. In an NSD using a Check Point firewall type, when the link is not stable it is possible to end up with a situation when the peer sends a Delete Notify, but the Gateway cannot delete Phase 2 SA because the corresponding Phase 1 SA has already been deleted.

  • Fixed Issue 54846: VMware SD-WAN SNMP MIBs use counters for Jitter, Latency, and Packet Loss.

    In VMware SNMP MIBs, Latency, Jitter, and Packet Loss are defined as Counter64 which is not appropriate for these types. Counters should be used for data types that are ever increasing values and which never reset in SNMP like bytes Tx/Rx. In contrast, latency, jitter, and packet loss do not have ever increasing values but dynamically adjusted values and should not use counters.

  • Fixed Issue 55327: The SSH connection from a VMware SD-WAN Gateway to a VMware SD-WAN Edge may not work if the tunnel from the Edge to the Gateway continuously flaps.

    If the tunnel from Edge to Gateway flaps continuously, the route entry installed in the Edge for allowing the SSH connection from the Gateway may get deleted and cause the SSH connection to fail.

  • Fixed Issue 57011: For a site configured with a High-Availability topology, whenever segments are added and then deleted on that site, one of the HA Edges may experience a Dataplane Service failure and if the service failure is on the Active Edge, the site would also experience an HA failover.

    When segments are added and then deleted from an HA site, there is the potential for stale segments (in other words, the deleted segments might still show up on one of the Edges in the HA pair). Due to this mismatch in segment information between the HA Edges, any event meant for the stale segment might be sent to the other Edge resulting in a Dataplane Service failure, an HA failover if the service failure is on the Active Edge, and the generation of a core dump that will be found on a diagnostic bundle taken after the failover.

  • Fixed Issue 58759: An SNMP walk from a remote cloud location to a WAN interface on a VMware SD-WAN Edge times out.

    Pinging the same WAN interface works as expected. SNMP reply packets are dropped on the Edge because of improper steering policies being applied to those packets.

  • Fixed Issue 65186: For a customer site using multiple WAN links, if there is a business policy configured to use one link with a Preferred or Mandatory policy, the traffic type covered by the business policy continues to be load balanced across all available links.

    Even though the Business Policy is configured to route traffic to one WAN Link using a mandatory or preferred configuration, traffic would be load balanced on multiple WAN links.

  • Fixed Issue 66325: Customer traffic that should be matching a BGP learned route is instead matching a business policy with the potential for disrupting customer traffic.

    If a customer enterprise uses a business policy that configures a Source as the IP of a routed client and the destination as Internet, traffic that should match a BGP learned route instead uses the business policy, including whatever the traffic classification is for that policy (for example, Real Time), which can cause customer traffic disruptions.

  • Fixed Issue 67458: When a VMware SD-WAN Hub Edge with a large number of Spoke Edges is upgraded to Release 4.2.1 or later, some tunnels to other Spoke Edges will not come up for the Hub Edge.

    A large number of Spoke Edges is understood at ~1000 or more. This issue is not consistent, but generally ~1/3rd of the VeloCloud Management Protocol (VCMP) tunnels are not established between the Hub Edge and the connected Spoke Edges. This is caused by the Hub Edge ignoring the

    MP_INITs as the number of half open TDs exceeds the Hub Edge's upper limit.

    Without the fix, a user would need to restart the Edge Service to restore full tunnel connectivity.

  • Fixed Issue 67745: A VMware SD-WAN Edge which has a WAN link with some ISP provided routers may experience customer traffic issues if that ISP route goes down and then comes up.

    When a WAN link from the Edge to some ISP routers (this issue was found with an ISP router used by Spectrum) goes down or the ISP router goes down and then and back up, the ISP router performs a diagnostic which includes briefly assigning a private IP to the Edge in the subnet 192.168.100.0/24, and then after that it assigns the public IP address. However the Edge installs the connected route for 192.168.100.0/0 and it is not cleared after it gets the public IP address.

  • Fixed Issue 67889: SNMPv3 polling may not work properly for a customer polling multiple VMware SD-WAN Edges.

    The issue is that VMware SD-WAN allows its SNMP process (snmpd) to generate a random engine_id for each Edge and in many cases this engine-id is duplicated for multiple Edges. When that is the case one of the Edges with the same engine-id will not report statistics back to the collector.

  • Fixed Issue 68044: A VMWare SD-WAN Edge using a VNF may experience a Dataplane Service failure and restart.

    To monitor whether the traffic passes via the VNF, the Edge process sends periodic Gratuitous ARPs (GARP). There is the potential for memory corruption when the timer for GARPs runs at the same time the Edge updates the VNF configuration.

  • Fixed Issue 68224: A VMware SD-WAN Edge may experience a Dataplane Service failure and restart when removing routes from a PE (Provider Edge) at a high scale.

    High scale is defined as 100K routes or greater. The issue occurs when one thread has to wait too long to obtain a route table lock and the mutex monitor induces a Dataplane Service failure to clear the issue by restarting the service. The issue is caused when a different thread holds the route table lock while trying to do a BGP redistribute hash table lookup and this process takes longer than specified. The process takes too long because the hash process does not distribute the nodes evenly which results in an excess of hash table collisions and thus the lookups end up being a linked list search.

  • Fixed Issue 68785: DHCP INFORM packets are dropped by the VMware SD-WAN Edge software when received on an interface configured as a DHCP relay.

    DHCP clients can request additional network information like DNS server or the Gateway address using the DHCP INFORM message once it has acquired an IP address. When the Edge is configured as a relay agent, these INFORM messages should be forwarded to the DHCP server but are getting dropped.

  • Fixed Issue 68829: For VMware SD-WAN Edge LTE models (in other words, the Edge 510-LTE or Edge 610-LTE), IPv6 paths are not formed over their LTE interfaces.

    The udp6 packets are sent out with a 0 checksum, causing them to be dropped on the next hop. This resulted in the SD-WAN management paths being in an INIT state. The correct behavior is to populate checksum for udp6 packets.

  • Fixed Issue 68840: For a customer using a High-Availability topology, SNMP polling is not able to retrieve LAN and WAN information from the VMware SD-WAN Standby Edge.

    For HA SNMP GET, the Standby LAN/WAN count (vceHaStandbyLanItfNum and vceHaStandbyWanItfNum) is displayed either partially or not all.

  • Fixed Issue 68923: On a customer enterprise using BGP, a default route may be redistributed to a BGP peer though the reachable status for the installed default route is set to 'False'.

    If a static route is configured on an VMware SD-WAN Edge pointing to any Edge interface and that BGP peer learns the default route from the Edge and that interface is later turned off, which changes the reachable flag for that route to False, the route continues to be advertised. It is equally true that a route that is not being redistributed because the interface was down, but then when the interface comes up and marks the route status as 'True', the route would continue to not be redistributed. The cause in both instances is the Edge not readvertising the route on an interface status change that reflects the new route status.

  • Fixed Issue 69462: Partner Gateway (PG) and NAT routes learned on a VMware SD-WAN Gateway are not initially advertised to a Non SD-WAN Destination (NSD) via BGP.

    The routes are advertised after toggling the “secure BGP routes” on the customer partner handoff. The cloud routes are not getting redistributed because when the Partner Gateway BGP comes up before NSD-BGP, the NSD-BGP configuration rewrites the redis table which results in some of the PG routes added to the redis table being omitted. The fix ensures if the redis table is already present, NSD-BGP does not rewrite it.

  • Fixed Issue 69497: The VMware SD-WAN MIBs shows vceLinkVpnState SNMP object even though that is no longer a valid object.

    VMware SD-WAN no longer shows a differentiated VPN state on the VMware SD-WAN Orchestrator but still exposes this in SNMP. To be specific, the SNMP Collector polls for SNMP OID 1.3.6.1.4.1.45346.1.1.2.3.2.2.1.26, which it should no longer do.

  • Fixed Issue 69681: If a VMware SD-WAN Edge is configure with Hot Standby WAN links and also uses SNMP polling, the user will observe SNMP errors.

    Error message would be similar to following:

    Cause of issue is that SNMP path states do no include Hot Standby link states and this causes SNMP issues including error messages.

  • Fixed Issue 69724: A user may observe packet loss on a VMware SD-WAN Gateway due to invalid packet length.

    Due to an issue in the packet pipeline, a packet can be categorized as non-secure and the MTU checks are done based on that classification. However, later it is sent over a management tunnel which has more overhead and as a result, this packet can be dropped at the last moment as its size is more than the link MTU.

  • Fixed Issue 70041: A partner upgrading their VMware SD-WAN Gateways to Release 4.3.0 would observe that they are no longer able to ping the Gateway's VRF IP address.

    When the ping fails, route_drop counter increments. Pinging the VRF IP address from a handoff interface was removed with 4.5.0 and is restored with the fix found on Release 4.5.1.

  • Fixed Issue 70129: When syslog is configured for use on a large scale VMware SD-WAN Gateway, the /var/log folder may get filled up with syslog log files in a short period of time.

    Large scale is understood as a Gateway with ~4K peers and ~6K tunnels where there are usually ~100-150K flows and ~50-100K NAT entries. Short period can be as little as 24 hours with a syslog.log file of >3.2Gb in size. This is caused by some NAT logs being directed to the /var/log that should be directed to a different folder.

  • Fixed Issue 70154: For a customer enterprise where the Stateful Firewall is activated, the user will observe packet drops when sending bidirectional pings between branch clients with the same ICMP ID.

    If a ping is initiated from Client A in Branch 1 to Client B in Branch 2 and vice versa, the ICMP states for both the pings will be tracked with the same flow object if the ICMP ID is the same and this can lead to multiple packet drops because of the sequence number check.

    Without this fix, the workaround is to either turn off the Stateful Firewall or to generate ping with different ICMP IDs.

  • Fixed Issue 70311: A VMware SD-WAN Edge may experience a Dataplane Service Failure and restart as a result.

    During the Edge service restart, customer traffic would be disrupted for ~15-30 seconds. This issue occurs inconsistently, but when it does occur the Edge is tearing down an IKE security association (SA). This typically only occurs when: the SA timer (as configured on the VMware SD-WAN Orchestrator) expires; or the user modifies the IPSec configuration on the Orchestrator.

  • Fixed Issue 70349: In rare instances NAT'd traffic stops working on a VMware SD-WAN Gateway.

    Restarting the Gateway's service will clear the condition. Due to a race condition on the Gateway's dataplane process, the Gateway may fail to establish a connection with natd (the daemon in the Gateway that manages NAT allocations) and so the Gateway will not be able to allocate NAT entries. This will cause all flows NAT'd via the Gateway to fail.

  • Fixed Issue 70438: Customers that have traffic which relies on NAT may experience disruptions in this traffic when a VMware SD-WAN Gateway is upgraded to Release 4.5.0, or restarts while running 4.5.0.

    On Gateway startup there may be a race condition that deletes the NAT entry from "gwd_cloud_read" while the NAT entry could already be cached in the flow (fc->nat_tup) in the SEND_TO_WAN direction. Even though the NAT entry is deleted, the flow will still use the cached nat_tup for applying NAT. Meanwhile another flow can get the same source port allocated with a new NAT entry.

  • Fixed Issue 70586: When a routed interface on a VMware SD-WAN Edge is configured for 802.1x (uses RADIUS authentication), clients connected on that interface get silently de-authenticated whenever any other interface flaps (in other words, when any non-802.1x interface goes down and up in quick succession), and all of their traffic gets dropped until the client disconnects and then reconnects to the Edge.

    The Edge is not checking that the interface that flapped is actually the one that had 802.1x clients authenticated and thus treats any interface flap is if it were a 802.1x interface flap and acts accordingly.

    Without the fix, the only workaround is to force the client to physically disconnect and reconnect to get re-authenticated again.

  • Fixed Issue 70590: An attempt to generate a diagnostic bundle on a VMware SD-WAN Gateway using Release 4.3.0 may fail.

    Diagnostic bundles generated on a Gateway running Release 4.3.0 fail due to the diagnostic bundle exceeding the size limit configured on the Orchestrator. The excessive size of the diagnostic bundle is caused by audit logs that grow large over time.

    Without this fix, the only way to successfully generate a diagnostic bundle on a Gateway is for an Operator User to log into the Gateway and, prior to triggering the diagnostic bundle on the Orchestrator, the Operator User needs to delete the audit log files that are under the Gateway's /var/log/audit directory.

  • Fixed Issue 70855: A VMware SD-WAN Gateway may drop traffic originating from the VMware SD-WAN Orchestrator and as a result may cause any Gateway configuration update from the Orchestrator to fail.

    When the Gateway load is high (for example, ~1.6 million flows on the Gateway with a NAT object count of ~800K), the number of packet buffers in the system will be depleted and this can sometimes cause the Orchestrator traffic to be dropped on the Gateway. Once the Gateway enters this state, the Orchestrator traffic is always dropped even if packet buffers become available.

    Without the fix, the only remediation of this issue is to restart the Gateway.

  • Fixed Issue 70919: On a VMware SD-WAN Edge using Release 4.2.1 GA that is also Wi-Fi capable, users may not be able to connect to the Edge's Wi-Fi and no SSID is broadcast.

    When the Edge is not broadcasting Wi-Fi, the wlan0 interface (WLAN1 on the Orchestrator UI) is not detected when checking logs. This is the result of an exception in the W-Fi card's firmware that occurs under heavy traffic, this exception causes the Wi-Fi to fail.

    In the kernel log a user would observe messages similar to the following:

    The remediation for this issue is a script that detects a firmware failure and reloads the Wi-Fi kernel modules. As part of the module reload, the script also performs a PCIe reset of the Wi-Fi device which restarts the firmware and prepares the device for use. The detection of a failure and the subsequent recovery may take from 30-40 seconds and during this time, Wi-Fi will not be available. This should be understood as a defensive fix versus a complete fix that prevents the issue from occurring in the first place.

    Without the script, the user must reboot or power cycle the Edge to restore Wi-Fi.

  • Fixed Issue 70933: After a configuration profile migration, a VMware SD-WAN Edge where High Availability is used may experience multiple restarts.

    During a configuration profile migration, only the device settings configuration is synchronized immediately with the Standby Edge. The remaining configurations are synchronized only in response to a heartbeat from the Standby Edge. When an Active Edge restarts to apply the latest configuration prior to receiving the heartbeat from the Standby Edge, the result will be a configuration mismatch between the Active and Standby Edge and this will cause multiple Edge restarts to synchronize the configurations of both HA Edges.

  • Fixed Issue 70954: VMware SD-WAN Edge may experience multiple Dataplane Service Failures if they the Edge has a Business Policy configured with a mandatory link to a Zscaler Cloud Security Service (CSS) and the interface for that mandatory link fails.

    The Edge should be dropping the traffic to Zscaler when the mandatory interface drops, versus experiencing a service failure.

  • Fixed Issue 71052: When the number of customer enterprises connecting to a VMware SD-WAN Gateway is greater than 285, the Gateway experiences a Dataplane Service failure.

    Beginning with Gateway Release 4.3.0, the Gateway's ability to monitor customers was enhanced by adding new counters to track the packet and flow related information at the customer enterprise level. The issue is that the number of counters initialized for the customer enterprises exhausts after 285 customer enterprises, and the counter initialization for any further new customer will fail, causing the Gateway's Dataplane Service to fail and forcing a service restart.

  • Fixed Issue 71189: After downgrading from Release 4.5.0 and again upgrading back to 4.5.0, the VMware SASE Orchestrator is not saving a routed interface configuration upon enabling the IPv6 option.

    After downgrading from 4.5.0 and then reupgrading back to 4.5.0, the Orchestrator throws an error "IPv6 wanoverlay must be disabled when IPv4 wanoverlay is disabled for <interface name>" when trying to enable IPv6 option.

    Without the fix, prior to saving the configuration, check the "Enable WAN Overlay" checkbox and then uncheck it.

  • Fixed Issue 71314: On a site with a High-Availability topology, customer may observe multiple HA failover events that result in the VMware SD-WAN Edge service being shut down with a resulting impact in customer traffic quality.

    This issue is inconsistent but when it occurs it is connected to a split-brain issue arising out of a link flap where the HA Edge experiences a dead-lock issue between the HA process and the process that deals with interface state change. The deadlock results in a triple Dataplane Service failure for each HA Edge (with the resulting HA failover events that would follow from this) culminating in the Dataplane Service being shut down. An Edge where its Dataplane Service is not enabled will pass traffic and be able to make configuration changes as the Management Service remains operational. But Dynamic Multipath Optimization is no longer used which means no traffic prioritization, no link steering, and no error correction as all traffic will be sent direct to the cloud, and this will negatively impact high priority customer traffic until it is cleared.

    Without the fix, if this condition is encountered, the user can use Remote Actions > Reboot Edge to restore the Dataplane Service for each HA Edge with the resulting HA Failovers that would follow from this action.

  • Fixed Issue 71465: When configuring a Non SD-WAN Destination (NSD) via Gateway on a VMware SASE Orchestrator UI, under BGP the UI shows the options to create an IPv6 filter when configuring the BGP settings.

    The key problem is that IPv6 is not supported for NSD via Gateways for Release 4.5.x. The BGP editor page is common to the BGP configuration on the Configure > Device Settings page for both the Edge and Profile, the NSD via Gateway through the Configure > Network Services page, and the Partner Gateway Handoff through the Configure > Customer page. The IPv6 changes are done for BGP configuration at the Edge or Profile Configure > Device Settings page and those changes are getting displayed on the NSD via Gateway and Partner Gateway Handoff pages, where it is not supported.

  • Fixed Issue 71720: On the Gateway Monitoring page of the VMware SASE Orchestrator, an Operator User may not see an accurate count for the number of handoff queue drops occurring on a VMware SD-WAN Gateway.

    When a Gateway is oversubscribed this results in handoff queue drops due that should be reported to the Orchestrator and recorded on the UI's Gateway > Monitor page on the Handoff Queue Drops graph for that particular Gateway. In this case, the Gateway is still reporting the handoff queue drops to the Orchestrator but due to a change in drop accounting, the Orchestrator does not report the overcapacity drops and this results in the Handoff Queue Drops graph showing a consistent value of zero even if there are drops on the Gateway.

  • Fixed Issue 71785: SNMP walk may not work properly on an Edge running release 4.3.0 or later.

    When an Edge is upgraded to Release 4.3.0 some IP's that SNMP uses are blocked and the Firewall settings have to be changed to allow traffic on Port 161.

  • Fixed Issue 71788: For a customer deployed with a High-Availability topology, there is a potential for the site to experience an unexpected HA failover that includes a disruption of customer of traffic.

    This is an inconsistent issue that is not readily recreated but where encountered, the HA Edge's periodic handler thread runs more than 180ms during flow synchronization between the Active and Standby Edge and this causes a deadlock and triggers a SIGKILL message on the Active Edge's mutex mon thread resulting in a core and an Edge reboot.

  • Fixed Issue 71977: For a customer using VMware Edge Network Intelligence, when Analytics is enabled for a VMware SD-WAN Edge, this may result in multiple Dataplane Service failures and result in the Edge service restarting for each failure.

    This issue is triggered when a user enables Analytics while the number of sessions created dynamically on the Edge exceeds the maximum allowed limit. The service failures disrupt customer traffic and if there are three consecutive failures the Dataplane Service will be shut down for that Edge. While customer traffic will still pass and the Edge can be configured through the Orchestrator, customer traffic will not benefit from Dynamic Multipath Optimization (DMPO). The lack of DMPO means customer traffic quality may be impacted because there is no traffic prioritization, link steering, or error correction.

    A user would need to reboot the Edge to recover the Dataplane Service. This could be done through the Orchestrator using Remote Actions > Reboot Edge but should only be done in a service window as this would be additional 2-3 minutes of customer traffic disruption.

  • Fixed Issue 72100: For a site using an Enhanced High-Availability topology where a pair of VMware SD-WAN Edge 610's are used for that deployment, tunnels do not establish via the WAN link on the Standby Edge 610.

    The issue is seen only in case of Edge Model 610 where tunnels are not being established via the Standby Edge. Prior to enabling HA, if the VLAN on GE1 interface does not have an IP address then the SD-WAN service does not map the hardware switch port to GE1 once HA is enabled. As a result the packets are dropped on the Standby Edge.

    Without the fix, the workaround for this issue is to add an IP address to the VLAN so that the interface GE1 is mapped to a port on the hardware switch.

  • Fixed Issue 72270: For Edges deployed in a High-Availability, software upgrades that also include a firmware upgrade may cause the HA Edges to be both unexpectedly upgrade and reboot together and trigger customer traffic disruption and OSPF or BFD route learning.

    This is true of Edge 3x00 models that upgrade to a build that includes a CPLD firmware upgrade. By design the Standby Edge upgrades first and then fails over to Active so that Active can then upgrade, but the Active is either waiting for the failover or in the absence of a failover observes a five minutes delay before upgrading also. The Standby Edge is also upgrading it's firmware including the OS kernel this may take longer than five minutes and would create a scenario where both the Standby and Active Edge are upgrading and rebooting simultaneously and causing disruption to customer traffic and forcing OSPF or BFD routes to be relearned which itself is disruptive. The fix here simply extends the Standby's timer to ensure that only one Edge is upgrading at a time.

  • Fixed Issue 72358: If the IP address of a VMware SASE Orchestrator DNS name changes, the Management Service on each VMware SD-WAN Gateway connected to that Orchestrator fails to re-resolve it properly.

    The Management Service on the Gateway periodically checks the DNS resolution of the Orchestrator's DNS name to see if has changed recently and so that it can connect to the right host. There is an issue with the resolution code such that all of these resolution checks fail, and the Gateway keeps using the old address.

    Without the fix, an Operator should not change the IP address of the Orchestrator and if it must be changed, all connected Gateways will have to be reactivated.

  • Fixed Issue 72425: OSPF routes are not advertised to the remote sites, even though a user can see the route received by the local VMware SD-WAN Edge.

    The issue is the result of a race condition in the Edge while processing OSPF NSM (Neighbor State Machine) events and route additions. In the problem state, while adding the OSPF route to the RIB (Routing Information Base), the Edge does possess the OSPF neighbor information because the Edge has not processed the OSPF NSM state event. Due to this, the Edge adds the OSPF route with neighbor IP as 0. If the neighbor IP is 0 the Edge does not synchronize the route to the Orchestrator or advertise it to the Gateway and this is why the route is not seen in either Overlay Flow Control (OFC) or on the Gateway.

  • Fixed Issue 72498: A customer may observe a VMware SD-WAN Edge consuming an increasingly large percentage of its memory and in models with smaller amounts of memory available (for example, Edge 510, 520, 610s) the Edge may initiate a service restart to clear the memory, which will cause a 5-10 second disruption of customer traffic.

    This issue is caused by a memory leak. In a network deployment where Dynamic Edge to Edge is enabled and the Edge is dynamically building and tearing down a high number of tunnels with other Edges in the network, the Edge is not cleaning up the old IKE's from torn down tunnels and this slowly consumes memory over time with the potential in smaller memory Edges to reach a critical level, causing an Edge service restart to clear the memory.

    Without the fix, a user could preemptively restart the Edge service in a maintenance window to clear the memory. But the Edge memory would just begin to slowly leak again.

  • Fixed Issue 72567: If a user deliberately configures asymmetric routing via an underlay (MPLS with no WAN-overlay) in the forward path through a fixed_link Business Policy and then via an overlay (for example, a Hub Edge) in the reverse path, the flow will break and traffic for that flow will not be successful.

    This is a case of asymmetric routing of cloud flows by design in which the Edge has a business policy to force flows out of the underlay (INTF_ROUTED). The remote Edge routes the response back via a Hub Edge (overlay). The remote Edge would see the flow as a new, locally initiated flow, and sends a QoS synchronization to update the flow routing parameters, which leads to a break in the flow routing. In the fix, the QoS synchronization is rejected to prevent the configured link_mode from getting overwritten.

  • Fixed Issue 72688: VMware SD-WAN Edge randomly restarts its Dataplane Service with resulting service interruptions due to the restarts.

    Packets pinned to a decryption thread, when they float to another decryption thread are rejected by the non-owning thread. In the process of rejecting packets, the associated QAT crypto reference was incorrectly released leading to exceptions in the Dataplane Service and a failure and restart.

  • Fixed Issue 72841: VMware SD-WAN Orchestrator is inconsistent in generating "Gateway up" events. Whenever the Gateway state is changed from OFFLINE to CONNECTED, "Gateway up" the event is not logged in every instance.

    The issue occurs inconsistently but when it does happen there are more than 2 Gateways and for both the Gateways the updateStateChange is triggered at the same time. The more Gateways available which all experience a state change at the same time the more likely the issue will occur.

  • Fixed Issue 72859: A VMware SD-WAN Edge may experience a Dataplane Service Failure and restart as a result, causing customer traffic disruption of 10-15 seconds or a High-Availability failover depending on the customer topology.

    This issue is caused when the Edge receives packet fragments where the protocol is undefined or something unexpected. When that occurs the Edge drops the packet, but prior to dropping the packet, the Edge performs a lookup and the Edge service always expects this lookup to succeed, even though it may actually fail. And when this lookup fails it is triggering an issue in the Edge Dataplane Service and causing it to fail and restart. The issue is fixed by including a NULL check which prevents the Dataplane Service from failing on a failed packet lookup.

  • Fixed Issue 72925: For customers who do SNMP polling for monitoring their enterprise and who deploy lower model VMware SD-WAN Edges (for example, Edge models 510, 520, or 610) which are also running a 4.x software release, SNMP polling takes exceptionally long to process and can even timeout.

    This issue significantly reduces the effectiveness of SNMP polling for network monitoring when using Edges in the 510, 5x0, and 6x0 series. This issue is caused by the Release 4.x SNMPagent taking an unnecessarily long amount of time in traversing the debug command list, which is not actually required for the SNMP process.

  • Fixed Issue 72939: A user may observe a VMware SD-WAN Gateway's stale flow count increasing steadily until the count reaches the maximum flows supported for that Gateway specifications and thus no longer create new flows which disrupts users connected to that Gateway.

    The increasing stale flow count is connected to sites sending traffic from a Non SD-WAN Destination to a Partner Gateway handoff. There are two functions that have been added "gw_nvs_to_pg_pkt" & "gw_send_nvs_pkt", where flow counts are not being released after the lookup.

    Without the fix, the only remediation is to restart the Gateway service.

  • Fixed Issue 73251: Users who need to authenticate via RADIUS may find they are unable to authenticate because fragmented traffic is failing to be sent from the VMware SD-WAN Edge.

    RADIUS traffic is always fragmented and this issue impacts users trying to authenticate on a wireless link even more so. When this issue occurs, the fragmented packet count gets beyond what DPDK can handle on the affected particular interface. The fix proactively resets the fragmentation count to avoid traffic disruption.

  • Fixed Issue 73320: When a VMware SD-WAN Edge interface is configured with a DHCP address type and when the IP address assigned to the interface is updated, some of the direct flows can fail due to NAT failure.

    When the DHCP lease expires, the IP address is removed for the interface. Before a new IP is assigned to the interface, NAT entries for any new flows during this transient time will be created with "0.0.0.0" as the source NAT IP and the packets will be dropped. Once a valid IP address is assigned to the interface, the existing NAT entry is not deleted and a new NAT entry is not created with a valid IP causing the traffic to be dropped.

  • Fixed Issue 73830: System Center Configuration Manager (SCCM) application traffic is being misclassified by the VMware SD-WAN Edge as Business Intelligence Service (BITS) traffic and customers using Business Policy or Firewall Rules designed for SCCM traffic will find that traffic impacted.

    The Edge's Deep Packet Inspection (DPI) engine is misclassifying the SCCM application packets as BITS traffic and if there are Business Policy or Firewall Rules designed to steer that traffic or ensure that traffic is allowed by Firewall rules, the misclassification my result in SCCM traffic being blocked with a resulting disruption to the customer. The remediation for this issue involved amending the default 4.5.1 and later application maps to ensure that this misclassification is prevented.

  • Fixed Issue 73987: A VMware SD-WAN Edge may experience a Dataplane Service Failure when attempting to generate a diagnostic bundle.

    The key part of the diagnostic bundle that is causing the issue is when the Edge needs the logs for "vcdgbdump -r remote_routes". In this scenario, when there are a large number of remote routes, traffic is running which is using those routes, and a diagnostic bundle collection is triggered, there is the potential for lock contention in such cases.

  • Fixed Issue 74225: A VMware SD-WAN Gateway or SD-WAN Edge may experience traffic issues and observe packets with zero MAC addresses in logs.

    Even if a Gateway or Edge runs a build that includes the fix for issue #62552, which also addressed a Gateway or Edge sending out packets with a 00:00:00:00 MAC address, there were locations within the Gateway/Edge dataplane process that could still send packets with a source and/or destination MAC address of zero. In these locations the ARP state machine was not validating the source and destination MAC Address.

    The only way to reliably know this issue is hitting is to check logs for zero MAC addresses, specifically ones using tcpdump.

  • Fixed Issue 74306: A user making a Skype call behind a VMware SD-WAN Edge may experience poorer than expected call quality.

    By default, the VMware SD-WAN service classifies all Skype and MS Teams packets, including call packets, as APP_CLASS_INTERNET_INSTANT_MESSAGING (10) class. Instant Messaging class has a lower priority and thus calls could be impacted by being deprioritized for bandwidth and link quality. The fix changes packets matching Skype and MS Teams to APP_CLASS_BUSINESS_COLLABORATION (5) which means calls would receive the expected High Priority/Real Time quality level treatment.

    Without this fix, the workaround was to customize the application map so that Skype and MS Teams were classified as Business Collaboration.

  • Fixed Issue 74316: A VMware SD-WAN Spoke Edge may not connect to any or all of the assigned Hub Edge Clusters, even if the Edge has a service restart or a full reboot.

    There is an issue with the cluster reassignment logic which creates cluster assignment mapping without the cluster member’s endpoint information in a specific Cluster-member-to-Super-Gateway overlay flap scenario. As a result, Spoke Edges assigned to the Hub Cluster member subsequently fails to receive the endpoint information of the Hub Cluster member leading to no overlays between Spoke Edges and Hub Clusters.

    Without the fix the only way to temporarily remediate the condition is for someone with Gateway access to trigger a cluster reassignment manually on the Super Gateways.

  • Fixed Issue 74446: Gateway Handoff Queue Drop counters are not serving the purpose of identifying traffic spikes when observed on the VMware SD-WAN Orchestrator UI.

    Gateway Handoff Queue Drop is not sufficiently granular to identify traffic spikes. The fix for this adds new watermark counters: wmark_1min and wmark_5mins which will give the maximum depth of a handoff queue within 1 minute and 5 minutes respectively.

  • Fixed Issue 74450: VMware SD-WAN Orchestrator does not export watermark counters to an external application like Telegraf or Prometheus.

    This ticket is linked 74446 where that fix creates watermark counters for Gateway Handoff Queue Drops. This ticket ensures those metrics can be collected by an application like Telegraf or Prometheus.

  • Fixed Issue 75117: A user may observe when consulting diagnostic bundle logs a large number of 'DNS Cache Max Limit Reached. Failed to add cache entry' entries that crowds out other log entries of greater importance.

    On a VMware SD-WAN Hardware Edge, if DNS queries exceeds 32K, this will result in continuous DNS Cache logging that crowds out other log entries. This hinders a user's ability to troubleshoot another issue by consulting the logs in a diagnostic bundle.

  • Fixed Issue 75121: A customer may observe an unreachable route for a backhaul flow which leads to packet drops.

    There was an issue in the backhaul and interface flow route lookup code which was picking the first route even if it was unreachable. Since this unreachable route is not good to carry the packet, the packets were getting dropped. The resolution mandates a check on route reachability for all types of flows.

  • Fixed Issue 75186: When software bonding is setup on either a VMware SASE Orchestrator or a VMware SD-WAN Gateway, the bonding interface MAC address is not unique across the system which may cause MAC address conflicts in the network.

    This will only occur if the systems are deployed on the same subnet with software bonding enabled. The MAC address for the software bonding interface is derived from /etc/machine-id. That file is not randomized during the deployment on Orchestrator and Gateway versions prior to Release 4.2.1.

    A customer who uses software bonding on an Orchestrator or Gateway with systems deployed on the same subnet from a software image prior to 4.2.1, should contact VMware support for the resolution.

  • Fixed Issue 75309: A user may observe multicast traffic drop for a specific group on a VMware SD-WAN Spoke Edge.

    Multicast traffic for a specific group is not received at the Spoke Edge when traffic is sourced from a Hub Edge. IGMP groups are not populated in the IGMP group table of the Spoke Edge which leads to traffic loss. This can be observed in the Spoke Edge's diagnostic bundle logs for igmp.

  • Fixed Issue 75578: When running the Remote Diagnostic "Interface Status" the output does not display the speed and mode for the High-Availability enabled interface.

    Interface status from the remote diagnostics does not display the speed and mode for the HA enabled interface. The HA interface, which connects the two Edges, is usually GE1 on most Edge platforms.

  • Fixed Issue 75772: For a customer using Edge Network Intelligence where a VMware SD-WAN Edge has Analytics active, the Edge may experience a memory leak that results in the Edge restarting to clear the memory leak.

    Where Analytics is enabled and DHCP is enabled on an Edge interface, the client connectivity events cause the memory usage to increase. Over a sufficient period of time the memory usage may reach a critical threshold and the Edge would defensively restart the Edge service to restore normal memory levels. As with any memory leak issue, the smaller the initial memory footprint (for example, an Edge 510, 520, or 610) the more vulnerable the Edge is to having an Edge restart occur.

  • Fixed Issue 75827: A VMware SD-WAN Gateway may experience a Dataplane Service failure and restart as a result.

    In the logs, a user would observe a Gateway process failure with de2e_info_reply_msg_recieved. The root cause is a NULL pointer exception that is not handled in the Gateway process which triggers an exception and causes the service failure and restart.

  • Fixed Issue 75890: The loopback IP address of a VMware SD-WAN Spoke Edge appears as reachable for ~2 minutes in a regional Hub Edge via a datacenter Hub Edge even though there is no path between these Hub Edges and the Spoke Edge.

    The routes remain true for 2 minutes and then finally get cleaned up after 2 minutes as part of the cleanup for peer unreachable routes. This only happens to the loopback IP address and all other Spoke Edge IP addresses are properly marked as False in this scenario.

  • Fixed Issue 75968: When a user configures a local IP address hand-off for a VMware SD-WAN Gateway, and then removes that handoff, the handoff is not removed from the Gateway which causes the tunnel path to go down.

    This issue can cause major disruptions since the Gateway cannot build a new tunnel.

    Without the fix, the user would need to restart the Gateway to clear the issue.

  • Fixed Issue 75989: The VMware SD-WAN Gateway's Telegraf agent may fail to export metrics collected from vcg_metrics.sh.

    The Gateway's Telegraf logs show "vcgCommand timeout" error, meaning the script was not able to be executed within the timeout specified.

  • Fixed Issue 75992: A customer enterprise with multiple VMware SD-WAN Edges where some Edges are running a 3.4.x Edge version and other Edges are running a 4.3.x Edge version, the customer may observe service and traffic disruptions.

    In some customer deployments using a VMware SD-WAN Gateway running Gateway version 4.3.0 and Edges running a mix of 3.4.x and 4.3.x versions, The Edge running 3.4.x have been seen with some invalid routes which have a non-zero network address but a zero mask. When such route gets installed into the FIB, this causes the routing issue.

  • Fixed Issue 76005: A NAT 1:1 Rule may not be honored after a WAN link flaps on a VMware SD-WAN Edge.

    When looking for a 1:1 NAT rule match, if a particular interface is down, it is skipped by the Edge. So if a flow is created when an interface is down, even after the interface comes up, the Edge does not honor it.

  • Fixed Issue 76165: A VMware SD-WAN Edge may experience a Dataplane Service failure and restart as a result.

    When an Edge is part of an enterprise with a large number of routes (for example, ~20K routes), if a diagnostic bundle is collected, the Edge CPU may become starved during the process of generating logs and experience a service failure restart because of the CPU starvation.

  • Fixed Issue 76315: An Operator User may observe a VMware SD-WAN Gateway dropping the first few packets of every flow the Gateway sends out.

    The drop reason will be listed as gwrouting_vpn_unreachable. The issue is the result of a processing delay of the management protocol's QoS (Quality of Service) synchronization messages which causes the first few packets of every flow to be incorrectly classified and dropped.

  • Fixed Issue 76564: For a site configured with a High-Availability topology, when the VMware SD-WAN Edge's WAN interface is either enabled or not enabled using the VMware SD-WAN Orchestrator UI, the site may experience a "split-brain" Active/Active state which is disruptive to customer traffic.

    When an Edge's WAN Settings are changed, the Edge's network service is restarted and this leads to temporary HA packet loss which fools the Orchestrator into thinking the Active Edge is down and promotes the Standby Edge to Active which leads to the Active/Active "split-brain" state.

  • Fixed Issue 76586: For a customer with a Hub/Spoke topology, a customer may observe that the VMware SD-WAN Spoke Edge is shifting from 'Direct' to 'Gateway' upon receiving replies from the Hub Edge for WAN overlay enabled interfaces and leads to the flow breaking with the resulting disruption in customer traffic for that flow.

    This is a scenario where the customer's intention is to asymmetrically route flows via the underlay (MPLS with no WAN-overlay) on the forward path through a fixed link Business Policy and via the overlay (Hub Edge) in the reverse path and this leads to the flow breaking.

    The cause of this flow breakage is that when the remote end routes respond back via a Hub Edge (Overlay), the remote Edge views this flow as a new, locally initiated flow and sends a QoS synchronization request to update the flow routing parameters, which leads to a break in the flow routing. The QoS synchronization request is rejected to prevent the configured link mode from getting overwritten.

  • Fixed Issue 76726: A VMware SD-WAN Gateway may experience a Dataplane Service Failure and generate a core.

    The cause of the Gateway service failure and core is that management protocol ctrl packet exchanges between an Edge and the Gateway, the Gateway may fail because of the asserts and this would cause an outage for multiple customers. This issue is fixed by removing asserts and having proper error handling.

  • Fixed Issue 77357: A VMware SD-WAN Edge 3400 or 3800 that is deployed in Japan may lock up and reboot spontaneously.

    The Edge 3400 and 3800 have an incorrect voltage warning threshold (100V) set in the system's Baseboard Management Controller (BMC), which happens to match the 100V electrical supply in Japan. The result for an Edge 3400 or 3800 in this region is a continuous series of power supply alarms and if the volume of alarms is sufficiently frequent, the Edge will lock up and reboot.

    Note: This is a follow up to Edge issue 51291 which, while the fix was successful in resolving the issue, the fix was not lasting because the manually set voltage thresholds were getting cleared out by a CPLD upgrade. Otherwise the symptom and description are the same but the fix here ensures the voltage thresholds persist over any subsequent Edge upgrade.

  • Fixed Issue 78003: For a customer using a Hub/Spoke topology, static tunnels from the VMware SD-WAN Spoke Edge to a Hub Edge may not form.

    Typically if there are a large number of Dynamic Edge-to-Edge tunnels already established on the Spoke Edge, the maximum tunnel number check is hit on the Spoke for the static tunnel, and this check prevents static tunnel formation from the Spoke to the Hub.

  • Fixed Issue 78252: A VMware SD-WAN Edge may experience a Dataplane Service failure with a SIGXCPU message when deployed in a large scale enterprise.

    This issue is caused by a BGP/OSPF worker thread running with low priority holding the lock on a object for which the other high priority threads are waiting and this results in a priority inversion leading to the Edge service failure with SIGXCPU message.

  • Fixed Issue 78276: On a VMware SD-WAN Gateway, running the debug.py -qos_net fails if the VMware SD-WAN Edge's name includes non-ASCII characters.

    An example of this in the field was the use of Chinese characters but it applies to any non-ASCII characters and can be observed as follows: change an Edge name to include non-ASCII characters and reboot the Edge. Then a Gateway connected to the Edge run the CLI command: 'debug.py --list 3', to get the Edge's logical ID. Then run the Gateway CLI command: 'debug.py -qos_net [logical ID] all stats' and the user would observe the command fails.

  • Fixed Issue 78391: Traffic with the Speedtest application classification is not working properly.

    Both speedtest.net and fast.com have newly added speedtest server IP addresses that are missing in the default application map and as a result the Business Policy that deals with these application is not being applied.

    If not upgraded to Release 4.5.1, an Operator could add the required speedtest IPs to an existing application map using the VMware SASE Orchestrator's Application Map Editor.

  • Fixed Issue 78637: A VMware SD-WAN Gateway may experience a Dataplane Service failure with a SIGXCPU message and a resulting service restart.

    The Gateway will also generate a core. The issue is traced to when the Gateway receives a packet with the optional TLV length as 0, the Gateway ends up in an infinite loop which results both in the SIGXCPU and resulting service restart and core.

  • Fixed Issue 78678: On a site deployed with a High Availability topology, the VMware SD-WAN Edge performing the Standby role may get rebooted while processing synchronization messages from the Active Edge.

    When the Standby Edge is handling a high number of flow synchronization messages, the SD-WAN service may detect a buffer overflow condition and trigger a reboot of the Standby Edge.

  • Fixed Issue 79132: For a VMware SD-WAN Edge configured as a Spoke in a Hub/Spoke topology, a public link displays the wrong download bandwidth capacity when looking at Monitor > Edge on the VMware SASE Orchestrator UI.

    For the Spoke Edge's public links, the link statistics are loaded with the value measured against the Spoke's connected Hub Edge instead of the Primary Gateway, which is then uploaded to the Orchestrator and displayed for the download bandwidth.

    This issue is triggered when the Spoke Edge's Primary Gateway is restarted after The Spoke Edge establishes Spoke/Hub tunnels. So the only way to avoid this issue without the fix is to ensure the Gateway is not restarted post-tunnel establishment with the Spoke and Hub Edges.

  • Fixed Issue 79261: Office 365 / Microsoft 365 application traffic is misclassified as Tencent Meeting (VooV Meeting) application traffic on the VMware SD-WAN Edge.

    This can be disruptive for customers who rely on Business or Firewall Policies for routing and prioritizing Office 365 / Microsoft 365 traffic only to have that traffic classified as Tencent Meeting and thus hitting a completely different rule. Issued is traced to incorrect application map subnets for Tencent that are corrected for the 4.5.1 default application map. A customer not using 4.5.1 can have this corrected by an Operator who edits their application map through the Orchestrator Application Map Editor to correct the Tencent subnets.

  • Fixed Issue 79437: A VMware SD-WAN Gateway deployed on a server using a Intel X710 NIC with SR-IOV enabled for interfaces may fail to deploy.

    If this failure occurs, the Operator would see the X710 SR-IOV interfaces removed from DPDK and not seen when running debug.py --dpdk_ports_d. In addition, /opt/vc/etc/dpdk.json does not display the SR-IOV interfaces. Deploying a Gateway to the 4.5.1 Release ensures this issue would not be encountered.

  • Fixed Issue 79572: For a site deployed with an High-Availability topology, if a user turns off HA, the now standalone VMware SD-WAN Edge may lose be deactivated and lose contact with the VMware SASE Orchestrator and stop passing traffic entirely.

    When High-Availability is turned off, the Standby Edge applies the configuration and moves to a deactivated state. However, due to a race condition, the Standby Edge sometimes is able to send the Orchestrator a heartbeat and this fools the Orchestrator into sending a deactivation order to the Active Edge. There is significant impact to the customer when this issue occurs because all customer traffic will drop immediately and not resume until one of the Edges is reactivated.

    Without the fix, customers should either: a) Turn off HA in a maintenance window where there is an option for someone to reactivate the Edge should it encounter this issue, or b) power down and physically remove the Standby Edge from the customer deployment and only then turn off HA as the Standby Edge would not be able to send the spurious heartbeat that triggers the issue.

  • Fixed Issue 80006: When upgrading a VMware SASE Orchestrator or a VMware SD-WAN Gateway, the upgrade script fails to upgrade FIPS modules when the system is in FIPS mode (mode=compliant).

    FIPS packages are copied to a special system location (/var/lib/velocloud.fips) during system upgrade but the system installer does not pick them up. For this issue, the FIPS repositories are not updated in the preinstall script and thus new packages are not available during the system upgrade step.

    This issue is not encountered when upgrading either component to Release 4.5.1. However should an Operator upgrade an Orchestrator or Gateway to a Release without the fix, the Operator would need to upgrade the FIPS modules manually.

  • Fixed Issue 80010: For a customer enterprise using a Hub/Spoke topology where SD-WAN Reachable is also configured, the Spoke to Gateway path (using a public WAN link) via the Hub path does not come up if the Spoke-to-Hub path is point-to-point.

    The SD-WAN Reachable feature, which is a passthrough for a Spoke Edge to connect to a Gateway through a connected Hub, is not supported if the Spoke Edge and the Hub Edge are connected by a point-to-point link (in other words, the Spoke's IP address matches the connected route on the Hub). The fix for this issue adds this functionality.

  • Fixed Issue 80196: A VMware SD-WAN Gateway may experience a Dataplane Service failure with a SIGXCPU message and the Gateway restarts the service to recover with an impact to Gateway traffic for 15-30 seconds.

    This issue is seen at high throughput for that Gateway relative to its throughput capacity and thus is more likely to happen in large scale deployments (for example, 4K Edges and 6K tunnels). When traffic at a high rate hits the Gateway, in some instances the Gateway will experience a thread lock and generates a core while restarting. In the core the user would observe: "Program terminated with signal SIGXCPU, CPU time limit exceeded".

  • Fixed Issue 80479: A VMware SD-WAN Gateway may experience a Dataplane Service failure and the Gateway restarts the service to recover with an impact to Gateway traffic for 15-30 seconds.

    This issue can occur if a VMware SD-WAN Edge connected to the Gateway with Edge-to-Edge (E2E) enabled and a Loopback interface route advertised. When a user turns off E2E for this Edge, this triggers a route initiation but the loopback route is not deleted, and the route updates the profile flag of the route. Next, if the user removes the advertise for the Loopback route, this deletes the route from the FIB but remains stale in the E2E table on the Gateway. If the Loopback route is then readvertised and is added to the FIB and after that enable E2E which again just updates the flag, even though the route is present in the Gateway's E2E table (which is stale), the actual route ref_count is not correct. Finally, if the tunnel is torn down, this is what triggers the Dataplane Service failure on the Gateway.

    Without the fix, an Operator would need to make sure routes are withdrawn before a profile is changed for the Edge.

  • Fixed Issue 80496: Ping from a VMware SD-WAN Edge to a remote Edge's branch loopback IP address over a SD-WAN tunnel may not work.

    Issue is seen for a ping with a large enough packet size to cause fragmentation. When the ping is initiated with a large packet size from an Edge to a remote branch Edge's loopback IP address, the fragmented ICMP reply is reaching the Edge initiating the ping but does not reach the ping application since the next fragment is dropped.

  • Fixed Issue 80897: For a customer enterprise where VMware SD-WAN Edges are connected to VMware SD-WAN Partner Gateways, users may observe poor performance for customer traffic.

    The poor performance is the result of routing issues stemming from the Partner Gateway distributing routes to the Edges where preferred secure static routes are available but the Edge does not properly label these routes as secure. The result is the Edge potentially advertising non-preferred non-secure routes over secure routes since all routes are treated equally when the expected behavior is to always prefer secure routes over non-secure routes.

    Note: Both the Partner Gateway and customer Edges must be upgraded to a build that includes this fix to resolve the issue.

  • Fixed Issue 81196: User cannot login to the Local UI of a deactivated VMware SD-WAN Edge with the factory default password.

    A user can "Deactivate" an Edge either through the Local UI using Reset Setting > Reset Configuration, or on the VMware SASE Orchestrator by selecting Remote Actions > Deactivate for the Edge. After the user "Deactivates" the Edge, all configurations should be cleared and the login credentials of the local UI should revert to the default values, but they do not. The credentials remain the same as before the deactivation and if a user had changed the Local UI default credentials to some other value, that new value persists past the Edge Deactivation. If the local UI credentials were never updated, then the user would not any issue as the credentials before and after deactivation would remain as the defaults.

  • Fixed Issue 81221: If a customer configures a 1:1 NAT rule for a VMware SD-WAN Edge and that Edge is rebooted, the rule no longer works.

    After the reboot, the Edge assigns the NAT address as the Edge interface address where the NAT rule is being applied and thus no tunnels are being built for traffic matching that rule.

    Without the fix, the only remediation is to run the Remote Diagnostic "Flush NAT", which flushes the entire NAT table and reestablishes correct NAT rule operation.

  • Fixed Issue 81224: On a site deployed with a High Availability topology, when the site experiences an HA failover, the OSPF route tags may not propagate post-HA failover.

    On an HA failover, OSPF external LSA's (link state advertisements) do not have a route tag, which leads to improper routing with an adverse affect on customer traffic.

  • Fixed Issue 81517: On a site deployed with an Enhanced High Availability topology which uses VMware SD-WAN Edge models 6x0's, the HA link state is not updating properly.

    The HA link is the link that connects the Enhanced HA Edge pair and if this link does not update properly the site may have issues.

  • Fixed Issue 81575: A VMware SD-WAN Gateway deployed using a VMware OVA-based VM which uses Intel x710 SR-IOV-based NICs may have connectivity issues after a Gateway software upgrade.

    The issue arises out of the iavf Linux Virtual Function Drivers not installing correctly on a Gateway upgrade and as a result x710 SR-IOV based NICs do not work on the upgraded Gateway. This issue does not affect a Gateway which uses Virtio or VMNet based NICs.

  • Fixed Issue 81809: When a user attempts to SSH to a VLAN IP on a VMware SD-WAN Edge from a remote client sitting behind another Edge or even from a VMware SD-WAN Gateway, the SSH attempt fails.

    An SSH attempt from a LAN client to an Edge VLAN IP works properly. Originally the Edge's Management IP was used to SSH to the Edge. However, after the Edge Management IP was deprecated, there was no option for the user to SSH to the Edge (via overlay from a remote Edge client) as the Loopback IP still doesn't support SSH.

  • Fixed Issue 81920: On a VMware SD-WAN Gateway deployed using a KVM-based VM which uses Intel x710 SR-IOV-based NICs may have connectivity issues after a Gateway software upgrade.

    The issue arises out of the iavf Linux Virtual Function Drivers not installing correctly on a Gateway upgrade and as a result x710 SR-IOV based NICs do not work on the upgraded Gateway. This issue does not affect a Gateway which uses Virtio or VMNet based NICs.

  • Fixed Issue 82182: For a VMware SD-WAN Edge Model 510 or 510-LTE which is running Edge Release 5.0.0.0, when a user attempts a service restart of the Edge, the Edge may also reboot.

    An Edge reboot would disrupt customer traffic for 2-3 minutes while the Edge was going through the reboot process. On an Edge 510/510-LTE, there is a Wi-Fi device hang monitoring script which may fail to stop during the Edge service restart and this triggers the reboot.

    Without the fix a user would need to restart the Edge service, but Edge service restarts for these models should only be done in a maintenance window or with the understanding this issue may arise.

  • Fixed Issue 82188: For VMware SD-WAN LTE models (Edge 510-LTE, 610-LTE), when the IPv6 setting is toggled off, the tunnels via CELL interface may fail.

    When the IPv6 checkbox in Device Settings is turned off, the internal state of the CELL interface goes to UNKNOWN. This is not updated even when the IPv6 setting is toggled to on for the CELL interface. Due to this the tunnels via the CELL interface would fail. If the LTE Edge is only using CELL interfaces for traffic this result in the Edge going offline, and would drop all Edge traffic which would be very disruptive to the customer.

    Without the fix, a user would need to restart the Edge Service after toggling the IPv6 setting off. If the Edge is offline because it only uses the CELL interface for bandwidth, a local user would need to power cycle the Edge to recover it.

  • Fixed Issue 82314: When a VMware SD-WAN Gateway is upgraded it may throw an exception with a loss of connectivity when using Intel x710 PCIe pass-through based NICs.

    When the Gateway is upgraded, part of the upgrade is a kernel change and the i40e driver installation fails and is not available on the Gateway. Because of the unavailability of i40e drivers, all x710 PCIe pass-through based NICs will not operate properly on the Gateway with either a resulting performance degradation or a loss of connectivity. This issue does not affect a Gateway which uses Virtio or VMNet based NICs.

  • Fixed Issue 82522: When high throughput traffic hits a VMware SD-WAN Edge, there may be a drop in actual throughput observed on the Edge.

    Under high throughput, the Edge's NDP (Neighbor Discovery Protocol) thread is acquiring lock twice even for NDP entries which were marked reachable and further processing was not required. These duplicate locks cause the throughput to decrease due to the additional processing.

  • Fixed Issue 82790: On VMware SD-WAN Gateways deployed in Azure environments, the Gateway interface counters are not exported to the Wavefront monitoring service.

    Azure is an environment where DPDK is deactivated, and only DPDK interface counters (throughput rate, PPS, and drop counters) are exported to the Wavefront service. This leads to reduced monitoring abilities in platforms like Azure where DPDK is deactivated.

  • Fixed Issue 83029: For either a standalone VMware SD-WAN Edge or a site deployed with a High-Availability topology where one or more PPPoE links are used, if the PPPoE endpoint IP changes after either an Edge interface flap for that PPPoE link or when an HA site experiences a failover, traffic would not pass on the affected PPPoE link(s).

    On a site that uses PPPoE links, along with a change in the PPPoE endpoint IP, the impact would mean no customer traffic would pass. The issue is caused due to the presence of a stale default route, which is a route using the old IP address of the PPPoE endpoint on the Edge that is not deleted after a new PPPoE endpoint IP address is received.

    Without the fix, an onsite user would need to either disconnect each PPPoE cable and reconnect it to force a renegotiation or reboot the Edge, which would also force a renegotiation.

  • Fixed Issue 83402: On a VMware SD-WAN Edge with multiple WAN links, one or more WAN links may stop passing traffic.

    On the WAN link(s) that stop passing traffic, the DHCP acquired address is not renewed and the WAN interface's address is lost. Issue occurs when there are multiple interfaces acquiring IP addresses using DHCP and the DHCP server is in a different network from the client. The outgoing interface of a DHCP renew unicast packet is determined through route lookup. Since there are multiple default routes with different metric values learned through different interfaces, the DHCP request packets might get sent out of a different interface.

    Without the fix, an onsite user would need to unplug and then plug back in the affected WAN link from the Edge to force it to get its IP address again.

  • Fixed Issue 83928: A VMware SD-WAN Edge may experience high CPU usage and poor customer traffic performance.

    Users would also be able to observe poor QoE scores when looking at the Orchestrator's Monitor > Edge > QoE screen for that Edge. The issue is caused by an ACL (Access Control List) rule getting instantiated multiple times in the Edge and it is stressing the Edge's CPU capacity to process this many ACL rules at once and this results in the Edge being unable to process customer traffic properly.

  • Fixed Issue 83946: VMware SD-WAN Edge LAN-side clients may observe disruptions in traffic, and for a site using RADIUS authentication, client users may observe authentication failures.

    Large packets will be fragmented and these fragmented packets can be dropped by the Edge. The packets are dropped due to a memory leak during fragment IP identification translation during some error scenarios and if the Edge limit for fragmented packets is exceeded, then further fragmented packets will be dropped by the Edge.

    For customers using RADIUS where large packets from a wireless client to an Edge using RADIUS authentication are involved this can cause authentication failures. For example, large packets from a wireless LAN controller (WLC) to a RADIUS server may be dropped.

  • Fixed Issue 84359: When a VMware SD-WAN Edge interface flaps, it is possible that multiple IPv4 addresses can be assigned to it.

    When an interface, configured with a DHCP client flaps (goes down and up in rapid succession), the entire DHCP client process is carried out again and there could be scenarios where a different IP address is acquired each time. In this case, the older IP address is not cleared and stale.

    Without this fix, the only way to remediate the issue if for a user has to manually delete the IP addresses from the interface through the Linux shell using the "ip address del" command.

  • Fixed Issue 84825: When a large bulk routing configuration is applied on a VMware SD-WAN Edge in a single step, the Edge may experience repeated Dataplane Service failures resulting in repeated restarts of the Edge service to recover from each failure.

    When a standalone (non-HA) Edge encounters this issue, there is significant impact to customer traffic because while a single Edge service restart disrupts traffic for ~15 seconds, repeated Edge service restarts would result in disruptions of ~60 seconds or more. On a site with a High-Availability topology, the customer would observe repeated failovers resulting from the Edge service restarts which would also disrupt customer traffic.

    This issue occurs when a bulk routing configuration involving a large number of neighbors and route-maps is applied on an Edge in a single step. The Edge system faces great stress while converting these configurations into command specifications and applying them on routing protocols in a short span of time and this causes the repeated Edge service failures and restarts.

    On an Edge build without the fix, to mitigate the risk of this issue a customer user would need to do the following:

    • Instead of applying a large configuration in a single step, the configuration should be broken into multiple smaller sections with each section applied separately.

    • The number of routing filters should be minimized.

    • The Edge should only be deliberately restarted in a maintenance window and Edge service restarts should be generally avoided if there are a number of routing filters configured, as the entire Edge configuration is applied at once during restart which would greatly increase the risk of encountering this issue.

  • Fixed Issue 84847: Customers using either USB-based LTE modems or VMware SD-WAN Edge LTE models (510-LTE or 610-LTE) may experience intermittent issues with building tunnels from the CELL interface after the modem is reset.

    When the LTE modem is reset in one of the following scenarios:

    On an Edge using a USB modem, by removing and re-plugging in the modem from the USB port.

    On an Edge-LTE, after an Edge reboot or by resetting the CELL1 interface via the Test & Troubleshoot > Remote Diagnostics > Reset USB Modem > CELL1.

    In either scenario the underlying network device changes from wwan0 to wwan1 and the Edge does not honor this new name because it appears to be a duplicate interface.

    Without the fix the workaround to restore the LTE interface is to restart the Edge Service through Remote Actions > Restart Service.

  • Fixed Issue 86103: For a customer enterprise that uses RADIUS authentication, client users at some sites may be unable to connect to VMware SD-WAN Edges and pass traffic.

    The issues is caused by the Edge incorrectly categorizing fragmented RADIUS packets with the DF (Don't Fragment) bit set in the IP header as non-fragmented. One or more of these packets fails to reach multiple Edges with the result that traffic that relies on RADIUS authentication will not pass for those Edges. This issue can occur in any topology including Hub/Spoke and simple Branch-to-Branch.

    Without the fix the only workaround is to configure the RADIUS server to not set the DF bit in the IP header while sending fragmented packets.

  • Fixed Issue 86314: A VMware SD-WAN Edge may perform an incorrect Stateful Firewall rule lookup when LAN-side NAT flow is initiated by a remote peer.

    When a user configures a LAN-side source NAT (for example, to hide an internal IP subnet behind the Edge) on an Edge where the Stateful Firewall is in use, and a flow is initiated by a remote peer, an erroneous firewall lookup is done for the first return packet.

    For example, suppose that an Edge has the following configuration:

    LAN-side NAT: [source] inside address: 10.0.2.25/32 outside address: 7.0.2.25/32

    Static route: 7.0.2.25/32 [advertise] next hop: 10.0.2.1

    A remote client sending a ping to 7.0.2.25 from 10.0.1.25 would result in two firewall rule lookups on the Edge. The first incoming packet would result in a firewall lookup for 10.0.1.25 > 7.0.2.25, and then the first return packet would result in a firewall rule lookup using the non-NAT IP for 10.0.2.25 > 10.0.1.25. This second firewall rule lookup is done in error.

    Without this fix, the user would need to create an additional firewall rule to allow the first return packet of the flow.

  • Fixed Issue 86808: Some BGP routes are advertised when they should not be as per BGP filters (or not advertised when they should be).

    For a given route-map rule, the Edge could either have a prefix-list or a community-list configuration for the Edge's routing based on the rule match-type. However, for route-map unapply functions, the Edge is trying to delete both the prefix-list and community-list for each rule, one of which must be non-existent.

    Previously, this did not cause any issues as the commands for non-existent prefix-lists and/or community-lists used to be sent to the Edge's routing process as a separate vtysh command, which would just end up being no-ops and would not impact other commands. At that time, this was a deliberate call as it kept things simple in the route-map unapply functions.

    However, as part of the fix issue #84825, the Edge started batching multiple prefix-lists/community-list removal vtysh commands together to be sent to

    the Edge's routing process. Now, when trying to delete non-existent prefix-list/community-list causes the whole command batch to fail and filling the Edge with stale prefix-lists/community-lists configuration in the Edge's routing process.

    On an Edge without this fix, the only way to remediate the issue is to restart the Edge Service.

    Note: In editions of the 4.5.1 Release Notes up to 08-22-2022, this issue was listed as a known issue. This was not correct as the fix for #86808 was included in the original GA build for Release 4.5.1 and the issue has never been present in any Release 4.5.1 build.

Orchestrator Resolved Issues

Resolved in Orchestrator Version R451-20220831-GA

Orchestrator version R451-20220831-GA was released on 09-06-2022 and is the updated 3rd Orchestrator rollup build for Release 4.5.1. This build replaces the original 3rd rollup build R451-20220810-GA, which was released on 08-11-2022. R451-20220831-GA adds no new fixed issues over R451-20220810-GA but does include performance and troubleshooting changes required by VMware Engineering.

Important:

Customers must only use the R451-20220831-GA build and not use R451-20220810-GA.

Build R451-20220831-GA addresses the following critical issues since the 2nd Orchestrator rollup, version R451-20220612-GA.

  • Fixed Issue 80735: When a user changes the configuration of a Profile that is also assigned to one or more VMware SD-WAN Edges, the BGP filters on the Profile level are removed.

    The user would observe "ERR: invalid filter ref" on the VMware SASE Orchestrator UI in the places where previously the user could see the details of the BGP filter. The impact to a customer's networking that relies on BGP would be significant and the only way to restore the BGP filters would be to manually restore them.

  • Fixed Issue 83507: A VMware SASE Orchestrator updated to a new build continued using the same, older version of the MySQL server software.

    The Orchestrator build process was pinning the SQL server version being used and this resulted in the MySQL server software not including the latest bug fixes which may result in Orchestrator-level database issues.

  • Fixed Issue 88120: On the VMware SASE Orchestrator, when looking at the Monitor > Edge > Overview page on the Classic UI versus the New UI, there are discrepancies in link status when displaying in "Live Mode".

    The specific discrepancy is on the New UI where a WAN link may display a "Standby" status when it should be displaying a "Stable" status. The Classic UI displays the link status properly.

  • Fixed Issue 90067: When a VMware SASE Orchestrator is upgraded to 4.5.1 or 5.0.0, the Operator may observe high CPU usage and load issues.

    During the upgrade the Orchestrator loses a critical system property: edge.learnedRoute.maxRoutePerCall. This property caps the number of routing protocol events that can be received by the Orchestrator at any one time. In the absence of this property, an Orchestrator could be flooded with routing protocol events that place it under a high load which can impact the Orchestrator's performance. The fix ensures that system property edge.learnedRoute.maxRoutePerCall persists over Orchestrator upgrades.

  • Fixed Issue 91179: For a VMware SD-WAN Edge which has a WAN link configured as Hot Standby, if the Hot Standby link's status is standby, the VMware SASE Orchestrator's New UI displays the incorrect status for the Hot Standby link (Active).

    The Orchestrator's Classic UI shows the correct status for the link (Idle), so this is limited to the New UI only. The issue is caused by the New UI not getting the correct update on the change of status for a Hot Standby WAN link.

  • Fixed Issue 92082: For a customer using VMware Cloud Web Security, the customer may observe that the Content Filtering rules do not honor the configured domain.

    The Content Filtering rules override the configured domain provided if the user has also selected ALL for Categories. Or, if the user selects NONE for Categories, the wizard defaulted this choice to mean ALL Categories, hence the domains were not honored here as well. This is caused by an issue in the content filtering wizard and API. If the user configures at least one Category, the Domain is honored.

    On an Orchestrator without this fix, the user would need to configure specific categories along with domains, and then the Orchestrator would honor domains in content filtering.

Resolved in Orchestrator Version R451-20220612-GA

Orchestrator version R451-20220612-GA was released on 06-13-2022 and is the 2nd Orchestrator rollup for Release 4.5.1.

This Orchestrator rollup build addresses the below critical issue since the 1st Orchestrator rollup, version R451-20220520-GA.

  • Fixed Issue 86848: When a customer administrator makes a failed login attempt using the Native (username/password) method to their customer enterprise on the VMware SASE Orchestrator, the Orchestrator does not log the failed attempt on the Monitor > Events page of the UI.

    The Orchestrator should log every login attempt whether it is successful or not to ensure proper accountability of all user accounts and to all the administrators to detect unusual login activity. The issue is caused by the Orchestrator not including the 'enterpriseId' metadata to a failed username/password authorization attempt. This only affects customer users using Native (username/password) authorization and customer enterprises using Single Sign On (SSO) are not impacted by this issue.

  • Fixed Issue 89800: When a user updates the Segment Property on the VMware SASE Orchestrator, the Edge tunnels to their Zscaler Cloud Security Service (CSS) go down and traffic routed to Zscaler is dropped.

    If a user has a CSS configured under Configure > Network Service (any CSS type) and then configures the FQDN and PSK authentication details at Configure > Edge > Device > Cloud Security Service using Edge Override, when a user updates any Segment in the Configure > Segment section of the Orchestrator, the Edge's CSS authentication configuration is deleted and the Edge can no longer connect to the Zscaler peer.

Resolved in Orchestrator Version R451-20220520-GA

Orchestrator version R451-20220520-GA was released on 05-24-2022 and is the 1st Orchestrator rollup for Release 4.5.1.

This Orchestrator rollup build addresses the below critical issue since the initial GA version R451-20220517-GA.

  • Fixed Issue 84214: When an Operator user is on the Gateways page of a VMware SASE Orchestrator UI, they may be unable to assign a particular Gateway for the role of Super Gateway.

    When a Gateway is already assigned the role of both Super and Alternate Super Gateway, and the Operator tries to edit the Super Gateway assignment of an enterprise from the Customer Usage list on the Gateways > Configure Gateways screen, the UI does not correctly find associated data about the Super Gateway and the Assign Super Gateway dialog does not show up while also throwing an error in the console.

  • Fixed Issue 86546: For customers using VMware Secure Access, a user may not be able to use Secure Access on some SASE PoPs, and some may even show as offline on the VMware SASE Orchestrator.

    VMware Gateways that are not configured for use with Secure Access (in other words, Gateways which do not have a geneve tunnel with the tunnel server on the PoP) are also given information about the Secure Access service by the Orchestrator. This leads to a broken route being picked in some instances for routing customer traffic. This issue can be encountered only when more than one Gateway is assigned per PoP per Gateway pool on a particular Orchestrator.

    On an Orchestrator that does not have the fix for this issue, the workaround is to add and keep only one Gateway per PoP in each Gateway Pool so that this Gateway always gets picked for Secure Access and the establishing of the correct route.

  • Fixed Issue 89349: A user logging into a VMware SASE Orchestrator which uses the 4.5.1 Orchestrator build R451-20220517-GA would observe a Beta license agreement on the login page.

    A user is given the erroneous impression they are using a Beta build versus a VMware SASE qualified GA build when logging into the Orchestrator. The R451-20220517-GA Orchestrator build is a genuine GA build except the Beta license agreement was mistakenly left in and its display is purely cosmetic.

Resolved in Orchestrator Version R451-20220517-GA

The below issues have been resolved since Orchestrator Version R450-20220502-GA. 

Note:

Release 5.1.0 contains all Orchestrator fixes that are listed in the 4.5.0 Release Notes.

  • Fixed Issue 45078: When configuring a VNF for a customer on the VMware SD-WAN Orchestrator, if a VNF state is configured at the Profile level one way, and then configured a different way at the site level using Edge Override, when Edge Override is later deactivated, the site continues to use the Edge Override settings and does not revert back to the Profile settings as expected.

    This issue occurs when configuring a VNF Insertion parameter on a Configuration Profile where the opposite setting is configured for a site using Edge Override and later Edge Override is itself deactivated, but the setting persists.

  • Fixed Issue 53751: Connected route validation in static route settings fails in a VLAN without cidrIp and cidrPrefix.

    The issue occurs when the user create a VLAN without a cidrIp and cidrPrefix.

  • Fixed Issue 54015: User is able to configure an ICMP probe name with special characters and scripts on the VMware SASE Orchestrator.

    On configuring an ICMP Probe with script '<script>alert('test');</script>' and 'test<script>alert('test');</script>' as input for ICMP probe name it shows error as “An unsafe character “<” and “>” was found in a field while saving". What should be returned is: {"error":{"code":-32603,"message":"script injection error"}}.

  • Fixed Issue 59784: When a user is attempting to access a screen on a VMware SASE Orchestrator that that user does not have the permissions to access, the user is shown an empty screen with an endless spinner.

    User is not automatically redirected to an "Access Denied" screen when trying access a prohibited screen/route (for example, Device Settings where they have a read only role) via a "deep link", which is a link that points to the specific prohibited section of the customer enterprise versus just loading the main page.

  • Fixed Issue 62624: User sees the Customer name when attempting to uncheck the Partner Gateway checkbox while the Partner Gateway is in use.

    When a user unchecks the Partner Gateway checkbox for a particular Gateway on the VMware SD-WAN Orchestrator UI while the Gateway is used by one or more customers and a customer profile as well, the Orchestrator shows only the name of the Profile and Edge not the name of the customer names using the Gateway.

  • Fixed Issue 67209: When configuring a Cloud Security Service (CSS) on a VMware SASE Orchestrator, the Cloud Name does not align with the CSS configured.

    The lack of alignment between the Cloud Name and the CSS may cause confusion for the customer.

  • Fixed Issue 67912: Tunnel overall status is different when looking at the Network Services page versus the Edge Monitoring page on a VMware SASE Orchestrator.

    When a user goes to the Edge Monitoring page and checks the tunnel's overall status, the user may observe a different status from what they on the Network Services monitoring page when checking a tunnel's overall status in the Cloud Security Service (CSS) section which results in confusion as to the real status of that tunnel.

  • Fixed Issue 68387: When a user attempts to create a new operator user or customer administrator with a non-native type (in other words, a user not using a username/password but SSO or RADIUS authentication), the VMware SASE Orchestrator still requires the user to input a password for the new user.

    When a user tries to create a new customer or operator user, the Orchestrator UI does not clear the password field. And on the back-end, instead of first checking if user type is non-native, it first runs a password strength check and throws an error.

  • Fixed Issue 68463: When using the New UI on the VMware SASE Orchestrator and looking at the QoE section, the wrong graph values are shown.

    When going on QoE in the old UI "Latency Fair" is displayed on the graph, whereby when visiting the new UI (for the same Edge and time) "Jitter Fair" is displayed. This is caused by QoE being incorrectly mapped on the New UI.

    Without the fix, the only workaround is to use the Old Orchestrator UI to confirm correct QoE values.

  • Fixed Issue 69181: If a user configures a secondary fronted by IPsec Gateway in the Device Settings > Gateway Handoff section of the VMware SASE Orchestrator, the IPsec tunnel is not established with the secondary fronted by IPsec Gateway.

    The user would not observe the IPsec configuration applied and present when looking at debug.py --gateways.

  • Fixed Issue 69514: Report generation may fail with error "Failed to update blob data for PDF".

    When the user tries to create an offline report, the report generation fails with an Internal error. This was happening due to a wrong version for the chart generation library

  • Fixed Issue 69534: For a customer using VMware Edge Network Intelligence, when a user clicks anywhere on the main Monitoring page of the customer in the Orchestrator's New UI, the links for 'Application Analytics' and 'Branch Analytics' disappear.

    If the user reloads the page, those links will reappear until the user clicks anywhere on the same page and then they are gone once again. Because of this the user will not be able to view "Application Analytics" and "Branch Analytics" monitoring data unless they know to reload the page.

  • Fixed Issue 69869: When using the New UI on the VMware SASE Orchestrator, a customer's BFD configuration is editable for customer administrators even in when the selected segment is not delegated to the customer.

    Where a customer enterprise has two segments, and the first segment is editable for customer administrators, and another segment is not delegated to the customer enterprise. The ability to configure BFD is not blocked when a customer administrator switches from the editable segment to the non-editable segment.

  • Fixed Issue 69932: When using the New UI on the VMware SASE Orchestrator, if an Operator or Partner Administrator switches between customers several times, the application switcher (which selects SD-WAN/Cloud Web Security/Secure Access/Edge Network Intelligence) may disappear from the UI.

    In addition, when switching between customers, a customer may show the settings for the previous customer (for example, having Cloud Web Security available when that service is not enabled for that customer.)

  • Fixed Issue 69981: The tunnel overall status shows differently when looking at the Monitor > Edges page versus the Monitor > Network Services page on the VMware SASE Orchestrator.

    When a user consults the Monitor > Edges page and notes a tunnel's overall status and then compares this status with what is seen on the Monitor > Network Services page, the user may observe the tunnel's overall status in the Cloud Security Service (CSS) section will not match the status seen on the Monitor > Edges page.

  • Fixed Issue 70018: A VMware SD-WAN Orchestrator running Release 4.3.0 or higher may not be able to form a Disaster Recovery pair.

    The root cause prevents the Orchestrator from getting free disk space size necessary to do DR and the DR pairing may fail as a result with an error message "not enough disk space to copy DB".

  • Fixed Issue 70350: On the VMware SASE Orchestrator's New UI, any user can configure OSPF settings even if they do not have priviliges to do so.

    The New UI does not include a user privilege check for Device Settings > OSPF configuration.

  • Fixed Issue 70528: Custom composite roles for a customer are not displayed under session tracking section of the authentication page of the VMware SASE Orchestrator and cannot be configured for session limits.

    This issue is encountered on an Orchestrator running a 4.5.0 version in which the custom composite role capability was introduced. After creating a custom role, the customer will not be able to configure session limits for the new role.

  • Fixed Issue 71242: User is not be able to configure BFD in the VMware SASE Orchestrator UI while configuring Non SD-WAN Destination (NSD) via Gateway.

    On configuring NSD via Gateway through Configure > Network Services, on opening BFD dialog box, the UI does not provide any option to configure the BFD parameters.

  • Fixed Issue 71778: When a VMware SASE Orchestrator deployed in a Disaster Recovery (DR) topology is upgraded to Release 4.5.0, an Operator is not able to manually switch Gateways for a Non-SD-WAN Destination (NSD) object.

    The API call to switch the Gateway started enforcing request validation from 4.5.0, however the Orchestrator UI client did not provide a necessary parameter for the API.

  • Fixed Issue 71898: When configuring a RADIUS Service, if a configuration field is left empty and the user attempts to save the configuration, the VMware SASE Orchestrator UI throws a general error.

    The error message "An unexpected error has occurred" gives no insight to the user as to what needs to be corrected. In addition the configuration field that needs to be corrected is not highlighted.

  • Fixed Issue 72039: When working on the Configure > Device page of the VMware SASE Orchestrator, the user is unable to sort features by those that are segment-aware and those that are segment-agnostic.

    Some device settings are applied across segments versus those that must be configured for each segment, and there is no way to just sort the UI to view those settings for ease of use.

  • Fixed Issue 72444: When a user configures IPv4 and IPv6 BGP neighbors for a VMware SD-WAN Edge and then attempts to turn off the BGP settings using the On/Off sliding button, the configuration is not saved and the VMware SASE Orchestrator UI says "No Changes".

    In this rare case where a user is both configuring BGP setting but also turning off BGP, the users actions are not being saved for BGP settings as they should.

  • Fixed Issue 73234: For a customer enterprise where Dynamic Cost Calculation (DCC) is not used and there are a large number of learned routes, the VMware SASE Orchestrator may take a long time to advertise BGP routes after BGP is either initiated or restarted.

    This issue affects an enterprise with at least a few hundred learned routes and where there is an even greater number of routes (for example, ~2K routes), the Orchestrator can take 10 minutes or longer to advertise all the routes. The fix for the issue improves the speed of route convergence in the absence of DCC.

  • Fixed Issue 74251: The VMware SD-WAN Edge does not honor the port number in a LAN-side NAT rule that was configured through the Orchestrator API.

    This issue does NOT impact users who configure LAN-side NAT rules through the Orchestrator UI. The port configured as part of the LAN side NAT rule is not pushed properly by the VMware SASE Orchestrator to the Edge. In cases where a LAN-side NAT rule is configured via the updateConfigurationModule API method and a string value was passed for "insidePort" or "outsidePort", the Orchestrator API previously allowed the request. However, the Edge does not honor those string values (it expects integers instead). The Orchestrator API validation logic has been modified to reject string values (and now behaves in a manner consistent with what was already documented in the API reference).

  • Fixed Issue 74592: Link statistic queries for a longer period of time (for example, one year) take a long time to return a result on the VMware SASE Orchestrator.

    The queries take a long time because of the absence of enterpriseLogicalID and the Orchestrator lacking sufficient checks in the code to ensure the timestamp format.

  • Fixed Issue 75186: When software bonding is setup on either a VMware SASE Orchestrator or a VMware SD-WAN Gateway, the bonding interface MAC address is not unique across the system which may cause MAC address conflicts in the network.

    This will only occur if the systems are deployed on the same subnet with software bonding enabled. The MAC address for the software bonding interface is derived from /etc/machine-id. That file is not randomized during the deployment on Orchestrator and Gateway versions prior to Release 4.2.1.

    A customer who uses software bonding on an Orchestrator or Gateway with systems deployed on the same subnet from a software image prior to 4.2.1, should contact VMware support for the resolution.

  • Fixed Issue 75431: When examining a Non SD-WAN Destination (NSD) configuration on the VMware SASE Orchestrator's New UI, the 'Local Auth ID' (FQDN information) is missing.

    A user navigating to Monitor > Network Services > Non SD-WAN Destinations via Gateway and then selecting Details would not see the 'Local Auth ID' field which contains the FQDN information. This information displays properly on the Classic UI.

  • Fixed Issue 75656: In some instances, the VMware SASE Orchestrator after processing a VMware SD-WAN Edge's routing events replies with an empty ack (which as bad as a non reply) and this results in increased load on the Orchestrator.

    When an Orchestrator sends an empty ack to an Edge's routing events, the Edge retries the routes continuously causing the increase in load on the Orchestrator.

  • Fixed Issue 75879: In some instances, the VMware SASE Orchestrator does not reply to a VMware SD-WAN Edge's routing events and this results in an increased load on the Orchestrator.

    When an Orchestrator does not reply to an Edge's routing events, the Edge retries the routes continuously causing the increase in load on the Orchestrator. This is the permanent fix for this issue.

  • Fixed Issue 75895: An Edge Cloud Security Service (CSS) tunnel alert generation may be skipped for some CSS tunnel events.

    When a customer has an Edge configured with a Cloud Security Service, and there are tunnel Up/Down events on the Edge at the same time, the VMware SASE Orchestrator may fail to generate alerts for all events.

  • Fixed Issue 76001: Dates in graphs on the Monitor > Edges pages may differ slightly between the Classic UI and the New UI user interfaces on the VMware SASE Orchestrator.

    If a user navigates to the Monitor Edges list page in both the Classic UI and the New UI and then selects an Edge and views the Transport tab (Classic UI) or Links tab (New UI), they would note that the times do not exactly match.

  • Fixed Issue 76008: When cloning an enterprise on a VMware SASE Orchestrator, if the chosen Operator Profile is not originally assigned to the parent enterprise, the clone operation fails.

    If the new enterprise is assigned a software image that is NOT assigned to the enterprise being cloned, then the API call will fail with an error.

    Without the fix, the workaround for this issue is to initially assign the new enterprise the same software image as the cloned enterprise and then change to the desired operator profile afterwards.

  • Fixed Issue 76016: When a Partner Gateway handoff is set after upgrading a VMware SASE Orchestrator to Release 4.3.0 or later, any subcomponent inside the handoff, like BGP/BFD/static routes are not allowed to be deleted from the Orchestrator UI or API by a Partner administrator.

    This issue occurred due to a regression in logic for handling subcomponent configuration for deletion on the Orchestrator's backend. Previously a fix was added for mixed gateway pools (cloud + partner gateways) where if a Partner administrator modified a Partner Gateway, it does not affect the Cloud Gateway handoff configuration. However this fix did not account for cases where deletion of subcomponents is involved.

    Without the fix, there are two workarounds: a) The partner could request an Operator user (for example, VMware Support) could perform the deletion without any problems. It affects only MSP users. or b) The entire handoff for the Partner Gateway could be removed and recreated with the revised configuration.

  • Fixed Issue 76078: Some role privileges are removed if a VMware SASE Orchestrator is upgraded to 4.5.0. If the role customization package contains that list of privileges then the package will not be applied in the Orchestrator.

    After an Orchestrator upgrade from a lower release to 4.5.0, the Orchestrator does not apply existing role customization packages when the list of removed privileges in 4.5.0 is available in that package.

  • Fixed Issue 76442: Spurious API validation errors observed when updating a customer's Partner Gateway handoff configurations following the removal of a Gateway from the customer's assigned Gateway Pool.

    When a Partner Gateway for which handoff settings had previously been configured is removed from a Gateway pool, the Orchestrator does not purge handoff settings for that Gateway from customer-level handoff settings. As a result, subsequent attempts to update the customer-level handoff settings via API fail unless the API client deliberately removes the stale Gateway settings.

    Without the fix, the customer can work around this issue by doing the following: when updating customer Gateway Handoff settings, API clients can proactively ensure that all gateways represented in the settings are currently present in the customer gateway pool.

  • Fixed Issue 76508: When a VMware SASE Orchestrator is upgraded, customers who use BGP may notice that under Network Services > BGP Neighbor State, their VMware SD-WAN Edges show a state of REMOVED when the Edge BGP routes are actually ESTABLISHED.

    When this issue is encountered, the actual Edge BGP state can be confirmed using Remoted Diagnostics > Troubleshoot BGP - Show BGP Summary. The issue is caused by a combination of an Edge sending the wrong Segment ID to the Orchestrator for the BGP Neighbor Summary attribute combined with a 4.5.0 Orchestrator code change that introduces the REMOVED state in the BGP Neighbor Summary. The BGP Neighbor Summary compares the previous record and calculates whether the connection is ESTABLISHED or REMOVED on the basis of a unique key comprised of the BGP Neighbor Address and the Segment ID. Due to the inaccurate Segment ID, the Orchestrator UI displays the REMOVED record as the latest.

    Should a customer encounter this issue on an Orchestrator without the fix, bouncing BGP and reestablishing the routes will correct the BGP status for all affected Edges.

  • Fixed Issue 77515: When a Partner Administrator with a Superuser role assigns a software image to a customer on the Configure > Customer page, an error is thrown when attempting to save the change and the action fails.

    For a partner user while logging the event 'Modified the assigned software image list' to update 'Assigned Software/Firmware Images', if no Software/Firmware Images were removed for that event under 'removedSoftwareImages', it was processing and listing images from all the imageUpdate modules present in VELOCLOUD_CONFIGURATION_MODULE for the particular operator. In effect the logic is broken for this process when a partner user performs the action and this causes the error and the failure to change customer's default software image.

  • Fixed Issue 78684: For a customer enterprise which has uses an Azure type Non SD-WAN Destination via Gateway, a re-sync may not propagate the static routes in the configuration as expected.

    From the Orchestrator UI, when a subnet is added or removed, a set of parameters is passed via the API. But during re-sync the parameters are not similar to the one that API is expecting. Because of this, the re-sync option is not working correctly. When this issue occurs even if the subnets were updated from the Azure environment, it may not properly propagate to other places of the configuration.

  • Fixed Issue 79981: Some user interface terms in localized versions of the VMware SASE Orchestrator UI lack clarity for global customers looking at non-English versions of the UI.

    Interface term translations are improved in the 4.5.1 Orchestrator. These improvements are derived from customer feedback and result in an overall improved user interface translation for global customers.

  • Fixed Issue 80197: When a user creates new Cloud Security Service (CSS) tunnels (GRE) to Zscaler, from the VMware SD-WAN Edge perspective and from Zscaler's perspective these tunnels were successfully deployed and tunnel events are visible but the tunnel status is not shown on the VMware SASE Orchestrator.

    The default tunneling protocol option for automated deployment is IPsec. When the user selects the GRE option and toggles automated deployment to 'True', the default option is being saved as GRE which causes the Non SD-WAN Destination records to be skipped and monitoring missed tunnel event.

    Without the fix, a user needs to create a CSS provider without ever toggling back and forth on the Tunneling Protocol checkbox.

  • Fixed Issue 80930: If a user adds a new Business Policy to a customer enterprise that uses Non SD-WAN Destinations (NSD) via Edge where IPsec tunnels are configured, there is the potential for creating duplicate IPsec tunnel entries.

    When this issue occurs, the duplicate entries can be observed in the Configure > Device Settings > Branch to Non SD-WAN Destination via Edge section of the Orchestrator UI when a Business Policy is added. These duplicate entries need to be deleted and the configuration details need to be re-entered when this issue is encountered.

  • Fixed Issue 80963: When changing the time interval range on the Edge > Monitor > QoE tab, a sample reflected at one timestamp may change and shift over to a new timestamp depending on the selected time interval range.

    A rounding issue in sample sizes causes timestamps to shift earlier for longer time ranges queried on the QoE tab. For example, an event that occurred at 10:02 A.M. will show properly for small time ranges (for example, one hour), but when querying for a larger time range (for example, six hours), the sample might instead show up earlier at say 10:00 A.M., which results in confusion for the user.

  • Fixed Issue 81835: The Monitor > Edge > QoE page of the VMware SASE Orchestrator UI may not accurately represent a WAN link's status (whether it is online, offline, or degraded) or accurately represent link metrics for the time period selected.

    Different time intervals can lead to the QoE graph showing different results for a WAN link status. And for a link's metrics, the QoE graph may present a particular QoE value (latency, loss, or jitter) that does not reflect the real metric value at that exact time.

    This issue is caused by multiple WAN links belonging to different enterprises being assigned the same link logical ID which leads to a malfunction in the Orchestrator's link data backfill process. The Orchestrator erroneously assumes the WAN link logical Id to be unique because it is not tied to a customer's enterprise ID. This allows for duplicate link logical IDs and the possibility of incorrect link metrics and status.

    The fix for this issue stores the link keys in the Orchestrator's database as a combination of the customer enterprise logical ID and the WAN link's logical ID, ensuring each WAN link is unique.

  • Fixed Issue 82725: A VMware SASE Orchestrator may not generate the password reset link correctly.

    This issue occurs when the URL for the Orchestrator is not exactly https://domain/ or https://domain/operator/. However, if for example the URL is https://domain/test/ the password reset link does not work and directs you back to the login page.

    On Orchestrators without the fix: If the Orchestrator URL cannot be corrected to a URL as shown above, the only option is for a Superuser or Operator to manually enter a new password for the user and then share that with the affected user so that they could in turn reconfigure a different password for themselves once they were successfully logged back in.

  • Fixed Issue 82839: If a user performs an IPsec automation deletion action for a Zscaler Cloud Security Service tunnel on a VMware SASE Orchestrator, the action also deletes all the VPN credentials associated with the respective Zscaler location resulting in deletion of the location itself.

    The IPsec automation deletion action should only delete the VPN credential associated with it from the Zscaler Location. All other VPN credentials associated to the respective Zscaler Location should remain untouched

  • Fixed Issue 83165: An Operator user is not able to transfer a Customer to a Partner on the VMware SASE Orchestrator with the reason that they do not have the same Gateway Pool, even though both do have the same Gateway Pool.

    This is caused by an API call network/getNetworkEnterpriseProxies not returning the Gateway Pool details and leading the Orchestrator to think the Partner and Customer do not have the same Gateway Pool and rejecting the assignment.

  • Fixed Issue 83699: When a VMware SD-WAN Gateway is set to quiesced mode from the VMware SASE Orchestrator's New UI, when the user selects a new replacement Gateway, the Orchestrator does not allow any configuration changes to the replacement Gateway.

    This issue happens after activating the Non SD-WAN Destination migration process via the Orchestrator's New UI part of which is selecting a New Gateway, which is the Gateway replacing the quiesced Gateway. Once that New Gateway is designated as the replacement Gateway, when attempting to make any configuration change to the replacement Gateway the Operator would observe an error message thrown similar to: "GATEWAY_SERVICE_STATE_INVALID: Cannot change the state of the gateway to null, as it is already used as a replacement gateway".

Cloud Web Security Resolved Issues

Resolved in Version R451-20220517-GA

The below issues have been resolved since Version R450-20220502-GA.

  • Fixed Issue 69211: When adding a rule for URL Filtering to the Cloud Web Security service, the UI does fully display the domain name prompt for Select Source and Destination.

    The domain should be fully displayed to ensure it has been properly configured.

  • Fixed Issue 69346: A user is able to delete a Cloud Web Security policy which is associated with Secure Access traffic.

    User navigated to the Cloud Web Security UI and Configure tab and selects a Security Policy that is associated with Secure Access traffic and they elect to delete it, the attempt is successful when it should fail and throw an error.

  • Fixed Issue 69959: Traffic loss is observed when the VMware Cloud Web Security service is toggled.

    After a user activates or deactivates Cloud Web Security, default routes for Cloud Web Security are marked as False though the probe is up, and due to the reachability of the CWS routes not being updated, the result is traffic getting dropped.

  • Fixed Issue 70005: When using VMware Cloud Web Security, a user can edit an existing Security Policy and rename it an empty or blank name and save it.

    A user cannot create a Security Policy with an empty/blank name but can edit an existing policy to configure the name to be blank and the Orchestrator permits the change and does not throw an error.

  • Fixed Issue 71073: For customers using the VMware Cloud Web Security service, if CASB is configured as off for a customer enterprise, a customer administrator would still see a VMware SASE Orchestrator UI that includes that feature.

    This only affects customer administrators. Operators and partner administrators would see the correct UI screen.

  • Fixed Issue 74013: The VMware SASE Orchestrator does not adjust for a change in the CASB license from Advanced to Standard.

    If a customer starts with an Advanced CASB license and configures a rule specific to that license, and the customer is later switched to a Standard CASB license, the user can change the rule created by the Advanced CASB license and apply the changes.

  • Fixed Issue 86083: For a customer using the VMware Cloud Web Security service, the Cloud Web Security Event Log is missing important details.

    The Event log does not include the following details:

    • For configuration changes, the username is included in the event but not what change was made.

    • When a Cloud Web Security rule is deleted, the Orchestrator does not include the deleted data with the event.

    • Turning Authentication on or off does not include a timestamp for the event.

    • For CASB events, only the application ID is included, not the application name.

Secure Access Resolved Issues

Resolved in Version R451-20220517-GA

The below issues have been resolved since Version R450-20220502-GA.

  • Fixed Issue 70493: When a customer edits a VMware Secure Access service configuration and deactivates/removes association of a Cloud Web Security policy, saving the configuration fails.

    Editing a Secure Access service configuration where a Cloud Web Security policy is being removed fails with an "Invalid CWS Policy" error.

  • Fixed Issue 71078: A customer may observe that editing a Secure Access deployment fails when the customer subnet bits are modified causing the Secure Access service to be rendered unusable.

    Without a fix, the customer can delete the existing SA service and recreate it with the new customer subnets and that would cause the deployment to be successful.

  • Fixed Issue 70098: Deploying a VMware Secure Access service will complete but fail to fetch the configuration from the UEM OR route traffic.

    The VMware Gateway receives 'ras' configuration as part of the control plane configuration. However, the configuration is seen to be empty for all Secure Access services associated with a Gateway. This is caused by a parsing error introduced by the recent changes in Secure Access enterprise object for tunnel service health check.

Known Issues

Open Issues in Release 4.5.1

Edge/Gateway Known Issues

  • Issue 14655:

    Plugging or unplugging an SFP adapter may cause the device to stop responding on the Edge 540, Edge 840, and Edge 1000 and require a physical reboot.

    Workaround: The Edge must be physically rebooted. This may be done either on the Orchestrator using Remote Actions > Reboot Edge, or by power-cycling the Edge.

  • Issue 25504:

    Static route costs greater than 255 may result in unpredictable route ordering.

    Workaround: Use a route cost between 0 and 255

  • Issue 25595:

    A restart may be required for changes to static SLA on a WAN overlay to work properly.

    Workaround: Restart Edge after adding and removing Static SLA from WAN overlay.

  • Issue 25742:

    Underlay accounted traffic is capped at a maximum of the capacity towards the VMware SD-WAN Gateway, even if that is less than the capacity of a private WAN link which is not connected to the Gateway.

  • Issue 25758:

    USB WAN links may not update properly when switched from one USB port to another until the VMware SD-WAN Edge is rebooted.

    Workaround: Reboot the Edge after moving USB WAN links from one port to another.

  • Issue 25855:

    A large configuration update on the Partner Gateway (e.g. 200 BGP-enabled VRFs) may cause latency to increase for approximately 2-3 seconds for some traffic via the VMware SD-WAN Gateway.

    Workaround: No workaround available.

  • Issue 25921:

    VMware SD-WAN Hub High Availability failover takes longer than expected (up to 15 seconds) when there are three thousand branch Edges connected to the Hub.

  • Issue 25997:

    The VMware SD-WAN Edge may require a reboot to properly pass traffic on a routed interface that has been converted to a switched port.

    Workaround: Reboot the Edge after making the configuration change.

  • Issue 26421:

    The primary Partner Gateway for any branch site must also be assigned to a VMware SD-WAN Hub cluster for tunnels to the cluster to be established.

  • Issue 28175:

    Business Policy NAT fails when the NAT IP overlaps with the VMware SD-WAN Gateway interface IP.

  • Issue 31210:

    VRRP: ARP is not resolved in the LAN client for the VRRP virtual IP address when the VMware SD-WAN Edge is primary with a non-global CDE segment running on the LAN interface.

  • Issue 32731:

    Conditional default routes advertised via OSPF may not be withdrawn properly when the route is turned off. Reenabling and then deactivating the route will retract it successfully.

  • Issue 32960:

    Interface “Autonegotiation” and “Speed” status might be displayed incorrectly on the Local Web UI for activated VMware SD-WAN Edges.

  • Issue 32981:

    Hard-coding speed and duplex on a DPDK-enabled port may require a VMware SD-WAN Edge reboot for the configurations to take effect as it requires turning off DPDK.

  • Issue 34254:

    When a Zscaler CSS is created and the Global Segment has FQDN/PSK settings configured, these settings are copied to Non-Global Segments to form IPsec tunnels to a Zscaler CSS.

  • Issue 35778:

    When there are multiple user-defined WAN links on a single interface, only one of those WAN links can have a GRE tunnel to Zscaler.

    Workaround: Use a different interface for each WAN link that needs to build GRE tunnels to Zscaler.

  • Issue 36923:

    Cluster name may not be updated properly in the NetFlow interface description for a VMware SD-WAN Edge which is connected to that Cluster as its Hub.

  • Issue 38682:

    A VMware SD-WAN Edge acting as a DHCP server on a DPDK-enabled interface may not properly generate “New Client Device" events for all connected clients.

  • Issue 38767:

    When a WAN overlay that has GRE tunnels to Zscaler configured is changed from auto-detect to user-defined, stale tunnels may remain until the next restart.

    Workaround: Restart the Edge to clear the stale tunnel.

  • Issue 39134:

    The System health statistic “CPU Percentage” may not be reported correctly on Monitor > Edge > System for the VMware SD-WAN Edge, and on Monitor > Gateways for the VMware SD-WAN Gateway.

    Workaround: Users should use handoff queue drops for monitoring Edge capacity not CPU percentage.

  • Issue 39374:

    Changing the order of VMware SD-WAN Partner Gateways assigned to a VMware SD-WAN Edge may not properly set Gateway 1 as the local Gateway to be used for bandwidth testing.

  • Issue 39608:

    The output of the Remote Diagnostic “Ping Test” may display invalid content briefly before showing the correct results.

  • Issue 39624:

    Ping through a subinterface may fail when the parent interface is configured with PPPoE.

  • Issue 39753:

    Turning off Dynamic Branch-to-Branch VPN may cause existing flows currently being sent using Dynamic Branch-to-Branch to stall.

  • Issue 40096:

    If an activated VMware SD-WAN Edge 840 is rebooted, there is a chance an SFP module plugged into the Edge will stop passing traffic even though the link lights and the VMware SD-WAN Orchestrator will show the port as 'UP'.

    Workaround: Unplug the SFP module and then replug it back into the port.

  • Issue 40421:

    Traceroute is not showing the path when passing through a VMware SD-WAN Edge with an interface configured as a switched port.

  • Issue 42278:

    For a specific type of peer misconfiguration, the VMware SD-WAN Gateway may continuously send IKE init messages to a Non-SD-WAN peer. This issue does not disrupt user traffic to the Gateway; however, the Gateway logs will be filled with IKE errors and this may obscure useful log entries.

  • Issue 42388:

    On a VMware SD-WAN Edge 540, an SFP port is not detected after deactivating and then reenabling the interface from the VMware SD-WAN Orchestrator.

  • Issue 42488:

    On a VMware SD-WAN Edge where VRRP is enabled for either a switched or routed port, if the cable is disconnected from the port and the Edge Service is restarted, the LAN connected routes are advertised.

    Workaround: There is no workaround for this issue.

  • Issue 42872:

    Enabling Profile Isolation on a Hub profile where a Hub cluster is associated does not revoke the Hub routes from the routing information base (RIB).

  • Issue 43373:

    When the same BGP route is learnt from multiple VMware SD-WAN Edges, if this route is moved from preferred to eligible exit in the Overlay Flow Control, the Edge is not removed from the advertising list and continues to be advertised.

    Workaround: Enable distributed cost calculation on the VMware SD-WAN Orchestrator.

  • Issue 44995:

    OSPF routes are not revoked from VMware SD-WAN Gateways and VMware SD-WAN Spoke Edges when the routes are withdrawn from the Hub Cluster.

  • Issue 45189:

    With source LAN side NAT is configured, the traffic from a VMware SD-WAN Spoke Edge to a Hub Edge is allowed even without the static route configuration for the NAT subnet.

  • Issue 45302:

    In a VMware SD-WAN Hub Cluster, if one Hub loses connectivity for more than 5 minutes to all of the VMware SD-WAN Gateways common between itself and its assigned Spoke Edges, the Spokes may in rare conditions be unable to retain the hub routes after 5 minutes. The issue resolves itself when the Hub regains contact with the Gateways.

  • Issue 46053:

    BGP preference does not get auto-corrected for overlay routes when its neighbor is changed to an uplink neighbor.

    Workaround: An Edge Service Restart will correct this issue.

  • Issue 46216:

    On a Non SD-WAN Destinations via Gateway or Edge where the peer is an AWS instance, when the peer initiates Phase-2 re-key, the Phase-1 IKE is also deleted and forces a re-key. This means the tunnel is torn down and rebuilt, causing packet loss during the tunnel rebuild.

    Workaround: To avoid tunnel destruction, configure the Non SD-WAN Destinations via Gateway/Edge or CSS IPsec rekey timer to less than 60 minutes. This prevents AWS from initiating the re-key.

  • Issue 46391:

    For a VMware SD-WAN Edge 3800, the SFP1 and SFP2 interfaces each have issues with Multi-Rate SFPs (i.e. 1/10G) and should not be used in those ports.

    Workaround: Please use single rate SFP's per the KB article VMware SD-WAN Supported SFP Module List (79270). Multi-Rate SFPs may be used with SFP3 and SFP4.

  • Issue 47084:

    A VMware SD-WAN Hub Edge cannot establish more than 750 PIM (Protocol-Independent Multicast) neighbors when it has 4000 Spoke Edges attached.

  • Issue 47664:

     In a Hub and Spoke configuration where Branch-to-Branch via Hub VPN is not enabled, trying to U-turn Branch-to-Branch traffic using a summary route on an L3 switch/router will cause routing loops.

    Workaround: Configure Cloud VPN to enable Branch-to-Branch VPN and select “Use Hubs for VPN”.

  • Issue 47681:

    When a host on the LAN side of a VMware SD-WAN Edge uses the same IP as that Edge’s WAN interface, the connection from the LAN host to the WAN does not work.

  • Issue 48166:

    A VMware SD-WAN Virtual Edge on KVM is not supported when using a Ciena virtualization OS and the Edge will experience recurring Dataplane Service Failures.

  • Issue 48502:

    In some scenarios, a VMware SD-WAN Hub Edge being used to backhaul internet traffic may experience a Dataplane Service Failure due the improper handling of backhaul return packets.

  • Issue 48530:

    VMware SD-WAN Edge 6x0 models do not perform autonegotiation for triple speed (10/100/1000 Mbps) copper SFP's.

    Workaround: Edge 520/540 supports triple speed copper SFPs but this model has been marked for End-of-Sale by Q1 2021.

  • Issue 48597: Multihop BGP neighborship does not stay up if one of the two paths to the peer goes down.

    If there is a Multihop BGP neighborship with a peer to which there are multiple paths and one of them goes down, user will notice that the BGP neighborship goes down and does not come up using the other available path(s). This includes the Local IP-loopback neighborship case too.

    Workaround: There is no workaround for this issue.

  • Issue 48666:

    IPsec-fronted Gateway Path MTU calculation does not account for 61 Byte IPsec overhead, resulting in higher MTU advertisement to LAN client and subsequent IPsec packet fragmentation.

    Workaround: There is no workaround for this issue.

  • Issue 50518:

    On a VMware SD-WAN Gateway where PKI is enabled, if >6000 PKI tunnels attempt to connect to the Gateway, the tunnels may not all come up because inbound SAs do not get deleted.

    Note:

    Tunnels using pre-shared key (PSK) authentication do not have this issue.

  • Issue 51436: For a site using an Enhanced High-Availability topology while deploying a VMware SD-WAN Edge using an LTE modem, if the site gets into a "split-brain" state, the HA failover takes ~5-6 minutes.

    As part of the recovery from a split-brain state, the LAN ports are brought down on the Active Edge and this impacts LAN traffic during the time the ports are down and until the site can recover.

    Workaround: There is no workaround for this issue.

  • Issue 53219: After a VMware SD-WAN Hub Cluster rebalances, a few Spoke Edges may not have their RPF interface/IIF set properly.

    On the affected Spoke Edges, multicast traffic will be impacted. What happens is that after a cluster rebalance, some of the Spoke Edge fail to send a PIM join.

    Workaround: This issue will persist until the affected Spoke Edge has an Edge Service restart.

  • Issue 53337: Packet drops may be observed with an AWS instance of a VMware SD-WAN Gateway when the throughput is above 3200 Mbps.

    When traffic exceeds a throughput above 3200 Mbps and a packet size of 1300 bytes, packets drops are observed at RX and at IPv4 BH handoff.

    Workaround: There is no workaround for this issue.

  • Issue 53934: In an enterprise where a VMware SD-WAN Hub Cluster is configured, if the primary Hub has Multihop BGP neighborships on the LAN side, the customer may experience traffic drops on a Spoke Edge when there is a LAN side failure or when BGP is not enabled on all segments.

    In a Hub cluster, the primary Hub has Multihop BGP neighborship with a peer device to learn routes. If the physical interface on the Hub by which BGP neighborship is established, goes down, then BGP LAN routes may not become zero despite BGP view being empty. This may cause Hub Cluster rebalancing to not happen. The issue may also be observed when BGP is not enabled for all segments and when there are one or more Multihop BGP neighborships.

    Workaround: Restart the Hub which had the LAN-side failure (or where BGP is not enabled).

  • Issue 57210: Even when a VMware SD-WAN Edge is working normally and is able to reach the internet, the LED in the Local UI's Overview page shows as "Red".

    The Edge's Local UI determines the Edge's connectivity by whether it can resolve a well-known name via Google's DNS resolver (8.8.8.8). If it cannot do so for any reason, then it thinks it is offline and shows the LED as red.

    Workaround: There is no workaround for this issue, except to ensure that DNS traffic to 8.8.8.8 can reach the destination and be resolved successfully.

  • Issue 61543: If more than one 1:1 NAT rule is configured on different interfaces with the same Inside IP, the inbound traffic can be received on one interface and the outbound packets of the same flow can be routed via different interface.

    For the NAT flows from Outside to Inside, the 1:1 NAT rules will be matched against the Outside IP and the interface where the packets are received. For the outbound packets of the same flow, the VMWare SD-WAN Edge will try to match the NAT rules again comparing the Inside IP and the outbound traffic can be routed via the interface configured in the first matching rule with "Outbound Traffic" enabled.

    Workaround: There is no workaround for this issue outside of ensuring no more than one 1:1 NAT rule is configured with a particular Inside IP address.

  • Issue 62701: For a VMware SD-WAN Edge deployed as part of an Edge Hub Cluster, If Cloud VPN is not enabled under the Global Segment but is enabled under a Non-Global Segment, a control plane update sent by the Orchestrator may cause all the WAN links to flap on the Hub Edge.

    The Hub Edge's WAN links going down, then up in rapid succession (flap) will impact real time traffic like voice calls. This issue was observed on a customer deployment where Cloud VPN was not enabled on the Hub Edge's Global segment, but the Cluster configuration was enabled which means this Hub Edge was part of a Cluster (and a Cluster configuration is applicable to all segments). When a configuration change is pushed to the Hub Edge, the Hub Edge's dataplane will start parsing data and will start with the Global Segment where it will see Cloud VPN not enabled and the Hub Edge erroneously thinks clustering is not enabled on this Global Segment. As a result, the Hub Edge will tear down all tunnels from the Hub's WAN link(s) which will cause link flaps on all that Edge's WAN links. For any such incident the WAN links only go down and recover a single time per control pane update.

    Workaround: The workaround is to activate Cloud VPN on all segments, meaning the Global Segment and all Non-Global Segments.

  • Issue 63629: In a network topology where the VMware SD-WAN Hub Edge and Spoke Edge have different IP family preferences (in other words, IPv4/IPv6 dual stack), the customer can see more bandwidth allocated to the peer than expected.

    If both families IPv4 and IPv6 are enabled, the Edge internally creates two different link objects. The bandwidth values are added for both of them when it should be added only for one.

    Workaround: The workaround for this issue is to not have different tunnel preferences if Hub/Spoke topology have dual stack enabled.

  • Issue 64627: A VMware SD-WAN Gateway may experience a Dataplane Service restart, causing 3-5 seconds of traffic disruption.

    When there are a large number of sub-paths configured on a VMware SD-WAN Edge's WAN links or there are frequent flaps of of the VeloCloud Management Plane (VCMP) tunnels, it may lead to the exhaustion of the counters and the ultimate Dataplane Service restart of the Gateway.

    Workaround: There is no workaround for this issue.

  • Issue 65560: Traffic from a customer to PE (Provider Edge) device fails.

    BGP neighborship between a Partner Gateway and Provider Edge does not get established when tag-type is selected as "none" on the handoff configuration. This is because ctag, stag values get picked from /etc/config/gatewayd instead of the handoff configuration on the Orchestrator when tag-type is "none".

    Workaround: Update the ctag, stag values to 0 each under vrf_vlan->tag_info in /etc/config/gatewayd. Do a vc_procmon restart.

  • Issue 67879: A Cloud Security Service (CSS) tunnel is deleted after a user changes a WAN Overlay setting from auto-detect to user-defined on a WAN interface setting.

    After saving the changes, the CSS tunnels do not come back up until the customer takes down and then puts back up the tunnel. Changing the WAN configuration will bring down the CSS tunnel and parse the CSS setup again. However, in some corner cases, the nvs_config->num_gre_links is 0 and the CSS tunnel fails to come up.

    Workaround: Deactivate and then reenable the CSS setup will bring the CSS tunnel up.

  • Issue 68057: DHCPv6 release packet is not sent from the VMware SD-WAN Edge on the changing of a WAN interface address mode from DHCP stateful to static IPv6 address and the lease remains active till reaching its valid time.

    The DHCPv6 client possesses a lease which it does not release when the configuration change is done. The lease remains valid till its lifetime expires in the DHCPv6 server and is deleted.

    Workaround: Without the fix, there is no way of remediating this issue as the lease would remain active till valid lifetime.

  • Issue 68851: If a VMware SD-WAN Edge and VMware SD-WAN Gateway each have the same TCP syslog server configured, the TCP connection is not established from the Edge to the syslog server.

    If the Edge and Gateway each have the same TCP server and if the syslog packets from the Edge are routed via the Gateway, the syslog server sends a TCP reset to the Edge.

    Workaround: Send the syslog packets direct from the Edge instead of routing via a Gateway or configure a different syslog server for the Edge and Gateway.

  • Issue 69284: For a site using a High-Availability topology where the Edges deploy VNF's in an HA configuration and are using Release 4.x, if these HA Edges are downgraded to a 3.4.x Release where HA VNF's are not supported, and then upgraded to 4.5.0, when the HA VNF's are reenabled, the Standby Edge VNF will not come up.

    The VNF state on the Standby Edge is communicated as down via SNMP. If the HA VNF pair is downgraded from a version supporting VNF-HA (release 4.0+) to a release which does not support it with VNF enabled on the Orchestrator. This issue will be seen when the Edge is upgraded back to a version supporting VNF-HA and it is enabled on the Orchestrator again.

    Workaround: VNF should first be deactivated in the case of an HA if the Edge is being downgraded to a version which does not support it.

  • Issue 69562: Traffic failure is observed on a VMware SD-WAN Gateway when both Partner Gateway-BGP and Non SD-WAN Destination-BGP have the same local Autonomous System Number (ASN).

    When both PG-BGP and NSD-BGP have the same local ASN and an NSD-BGP learned route is redistributed into a PG-BGP, the ASN will get prepended twice on the route before advertisement. This may cause some other BGP route to get preferred over this one due to a shorter AS path, possibly causing traffic matching the wrong route.

    Workaround: Without this fix, the workaround to this issues is to have different ASN's for PG-BGP and NSD-BGP.

  • Issue 71719: PPTP Connection is not Established along Edge to Cloud path.

    Connection to the PPTP server behind the VMware SD-WAN Edge does not get established.

    Workaround: There is no workaround for this issue, not even an Edge restart or reboot.

  • Issue 72358: If the IP address of a VMware SD-WAN Orchestrator DNS name changes, the VMware SD-WAN Gateway's management plane process fails to resolve it properly and the Gateway will be unable to connect to the Orchestrator.

    The Gateway's management process periodically checks the DNS resolution of the Orchestrator's DNS name, to see if it has changed recently so that the Gateway can connect to the right host. The DNS resolution code has an issue in it so that all of these resolution checks fail, and the Gateway will keep using the old address and thus no longer be able to connect to the Orchestrator.

    Workaround: Until this issue is resolved, an Operator User should not change the IP address of the Orchestrator. If the Orchestrator's IP address must be changed, all Gateways connecting to that Orchestrator will have to be reactivated.

  • Issue 75593: Customer deployments using BGP may experience issues with degraded performance because of suboptimal routing due to unexpected route preferences for uplink BGP routes.

    This issue is caused by a customer enterprise's BGP prefix advertise and preference values not being updated properly when the route prefix is updated from a non-uplink to an uplink or vice-versa, which results in asymmetric routing.

    Workaround: A customer without a fix for this issue can disable and enable BGP uplink community to restore correct routing.

  • Issue 76837: A customer using BGP may observe that a peer router is not sending traffic to a VMware SD-WAN Edge within its network.

    Troubleshooting the issue would reveal that the default route via default-originate is not being advertised by the Edge. The issue is caused by a route map string associated with the default route being truncated and so the Edge does not match the default route with anything in its route map, and this results in the peer router either dropping traffic or sending it using an invalid route where the traffic is blackholed.

    Workaround: A user would need to configure a static route on the peer router for the default route until it is possible to upgrade to an Edge version that includes the fix.

  • Issue 78050: A dataplane service failure occurs when the PPTP server is present on the LAN side.

    When PPTP server is present in the LAN side, and PPTP client from the Internet connects to it via inbound firewall rules, the Edge crashes due to PPTP control channel lookup failure. This control channel lookup is needed to ensure the GRE data channel is sent out via the same link back to PPTP client.

    Workaround: This issue is seen only if PPTP traffic is seen on the Edge. Do not use PPTP sessions.

  • Issue 81353: Packets might be dropped at Rx of interfaces due to low buffer.

    The Ring buffer setting was not a part of non-dpdk managed interfaces, which is being used by Azure platforms. NIC Rx ring buffer queues are set to low numbers.

    Workaround: No workaround is available for this issue. Rebooting may solve the issue temporary.

  • Issue 81859: When activating a VMware SD-WAN Edge 610-LTE, the CELL interface may not come up after the Edge completes its activation.

    This issue is not consistent but when it occurs it can have a major impact if the Edge 610-LTE's only public link is the mobile CELL link as the Edge would be effectively down and intervention for this Edge would need to be local in the form of someone power cycling the Edge to recover it.

    Workaround: If encountering this issue and the 610-LTE has other wired public WAN links, the user would need to either restart the Edge service through the Orchestrator using Remote Actions > Service Restart in a suitable maintenance window, or restart the Edge's modem to restore the CELL interface.

    If the 610-LTE only uses a CELL interface for internet, someone local to the Edge would have to power cycle the Edge as it would be inaccessible through the Orchestrator.

    If the 610-LTE Edge being activated only uses CELL for internet, the Edge should be activated with someone present to potentially power cycle it should it go down after completing activation.

  • Issue 82104: In rare cases, VMware SD-WAN Edges activated in a High Availability topology may be unable to communicate with a VMware SASE Orchestrator which will mark the site as down and preclude any intervention through the Orchestrator to the site.

    This issue occurs only when an unusual and invalid configuration is applied to the HA Edges. The configuration specifies that the HA port is configured as "trunk" (which should not be allowed), with zero VLANs (also should not be allowed), but where "all VLANs" are set. Instead of throwing an error at this configuration and preventing a user from activating HA for the Edges, the Orchestrator allows it and this configuration triggers a management plane failure on the HA Edges which no longer send a heartbeat to the Orchestrator and the Orchestrator marks the site as down.

    Workaround: Avoid using the configuration outlined above.

  • Issue 82415: A VMware SD-WAN Gateway deployed a KVM image with Intel® Ethernet Server Adapter X710; SR-IOV; and Bond0 does not respond if activated to Release 4.5.1 or 5.0.0.0.

    For a Gateway so deployed paths do not come up and the debug.py commands do not work.

    Workaround: Without this fix, Operators should avoid using these builds for this specific Gateway deployment until new builds with this issue fixed are rolled out.

  • Issue 83083: A VMware SD-WAN Gateway upgraded to Release 4.3.1 or later may experience a slow memory leak which can lead to the Gateway's service restarting to clear the memory.

    Gateway restarts can be disruptive to customer traffic for the 30-45 seconds it takes the for the Gateway service to restart. Each time an Operator user runs the debug.py --flow_dump all all all command on the Gateway, the Gateway will leak some of its memory. Running this debug command a sufficient number of times will cause the Gateway's memory usage to reach a critical level and trigger a Gateway service restart to clear the memory.

    Workaround: Avoid running the debug.py --flow_dump all all all command on the Gateway. If using this debug command is unavoidable, monitor the memory usage and schedule maintenance windows to preemptively restart the service to clear the memory prior to an unscheduled restart.

  • Issue 83212: When looking at the VMware SASE Orchestrator for Monitor > Edge > Transport, there is a discrepancy between the Link and Application statistics table.

    The Application and Link statistics should be match but the Application statistics show a higher value than the Link statistics. This issue is most commonly occurs where there is a VMware SD-WAN Edge Hub Cluster topology where the Spoke Edge uses a single WAN link. If this single WAN link experiences some loss, the packets are retransmitted and are accounted twice in Application statistics which results in the observed discrepancy.

    Workaround: There is no workaround for this issue but Applications statistics will be correct versus Link statistics when this issue is encountered.

  • Issue 84501: The NAS-IP address is set as a loopback IP address by default in RADIUS packets.

    The NAS-IP address is set as a loopback IP address by default in the RADIUS packets sent from the Edge (Authenticator) to the Radius Server. Set the NAS-IP address as the source interface IP address selected/configured with RADIUS Authentication settings. If Auto is selected as the source interface, the loopback IP will be set as NAS-IP by default.

    Workaround: Upgrade to the 5.0.1.0 release or later.

  • Issue 84790: When a VMware SD-WAN Edge with any model type other than 510/510-LTE is rebooted, the Edge may erroneously report the critical event Unable to launch service wifihang to the VMware SASE Orchestrator.

    The wifihang event message is designed for use only with the Edge 510/510-LTE models and alerts a customer to a problem with that Edge model's Wi-Fi process. When this event message is observed on any other Edge model, whether that model uses Wi-Fi or not (for example: the Edge 3400), the event message is spurious and the event can be safely ignored.

    Workaround: A user can safely ignore the wifihang event message on any Edge other than an Edge 510 or 510-LTE as it is spurious.

  • Issue 85154: When a VMware SD-WAN Virtual Edge on AWS with instance type C4.xlarge is upgraded from an older Edge release to Release 4.5.1, and is then downgraded back to an older Edge release, the Edge goes into a deactivated state where the Edge does not form management tunnels with the Gateway and Orchestrator.

    The cause of the issue is the Orchestrator erroneously deactivating the Edge because of what the Orchestrator detects as a serial number mismatch.

    Workaround: There is no workaround for this issue beyond NOT downgrading from Release 4.5.1 once the AWS Edge is on this release.

  • Issue 85459: An attempt to SSH either from an Edge LAN-side client to an Edge, or from a remote branch Edge client to an Edge may not work after LAN side NAT rules rules are configured.

    SSH reply packet packets coming from the Edge's SSH process go through the Edge's dataplane service and since LAN side NAT rules are configured, it is possible the SSH reply packets use LAN side NAT rules to go to different destination than the original client that generated the SSH traffic which causes an SSH attempt to an Edge to not work.

    Workaround: The only workaround is to remove the NAT rule.

  • Issue 87982: A VMware SD-WAN Edge using a Metanoia-type SFP module with a private PPPoE WAN link may be unable to establish BGP peering and connect to other sites.

    VLAN tagged packets using a private PPPoE link are corrupted by the Edge and never reach their destination as a result. This issue does not affect public PPPoE links.

    Workaround: The workarounds are either to not VLAN tag the traffic for PPPoE link or make the link public.

  • Issue 88604: For a site using a High Availability topology, if a WAN interface goes down and then comes back up on a VMware SD-WAN Standby Edge, the event is not recorded on the VMware SASE Orchestrator.

    A user does not have visibility on Standby Edge interface events, which is especially impactful on Enhanced HA deployments where the Standby Edge is also passing traffic.

    Workaround: There is no workaround for this issue.

  • Issue 88796: When deploying either a VMware SASE Orchestrator or a VMware SD-WAN Gateway and using an OVA on vSphere, the OVF properties set as part of the deployment (password, network information, etc.) are not applied to the image and the system cannot be accessed after deployment.

    This only affects new system deployed from OVA using OVF/vApp properties (versus using ISO files). This issue is caused by upstream changes to cloud-init in recent updates.

    Workaround: Without the fix, the workaround is for the Operator to deploy the system using a cloud-init user-data ISO file.

  • Issue 89235: Backhaul Traffic from a VMware SD-WAN Spoke Edge to the Internet may get dropped on the the VMware SD-WAN Edge used as the backhaul Hub.

    Due to a timing issue (post service restarts, power outage, or a config change) between the 'backhaul traffic from spoke' and 'the route advertised from the Spoke Edge,' the backhaul traffic will get dropped in the Hub Edge with a nsch_drop_stale_pi counter.

    Workaround: Running the Remote Diagnostic: Flush Flows on the Hub Edge remediates the issue.

  • Issue 89217: A VMware SD-WAN Edge in the 6x0 model line (610, 610N, 610-LTE, 620, 620N, 640, 640N, 680, 680N) may suddenly power off for no reason.

    The 6x0 Edge would have all lights off, both the front status LED and the rear Ethernet port lights, and can only be recovered by manually power cycling the Edge.

    The cause of the issue is traced to a PIC microcontroller exclusive to the Edge 6x0 line which uses a PIC firmware version of v20M or earlier (v20L, v20K, v20J). This issue can only occur when the 6x0 Edge uses a PIC version of v20M or earlier, but even with this version the odds of experiencing the power off issue are rare (approximately 1/1,000). The issue cannot occur on a 6x0 Edge with a PIC firmware version of v20N or later.

    Note:

    A 6x0 Edge's Firmware including PIC version can be determined on an Orchestrator using 5.x by going to the Monitor > Edge > Overview page for that Edge and clicking the dropdown information box next to the Edge name which includes the Edge Information, Device Version, and the Device Firmware. However this only works on an Edge using Release 4.5.1.

    The issue is resolved by upgrading the 6x0 Edge to Platform Firmware 1.3.1 (R131-20221216-GA), which includes PIC version v20N. To do this the 6x0 Edge must be connected to a VMware SASE Orchestrator using Release 5.x (5.0.0 or later), and the 6x0 Edge must first be upgraded to Edge hoftix build R5012-20230327-GA-107522. Once the 6x0 Edge is upgraded to R5012-20230327-GA-107522, the user would then update the 6x0 Edge Platform Firmware to version R131-20221216-GA in the same way that an Edge's software version is modified.

    For more information and a step-by-step guide to upgrading a 6x0 Edge to Platform Firmware 1.3.1, see the KB Article: VMware SD-WAN 6X0 model Edges may power off with no LEDs and require a power cycle to come back to a working state (88970). This KB article was updated on April 4th, 2023 to reflect the new Edge and Platform Software needed to resolve the issue.

    For information on uploading a Platform Firmware bundle to an Orchestrator, consult the Platform Firmware and Factory Images with New Orchestrator UI section of the VMware SD-WAN Operator Guide.

    For information on updating a 6x0 Edge’s Platform Firmware, consult the View or Modify Edge Information section of the VMware SD-WAN Administration Guide.

    Workaround: To recover the Edge from the problem state:​

    1. Disconnect the Edge from the power source.

    2. Wait 20 seconds.

    3. Reconnect the Edge to the power source.

    If you do not wish to upgrade the 6x0 Edge's platform firmware, the user can ensure the power to the Edge is consistent and does not flap rapidly or consistently. A good way to ensure a reliable power source is to connect the 6x0 Edge to a Uninterruptible Power Supply (UPS).

    If the user prefers to keep the Edge on a lower software release (for example, Release 4.3.1, or 4.5.1), the customer can temporarily upgrade the Edge to R5012-20230327-GA-107522, perform the Platform Firmware upgrade to version 1.3.1 (R131-20221216-GA) so that the PIC version is v20N, and then downgrade the Edge’s software back to their preferred version. Downgrading the 6x0 Edge's software to an earlier version does not also downgrade the Edge's Platform Firmware and the Edge would continue to use Platform Firmware version 1.3.1. In this use case the customer Edges would need to be on an Orchestrator using Release 5.x.

    If the 6x0 Edge is on an Orchestrator that does not use version 5.x and has experienced this issue and requires an update of its PIC firmware, the customer may reach out to VMware SD-WAN Support and they will manually update the Edge’s PIC version.

  • Issue 90884: For a customer enterprise using a Hub Cluster/Spoke topology, when a Hub Edge in the Cluster is reassigned to one or more Spoke Edges, the users at those Spoke Edge locations may experience traffic failure.

    Hub Edges in a Cluster can be reassigned as part of a Cluster rebalance when an enterprise upgrades their Edge software, so this issue may be observed post-upgrade. When this issue is encountered, the VMware SD-WAN Gateway does not send the new Spoke Edge routes to the Hub Edge because the Gateway is expecting all Hub Edges to have all Spoke Edge routes, and thus these routes are not in the Hub Edge's routing table. As a result traffic between the Spoke Edges and the Hub Edge in the Cluster is impacted because the forwarding path is down.

    Workaround: If the issue is encountered in an enterprise not using Gateways with a fix for this issue, it can be temporarily resolved by performing a route reinit on the Hub Edge. However, the issue may recur on a new Hub Edge reassignment.

  • Issue 91875: For a customer who has configured a WAN link as a Backup on a VMware SD-WAN Edge, they may observe the backup WAN link becoming active intermittently even though the conditions requiring the link to become active are not present.

    The issue is caused by a race condition on an Edge process that leads the Edge to erroneously think the backup WAN link is needed and proceeds to build a tunnel for that link which the Edge has no failsafe for detecting and tearing down this erroneous tunnel.

    Workaround: If you are using WAN links as backup links, it is recommended you upgrade the Edges where this is the case to hotfix build R451-20220714-GA-91875 which includes a specific fix for this issue.

  • Issue 92481: If a WAN interface on a VMware SD-WAN Edge is configured as not enabled on the VMware SASE Orchestrator, the interface will still be reported as 'UP' by SNMP.

    The key debug process for interfaces output does not include the physical port details for Edge WAN interfaces (for example, GE3 or GE4 on an Edge 6x0 or 3x00 model). As a result when SNMP polls those interfaces it always returns a result of UP regardless of how these interfaces are configured.

    Workaround: There is no workaround for this issue.

  • Issue 93055: On a customer site deployed with a High Availability topology, a VMware SD-WAN Edge may experience a Dataplane Service failure and restart as a result, triggering an HA failover.

    An HA failover on an Enhanced HA topology would cause disruption for traffic using a WAN link on the Active Edge. This issue is limited to builds prior to Release 5.x.

    When this issue is encountered the logs would show something similar to the following:

    ERROR [edged 7205(7212)] Process edged (pid 3022) exited on signal SIGABRT: restarting after 3.0 seconds INFO [Thread-3 7205(15804)] edged: posted ERROR event EDGE_SERVICE_FAILED

    Workaround: If the Edge has experienced this issue, the customer can upgrade to an Edge build with a hot fix for this issue. The hot fix build is R451-20221222-GA-93055 and can be uploaded to a Partner or Customer by contacting VMware SD-WAN Support.

  • Issue 94430: For a customer enterprise that uses a Hub/Spoke topology where multiple Hubs are deployed, a user behind a VMware SD-WAN Spoke Edge may observe issues with traffic that is destined for a Hub Edge.

    Client traffic issues occur when the Spoke Edge forwards traffic towards a Hub different than the one expected to receive the traffic. The issue is caused by the AS path length for the remote BGP routes not being calculated properly in certain scenarios. Because of this, the routes from the Hubs that should have a lower routing preference instead end up having a greater AS_PATH length and may be preferred.

    Workaround: If encountering this issue without a fix, the customer can withdraw and readvertise the route that is expected to be preferred.

  • Issue 94980: For a site deployed with a High Availability topology, the VMware SD-WAN Standby Edge may experience a Dataplane Service error and restart after a PPPoE WAN link is configured for the HA Edges.

    When examining the core generated by the Standby Edge, a user would see the message vc_is_use_cloud_gateway_set after the PPPoE link is configured.

    Workaround: There is no workaround for this issue beyond configuring PPPoE links in a maintenance window to manage the risk of this action.

  • Issue 95399: The issue is seen only when the customer monitors the events in the Orchestrator. Previously INTERFACE UP/DOWN events would get triggered when the interface goes down or up. Currently, it's not getting reported. The customer won't be able to see the EDGE_INTERFACE_UP or EDGE_INTERFACE_DOWN event.

    The issue is due to the dhclient addition in the 4.5.1 release. dhclient was not configured to send these events to the VMware SD-WAN Orchestrator.

    Workaround: The link alive /link dead events for monitoring can also be used. However, the exact EDGE INTERFACE UP and EDGE INTERFACE DOWN will not be available for monitoring.

  • Issue 97073: For a customer enterprise site using an Enhanced High Availability topology, after an HA failover the users may observe that one of the WAN links goes down.

    Due to Enhanced HA link cleanup not being done properly, even if the link is up on the Active Edge, SD-WAN sends packets meant for that link to the Standby Edge. This causes link connectivity to go down.

    Workaround: On an Enhanced HA site not using a build with a fix for this issue, the link state can be corrected by a reboot of the Active Edge which triggers an HA failover. However, depending on the sequence of events happening when Enhanced HA is setup for an interface, this may not work with the first attempt. If the initial reboot of the Active Edge which forces an HA failover does not work, initiate additional reboots until the link is recovered.

    This issue is fixed on all Release 4.5.2 Edge builds.

  • Issue 97272: On a site deployed with a High Availability topology and which uses OSPF, when the HA Edges get into an Active-Active "Split Brain" condition, the users may observe that all traffic drops.

    All traffic drops due to the Standby Edge setting the link-state advertisement (LSA) age to the maximum value (3600) in the core-router, which causes the removal of the default route. When operating properly, the core-router has the LSA age synchronized with the Active Edge, but in an Active-Active state is encountered, the Standby Edge is also Active and it sends the LSA to the core-router. Both Active and Standby Edge have the same router ID but are sending different LSA ages to the core router. On seeing the mismatch, the core router sets the LSA age to the maximum value of 3600 and leads to the OSPF default route being removed.

    Workaround: If a customer is experiencing this issue on an HA site, they can upgrade the Edge to a special hotfix build R451-20221123-GA-97272 which includes the fix for this issue. Contact VMware SD-WAN Support to have this build uploaded to your enterprise or partner portal. On an Edge without a fix for this issue the would need to restart OSPF on the Active Edge once the Active-Active state is resolved.

  • Issue 97759: For a VMware SD-WAN Gateway where Non SD-WAN Destinations via Gateway are connected, if the Gateway receives a packet from the NSD with a time to live (TTL) value of 1, the Gateway can experience a Dataplane Service failure and restart to recover.

    This issue can be experienced multiple times on the Gateway if the NSD is repeatedly sending packets with TTL=1 values. The issue is caused by the Gateway treating a TTL with value 1 as having value 0 which results in a double free memory condition and the resulting service failure.

    The issue is caused by the Gateway treating a TTL with value 1 as having value 0 which results in a double free memory condition and the resulting service failure.

    Workaround: There is no workaround beyond upgrading the Gateway to Release 4.5.2, which includes a fix for this issue.

  • Issue 97997: When Edge Network Intelligence Analytics is activated on a VMware SD-WAN Edge, the Edge may trigger an Edge Service restart, which causes 10-15 seconds of customer traffic disruption.

    When Analytics is enabled on the Edge, the Edge can experience an out of memory condition followed by a "double free" memory state. The Edge restarts its service to restore memory. The symptoms for this issue can happen multiple times while Analytics are activated.

    Workaround: There is no workaround for this issue beyond deactivating Analytics.

  • Issue 98136: For customer enterprises using a Hub/Spoke topology where Dynamic Branch To Branch VPN is configured, client users behind a SD-WAN Spoke Edge may observe that some traffic has unexpected latency resulting from the traffic using a sub-optimal path.

    Spoke Edge traffic that experiences this issue uses a route that was initially a non-uplink route for a Hub Edge not included in the Profile the Spoke Edge was using. A Dynamic Branch To Branch VPN tunnel can be formed from the Spoke Edge to the Hub Edge because of traffic being sent towards some other unrelated prefix and in this instance the non-uplink route is installed in the Spoke Edge.

    As a result of this non-uplink route, all traffic towards this prefix starts going through the Hub Edge and the non-uplink route becomes uplink (community change to uplink community) but the non-uplink route installed previously is not revoked and the traffic takes the Hub Edge path as long as the Dynamic Branch To Branch VPN tunnel remains up.

    Workaround: Wait for the Dynamic Branch To Branch VPN tunnel to tear down, after which the uplink route will not be installed in the Spoke Edge when a new Dynamic Branch To Branch VPN tunnel is formed towards the Hub Edge.

  • Issue 98979: A VMware SD-WAN Edge may experience a reboot due to an out of memory condition.

    Depending on how much memory is allocated by the Edge during runtime when combined with a subsequent core generation may trigger a kernel out of memory condition which in turn triggers a kernel panic and resulting reboot. An Edge reboot takes about 2-3 minutes to complete and customer traffic would be disrupted until the Edge completes the reboot.

    Workaround: There is no workaround for this issue.

  • Issue 100005: VMware SD-WAN Edge model 610 or 610-LTE returns incorrect value for ifSpeed - OID 1.3.6.1.2.1.2.2.1.5.

    When queried for interface speed, an Edge model 610 returns an incorrect value because the DPDK AF_PACKET driver has a hardcoded speed of 10G.

    Workaround: There is no workaround for this issue.

  • Issue 103662: For a customer site configured with any High Availability topology, the user may observe that the VMware SD-WAN Edge in the Standby role has rebooted multiple times.

     The reboots are a result of the HA site experiencing an Active/Active (Split-Brain) state that SD-WAN corrects by rebooting the Standby Edge. These Active/Active states are the result of heartbeats from the Active Edge being delayed by other Edge processes that lead the Standby Edge to think the Active Edge has gone offline and is thus promoted to active. As with any Active/Active situation, this issue is potentially disruptive to a site deployed with Enhanced HA since the Standby is also passing customer traffic through its WAN links.

    Workaround: The workaround is to increase the HA failover time from the default of 700ms to 7000ms.

  • Issue 112131: For a customer site deployed with a High Availability topology where the customer has an Edge Network Intelligence license, activating Analytics for the HA site may result in the site experiencing an Active/Active (Split-Brain) state and the Standby Edge rebooting to resolve the state.

    Once Analytics is activated for an HA site, the Edge Network Intelligence service may starve the Edge's dataplane threads, which delays HA packet processing. This results in both Edges declaring themselves as an Active Edge and SD-WAN has to reboot the Standby Edge to recover the situation. This issue can happen at any time, not just at the initial activation of Analytics. As with any Active/Active situation, this issue is potentially disruptive to a site deployed with Enhanced HA since the Standby is also passing customer traffic through its WAN links.

    Workaround: Avoid the use of Analytics on HA sites.

  • Issue 113291: When a user runs the Remote Diagnostics "Troubleshoot BGP - Show BGP Neighbor details", "Troubleshoot BGP - Show BGP Neighbor Received Routes", or "Troubleshoot BGP - Show BGP Neighbor Learned Routes" the VMware SASE Orchestrator may return an error with no results.

    The error reads No such neighbor in this view/vrf. This occurs if there is a stale BGP entry and is caused by a logical issue where the diagnostic fetches the vrf name where an Edge has a stale entry, the command returns two entries with the first being the stale one.

    Workaround: The stale BGP entry needs to be deleted to ensure a successful Remote Diagnostic.

  • Issue 118704: A user may observe abnormally high latency values for paths measured between SD-WAN Edges and SD-WAN Gateways even though actual Edge-to-Gateway packet latency is much lower.

    A race condition has been identified with clock synchronization resulting in latency values measured incorrectly. This issue is cosmetic and there is no performance impact to customer traffic but it does negatively impact a customer's ability to properly monitor Edge links and paths.

    Workaround: Clock synchronization can be reset via the Edge's CLI, please contact SASE Support if assistance is required to apply the workaround for this condition.

  • Issue 142366: For a customer enterprise site connected to Partner Gateways where one or more static routes are configured, client users working behind an SD-WAN Edge may observe intermittent traffic loss if a static route via the Primary Partner Gateway is unreachable.

    When the same static route is reachable via two or more Partner Gateways, if the route via the Partner Gateway in the Primary role is unreachable, traffic from an Edge can experience intermittent traffic loss. This issue is the result of the Edge API failing to properly check for reachability on a route lookup which causes the Edge to continue to use the Primary Partner Gateway even though reachability is false.

    Workaround: The issue can be temporarily remediated by the Partner shutting down the Primary Partner Gateway until the static route becomes reachable again. Shutting down the Primary Partner Gateway prevents the Edge from including it in route reachability lookups and ensures traffic matching that static route uses a secondary Partner Gateway. However this can be disruptive for all customers using this Partner Gateway as their Primary Gateway and should be done in a maintenance window by the Partner if possible.

Orchestrator Known Issues

  • Issue 21342:

    When assigning Partner Gateways per-segment, the proper list of Gateway Assignments may not show under the Operator option "View" Gateways on the VMware SD-WAN Edge monitoring list.

  • Issue 24269:

    Monitor > Transport > Loss not graphing observed WAN link loss while QoE graphs do reflect this loss.

  • Issue 25932:

    The VMware SD-WAN Orchestrator allows VMware SD-WAN Gateways to be removed from the Gateway Pool even when they are in use.

  • Issue 32335:

    The ‘End User Service Agreement’ (EUSA) page throws an error when a user is trying to accept the agreement.

    Workaround: Ensure no leading or trailing spaces are found in Enterprise Name.

  • Issue 32435:

    A VMware SD-WAN Edge override for a policy-based NAT configuration is permitted for tuples which are already configured at the profile level and vice versa.

  • Issue 32913:

    After Enabling High Availability, Multicast details for the VMware SD-WAN Edge are not displayed on the Monitoring Page. A failover resolves the issue.

  • Issue 33026:

    The ‘End User Service Agreement’ (EUSA) page does not reload properly after deleting the agreement.

  • Issue 35658:

    When a VMware SD-WAN Edge is moved from one profile to another which has a different CSS setting (e.g. IPsec in profile1 to GRE in profile2), the Edge level CSS settings will continue to use the previous CSS settings (e.g. IPsec versus GRE).

    Workaround: Deactivate and then reenable GRE at the Edge level to resolve the issue.

  • Issue 35667:

    When a VMware SD-WAN Edge is moved from one profile to another profile which has the same CSS setting but a different GRE CSS name (the same endpoints), some GRE tunnels will not show in monitoring.

    Workaround: Deactivate and then reenable GRE at the Edge level to resolve the issue.

  • Issue 36665:

    If the VMware SD-WAN Orchestrator cannot reach the internet, user interface pages that require accessing the Google Maps API may fail to load entirely.

  • Issue 38056:

    The Edge-Licensing export.csv file not show region data.

  • Issue 38843:

    When pushing an application map, there is no Operator event, and the Edge event is of limited utility.

  • Issue 39633:

    The Super Gateway hyper link does not work after a user assigns the Alternate Gateway as the Super Gateway.

  • Issue 39790:

    The VMware SD-WAN Orchestrator allows a user to configure a VMware SD-WAN Edge’s routed interface to have greater than the supported 32 subinterfaces, creating the risk that a user can configure 33 or more subinterfaces on an interface which would cause a Dataplane Service Failure for the Edge.

  • Issue 40341:

    Though the Skype application is properly categorized on the backend as Real Time traffic, when editing the Skype Business Policy on the VMware SD-WAN Orchestrator, the Service Class may erroneously display “Transactional”.

  • Issue 41691:

    User cannot change the 'Number of addresses' field although the DHCP pool is not exhausted on the Configure > Edge > Device page.

  • Issue 43276:

    User cannot change the Segment type when a VMware SD-WAN Edge or Profile has a partner gateway configured.

  • Issue 47269:

    The VMware SD-WAN 510-LTE interface may appear for Edge models that do not support an LTE interface.

  • Issue 47713:

     If a Business Policy Rule is configured while Cloud VPN is not enabled, the NAT configuration must be reconfigured upon enabling Cloud VPN.

  • Issue 47820:

    If a VLAN is configured with DHCP not enabled at the Profile level, while also having an Edge Override for this VLAN on that Edge with DHCP enabled, and there is an entry for the DNS server field set to none (no IP configured), the user will be unable to make any changed on the Configure > Edge > Device page and will get an error message of ‘invalid IP address []’ that does not explain or point to the actual problem.

  • Issue 48085:

    The VMware SD-WAN Orchestrator allows a user to delete a VLAN which is associated with an interface.

  • Issue 48737:

    On a VMware SD-WAN Orchestrator which is using the Release 4.0.0 new user interface, If a user is on a Monitor page and changes the Start & End time interval and then navigates between tabs, the Orchestrator does not update Start & End interval time to the new values.

  • Issue 49225:

    VMware SD-WAN Orchestrator does not enforce a limit of 32 total VLANs.

  • Issue 49790:

    When a VMware SD-WAN Edge is activated to Release 4.0.0, the activation is posted twice in Events.

    Workaround: Ignore the duplicate event.

  • Issue 50531:

    When two Operators of differing privileges use the same browser window when accessing the New UI on a 4.0.0 Release version of the VMware SD-WAN Orchestrator, and the Operator with lesser privileges tries to login after the Operator with higher privileges, that lesser privileged Operator will observe multiple errors stating that the "user does not have privilege".

    Note:

    There is no escalation in privileges for the Operator with lower privileges, only the display of error messages.

    Workaround: The next operator may refresh that page prior to logging in to prevent seeing the errors, or each Operator may use different browser windows to avoid this display issue.

  • Issue 51722: On the Release 4.0.0 VMware SD-WAN Orchestrator, the time range selector is no greater than two weeks for any statistic in the Monitor > Edge tabs.

    The time range selector does not show options greater than "Past 2 Weeks" in Monitor > Edge tabs even if the retention period for a set of statistics is much longer than 2 weeks. For example, flow and link statistics are retained for 365 days by default (which is configurable), while path statistics are retained only for 2 weeks by default (also configurable). This issue is making all monitor tabs conform to the lowest retained type of statistic versus allowing a user to select a time period that is consistent with the retention period for that statistic.

    Workaround: A user may use the "Custom" option in the time range selector to see data for more than 2 weeks.

  • Issue 60039: RMA Reactivation does not work when the VMware SD-WAN Edge model is changed.

    When performing an RMA Reactivation for a site where the Edge model is also being changed, the VMware SD-WAN Orchestrator does not save the model change making the reactivation link ineffective. This only affects RMA Reactivations where the Edge model is changed, an RMA Reactivation where the Edge model remains the same will work as expected.

    Workaround: If using a different Edge model for a site, the user would need to create a new Edge and manually apply all Edge-specific settings.

  • Issue 60522: On the VMware SASE Orchestrator UI, the user observes a large number of error messages when they try to remove a segment.

    The issue can be observed when adding a segment to a profile and the associating the segment with multiple VMware SD-WAN Edges. When the user attempts to remove the added segment from the profile, they will see a large number of error messages.

    Workaround: There is no workaround for this issue.

  • Issue 88910: On a VMware SASE Orchestrator running Release 4.5.1, a user is unable to run a WAN Link speed test for a VMware SD-WAN Edge using Remote Diagnostics > WAN Link Speed Test.

    When trying to use the WAN Link Speed Test diagnostic the user will not be able to choose an interface for the speed test as there is no drop down menu for the interfaces present.

    Workaround: If a user wants to force a bandwidth test for a WAN link, as a workaround the user can change the bandwidth measurement method for that WAN link, which triggers a retest automatically. This can be done as follows:

    1. On the Orchestrator UI for a particular Edge, go to Configure > Device and scroll down to WAN Settings.

    2. For the WAN link to be retested, select Edit for the link to be remeasured and then select Advanced to reveal additional settings.

    3. Under the Bandwidth Measurement menu change the Bandwidth Measurement method from the current to a different method (for example if the link uses Burst Mode, change to Slow Start, or vice versa) and click Update Link and then Save Changes at the top of the Device page.

    4. Once a measurement has been performed on the WAN link, change the measurement method back to the original method to ensure the most accurate measurement for the respective WAN link. Doing this triggers an additional bandwidth test.

check-circle-line exclamation-circle-line close-line
Scroll to top icon