Updated September 24th, 2021

VMware SD-WAN™ Orchestrator Version R430-20210810-GA
VMware SD-WAN™ Gateway Version R430-20210916-GA-VCG
VMware SD-WAN™ Edge Version R430-20210702-GA-61583

Check regularly for additions and updates to these release notes.

What is in the Release Notes

The release notes cover the following topics:

Recommended Use

This release is recommended for all customers who require the features and functionality first made available in Release 4.3.0.

Compatibility

Release 4.3.0 Orchestrators, Gateways, and Hub Edges support all previous VMware SD-WAN Edge versions greater than or equal to Release 3.2.0 
Note: this means releases prior to 3.2.0 are not supported.

The following interoperability combinations were explicitly tested:

Orchestrator

Gateway

Edge

Hub

Branch/Spoke

4.3.0

4.2.0

4.2.0

4.2.0

4.3.0

4.3.0

4.2.0

4.2.0

4.3.0

4.3.0

4.3.0

4.2.0

4.3.0

4.3.0

4.2.0

4.3.0

4.3.0

4.0.1

4.0.1

4.0.1

4.3.0

4.3.0

4.0.1

4.0.1

4.3.0

4.3.0

4.3.0

4.0.1

4.3.0

4.3.0

4.0.1

4.3.0

4.3.0

3.4.2

3.4.4

3.4.4

4.3.0

4.3.0

3.4.4

3.4.4

4.3.0

4.3.0

4.3.0

3.4.4

4.3.0

4.3.0

3.4.4

4.3.0

4.3.0

3.4.2

3.4.2

3.4.2

4.3.0

4.3.0

3.4.2

3.4.2

4.3.0

4.3.0

4.3.0

3.4.2

4.3.0

4.3.0

3.4.2

4.3.0

4.3.0

3.3.2 P2

3.3.2 P2

3.3.2 P2

4.3.0

4.3.0

3.3.2 P2

3.3.2P2

4.3.0

4.3.0

4.3.0

3.3.2 P2

4.3.0

4.3.0

3.3.2 P2

4.3.0

4.3.0

3.2.2

3.2.2

3.2.2

4.3.0

4.3.0

3.2.2

3.2.2

4.3.0

4.3.0

4.3.0

3.2.2

4.3.0

4.3.0

3.2.2

4.3.0

Note: Release 3.x did not properly support AES-256-GCM, which meant that customers using AES-256 were always using their Edges with GCM disabled (AES-256-CBC). If a customer is using AES-256, they must explicitly disable GCM from the Orchestrator prior to upgrading their Edges to a 4.x Release. Once all their Edges are running a 4.x release, the customer may choose between AES-256-GCM and AES-256-CBC.

Important Notes

BGPv4 Filter Configuration Delimiter Change for AS-PATH Prepending

Through Release 3.x, the VMware SD-WAN BGPv4 filter configuration for AS-PATH prepending supported both comma and space based delimiters. However, beginning in Release 4.0.0 and forward, VMware SD-WAN will only support a space based delimiter in an AS-Path prepending configuration.
Customers upgrading from 3.x to 4.x need to edit their AS-PATH prepending configurations to "replace commas with spaces" prior to upgrade to avoid incorrect BGP best route selection.

Zscaler Tunnels Now Use IKEv2

Once an Orchestrator and Gateways are upgraded to Release 4.3.0 or above, all Non SD-WAN Destinations via Gateway which use a Zscaler type will have their tunnels change to IKEv2 and no longer use IKEv1.

Extended Upgrade Time for Edge 3x00 Models

Upgrades to this version may take longer than normal (3-5 minutes) on Edge 3x00 models (i.e., 3400, 3800 and 3810). This is due to a firmware upgrade which resolves issue 53676. If an Edge 3400 or 3800 had previously upgraded its firmware when on Release 3.4.5, 4.0.2, or 4.2.1, then the Edge would upgrade as expected. For more information, please consult Fixed Issue 53676.

New Features

Azure Virtual WAN Automation from Edge

A customer can enable a Non SD-WAN Destination (NSD) via Edge to an Azure vWAN Hub with IPsec connectivity.

Note: For Release 4.3.0, only static routes are supported when automating connectivity from an Edge to an Azure vWAN.

Azure Private MEC (Multi-Access Edge Compute)

Support of SD-WAN Edge on Azure Private MEC will allow customers to enable low latency access to computing and storage services deployed on-premises and deploy CNFs, VNFs, and 3rd party applications.

BGP over IPsec on Edge and Gateway

BGP over IPsec allows users to enable BGP on Non SD-WAN Destination tunnels, augmenting the static routing offered today with the ability to learn and advertise routes dynamically with third party IPsec endpoints.

IPv6 on the WAN

VMware SD-WAN now supports dual-stack (IPv4/IPv6) or IPv6 only on the WAN transport side. IPv6 on the WAN addresses use cases such as transport IPv4 user traffic over IPv6 WAN, IPv4 over Mix IPv4/IPv6 Transport and IPv4 over Dual Stack Transport.

Multi-Segment Loopback Interface

This feature introduces loopback interfaces, which are logical interfaces that are always up and reachable. A loopback interface IP address can be used as the source IP address for assorted services such as Orchestrator Management Traffic, Authentication, DNS, NetFlow, Syslog, TACACS, BGP, and NTP. As loopback interfaces are always up and reachable, these services can receive the reply packets, if at least one physical interface configured for the Edge has layer 3 reachability. Loopback interfaces can also be used as the source interface for BGP as this ensures that when the BGP's interface state flaps, the BGP membership does not flap if there is at least one layer 3 connection available.

Bastion Orchestrator

For customers with strict security requirements (e.g., Financial and Government sectors), their security policies dictate that a production Orchestrator cannot be exposed to the internet. The VMware SD-WAN Bastion Orchestrator is a "sacrificial" Orchestrator where, if compromised, would not result in the loss of sensitive customer information or create a vulnerability to the SD-WAN system. The Bastian Orchestrator allows a customer to ”pre-stage” device configurations prior to enrolling the device into product services.

External Certificate Authority

Large enterprises and government customers planning to deploy an on-premise Orchestrator have stipulated that VMware SD-WAN must use the customer's own external certificate authority (CA) rather than the default self-signed Orchestrator certificate authority. The External CA feature allows the Orchestrator to offload certificate issuing for SD-WAN Edges and Gateways to a customer’s own external certificate authority.

FIPS Mode support for the Orchestrator and Gateway

A customer can now bring up both an Orchestrator and Gateways with Cloud-Init into Federal Information Processing Standard (FIPS) mode.  FIPS mode disables non-FIPS approved cipher suites. MD5 is disabled and SHA-1 is used. FIPS Mode is for use with new installations only, no existing Orchestrator or Gateway can be upgraded to this feature.

Image Signing and Verification

Customers want to ensure the integrity and authenticity of the software and firmware running on VMware SD-WAN devices. Image signing assures our customers that the software they install is authentic, untampered, and traceable. In addition, during the software upgrade process, the device verifies that all software upgrade images have been signed by VMware based on a root of trust securely stored on the device.

VMware SD-WAN in Virtual WAN

With this feature, a customer can deploy a VMware SD-WAN Edge using a managed application in an Azure Virtual WAN Hub. This enables customers to extend their VMware SD-WAN fabric from their Edges at branches to directly inside Azure Virtual WAN hubs and pair Azure Virtual WAN’s customizable routing intelligence with VMware SD-WAN’s last mile optimization using Dynamic Multi-Path Optimization (DMPO) to seamlessly access workloads across Azure regions while experiencing enhanced end user experience.

Support for New Hardware Platforms

Edge 510N, Edge 610N, Edge 620N, Edge 640N, and Edge 680N

VMware plans to introduce several new SD-WAN Edge hardware models that do not include integrated Wi-Fi. These include the Edge 510N, 610N, 620N, 640N, and 680N. These Edge models will be supported in this release beginning with Orchestrator build R430-20210615-GA. Any Wi-Fi configurations made in the VMware SD-WAN/SASE Orchestrator’s settings will not impact these Edge models.

Release limitations: In Release 4.3.0, the VMware SD-WAN Orchestrator will continue to display the WLAN interfaces for the non-Wi-Fi Edge devices even though Wi-Fi is not supported. 

Feature Enhancements

128 Segments

VMware SD-WAN now supports up to 128 segments to accommodate the needs of large enterprises and service providers.

ADSL2/2+ and VDSL2 Support for All Edge 6x0 Models

With Release 3.4.0, ADSL and VDSL SFP support was added for the Edge 610 and 610-LTE.  In Release 4.3.0 this support is also added to the remainder of the Edge 6x0 model line to include the Edge 620, 640, and 680. As with the Edge 610/610-LTE, the supported SFP module is the Metanoia SFP-V5311-T-R xDSL SFP adapter which operates according to VDSL2 and ADSL2/2+ specifications.

In addition, DSL parameters are now included on the VMware SD-WAN Orchestrator as part of the activation process which allows an Edge 6x0 to be activated using only a DSL link.  ADSL2/2+ support is enhanced by adding additional parameters for the PVC setting when configuring the SFP interface.

Azure Accelerated Networking Support

VMware SD-WAN vVCE adds support for Azure (Hyper-V) based SR-IOV.

Class of Service for Public Wireless Links

A customer can now configure class of service mapping under public wireless links.

Firewall Enhancements

Firewall functionality is now improved in two ways:

  • Updates to syslog message format for Firewall Logs.
  • Ability to configure audit comments in a Firewall Rule.

High Availability

Customers can now use Virtual Edges in uCPE deployments with High Availability on ESXi. This is the result of two enhancements:

  • Support for a unique MAC address on a High Availability interface. Instead of generating a common or shared virtual MAC when in HA, this feature uses the physical MAC address for hardware Edges and the assigned MAC address for virtual Edges.
  • Support for detecting Loss of Signal (LoS) on an HA Interface, which ensures there will be HA failover even if the ESXi deployment is using a virtual switch.

Orchestrator Localization

The VMware SD-WAN Orchestrator UI now supports localization in Spanish, Japanese, Korean, Traditional Chinese, and Portuguese. Please use the browser language setting to select the Orchestrator language.

Partner Role Enhancements

This partner administrator role is enhanced to do the following:

  • Generate and download Partner Gateway diagnostic bundles.
  • View Partner Gateway events and set up authentication mode.
  • Configure customer settings, and view and modify the customer's account.
  • Export a consolidated customer inventory.

Role Customization

Enhancements to the customer administrator roles and added customizations. Some of the use cases for these new customizations include:

  • Security Administrator: A Standard Administrator customized to only be able to modify the configurations under the Firewall tab.
  • Network Administrator:  A Standard Administrator customized to only be able to modify configurations found under the Device and Business Policy tabs.  
  • Enterprise Read-Only: In this use case, due to security/privacy reasons, customers would like to have their read-only users only able to view the rollup of OS sources (IOS,  Linux, Windows etc.) instead of showing individual source IP addresses.
  • For the Telco/Customer Co-Managed model, there is now the ability to customize a role that can modify LAN settings but not WAN settings.

Security Enhancements for Password Policy

Password strength can be configured for the following:

  • Minimum and maximum password character lengths are configurable. The default minimum password length is 8 characters, and the default maximum password length is 32 characters.
  • The password must contain numeric, lowercase, uppercase and/or special characters. Numeric and lowercase requirement are enabled by default.
  • Password must not match a list of the most commonly used passwords.
  • Password must include no repetitive or sequential characters (for example, “aaaaaa’, “1234abcd”). This is disabled by default.
  • Password must not contain the user's ID in whole or part. This is disabled by default.
  • New password must vary from the old password by a configurable number of characters. This is disabled by default.
  • Password expiration can be configured. This is disabled by default. When enabled, the default expiration is 30 days.
  • Old passwords cannot be used. The number of old passwords that can be retained is configurable. This is disabled by default. When enabled, the default is that the new password cannot match the last 5 passwords.

Note: These settings are configured on the SD-WAN Orchestrator by an Operator user under System Properties and can be set for the Operators on that Orchestrator and globally for all customers on that Orchestrator.

VMware Horizon

VMware SD-WAN service now includes complete coverage of the VMware Horizon application in the default application map, including the Horizon Blast protocols. This ensures that all business policies will match VMware Horizon traffic and flow diagnostics will correctly identify VMware Horizon traffic.

Zero Touch Provisioning (ZTP) - Push Activation

Previously, ZTP required deployment personnel to be physically local to the Edge while using an activation link pulled from the Orchestrator in the form of an email to activate an Edge, a method that scales poorly on large deployments. With the addition of Push Activation to our ZTP feature, customers may activate an Edge just by connecting it to power and the internet.

Zscaler Integration

As part of VMware's ongoing Zscaler partnership to improve the customer experience, Release 4.3.0 includes Zscaler Single Sign-On (SSO) integration and support for Layer 7 health checks.

Orchestrator API Changes Since Release 4.2.x

As of the 4.3.0 Orchestrator software release, VMware supports the following TLS cipher suites across all public VMware SD-WAN Orchestrator APIs:

  • TLS_AES_256_GCM_SHA384

  • TLS_AES_128_GCM_SHA256

  • ECDHE-RSA-AES256-GCM-SHA384

  • ECDHE-RSA-AES128-GCM-SHA256

Changes to the Orchestrator Portal API ("API v1")

The complete 4.3.0 VCO Portal API Reference is available via code.vmware.com.

In previous Release Notes, the API section was used to enumerate an extensive list of changes to the Orchestrator Portal API. With the number of interrelease changes that impact the Portal API increasing, and with the ongoing introduction of entirely new public APIs (such as SD-WAN API v2), we have decided to revisit this approach in the interest of keeping this section more concise.

Beginning with Release 4.3.0 and moving forward, the Release Notes will highlight only a subset of changes that our Engineering team has identified as posing some risk of disruption to existing API clients. Meanwhile, the API team will continue to provide a comprehensive change log for the Portal API as a downloadable artifact alongside the API Reference on code.vmware.com. We expect to adopt a similar practice for other APIs.

Maintainers of existing API clients may observe changes in API behavior stemming from the following software changes:

  • Issue 53798: Enforcement of more rigorous request syntax validation on API calls to enterprise/insertOrUpdateEnterpriseGatewayHandoff. This validation is based on precisely the same request schema that appears in our API Swagger documentation.
  • Issue 58407: Addition of incremental request validation logic which enforces a minimum value of 0 for the "limit" request parameter for all API methods that produce paginated result sets (e.g. event/getOperatorEvents, monitoring/getAggregateEnterpriseEvents, monitoring/getEnterpriseEdgeStatus, etc.).
  • Modification of privileges required to invoke "CRUD" (create, read, update, and delete) API methods pertaining to Object Groups. Whereas these methods previously required permissions to manage NETWORK_SERVICE entities, they now require permissions to manage OBJECT_GROUP resources. Users assigned standard Orchestrator roles should not observe any change in API behavior as a result of this change. 

Document Revision History

June 3rd, 2021. First Edition

June 7th, 2021. Second Edition

  • Added a new Gateway build: R430-20210605-GA as the most recent Gateway GA build.  This build adds a fix for Issue 64633, which is also added to this edition.

June 15th, 2021. Third Edition.

  • Added an Important Note regarding Non SD-WAN Destinations with a Zscaler type would have their tunnels change from IKEv1 to IKEv2 after both the Orchestrator and Gateways were upgraded to Release 4.3.0.

June 16th, 2021. Fourth Edition.

  • Changed the New Feature "Azure Private Edge Zones" to "Azure Private MEC (Multi-Access Edge Compute)" as a result of a Microsoft renaming of the product, effective June 16th, 2021.

June 30th, 2021. Fifth Edition.

  • Added a new section: Support for New Hardware Platforms. This section adds the following text:
    "VMware intends to introduce new SD-WAN Edge hardware models that do not include integrated Wi-Fi. These include the Edge 510N, Edge 610N, Edge 620N, Edge 640N, and Edge 680N. These Edge models will be supported in this release beginning with Orchestrator build R430-20210615-GA. Any Wi-Fi configurations made in the VMware SD-WAN/SASE Orchestrator’s settings will not impact these Edge models."
  • Added a new Orchestrator build, R430-20210615-GA to the Resolved Issues section.  This is also the new default build at the top of the Notes.
  • Added two new resolved tickets fixed by this new build: 62355 and 64716

July 2nd, 2021, Sixth Edition

  • Added a new Edge GA build R430-20210702-GA-61583 to the Resolved section. This Edge build addresses a new resolved issue #61583.

July 8th, 2021, Seventh Edition

  • Added a new Open Issue to Edge/Gateway Known Issues section: Issue #65885.

July 13th, 2021, Eighth Edition

  • Added a new Orchestrator GA build R430-20210709-GA to the Orchestrator Resolved Section. 
  • This Orchestrator build addresses the following fixed issues which are included in the R430-20210709-GA section: #59434, #65526, #65967, #66678, and #66679.

July 28th, 2021, Ninth Edition

  • Added a new Gateway GA build R430-20210727-GA-65293-67191 to the Edge/Gateway Resolved section.
  • This Gateway build adds the fixed issue #65293 and #67191, which are documented in this section.
  • Added a new Orchestrator GA build R430-20210719-GA to the Orchestrator Resolved section.
  • This Orchestrator build adds fixed issues #66203 and #67496, which are documented in this section.

August 5th, 2021, Tenth Edition

  • Added a new Orchestrator GA build R430-20210729-GA to the Orchestrator Resolved section.
  • This Orchestrator build adds fixed issues #66794 and #68577, which are documented in this section.

August 17th, 2021, Eleventh Edition

  • Added a new Orchestrator GA build R430-20210810-GA to the Orchestrator Resolved section.
  • This Orchestrator build adds fixed issues #65253 and #69046, which are documented in this section.
  • Corrected the wording for the Security Enhancements for Password Policy.

September 8th, 2021. Twelfth Edition.

  • Added a new Gateway build R430-20210903-GA-68994-65293-71052 to the Edge/Gateway Resolved section.
  • This Gateway build adds the fixed issue #68994 and #71052, which are documented in this section.
  • Removed Open Issue 49738 as it is not found on the 4.3.0 Build.

September 16th, 2021. Thirteenth Edition. 

  • Added to Important Notes the Note: BGPv4 Filter Configuration Delimiter Change for AS-PATH Prepending.

September 24th, 2021. Fourteenth Edition.

  • Added a new Gateway build R430-20210916-GA-VCG to the Edge/Gateway Resolved section.
  • This Gateway build adds the fixed issue #70416, #70590, #70855 and #71052, and #71513, which are documented in this section.

Resolved Issues

The resolved issues are grouped as follows.

Gateway Resolved Issues

Resolved in Version R430-20210916-GA-VCG

The below issues have been resolved since Gateway version R430-20210903-GA-68994-65293-71052.

  • Fixed Issue 70416: VMware SD-WAN Gateway may show a high CPU load that results in latency and packet loss for Edges using it as their Primary Gateway.

    This issue is caused by the Gateway's fast path threads (IKE, VCMP Data, etc.) spending between 15-20% of their cycles doing InetNtop operations. The fix for this issue removes InetNtop operations and replaces them with a more efficient data formatting process.

  • Fixed Issue 70590: An attempt to generate a diagnostic bundle on a VMware SD-WAN Gateway using Release 4.3.0 may fail.

    Diagnostic bundles generated on a Gateway running Release 4.3.0 fail due to the diagnostic bundle exceeding the size limit configured on the Orchestrator.  The excessive size of the diagnostic bundle is caused by audit logs that grow large over time.

    Without this fix, the only way to successfully generate a diagnostic bundle on a Gateway is for an Operator User to log into the Gateway and, prior to triggering the diagnostic bundle on the Orchestrator, the Operator User needs to delete the audit log files that are under the Gateway's /var/log/audit directory.

  • Fixed Issue 70855: A VMware SD-WAN Gateway may drop traffic originating from the VMware SD-WAN Orchestrator and as a result may cause any Gateway configuration update from the Orchestrator to fail.

    When the Gateway load is high (for example, ~1.6 million flows on the Gateway with a NAT object count of ~800K), the number of packet buffers in the system will be depleted and this can sometimes cause the Orchestrator traffic to be dropped on the Gateway. Once the Gateway enters this state, the Orchestrator traffic is always dropped even if packet buffers become available.

    Without the fix, the only remediation of this issue is to restart the Gateway.

  • Fixed Issue 71513: A user looking at the Gateways > Monitor tab on a VMware SD-WAN Orchestrator UI would observe that the Handoff Queue Drops always show a value of 0 if looking at a VMware SD-WAN Gateway using Release 4.3.0.

    Gateways running Release 4.3.0 or above do not report handoff queue drops to the Orchestrator due to incorrect formatting and this blocks the Operator from getting a clear picture of this particular troubleshooting data point.

___________________________________________________________________

Resolved in Version R430-20210903-GA-68994-65293-71052

The below issues have been resolved since Gateway version R430-20210727-GA-65293-67191.

  • Fixed Issue 68994: Customers who deploy a Non SD-WAN Destination (NSD) tunnel with any VMware SD-WAN Gateway may observe the tunnel flapping.

    This issue is observed at tunnel establishment or at IKE rekey. The Gateway deletes the security associations (SAs) based on IKESAID=0 which causes tunnel flapping. The tunnel automatically stabilizes, but the time needed to do this is not consistent and that can further the impact to customer traffic to the NSD.

  • Fixed Issue 71052: When the number of customer enterprises connecting to a VMware SD-WAN Gateway is greater than 285, the Gateway experiences a Dataplane Service failure.

    Beginning with Gateway Release 4.3.0, the Gateway's ability to monitor customers was enhanced by adding new counters to track the packet and flow related information at the customer enterprise level. The issue is that the number of counters initialized for the customer enterprises exhausts after 285 customer enterprises, and the counter initialization for any further new customer will fail, causing the Gateway's Dataplane Service to fail and forcing a service restart.

___________________________________________________________________

Resolved in Version R430-20210727-GA-65293-67191

The below issues have been resolved since Gateway version R430-20210605-GA.

  • Fixed Issue 65293: The throughput performance of a VMware SD-WAN Gateway deployed in AWS and running with Amazon's Elastic Network Adapter (ENA) driver is degraded when using Release 4.x.

    This issue will occur if the Gateway is upgraded to a 4.x build (from 3.x) or on a new deployment using a 4.x build. Gateways using Release 4.0.0 or later have DPDK v19.11, and starting from DPDK v19.02, Amazon's ENA driver uses low-latency queuing (LLQ). However, for LLQ to work efficiently the write-combine for memory setting must be enabled per the ENA reference guide.  If memory mapping is not write-combined, a Gateway deployed on AWS experiences high CPU usage, significantly impacting throughput. The fix for this issue enables write-combining on the ENA adapter for Gateways deployed on AWS.

  • Fixed Issue 67191: For a customer using a Cloud Security Service (CSS) who has also activated Layer 7 Health Check for that CSS, the L7 health check may return an erroneous failure and as a result the CSS tunnels would be torn down.

    When there are a large number of Non SD-WAN Destination (NSD) tunnels on a VMware SD-WAN Gateway, the Virtual Tunnel Interface (VTI) IP can fall out of the given subnet mask /24 range, which is defined for the probes to be processed by the Gateway dataplane service. This is what causes an erroneous L7 Health Check failure. The fix updates the mask to /16 to accept the L7 for processing in the Gateway's dataplane service.

Edge Resolved Issue

Resolved in Version R430-20210702-GA-61583

The below issue has been resolved since Edge version R430-20210528-GA.

  • Fixed Issue 61583: If a customer enables High Availability for a site, the VMware SD-WAN Edge will go offline and the site goes down and all customer traffic is disrupted. 

    When HA is enabled, the Edge will at a minimum go offline for ~5 minutes with customer traffic disrupted during that period. The Edge may be able to roll back to the previous configuration and resume operation after that ~5 minute period, though that Edge would continue to operate as a standalone with HA not possible.  However, if the Edge does not successfully roll back to the last configuration, then it will stay down until a local user performs a factory reset followed by an RMA Reactivation (with HA not enabled) to restore connectivity at that site.

    For more information please consult the KB article Enabling High Availability on a VMware SD-WAN Edge using Release 4.3.0 GA or 4.4.0 GA may cause the Edge to go offline. (84396)

___________________________________________________________________

Resolved in Version R430-20210605-GA

The below issue has been resolved since Gateway version R430-20210528-GA.

  • Fixed Issue 64633: A customer who uses a Non SD-WAN (NSD) via Gateway to connect to a VMware Cloud (VMC) on AWS peer may observe an intermittent traffic drop lasting ~30 seconds each time.

    This issue is observed with VMware Cloud (VMC) on AWS only. The peer starts an IKE rekey 30 seconds before the security association (SA) expiration and after each rekey the peer retains the old SA and uses it until its expiration, while the VMware SD-WAN Gateway deletes the inbound SA. The deletion of the inbound SA causes the traffic drop with this peer. The frequency of this issue is contingent on the peer's rekey policy. If the peer rekeys every 45 minutes, then this issue would happen every 45 minutes, if 12 hours, then every 12 hours. The traffic will recover automatically after ~30 seconds by itself, when the peer switches to the new SA.

Edge/Gateway Resolved Issues

Resolved in Version R430-20210528-GA

The below issues have been resolved since Edge version R420-20201218-GA and Gateway version R420-20210208-GA-53243-54800.

  • Fixed Issue 34037: A VMware SD-WAN Gateway may encounter a Dataplane Service failure after a peer tunnel drops.

    When a peer tunnel goes dead, the cleanup process takes the fc->vpi and assigns it to NULL. There seems to be a few packets in the pipeline which still had a reference to this fc. As part of processing those packets fc->vpi is accessed which is NULL and hence the Gateway process hits an exception and restarts.

  • Fixed Issue 34626: Invalid control packets from a VMware Edge to a VMware SD-WAN Gateway may cause the Gateway to suffer a Dataplane Service failure and restart.

    For the flows from Edge to Gateway, the Edge sends a control message to the Gateway to synchronize the business policy actions for each flow. If the control message has an invalid action, it can cause the Gateway to restart when trying to route the data packets of the flow with an invalid action set.

  • Fixed Issue 39232: On a VMware SD-WAN Edge with BGP or OSPF enabled, the Edge may encounter a Dataplane Service failure following a timeout waiting for a blocking socket to return from Tx.

    This is typically observed with the following signature:
    #0 0x00007f5c09f34754 in send () from deps/lib/libpthread.so.0
    #1 0x000000000092cc7a in vc_zeb_send_to_client (buf=0x7f5aa7fed580 "", length=<optimized out>, client=0x7f5ad402ec90) at /mnt/build/workspace/master-nightly-build/common/libs/drp/vc_zebra.c:188

    The Edge routing and dataplane processes communicate with a TCP socket on the localhost. Depending on the runtime of some threads, it is possible for the task queuing system (ksoftirqd) on the local core to receive minimal runtime to deliver packets to the opened socket to the routing process, leading to a blocked Tx call for the OSPF and/or BGP thread.

    Thread priority of the OSPF and BGP threads are now reclassified for all cores which leverage the use of the kernel scheduler to preempt rather than voluntarily yielding the core and allows for more runtime and cooperative pre-emption with ksoftirqd.

    Note: This same issue is also tracked in duplicate ticket 52127 which is omitted from these Notes.

  • Fixed Issue 39374: Bandwidth test will be triggered on both primary and secondary Gateway paths after changing the Partner Gateway order on the VMware SD-WAN Orchestrator UI and triggering the bandwidth test manually.

    Initially when a Partner Gateway configuration change is performed, a bandwidth test would be triggered on the primary Gateway path. When the order of Partner Gateways is changed and bandwidth test is triggered manually then the bandwidth test would be triggered on both the primary Gateway and secondary Gateway paths.  Bandwidth tests for both paths for all assigned Edges can heavily stress the Partner Gateway and possibly cause a Dataplane Service failure.

  • Fixed Issue 39501: On a site using a pair of VMware SD-WAN Edges in a High Availability topology, if the HA Interface (e.g. GE1 or LAN1) goes down, the Edges get into a split-brain state, meaning both will stay Active.

    An HA site in a split-brain state has severe consequences as customer traffic will not get routed for 3-4 minutes until the site recovers.

  • Fixed Issue 39654:  On a site using a pair of VMware SD-WAN Edges in a High Availability topology, an Edge transitioning from Active to Standby may encounter a Dataplane Service failure during flow creation.

    Flows are created only on the Active Edge. However, if there is a transition of HA state from Active to Standby during the flow creation process, it leads to the new Standby Edge's Dataplane Service failing as the software does not allow flow creation when an Edge is in a Standby state. Customer impact is minimal since it is the new Standby Edge with the issue, but there will be logging in Events about this issue.

  • Fixed Issue 40124: When a user disables the CELL1 interface on a VMware SD-WAN Edge 510-LTE from the VMware SD-WAN Orchestrator the CELL1 interface tunnels remain up.

    When CELL1 is disabled from the Orchestrator, it is not fully disabled, and tunnels remain up. The Monitor page also still shows the CELL1 interface.

  • Fixed Issue 41837: The NAT IP address of the source and destination gets printed in a VMware SD-WAN Edge's log instead of the original IP address.

    The Edge firewall logs should display the original source & destination IP address but instead the NAT'd IP gets printed, severely undermining the usefulness of the firewall logs.

  • Fixed Issue 43257: Routes from a VMware SD-WAN Partner Gateway are missing when there is a sequence of changes done for the Partner Gateway assignment for a VMware SD-WAN Edge.

    When a Partner Gateway is removed from the list of assigned Gateways, a corresponding route_event_queue is also removed. But when the Partner Gateway is added back, a route_event_queue is not created for the corresponding segments. As the VeloCloud Routing Protocol (VCRP) route event window creation fails with an error, further route distribution is not done properly, and this results in missing route updates from the Partner Gateway to the Edge.

    Without this fix, the only way to resolve this issue is to remove a Partner Gateway from all segments and add the Gateway again, which will properly update the segment information for the particular peer and help in recreation of corresponding VCRP window.

  • Fixed Issue 43278: If a user configures an outbound BGP filter to match the default route or prefix, and then sets an AS Path prepend, the default route or prefix is advertised to the BGP neighbor, but no AS Path prepend occurs.

    A BGP outbound neighbor filter set to match a prefix and set AS Path prepend does not work on any prefixes originated by using the BGP Advanced configuration "Networks" statement. This also does not work for a default route originated to a neighbor via the neighbor "DR Originate", via the BGP Advanced "Conditional" Default Route Advertise check box.

    Also when using a static route configuration configuration on the VMware SD-WAN Edge, neither a default route or a non-DR static route set to "Advertise" would be advertised to a BGP neighbor with the AS Path prepended; the only AS in the prefix is the Edge's own AS.

  • Fixed Issue 44832: Traffic from one Non SD-WAN Destinations (NSD) via Edge to another Non SD-WAN Destinations via Edge (i.e. 'hairpinning' or 'NAT loopback'), is dropped on the VMware SD-WAN Edge.

    NSD-to-NSD traffic via a VMware SD-WAN Edge was not supported until Release 4.3.0. With the addition of the new feature BGP over IPsec via Edge, Edges can now support NSD-NSD hairpin route exchanges along with other traffic.

  • Fixed Issue 46365: On a site using a pair of VMware SD-WAN Edges in a High Availability topology, if a user is performing an SNMP walk, they may observe a failed result on a WAN interface of the Standby Edge.

    If a user tries to SNMP walk on a WAN interface which is not connected to an active device, the response does not come back because the local kernel was not aware of the WAN link.

  • Issue 47244: On an activated VMware SD-WAN Edge 6x0 with DPDK enabled, some Copper SFPs, the Edge will show the link as 'UP' even when no cable is inserted on the VMware SD-WAN Orchestrator UI.

    This issue results in user confusion as to the actual status of a particular SFP port for an Edge in the 6x0 model line.

    Prior to this fix, the only way to resolve the issue was plugging in a random cable, and then unplugging that cable from the affected SFP port.

  • Fixed Issue 47664: In a Hub and Spoke configuration where Branch-to-Branch via Hub VPN is disabled, trying to U-turn Branch-to-Branch traffic using a summary route on an L3 switch/router will cause routing loops. 

    Spoke1 <-vcmp-> Hub <-GEx-> External Firewall <-GEx-> Hub Edge <-vcmp-> Spoke2

    In this topology, traffic from Spoke1 goes to the Hub via the VeloCloud Management Protocol (vcmp), then to an external firewall via a physical port and then back to the Hub and then to Spoke2 via vcmp. Note that packets from Spoke1 to Spoke2 have to traverse the Hub Edge twice (first while going from Spoke1 to the firewall and then a second time while going from the firewall to Spoke2). Prior to Release 3.4.0, this scenario worked properly because the Hub Edge would create two separate flows for handling the packets in the Hub (i.e., one flow for handling packets between Spoke1 and the external firewall and another for handling packets between Spoke2 and the external firewall.)

    Due to changes in the way flows are created and packets classified to flows (introduced in release 3.4.0), only a single flow is created on the Hub Edge in this scenario. This causes return traffic from the external firewall to be sent back to the Spoke from which it was just sent.

    Without this fix, the only workaround for this issue is to configure Cloud VPN to enable Branch-to-Branch VPN and select “Use Hubs for VPN”.

  • Fixed Issue 48211: On a VMware SD-WAN Edge which has the Firewall enabled, during a restart of the Edge's Dataplane Service, some iptable rules may not be installed due to "Resource unavailable" error.

    Iptables rule insertion may hit "Resource unavailable" error when called without a -w option, and this would result in an iptable rule miss, since there is no retry mechanism for the failure.

    Without the fix, if any "resource unavailable" errors were seen, the only way to resolve the issue it try to install the iptable rule again with -w option manually,

  • Fixed Issue 48958: A VMware SD-WAN Gateway may lose connectivity on a bonded interface.

    When the VeloCloud Management Protocol (VCMP) and the WAN port are set to use the same port, the Partner Gateway VLAN handoff configuration may cause the Gateway to go offline due to ARP resolutions failing. With this fix, when VCMP and the WAN port are the same, the VLAN handoff configuration from the VMware SD-WAN Orchestrator will be rejected in the Gateway.  Without the fix, the workaround is to not assign the same port for VCMP and WAN.

  • Fixed Issue 50233: A VMware SD-WAN Edge using Release 3.4.x or higher will not send information for physical LAN interfaces via SNMP. 

    In releases prior to 3.4.x when VMware SD-WAN used net-snmp, LAN interfaces were sent via SNMP. In Release 3.4.x, we added our own snmpagent, which fetches the data from the command debug.py --interfaces and that output does not have information about LAN interfaces. The fix adds LAN interfaces to that command so that the snmpagent can send the data via SNMP.

  • Fixed Issue 51025: When a WAN link flaps (alternates rapidly between an up and down state) on a VMware SD-WAN Edge, the route table entry for the routed interface’s default gateway may be removed and not reapplied.

    When an Edge encounters this issue, there is a link flap, and the default gateway route entry gets removed for the interface using that link, resulting in an empty route table for the interface. However, if left with an empty route table, Linux connection tracking (conntrack) will route to the next table by default causing all packets to egress through the wrong routed interface.

  • Fixed Issue 51165: API returns '500 Server Error' on an attempt to create a VMware SD-WAN Edge using an invalid Enterprise Profile ID.

    When a user sends "POST /sdwan/enterprises/{logicalId}/edges/" with payload having Edge Specific Profile ID as configurationId, or "POST /sdwan/enterprises/{logicalId of the second enterprise}/edges/ with configurationId that belongs to the first enterprise"; a '500 Internal Server Error' is returned. The expected result in this scenario is for the API to respond with '422 Unprocessable Entity' or generic 400 Bad Request, error message which mentions the invalid enterprise configuration ID.

  • Issue 51436: For a site using an Enhanced High-Availability topology while deploying a VMware SD-WAN Edge using an LTE modem, if the site gets into a "split-brain" state, the HA failover takes ~5-6 minutes.

    As part of the recovery from a split-brain state, the LAN ports are brought down on the Active Edge and this impacts LAN traffic during the time the ports are down and until the site can recover.

  • Fixed Issue 51502: Dynamic Branch-to-Branch tunnels are accepted by VMware SD-WAN Edges that do not have Branch-to-Branch VPN enabled.

    Edges always respond to dynamic tunnels, even when they do not have Branch-to-Branch VPN enabled. Tunnels can also form even if one side is set to 'To Edges within Profile' and the other side is set to 'To All Edges'.  There is impact in the field since inbound tunnels cannot be controlled. In one field case, Branch Edges form dynamic tunnels to Hub Edges of other regions which then causes other routing issues in the enterprise's backbone.

  • Fixed Issue 51837: If a user configures a number/integer value for some DHCP options, dnsmasq will fail for the VMware SD-WAN Edge.

    This issue is specific to DHCP Option 43 and a validation is added to ensure that numerical values will work when configuring for this options.

  • Fixed Issue 52710: High-Availability state displayed on the VMware SD-WAN Orchestrator is sometimes not accurate.

    One status that the VMware SD-WAN Edge does not push to the Orchestrator is "Failed". Under some scenarios, such as a Standby Edge going down or the Active Edge not being able to communicate with the Standby Edge, The HA state is not correctly displayed on the Orchestrator. This is due to the incorrect data being sent from the Edge to the Orchestrator because "Failed" is not an available state.

  • Fixed Issue 52991: A pair of VMware SD-WAN Edges configured for High Availability that also have VNF's deployed will output an incorrect HA state when a VNF is powered off.

    After a VNF is powered off the VNF HA state should show as 0 as the VNF is not active. VNF HA state is not monitored if the VNF is powered off so the Orchestrator displays the last state when the VNF is powered on.

  • Fixed Issue 53159: For a site deployed in a High Availability topology that also uses VNFs, the IP address assigned to a Standby VNF might be assigned by the DHCP server on the VMware SD-WAN Edge to a connected LAN side client, if the IP range of the DHCP server coincides with Standby's VNF.

    The IP address of the Standby VNF is not added in the reserved/already used IP addresses. As a result, it is possible that the DHCP server might give out the Standby VNF's IP address to a client device.

  • Fixed Issue 53518: Running a snmpwalk locally on a VMware SD-WAN Gateway does not work properly.

    When running a snmpwalk on a Gateway, the following issues are observed:

    1. Wrong Path index found. It should consist of {vcgVceUUID, vcgPathIfIntId, vcgPathGwAddrType, vcgPathGwAddr }
    2. Inconsistent Path counter. The expectation is  for the  path counter to represent all ALIVE paths found in debug.py --v --path.
    3. Inconsistent ARP counter. The Gateway snmpagent reports the counter including both ALIVE and DEAD ARP entry with only the details on ALIVE ones. Expectation is that it should either report ALIVE-only entries or all.
    4. Updates the snmpd.conf to allow all other IP access by default.
  • Fixed Issue 53415: On a customer enterprise where Edge Network Intelligence (ENI) is enabled, if the enterprise's VMware SD-WAN Edge has Wi-Fi enabled, the ENI page may show an incorrect MAC address for the Wi-Fi access point and the access point IP will show as 160.254.3.1.

    The issue is the result of a misconfiguration of the Wi-Fi access point MAC address being set to a value called 'selfMacAddress' and the access point IP address being always configured for 160.254.3.1 in the ENI page. The fix will derive the MAC address from the Wi-Fi interface wlan0, and the analytics interface's IP address.

  • Fixed Issue 53551: When activating a VMware SD-WAN Gateway using the command-line activate.py tool, a message about "No module named zmq" is emitted while the activation proceeds normally.

    This incorrect message is emitted for all Gateway activations. However, it does not affect the process of activation in any way (i.e., it is purely cosmetic).

  • Fixed Issue 53651: On a customer site using an Enhanced High-Availability topology, when making a configuration change to a VMware SD-WAN Edge device setting that requires an Edge service restart, two HA failovers in succession may occur.

    For a device setting configuration change that requires an Edge service restart, the HA module wrongly updates the LAN/WAN count to the VMware SD-WAN Gateway before the Edge service is restarted during configuration processing. As a result, when the initial HA failover happens and the current Active Edge's service restarts as part of being demoted to Standby, the Gateway misunderstands that the new Standby Edge has a better LAN/WAN count and sends a failover command to the newly promoted Active Edge, leading to the second failover.

    Note: for a list of Edge configuration changes that can trigger a service restart, please consult the KB article, VMware SD-WAN Edge configuration changes that can trigger a service restart (60247)

  • Fixed Issue 53676: On the VMware SD-WAN Edge 3x00 platform, very brief periods of input voltage instability, as short as 4 milliseconds, can cause the Edge to reboot.

    This issue is typically seen when using an Uninterruptible Power Supplies (UPS) that experiences slight output voltage instability when switching from line to battery. The fix for this issue upgrades the Edge’s firmware to tolerate 20-30ms of voltage instability prior to the Edge rebooting.

    Note: Upgrading the 3x00's firmware will extend the Edge's upgrade time to 3-5 minutes if the Edge did not previously have their firmware upgraded when using Release 3.4.5 or 4.0.2.

    For an Edge 3x00 model without this fix, the customer’s only option is to use a more sophisticated UPS that can switch its input without any output voltage instability. 

  • Fixed Issue 53750: A VMware SD-WAN Gateway may suffer a Dataplane Service failure and restart that service as a result.

    If a VMware SD-WAN Edge detects loss for the traffic received from the Gateway, it sends a negative-acknowledgement (NACK) message to the Gateway to retransmit the lost packets. The Gateway checks the retransmission slots to retransmit the packets. Ideally the Gateway should stop retransmission once all the slots are checked, but the Gateway checks the retransmission slots repeatedly until it reaches the sequence number in the NACK message, and this can cause the monitor thread in the Gateway to detect this as a hung thread and restart the gateway.

  • Fixed Issue 53789: In VMware SD-WAN Virtual Edges running under ESXi, /var/log/messages is filled with a spurious error message every 30 seconds.

    The spurious error message will show as GuestInfoGetDiskDevice: Missing disk device name; VMDK mapping unavailable for "/", fsName: "/dev/root" and is always logged into /var/log/messages, filling up /var/log/messages and its saved counterpart /velocloud/log/messages*, causing more important messages to be rotated out and lost when consulting the logs for the affected Edge.

  • Fixed Issue 53828: Cloud via Gateway traffic switches to Direct to Cloud traffic after a High Availability failover.

    After an HA failover, if the path to the VMware SD-WAN Gateway is not up when the traffic reaches the VMware SD-WAN Edge, the traffic goes 'Direct to Cloud' instead of 'Cloud via Gateway'.  This is bad for sensitive customer traffic as Direct to Cloud traffic does not use Dynamic Multi-Path Optimization and the attendant enhancements available with DMPO.

  • Fixed Issue 53929: On a customer site using an Enhanced High-Availability topology, after an HA failover, 'Cloud via Gateway' flows switch to a 'Direct to Cloud' path.

    After an HA failover, if the path to the VMWare SD-WAN Gateway is not up when the traffic reaches the VMware SD-WAN Edge, the traffic goes 'Direct to Cloud' instead of 'Cloud via Gateway'. This can have significant impact for flows that rely on Dynamic Multipath Optimizations like Realtime traffic (e.g. voice and video) because Direct traffic does not use these optimizations.

  • Fixed Issue 54136: On a site using a High Availability topology, a VMware SD-WAN Active Edge may initiate a Dataplane Service restart, resulting in an HA failover.

    The Active Edge consumes a high amount of memory when there are a high number of flows (1.9 million flows per second).  When memory consumption reaches a critical level, the Edge will restart to clear the memory and cause a failover.

  • Fixed Issue 54694: When a customer uses SNMP polling, SNMP monitoring delivers inaccurate measurements for outbound traffic

    The SNMP call for IF-MIB::ifHCOutOctets delivers TX packets instead of TX Bytes, resulting in inaccurate outbound octet counts which affects the customer's ability to monitor their enterprise. This issue is the result of the snmpagent process monitoring Tx packets versus Tx bytes.

  • Fixed Issue 55182: A Non SD-WAN Destination's routes are changed without refresh-route selected.

    When the OFC (Overlay Flow Control) global configuration order or NSD-global-advertise flags are changed, the changes which impact the NSD routes will be applied even though the refresh routes option is not turned on. This was caused by the IPsec tunnel flapping.

  • Fixed Issue 55349: A site deploying VNF's in a High Availability topology will encounter an HA failover and the VNF image will not download if the site previously deployed VNF's in HA and then the user disabled HA while the VNF's were active.

    When VNF HA is deployed on a HA pair of VMware SD-WAN Edges, after undeploying a previously deployed and up VNF HA on the same Edges, when the deployment is in progress (i.e., in the image download phase), a HA failover occurs due to the Standby Edge assuming that VNF is up on it. This sudden failover causes the Active Edge to stop trying to download the VNF image and VNF does not get deployed on both Edges. Without the fix, a user must reboot the HA Edges.

  • Fixed Issue 55827: BGP session where the BGP neighbor relationship includes MD5 authorization may fail to establish.

    A BGP MD5 password configured with special characters (e.g., $, where the password is $123) has issues and does not form a BGP session with the peer.  Without the fix, the recommended action is to configure BGP MD5 password with plain text.

  • Fixed Issue 55949: In some scenarios a Non SD-WAN Destination (NSD) via Gateway tunnel goes down and does not recover for a period of time.

    In a situation when a VMware SD-WAN Gateway triggers an IKE (Internet Key Exchange) rekey with any other NSD destination and the rekey attempt does not succeed due to a networking issue in the middle of the negotiation, the IKE rekey will keep retrying. When a link establishes back, it is possible that Dead Peer Detection (DPD) event will delete a newly created Phase1 Security Association (SA). This causes the IPsec SA to be deleted as well with some peers, most notably with Zscaler. When a peer deletes IPsec SA, the Gateway will not be able to detect it and a tunnel will be down until the next rekey time.  Without the fix, the only way to force this rekey is to bounce the tunnel by disabling and reenabling the affected NSD through the VMware SD-WAN Orchestrator.

  • Fixed Issue 56149: After Dynamic Cost Calculation (DCC) is enabled on a customer enterprise which is using BGP, a VMware SD-WAN Edge may show an incorrect route preference value for auto-corrected routes if the BGP route for the underlay route flaps.

    The impact to the customer is asymmetric routing due to the incorrect remote route preference, which results in higher latency and poor performance on all customer applications. After DCC is enabled, the new routing information base (RIB) preference value should be updated on the route and the route should be re-advertised to the VMware SD-WAN Gateway with the new RIB preference value which is then communicated to all Edges. The cause of the issue is that when the route is auto-corrected, this RIB preference is not updated in the peer Edge's FIB table, which retains the old, pre-DCC value. 

  • Fixed Issue 56346: A customer may observe Handoff Queue Drops when looking at a VMware SD-WAN Edge's Monitor > System page.

    A VCRP (VeloCloud Route Protocol) route event updates leads to handoff queue drops in the VCMP (VeloCloud Management Plane) data thread.  This is because when a route update is received, all the routes in the respective segment are invalidated. This leads to new route lookups in the data path. A particular function that is called as part of the route lookup does a costly hash enumerate operation leading to 40% increased VCMP data thread utilization.  For the instance when this issue was found in the field, the quantity of handoff queue drops was not sufficient to impact network performance.

  • Fixed Issue 56483: Packet loss, jitter, and latency values not showing in WAN link live monitoring on a VMware SD-WAN Orchestrator under the Monitor > Transport screen.

    A user is unable to get real time data for packet loss, jitter, or latency for a particular WAN link under Monitor > Transport, with the graph showing as a flat line. In addition, when looking at the Monitor > Edge > Overview screen, the all values for loss, jitter, and latency are expressed as '0'. Historical statistics will show correctly in Monitor > Transport, this issue only affects "Live Mode" statistics. 

  • Fixed Issue 56645: There are frequent WAN link flaps when a VMware SD-WAN Edge 610 is connected to certain Meraki Access Points.

    When an Edge 610 is connected to a Meraki M36 access point (or similar models), the Ethernet link encounters frequent link drops. This is the result of a driver issue on the Edge 610.

  • Fixed Issue 56876: VMware SD-WAN Edges performing dynamic branch to branch can exhaust available memory and trigger a kernel panic, even though Edge memory is not actually leaking.

    When dynamic tunnels are created, a small amount of memory is reserved for storing per-peer counters. When the dynamic tunnel is torn down, this memory is not cleaned up to optimize the bring up time the next time this same peer connects. On a small Edge (e.g., Edge 500, 510, 520, 610) which connects to many different destinations over time, this can eventually exhaust available memory and trigger a kernel panic and an Edge reboot.  Without this fix, a user needs to proactively restart the Edge's service if memory usage is greater than 90% of health statistics when looking at an Edge's Monitor > System screen on the VMware SD-WAN Orchestrator.

  • Fixed Issue 56931: A customer site that has configured a Non SD-WAN Destination (NSD) via Edge may show incorrect Edge Health Statistics on the VMware SD-WAN Orchestrator UI.

    When an NSD is configured from the Edge, the SD-WAN service sends health statistics from the Edge to the Orchestrator with a start time of 0 for the first time after reboot. This results in the Orchestrator displaying the wrong data after the Edge reboots.

  • Fixed Issue 57011: For a site configured with a High-Availability topology, whenever segments are added and then deleted on that site, one of the HA Edges may experience a Dataplane Service failure and if the service failure is on the Active Edge, the site would also experience an HA failover.

    When segments are added, and then deleted from an HA site, there is the potential for stale segments (i.e., the deleted segments might still show up on one of the Edges in the HA pair). Due to this mismatch in segment information between the HA Edges, any event meant for the stale segment might be sent to the other Edge resulting in a Dataplane Service failure, an HA failover if the service failure is on the Active Edge, and the generation of a core dump that will be found on a diagnostic bundle taken after the failover.

  • Fixed Issue 57169: User will observe the presence of stray BGP routes even when Multi-hop BGP peer is unreachable.

    .For non-global segments, the next hop tracking (NHT) update message is not processed by the BGP process when reachability goes down. As a result, when a Multi-hop BGP peer becomes unreachable, BGP-learnt routes may not get deleted right away. They will only get deleted once the BGP hold timer expires and the neighborship goes down.

  • Fixed Issue 57644: BFD session is not down when next hop reachability is down.

    BFD session in a non-global segment is not going down when a static route to reach the BFD peer IP address is deleted which means there is no route available to reach the peer IP address.

  • Fixed Issue 58061: A PPPoE interface's information cannot be fetched via SNMP.

    The IFMIB SNMP walk does not show information regarding the WAN interfaces that have PPPoE configured.

  • Fixed Issue 58075: On a VMware SD-WAN Edge where High Availability has been enabled. an SNMP walk/query will get timed out.

    SNMP query output will return only partial results and will ultimately get a timeout on a HA enabled Edge.

  • Fixed Issue 58535: When a customer has configured a Stateful Firewall, and under Network & Flood Protection has also configured a Denylist, the Denylist automatically sets itself to the most aggressive settings for new connections and the Stateful Firewall blocks any new connection.

    The issue has a critical impact for customers using a Stateful Firewall as it renders the Denylist feature unusable. Once the Denylist feature is enabled the Firewall Events are filled with the logs: "FLOOD_ATTACK_DETECTED" and "Blacklisting source: xxx.xxx.x.x exceeded CPS limit : 0 per source".  Where the IP address is the Edge's management IP address, and CPS = Connections Per Second. The New Connection Threshold limit is being set to 0% which effectively means any connection attempts will trigger the Denylist to block all connections.  The default value of New Connection Threshold is 25%.

  • Fixed Issue 58282: Flows destined to a private IP range matching the default route are allowed only if a business policy is configured with Network service as "Direct" and Link steering as "Mandatory".

    Packets to a private IP range will be dropped on the VMware SD-WAN Edge if it matches the default route and if the destination is Multipath. If the destination is Direct, the packets should not be dropped. But the packets are allowed only if Link steering is configured as "Mandatory" in the business policy.

  • Fixed Issue 58527: When running the Remote Diagnostic "List Active Flows", business policy name output is limited to 24 characters versus the expected 32 and the business policy name is trimmed to 24 characters from the actual 32 characters.

    In .edge.info, the configured biz_policy name is listed properly (even if it occupies the full length in biz_policy_name field). But while displaying the biz_policy_name in user_flow_dump/ flow_dump output, we are using only 24 chars to store the policy name. So, actual configured biz_policy is not completely displayed.

  • Fixed Issue 58567: On a site configured with a High Availability topology where VNF's are also configured, there may be frequent HA failovers due to a VNF being down.

    When a Check Point VNF is deployed with HA, the VMware SD-WAN Edge monitors the VNF state using SNMP queries. If the VNF state is marked down for 3 consecutive tries, the Edge determines the VNF to be down and initiates an HA switchover. The problem is that because a Check Point VNF can take greater than 1 second to respond to a SNMP queries intermittently, the Edge can come to an erroneous conclusion regarding the Check Point VNF's status and mark it down when it is up and make this mistake several times causing frequent HA failovers.

    Without the fix, the only way to address this issue is to increase the number of SNMP retries to a higher value than 3 before determining that VNF is down. This can be configured in /opt/vc/etc/vnf/default.json by modifying the field "snmp_retries" and powering the VNF off and then on.

  • Fixed Issue 58678: If Dynamic Edge-to-Edge is enabled and an invalid Dynamic Edge-to-Edge control message is received on a VMware SD-WAN Edge, the Edge can experience a Dataplane Service failure.

    To create a tunnel to the peer Edge, the Edge requests Dynamic Edge-to-Edge information from the VMware SD-WAN Gateway. If the reply message from the Gateway is corrupt, it can cause the Edge to restart as proper validation is missing for some of the fields.

  • Fixed Issue 58830: VMware SD-WAN Edge is dropping traffic from a routed client to a VCMP server with catch-all NAT subnet in Partner Gateway.

    Ping from an Edge routed client to VCMP traffic fails. Ping fails in the listed scenario where a default static route advertised from a Partner Gateway to an Edge and the Edge itself has the local default static route configured pointing to an underlay L3 switch next hop for routed client reachability. Here the Edge drops ICMP reply packet from VCMP with error as " rfc1918 cloud route match".

  • Fixed Issue 58985: When BGP and BFD sessions are configured with the same local_ip. If BFD/BGP local_ip is changed, the other session which is using the local_ip goes down.

    BGP and BFD use separate local_ip tables. When local_ip BFD configuration is modified, BFD local_ip is checked. As BFD is no longer using the IP address, the corresponding entry is removed from the common local_routes table (used by BGP and BFD), which impacts BGP, as it Is still using the local_ip. The fix ensures that when the BFD/ BGP local_ip is updated, a check is made to find whether the other process is still using the local_ip.
    If it is being used, it will ignore the removal.

  • Fixed Issue 59008: The link internalID for multiple USB links may be the same across several VMware SD-WAN Edges, causing incorrect USB link statistics on the VMware SD-WAN Orchestrator.

    The USB links in different Edges can have the same internal ID assigned. As a result, the monitoring across different Edges for a customer is impacted, as some data will be missed.

  • Fixed Issue 59527: On a site configure with a High Availability topology and where VNFs are deployed in HA as well, a customer may experience repetitive outages due to HA failover happening repeatedly.

    When a VNF HA is up and all the LAN-side links go down for both Edges, this issue is triggered and will continue until LAN connectivity is restored in at least one of the Edges in the HA pair.

  • Fixed Issue 59820: A VMware SD-WAN Spoke Edge may form a tunnel with a degraded Primary Hub Edge in some scenarios.

    This issue is seen in interoperability scenarios where the Spoke Edge runs a 3.X release and the Hub Edge and VMware SD-WAN Gateway run a 4.X release. The DCE control messages are misinterpreted resulting in the Spoke Edge forming a tunnel with the degraded Primary Hub Edge. This is due to incorrect message formatting of this DCE message being sent to the Spoke Edge by the Gateway.

  • Fixed Issue 59862: When running the Remote Diagnostic "Interface Status", the test may fail with message "Error Reading data for test".

    A stray USB modem entry left behind in the VMware SD-WAN Edge's network configuration file causes the issue. The actual action which causes the entry to be left is still being investigated, however this fix makes sure that the stray data is handled gracefully, and the test runs properly. On Edges that do not have the fix, a reboot of the Edge clears the stray entry, and the test will run properly.

  • Fixed Issue 60006: When HA is enabled on hardware-based VMware SD-WAN Edges like the 620 and 640, the Standby Edge may intermittently reboot.

    When HA is enabled on a 620 or 640, the Standby Edge may detect an Active/Active panic and reboot to recover from the Active/Active state.

  • Fixed Issue 60130: A site may experience intermittent periods of high packet loss and connectivity issues.

    This is caused by the API that checks for ARP resolution telling the Edge there is a successful ARP resolution for a device while delivering a MAC address of 00:00:00:00.  This address is kept in the ARP cache and any packets intended for the device where the MAC is listed as zero are dropped.  In this issue, many such instances of successful ARPs with zero MAC addresses are delivered causing high packet loss and connectivity issues.

    This fix corrects issues with the cached value of MAC addresses in a flow (the most common cause for the problem), however this fix does not address a rarer scenario where the ARP caches itself and then returns a zero MAC. That will be addressed in 62552. Other than having an Edge image with the fix, there is no workaround for this issue.

  • Fixed Issue 60225: When running the Remote Diagnostic "Interface Status" for a VMware SD-WAN Edge, the output on the VMware SD-WAN Orchestrator for SFP interfaces shows the incorrect speed and duplex information.

    The data on the Orchestrator is incorrect for SFP interfaces. For example, showing 0 Mbps / half-duplex where if viewed directly on the Edge, the data shows full duplex at 1000 Mbps, or something similar.

  • Fixed Issue 60523: Ping fails to a routed-client IP address if a SLA probe is enabled.

    ICMP response packet fails to process by the Edge Dataplane Service If a SLA probe is enabled for the routed client IP address.  Without the fix the only way to resolve this was to disable the ICMP probe.

  • Fixed Issue 61388: SNP route used for BGP-peering is getting advertised to remote Edges if that prefix is received from any BGP peer.

    If the customer advertises the local_ip used for BGP peership via BGP, there creates a datacenter static route for the same prefix advertised to other peers instead of a BGP dynamic route.

  • Fixed Issue 61400: On a customer enterprise where either Edge-to-Edge via Gateway or Dynamic Edge-to-Edge is enabled, when a user sends ICMP packets they may observe two different MTU values.

    If a user sets the interface MTU value and sends packets to the VMware SD-WAN Edge with DF bit set, the Edge might return two different ICMP error message with two different MTU values. The different value represents the interface MTU versus the tunnel MTU.  This is a day one issue and there is no customer impact.

  • Fixed Issue 61433: In a cascaded Hub/Spoke topology where one VMware SD-WAN Edge is a Spoke to a Hub-cluster and also a Hub to another Spoke, the underlay routing changes on one of the Hub-cluster members might delete the routes from this Spoke/Hub Edge.

    In case of cascaded Hub topology, the route removal on a Hub-cluster member triggered an unintentional delete message from the VMWare SD-WAN Gateway to the Spoke Edge also serving as a Hub Edge. That message was due to an update from the deep Spoke, that should have been ignored on the Gateway, but due to this issue, the Gateway sent a delete message to the Spoke/Hub Edge, resulting in the Edge losing those underlay routes (learned from the Hub cluster).

    Without the fix in this build, there is no workaround for this issue involving a cascaded Hub topology. It is advised to avoid making an Edge a Spoke to a Hub-cluster and also a Hub to a deep Spoke, otherwise this issue may appear.

  • Fixed Issue 61543: If more than one 1:1 NAT rule is configured on different interfaces with the same Inside IP, the inbound traffic can be received on one interface and the outbound packets of the same flow can be routed via different interface.

    For the NAT flows from Outside to Inside, the 1:1 NAT rules will be matched against the Outside IP and the interface where the packets are received. For the outbound packets of the same flow, the VMWare SD-WAN Edge will try to match the NAT rules again comparing the Inside IP and the outbound traffic can be routed via the interface configured in the first matching rule with "Outbound Traffic" enabled.

  • Fixed Issue 61596: There is a performance degradation when Partner BGP is enabled with secure option and a static route is configured as unsecure, or vice versa.

    The performance degradation is caused by a miscalculation of IP address maximum length when an unsecure static route picks up the secure flag from a BGP configuration. Initially the VMware SD-WAN Gateway does a route lookup and if it finds an insecure static route, the Gateway then checks if the BGP is enabled or not.  If BGP is enabled, the Gateway checks to see the encryption set and then picks up the encryption set for BGP which is secure, and then fragmentation happens because the secure option is more conservative than insecure.

  • Fixed Issue 61739: VMware SD-WAN Edge is reporting the incorrect IP address to the VMware SD-WAN Orchestrator which results in Monitor pages showing the wrong IP address.

    This can be seen on Monitor > Edge > Sources/Destinations and on Edge Events for New Client Device events. A client device connected to the Edge which tries to extend the lease timer will send a packet requesting the lease extension and as soon as the Edge receives this packet the client device's source IP becomes reversed in the Source tab on the Monitor page. It is understood as a cosmetic issue and does not affect any underlying functionality beyond the confusion over the inaccurate data.

  • Fixed Issue 62004: VMware SD-WAN Edge reports high memory usage to the VMware SD-WAN Orchestrator even though the actual memory usage of the processes running on the Edge is low.

    The Edge calculates the system memory usage by reading the /proc/meminfo file. The calculation excludes the free memory, buffers, and cached memory from the total memory to get the memory usage. The Edge does not exclude the slab reclaimable memory from the total memory and hence the memory usage reported can be high sometimes if reclaimable memory is high.  This results in users seeing high memory utilization numbers for an Edge that are not actually valid.

  • Fixed Issue 62620: On a site deployed with a High Availability topology, direct traffic to some of the destinations might stop working after HA failover.

    The flows from the Active VMware SD-WAN Edge are synced to the Standby Edge along with the port allocated for the NAT entry so that when there is a failover, there is no disruption to traffic after failover. The port allocated on the Standby Edge is never freed even after the flow expires. So when there is a failover, there is a possibility of NAT port exhaustion and NAT failure. As a result, packets can get dropped in the Edge.

  • Fixed Issue 62890: Application IDs are different when viewing statistics in Edge Network Intelligence and the VMware SD-WAN Orchestrator.

    There are two different ways that applications are learned:
    1. On the first packet, via a database of well-known SaaS applications IP addresses and ports (e.g. Office 365), or by learning the IP of an application the Orchestrator had previously learned.
    2. After a series of packets are analyzed using deep packet inspection (DPI).

    The Edge Network Intelligence Application IDs were not being updated by #2. That means that first-packet applications like Office365 are visible, but applications that require DPI (e.g. AnyDesk) are seen in the Orchestrator but not Edge Network Intelligence.

Orchestrator Resolved Issues

Resolved in Version R430-20210810-GA

The below issues have been resolved since Orchestrator version R430-20210729-GA.

  • Fixed Issue 65253: When configuring a Firewall Rule, the drop down list for Object Groups is unusable on the VMware SD-WAN Orchestrator UI when 20+ groups are configured.

    Even with 5+ Object Groups (Address Group, Port Group) configured, the Object Group drop down list appears near the bottom of the browser screen.  With 20+ rules the Object Groups list is completely out of the screen, and it’s impossible to see it unless the user zooms out a lot on the browser but by then the text is so tiny as to also be unusable.

  • Fixed Issue 69046: On a VMware SD-WAN Orchestrator, connected VMware SD-WAN Edges may not receive their routing updates and as a result continue trying to use old routes.

    When Edges queue more than 250 files in a span of 15 seconds and all of these files are small, containing only one or two route events, the routing events file enqueue job is not creating jobs for the consumers to consume. As a result, file count keeps on increasing in the file processing queue and the enqueue job become long running as the file count increases to a large number. When hitting this issue, there are many pending routing events files with the count consistently increasing. Even though the Orchestrator's upload process which handles routing requests is responding with ACKs for all routing events immediately, the Edges are not receiving the ACKs. Instead "Connection timed out" messages are seen in Edge logs. This impacts not only the Edges not getting routing events but also places stress on the Orchestrator's processing. 

___________________________________________________________________

Resolved in Version R430-20210729-GA

The below issues have been resolved since Orchestrator version R430-20210719-GA.

  • Fixed Issue 66794: If a VMware SD-WAN Orchestrator is upgraded to Release 4.3.0, a user may be unable to configure Port Forwarding or 1:1 NAT rules for a VMware SD-WAN Edge.

    For an Edge where the firewall is on but has no rules configured, on the Edge's Configure > Firewall page if the user configures a Port Forwarding rule or a 1:1 NAT rule and attempts to save that rule, the VMware SD-WAN Orchestrator will not save the rule and instead displays a 'networkSegments is not iterable' error on the page. This issue is caused by the Orchestrator using segmentID as the array index for array networkSegments.

  • Fixed Issue 68577: A VMware SD-WAN Orchestrator upgraded to Release 4.3.0 may have a UI with broken functionality and missing IPv6 defaults.

    The Orchestrator UI will not allow a user to load profiles and other settings will not save with the Orchestrator web page hanging indefinitely. This issue occurs if there is an invalid configuration found by the Orchestrator during the upgrade which triggers a "patch application failure" (this will be noted in Orchestrator events). Once an invalid configuration triggers the patch failure, the remaining configurations in the patch queue get skipped, including the IPv6 object.  The Orchestrator UI fails to load the page due to the assumption of the missing IPv6 related object being populated.

___________________________________________________________________

Resolved in Version R430-20210719-GA

The below issues have been resolved since Orchestrator version R430-20210709-GA.

  • Fixed Issue 66203: A Partner Administrator is unable to edit Partner Gateway handoffs for a customer who is assigned a Gateway pool which includes both partner managed Gateways and cloud hosted Gateways on a VMware SD-WAN Orchestrator.

    Gateway handoffs are configured on the Configure > Customer page of the Orchestrator. When a mixed Gateway Pool is assigned to a customer containing managed partner Gateways and cloud hosted Gateways (Gateways managed by VMware), the Partner Administrator is not able to modify the Gateway handoff configuration for the managed Gateways.

  • Fixed Issue 67496: A VMware SD-WAN Orchestrator upgraded to Release 4.3.0 has a minor performance regression with regard to resource utilization.

    The issue would not be noticed by a customer at the enterprise level, however the Orchestrator administrator would note a ~10% increase in resource utilization after upgrading to 4.3.0.  The Orchestrator performance issues were caused by several database queries. Those queries each had a very minor inefficiency which could impact Orchestrator performance on very large scale deployments (6000+ Edges). The fix addresses those issues and restores Orchestrator performance to expected or better when compared with the previous release.

___________________________________________________________________

Resolved in Version R430-20210709-GA

The below issues have been resolved since Orchestrator version R430-20210615-GA.

  • Fixed Issue 59434: When a user navigates away from the Configure > Edge > Device page on the VMware SD-WAN Orchestrator UI, the web page shows a navigation pop-up that reads "The changes you made will be lost if you navigate away from this page" even though the user had made no changes on the Device Settings page.

    The Orchestrator has data that shows a change has been made even though no change has been made, so the popup appears asking customers to save. This is a result of the wrong object being compared to the existing object to check the changes. Because of a wrong comparison, data was considered as modified. The fix replaces the wrong object with the correct object to compare and thus ensure no false requests to save.

  • Fixed Issue 65526: The VMware SD-WAN Orchestrator generates Alerts and Events for a VMware SD-WAN Edge in a "Degraded" state which never reaches an "Offline/Down" state.

    When a VMware SD-WAN Edge initially loses connectivity to the Orchestrator (on a heartbeat check), this state is called "Degraded".  Should the Edge loss of connectivity to the Orchestrator continue, the Edge would then be marked as Offline/Down, and this second state is when an "Edge Down" Event should be posted on the Orchestrator's Monitor > Events page and a matching Alert sent out as appropriate to a Customer's Alerts configuration.  However, the Orchestrator is generating an Event and sending an Alert for an Edge in a Degraded state, resulting in a possibly large number of spurious Edge Down Events and Alert notifications for the customer.

  • Fixed Issue 65967: For a customer performing an upgrade of an on-premise VMware SD-WAN Orchestrator to Release 4.3.0, the Orchestrator services may not come up after the upgrade is complete and the Orchestrator will appear to be down.

    This issue is the result of an upgrade script which is not able to handle invalid data sent by certain versions of VMware SD-WAN Edges. Some of the internal services in the Orchestrator cannot start properly and restarting the portal and upload services does not clear up the issue. The fix for this issue includes a patch which is refactored to skip invalid configurations and log an Operator Event with all IDs, so that the Operator can fix them later.

  • Fixed Issue 66678: When a VMware SD-WAN Orchestrator is upgraded to Release 4.3.0, the Non SD-WAN Destination via Gateway tunnels may be torn down and not rebuilt.

    This issue is caused by a validation defect introduced within a feature added for Release 4.3.0. This defect causes the VMware SD-WAN Gateway heartbeat to fail which results in the Orchestrator not pushing Gateway configurations to their respective Gateways. Since the configurations are not sent, the NSD tunnels, which are part of the Gateway configuration, are not propagated to the Gateways and the tunnels eventually go down and do not recover. This issue affects Orchestrators which have been in use for a long time and where some of the NSD peers in these Orchestrators were not associated with any segment. Since the heartbeat failure is associated with Gateways, multiple customers could face an NSD via Gateway tunnel down issue.

  • Fixed Issue 66679: A user may find a VMware SD-WAN Orchestrator portal unresponsive after it is upgraded to 4.3.0.

    Post-4.3.0 Orchestrator upgrade, the backend process does not start up as expected due to a side effect of the Bastion Orchestrator feature where Redis is being used as an intermediary for ensuring bastion settings have been configured. This causes the Orchestrator backend to run into an infinite loop because it is receiving Pub/Sub messages on 'Edge' channel subscriptions.

___________________________________________________________________

Resolved in Version R430-20210615-GA

The below issues have been resolved since Orchestrator version R430-20210527-GA.

  • Support for VMware SD-WAN Edge models 510N, 610N, 620N, 640N, and 680N.

    Orchestrator build R430-20210615-GA supports Edge models 510N, 610N, 620N, 640N, and 680N. These models do not include integrated Wi-Fi. For more details please consult the Support for New Hardware Platforms section earlier in the Release Notes. 

  • Fixed Issue 62355: User is unable to configure BGP options for a Non SD-WAN Destination with a Palo Alto Networks type.

    A prior ticket removed the ability to configure BGP from Non SD-WAN Destinations (NSD) that did not support BGP.  However, an NSD with a Palo Alto Networks type does support BGP and the ability to configure BGP was inadvertently removed from this type as well. The fix here restores those BGP configuration fields to the Palo Alto Networks type.

  • Fixed Issue 64716: User is unable to generate reports on the VMware SD-WAN Orchestrator.

    When the user attempts to generate a report it will fail and with 'Error' displayed in the Status column of the Reports page. The issue was introduced when one of the dependent packages was updated, and the updated package introduced a defect that caused all generated reports to fail.

___________________________________________________________________

Resolved in Version R430-20210527-GA

The below issues have been resolved since Orchestrator version R421-20210326-GA.

  • Issue 20900: If the MaxMind geolocation service is enabled and cannot reach the MaxMind server, new VMware SD-WAN Edge activations will not work.

    The Edge creates an HTTPS connection to the VMware SD-WAN Orchestrator in order to activate. The default timeout for the request is 120 seconds and for the proxied connection, it is 60 seconds. As the Orchestrator is attempting to geolocate the Edge (IPv4 remote address) uploads waits for the response from the MaxMind service in order to proceed with the activation. Hence, after 60 seconds, NGINX stops for the upload service’s response and closes the connection. Therefore the activation fails because of a 504 timeout from the NGINX. 

    With the new system property service.maxmind.timeout.seconds, the Maxmind API call is made with a custom timeout. If the timeout is reached, the call proceeds with the activation workflow and hence the Edge gets successfully activated.

  • Fixed Issue 29701: A user can configure routed interface's IP address with a Network ID format when using the VMware SD-WAN Orchestrator UI.

    The Orchestrator does not check or throw an error if the user configures a routed interface's IP address with a network ID type format (e.g., “1.2.3.0”). If this error is not caught it can cause issues in customer traffic flow. The fix for this issue ensures that a validation is performed for the routed interface's IP address and throws an error if the format is not correct.

  • Fixed Issue 30066: On the Customer's Gateway page, the customer will see the Partner Gateway count as zero despite having a Partner Gateway assigned. 

    The VMware SD-WAN Orchestrator uses the wrong parameter to determine if a Partner Gateway is assigned. As a result, the condition always shows false, and the count displays as zero.  The fix uses the correct parameter to check if the Partner Gateway is assigned, and ensures the correct Partner Gateway count is displayed.

  • Fixed Issue 31416: Configuration updates take more than six minutes to complete on larger customer enterprises and may time out as a result. 

    A large customer enterprise is understood as having at least 16 segments, at least 50 Hub Edges and 10K Spoke Edges where the user would assign a spoke profile to the 10K Spoke Edges and a Hub profile to the 50 Hub Edges and Edge-to-Edge is configured via 5 Hub Edges in each segment of Spoke profile.  In such an enterprise, a configuration update may take so long as to time-out and significantly impact the ability of the administrators to configure their network. The fix ensures that even enterprises on this scale successfully complete any configuration updates no matter how large.

    Without the fix, the only way to prevent timeouts on the UI was to either enable Asynchronous API (session.options.asyncPollingMaxCount: 50) or to set vco.enterprise.events.configuration.diff.enable to False. The latter disables configuration diff events, but improves overall performance of any configuration related API.

  • Fixed Issue 33026: When a user deletes an End User Services Agreement (EUSA) from the User Agreements page on the VMware SD-WAN Orchestrator UI, the page does not refresh properly.

    When the user deletes the last entry of the EUSA, the page needs to be refreshed but the Orchestrator does not trigger the refresh. The fix rewrites the validation logic for page refresh to ensure the expected result.

  • Fixed Issue 38536: User is unable to delete a VMware SD-WAN Edge when using the VMware SD-WAN Orchestrator with the error "Error processing item. Please try again".

    This issue occurs because the network allocation for an Edge is not present in the database and this breaks the process for deleting the Edge.

  • Fixed Issue 39035: For customer enterprises with a large number of Edges, Profiles, and Segments, when an administrator attempts to change the segment type, applying the configuration may take so long to complete that the process times out and the configuration change is not applied.

    A large customer enterprise is understood as +3K Edges with +1K Edges activated under a single profile where Edge-to-Edge via Gateway is enabled, and each Edge is configured with 16 segments.  Under these conditions a segment change may not complete in the required time window (6 minutes) and time out.

  • Fixed Issue 42243: User is unable to edit a Firewall rule that includes both an IP address and a domain name.

    Once a firewall rule is created with both an IP address and a domain name which is observable on the Firewall Rule page, a user cannot make changes to this rule.  For example deleting the domain name will throw an error "Please fix the problem below and try again" where the problem is removing that domain name.

  • Fixed Issue 44755: For a site using a VMware SD-WAN VNF Edge where the Edge is powered on and VNF insertion has been performed, if a user downloads a CSV from the Edge > Monitor list, the CSV shows the state incorrectly as "Powered On (Insertion Not Enabled)".

    The issue only affects the downloaded CSV report.  The VMware SD-WAN Orchestrator UI will display the correct state "Powered On (Insertion Enabled)" for a VNF Edge so enabled.

  • Fixed Issue 45293: If the VLAN option "ICMP Echo Response" is turned off on a VMware SD-WAN Edge using Edge Override, and then later the Edge Override is itself turned off which should force the Edge to use the Profile settings for this option, the Edge does not take the Profile's "ICMP Echo Response" setting, and that option remains off even if the Profile has it as on.

    This impacts a customer expecting a particular Edge VLAN to ping back an ICMP packet and the VMware SD-WAN Orchestrator indicates that it is configured to do so because the Edge should be pulling the configuration from the Profile. 

  • Fixed Issue 46254: A newly activated VMware SD-WAN Edge may lack DHCP and VLAN settings on one or more routed interfaces.

    The email activation URL should include the entire Edge-specific configuration that user has created for that Edge at the time the activation email is sent. With this issue, if the Edge has VLAN settings on a DCHP enabled interface, the email link may not include the DHCP and VLAN configurations for the Edge and when the Edge is activated, the user observes these settings missing for that Edge on the VMware SD-WAN Orchestrator.

  • Fixed Issue 46996: When a user switches a VMware SD-WAN Edge from one profile to another by using the VMware SD-WAN Orchestrator, the browser screen may freeze up and require a refresh of the browser to continue.

    The Orchestrator UI screen will become gray after confirming the profile switch when using Configure > Edges and just stay that way until the refresh is performed.  The action to change the profile is in fact successful and can be confirmed once the browser refresh is performed. 

  • Fixed Issue 47990: A user may not see the 'Interface' option while configuring a static route on the VMware SD-WAN Orchestrator.

    If a user configures a VLAN interface with an IP address and then later removes the IP address, when trying to create a static route, the Orchestrator still thinks the route is for a VLAN interface and will not let the user select the 'Interface' option for the static route entry. The user can still save the configuration and the route will show up in the route table, but the reachability flag will show as FALSE.

  • Fixed Issue 48068: An Operator with a Business role has the option to delete a VNF even though that VNF is in use.

    The result of the Business Operator's deletion of an in-use VNF will cause the Orchestrator to throw an "An Unexpected Error Has Occurred" message on the UI.

  • Fixed Issue 48131: If an Operator with a Business role logs into the Standby instance of a VMware SD-WAN Orchestrator that is configured to use Disaster Recovery (DR), the browser page will show as loading but never actually load.

    Not even a browser refresh will clear this issue. This fix for this issue includes an error message instructing the Operator with a Business role to please log into the correct instance of the Orchestrator (i.e., the Active instance of the pair).

  • Fixed Issue 48459: A customer administrator with a Customer Support role will be unable to see the Local Auth ID section of a Non SD-WAN Destination (NSD) on a VMware SD-WAN Orchestrator.

    The customer support user would be consulting the Configure > Network Services section of the Orchestrator to examine NSD details. The issue is the result of missing code to display the Local Auth ID details in the Orchestrator's read only file.  The fix adds the corresponding code and files to display the Local Auth ID in read only mode for support users.

  • Fixed Issue 48461: If a user turns off Alerts at the Edge level, the configuration is not honored for Computer Security Service (CSS) tunnel alerts.

    Alerts can be turned off for a particular VMware SD-WAN Edge by going to Configure > Edge > Overview and unchecking the Enable Alerts box. The expectation when this box is unchecked is that the Edge would not send any alerts of any type for any reason. However, if the Edge is configured to use a CSS, those users configured to receive alerts would receive them for CSS tunnel events on this Edge (i.e., CSS Tunnel Up or Down).

  • Fixed Issue 48633: A customer administrator with a Customer Support role will be unable to see several configuration details of a Non SD-WAN Destination (NSD) on a VMware SD-WAN Orchestrator.

    This is virtually identical to 48459 but covers different parameters of an NSD configuration including username and hostname. The customer support user would be consulting the Configure > Network Services section of the Orchestrator to examine NSD details. The issue is the result of missing code to display the Local Auth ID details in the Orchestrator's read only file.  The fix adds the corresponding code and files to display the Local Auth ID in read only mode for support users.

  • Fixed Issue 48641: An Operator with a Support role is unable to see a VMware SD-WAN Gateway's 'Gateway Authentication Mode' on the Gateways page of the VMware SD-WAN Orchestrator UI.

    As a Support Operator is read-only for configurations there was a miss in allowing such an Operator to at least view the Gateway's Authentication Mode, instead only Operators with configuration privileges could see this field.

  • Fixed Issue 48642: An Operator with a Support role is unable to view the 'License' field on the Configure > Edge > Edge Overview section of the VMware SD-WAN Orchestrator.

    Previously there was no code for viewing the License field in read only, which meant Support Operators could not see the field. With this fix, the License field may also be viewed in read only.

  • Fixed Issue 48645: An Operator with a Support role is able to configure the 'Deprecated' field on an Edge Software Image.

    A Support Operator, if they navigate to the Software Images section of the VMware SD-WAN Orchestrator UI, can select an Edge image to examine and the 'Deprecated' field should be grayed out for that Operator as they have read only privileges. This issue is the result of a lack of condition/validation checks to check whether the Operator has privileges to edit the checkbox or not.

  • Fixed Issue 48915: On the Cloud Security Service (CSS) Events page, the CSS Type does not display properly.

    CSS Events are found on the Network Services page of the VMware SD-WAN Orchestrator.  The page title shows raw code (e.g., "UIPlugins.genericCloudSecurityService.title") instead of an actual title string. The fix adds a title string for the raw code that was previously displayed.

  • Fixed Issue 49395: A VMware SD-WAN Edge shows a systemUpSince timestamp from when the device is first provisioned and not from when the Edge is first activated.

    The issue can be seen by selecting the Edge information drop down on Monitor > Edges > Edge Overview. The indicated systemUpSince time prior to Edge activation has no meaning and hinders troubleshooting of the Edge.

  • Fixed Issue 49997: If VMware Edge Network Intelligence Analytics Mode is enabled on a VMware SD-WAN Orchestrator, when a new operator user is created that operator is not able to connect with the analytics sections of the Orchestrator UI.

    Operator users created after activation of analytics mode should be able to access the VMware Edge Network Intelligence UI of all enterprise customers that have enabled support access, and that is not the case with this issue.

  • Fixed Issue 50082: When changing the DHCP lease time via API or by modifying the values in the VMware SD-WAN Orchestrator UI via the inspect element breaks the Orchestrator UI as the UI only supports the default values of 900, 3600, 14400, 43200, 86400, and 604800.

    This issue can be seen with the following steps:

    1. Create a new profile.
    2. Change the default VLAN DHCP lease time to something other than 900, 3600, 14400, 43200, 86400 and 604800.
    3. Assign that Profile to an Edge.
    4. Select the 'Edit' button of the VLAN within the Edge.
    5. Nothing happens.

    The fix adds a validation for the DHCP lease timer as it should be one of the [900, 3600, 14400, 43200, 86400 and 604800] values.  A user will not be allowed to create a profile with an invalid DHCP lease time value via either the API or the Orchestrator UI.

  • Fixed Issue 51608: The Remote Diagnostic 'DNS Test' does not accept a domain name with an underscore '_' character.

    When attempting to enter in a domain name with an underscore symbol (e.g., _test.com) for the DNS Test, the VMware Orchestrator will return an error reading: "_test.com is invalid".  The fix adds an underscore symbol for domain name validation.

  • Fixed Issue 51620: A VMware SD-WAN Edge interface is not reset to 'none' for static routes when a switched port is converted to a routed interface.

    Issue is after changing the interface mode from routed to switched, the static route dropdown reset does not happen. The fix triggers the dropdown reset functionality on interface mode changes.  This issue is not seen when changing from switched to routed.

  • Fixed Issue 51707: When a user configures a VLAN without a name, the VMware SD-WAN Orchestrator throws the wrong error.

    The Orchestrator is displaying: "name must not contain < and >" error, instead of the correct error: "Name is required for VLAN", when creating a VLAN without a name.

  • Fixed Issue 51714: A user is able to configure the 'Time offset' field under DHCP as a decimal value, which cause the dnsmasq to fail.

    On a VMware SD-WAN Orchestrator, if a user goes to Configure > Edge > Device settings and selects a routed interface where they enable DHCP and in the options add Time offset with value of, for example, 0.11 and save the changes. The user could check the Events page and notice the following: 'observed dnsmasq[20245]: FAILED to start up error'.  The fix includes a validation check to ensure that decimals are not allowed for the Time offset field.

  • Fixed Issue 52034: User is able to configure a non-integer value for the Local UI Port on the VMware SD-WAN Orchestrator.

    The is found on the Configure > Firewall page for an Edge or Profile under the "Edge Access" section.  A user could configure a non-integer like 99.99 as the Local UI Port and the Orchestrator would not throw an error and accept it. The customer impact was lessened by the fact that the Edge dataplane was handling this parameter where, in the event of a non-integer value, it assigns the default port 443.  The fix now includes a validation for this value to ensure it is an integer.

  • Fixed Issue 52107: An Operator with a Support role is unable to see the region field for a VNF configuration on a VMware SD-WAN Orchestrator.

    A Support Operator when viewing the VNF Service Management Configuration box would see the Region field as a series of security dots. The fix allows the Region field to be seen as read only.

  • Fixed Issue 52293: Tunnel status tooltip shows the wrong interface name in Monitor > Non SD-WAN Destinations via Edge if there are multiple tunnels.

    On the VMware SD-WAN Orchestrator when the user has more than one tunnels configured on a NSD via Edge, the UI shows the same interface name for all the tunnels. The first interface name is being copied for all the entries.

  • Fixed Issue 52317: The Gateway's Partner off field names under the Gateway Pool module on the Customer Configuration page is truncated.

    If a Partner or Operator user goes to Configure > Customer on a VMware SD-WAN Orchestrator, and scrolls down to Gateway Pool, the user will see, instead of the Partner Gateway name, a truncated name displayed (e.g., "Pa..." and "C..."). 

  • Fixed Issue 52329: On a VMware SD-WAN Orchestrator's New UI, the pie chart in Customer View does not display any numerical values when a user hovers over it as it would for pie charts on the Edge Overview page.

    User is unable to see number of items of specific status when they hover over a series on Total Customers or Total Edges pie charts.

  • Fixed Issue 52379: The VMware SD-WAN Orchestrator sends out an ‘Edge Down’ alert email if the VMware SD-WAN Edge recovers within the configured delay interval.

    Administrators can be falsely alerted of an Edge being down in their network even though they configured a delay to allow an Edge to be down for a period of time before triggering that alert.

  • Fixed Issue 52729: When configuring a VNF, if the user does not include correct values for mandatory fields, the VMware SD-WAN Orchestrator UI displays "An unexpected error has occurred" without providing any guidance on what fields need to be configured.

    A user could provide an invalid hostname and the error message is just the vague "An unexpected error has occurred" versus explicitly stating that the hostname is invalid. The fix adds a proper validation for these VNF fields and more explicit error messages.

  • Fixed Issue 52820: On VMware SD-WAN Orchestrators with a large number of VMware SD-WAN Edges, a configuration change made via the UI may result in a timeout error being thrown.

    If a configuration request is not communicated to the Orchestrator within 90 seconds, the timeout error is thrown.  The fix manages large scale Orchestrators (at least 2000 Edges) by automatically configuring an asymmetric API system.

  • Fixed Issue 52835: On a VMware SD-WAN Orchestrator using the New UI, Non SD-WAN Destinations via Edge tunnel service name is not updating properly in the Monitor Edge screen.

    When a user opens Monitor > Edges list, the user will see a technical service name in the 'Edge tunnels' column tooltip instead of a user-friendly display name. For example, Non SD-WAN Destinations via Edge tunnel Name "Generic IKEv2 router" is updating wrongly as "Generic ik v2 router".

  • Fixed Issue 52851: A VMware SD-WAN Gateway activation event is shown in the VMware SD-WAN Orchestrator UI with an incorrect description which reads, "Edge activation".

    This is seen in Operator Events every time a new Gateway is activated.

  • Fixed Issue 52863: The VMware SD-WAN Orchestrator UI allows non-standard BGP timer configurations and does not throw an error.

    While enabling Partner handoff configuration at the customer configuration page, when a user configures the BGP keep/hold timers on the Orchestrator that do not comply with the BGP standard in RFC 4271, the Orchestrator allows the configuration to be saved. However, on the VMware SD-WAN Edge itself, FRR changes the keep/hold values to comply with standards. For instance, if a user configures a keep 2 second/hold 5 second on the Orchestrator, the Edge FRR will change the keep value to 1 sec so that 3 x keep = (less or equal to hold).

  • Fixed Issue 52885: A partner or customer administrator is able to change their login password by only filling out the first field and not needing to fill in the confirmation field.

    This is true for all user roles at either the partner or customer level.  This can result in an administrator thinking they entered one password when it reality it is not exactly what they intended which is what the confirmation field is designed to prevent.

  • Fixed Issue 52903: When a user tries to add a new Cloud Security Service (CSS) provider without the correct API token, the VMware SD-WAN Orchestrator UI returns "Invalid Username or password".

    This has been seen with the most common CSS: Zscaler. There are two other issues associated with this ticket: The password and token fields do not take in input when in clear text.  And a token with the right prefix will be treated as the correct token even if it is not. The fix for all these issues is improved validation of CSS API credentials.

  • Fixed Issue 52918: On a customer enterprise that deploys at least one VNF, if at some point the VNF deployment details do not match the actual deployment details, a VMware SD-WAN Orchestrator using the New UI will not issue a warning about the mismatch.

    The Old UI works properly for this scenario.  The error message that should pop up reads "The current VNF deployment does not match the configured state for this Edge.  This typically happens when a recent change is pending application by the Edge and usually is resolved after a few heartbeat cycles.  See Edge events for additional information."

  • Fixed Issue 52934: Azure Automation for Network Services, either Non SD-WAN Destination via Gateway, or Non SD-WAN Destination via Edge will not start.

    User would see this issue when configuring adding/deleting NSD-via-GW or NSD-via-Edge for type Azure. User may also see that when the WAN link IP is changed the update action to Azure will not get processed.  Without the fix, the only way to correct this issue is to restart the VMware SD-WAN Orchestrator's backend process.

  • Fixed Issue 53076: The VMware SD-WAN Orchestrator allows a user to enter in a Mobile Phone # for their administrator account with a numerical format that does not work correctly.

    This issue can cause a two-factor authorization user to get locked out as the user will not get their SMS text to complete 2FA. If the mobile number is entered in the format 00xxxxxxxx, the VMware SD-WAN Orchestrator does not check the field and accepts the number. It even accepts clearly wrong numbers such as "+-+00". But when someone tries to log in with that user, there is an error: "The SMS service encountered the following error:[object Object]". If the number is changed to +<country_code>xxxxxxxxxx it will work properly.  The fix includes a validation check for the Mobile # and the Orchestrator will not allow phone numbers that cannot work.

  • Fixed Issue 53428: When a user configures two customer enterprises with the same account number, the VMware SD-WAN Orchestrator displays an "Internal error" message.

    If a user provides an account number for an enterprise in the System Setting page, and that account number already exists for some other enterprise, then on saving the changes, Internal error was shown in the UI. The error message is vague and does not tell the user what needs to be corrected.  

  • Fixed Issue 53525: When using the New UI on a VMware SD-WAN Orchestrator and viewing the Edge overview page, the Links column does not show the state of the link (e.g., Backup, Standby).

    This link state information is correctly shown on the Old UI and with this fix will show as expected on the New UI.

  • Fixed Issue 53652: When a customer enterprise that is using a custom application map is upgraded from 3.x to 4.x, the customer may observe random names for their custom applications created prior to the upgrade.

    Whenever a custom application map is configured with an Application ID (appId) which already exists as part of a default initial application map, the VMware SD-WAN Orchestrator will always show the display name of the default initial application map and override the customer defined name. This is also true when the Orchestrator is upgraded from a lower version to a higher version and the higher version default initial application map has an appid which conflicts with an appId of the custom applications created in a lower version.  After the Orchestrator upgrade, those custom applications will show an incorrect display name which is the display name of the appid for the higher version's default initial application map.

  • Fixed Issue 53752: A customer enterprise migration fails when it is attempted from a VMware SD-WAN Orchestrator using a 3.4.x release to an Orchestrator using release 4.2.0.

    The latest enterprise migration tool did not support Release 4.2.x, and this was the cause of migration failures.

  • Fixed Issue 53767: Customer may see the HTML codes on a business policy when looking at them on the VMware SD-WAN Edge UI.

    The dynamic messages were assigned to a variable and the variable is appended to a string, so the entire value of the variable is displayed as a string including the HTML tags. The fix moves the place where the variable renders the text into a new tag.

  • Fixed Issue 53798: Customer can configure an invalid property type or value to create a Partner Gateway handoff.

    VMware SD-WAN Orchestrator is missing a syntax validation for the insertion or update of an enterprise Gateway Handoff that is corrected in this build.

  • Fixed Issue 53882: Customer can edit the BFD rules on a VMware SD-WAN Edge even when Edge override is turned off.

    The VMware SD-WAN Orchestrator is missing a validation check to see if Edge override is turned off when attempting to edit BFD rules.

  • Fixed Issue 53857: A VMware SD-WAN Orchestrator deployment which uses a KVM image based on Release 4.0.0 will fail to deploy.

    The reason for the failure is that the KVM image has an incorrect virtual disk size and the volumes will not expand to the required size.  On a deployment, the Orchestrator scripts automatically expand Orchestrator volumes to take 80% of the maximum size of the underlying disks (physical volumes).  In this case, because of the incorrect virtual size, that expansion is inadequate for Orchestrator database requirements and the deployment fails.  It is possible to deploy an Orchestrator using an older build without this fix, but the volumes must be resized manually.

  • Fixed Issue 53987: On a VMware SD-WAN Orchestrator, the Edge Event 'Link MTU Detected' is not searchable on the Orchestrator UI.

    This issue has been observed on Orchestrators using release 4.0.x and higher.  While doing an event search in the Orchestrator UI under the Events page, 'Link MTU Detected'" is not available in the Event list for filter, making it difficult to isolate that event as part of a troubleshooting effort.

  • Fixed Issue 54035: If VMware Edge Network Intelligence Analytics Mode is enabled, packets destined for the syslog daemon, aruba daemon and snmptrap daemon are dropped on the VMware SD-WAN Edge and this data will not show on the Edge Network Intelligence viewer.

    The packets destined to the Edge Network Intelligence daemons (syslogd, amond and snmptrapd) are dropped in the Edge dataplane process due to missing corresponding iptable rules. As a result the corresponding stats are not received in the Edge Network Intelligence backend.

  • Fixed Issue 54546: The VMware SD-WAN Orchestrator UI does not display the Cloud Security Service or Non SD-WAN via Edge tunnels correctly on the Monitor > Edges page.

    The issue may happen when there are multiple VMware SD-WAN Edges that use WAN links through USB interfaces for CSS or NSD via Edge tunnels. The Orchestrator is sorting the tunnel events by time and the latest event per ‘datakey’ is being used to determine the effective state. Since the key’s value is same for many entries, some of the tunnels got left out. This is a display issue only with no customer impact beyond showing a false status.

  • Fixed Issue 54861: On enabling a Cloud Security Service (CSS) Zscaler automated provider after switching from a manual IPsec provider, the old VPN credentials are not cleared from the VMware SD-WAN Orchestrator's Device Settings, with the result that automation does not begin.

    The user assigns a manual CSS IPsec provider to the VMware SD-WAN Edge and configures its VPN credentials. Then the user switches to automated CSS provider and the old VPN credentials are not cleared by the UI. This leads to the automation not engaging. Without the fix the only way to resolve the issue is to clear all the pre-existing VPN credentials manually from the Orchestrator UI.

  • Fixed Issue 55205: When a user hovers over the QoE graph on the Monitor > QoE page of the VMware SD-WAN Orchestrator, the tooltip lists the packet delay values in the wrong order.

    The current order the values are shown is jitter, then latency.  But jitter is a packet delay variation and should be properly listed after latency since latency is the straightforward expression of packet delay.

  • Fixed Issue 55259: When an administrator creates a new VMware SD-WAN Edge on the VMware SD-WAN Orchestrator UI, the "Set Location" field is missing.

    With this issue, the administrator can create the Edge but without location information and the Orchestrator cannot perform automatic geolocation for the Edge and assign the correct Gateways.  The administrator has to perform an additional step after the Edge is created to go in and fill out the Edge location information on the Configure > Edge > Edge Overview page.

  • Fixed Issue 55294: A keyboard only user must cycle through every navigation link every time they navigate to a new page on the VMware SD-WAN Orchestrator UI.

    Each page is missing a' Skip to Main' content link. When keyboard-only users interact with the application, they use the tab key to jump from link to link. Users need to tab through every navigation link every time they come to a new page just to get to the main content.

  • Fixed Issue 55367: On the VMware SD-WAN Orchestrator's New UI, on the Monitor > Applications page there is a print button that is not labeled.

    While the print button conforms visually to what a print button should look like, there is no text to explicitly communicate to a user what this button does or how to use it.

  • Fixed Issue 55861: A Non SD-WAN Destination via Gateway with Check Point type may have the wrong NSD type stored on the VMware SD-WAN Orchestrator which will lead to tunnel failures.

    The correct tunnel type for a Check Point NSD is "Other".  However, after an Orchestrator upgrade, the NSD type stored on the Orchestrator may read "Checkpoint" versus "Other", and in this instance, "Checkpoint" is actually incorrect.  Since the Orchestrator is sending the wrong NSD type to the Gateway, the tunnels may not form.

  • Fixed Issue 55871: Some API calls to REST APIv2 (/sdwan) HTTP cause the server to produce HTTP 500 errors.

    In some cases where customer data does not conform precisely to the schema that the API expects, the API produces an HTTP 500 error rather than return data which is inconsistent with the documented API schema. This behavior was driven by a design decision that has since been revisited. Calls to "GET /enterprises", "GET /enterprises/{enterpriseLogicalId}/edges", and "GET /enterprises/{enterpriseLogicalId}/clientDevices" are known to be affected.

  • Fixed Issue 56545: DHCP negative acknowledgment (NAK) packets will not be forwarded to the client when the VMware SD-WAN Edge is configured as a DHCP relay agent and the DHCP server is also located in the same branch.

    f the Edge is configured as a DHCP relay agent and the DHCP server is located in the same branch, then when the IP address which is not in the range of the DHCP server is requested, the DHCP NAK packets from the DHCP server will be dropped by the Edge acting as the DHCP relay agent.

  • Fixed Issue 56763: On a VMware SD-WAN Orchestrator using Release 4.x or later with reports enabled, if a report fails to generate for whatever reason, all subsequent reports for all customers using the Orchestrator will also fail to generate until the Orchestrator's backend service is restarted.

    This issue has a significant impact on an affected Orchestrator because all customers using the Orchestrator will be unable to get reports until the Orchestrator backend service is restarted. This issue is caused by a single report failure which puts the reporting service into a bad state from which it cannot recover outside of restarting the backend service on the Orchestrator.  This is because new report generation does not occur independent of prior report generation.  The fix ensures that the reporting service continues to generate new reports independently of a report generation failure.

  • Fixed Issue 56824: On a VMware SD-WAN Orchestrator using Release 4.2.x, delivery of alerts to Webhook recipients fails when the recipient URL includes an explicit port number.

    Users that had previously configured Webhook recipient URLs that included explicit port numbers may observe that alert delivery fails indefinitely to those recipients. Without the fix in this build, an administrator would need to configure a reverse proxy to pass requests along to the original Webhook recipient and update the Webhook recipient URL to point to the reverse proxy.

  • Fixed Issue 56896: User could experience API failures and Gateway timeouts.

    This issues is the result of a VMware SD-WAN Orchestrator's disk storage becoming full due to an accumulation of files. This accumulation occurs because there is a way to disable flow stats processing for a VMware SD-WAN Edge or a list of Edges which resembles a block-list / deny-list feature. Although the flow processing is skipped for these Edges, the issue is that the files remain on the Orchestrator's disk without getting deleted. In the field found instance of this issue there was sufficient monitoring in place to catch the issue and prevent any user experience issues, but on an Orchestrator where there was less monitoring this could impact customer traffic. Without this fix, an Orchestrator operator would need to manually delete files on the Orchestrator's disk storage with a timestamp of more than 24 hours.

  • Fixed Issue 56909: On a VMware SD-WAN Orchestrator using Release 4.x, report generation may fail when a backup link is included.

    If a link has no link statistics records, the report generation throws an error.  A link set to backup will generate no link statistics if it strictly remains backup during the period selected for the report. Without this fix the only way to generate a report is to unselect the backup link while generating a report so that the link has some statistical data for its record.

  • Fixed Issue 57046: At the creation of a customer on the VMware SD-WAN Orchestrator, Operator access is disallowed. But when the Orchestrator is upgraded to a 4.0.x release, the MODIFY_ENTERPRISE_CUSTOM_ROLES and VIEW_PATH_STATS privileges are added by system patches without checking the Operator access.

    Customer settings show permissions delegated to the Operator, and yet the Operator does not have the permissions to, for example, read network services or modify Edges/Profiles. So there is a false status displayed for what an Operator may do on that customer's enterprise.

  • Fixed Issue 57087: When the user tries to switch a VMware SD-WAN Edge`s profile from the Configure > Edge screen, the user will observe a validation error which includes a notification box with a generic error message instead of the actual reason.

    The generic error seen reads "Error processing item. Please try again". For the actual validation error reason, the user had to check a web browser's debugging console. After the fix, the appropriate validation error/reason for failure is displayed.

  • Fixed Issue 57163: Customer cannot receive notifications via SNMP trap for Cloud Security Service (CSS) or Non SD-WAN Destinations (NSD) via Edge tunnel alerts.

    The issue occurs when a customer wants to use a SNMP trap to receive CSS/NSD via Edge tunnel alerts, but the SNMP traps are not being triggered for those events.

  • Fixed Issue 57193: When configuring Reports on the VMware SD-WAN Orchestrator, a user is unable to schedule recurring reports when using option "Last Year".

    The Orchestrator throws a validation error when this is attempted. This is a UI limitation on the browser window size.

  • Fixed Issue 58127: A user will observe several issues with Enterprise Reporting.

    The issues include an ID field on the reports that should not be shown, the username is missing in the generated report, and a 50 report limitation.  This ticket fixes the first two issues.  The 50 report limitation is a System Property configuration designed to prevent Orchestrator performance issues. While an Orchestrator administrator could increase the report cap, it would entail some performance risk on the Orchestrator.

  • Fixed Issue 58288: When using the New UI on the VMware SD-WAN Orchestrator, an incorrect number of tunnels per status are displaying on the Monitor Edges > "Edge tunnels" column.

    When a user opens Monitor Edges list, they will see an incorrect number of tunnels per status in "Edge tunnels" column.

  • Fixed Issue 58627: Users configured to receive Alerts may receive a Link Up Alert when in reality the link remains down.

    Sometimes after a link is marked as 'Down', statistics for that link that were generated before the link went down may not be sent to the VMware SD-WAN Orchestrator for up to a minute after the event.  Once the Orchestrator receives these lagging link statistics, it is fooled into thinking the Link is back up and thus triggers a Link Up alert if the Alert settings are aggressive (e.g. 0 minute delay).  The fix ensures that the Orchestrator does not interpret delayed link statistics as indicating that the link is now up.

  • Fixed Issue 58996: Updating BFD fields from the VMware SD-WAN Orchestrator UI handles validations and works without any issues. But the corresponding API request is missing the validations. Also, the JSON schema is missing for BFD fields.

    The API request is missing JSON schema information which is used to generate the documentation of API. Also, the API request is missing the server side validation for the corresponding fields which were properly validated from UI.

  • Fixed Issue 59094: When an operator is attempting to upgrade a VMware SD-WAN Orchestrator, the update script does not provide a proper warning message about the schema update requirements.

    If an operator misses the step to apply schema changes on the larger tables, there could be an error on the Orchestrator services.  Also there is not an easy way to find out what changes are missing. This fix addresses this issue when, upon a backend service restart, it will regenerate any missing schema changes required on a large table.

  • Fixed Issue 59689: When using the New UI on a VMware SD-WAN Orchestrator with an exceptionally high number of enterprise and Edges, the Monitor > Firewall Logs page may load slowly or stop responding completely.

    This issue has been seen on a hosted Orchestrator with 200+ customer enterprises and thousands of Edges.

  • Fixed Issue 59967: After a VMware SD-WAN Orchestrator is upgraded to a 4.2.x release or higher, when an Operator user attempts to access the Configure > Business Policy or Configure > Firewall policy pages, the page will not load, and the user will see an error.

    The error reads "An unexpected error has occurred". This affects Operator users, not Partner or Customer Administrators. The pages do not load due to a missing READ: OBJECT_GROUP privilege for Operators, meaning the Orchestrator does not recognize an Operator as having the necessary privileges to access the Business Policy and Firewall pages.

  • Fixed Issue 59987: In the Edge activation email, the custom message displays *****, instead of the actual text.

    Earlier message key was added in the sanitization list. And that is why, while sending any email, like an Edge activation, password reset (eg., in the email text, in event creation etc.) the message key was getting obscured. But only for the activation email does the message key need to be displayed as that field is used for sending some useful custom information to the user.

  • Fixed Issue 60039: RMA Reactivation does not work when the VMware SD-WAN Edge model is changed.

    When performing an RMA Reactivation for a site where the Edge model is also being changed, the VMware SD-WAN Orchestrator does not save the model change and instead assigns a Virtual Edge to the site, making the reactivation link useless. This only affects RMA Reactivations where the Edge model is changed, an RMA Reactivation where the Edge model remains the same will work as expected.

  • Fixed Issue 60366: A user using token based authentication cannot generate a diagnostic bundle on the VMware SD-WAN Orchestrator.

    Users would not be able to generate Diagnostics Bundles when using Token Based Authentication.

  • Fixed Issue 60405: pathType data field is absent on the VMware SD-WAN Orchestrator that is upgraded to 4.3.x.

    A new field `pathType` was introduced in the PathStats table. The field was only observed in a newly deployed Orchestrator and not in the upgraded Orchestrators. The fix makes sure that the 'pathType' data field is present in both, newly deployed Orchestrators as well as upgraded ones.

  • Fixed Issue 60444: If a user is configuring a specific rate limit for a customer/partner and then disables that policy using the `enabled: false` attribute may affect rate limit numbers.

    The issue comes out of this particular use case where a user defines a policy but then disables it in the policy itself.  Without the fix, the workaround for this problem is to not add the policies in the rate limit policies array if they are supposed to be disabled. Basically skip all policies which are disabled. That way only the existing rate limit policies will be applicable.

  • Fixed Issue 61000: Newly created Operator Profiles may not be selectable from the Partner Overview page on the VMware SD-WAN Orchestrator UI.

    When an Orchestrator has over 100 Operator Profiles, and then a user attempts to select some of them from the Partner Overview page, only 100 will be displayed in the UI. Without the fix, the only way to address this is to have VMWare SD-WAN Technical Support assign the requested Operator Profile.

  • Fixed Issue 61312: A VMware SD-WAN Orchestrator may encounter an issue where routes are no longer updated and the CPU utilization of the Orchestrator is near 100%, especially after the Orchestrator is upgraded.

    This issue manifests when an Edge sends ~2K+ route updates to the Orchestrator's routing API. In those scenarios where the Orchestrator is unable to process the entire set of routes sent on a particular API call within 60 seconds, it results on a timeout for that call which in turn results in the API call being rejected entirely. The Edge receives this rejection and attempts to push the same 2K+ routes to the Orchestrator again, leading to the same scenario as before which creates a loop that overloads the Orchestrator's vCPU resources. When present this issue can prevent route updates from being processed. 

    To address this issue, two system properties have been added:

    edge.learnedRoute.maxRoutePerCall  This property ensures only a limited number of routes are processed from an Edge. If the property value is ‘200’, then 200 routes will be processed per Edge request which ensures that an acknowledgment is sent to the Edge on time.

    vco.learnedRoute.simultaneous.maxQueue  This property ensures only the configured number of Edges may have route requests queued at a time. If the property value is ‘8’, then only 8 Edges would be permitted to send route requests at a time and those in excess of the configured value would be rejected immediately prior to the routes being processed.

  • Fixed Issue 61852: Monitor > Firewall Logs page in the New UI does not display correct pagination information.

    The page row count is incorrect for this section.

  • Fixed Issue 62038: When client visibility is set to MAC mode on the VMware SD-WAN Orchestrator,  the Edge > Monitoring > Sources page will never display the latest IP for a given MAC address.

    In a given branch, if a client changes its IP address then the user will not be able to see the latest IP address for that client on the Edge Monitoring page. Instead, the user will see a stale IP address. This issue is the result of business logic querying the wrong database.

Known Issues

Open Issues in Release 4.3.0

The known issues are grouped as follows.

Edge/Gateway Known Issues
  • Issue 14655:

    Plugging or unplugging an SFP adapter may cause the device to stop responding on the Edge 540, Edge 840, and Edge 1000 and require a physical reboot.

    Workaround: The Edge must be physically rebooted.  This may be done either on the Orchestrator using Remote Actions > Reboot Edge, or by power-cycling the Edge.

  • Issue 25504:

    Static route costs greater than 255 may result in unpredictable route ordering.  

    Workaround: Use a route cost between 0 and 255

  • Issue 25595:

    A restart may be required for changes to static SLA on a WAN overlay to work properly.  

    Workaround: Restart Edge after adding and removing Static SLA from WAN overlay

  • Issue 25742:

    Underlay accounted traffic is capped at a maximum of the capacity towards the VMware SD-WAN Gateway, even if that is less than the capacity of a private WAN link which is not connected to the Gateway.  

  • Issue 25758:

    USB WAN links may not update properly when switched from one USB port to another until the VMware SD-WAN Edge is rebooted.  

    Workaround: Reboot the Edge after moving USB WAN links from one port to another.

  • Issue 25855:

    A large configuration update on the Partner Gateway (e.g. 200 BGP-enabled VRFs) may cause latency to increase for approximately 2-3 seconds for some traffic via the VMware SD-WAN Gateway.

    Workaround: No workaround available.

  • Issue 25921:

    VMware SD-WAN Hub High Availability failover takes longer than expected (up to 15 seconds) when there are three thousand branch Edges connected to the Hub.  

  • Issue 25997:

    The VMware SD-WAN Edge may require a reboot to properly pass traffic on a routed interface that has been converted to a switched port.  

    Workaround: Reboot the Edge after making the configuration change.

  • Issue 26421:

    The primary Partner Gateway for any branch site must also be assigned to a VMware SD-WAN Hub cluster for tunnels to the cluster to be established.  

  • Issue 28175:

    Business Policy NAT fails when the NAT IP overlaps with the VMware SD-WAN Gateway interface IP.  

  • Issue 31210:

    VRRP: ARP is not resolved in the LAN client for the VRRP virtual IP address when the VMware SD-WAN Edge is primary with a non-global CDE segment running on the LAN interface. 

  • Issue 32731:

    Conditional default routes advertised via OSPF may not be withdrawn properly when the route is turned off. Re-enabling and disabling the route will retract it successfully. 

  • Issue 32960:

    Interface “Autonegotiation” and “Speed” status might be displayed incorrectly on the Local Web UI for activated VMware SD-WAN Edges.

  • Issue 32981:

    Hard-coding speed and duplex on a DPDK-enabled port may require a VMware SD-WAN Edge reboot for the configurations to take effect as it requires disabling DPDK.

  • Issue 34254:

    When a Zscaler CSS is created and the Global Segment has FQDN/PSK settings configured, these settings are copied to Non-Global Segments to form IPsec tunnels to a Zscaler CSS.

  • Issue 35778:

    When there are multiple user-defined WAN links on a single interface, only one of those WAN links can have a GRE tunnel to Zscaler. 

    Workaround: Use a different interface for each WAN link that needs to build GRE tunnels to Zscaler.

  • Issue 35807:

    A DPDK routed interface will be disabled completely if the interface is disabled and re-enabled from the VMware SD-WAN Orchestrator. 

  • Issue 36923:

    Cluster name may not be updated properly in the NetFlow interface description for a VMware SD-WAN Edge which is connected to that Cluster as its Hub.

  • Issue 38682:

    A VMware SD-WAN Edge acting as a DHCP server on a DPDK-enabled interface may not properly generate “New Client Device" events for all connected clients.

  • Issue 38767:

    When a WAN overlay that has GRE tunnels to Zscaler configured is changed from auto-detect to user-defined, stale tunnels may remain until the next restart.

    Workaround: Restart the Edge to clear the stale tunnel.

  • Issue 39134:

    The System health statistic “CPU Percentage” may not be reported correctly on Monitor > Edge > System for the VMware SD-WAN Edge, and on Monitor > Gateways for the VMware SD-WAN Gateway.

    Workaround: Users should use handoff queue drops for monitoring Edge capacity not CPU percentage.

  • Issue 39608:

    The output of the Remote Diagnostic “Ping Test” may display invalid content briefly before showing the correct results.

  • Issue 39624:

    Ping through a subinterface may fail when the parent interface is configured with PPPoE.

  • Issue 39659:

    On a site configured for Enhanced High Availability, with one WAN link on each VMware SD-WAN Edge, when the standby Edge has only PPPoE connected and the active has only non-PPPoE connected, a split brain state (active/active) may be possible if the HA cable fails.

  • Issue 39753:

    Disabling Dynamic Branch-to-Branch VPN may cause existing flows currently being sent using Dynamic Branch-to-Branch to stall.

  • Issue 40096:

    If an activated VMware SD-WAN Edge 840 is rebooted, there is a chance an SFP module plugged into the Edge will stop passing traffic even though the link lights and the VMware SD-WAN Orchestrator will show the port as 'UP'. 

    Workaround: Unplug the SFP module and then replug it back into the port.

  • Issue 40421:

    Traceroute is not showing the path when passing through a VMware SD-WAN Edge with an interface configured as a switched port.

  • Issue 42278:

    For a specific type of peer misconfiguration, the VMware SD-WAN Gateway may continuously send IKE init messages to a Non-SD-WAN peer. This issue does not disrupt user traffic to the Gateway; however, the Gateway logs will be filled with IKE errors and this may obscure useful log entries.

  • Issue 42388:

    On a VMware SD-WAN Edge 540, an SFP port is not detected after disabling and reenabling the interface from the VMware SD-WAN Orchestrator.

  • Issue 42488:

    On a VMware SD-WAN Edge that has a switched port with VRRP enabled, if the cable is disconnected and the Edge Service is restarted, the LAN connected routes are advertised.

    Workaround: There is no workaround for this issue.

  • Issue 42872:

    Enabling Profile Isolation on a Hub profile where a Hub cluster is associated does not revoke the Hub routes from the routing information base (RIB).

  • Issue 43373:

    When the same BGP route is learnt from multiple VMware SD-WAN Edges, if this route is moved from preferred to eligible exit in the Overlay Flow Control, the Edge is not removed from the advertising list and continues to be advertised.

    Workaround: Enable distributed cost calculation on the VMware SD-WAN Orchestrator.

  • Issue 44526: For an enterprise where two different sites deploy their VMware SD-WAN Edges as Hubs while also using a high-availability topology, and each site uses the other Hub site as a Hub in its profile.  If one of the Hub sites triggers an HA failover, it may take up to 30 minutes for both Hub Edges to reestablish tunnels with each other. 

    On an HA failover, both Hub Edges try to initiate a tunnel with each other at the same time and neither replies to the peer, the packet exchange between both Hubs occurs, but IKE never succeeds. This leads to a deadlock that has been observed to take up to 30 minutes to resolve on its own. The issue is intermittent and does not occur after every HA failover. 

    Workaround: To prevent this issue from occurring, the customer should configure only one of the two HA Hub sites to use the other Hub site as a Hub for itself.  For example, where there are two HA Hub sites, Hub1 and Hub2, Hub1 could have Hub2 as a Hub for itself in its profile, but Hub2 must not use Hub1 as a Hub in its profile.

  • Issue 44995:

    OSPF routes are not revoked from VMware SD-WAN Gateways and VMware SD-WAN Spoke Edges when the routes are withdrawn from the Hub Cluster.

  • Issue 45189:

    With source LAN side NAT is configured, the traffic from a VMware SD-WAN Spoke Edge to a Hub Edge is allowed even without the static route configuration for the NAT subnet.

  • Issue 45302:

    In a VMware SD-WAN Hub Cluster, if one Hub loses connectivity for more than 5 minutes to all of the VMware SD-WAN Gateways common between itself and its assigned Spoke Edges, the Spokes may in rare conditions be unable to retain the hub routes after 5 minutes. The issue resolves itself when the Hub regains contact with the Gateways.

  • Issue 46053:

    BGP preference does not get auto-corrected for overlay routes when its neighbor is changed to an uplink neighbor.

    Workaround: An Edge Service Restart will correct this issue.

  • Issue 46137:

    A VMware SD-WAN Edge running 3.4.x software does not initiate a tunnel with AES-GCM encryption even if the Edge is configured for GCM.

  • Issue 46216:

    On a Non SD-WAN Destinations via Gateway or Edge where the peer is an AWS instance, when the peer initiates Phase-2 re-key, the Phase-1 IKE is also deleted and forces a re-key.  This means the tunnel is torn down and rebuilt, causing packet loss during the tunnel rebuild.

    Workaround: To avoid tunnel destruction, configure the Non SD-WAN Destinations via Gateway/Edge or CSS IPsec rekey timer to less than 60 minutes.  This prevents AWS from initiating the re-key.

  • Issue 46391:

    For a VMware SD-WAN Edge 3800, the SFP1 and SFP2 interfaces each have issues with Multi-Rate SFPs (i.e. 1/10G) and should not be used in those ports.

    Workaround: Please use single rate SFP's per the KB article VMware SD-WAN Supported SFP Module List (79270).  Multi-Rate SFPs may be used with SFP3 and SFP4.

  • Issue 46918:

    A VMware SD-WAN Spoke Edge using the 3.4.2 Release does not update the private network id of a Cluster Hub node properly.

  • Issue 47084:

    A VMware SD-WAN Hub Edge cannot establish more than 750 PIM (Protocol-Independent Multicast) neighbors when it has 4000 Spoke Edges attached.

  • Issue 47355:

    When the same route is learned via local underlay BGP, Hub BGP and/or statically configured on the Partner Gateway, the sorting order of the routes is incorrect with the Hub BGP being preferred over the underlay BGP.

  • Issue 47681:

    When a host on the LAN side of a VMware SD-WAN Edge uses the same IP as that Edge’s WAN interface, the connection from the LAN host to the WAN does not work.

  • Issue 47787:

    A VMware SD-WAN Spoke Edge configured with a backhaul business policy incorrectly sends traffic via the VMware SD-WAN Gateway path if that flow is initiated from the Hub Edge to that Spoke Edge.

  • Issue 48166:

    A VMware SD-WAN Virtual Edge on KVM is not supported when using a Ciena virtualization OS and the Edge will experience recurring Dataplane Service Failures.

  • Issue 48175:

    A VMware SD-WAN Edge running Release 3.4.2 will form an OSPF adjacency on a non-global segment if the non-global segment has an interface configured in the same IP range as an interface configured on the global segment

  • Issue 48502:

    In some scenarios, a VMware SD-WAN Hub Edge being used to backhaul internet traffic may experience a Dataplane Service Failure due the improper handling of backhaul return packets.

  • Issue 48530:

    VMware SD-WAN Edge 6x0 models do not perform autonegotiation for triple speed (10/100/1000 Mbps) copper SFP's.

    Workaround: Edge 520/540 supports triple speed copper SFPs but this model has been marked for End-of-Sale by Q1 2021.

  • Issue 48597: Multihop BGP neighborship does not stay up if one of the two paths to the peer goes down

    If there is a Multihop BGP neighborship with a peer to which there are multiple paths and one of them goes down, user will notice that the BGP neighborship goes down and does not come up using the other available path(s). This includes the Local IP-loopback neighborship case too.

    Workaround: There is no workaround for this issue.

  • Issue 48666:

    IPsec-fronted Gateway Path MTU calculation does not account for 61 Byte IPsec overhead, resulting in higher MTU advertisement to LAN client and subsequent IPsec packet fragmentation.

    Workaround: There is no workaround for this issue.

  • Issue 49172:

    A Policy Based NAT rule configured with the same NAT subnet for two different VMware SD-WAN Edges does not work.

  • Issue 50518:

    On a VMware SD-WAN Gateway where PKI is enabled, if >6000 PKI tunnels attempt to connect to the Gateway, the tunnels may not all come up because inbound SAs do not get deleted.

    Note: Tunnels using pre-shared key (PSK) authentication do not have this issue.

  • Issue 51036: ifSpeed reports a 0 value for operational VMware SD-WAN Edge interfaces when polling the Edge via SNMP.

    This is an expected behavior for DPDK enabled ports. Currently the only way to get speed values for DPDK enabled ports is by using the command "debug.py --dpdk_ports". But the SNMP module running on an Edge does not rely on this command to extract speed values for DPDK-enabled ports. SNMP only queries via the kernel interface, which unfortunately does not populate speed values for dpdk_ports.

  • Issue 51428: Multicast traffic loss may be observed on a site where the VMware SD-WAN Edge has a sub-interface configured with PIM.

    When a sub-interface configured with PIM is moved from a segment to another on the fly, pimd (the process that manages PIM) may restart and the site would experience intermittent multicast traffic loss.

    Workaround: Disable the sub-interface first, and then move the sub-interface to another segment. Once moved, re-enable the sub-interface.

  • Issue 52104: For an enterprise using a Hub/Spoke topology, existing flows are dropped on a VMware SD-WAN Spoke Edge when recovering from a Hub Edge failover for a given tuple.

    The sequence of events leads to this issue when a primary Hub Edge recovers from failover:
    1. When the primary Hub Edge goes down, the route is removed from the FIB for that primary Hub Edge while retaining the routes in the RIB.
    2. Existing flows will now switch to the secondary Hub Edge.
    3. When the primary Hub comes back up, a tunnel is immediately established between the primary Hub from the Spoke Edge.
    4. Routes in the RIB which were previously learned from the primary Hub via the Gateway are scanned and routes are installed in the FIB pointing to this primary Hub.
    5. Traffic will switch back to the primary Hub whereas the primary Hub would not have learned the routes from its BGP neighbor.
    6. This causes route lookup to match on the default route and return traffic is marked with a backhaul flag.
    7. The Spoke Edge is not expecting return traffic with a backhaul flag set and this leads to traffic being dropped. 

    Workaround: On the Hub Edge, run the Remote Diagnostic "Flush Flows" for the given tuple and traffic will be restored.

  • Issue 52483: If underlay accounting is enabled for an interface, the VMware SD-WAN Edge wrongly forwards the traffic back to the same interface instead of forwarding to the overlay.

    This behavior is caused by an issue with underlay accounting and a recursive route resolution.

    Workaround: Disable underlay accounting for the affected interface.

  • Issue 52955: DHCP decline is not sent from Edge and DHCP rebinding is not restarted after DAD failure in Stateful DHCP.

    If DHCPv6 server allocates an address which is detected as duplicate by the kernel during a DAD check then the DHCPv6 client does not send a decline. This will lead to traffic dropping as the interface address will be marked as DAD check failed and will not be used. This will not lead to any traffic looping in the network but traffic blackholing will be seen.

    Workaround: There is no workaround for this Issue.

  • Issue 53147: Non default hop limit values advertised in the router advertisements are not honored on the VMware SD-WAN Edge. The hop limit value of the tunnel is always set to 64.

    The default value of hop limit is 64. If the desire is to have a non-default hop limit value advertised through router advertisements, the Edge does not process the hop limit fields in the packet and the values remain as 64.

    Workaround: There is no workaround to this issue.

  • Issue 53219: After a VMware SD-WAN Hub Cluster rebalances, a few Spoke Edges may not have their RPF interface/IIF set properly.

    On the affected Spoke Edges, multicast traffic will be impacted. What happens is that after a cluster rebalance, some of the Spoke Edge fail to send a PIM join.

    Workaround: This issue will persist until the affected Spoke Edge has an Edge Service restart.

  • Issue 53337: Packet drops may be observed with an AWS instance of a VMware SD-WAN Gateway when the throughput is above 3200 Mbps.

    When traffic exceeds a throughput above 3200 Mbps and a packet size of 1300 bytes, packets drops are observed at RX and at IPv4 BH handoff.

    Workaround: There is no workaround for this issue.

  • Issue 53359: BGP/BFD session may fail during some DDoS attack scenarios.

    If traffic is flooded from the client connected to the routed interface to the LAN client, the BGP/BFD session can fail. Also when real-time high priority traffic is flooded to the overlay destination, the BGP/BFD session can fail.

    Workaround: There is no workaround for this issue.

  • Issue 53687: When a VMWare SD-WAN Spoke Edge has a preference for a IPv6/v4 tunnel, the non-preferred v4/v6 tunnels MTU will influence the MTU seen in preferred tunnels too.

    An Edge (Spoke or Hub) maintains a system level MTU, which is the minimum of all the link MTU's and this MTU is exchanged as the advertised MTU. Since a non-preferred link's MTU can still be considered for determining the system level MTU, a lower MTU can be advertised than the actual path MTU.

    Workaround: There is no workaround for this issue.

  • Issue 53830: On a VMware SD-WAN Edge, some of the routes in BGP view may not have the correct preference and advertise values when DCC flag is enabled causing incorrect sorting order in the Edge's FIB.

    When Distributed Cost Calculation (DCC) is enabled in a scaled scenario with a large number of routes on an Edge, when looking at an Edge diagnostic bundle for the log bgp_view some of the routes may not be correctly updated with the preference and advertise values.  This issue, if found at all, would be a found in a few Edges as part of a large enterprise (100+ Spoke Edges connected to either Hub Edges or Hub Clusters).  

    Workaround: This issue can be addressed by either relearning the underlay BGP routes or performing a "Refresh" option on the OFC page of the VMware SD-WAN Orchestrator for the affected routes. Please note that performing a "Refresh" of a route would re-learn the routes from all the Edges in the enterprise.

  • Issue 53934: In an enterprise where a VMware SD-WAN Hub Cluster is configured, if the primary Hub has Multihop BGP neighborships on the LAN side, the customer may experience traffic drops on a Spoke Edge when there is a LAN side failure or when BGP is disabled on all segments.

    In a Hub cluster, the primary Hub has Multihop BGP neighborship with a peer device to learn routes. If the physical interface on the Hub by which BGP neighborship is established, goes down, then BGP LAN routes may not become zero despite BGP view being empty. This may cause Hub Cluster rebalancing to not happen. The issue may also be observed when BGP is disabled for all segments and when there are one or more Multihop BGP neighborships.

    Workaround: Restart the Hub which had the LAN-side failure (or BGP disabled).

  • Issue 54099: Fragmented IPv6 packets will be dropped by a VMware SD-WAN Edge.

    Any fragmented IPv6 packet is going to get dropped by an Edge.

    Workaround: There is no workaround for this issue.

  • Issue 54378: IPv6 static address enabled interface duplicate address detection (DAD) check fails due to NA drops. 

    Static address DAD check will not happen and if there is a duplicate address in the network for configured static address then that will not be detected.

    Workaround: There is no workaround for this issue.

  • Issue 54536: Duplicate address detection (DAD) check not triggered after a VMware SD-WAN Edge reboot.

     If there is a duplicate address in the network, then that would not be detected if this DAD check is not performed after reboot.

    Workaround: There is no workaround for this issue.

  • Issue 54687: DHCPv6 solicit message is not sent by a VMware SD-WAN Edge after the server is provided with a configuration of T1 values greater than T2.

    If a DHCPv6 server initially provides T1 value greater than T2, then the Edge does not accept the provided prefix, but even after this configuration has been corrected on the server, the Edge will not send a DHCPv6 solicit message after attempting three times.  At that point only when the Edge's Dataplane Service is restarted will the issue get cleared.

    Workaround: Restart the Edge's service.

  • Issue 54731: Renew messages are sent with a high frequency until the rebind time (t2) is reached when a user changes the value of the IPv6 address range in the server.

    When an IP address which has been assigned to a VMware SD-WAN Edge is removed from the valid range of addresses provided by a server, the client keeps sending renew messages to the server until T2 time is reached. This may result in a customer user observing a large quantity of DHCPv6 traffic.

    Workaround: There is no workaround for this issue.

  • Issue 56218: For a customer site deployed with a High-Availability topology or where HA has just been enabled, when the Edges are upgrade from 3.2.x to 3.4.x, the Standby Edge may go down.

    When HA is enabled or the HA Edges are upgraded from 3.2.2 to 3.4.x after a WAN setting is configured using the Local UI, the HA interface (e.g. LAN1 or GE1 depending on the Edge model) will be removed from the Standby Edge and HA status will be set to HA_FAILED on the VMware SD-WAN Orchestrator.

    Workaround: Reboot the Standby Edge to recover it

  • Issue 56454: Configure both IPv4 and IPv6 links as auto-discovered links on an interface and then have tunnels formed through the non-preferred link as well. The link stats do not display consolidated information of a IPv4 and IPv6 link.

    When an interface has both IPv4 and IPv6 overlays configured as auto-discovered overlays and tunnels formed over both links, the links stats reflect only the status of the preferred link. The traffic information or the status of the non-preferred link is not correctly reflected. As a result, the statistics seen for the link on the Edge > Monitoring page which include the bandwidth and throughput should be used as a guideline to measure the performance of the tunnels formed over the preferred IP address family only.

    Workaround: There is no workaround for this issue.

  • Issue 57957: If a DPDK interface is changed from Autonegotiate=on to Autonegotiate=off, the Edge unloads the KNI driver and loads the Linux driver for that interface during the Edge Service restart sequence (from vc_dpdk.py).

    After loading up the new Linux interface and nameif'ing it, vc_dpdk.py also needs to invoke "set_interface_neg.py" to apply the auto-negotiation settings. However, because of the new auto-negotiation settings and the Linux driver is reloaded, the bare metal interface is no longer under DPDK control.

  • Issue 58453: Some Office365 packets are misclassified as SSL packets by the VMware SD-WAN Edge.

    The VMware SD-WAN Deep Packet Inspection (DPI) engine is sometimes misclassifying packets that should be classified as Office365 as SSL instead.  The impact is that these flows will be treated as SSL flows versus Office365 flows and that may mean they are treated with less priority, impacting the user experience.

  • Issue 59970: Customer will observe traffic drop from a VMware SD-WAN Edge to the datacenter through a Zscaler IaaS, when switching from a primary to a secondary Gateway.

    When the primary Gateway goes down and the Orchestrator switches to the secondary Gateway, the current traffic flow reverses path from a Zscaler Enforcement Noted (ZEN) does not work.

    Workaround: The workaround would be to reinitiate all traffic flow. The Zscaler is notified about this issue and they confirmed that reverse traffic path is not working properly on their side.

  • Issue 61882: When a security parameter configuration change is made (e.g., SA lifetime change) from the VMware SD-WAN Orchestrator, the customer may observe traffic drop for a period of time.

    This has been seen on a large-scale deployment with +1000 Edges in a Hub/Spoke topology. If security parameters (lifetime, cryptography algorithms, authentication mode) are changed this will bring down current tunnels, which will then rebuild. In a large-scale deployment this can cause issues with traffic stability. The responder side (Hub Edge) may not be able to handle all the tunnels in time and this can cause a traffic drop. Eventually tunnels will establish, but it can take some time based on the existing number of tunnels. 

    Workaround: It is recommended to perform configuration changes in a maintenance window, since recovery time is unknown based on the existing number of tunnels.

  • Issue 62552: A site may experience intermittent periods of high packet loss and connectivity issues.

    This is caused by the API that checks for ARP resolution telling the Edge there is a successful ARP resolution for a device while delivering a MAC address of 00:00:00:00.  This address is kept in the ARP cache and any packets intended for the device where the MAC is listed as zero are dropped.  In this issue, many such instances of successful ARP's with zero MAC addresses are delivered causing high packet loss and connectivity issues.

    Note: Both issue 60130 and this issue have the same underlying behavior and cause but the expected fixes for each ticket differ. 60130 will have a defensive workaround fix while 62552 will have a complete fix that prevents any recurrence of this issue.

    Workaround: There is no workaround for this issue.

  • Issue 62685: If LAN side NAT is configured with the same outside IP for different LAN subnets with NAT type as source, traffic destined for the Cloud will not work.

    For the outside IP used in LAN side NAT rules, we configure a static route and advertise it to the remote branches. For the return traffic to be routed to the correct the LAN subnet, route lookup should be done based on the Inside IP configured in the LAN side NAT rule instead of the next hop in the static route. But for the return traffic from cloud, the route lookup is done based on the next hop in the static route and traffic can get routed to the wrong LAN subnet.

    Workaround: Use a different Outside IP for different LAN subnets.

  • Issue 62725: A VMware SD-WAN Edge in a network which uses BGP may experience high memory usage under certain rare conditions.

    If an Edge learns a BGP route with a next hop IP address which is different from the peer IP address, the next hop will be tracked for reachability by the Edge's Next Hop Tracking (NHT) module. If BGP is then disabled when the tracked IP address is unreachable on the Edge, the tracked NHT entry may not get deleted. In rare cases where there are a lot of stale NHT entries, high Edge memory usage can be seen.

    Workaround: Reboot the Edge to delete the memory leaking NHT entries. 

  • Issue 62897: The debug command tcpdump does not work properly on a VMware SD-WAN Gateway.

    Execute tcpdump command on eth0 or eth1 interface of Gateway, the output is not correct. tcpdump.sh and vctcdump are also not working. An attempted fix was attempted, an AppArmor complain profile for vctcpdump (based on the tcpdump profile) that allows tcpdump to inherit vctcpdump's confinement was added, but tcpdump still is not working. Essentially AppArmor causes stdout to stop working for tcpdump. This is a known issue with AppArmor.

    Workaround: "pipe tcpdump output to cat. e.g. tcpdump -nnplei eth0 | cat"

  • Issue 63125: When MTU is increased for any interface/link on a VMware SD-WAN Hub Edge, it will not be reflected in the path MTU on the Spoke Edge (for the paths with that Hub Edge).

    If the user increases the MTU of an interface or link on a Hub, the Spoke Edge path does not pick up the changed MTU setting.

    Workaround: Reboot the Spoke Edge, the increased MTU will be reflected in the path MTU on the Spoke.

  • Issue 63362: On a site that uses an Enhanced High Availability topology, a DHCP/PPPoE enabled interface stops sending traffic after the VMware SD-WAN Standby Edge is either rebooted or power cycled.

    In an Enhanced HA setup, if DHCP/PPPoE is enabled on a proxy interface (i.e., HA link state set to USE_PEER), it fails to get the address from the server after the Standby Edge reboots or power cycles.

    Workaround: Either change the dynamic address to a static address type or do a forced HA failover to get the IP address from the server.

  • Issue 65885: For a customer who has deployed a Non SD-WAN Destination (NSD) via Gateway and is using a redundant Gateway configuration, if the Primary Gateway goes down, there is a race condition in which before the PG-BGP peership on the Primary Gateway comes up, the NSD already advertises the legacy routes learned via the Redundant Gateway to the Primary Gateway.

    Since BGP multipath is not supported, the route which the Primary Gateway was supposed to learn over the handoff interface appears to be learned as the datacenter from the NSD. Though these are valid routes, this should not happen since traffic should go via the NSD > Primary Gateway > Handoff Interface when the Primary Gateway is up. The impact to the customer is not in terms of performance as the traffic still reaches the destination, but it negatively impacts a customer's network management. The customer expects traffic to come via one route and will install QoS policies to manage it, but the traffic is going via another route not expected in the QoS policy.

    Workaround: The customer should filter outbound routes on the Non SD-WAN Destination via Gateway so that it does not advertise a route learned via a Redundant Gateway to a Primary Gateway.

Orchestrator Known Issues
  • Issue 19566:

    After High Availability failover, the serial number of the standby VMware SD-WAN Edge may be shown as the active serial number in the Orchestrator.

  • Issue 21342:

    When assigning Partner Gateways per-segment, the proper list of Gateway Assignments may not show under the Operator option "View" Gateways on the VMware SD-WAN Edge monitoring list.

  • Issue 24269:

    Monitor > Transport > Loss not graphing observed WAN link loss while QoE graphs do reflect this loss. 

  • Issue 25932:

    The VMware SD-WAN Orchestrator allows VMware SD-WAN Gateways to be removed from the Gateway Pool even when they are in use.

  • Issue 32335:

    The ‘End User Service Agreement’ (EUSA) page throws an error when a user is trying to accept the agreement.

    Workaround: Ensure no leading or trailing spaces are found in Enterprise Name.

  • Issue 32435:

    A VMware SD-WAN Edge override for a policy-based NAT configuration is permitted for tuples which are already configured at the profile level and vice versa.

  • Issue 32856:

    Though a business policy is configured to use the Hub cluster to backhaul internet traffic, the user can unselect the Hub cluster from a profile on a VMware SD-WAN Orchestrator that has been upgraded from Release 3.2.1 to Release 3.3.x.

  • Issue 32913:

    After Enabling High Availability, Multicast details for the VMware SD-WAN Edge are not displayed on the Monitoring Page. A failover resolves the issue.

  • Issue 34828:

    Traffic cannot pass between a VMware SD-WAN Spoke Edge using release 2.x and a Hub Edge using release 3.3.1.

  • Issue 35658:

    When a VMware SD-WAN Edge is moved from one profile to another which has a different CSS setting (e.g. IPsec in profile1 to GRE in profile2), the Edge level CSS settings will continue to use the previous CSS settings (e.g. IPsec versus GRE). 

    Workaround: Disable and then reenable GRE at the Edge level to resolve the issue.

  • Issue 35667:

    When a VMware SD-WAN Edge is moved from one profile to another profile which has the same CSS setting but a different GRE CSS name (the same endpoints), some GRE tunnels will not show in monitoring.

    Workaround: Disable and then reenable GRE at the Edge level to resolve the issue.

  • Issue 36665:

    If the VMware SD-WAN Orchestrator cannot reach the internet, user interface pages that require accessing the Google Maps API may fail to load entirely.

  • Issue 38056:

    The Edge-Licensing export.csv file does not show region data.

  • Issue 38843:

    When pushing an application map, there is no Operator event, and the Edge event is of limited utility.

  • Issue 39633:

    The Super Gateway hyper link does not work after a user assigns the Alternate Gateway as the Super Gateway.

  • Issue 39790:

    The VMware SD-WAN Orchestrator allows a user to configure a VMware SD-WAN Edge’s routed interface to have greater than the supported 32 subinterfaces, creating the risk that a user can configure 33 or more subinterfaces on an interface which would cause a Dataplane Service Failure for the Edge.

  • Issue 40341:

    Though the Skype application is properly categorized on the backend as Real Time traffic, when editing the Skype Business Policy on the VMware SD-WAN Orchestrator, the Service Class may erroneously display “Transactional”.

  • Issue 41691:

    User cannot change the 'Number of addresses' field although the DHCP pool is not exhausted on the Configure > Edge > Device page.

  • Issue 43276:

    User cannot change the Segment type when a VMware SD-WAN Edge or Profile has a partner gateway configured.

  • Issue 47269:

    The VMware SD-WAN 510-LTE interface may appear for Edge models that do not support an LTE interface.

  • Issue 47713:

    If a Business Policy Rule is configured while Cloud VPN is disabled, the NAT configuration must be reconfigured upon enabling Cloud VPN.

  • Issue 47820:

    If a VLAN is configured with DHCP disabled at the Profile level, while also having an Edge Override for this VLAN on that Edge with DHCP enabled, and there is an entry for the DNS server field set to none (no IP configured), the user will be unable to make any changed on the Configure > Edge > Device page and will get an error message of ‘invalid IP address []’ that does not explain or point to the actual problem.

  • Issue 48085:

    The VMware SD-WAN Orchestrator allows a user to delete a VLAN which is associated with an interface.

  • Issue 48737:

    On a VMware SD-WAN Orchestrator which is using the Release 4.0.0 new user interface, If a user is on a Monitor page and changes the Start & End time interval and then navigates between tabs, the Orchestrator does not update Start & End interval time to the new values.

  • Issue 49225:

    VMware SD-WAN Orchestrator does not enforce a limit of 32 total VLANs.

  • Issue 49790:

    When a VMware SD-WAN Edge is activated to Release 4.0.0, the activation is posted twice in Events.

    Workaround: Ignore the duplicate event.

  • Issue 50531:

    When two Operators of differing privileges use the same browser window when accessing the New UI on a 4.0.0 Release version of the VMware SD-WAN Orchestrator, and the Operator with lesser privileges tries to login after the Operator with higher privileges, that lesser privileged Operator will observe multiple errors stating that the "user does not have privilege".

    Note: There is no escalation in privileges for the Operator with lower privileges, only the display of error messages.

    Workaround: The next operator may refresh that page prior to logging in to prevent seeing the errors, or each Operator may use different browser windows to avoid this display issue.

  • Issue 51722: On the Release 4.0.0 VMware SD-WAN Orchestrator, the time range selector is no greater than two weeks for any statistic in the Monitor > Edge tabs.

    The time range selector does not show options greater than "Past 2 Weeks" in Monitor > Edge tabs even if the retention period for a set of statistics is much longer than 2 weeks.  For example, flow and link statistics are retained for 365 days by default (which is configurable), while path statistics are retained only for 2 weeks by default (also configurable).  This issue is making all monitor tabs conform to the lowest retained type of statistic versus allowing a user to select a time period that is consistent with the retention period for that statistic.

    Workaround: A user may use the "Custom" option in the time range selector to see data for more than 2 weeks.

check-circle-line exclamation-circle-line close-line
Scroll to top icon