Updated March 24th, 2022
VMware SD-WAN Orchestrator Version R401-20211218-GA
Check regularly for additions and updates to these release notes.
What's in the Release NotesThe release notes cover the following topics:
This release is recommended for all customers who require the features and functionality first made available in Release 4.0.0, as well as those customers impacted by the issues listed below which have been resolved since Release 4.0.0.
Release 4.0.1 Orchestrators, Gateways, and Hub Edges support all previous VMware SD-WAN Edge versions greater than or equal to Release 3.0.0
Note: this means releases prior to 3.0.0 are not supported.
The following interoperability combinations were explicitly tested:
Note: Interoperability with Release 3.x was explicitly tested as part of Release 4.0.0 testing, and there are no protocol changes between Release 4.0.0 and Release 4.0.1 that required explicitly re-testing more combinations than is shown in the above table.
Release 3.x did not properly support AES-256-GCM, which meant that customers using AES-256 were always using their Edges with GCM disabled (AES-256-CBC). If a customer is using AES-256, they must explicitly disable GCM from the Orchestrator prior to upgrading their Edges to a 4.x Release. Once all their Edges are running a 4.x release, the customer may choose between AES-256-GCM and AES-256-CBC.
Doing the above will prevent interoperability issues between 3.x and 4.x Edges.
BGPv4 Filter Configuration Delimiter Change for AS-PATH Prepending
Through Release 3.x, the VMware SD-WAN BGPv4 filter configuration for AS-PATH prepending supported both comma and space based delimiters. However, beginning in Release 4.0.0 and forward, VMware SD-WAN will only support a space based delimiter in an AS-Path prepending configuration.
Customers upgrading from 3.x to 4.x need to edit their AS-PATH prepending configurations to "replace commas with spaces" prior to upgrade to avoid incorrect BGP best route selection.
Beginning in Release 4.0.0, Edge Licensing is enabled by default and it is mandatory for a user to assign an Edge license type when creating a new VMware SD-WAN Edge. This requirement helps VMware SD-WAN to track customer subscriptions and simplifies and standardizes the Edge activation report sent by partners. VMware does not enforce the license parameters and assigning an Edge license type does not change the behavior of the Edge in any way.
To learn more, please consult the Edge Licensing documentation.
Limitation When Disabling Autonegotiation on VMware SD-WAN Edge Models 520, 540, 620, 640, 680, 3400, 3800, and 3810
When a user disables autonegotiation to hardcode speed and duplex on ports GE1 - GE4 on a VMware SD-WAN Edge model 620, 640 or 680; on ports GE3 or GE4 on an Edge 3400, 3800, or 3810; or on an Edge 520/540 when an SFP with a copper interface is used on ports SFP1 or SFP2, the user may find that even after a reboot the link does not come up.
This is caused by each of the listed Edge models using the Intel Ethernet Controller i350, which has a limitation that when autonegotiation is not used on both sides of the link, it is not able to dynamically detect the appropriate wires to transmit and receive on (auto-MDIX). If both sides of the connection are transmitting and receiving on the same wires, the link will not be detected.
If the peer side also does not support auto-MDIX without autonegotiation, and the link does not come up with a straight cable, then a crossover Ethernet cable will be needed to bring the link up.
For more information please see the KB article Limitation When Disabling Autonegotiation on VMware SD-WAN Edge Models 520, 540, 620, 640, 680, 3400, 3800, and 3810 (87208).
November 12, 2020. First Edition.
November 23, 2020. Second Edition.
- Added a new Orchestrator build: R401-20201118-GA; added an Orchestrator Resolved section for this build which includes the three new tickets resolved in this build: #52427, #52689, and #52762.
November 25, 2020. Third Edition.
- Added a new Gateway hotfix build: R401-20201124-GA-53090; added a Gateway Resolved section for this build which includes the new ticket resolved in this build: #53090.
November 26, 2020. Fourth Edition.
- Added the section "Important Notes"; to that section added a note on "Edge Licensing" regarding two important behavior changes on a 4.0.0 and higher Orchestrator and that had been omitted from previous editions of the 4.0.0 and 4.0.1 Release Notes.
December 8th, 2020. Fifth Edition.
- Added a new Orchestrator build: R401-20201202-GA; added an Orchestrator Resolved section for this build which includes the three new tickets resolved in this build: #51454, #52491, and #52805.
- Added a fixed ticket #52488 to the Orchestrator build 20201118. This ticket had been omitted from the Second Edition Notes and later but the fix was indeed a part of this build.
- Added a new ticket to the Edge/Gateway Open Issues section: #53219
December 16th, 2020. Sixth Edition.
- Added a new Orchestrator build: R401-20201215-GA; added an Orchestrator Resolved section for this build which includes the five new tickets resolved in this build: #51517, #52491, #52682, #53411, #53558
- Amended the description for #52491 of the R401-20201202-GA build to say that the fix in this build is only partial and that the full fix is in the R401-20201215-GA build.
- Added a new ticket to Orchestrator Known Issues: #53652.
September 16th, 2021. Seventh Edition.
- Added to Important Notes the Note: BGPv4 Filter Configuration Delimiter Change for AS-PATH Prepending.
December 21st, 2021. Eighth Edition.
- Added a new Orchestrator build R401-20211218-GA to Orchestrator Resolved Issues. This Orchestrator build remediates CVE-2021-44228, the Apache Log4j vulnerability, by updating to Log4j version 2.16.0. For more information on the Apache Log4j vulnerability, please consult the VMware Security Advisory VMSA-2021-0028.5.
- Added to Important Notes the Note: Limitation When Disabling Autonegotiation on VMware SD-WAN Edge Models 520, 540, 620, 640, 680, 3400, 3800, and 3810. This note covers an issue that may be encountered when configuring a forced speed on some Ethernet ports of the listed Edge models.
March 24th, 2022, Ninth Edition
- Added Issue #84825, to the Edge/Gateway Known Issues section.
The resolved issues are grouped as follows.Gateway Resolved Issue
Resolved in Gateway Release R401-20201124-GA-53090
The below issue is resolved since Gateway version R401-20201110
- Fixed Issue 53090: Tunnels to a Non SD-WAN Site via Gateway may fail after an IKE Phase 1 rekey or when a tunnel goes down and IKE Phase 1 must be reinitiated.
When the VMware SD-WAN Gateway initiates tunnels to non SD-WAN sites, the security association (SA) involves a source and destination port. Once there is an SA, both these ports need to be populated correctly to ensure that IKE Phase 1 SA is recreated properly. As referenced in the VMware SD-WAN Release 4.0.0 Release Notes, VMware SD-WAN Release 4.0.0 replaced the IPsec libraries with FIPS 140-2 compliant IPsec libraries. As part of this replacement, the SA destination port was inadvertently set incorrectly on the second and subsequent negotiation attempts for a given peer.
For some non SD-WAN sites, the peer discards this invalid port and returns a proper Traffic Selector (TS). For peer sites which do not correct this, the peer responds with the invalid port in the TS which the VMware SD-WAN Gateway rejects and this results in Gateway tunnels failing at rekey or when a tunnel goes down and IKE Phase 1 must be reinitiated.
A customer is more likely to encounter this issue if the Non SD-WAN Site is misconfigured (e.g. the site is configured to use IKEv1 when the peer site uses IKEv2).
Resolved in Edge and Gateway Release R401-20201110
The below issues have been resolved since Edge and Gateway version R400-20201002
- Fixed Issue 33195: PIM joins may fail on the VMware SD-WAN Hub when it has more than 1280 multicast neighbors.
In a scale environment of 1280 multicast customer sites or more, even though PIM JOIN is successful for the spokes, their logical IDs are missing in HUB's MCR table.
- Fixed Issue 46482: If a site using VMware SD-WAN Edge 540’s configured in High-Availability is upgraded to Edge software release 3.4.1, 3.4.2, or 3.4.3, the Standby Edge will no longer be detected, and the VMware SD-WAN Orchestrator will display this site’s HA status as “Standby Failed”.
When the Edge is upgraded to Release 3.4.1, 3.4.2, or 3.4.3, the HA Interface (LAN1 or GE1 depending on model) is configured as a trunk port, causing the HA Interface to fail to come up and this causes the Standby Edge to no longer be detected.
- Fixed Issue 47084: A VMware SD-WAN Hub Edge cannot establish more than 750 PIM neighbors with 4000 Spoke Edges.
PIM neighbors are not scaleable to the supported limit (~1600) in a scenario where the 4000 Spoke Edges are split into two configuration profiles and in which one profile has multicast enabled, and the other has multicast disabled.
- Fixed Issue 47774: In rare cases, a VMware SD-WAN Edge may experience a Dataplane Service Failure when a route is deleted from the overlay and this same route is simultaneously added to the underlay.
This issue was discovered during a stress test and has not been observed in the field.
- Fixed Issue 48011: For an enterprise with a Hub/Spoke topology using multicast, if a Rendezvous Point (RP) becomes unavailable, the VMware SD-WAN Edge's PIM entry for that RP remains resolved.
The PIM registered prefix for next-hop tracking goes down on the VMware SD-WAN Edge and the PIM process is not notified about that by the Edge's dataplane service.
- Fixed Issue 48291: IGMP query and PIM hello messages are not pushed to a VMware SD-WAN Edge based on the configured values.
The configured IGMP query timer and PIM hello timer as configured on the VMware SD-WAN Orchestrator is not pushed to the Edge.
- Fixed Issue 49092: A VMware SD-WAN Edge may experience an Edge Dataplane Failure when the Edge receives transit flow traffic via the overlay and it switches to underlay before a flow entry is created.
In this scenario, multiple micro flow entries are created for the same flow. When the flow is flushed, only one of the micro flow entries is removed and when the Edge receives the same transit flow, the flow lookup returns the stale micro flow which triggers the Edge Dataplane Service Failure.
- Issue 50433: When a secondary IP address is deleted from a routed interface on a VMware SD-WAN Edge, the corresponding route may not be removed from the peer Edges.
When a secondary IP is configured on the Edge, a connected route is installed locally and the route is distributed to the peer Edges. When the secondary IP is deleted, the connected route is removed locally but the route remains in the routing table of the peer Edges.
- Fixed Issue 50067: NAT is skipped for the traffic routed via subinterface when "NAT direct" is enabled on the subinterface and disabled on the parent interface.
The "NAT direct" configuration of the parent interface is always checked to apply NAT for the traffic routed via the sub-interface. The sub-interface "NAT direct" configuration has no effect on the traffic and the traffic goes out without NAT if "NAT direct" is enabled on sub-interface and disabled on the parent interface.
- Fixed Issue 50231: For a customer whose enterprise uses the special MGMT-IP build for their VMware SD-WAN Edges (i.e., where the Edge Management IP is enabled for customers with requirement for global segment loopback), the Edge Management IP Address is not redistributed to BGP/OSPF.
Local/remote Management IP of VMware SD-WAN Edges is not being redistributed into underlay BGP/OSPF protocols. This will affect customers that rely on reachability to Edge's Management IP for monitoring purposes.
- Fixed Issue 50399: The VMware SD-WAN Gateway does not check the IKE policy configured for a Non SD-WAN Site via Gateway versus the incoming IKE version from the peer site and, when confronted with an IKE mismatch, automatically overwrite the VMware IKE version to match the peer side IKE version. As a result, both the Gateway and VMware SD-WAN Edge are processing packets for a tunnel with an IKE version mismatch.
If a Gateway is configured with IKEv1 Generic profile it will accept IKEv2 packets and process it. This issue would also happen for a IKEv2 generic profile. The generated SA will cause confusion because of the mismatch and traffic will fail.
- Fixed Issue 50494: In extremely rare instances, a VMware SD-WAN Gateway may experience a Dataplane Service failure during a tunnel tear down/rebuild (i.e., “flap”) as a result of a race condition in accessing data in an internal routing structure.
In these rare instances, the memory used by route messages may get filled with invalid data causing the route message parsing to fail and stop. This causes the Gateway Dataplane Service Failure with a resulting service restart that fully recovers the Gateway.
- Fixed Issue 50835: A site using a Hub and Spoke topology where all the VMware SD-WAN Edges are hardware (versus virtual) and also uses a Non SD-WAN Destination via Gateway, if Spoke Edge traffic destined for the datacenter site is backhauled through the Hub Edge before being sent to the datacenter, that traffic will fail.
This is caused by having the wrong packet encapsulation for IPsec traffic destined for the Non SD-WAN Destination via Gateway. On a Hub Edge that is hardware based and has a Non SD-WAN Destination via Gateway configured, the traffic initiated from the Spoke Edge which traverses through the Hub Edge IPsec tunnel to the Non SD-WAN Destination is encapsulated with the wrong offset on the Hub Edge side. All IPsec tunnel packets will be dropped on the Non SD-WAN Destination side.
- Fixed Issue 50841: For a VMware SD-WAN Edge where stateful firewall is enabled, if tunnels to other sites are torn down and then rebuilt in rapid succession (i.e., “flap”), some packets may leak.
The risk for a customer is that once a large number of packets have been leaked, the Edge will stop forwarding all traffic and thus lose connection to the VMware SD-WAN Orchestrator and other Edges in the network until the affected Edge's service is restarted.
- Fixed Issue 50899: A newly configured FQDN for an existing Cloud Security Service Site (CSS) does not immediately take effect.
For an existing CSS configuration, when a FQDN is added, it does not get updated on the Edge and the CSS tunnel does not use the newly added FQDN. As a result a user may observe the tunnel to the CSS peer site to be failing even though the expectation is that the FQDN is valid and being applied.
- Fixed Issue 50929: In rare cases where a VMware SD-WAN Edge has an extremely high number of TCP and UDP flows, there are unnecessary system level logs that fill up the filesystem causing the Edge's to experience a Dataplane Service Failure.
While rare, this issue is more likely to impact entry level Edge models like the 500, 510, 520, and 610 which have less RAM and smaller SSD's. Because these system log files completely fill up the Edge's file system the Edge will likely experience three Dataplane Service Failure's in succession (this will be seen in the Events) and need to be restarted by a user.
- Fixed Issue 50942: When a VMware SD-WAN Gateway is upgraded to Release 3.4.4 or 4.0.0 it may experience Dataplane Service Failures multiple times and become disabled.
When the Gateway is upgraded, the Gateway's dataplane service must encode and decode the BGP route attributes between itself and the BGP process and in this ticket that effort is causing the service to fail. The Gateway would need to be restarted to clear this issue.
- Fixed Issue 50961: On a site using a High-Availability topology, when a user configures new credentials for the VMware SD-WAN Edge's Local User Interface, the configuration is not being applied to the Standby Edge.
Because the Standby Edge does not have the new Local UI credentials, if there is an HA failover that promotes the Standby Edge to Active, a local user trying to log into the Edge with the new credentials will not be successful.
- Fixed Issue 51010: When stateful firewall is enabled, a TCP session that is initiated using the same ports as a session closed within the past 4-minutes (RFC 5382 timeout period) will be dropped.
The existing stateful firewall implementation does not allow SYN to reopen a session, which is a behavior other leading firewalls do allow.
- Fixed Issue 51506: For a site using Enhanced High-Availability topology, if the Active Edge loses all WAN connectivity while the Standby Edge loses all LAN connectivity, HA Failover does not occur and the site has no connectivity.
In an Enhanced HA deployment, the Standby Edge must be able to communicate with the Active Edge through its LAN ports for Enhanced HA to work properly. If the Active Edge loses WAN connectivity it would be expected to tunnel through the Standby Edge to use its WAN connectivity and allow the site to continue passing traffic. In this scenario the Active Edge cannot reach the internet on its WAN link(s) but also cannot reach the Standby Edge's still working WAN links while also not being able to initiate a failover. The result is a site that is completely down and unable to pass traffic.
- Fixed Issue 51291: A VMware SD-WAN Edge 3400 or 3800 that is deployed in Japan may lock up and reboot spontaneously.
The Edge 3400 and 3800 have an incorrect voltage warning threshold (100V) set in the system's Baseboard Management Controller (BMC), which happens to match the 100V electrical supply in Japan. The result for an Edge 3400 or 3800 in this region is a continuous series of power supply alarms and if the volume of alarms is sufficiently frequent, the Edge will lock up and reboot.
Orchestrator version R401-20211218-GA
Orchestrator version R401-20211218-GA was released on 12-20-2021. This Orchestrator build remediates CVE-2021-44228, the Apache Log4j vulnerability, by updating to Log4j version 2.16.0. For more information on the Apache Log4j vulnerability, please consult the VMware Security Advisory VMSA-2021-0028.5.
Resolved in Version R401-20201215-GA
The below issues have been resolved since Orchestrator version R401-20201202-GA.
- Fixed Issue 51517: When editing an application map using the Application Map Editor, a custom application may show a display name of "Unknown virtual protocol".
When a custom application is created and saved, and a user navigates to a business policy screen or to the Edge Monitoring > Applications page, and come back and check the display name of the custom app in Application Map Editor, that screen shows the display name as "Unknown virtual protocol" for all the custom apps. The .json entry for the application map is correct, but the Orchestrator reverts back to the fallback name “Unknown Virtual protocol”.
- Fixed Issue 52491: When a new port forwarding rule is added in a VMware SD-WAN Orchestrator, the configuration cannot be saved.
This issue occurs when a segment is mapped to a port forwarding rule after deleting a few segments. The user is unable to save a firewall configuration with error "Cannot read property 'logicalId' of undefined".
- Fixed Issue 52682: For a VMware SD-WAN Orchestrator deployed in a Disaster Recovery (DR) topology, the Standby Orchestrator may continually attempt to copy reports and Orchestrator diagnostic bundles it has already copied from active VMware SD-WAN Edges and Gateways and both will continually report receipt of "new managementPlane configuration" from the Standby Orchestrator.
Edges and Gateways report frequent receipt of "new managementPlane configurations" and the Orchestrator disaster recovery will repeatedly go through a cycle of failing to synchronize, and then synchronizing due to the Standby Orchestrator continuously attempting to open SSH connections to active to copy files it has already copied.
- Fixed Issue 53411: When configuring an VMware SD-WAN Edge interface with PPPOE addressing type, the "username" configured is seen correctly through either the Orchestrator UI or an API, but propagated to the Edge with the value "null".
This is caused due to a part of code using "name" instead of "username" when picking attributes from addressing type object to be added as part of the composite configuration sent down to the Edge. As the "name" key does not exist on the object, "username" is added as "null".
- Fixed Issue 53558: When looking at various Edge > Monitoring pages, there may be large sections of time where no statistical data is displayed.
Some of the incoming statistics had an unexpected field value (specifically, negative flow counts). This caused the statistics collection process to throw an error and this, in turn, caused a slower statistic collection rate to the point that collections halted altogether for sometimes large periods of time, and no statistical data would output to the relevant Monitoring pages on the Orchestrator UI.
Resolved in Version R401-20201202-GA
The below issues have been resolved since Orchestrator version R401-20201118-GA.
- Fixed Issue 51454: On a VMware SD-WAN Orchestrator running release 4.0.0, a customer may be unable to create, read, update or delete non-global segment business policies if the segment has a 'segmentId' bigger than the total amount of segments assigned to a VMware SD-WAN Edge's configuration profile.
This issue surfaces only when one or more segments is deleted. The Orchestrator UI uses an incorrect index to access segments and a segment deletion(s) can trigger this issue as a result.
- Partially Fixed Issue 52491: When a new port forwarding rule is added in a VMware SD-WAN Orchestrator, the configuration cannot be saved.
Note: The fix for this issue in build R401-20201202-GA is only partial. The complete fix for this issue is in R401-20201215-GA.
This issue occurs when a segment is mapped to a port forwarding rule after deleting a few segments. The user is unable to save a firewall configuration with error "Cannot read property 'logicalId' of undefined".
- Fixed Issue 52805: When a Partner Administrator with a Business Specialist role logs onto a Release 4.x Orchestrator after an administrator with a more powerful role (i.e. Superuser or Standard) was previously logged in and tries to use the new user interface, the Monitoring pages have missing data and "User does not have privilege" errors will be present.
This occurs on the New Orchestrator UI when different user roles log in and out of the same browser. The New Orchestrator UI pages will display user errors regarding wrong permissions on several pages. This is caused by user permissions not resetting when logging in/out (corrected by reloading a page). The Business Specialist does not actually have the privileges of the higher echelon administrator and this is reflected in the errors on the Monitoring pages.
Resolved in Version R401-20201118-GA
The below issue has been resolved since Orchestrator version R401-20201111-GA.
- Fixed Issue 52427: On a VMware SD-WAN Orchestrator upgraded to Release 4.x, the outputs for Top Devices and Top Destinations are missing on the Monitor > Application page.
When a user clicks on a graph on the Applications page, the resulting dialog contains only the Transport Groups box. The boxes for Top Devices and Top Destinations are missing.
- Fixed Issue 52488: A Partner or Customer Administrator when using the New Orchestrator UI on a 4.x Orchestrator, if the user is looking at one of the Monitoring screens (e.g Applications, Sources, Destinations, etc.) and attempts to find additional detail by selecting the arrow, the user is redirected back to the main screen.
When the user is in a detailed break-out box (e.g. Top Applications, Top Destinations, etc.) and they select the "drill down" arrow on the upper right of the box, the user should be directed to a screen that filters for that data type with a graph and data providing greater granularity. Instead the user is redirected to back to the main screen for that Monitoring type (e.g. Applications, Destinations, etc).
- Fixed Issue 52689: For a VMware SD-WAN Orchestrator configured for Disaster Recovery (DR), after an upgrade to Release 4.x users may see loss of data on the Edge > Monitoring tabs.
This issue affects DR Orchestrator production deployments which has one or more Edge sites which have recorded more than 30 million client devices over the life of that site. Where that is the case, one of the backend processes responsible for performing data migration from the old datastore (MySQL) to the new datastore (ClickHouse) consumes high levels of memory in a short period of time due to the large amounts of database reads. This leads to continuous process restarts thereby affecting the outcome of the data migration.
- Fixed Issue 52762: For a VMware SD-WAN Orchestrator configured for Disaster Recovery (DR), after an upgrade to Release 4.x a user may see loss of data on the Edge > Monitoring tabs.
In Release 4.0.0, link statistics need to be migrated from the old datastore (MySQL) to the new datastore (ClickHouse). The migration process inserts data inefficiently in certain specific scale scenarios into the database thereby forcing the database to error out. The issue is isolated to inserts in smaller batches as opposed to inserts in larger batches. When a user encounters this issue they will observe "Too many parts" and "Too many simultaneous queries" errors.
Resolved in Version R401-20201111-GA
The below issues have been resolved since Orchestrator version R400-20201023-GA.
- Fixed Issue 45131: VMware SD-WAN Edges and VMware SD-WAN Gateways may be seen as offline on the VMware SD-WAN Orchestrator as heartbeats to the Orchestrator fail intermittently.
The user will observe VMware SD-WAN Gateways and VMware SD-WAN Edges being offline causing a false positive for either Gateway or Edge Down alerts being sent out to the configured recipients. This is primarily a display issue with the added nuisance of alert emails, but there is no dataplane impact because of this issue.
- Fixed Issue 47279: The IKE/IPSEC Template is not correct for the Non SD-WAN Destinations via Gateway type “Generic Firewall” (Policy Based VPN).
When displaying the IKESA template for a Generic Firewall, the lifetime value is the incorrect value which may result in configuration mismatches with the peer.
- Fixed Issue 47910: When a Non SD-WAN Destinations via Gateway with type Checkpoint has its configuration modified through the Monitor > Network Services screen, the primary VPN goes down due to the VMware SD-WAN Orchestrator pushing a configuration update which includes the wrong NVS type.
Changing the configuration on a Checkpoint Non SD-WAN Destinations via Gateway could not be performed without bringing down the primary VPN.
- Fixed Issue 48463: The filter option is not working on the Create Edge > Edge License section of the VMware SD-WAN Orchestrator UI.
When creating a VMware SD-WAN Edge on the Orchestrator, searching for a specific Edge license in the dropdown menu returns no results. This is especially problematic because choosing a license for a newly created Edge is mandatory beginning with Release 4.0.0.
- Fixed Issue 48895: After upgrading a VMware SD-WAN Orchestrator to 3.4.x, profiles that do not contain VLAN1 are unable to be modified.
This issue will occur when the VLAN with ID = 1 is removed from the profile and new VLAN(s) are added. In such a case the VLAN will not be set in interface settings when a VMware SD-WAN Edge is enabled in a profile and only VLAN with ID = 1 can be used in the Edge model interface settings.
- Fixed Issue 48951: Customer Administrators with a Standard role can still edit BGP settings even after the Standard role has been set to read-only for that setting through role customization.
After a VMware SD-WAN Orchestrator is upgraded to Release 3.4.2, a user with a Customer Standard Administrator role who previously had read-only access to edit BGP settings would find that they may perform BGP settings changes at the Edge level (though not at the Profile level).
- Fixed Issue 49927: API calls to 'monitoring/getAggregateEdgeLinkMetrics' API method covering a sufficiently large number of Edges fail with a database syntax error.
API calls to monitoring/getAggregateEdgeLinkMetrics API method covering ~2000 Edges fail with the API end-user observing a "Max query size exceeded" error.
- Fixed Issue 50119: On the Administration > System Settings page, the purpose of the "Grant access to <Partner_Name> Support" setting is not described accurately.
If a Customer Administrator Superuser disabled this setting, based on the description, they would expect a Partner Administrator to no longer be able to edit, configure, or troubleshoot their enterprise. In contrast, this setting is actually meant to regulate a Partner's ability to view and manage enterprise users and user identifiable traffic statistics.
- Fixed Issue 50992: Dispatch of configuration updates that impact the VMware SD-WAN Edge control plane configuration may be delayed in exceedingly rare cases where a device clock is skewed in the future prior to initial activation.
In rare cases where an Edge clock is skewed in the future prior to activation and the Edge is configured at or prior to activation time with interface settings overrides, the Edge can trick the VMware SD-WAN Orchestrator into erroneously withholding configuration updates that impact the Edge control plane configuration (e.g. VPN, cloud security, etc.).
- Fixed Issue 51152: A user may observe an unexpected drop in a chart on the Edge > Monitoring tab in the VMware SD-WAN Orchestrator UI. This drop will occur at different places on the chart every time the user refreshes the UI monitoring page.
The VMware SD-WAN Edge exports statistics every 5 minutes. However, the Orchestrator process that constructs the time series for the monitoring charts may choose an interval that is less than 5 minutes. Due to this discrepancy, data points of the time series will miss a given interval and will be accounted for in the next interval.
- Fixed Issue 51227: A user is not able to select a time range of up to 1 year for the Edge Monitoring tabs: Application, Sources, and Destination.
There was a missing configuration for the VMware SD-WAN Orchestrator UI that would permit a user to select a time range of up to 1 year for the Monitoring tabs - Application, Sources, and Destination. Instead, the user could only select a time range of no greater than 2 weeks.
- Fixed Issue 52271: When a VMware SD-WAN Orchestrator is upgraded to Release 4.0.0, a user is not able to change Edge-overridden Cloud Security Service (CSS) configurations if there is a business policy using that CSS present on any VMware SD-WAN Edge in a customer enterprise.
The user would encounter this issue on Configure > Edge > Device, in the Cloud Security Service section. This issue impacts a customer because the user is unable to change the CSS provider. This issue is the result of an added UI validation on 4.0.0 that is designed to prevent a user from changing an Edge-overridden CSS provider if that CSS is being used for any Edge-level business policy of the same Edge. However, instead of limiting the check to business policies of the configured Edge, the Orchestrator checks all of the Edges in the customer's enterprise.
Open Issues in Release 4.0.1
The known issues are grouped as follows.Edge/Gateway Known Issues
- Issue 14655:
Plugging or unplugging an SFP adapter may cause the device to stop responding on the Edge 540, Edge 840, and Edge 1000 and require a physical reboot.
Workaround: The Edge must be physically rebooted. This may be done either on the Orchestrator using Remote Actions > Reboot Edge, or by power-cycling the Edge.
- Issue 25504:
Static route costs greater than 255 may result in unpredictable route ordering.
Workaround: Use a route cost between 0 and 255
- Issue 25595:
A restart may be required for changes to static SLA on a WAN overlay to work properly.
Workaround: Restart Edge after adding and removing Static SLA from WAN overlay
- Issue 25742:
Underlay accounted traffic is capped at a maximum of the capacity towards the VMware SD-WAN Gateway, even if that is less than the capacity of a private WAN link which is not connected to the Gateway.
- Issue 25758:
USB WAN links may not update properly when switched from one USB port to another until the VMware SD-WAN Edge is rebooted.
Workaround: Reboot the Edge after moving USB WAN links from one port to another.
- Issue 25855:
A large configuration update on the Partner Gateway (e.g. 200 BGP-enabled VRFs) may cause latency to increase for approximately 2-3 seconds for some traffic via the VMware SD-WAN Gateway.
Workaround: No workaround available.
- Issue 25921:
VMware SD-WAN Hub High Availability failover takes longer than expected (up to 15 seconds) when there are three thousand branch Edges connected to the Hub.
- Issue 25997:
The VMware SD-WAN Edge may require a reboot to properly pass traffic on a routed interface that has been converted to a switched port.
Workaround: Reboot the Edge after making the configuration change.
- Issue 26421:
The primary Partner Gateway for any branch site must also be assigned to a VMware SD-WAN Hub cluster for tunnels to the cluster to be established.
- Issue 28175:
Business Policy NAT fails when the NAT IP overlaps with the VMware SD-WAN Gateway interface IP.
- Issue 31210:
VRRP: ARP is not resolved in the LAN client for the VRRP virtual IP address when the VMware SD-WAN Edge is master with a non-global CDE segment running on the LAN interface.
- Issue 32731:
Conditional default routes advertised via OSPF may not be withdrawn properly when the route is turned off. Re-enabling and disabling the route will retract it successfully.
- Issue 32960:
Interface “Autonegotiation” and “Speed” status might be displayed incorrectly on the Local Web UI for activated VMware SD-WAN Edges.
- Issue 32981:
Hard-coding speed and duplex on a DPDK-enabled port may require a VMware SD-WAN Edge reboot for the configurations to take effect as it requires disabling DPDK.
- Issue 34254:
When a Zscaler CSS is created and the Global Segment has FQDN/PSK settings configured, these settings are copied to Non-Global Segments to form IPsec tunnels to a Zscaler CSS.
- Issue 35778:
When there are multiple user-defined WAN links on a single interface, only one of those WAN links can have a GRE tunnel to Zscaler.
Workaround: Use a different interface for each WAN link that needs to build GRE tunnels to Zscaler.
- Issue 35807:
A DPDK routed interface will be disabled completely if the interface is disabled and re-enabled from the VMware SD-WAN Orchestrator.
- Issue 36923:
Cluster name may not be updated properly in the NetFlow interface description for a VMware SD-WAN Edge which is connected to that Cluster as its Hub.
- Issue 38682:
A VMware SD-WAN Edge acting as a DHCP server on a DPDK-enabled interface may not properly generate “New Client Device" events for all connected clients.
- Issue 38767:
When a WAN overlay that has GRE tunnels to Zscaler configured is changed from auto-detect to user-defined, stale tunnels may remain until the next restart.
Workaround: Restart the Edge to clear the stale tunnel.
- Issue 39134:
The System health statistic “CPU Percentage” may not be reported correctly on Monitor > Edge > System for the VMware SD-WAN Edge, and on Monitor > Gateways for the VMware SD-WAN Gateway.
Workaround: Users should use handoff queue drops for monitoring Edge capacity not CPU percentage.
- Issue 39374:
Changing the order of VMware SD-WAN Partner Gateways assigned to a VMware SD-WAN Edge may not properly set Gateway 1 as the local Gateway to be used for bandwidth testing.
- Issue 39608:
The output of the Remote Diagnostic “Ping Test” may display invalid content briefly before showing the correct results.
- Issue 39624:
Ping through a subinterface may fail when the parent interface is configured with PPPoE.
- Issue 39659:
On a site configured for Enhanced High Availability, with one WAN link on each VMware SD-WAN Edge, when the standby Edge has only PPPoE connected and the active has only non-PPPoE connected, a split brain state (active/active) may be possible if the HA cable fails.
- Issue 39753:
Disabling Dynamic Branch-to-Branch VPN may cause existing flows currently being sent using Dynamic Branch-to-Branch to stall.
- Issue 40096:
If an activated VMware SD-WAN Edge 840 is rebooted, there is a chance an SFP module plugged into the Edge will stop passing traffic even though the link lights and the VMware SD-WAN Orchestrator will show the port as 'UP'.
Workaround: Unplug the SFP module and then replug it back into the port.
- Issue 40421:
Traceroute is not showing the path when passing through a VMware SD-WAN Edge with an interface configured as a switched port.
- Issue 42278:
For a specific type of peer misconfiguration, the VMware SD-WAN Gateway may continuously send IKE init messages to a Non-SD-WAN peer. This issue does not disrupt user traffic to the Gateway; however, the Gateway logs will be filled with IKE errors and this may obscure useful log entries.
- Issue 42388:
On a VMware SD-WAN Edge 540, an SFP port is not detected after disabling and reenabling the interface from the VMware SD-WAN Orchestrator.
- Issue 42488:
On a VMware SD-WAN Edge where VRRP is enabled for either a switched or routed port, if the cable is disconnected from the port and the Edge Service is restarted, the LAN connected routes are advertised.
Workaround: There is no workaround for this issue.
- Issue 42872:
Enabling Profile Isolation on a Hub profile where a Hub cluster is associated does not revoke the Hub routes from the routing information base (RIB).
- Issue 43373:
When the same BGP route is learnt from multiple VMware SD-WAN Edges, if this route is moved from preferred to eligible exit in the Overlay Flow Control, the Edge is not removed from the advertising list and continues to be advertised.
Workaround: Enable distributed cost calculation on the VMware SD-WAN Orchestrator.
- Issue 44526: For an enterprise where two different sites deploy their VMware SD-WAN Edges as Hubs while also using a high-availability topology, and each site uses the other Hub site as a Hub in its profile. If one of the Hub sites triggers an HA failover, it may take up to 30 minutes for both Hub Edges to reestablish tunnels with each other.
On an HA failover, both Hub Edges try to initiate a tunnel with each other at the same time and neither replies to the peer, the packet exchange between both Hubs occurs, but IKE never succeeds. This leads to a deadlock that has been observed to take up to 30 minutes to resolve on its own. The issue is intermittent and does not occur after every HA failover.
Workaround: To prevent this issue from occurring, the customer should configure only one of the two HA Hub sites to use the other Hub site as a Hub for itself. For example, where there are two HA Hub sites, Hub1 and Hub2, Hub1 could have Hub2 as a Hub for itself in its profile, but Hub2 must not use Hub1 as a Hub in its profile.
- Issue 44832:
Traffic from one Non SD-WAN Destinations via Edge to another Non SD-WAN Destinations via Edge (i.e. 'hairpinning' or 'NAT loopback'), is dropped on the VMware SD-WAN Edge.
- Issue 44995:
OSPF routes are not revoked from VMware SD-WAN Gateways and VMware SD-WAN Spoke Edges when the routes are withdrawn from the Hub Cluster.
- Issue 45189:
With source LAN side NAT is configured, the traffic from a VMware SD-WAN Spoke Edge to a Hub Edge is allowed even without the static route configuration for the NAT subnet.
- Issue 45302:
In a VMware SD-WAN Hub Cluster, if one Hub loses connectivity for more than 5 minutes to all of the VMware SD-WAN Gateways common between itself and its assigned Spoke Edges, the Spokes may in rare conditions be unable to retain the hub routes after 5 minutes. The issue resolves itself when the Hub regains contact with the Gateways.
- Issue 46053:
BGP preference does not get auto-corrected for overlay routes when its neighbor is changed to an uplink neighbor.
Workaround: An Edge Service Restart will correct this issue.
- Issue 46137:
A VMware SD-WAN Edge running 3.4.x software does not initiate a tunnel with AES-GCM encryption even if the Edge is configured for GCM.
- Issue 46216:
On a Non SD-WAN Destinations via Gateway or Edge where the peer is an AWS instance, when the peer initiates Phase-2 re-key, the Phase-1 IKE is also deleted and forces a re-key. This means the tunnel is torn down and rebuilt, causing packet loss during the tunnel rebuild.
Workaround: To avoid tunnel destruction, configure the Non SD-WAN Destinations via Gateway/Edge or CSS IPsec rekey timer to less than 60 minutes. This prevents AWS from initiating the re-key.
- Issue 46391:
For a VMware SD-WAN Edge 3800, the SFP1 and SFP2 interfaces each have issues with Multi-Rate SFPs (i.e. 1/10G) and should not be used in those ports.
Workaround: Please use single rate SFP's per the KB article VMware SD-WAN Supported SFP Module List (79270). Multi-Rate SFPs may be used with SFP3 and SFP4.
- Issue 46628:
The GE5 and GE6 ports on a VMware SD-WAN Edge 620/640/680 do not detect a link if the ports are configured with 100 Mbps and duplex.
- Issue 46918:
A VMware SD-WAN Spoke Edge using the 3.4.2 Release does not update the private network id of a Cluster Hub node properly.
- Issue 47084:
A VMware SD-WAN Hub Edge cannot establish more than 750 PIM (Protocol-Independent Multicast) neighbors when it has 4000 Spoke Edges attached.
- Issue 47244:
On an activated VMware SD-WAN Edge 6x0 with DPDK enabled, some Copper SFPs, the Edge will show the link as 'UP' even when no cable is inserted on the VMware SD-WAN Orchestrator UI.
Workaround: Plugging and unplugging a cable removes the false state.
- Issue 47355:
When the same route is learned via local underlay BGP, Hub BGP and/or statically configured on the Partner Gateway, the sorting order of the routes is incorrect with the Hub BGP being preferred over the underlay BGP.
- Issue 47664:
In a Hub and Spoke configuration where Branch-to-Branch via Hub VPN is disabled, trying to U-turn Branch-to-Branch traffic using a summary route on an L3 switch/router will cause routing loops.
Workaround: Configure Cloud VPN to enable Branch-to-Branch VPN and select “Use Hubs for VPN”.
- Issue 47681:
When a host on the LAN side of a VMware SD-WAN Edge uses the same IP as that Edge’s WAN interface, the connection from the LAN host to the WAN does not work.
- Issue 47787:
A VMware SD-WAN Spoke Edge configured with a backhaul business policy incorrectly sends traffic via the VMware SD-WAN Gateway path if that flow is initiated from the Hub Edge to that Spoke Edge.
- Issue 48166:
A VMware SD-WAN Virtual Edge on KVM is not supported when using a Ciena virtualization OS and the Edge will experience recurring Dataplane Service Failures.
- Issue 48175:
A VMware SD-WAN Edge running Release 3.4.2 will form an OSPF adjacency on a non-global segment if the non-global segment has an interface configured in the same IP range as an interface configured on the global segment
- Issue 48488:
Business policy rule is not overridden if the outbound traffic box is not checked (for the Traffic initiated from remote peer and allowed by 1:1 NAT rule).
Workaround: Please check “Outbound Traffic” in 1:1 NAT.
- Issue 48502:
In some scenarios, a VMware SD-WAN Hub Edge being used to backhaul internet traffic may experience a Dataplane Service Failure due the improper handling of backhaul return packets.
- Issue 48530:
VMware SD-WAN Edge 6x0 models do not perform autonegotiation for triple speed (10/100/1000 Mbps) copper SFP's.
Workaround: Edge 520/540 supports triple speed copper SFPs but this model has been marked for End-of-Sale by Q1 2021.
- Issue 48666:
IPsec-fronted Gateway Path MTU calculation does not account for 61 Byte IPsec overhead, resulting in higher MTU advertisement to LAN client and subsequent IPsec packet fragmentation.
Workaround: There is no workaround for this issue.
- Issue 49172:
A Policy Based NAT rule configured with the same NAT subnet for two different VMware SD-WAN Edges does not work.
- Issue 49738:
In some cases, when a VMware SD-WAN Spoke Edge is configured to use multiple Hub Edges, the Spoke Edge may not form tunnels to one of the Hubs configured in the Hub list.
- Issue 50518:
On a VMware SD-WAN Gateway where PKI is enabled, if >6000 PKI tunnels attempt to connect to the Gateway, the tunnels may not all come up because inbound SAs do not get deleted.
Note: Tunnels using pre-shared key (PSK) authentication do not have this issue.
- Issue 53219: After a VMware SD-WAN Hub Cluster rebalances, a few Spoke Edges may not have their RPF interface/IIF set properly.
On the affected Spoke Edges, multicast traffic will be impacted. What happens is that after a cluster rebalance, some of the Spoke Edges fail to send a PIM join.
Workaround: This issue will persist until the affected Spoke Edge has an Edge Service restart.
- Issue 84825: For a site deployed with a High-Availability topology where BGP is configured, if the site has greater than 512 BGPv4 match and set rules configured, the customer may observe the HA Edge pair continuously failing over without ever recovering.
Greater than 512 BGPv4 match and set rules is understood as a customer configuring more than 256 such rules on the inbound filter and 256 rules on the outbound filter. This issue would be disruptive to the customer as the repeated failover would cause flows for real time traffic like voice calls to be continuously dropped and then recreated. When HA Edges experience this issue, the process that synchronizes Edge CPU threads fails causing the Edge to reboot to recover, but the promoted Edge also experiences the same issue and reboots in turn with no recovery reached at the site.
Workaround: Without a fix for this issue, the customer must ensure that no more than 512 BGPv4 match and set rules are configured for an HA site.
If a site is experiencing this issue and has more than 512 BGP/v4 match and set rules configured, the customer must immediately reduce the number of rules to 512 or less to recover the site.
Alternatively, if the customer must have more than 512 BGPv4 match and set rules, they can downgrade the HA Edges to Release 3.4.6 where this issue is not encountered, but at the cost of Edge features found in later releases. This can only be done if their Edge model is supported on 3.4.6 and the customer should confirm that is so before downgrading.
- Issue 19566:
After High Availability failover, the serial number of the standby VMware SD-WAN Edge may be shown as the active serial number in the Orchestrator.
- Issue 20900:
If the MaxMind geolocation service is enabled and cannot reach the MaxMind server, new VMware SD-WAN Edge activations will not work.
- Issue 21342:
When assigning Partner Gateways per-segment, the proper list of Gateway Assignments may not show under the Operator option "View" Gateways on the VMware SD-WAN Edge monitoring list.
- Issue 24269:
Monitor > Transport > Loss not graphing observed WAN link loss while QoE graphs do reflect this loss.
- Issue 25932:
The VMware SD-WAN Orchestrator allows VMware SD-WAN Gateways to be removed from the Gateway Pool even when they are in use.
- Issue 32335:
The ‘End User Service Agreement’ (EUSA) page throws an error when a user is trying to accept the agreement.
Workaround: Ensure no leading or trailing spaces are found in Enterprise Name.
- Issue 32435:
A VMware SD-WAN Edge override for a policy-based NAT configuration is permitted for tuples which are already configured at the profile level and vice versa.
- Issue 32856:
Though a business policy is configured to use the Hub cluster to backhaul internet traffic, the user can unselect the Hub cluster from a profile on a VMware SD-WAN Orchestrator that has been upgraded from Release 3.2.1 to Release 3.3.x.
- Issue 32913:
After Enabling High Availability, Multicast details for the VMware SD-WAN Edge are not displayed on the Monitoring Page. A failover resolves the issue.
- Issue 33026:
The ‘End User Service Agreement’ (EUSA) page does not reload properly after deleting the agreement.
- Issue 34828:
Traffic cannot pass between a VMware SD-WAN Spoke Edge using release 2.x and a Hub Edge using release 3.3.1.
- Issue 35658:
When a VMware SD-WAN Edge is moved from one profile to another which has a different CSS setting (e.g. IPsec in profile1 to GRE in profile2), the Edge level CSS settings will continue to use the previous CSS settings (e.g. IPsec versus GRE).
Workaround: Disable and then reenable GRE at the Edge level to resolve the issue.
- Issue 35667:
When a VMware SD-WAN Edge is moved from one profile to another profile which has the same CSS setting but a different GRE CSS name (the same endpoints), some GRE tunnels will not show in monitoring.
Workaround: Disable and then reenable GRE at the Edge level to resolve the issue.
- Issue 36665:
If the VMware SD-WAN Orchestrator cannot reach the internet, user interface pages that require accessing the Google Maps API may fail to load entirely.
- Issue 38056:
The Edge-Licensing export.csv file not show region data.
- Issue 38843:
When pushing an application map, there is no Operator event, and the Edge event is of limited utility.
- Issue 39633:
The Super Gateway hyper link does not work after a user assigns the Alternate Gateway as the Super Gateway.
- Issue 39790:
The VMware SD-WAN Orchestrator allows a user to configure a VMware SD-WAN Edge’s routed interface to have greater than the supported 32 subinterfaces, creating the risk that a user can configure 33 or more subinterfaces on an interface which would cause a Dataplane Service Failure for the Edge.
- Issue 40341:
Though the Skype application is properly categorized on the backend as Real Time traffic, when editing the Skype Business Policy on the VMware SD-WAN Orchestrator, the Service Class may erroneously display “Transactional”.
- Issue 41691:
User cannot change the 'Number of addresses' field although the DHCP pool is not exhausted on the Configure > Edge > Device page.
- Issue 43276:
User cannot change the Segment type when a VMware SD-WAN Edge or Profile has a partner gateway configured.
- Issue 44153:
The VMware SD-WAN Orchestrator does not consistently send alert emails to the email addresses configured in the 'Alerts and Notifications' section.
- Issue 46254:
During a VMware SD-WAN Edge activation, the VMware SD-WAN Orchestrator does not detect a changed WAN link MTU or the presence of a VLAN ID for DHCP configured interfaces.
- Issue 47269:
The VMware SD-WAN 510-LTE interface may appear for Edge models that do not support an LTE interface.
- Issue 47713:
If a Business Policy Rule is configured while Cloud VPN is disabled, the NAT configuration must be reconfigured upon enabling Cloud VPN.
- Issue 47820:
If a VLAN is configured with DHCP disabled at the Profile level, while also having an Edge Override for this VLAN on that Edge with DHCP enabled, and there is an entry for the DNS server field set to none (no IP configured), the user will be unable to make any changed on the Configure > Edge > Device page and will get an error message of ‘invalid IP address ’ that does not explain or point to the actual problem.
- Issue 48085:
The VMware SD-WAN Orchestrator allows a user to delete a VLAN which is associated with an interface.
- Issue 48737:
On a VMware SD-WAN Orchestrator which is using the Release 4.0.0 new user interface, If a user is on a Monitor page and changes the Start & End time interval and then navigates between tabs, the Orchestrator does not update Start & End interval time to the new values.
- Issue 49225:
VMware SD-WAN Orchestrator does not enforce a limit of 32 total VLANs.
- Issue 49790:
When a VMware SD-WAN Edge is activated to Release 4.0.0, the activation is posted twice in Events.
Workaround: Ignore the duplicate event.
- Issue 50531:
When two Operators of differing privileges use the same browser window when accessing the New UI on a 4.0.0 Release version of the VMware SD-WAN Orchestrator, and the Operator with lesser privileges tries to login after the Operator with higher privileges, that lesser privileged Operator will observe multiple errors stating that the "user does not have privilege".
Note: There is no escalation in privileges for the Operator with lower privileges, only the display of error messages.
Workaround: The next operator may refresh that page prior to logging in to prevent seeing the errors, or each Operator may use different browser windows to avoid this display issue.
- Issue 53652: The VMware SD-WAN Orchestrator Web UI's Application viewer displays a random name for a custom application created prior to the Orchestrator upgrading to Release 4.0.1.
Whenever a custom application is configured with an appId which already exists as part of the default initial application map, the custom application always shows the display name of the default initial application map and overrides the customer defined name. Even when the Orchestrator is upgraded from a lower version to a higher version and the higher version's default initial application map has appIds that conflicts with the appIds of a custom application created in a lower version, then after upgrade for those custom applications, the display name is shown incorrectly (showing the display name of the higher Orchestrator version's default initial application map against the display name of the lower version custom application).
Workaround: Download the application map, hand-fix the appIds of the custom application (i.e., change the appIds of the custom application starting from 7000, 7001, and so on). Upload the newly modified Application Map. Duplicate the current operator profile and just change the application map to newly modified application map and then assign this duplicate operator profile to the Edges in the customer enterprise.
- Issue 51722: On the Release 4.0.0 VMware SD-WAN Orchestrator, the time range selector is no greater than two weeks for any statistic in the Monitor > Edge tabs.
The time range selector does not show options greater than "Past 2 Weeks" in Monitor > Edge tabs even if the retention period for a set of statistics is much longer than 2 weeks. For example, flow and link statistics are retained for 365 days by default (which is configurable), while path statistics are retained only for 2 weeks by default (also configurable). This issue is making all monitor tabs conform to the lowest retained type of statistic versus allowing a user to select a time period that is consistent with the retention period for that statistic.
Workaround: A user may use the "Custom" option in the time range selector to see data for more than 2 weeks.