SD-WAN Edge Performance and Scale Data

Introduction

The tests represent common deployment scenarios to provide recommendations that apply to most deployments. The test data herein are not all-inclusive metrics, nor are they performance or scale limits. There are implementations where the observed performance exceeds the test results and others where specific services, extremely small packet sizes, or other factors can reduce performance below the test results.

Customers are welcome to perform independent tests, and results could vary. However, recommendations based on our test results are adequate for most deployments.

VMware SD-WAN Edge

VMware SD-WAN Edges are zero-touch, enterprise-class appliances that provide secure optimized connectivity to private, public, and hybrid applications as well as compute and virtualized services. VMware SD-WAN Edges perform deep application recognition of traffic flows, performance metrics measurements of underlay transport and apply end-to-end quality of service by applying packet-based link steering and on-demand application remediation, in addition to supporting other virtualized network services.

Throughput Performance Test Topologies

Figure 1. FIGURE 1: Throughput performance test topology for devices 1 Gbps or lower

Figure 2. FIGURE 2: Throughput performance test topology for devices above 1 Gbps

Test Methodology

This subsection details the performance and scale test methodology used to derive the results.

Performance Test Methodology

The testing methodology for Edges uses the industry benchmarking standard RFC 2544 as a framework to execute throughput performance testing. There are specific changes to the type of traffic used and configurations set during testing, described below:

Performance is measured using a fully operational SD-WAN network overlay (DMPO tunnels) test topology in order to exercise the SD-WAN features and obtain results that can be used to appropriately size WAN networks. Testing is conducted using stateful traffic that establishes multiple flows (connections) and are a mix of well-known applications. The number of flows depends on the platform model being tested. Platforms are divided by expected aggregate performance of under 1 Gbps and over 1 Gbps models. Typically, hundreds of flows are needed to fully exercise and determine max throughput of platforms expected to perform under 1 Gbps, and thousands of flows are used to exercise platforms of over 1 Gbps.
The traffic profiles simulate two network traffic conditions:
- Large Packet, a 1300-byte condition.
- IMIX, a mix of packet sizes that average to a 417-byte condition.
These traffic profiles are used separately to measure maximum throughput per profile.
Performance results are recorded at a packet drop rate (PDR) of 0.01%. The PDR mark provides a more realistic performance result which accounts for normal packet drop that may occur within the SD-WAN packet pipeline in the device. A PDR of 0.01% does not impact application experience even in single link deployment scenarios.
- The device under test is configured with the following DMPO features; IPsec encrypted using AES-128 and SHA1 for hashing, Application Recognition, link SLA measurements, per-packet forwarding. Business Policy is configured to match all traffic as bulk/low priority to prevent DMPO NACK or FEC from executing and incorrectly altering the traffic generator’s packet count tracking.

Test Results

VMware SD-WAN Edge Performance and Scale Results

Performance metrics are based on the Test Methodology detailed above.

Switched Port Performance: VMware SD-WAN Edges are designed to be deployed as gateway routers between the LAN and the WAN. However, the Edges also provide the flexibility of meeting a variety of other deployment topologies. For example, SD-WAN Edges can have their interfaces configured to operate as switched ports—allowing the switching of LAN traffic between various LAN interfaces without the need for an external device.

An Edge with its interfaces configured as switched ports is ideal for small office deployments where high throughput is not required, as the additional layer of complexity required to handle traffic switching reduces the overall performance of the system. For most deployments, VMware recommends using all routed interfaces.

Note:

The Edge device's Maximum Throughput is the sum of throughput across all interfaces of the Edge under test.
Overall traffic is the “aggregate” of all traffic flows going to and from an Edge device.

Table 1. Physical Edge Appliances
VMware SD-WAN Edge	510, 510N	510-LTE	520	520V	540	610, 610C, 610N	610-LTE	620, 620C, 620N
Maximum Throughput Large Packet (1300-byte)
Routed Mode All Ports	350 Mbps	350 Mbps	350 Mbps	350 Mbps	1 Gbps	350 Mbps	350 Mbps	1.5 Gbps
Switched Mode All Ports	200 Mbps	200 Mbps	200 Mbps	200 Mbps	650 Mbps	300 Mbps	300 Mbps	700 Mbps
Maximum Throughput Internet Traffic (IMIX)
Routed Mode All Ports	200 Mbps	200 Mbps	200 Mbps	200 Mbps	500 Mbps	200 Mbps	200 Mbps	750 Mbps
Routed Mode All Ports with Edge Network Intelligence activated.	200 Mbps	200 Mbps	200 Mbps	200 Mbps	400 Mbps	200 Mbps	200 Mbps	600 Mbps
Routed Mode All Ports with IPS and Stateful Firewall activated.	140 Mbps	140 Mbps	140 Mbps	140 Mbps	350 Mbps	150 Mbps	150 Mbps	500 Mbps
Switched Mode All Ports	80 Mbps	80 Mbps	80 Mbps	80 Mbps	200 Mbps	150 Mbps	150 Mbps	250 Mbps
Other Scale Vectors
Maximum Tunnel Scale	50	50	50	50	100	50	50	100
Flows Per Second	2,400	2,400	2,400	2,400	4,800	2,400	2,400	4,800
Flows Per Second with Edge Network Intelligence activated	1,200	1,200	1,200	1,200	2,400	1,200	1,200	2,400
Maximum Concurrent Flows	240K	240K	240K	240K	480K	240K	240K	480K
Maximum Concurrent Flows with Edge Network Intelligence activated	120K	120K	120K	120K	240K	120K	120K	240K
Maximum Number of Routes	100K	100K	100K	100K	100K	100K	100K	100K
Maximum Number of Segments	128	128	128	128	128	128	128	128
Maximum Number of NAT Entries	80K	80K	80K	80K	150K	80K	80K	150K

Table 2.
VMware SD-WAN Edge	640, 640C, 640N	680, 680C, 680N	840	2000	3400, 3400C	3800, 3800C	3810
Maximum Throughput Large Packet (1300-byte)
Routed Mode All Ports	3 Gbps	6 Gbps	4 Gbps	10 Gbps	7 Gbps	10 Gbps	10 Gbps
Switched Mode All Ports	1 Gbps	1 Gbps	1 Gbps	1.2 Gbps	1.2 Gbps	1.2 Gbps	1.2 Gbps
Maximum Throughput Internet Traffic (IMIX)
Routed Mode All Ports	1 Gbps	2 Gbps	1.5 Gbps	5 Gbps	2.5 Gbps	5 Gbps	5 Gbps
Routed Mode All Ports with Edge Network Intelligence activated.	800 Mbps	1.6 Gbps	1.2 Gbps	4 Gbps	2 Gbps	4 Gbps	4 Gbps
Routed Mode All Ports with IPS and Stateful Firewall activated	700 Mbps	1.5 Mbps	1 Gbps	3.5 Gbps	1.7 Gbps	3.5 Gbps	3.5 Gbps
Switched Mode All Ports	350 Mbps	350 Mbps	350 Mbps	350 Mbps	900 Mbps	900 Mbps	900 Mbps
Other Scale Vectors
Maximum Tunnel Scale	400	800	400	6,000	4,000	6,000	6,000
Flows Per Second	19,200	19,200	19,200	38,400	38,400	38,400	38,400
Flows Per Second with Edge Network Intelligence activated	9,600	9,600	9,600	19,200	19,200	19,200	19,200
Maximum Concurrent Flows	1.9M	1.9M	1.9M	3.8M	1.9M	3.8M	3.8M
Maximum Concurrent Flows with Edge Network Intelligence activated	960K	960K	960K	960K	960K	960K	960K
Maximum Number of Routes	100K	100K	100K	100K	100K	100K	100K
Maximum Number of Segments	128	128	128	128	128	128	128
Maximum Number of NAT Entries	650K	650K	650K	960K	960K	960K	960K

Note:

Large Packet performance is based on a large packet (1300-byte) payload with AES-128 encryption and DPI turned on.
Internet Traffic (IMIX) performance is based on an average packet size of 417-byte payload with AES-128 encryption and DPI turned on.
Edge Network Intelligence peformance numbers were measured with a 400-byte payload.
IPS and Stateful Firewall performance numbers were measured using a payload with an average packet size of 417-bytes and AES-128 encryption and Deep Packet Inspection (DPI) turned on.

Important: Maximum Tunnel Scale is understood as the total number of tunnels an Edge model can establish at one time with all other sites. However, the maximum number of tunnels an Edge can establish with another Edge or Gateway is 16, regardless of Edge model or type. Each public WAN link an Edge uses establishes a tunnel with each WAN link the peer Edge or Gateway has.

For example: Edge 1 with public WAN links A, B, C, and D connects to Edge 2 with public WAN links E, F, G, and H. Edge 1's WAN link A establishes a tunnel with each of Edge 2's WAN links E, F, G, and H for a total of 4 tunnels for WAN link A to Edge 2. And this follows for Edge 1's other WAN links B, C, and D. Each establishes tunnels with Edge 2's four public WAN links and so four WAN links with 4 tunnels each results in Edge 1 having 16 total tunnels to Edge 2. In this example, no additional tunnels can be established between the two Edges if an additional WAN link is added to either Edge as the maximum has been reached.

Tip: Multiple SD-WAN Edges can be deployed in a cluster for multi-gigabit performance.

Table 3. Edge Maximum Throughput When a Firewall VNF is Actively Service Chained:
Edge Model	520V	620, 620C, 620N	640, 640C, 640N	680, 680C, 680N	840	3400, 3400C	3800, 3800C	3810
Max. Throughput with FW VNF (1300-byte)	100 Mbps	300 Mbps	600 Mbps	1 Gbps	1 Gbps	2 Gbps	3 Gbps	3 Gbps

Table 4. Enhanced High-Availability (HA) Link Performance
Edge Model	510, 510N	510-LTE	520, 520v	610, 610C, 610N	610-LTE	620, 620C, 620N
Maximum Throughput (IMIX) Across Enhanced HA Link	200 Mbps	200 Mbps	100 Mbps	200 Mbps	200 Mbps	500 Mbps


Edge Model	640, 640C, 640N	680, 680C, 680N	840	2000	3400, 3400C	3800, 3800C	3810
Maximum Throughput (IMIX) Across Enhanced HA Link	800 Mbps	800 Mbps	800 Mbps	800 Mbps	800 Mbps	800 Mbps	800 Mbps

Important: Performance with Edge Network Intelligence activated:

There is a performance impact of up to 20% when analytics are activated.
Flow capacity is reduced by half when analytics are activated due to the additional memory and processing required for analysis.

Platform Independent Edge Scale Numbers

The Edge Scale numbers listed in the following table are platform independent and are valid for all Edge models, both hardware and virtual.

Note: The listed maximum value for each feature represents the supported limits that have been tested and verified by VMware. In some cases, customers may exceed values higher than that is listed in the table. If a customer exceeds the published maximum value, the environment may work, but VMware cannot guarantee that it would.


Feature	Supported Number
Feature	IPv4	IPv6
Maximum number of Port Forwarding rules on a single segment	128	128
Maximum number of Port Forwarding rules across 16 segments	128	128
Maximum number of Port Forwarding rules across 128 segments	128	128
Maximum number of Outbound Firewall Rules on a single segment	2040	2040
Maximum number of Outbound Firewall Rules across 16 segments	2040	2040
Maximum number of Outbound Firewall Rules across 128 segments	2040	2040
Maximum number of 1:1 NAT rules on a single segment	128	128
Maximum number of 1:1 NAT rules across 16 segments	128	128
Maximum number of 1:1 NAT rules across 128 segments	128	128
Maximum number of LAN side NAT rules on a single segment	256	-
Maximum number of LAN side NAT rules across 16 segments	256	-
Maximum number of LAN side NAT rules across 128 segments	256	-
Maximum number of Object Groups (1000 business policies, each business policy assigned to one object group, each object group supports 255 address groups)	1000	1000

Virtual Edge

Table 5. Private Cloud (Hypervisors)
Edge Device	Maximum Throughput		Maximum Number of Tunnels	Flows Per Second	Maximum Concurrent Flows	Maximum Number of Routes	Maximum Number of Segments
ESXi Virtual Edge (2-core, VMXNET3)	2 Gbps (1300-byte) 800 Mbps (IMIX)		50	2400	240K	35K	128
KVM Virtual Edge (2-core, Linux Bridge)	500 Mbps (1300-byte) 200 Mbps (IMIX)		50	2400	240K	35K	128
KVM Virtual Edge (2-core, SR-IOV)	1.25 Gbps (1300-byte) 600 Mbps (IMIX)		50	2400	240K	35K	128
ESXi Virtual Edge (4-core, VMXNET3)	2 Gbps (1300-byte) 1.5 Gbps (IMIX)		400	19200	1.9M	35K	128
ESXi Virtual Edge (4-core, SR-IOV)	2 Gbps (1300-byte) 1.5 Gbps (IMIX)		400	19200	1.9M	35K	128
KVM Virtual Edge (4-core, Linux Bridge)	1 Gbps (1300-byte) 350 Mbps (IMIX)		400	4800	480K	35K	128
KVM Virtual Edge (4-core, SR-IOV)	2 Gbps (1300-byte) 1 Gbps (IMIX)		400	19200	1.9M	35K	128
ESXi Virtual Edge (8-core, VMXNET3)	5 Gbps (1300-byte) 2.5 Gbps (IMIX)		800	38400	1.9M	35K	128
ESXi Virtual Edge (8-core, SR-IOV)	Version 3.4 or older: 5 Gbps (1300-byte) 2.5 Gbps (IMIX)	Version 4.0 or newer: 9 Gbps (1300-byte) 4 Gbps (IMIX)	800	38400	1.9M	35K	128
KVM Virtual Edge (8-core, SR-IOV	Version 3.4 or older: 3.5 Gbps (1300-byte) 1 Gbps (IMIX)	Version 4.0 or newer: 9 Gbps (1300-byte) 3 Gbps (IMIX)	800	38400	1.9M	35K	128


	2 vCPU	4vCPU	8vCPU	10vCPU
Minimum Memory (DRAM)	8 GB	16 GB	32 GB	32 GB
Minimum Storage	8 GB	8 GB	16 GB	16 GB
Supported Hypervisors	Software Version 3.4 or older: ESXi 6.0, 6.5U1, 6.7U1 KVM Ubuntu 14.04 LTS or 16.04 Software version 4.0 and above: ESXi 6.5U1, 6.7U1, 7.0 KVM Ubuntu 16.04 and 18.04
Supported Public Cloud	AWS, Azure, GCP, and Alibaba
Support Network I/O	SR-IOV, VirtIO, VMXNET3
Recommended Host Settings	CPUs at 2.0 GHz or higher CPU Instruction set: AES-NI AVX2 or AVX512 SSE3, SSE4, and RDTSC instruction sets Hyper-threading deactivated

Note: Performance metrics are based on a system using an Intel ^® Xeon ^® CPU E5-2683 v4 at 2.10 GHz.

Public Cloud

Table 6. Amazon Web Services (AWS)
AWS Instance Type	c5.large	c5.xlarge	c5.2xlarge	c5.4xlarge
Maximum Throughput	100 Mbps (1300-byte) 50 Mbps (IMIX)	200 Mbps (1300-byte) 100 Mbps (IMIX)	1.5 Gpbs (1300-byte) 450 Mbps (IMIX)	4 Gbps (1300-byte) 1 Gbps (IMIX)
Maximum Tunnels	50	400	800	2,000
Flows Per Second	1,200	2,400	4,800	9,600
Maximum Concurrent Flows	125,000	250,000	550,000	1.9M
Maximum Number of Routes	35,000	35,000	35,000	35,000
Maximum Number of Segments	128	128	128	128

Note: c5.2xlarge and c5.4xlarge performance and scale numbers are based on AWS Enhanced Networking (ENA SR-IOV drivers) being ‘activated’.

Table 7. Microsoft Azure
Azure VM Series	D2d v4	D4d v4	D8d v4	D16d v4
Maximum Throughput	100 Mbps (1300-byte) 50 Mbps (IMIX)	200 Mbps (1300-byte) 100 Mbps (IMIX)	1 Gbps (1300-byte) 450 Mbps (IMIX)	1 Gbps (1300-byte) 450 Mbps (IMIX)
Maximum Tunnels	50	400	800	2000
Flows Per Second	1,200	2,400	4,800	4,800
Maximum Concurrent Flows	125,000	250,000	550,000	550,000
Maximum Number of Routes	35,000	35,000	35,000	35,000
Maximum Number of Segments	128	128	128	128

Note: Azure Accelerated Networking is supported with a limited availability. Please contact your sales representative for details.

Table 8. Google Cloud Platform
GCP Instance Type	n2-highcpu-4	n2-highcpu-8	n2-highcpu-16
Maximum Throughput	850 Mbps (1300-byte) 500 Mbps (IMIX)	4.5 Gbps (1300-byte) 1.6 Gbps (IMIX)	6.5 Gbps (1300-byte) 1.9 Gbps (IMIX)
Maximum Tunnels	50	400	800
Flows Per Second	1,200	2,400	4,800
Maximum Concurrent Flows	125,000	250,000	550,000
Maximum Number of Routes	35,000	35,000	35,000
Maximum Number of Segments	128	128	128

Use of DPDK on VMware SD-WAN Edges

To improve packet throughput performance, VMware SD-WAN Edges take advantage of Data Plane Development Kit (DPDK) technology. DPDK is a set of data plane libraries and drivers provided by Intel for offloading TCP packet processing from the operating system kernel to processes running in user space and results in higher packet throughput. For more details, see https://www.dpdk.org/.

Edge hardware models 620 and higher and all virtual Edges use DPDK by default on their routed interfaces. Edges do not use DPDK on their switched interfaces. A user cannot activate or deactivate DPDK for an Edge interface.