Certain types of iSCSI adapters depend on the VMkernel networking. These adapters include the software or dependent hardware iSCSI adapters, and the VMware iSCSI over RDMA (iSER) adapter. If your ESXi environment includes any of these adapters, you must configure connections for the traffic between the iSCSI or iSER component and the physical network adapters.
Configuring the network connection involves creating a virtual VMkernel adapter for each physical network adapter. You use 1:1 mapping between each virtual and physical network adapter. You then associate the VMkernel adapter with an appropriate iSCSI or iSER adapter. This process is called port binding.
- You can connect the software iSCSI adapter with any physical NICs available on your host.
- The dependent iSCSI adapters must be connected only to their own physical NICs.
- You must connect the iSER adapter only to the RDMA-capable network adapter.
For specific considerations on when and how to use network connections with software iSCSI, see the VMware knowledge base article at http://kb.vmware.com/kb/2038869.
Multiple Network Adapters in iSCSI or iSER Configuration
If your host has more than one physical network adapter for iSCSI or iSER, you can use the adapters for multipathing.
You can use multiple physical adapters in a single or multiple switch configurations.
In the multiple switch configuration, you designate a separate vSphere switch for each virtual-to-physical adapter pair.
An alternative is to add all NICs and VMkernel adapters to the single vSphere switch. The number of VMkernel adapters must correspond to the number of physical adapters on the vSphere Standard switch. The single switch configuration is not appropriate for iSER because iSER does not support NIC teaming.
For that type of configuration, you must override the default network setup and make sure that each VMkernel adapter maps to only one corresponding active physical adapter, as the table indicates.
| VMkernel Adapter (vmk#) | Physical Network Adapter (vmnic#) |
|---|---|
| vmk1 (iSCSI1) | Active Adapters vmnic1 Unused Adaptersvmnic2 |
| vmk2 (iSCSI2) | Active Adapters vmnic2 Unused Adaptersvmnic1 |
You can also use distributed switches. For more information about vSphere distributed switches and how to change the default network policy, see the vSphere Networking documentation.
- Physical network adapters must be on the same subnet as the storage system they connect to.
- (Applies only to iSCSI and not to iSER) If you use separate vSphere switches, you must connect them to different IP subnets. Otherwise, VMkernel adapters might experience connectivity problems and the host fails to discover the LUNs.
- The single switch configuration is not appropriate for iSER because iSER does not support NIC teaming.
- Array target iSCSI ports are in a different broadcast domain and IP subnet.
- VMkernel adapters used for iSCSI connectivity exist in different broadcast domains, IP subnets, or use different virtual switches.
Note: In iSER configurations, the VMkernel adapters used for iSER connectivity cannot be used for converged traffic. The VMkernel adapters that you created to enable connectivity between the ESXi host with iSER and the iSER target must be used only for iSER traffic.
Best Practices for Configuring Networking with Software iSCSI
When you configure networking with software iSCSI, consider several best practices.
Software iSCSI Port Binding
You can bind the software iSCSI initiator on the ESXi host to a single or multiple VMkernel ports, so that iSCSI traffic flows only through the bound ports. Unbound ports are not used for iSCSI traffic.
When port binding is configured, the iSCSI initiator creates iSCSI sessions from all bound ports to all configured target portals.
| VMkernel Ports | Target Portals | iSCSI Sessions |
|---|---|---|
| 2 bound VMkernel ports | 2 target portals | 4 sessions (2 x 2) |
| 4 bound VMkernel ports | 1 target portal | 4 sessions (4 x 1) |
| 2 bound VMkernel ports | 4 target portals | 8 sessions (2 x 4) |
No Port Binding
If you do not use port binding, the ESXi networking layer selects the best VMkernel port based on its routing table. The host uses the port to create an iSCSI session with the target portal. Without the port binding, only one session per each target portal is created.
| VMkernel Ports | Target Portals | iSCSI Sessions |
|---|---|---|
| 2 unbound VMkernel ports | 2 target portals | 2 sessions |
| 4 unbound VMkernel ports | 1 target portal | 1 session |
| 2 unbound VMkernel ports | 4 target portals | 4 sessions |
Software iSCSI Multipathing
Example 1. Multiple paths for an iSCSI target with a single network portal
If your target has only one network portal, you can create multiple paths to the target by adding multiple VMkernel ports on your ESXi host and binding them to the iSCSI initiator.
In this example, all initiator ports and the target portal are configured in the same subnet. The target is reachable through all bound ports. You have four VMkernel ports and one target portal, so total of four paths are created.
Without the port binding, only one path is created.
Example 2. Multiple paths with VMkernel ports in different subnets
You can create multiple paths by configuring multiple ports and target portals on different IP subnets. By keeping initiator and target ports in different subnets, you can force ESXi to create paths through specific ports. In this configuration, you do not use port binding because port binding requires that all initiator and target ports are on the same subnet.
ESXi selects vmk1 when connecting to Port 0 of Controller A and Controller B because all three ports are on the same subnet. Similarly, vmk2 is selected when connecting to Port 1 of Controller A and B. You can use NIC teaming in this configuration.
| Paths | Description |
|---|---|
| Path 1 | vmk1 and Port0 of Controller A |
| Path 2 | vmk1 and Port0 of Controller B |
| Path 3 | vmk2 and Port1 of Controller A |
| Path 4 | vmk2 and Port1 of Controller B |
Routing with Software iSCSI
You can use the esxcli command to add static routes for your iSCSI traffic. After you configure static routes, initiator and target ports in different subnets can communicate with each other.
Example 1. Using static routes with port binding
In this example, you keep all bound VMkernel ports in one subnet (N1) and configure all target portals in another subnet (N2). You can then add a static route for the target subnet (N2).
Use the following command:
# esxcli network ip route ipv4 add -gateway 192.168.1.253 -network 10.115.179.0/24
Example 2. Using static routes to create multiple paths
In this configuration, you use static routing when using different subnets. You cannot use the port binding with this configuration.
You configure vmk1 and vmk2 in separate subnets, 192.168.1.0 and 192.168.2.0. Your target portals are also in separate subnets, 10.115.155.0 and 10.155.179.0.
You can add the static route for 10.115.155.0 from vmk1. Make sure that the gateway is reachable from vmk1.
# esxcli network ip route ipv4 add -gateway 192.168.1.253 -network 10.115.155.0/24
You then add static route for 10.115.179.0 from vmk2. Make sure that the gateway is reachable from vmk2.
# esxcli network ip route ipv4 add -gateway 192.168.2.253 -network 10.115.179.0/24
When connecting with Port 0 of Controller A, vmk1 is used.
When connecting with Port 0 of Controller B, vmk2 is used.
Example 3. Routing with a separate gateway per vmkernel port
Starting with vSphere 6.5, you can configure a separate gateway per VMkernel port. If you use DHCP to obtain IP configuration for a VMkernel port, gateway information can also be obtained using DHCP.
To see gateway information per VMkernel port, use the following command:
Name IPv4 Address IPv4 Netmask IPv4 Broadcast Address Type Gateway DHCP DNS ---- -------------- ------------- -------------- ------------ -------------- -------- vmk0 10.115.155.122 255.255.252.0 10.115.155.255 DHCP 10.115.155.253 true vmk1 10.115.179.209 255.255.252.0 10.115.179.255 DHCP 10.115.179.253 true vmk2 10.115.179.146 255.255.252.0 10.115.179.255 DHCP 10.115.179.253 true
With separate gateways per VMkernel port, you use port binding to reach targets in different subnets.
Managing iSCSI Network
Special considerations apply to network adapters, both physical and VMkernel, that are associated with an iSCSI adapter.
After you create network connections for iSCSI, an iSCSI indicator becomes enabled in the vSphere Client. The indicator shows that a particular virtual or physical network adapter is iSCSI-bound. To avoid disruptions in iSCSI traffic, follow these guidelines and considerations when managing iSCSI-bound virtual and physical network adapters:
- Make sure that the VMkernel network adapters are assigned addresses on the same subnet as the iSCSI storage portal they connect to.
- iSCSI adapters using VMkernel adapters cannot connect to iSCSI ports on different subnets, even if the iSCSI adapters discover those ports.
- When using separate vSphere switches to connect physical network adapters and VMkernel adapters, make sure that the vSphere switches connect to different IP subnets.
- If VMkernel adapters are on the same subnet, they must connect to a single vSwitch.
- If you migrate VMkernel adapters to a different vSphere switch, move associated physical adapters.
- Do not make configuration changes to iSCSI-bound VMkernel adapters or physical network adapters.
- Do not make changes that might break association of VMkernel adapters and physical network adapters. You can break the association if you remove one of the adapters or the vSphere switch that connects them. Or if you change the 1:1 network policy for their connection.
iSCSI Network Troubleshooting
A warning sign indicates non-compliant port group policy for an iSCSI-bound VMkernel adapter.
Problem
The VMkernel adapter's port group policy is considered non-compliant in the following cases:
- The VMkernel adapter is not connected to an active physical network adapter.
- The VMkernel adapter is connected to more than one physical network adapter.
- The VMkernel adapter is connected to one or more standby physical adapters.
- The active physical adapter is changed.
Solution
Set up the correct network policy for the iSCSI-bound VMkernel adapter. See Configure Port Binding for iSCSI or iSER on ESXi.
