iSCSI Multipathing: Essential to ensure high availability and bandwidth in your iSCSI SAN
iSCSI connectivity is critical when used for SAN in a datacenter. High Availability and guaranteed bandwidth availability are essential for successful operation with the desired up-time and performance requirements. Different multipathing schemes are used for the iSCSI connections to ensure high availability and bandwidth. Multipathing in simple terms is the availability of redundant physical components like, adapters, cables, switches, to create redundant logical iSCSI paths between an iSCSI initiator and an iSCSI target. Having multiple paths ensures high availability in case of a failure and higher bandwidth. There are three main types of multipathing for iSCSI.
Active/Standby Multipathing
Active/Active multipathing
Link Aggregation: IEEE 802.3ad Link Aggregation Control Protocol (LACP) and EtherChannel
Link Aggregation refers to the concept of aggregating two or more iSCSI links into a single link. IEEE standard 802.3ad and EtherChannel both independently specify a method for aggregating multiple Ethernet links to form a single network link. The packet traffic between an initiator and a target is distributed among aggregated links using a hash function based on a variety of parameters, including source and destination MAC addresses, source and destination IP address, or other. The paths are generally converged at the NIC driver layer or the operating system interface layer. A single IP address is assigned to the set of links on both the initiator and the target.
IEEE 802.3ad standard does not require any particular distribution algorithm but the distribution algorithm used needs to meet the following criteria.
a) Mis-order frames that are part of any given conversation, or
These conditions are met by ensuring that all frames that compose a given conversation are transmitted on a single link in the order that they are generated. This means that the traffic might not be uniformly distributed across the aggregated links. Generally, there are enough conversations to, more or less, distribute the traffic between the aggregated links but if the traffic is composed of only one conversation, i.e. one initiator/destination, then at any given time, only one of the aggregated links will be used. In such a scenario, the effective bandwidth will be equivalent to the bandwidth of a single port. If this is a 10Gb environment and if only iSCSI traffic is passing through this port, this may be OK. Regardless, this will provide redundancy and will make the iSCSI link highly available. This is an important concept: Link Aggregation does not always mean a fatter pipe.
Using a round-robin algorithm, instead MAC or IP hash, would correct this problem, but it does not meet the criteria for the 802.3ad standard. EtherChannel and other nonstandard mechanisms do allow for the use of round-robin distribution. Using EtherChannel would provide the opportunity to get rid of this issue but not all NIC and switch hardware support EtherChannel.
As evident, the end-user performance and redundancy requirements, available hardware and other factors need to be carefully considered before deciding to use Link Aggregation.
Active/Standby Multipathing:
Active/Standby multipathing creates high-availability but does not increase the bandwidth of the solution. In case of Active/Standby Multipathing, iSCSI initiator will initiate a TCP connection per port to the iSCSI target. That means, for two NIC ports, two TCP connections will be established. One of these TCP connections will be the primary connection, which will be active; and other ones will be standby at any given time. Data will flow down the active connection until active connection fails. In case of such a failure, traffic will then be diverted to the standby connection. Since the standby connection is fully established, failover is very fast.
The choice between Active/Standby and Active/Active multipathing also depends upon the iSCSI SAN storage array. Not all array controllers support Active/Active and Active/Standby both. Some might support either of the two and some might support both.
Also, it is important to note that Link Aggregation can be used with Active/Active or Active/Standby multipathing. As mentioned before, Link Aggregation creates a single link by aggregating multiple links. These "aggregated single links" can be multipathed in Active/Active or Active/Standby mode. This is especially useful when some aspect of the design dictates the use of Active/Standby multipathing. In this scenario, multipathing is not going to increase your bandwidth. In this case, Link Aggregation can be used to improve total bandwidth by creating Active/Standby multipathing between two aggregating links of two. For example, if you need 20Gb of storage bandwidth with Active/Standby multipathing, you would need 4x 10 Gb ports in two bundles of two to achieve this.
Active/Active Multipathing:
Active/Active multipathing increases the available bandwidth between an iSCSI initiator and target. In Active/Active multipathing, bandwidth of both of the adapters can be used. This is, of course, dependent upon the algorithm used for load balancing. If the load balancing algorithm performs hashing of IP or MAC, then both of the links may not be used simultaneously all the time. If round-robin or other similar algorithms are used, efficiency simultaneous use of both of the adapters is possible.
Active/Active multipathing is performed by using a software agent (a multipathing driver) in the software stack above the SCSI layer. This multipathing module presents the two iSCSI initiators as a single path or link to the disk driver. As you can imagine, this is only possible when both of the iSCSI initiators are connected to the same end device (a LUN or datastore). This is ensured by using SCSI inquiry commands to check for commonality in device paths, usually the LUN serial number.
As mentioned before, Link Aggregation can also be used with Active/Active multipathing, although it may not provide any significant benefits.
Active/Active multipathing is a great way to ensure redundancy on the iSCSI connection and also to utilize the bandwidth of all available adapters used to create the iSCSI connection. It may not always be possible to use Active/Active multipathing from storage array compatibility, iSCSI adapter support, OS/hypervisor support or cost perspective. When Active/Active multipathing is not possible, Active/Standby or Active/Standby along with Link Aggregation also provide a good way of ensuring high availability and bandwidth for iSCSI connection.