Saturday, May 6, 2017

Performance NVMe Active/Active Fibre Channel Array

Recently, we worked with a local college in Michigan on a new highly available (HA) all-flash Fibre Channel storage array. This new high performance storage solution will be used in a mature VMware Horzion View (VDI - Virtual Desktop Infrastructure) environment.

The college wanted to replace their aging single-head SSD Fibre Channel disk arrays, with something faster, and it had to be fully redundant. Their current ESOS disk arrays arrays contain only a single controller head, which does not tolerate a controller failure, and does not allow for non-disruptive updates.

Parodyne and Advanced HPC partnered to develop a new all-flash NVMe dual-head ESOS storage array. We used a Mercury RM240 2U NVMe CiB (cluster-in-a-box). It contains (40) 2.5" NVMe drive bays (20 in the front, 20 in the middle) and two full servers inside the 2U chassis, that can both access the dual-port NVMe drives concurrently.

Hardware/Software Configuration & Testing

The NVMe CiB from Advanced HPC was loaded with (24) Intel 1TB DC D3600 2.5" PCIe NVMe 3.0 2x2 dual-port MLC sold state drives, and each server contains (2) Intel Xeon E5-2690v4 2.6GHz fourteen-core 135W processors, 64GB DDR4 2400 MHz memory, and (1) QLogic dual-port PCIe 3.0 16 Gbit/s Fibre Channel host bus adapter.

The servers were loaded with ESOS (Enterprise Storage OS) 1.1.1 from Parodyne, and configured with md-cluster RAID1, and used LVM to carve up a total of (24) 400GB logical volumes on top of the (12) RAID1 (mirror) arrays (2 logical volumes per RAID1 array).

Before presenting the LUN's across the 16 Gb Fibre Channel SAN, the raw performance of the NVMe RAID1 arrays was tested, directly on the ESOS servers. This gives a theoretical maximum performance number, before testing across the SAN.

(24) 1TB NVMe drives with md-cluster RAID1 (12 mirror sets) across both NVMe CiB server nodes:
  • 100% random, 100% read, 4K blocks: 8,732,041.8 IOPS
  • 100% random, 100% write, 4K blocks: 2,697,346.1 IOPS
  • 100% sequential, 100% read, 4M blocks: 36.4 gigaBYTES/sec = 291.2 gigabits/sec
  • 100% sequential, 100% write, 4M blocks: 10.45 gigaBYTES/sec = 83.6 gigabits/sec

Nearly 9 million 4K read IOPS, and a whopping 36 gigabytes (big B) per second of read throughput! All of that in just 2U's and its fully redundant. We're able to obtain nearly the full performance of the (24) NVMe drives for reads, even though we're using RAID mirroring. With RAID1 (mirroring) we get the performance of both drives in an array for reads, but only one drive for writes. We're still getting excellent write performance as well with nearly 2.7 million 4K read IOPS, and 10 gigabytes per second. And remember, these are MLC SSD's so the write performance is much less compared to reads.

Now let's take a look at the performance across the 16 Gb Fibre Channel SAN. The Fibre Channel SAN consists of (2) Brocade 6510 16 Gb Fibre Channel switches. (4) Dell PowerEdge 730 servers were used on the initiator side to generate the load, and each Dell initiator server has a dual-port 16 Gb QLogic HBA in it. Each Dell PowerEdge server is running ESOS 1.1.1 (for the initiator stack).

Here are the results across the 16 Gb Fibre Channel SAN:
  • 100% random, 100% read, 4K blocks: 1,164,274.2 IOPS
  • 100% random, 100% write, 4K blocks: 909,399.8 IOPS
  • 100% sequential, 100% read, 4M blocks: 7,103 megaBYTES/sec = 56.824 gigabits/sec
  • 100% sequential, 100% write, 4M blocks: 6,850 megaBYTES/sec = 54.8 gigabits/sec

So, with this round of tests, its clear the SAN is our limiting factor. Still, 1.1 million 4K read IOPS from a 2U box across Fibre Channel is nothing to sneeze at! 909K 4K write IOPS were observed, and while a far greater number of write IOPS can be obtained directly on the raw storage, there is much more overhead for writes vs. reads. And for throughput we're seeing 56 Gbps for reads and 54 Gpbs for writes. On the ESOS side, we have (4) 16 Gb Fibre Channel ports, which yields 64 Gb of theoretical bandwidth, so we're very close to that.

ATTO Gen 6 Celerity 32 Gb Fibre Channel Adapters

While testing the ESOS NVMe CiB, we were able to get our hands on some ATTO 32 Gb Fibre Channel HBA's to test on the target (ESOS) side. Recently support for ATTO Fibre Channel adapters was added to ESOS, and now ESOS is fully certified for use with ATTO FC Gen 6 HBA's and switches.

We swapped the two dual-port QLogic 16 Gb HBA's in the ESOS CiB servers for two ATTO Gen 6 Celerity 32 Gb dual-port HBA's. Even though the ATTO adapters support linking at 32 Gb, the SAN we tested with was only 16 Gb Fibre Channel. On the initiator side, we continued to push I/O with QLogic adapters, so the only change that was made in the configuring was swapping the (2) QLogic adapters for the (2) ATTO adapters on the target side. Using the ATTO adapters resulted in a net performance gain of nearly ~200K 4K IOPS! So there is clearly a performance advantage when using the ATTO FC adapters versus QLogic.


Combining Parodyne's ESOS with Advanced HPC's NVMe CiB solution creates a very high performance, cost effective all-flash SAN disk array. ESOS Commander (the ESOS GUI) fully supports this dual-head NVMe solution, from initial cluster setup, all the way through mapping the logical volumes as LUN's. This initial hardware costs are greatly reduced, and the maintenance/support is only a fraction of what's typical from proprietary storage solutions.

For a more in-depth, technical review of the ESOS NVMe CiB solution, please visit this link.


  1. You really should consider using Infiniband for this.

    1. Yeah, for sure... InfiniBand/SRP or some other RDMA technology would certainly yield better performance results across the SAN. The system itself is capable of much more performance than we can push across our Fibre Channel SAN.