Thursday, September 21, 2017

Parodyne/ESOS on YouTube

Finally! We now have a home on YouTube with demonstration videos of ESOS itself and the Parodyne commercial software products for ESOS. So far we have a cache of about 10 demo / how-to videos from the basics of installing ESOS to a USB flash drive, up to configuring a dual-head ZFS HA disk array, and everything in-between (well, almost everything).

Visit the Parodyne YouTube Channel

For all of our ESOS Commander users with dual-head arrays, these videos serve as an excellent resource, especially for new Parodyne users. Even with a full-featured GUI product, the initial (one-time) configuration of ESOS dual-head arrays (clusters) can be somewhat tedious. Follow these demonstration videos, and please don’t hesitate to contact Parodyne support for any questions, or if you’d like to schedule a screen-sharing session for a guided configuration.

We’ll continue to grow our YouTube channel and upload additional videos as new features are developed in ESOS and the GUI/API. If we’re missing any videos that you’d like to see please contact us and share your ideas / requests! And please, pretty please, if you’re reading this… subscribe to our YouTube channel so we can get one of those fancy short URL’s!

Sunday, May 7, 2017

ESOS Certifies Support for ATTO Fibre Channel Gen 6 HBA's

Enterprise Storage OS (ESOS) by Parodyne is now certified for use with ATTO Gen 6 Fibre Channel adapters! On the ESOS target side, users can now use 16 Gb or 32 Gb Gen 6 ATTO Fibre Channel HBA’s for target interfaces. In addition to the target support, ESOS is also fully supported by ATTO’s MultiPath Director driver on the initiator side. The MultiPath Director software/driver is a full featured MPIO (mulitpath I/O) driver that supports Windows, macOS, and Linux initiators.

How to Get ESOS with ATTO

Parodyne customers / registered users can download the “ESOS with ATTO FC” installation package in the Parodyne Portal. This ESOS image contains the ATTO Celerity 16/32 Gb Gen 6 HBA driver, and the ATTO SCST target driver. Simply boot up your storage array that contains ATTO Gen 6 (16/32 Gb) Fibre Channel host bus adapters and you’re ready to push some serious I/O!

ATTO Adapters are Faster

Yes, vendors claim their products are faster than others, but this is the real deal. We tested the performance of the ATTO FC adapters vs. QLogic HBA’s in our lab. We used (4) Dell PowerEdge servers that each contained a dual-port QLogic 16 Gb FC HBA for our initiator systems to generate load (running ESOS 1.1.1). On the target side, we used a NVMe CiB from Advanced HPC with a dual-port QLogic 16 Gb FC HBA in each server node (2 total) running ESOS 1.1.1 as well. The initiator and target systems were connected via (2) 16 Gb Fibre Channel switches.

We then generated I/O from the (4) ESOS initiator systems using the ‘fio’ tool to generate a 100% random, 100% read, 4K I/O size load to several “null I/O” devices that were configured on the ESOS target side. We ran the fio tool for (30) seconds concurrent on each initiator system and here is what we got:
  • Initiator Host 1: 284,957 IOPS
  • Initiator Host 2: 293,485 IOPS
  • Initiator Host 3: 261,098 IOPS
  • Initiator Host 4: 261,005 IOPS

Total IOPS across all 4 hosts: 284,957 + 293,485 + 261,098 + 261,005 = 1,100,545 IOPS

Okay, so 1.1M 4K IOPS is very nice! Now, we kept our same setup, except on the ESOS target side, we swapped the QLogic adapters for ATTO Celerity 32 Gb dual-port cards (still 16 Gb FC SAN). The QLogic cards in the initiator systems stayed the same for this test, we only swapped the QLogic for ATTO in the ESOS target systems (the Supermicro CiB). Same tests running for (30) seconds concurrently on all four initiator systems:
  • Initiator Host 1: 335,105 IOPS
  • Initiator Host 2: 332,412 IOPS
  • Initiator Host 3: 304,175 IOPS
  • Initiator Host 4: 314,794 IOPS

Total IOPS across all 4 hosts: 335,105 + 332,412 + 304,175 + 314,794 = 1,286,486 IOPS

That’s nearly a 200K IOPS performance increase, just by using ATTO adapters on the target side! Imagine if we were able to swap QLogic for ATTO on the initiator side as well…

ESOS and ATTO MultiPath Director

When using ATTO Fibre Channel adapters on the initiator side, you get to use ATTO’s MultiPath Director on the initiator systems, for free (comes with ATTO FC adapters)! The MultiPath Director is a MPIO driver for your initiator systems, and includes a nifty GUI utility for status/configuration. For the Linux users out there, MultiPath Director is a direct replacement for dm-multipath / multipath-tools. No more messing with or tweaking the multipath.conf configuration file… with ATTO MultiPath Director, it just works!

There is one special configuration change needed on the ESOS target side in order for the ATTO MultiPath Director driver to properly identify the LUN’s (mapped devices). For each SCST device, you’ll need to set the “naa_id” attribute value to a unique NAA ID (Network Address Authority). The ID value must be formatted according to T11 Network Address Authority (NAA) Naming Format (see this page for more information) which is typically a 8 or 16 byte long hex string, which is unique among the other SCST devices. By default, SCST generates a EUI-64 ID for each device, but when setting the “naa_id” attribute, this will cause a NAA ID to be sent in the SCSI INQUIRY response instead.

You can easily set this attribute using ESOS Commander (the ESOS GUI). Browse to the Devices & Mappings tab for a selected host or node. Then select the SCST device that you’d like to set a NAA ID value for. After selecting the device, the device attribute editor appears on the right, browse to the “naa_id” line, and set the value, then click apply. Repeat for each SCST device, and ensure the NAA ID values used are unique among these devices.


ATTO Fibre Channel adapters are an excellent fit for ESOS storage arrays, and easily outperform standard QLogic HBA’s. In addition to being supported on the target side, ATTO HBA’s used on the initiator side with ATTO’s MultiPath Director fully support ESOS-based disk arrays.

Saturday, May 6, 2017

Performance NVMe Active/Active Fibre Channel Array

Recently, we worked with a local college in Michigan on a new highly available (HA) all-flash Fibre Channel storage array. This new high performance storage solution will be used in a mature VMware Horzion View (VDI - Virtual Desktop Infrastructure) environment.

The college wanted to replace their aging single-head SSD Fibre Channel disk arrays, with something faster, and it had to be fully redundant. Their current ESOS disk arrays arrays contain only a single controller head, which does not tolerate a controller failure, and does not allow for non-disruptive updates.

Parodyne and Advanced HPC partnered to develop a new all-flash NVMe dual-head ESOS storage array. We used a Mercury RM240 2U NVMe CiB (cluster-in-a-box). It contains (40) 2.5" NVMe drive bays (20 in the front, 20 in the middle) and two full servers inside the 2U chassis, that can both access the dual-port NVMe drives concurrently.

Hardware/Software Configuration & Testing

The NVMe CiB from Advanced HPC was loaded with (24) Intel 1TB DC D3600 2.5" PCIe NVMe 3.0 2x2 dual-port MLC sold state drives, and each server contains (2) Intel Xeon E5-2690v4 2.6GHz fourteen-core 135W processors, 64GB DDR4 2400 MHz memory, and (1) QLogic dual-port PCIe 3.0 16 Gbit/s Fibre Channel host bus adapter.

The servers were loaded with ESOS (Enterprise Storage OS) 1.1.1 from Parodyne, and configured with md-cluster RAID1, and used LVM to carve up a total of (24) 400GB logical volumes on top of the (12) RAID1 (mirror) arrays (2 logical volumes per RAID1 array).

Before presenting the LUN's across the 16 Gb Fibre Channel SAN, the raw performance of the NVMe RAID1 arrays was tested, directly on the ESOS servers. This gives a theoretical maximum performance number, before testing across the SAN.

(24) 1TB NVMe drives with md-cluster RAID1 (12 mirror sets) across both NVMe CiB server nodes:
  • 100% random, 100% read, 4K blocks: 8,732,041.8 IOPS
  • 100% random, 100% write, 4K blocks: 2,697,346.1 IOPS
  • 100% sequential, 100% read, 4M blocks: 36.4 gigaBYTES/sec = 291.2 gigabits/sec
  • 100% sequential, 100% write, 4M blocks: 10.45 gigaBYTES/sec = 83.6 gigabits/sec

Nearly 9 million 4K read IOPS, and a whopping 36 gigabytes (big B) per second of read throughput! All of that in just 2U's and its fully redundant. We're able to obtain nearly the full performance of the (24) NVMe drives for reads, even though we're using RAID mirroring. With RAID1 (mirroring) we get the performance of both drives in an array for reads, but only one drive for writes. We're still getting excellent write performance as well with nearly 2.7 million 4K read IOPS, and 10 gigabytes per second. And remember, these are MLC SSD's so the write performance is much less compared to reads.

Now let's take a look at the performance across the 16 Gb Fibre Channel SAN. The Fibre Channel SAN consists of (2) Brocade 6510 16 Gb Fibre Channel switches. (4) Dell PowerEdge 730 servers were used on the initiator side to generate the load, and each Dell initiator server has a dual-port 16 Gb QLogic HBA in it. Each Dell PowerEdge server is running ESOS 1.1.1 (for the initiator stack).

Here are the results across the 16 Gb Fibre Channel SAN:
  • 100% random, 100% read, 4K blocks: 1,164,274.2 IOPS
  • 100% random, 100% write, 4K blocks: 909,399.8 IOPS
  • 100% sequential, 100% read, 4M blocks: 7,103 megaBYTES/sec = 56.824 gigabits/sec
  • 100% sequential, 100% write, 4M blocks: 6,850 megaBYTES/sec = 54.8 gigabits/sec

So, with this round of tests, its clear the SAN is our limiting factor. Still, 1.1 million 4K read IOPS from a 2U box across Fibre Channel is nothing to sneeze at! 909K 4K write IOPS were observed, and while a far greater number of write IOPS can be obtained directly on the raw storage, there is much more overhead for writes vs. reads. And for throughput we're seeing 56 Gbps for reads and 54 Gpbs for writes. On the ESOS side, we have (4) 16 Gb Fibre Channel ports, which yields 64 Gb of theoretical bandwidth, so we're very close to that.

ATTO Gen 6 Celerity 32 Gb Fibre Channel Adapters

While testing the ESOS NVMe CiB, we were able to get our hands on some ATTO 32 Gb Fibre Channel HBA's to test on the target (ESOS) side. Recently support for ATTO Fibre Channel adapters was added to ESOS, and now ESOS is fully certified for use with ATTO FC Gen 6 HBA's and switches.

We swapped the two dual-port QLogic 16 Gb HBA's in the ESOS CiB servers for two ATTO Gen 6 Celerity 32 Gb dual-port HBA's. Even though the ATTO adapters support linking at 32 Gb, the SAN we tested with was only 16 Gb Fibre Channel. On the initiator side, we continued to push I/O with QLogic adapters, so the only change that was made in the configuring was swapping the (2) QLogic adapters for the (2) ATTO adapters on the target side. Using the ATTO adapters resulted in a net performance gain of nearly ~200K 4K IOPS! So there is clearly a performance advantage when using the ATTO FC adapters versus QLogic.


Combining Parodyne's ESOS with Advanced HPC's NVMe CiB solution creates a very high performance, cost effective all-flash SAN disk array. ESOS Commander (the ESOS GUI) fully supports this dual-head NVMe solution, from initial cluster setup, all the way through mapping the logical volumes as LUN's. This initial hardware costs are greatly reduced, and the maintenance/support is only a fraction of what's typical from proprietary storage solutions.

For a more in-depth, technical review of the ESOS NVMe CiB solution, please visit this link.

Saturday, April 8, 2017

High Availability (HA) with ESOS

ESOS (Enterprise Storage OS) fully supports creating single-head arrays, as well as dual-head, highly available (HA) disk arrays. There are several different methods or technologies that facilitating creating these HA (dual-head) disk arrays in ESOS. We'll cover each of the methods in this article, along with some of the pros and cons. There are also some differences in supported RAID / logical device technologies when configuring a dual-head ESOS storage array, when compared to options supported in single-head arrays, and we'll cover those here too.

ESOS HA Options

We have three supported methods for achieving high availability in ESOS dual-head configurations:
  • Replication using DRBD
  • Linux md-cluster RAID1 (mirror) arrays
  • Broadcom/Avago/LSI Syncro hardware RAID controllers

These three choices would be used for the underlying back-end storage technology. You can then run Logical Volume Manager (LVM) on top of these back-end storage devices to provision logical storage devices.

Why Limited Choices

Why isn't ZFS, or bcache, or technology X supported for dual-head (HA) ESOS arrays? Its simple: We have to be able to open and access the underlying back-end storage device concurrently from more than one node. You can't mount a standard Linux file system (eg, XFS) on more than one node at a time. You can't activate a ZFS storage pool on more than one node at a time.

"Well, vendor X supports running ZFS in a cluster/HA environment, why can't ESOS do it." Right, we could do that, but those solutions trick the initiators and/or present a faux block device on the non-active/non-primary node, and they use an ALUA policy that tells the initiators that they "shouldn't" read or write data on a path where the target is not in an accessible mode. But what if they do? SCST is supposed to deny these requests and make them fail at that level, but what if the ALUA states are wrong for an instant, and a write is accepted on a dummy device but is actually discarded?Corruption, that's what happens. In ESOS we're playing it safe. In the SCST project there is work being done to better support those types of active/passive back-end storage devices, but its not ready yet. So will we support things like ZFS in a cluster eventually? Probably, yes. But again, for now, we keep the choices limited for performance and safety reasons.

Replication via DRBD

For this HA method, it requires two (2) completely "independent" storage servers. So as an example, lets say we want a 4U 72-drive SAS storage server filled with 2 TB drives. You would use two of these, and data would be replicated synchronously between the two storage servers (each has matching storage amounts/configuration). So you can still make use of things like hardware RAID controllers, RAID5/6, or use MD software RAID with HBA's, or whatever local storage configuration you like on the storage servers.

  • Full redundancy, as you have two fully function servers with all the storage in each; you could even lose a RAID array completely on a node, and the data would still be intact since there is a second node with the same storage
  • Proven reliability with LINBIT's DRBD -- used in data centers throughout the world for critical high-end projects
  • Could even satisfy some minor DR/BC requirements if you put the storage nodes in different rooms or buildings (depending on the connectivity needed)

  • Can hamper performance, especially in the case of random I/O (IOPS) performance, as every write has to be sync'd on the first and second nodes
  • Requires a full matching (recommended) set of hardware, which costs more

Linux md-cluster RAID1 Arrays

This is the popular Linux software RAID but supports opening/accessing (running) an MD RAID1 arrays from more than one node concurrently. Its a relatively new addition to the Linux kernel, but was deemed stable since Linux 4.9.x (what we use in ESOS). Parity levels (eg, RAID5 or RAID6) are not supported with md-cluster. Only RAID1 (mirroring) but RAID1 is the highest performing option for redundancy, and can be combined with LVM striped logical devices to create a RAID10 configuration. The hardware options for setups employing md-cluster RAID1 arrays would be external SAS enclosures that support dual I/O controllers, and using dual-domain SAS disks (HDD or SSD). There are also dual-head SAS CiB (cluster-in-a-box) options which put a shared set of SAS disks and two server nodes in a single chassis -- very slick. And now for the extreme HA high performance option, they have NVMe CiB units (uses dual-port NVMe drives).

  • Using md-cluster RAID1 is the best option for performance, there is some overhead for writes but its tolerable; reads get the performance of two drives, and for writes the performance of a single drive
  • Simple configuration, and uses the well-known and highly regarded Linux MD software RAID stack
  • Makes use of dual-port NVMe drives, dual-domain SAS drives, and dual I/O module SAS enclosures which can even be daisy-chained to get a lot of storage out of one set of dual-head ESOS storage controllers

  • It's RAID1 (mirroring), so you lose half of your storage, which can be expensive
  • Requires the higher-end NVMe and SAS drives which are typically more costly

The Syncro Hardware RAID Controller

Ah, the Syncro... it never even made it to it's prime. This is a hardware SAS RAID controller "kit" which includes two MegaRAID-based RAID controller cards, one for each server. All of the communication between the two nodes for the HA portion is handled by the adapters themselves via SAS connectivity. Unfortunately, Broadcom recently acquired Avago and killed the Syncro line of products. Its such a simple solution, and works well for creating ESOS dual-head arrays, so if you are able to still find one of these controller kits, it'd be a worthwhile investment. Broadcom may not be producing/selling these kits, but they are still obligated to support them for at least the next few years. For the hardware options used with these, a SAS CiB (cluster-in-a-box) or two storage servers each with a Syncro controller connected to dual I/O SAS enclosures with dual-domain SAS drives (SSD or HDD). The SAS enclosures can be daisy chained, and each Syncro controller kit supports up to (120) devices.

  • Simple hardware RAID solution that provides high availability -- just plug in the adapters, and provision MegaRAID virtual drives as usual
  • Supports parity RAID levels (eg, RAID5 or RAID6) as well as the traditional RAID0 / RAID1 levels, and even the nested RAID levels (eg, RAID50, RAID10).
  • Performance is great, although storage should only be accessed by one of the nodes (the owning node)

  • The Syncro kit is somewhat expensive (~ $5,000) and requires the typically more costly dual-domain SAS drives
  • Must use SAS drives (HDD or SSD) that are on the hardware compatibility list
  • It's EOL'd and support is now limited, and no more units are being produced

Logical Devices and Active/Active Configurations

While the above HA methods in ESOS configure the underlying storage devices, and make usable RAID volumes available to both ESOS storage nodes, it doesn't address creating perhaps smaller logical devices from those RAID or back-end storage devices. And of course you don't need to creating sub-volumes from the large storage devices, you could just map those storage devices directly to LUN's. But if you do want to create sub-volume logical storage devices, ESOS fully supports this using Logical Volume Manager (LVM).

Using LVM you'll create physical volumes (PV's) from the DRBD or md-cluster RAID devices, and then create one more volume groups (VG's). Finally you'll create your logical volumes (LV's) on the LVM volume groups. These can be striped logical volumes creating a RAID0 effect. This could be combined with either HA storage technology which may increase performance by striping data across redundant storage devices.

You can creating either an "active/active" storage configuration where some devices are accessed primarily from one node, and another set of devices is primary accessed from the opposing node. Its up to the user to "round-robin" placement or mapping of these devices in different SCST device groups (which controls the ALUA path'ing policy). An active/active configuration would be good if you have matching hardware, with the same performance capability on each.

You can also create an "active/passive" configuration, where all devices are only accessed from one node primarily. This type of setup would be good if you have asymmetrical nodes, where maybe one node is faster or more capable performance wise than the other.

Wrapping Up

Yes, that was a lot to cover, and we wanted to explain the different HA methods used in ESOS, why we use them, and the various pros/cons of each. While these may seem pretty deep, all of the configurations described in this post are fully supported by the ESOS GUI (ESOS Commander). Using the GUI to produce these dual-head ESOS arrays using any of the supported HA technologies couldn't be easiser. Install the ESOS RPC Agent on both controller nodes, install the ESOS Commander application on your desktop machine, and click through the wizards to create your highly available ESOS SAN disk array. All SAN technologies that ESOS supports (iSCSI, Fibre Channel, FCoE, InfiniBand/SRP, and even NVMeoF) work with the HA methods described above.

Wednesday, April 5, 2017


The official Graphical User Interface for ESOS (Enterprise Storage OS) disk arrays is here! Managing single-head and dual-head ESOS arrays is a snap. You can now manage all of your arrays from one easy-to-use interface: Complete storage provisioning functionality, including support for ZFS; performance statistics; guided wizards for common configuration tasks; advanced views for power users; and much more.

How It Works

The GUI (“ESOS Commander”) is installed on a local desktop machine (physical or virtual) running Windows, macOS, or Linux. For each ESOS server that you’d like to manage, a lightweight RPC agent is installed directly on the server which provides the ESOS API. Communication is encrypted between the GUI and ESOS arrays, and X.509 certificates are used for authentication.

And that’s it: Just two components to install and license. The machine that ESOS Commander is installed on can be “shared” by multiple administrators using remote desktop services. No licensing individual features/functions, and no capacity limits -- you get the full power of ESOS.

Guided Wizards

Using the wizards in ESOS Commander makes provisioning storage simple and straightforward, even for the novice SAN administrator. For both single-head and dual-head (clustered) ESOS arrays, the wizard functions guide the user through creating back-end RAID storage devices, carving up logical storage devices, configuring servers/initiators, and mapping the storage devices as LUN’s to the remote servers.

The ESOS Commander wizards make configuring ESOS clusters (dual-head arrays) a breeze. From the initial cluster setup, all the way to mapping devices as LUN’s, the wizards will perform operations on both cluster nodes as needed.

Advanced Provisioning

ESOS Commander also accommodates the power user, for tweaking, and advanced storage configurations. You can use the tabs across the top of a selected host and drill into each functional area for advanced features.

For ESOS clusters (dual-head arrays) the cluster configuration tab provides an advanced interface to the standard Linux cluster stack. You can manage resources (clone sets), set various cluster properties, and even edit the raw cluster configuration and apply the changes live!

More Information

Contact Parodyne today: We can help you spec a custom high-performance SAN disk array, and save you money! Or if you have some spare server hardware in your data center or lab, create an account on Parodyne and try ESOS Commander for free.