RTC Magazine

Industry standard fabric technologies have gained significant traction versus proprietary solutions in the embedded market place, especially for performance-driven applications. Just a couple of years ago, terabit switching performance was reserved only for super optimized (and super expensive) systems based on multi-chip silicon fabrics that took years to develop, tune and eventually productize. Developers realize the costs associated with maintaining these fabrics from both a hardware and software perspective is too high, in addition to the investment required to scale these solutions for future generations.

InfiniBand, an industry standard interconnect technology that is backed by a well supported trade association, has not only established itself as an out-of-the-box interconnect used to cluster commodity servers and storage into an efficient supercomputer, but has also found a home in traditional performance networking and industrial applications. InfiniBand is the only standard fabric that supports up to 60 Gbit/s blade-to-blade performance (by using 12 SerDes interfaces running in parallel at 5 Gbits/s) inside a single chassis and can expand this fabric externally to other systems via copper or fiber cables.

More typical of today’s deployment are links comprised of 4 SerDes interfaces running 2.5 Gbits/s or 5 Gbits/s for aggregate 10 Gbit/s or 20 Gbit/s data rates. A single silicon chip available from Mellanox acts as the switch fabric that supports nearly a Terabit of switching capacity, and multiple chips can be aggregated to easily scale well beyond this. All of this performance is available off-of-the-shelf without having to worry about complex protocol development and costly, time-consuming silicon tape-outs.

One of the major costs associated with proprietary fabrics is the need to bridge from a customized physical and link layer protocol to something standard. Typically expensive, large FPGAs are used to bridge proprietary switch fabrics to a standard bus interface so an NPU or general-purpose processor can eventually do something useful with the data. InfiniBand solves this problem with low-priced, standard silicon adapters the size of a U.S. dime, which connect the fabric to PCI Express. PCI Express has quickly become the de facto I/O on-ramp to processing complexes of blades and has had a profound effect on the proliferation of InfiniBand technology.

Prior to PCI Express, fabrics latched onto PCI-X, which provided blades with I/O throughput up to ~8.5 Gbits/s. A PCI Express x8 significantly increases the bandwidth to processing blades up to 20 + 20 Gbits/s or 40 Gbits/s full duplex. This perfectly balances the bandwidth of an InfiniBand DDR adapter on a blade and enables it to be the highest performing industry standard fabric on the market today. Once this connection is made, commodity x86, PowerPC, MIPS and even specialized network processing solutions can take advantage of the high-throughput fabric and communicate with the rest of the system.

Another problem proprietary solutions face is that once the fabric is available from a hardware perspective, following suit with complex software drivers and communication stacks can take months to debug and tune for maximum utilization and throughput from one blade to another. InfiniBand solves this with open source drivers and upper layer protocol stacks that are included directly with every Linux kernel distribution (version 2.6.11 and beyond). The organization responsible for the development and maintenance of this software support, OpenIB.org, is also working on standard Windows drivers, and several vendors offer WindRiver VxWorks packages as well a plethora of Unix, Solaris and MacOS support.

All of this widely available software has tuned drivers that exploit InfiniBand’s remote direct memory access (RDMA) capability, which allows applications running on processing blades to directly pass large blocks of data back and forth without getting the kernel involved, freeing CPU cycles to focus on application processing. Often referred to as kernel bypass, RDMA enables two applications on different blades in a system to communicate with a measured 2.7us of latency compared to greater than 50us latencies observed on standard Gigabit Ethernet fabrics. The end result is a more efficient embedded computing blade, and ultimately peak performance for the entire system.

InfiniBand in Standard AdvancedTCA Chassis

Combining the InfiniBand fabric with a standard chassis provides even greater benefits to the end user including increased reliability and bandwidth with lower resource utilization, at a much lower cost point than proprietary blade installations. One such vendor, Diversified Technology (DTI), has introduced an InfiniBand ATCA solution compliant to the PICMG 3.2 specification. That company’s Targa B solution features an InfiniBand fabric comprised of a switch card and a CPU blade. By using InfiniBand as the fabric in ATCA, users can immediately benefit from the bandwidth and reliability of InfiniBand in a completely industry standard form-factor.