RTC Magazine

Optical backplanes have been long anticipated in the embedded computer marketplace. They have been seen by some as the ultimate solution for higher bandwidth interconnects. With the release of two ANSI-VITA standards addressing backplane optical interfaces, the practicality of backplane-based optical solutions has moved several steps closer to becoming a reality.

Although optical routers and optical networking hardware have become an accepted part of the commercial networking infrastructure, implementation of optical signals in the embedded computing industry has been limited to the front panel, where existing standard optical interfaces are more easily integrated.

Examples of such existing front panel optical interfaces are the Quad Small Form-factor Pluggable (QSFP and QSFP+) specifications developed by the Small Form Factor Committee (SFF), an ad hoc industry group that has addressed a variety of computer hardware specifications. Some well accepted serial protocols have established optical roadmaps such as Fibre Channel and InfiniBand. In addition, a wide array of industry groups and technical conferences address optical interfaces and optical computing: Optical Internetworking Forum (OIF), National Fiber Optic Engineers Conference (NFOEC), The International Workshop on Optical Supercomputing (OSC) and even VITA’s Optical Computing Forum (Op-Comp), to name a few.

With all the various industry efforts related to optical interfaces and optical computing, why does the embedded computing industry continue to depend exclusively on copper backplane interconnects? The answer is that thus far, electrical signaling has provided the performance needed at a reasonable price. In addition, the 12- to 24-month development cycle typical of the embedded computer industry has been too short for optical solutions to displace electrical signaling solutions.

Two new standards released in 2011, ANSI-VITA 66.0 (Optical Interconnect on VPX – Base Standard) and ANSI-VITA 66.1 (Optical Interconnect on VPX – MT Variant) are laying the groundwork for migration to optical technologies within the embedded computing industry. These two standards are the first in a family of documents that will define a variety of backplane optical interfaces compatible with the existing Eurocard form factor used by the VPX architecture (Figure 1). Since the Eurocard architecture is utilized by a number of other popular embedded computing standards, these new optical connector interfaces have the potential to rapidly migrate to other platforms if they first become established within the VPX community.

There are three basic system implementations that are now possible based upon the VITA 66.1 interconnect: 1) fiber optic I/O from the chassis to external points such as sensor arrays, 2) direct slot-to-slot fiber optic links providing high bandwidth paths between pairs of cards within a system and 3) switched optical data planes with multiple payload cards connected to an optical switch slot.

It is important to understand that the VITA 66.1 interconnect is not entirely new. It is based upon the existing well-established MT and MPO ferrule technology first introduced as an EIA standard around 1993. VITA 66.1 defines a VPX-compatible module that positions the MPO multi-fiber ferrule with sufficient precision to allow it to be used as a backplane interface for conventional VPX plug-in cards. The VITA 66.1 standard provides for up to two MT ferrules in a 1-inch long module. Each of the two MT ferrules can hold up to 24 optical fibers.

This means that a 6U VPX module could support up to six VITA 66.1 modules for a total of 288 optical fibers. Of course this represents a theoretical maximum and it is not expected that any card would implement more than one or two modules for a total of between 24 and 96 optical fibers. The reason for the wide range is that although an individual MT ferrule today can hold up to 24 fibers, 12 is a much more common implementation.

Another important aspect of the VITA 66.1 technology is that it only defines the positioning of MT ferrules. The MT ferrule allows the use of a variety of fiber types and these fibers could be driven by increasingly capable optical engines. So, although the bandwidth of each fiber with today’s optical engines is typically 10 Gbaud, 14.4 and 28 Gbaud optical engines are becoming available, and in the future multi-mode fiber systems could support far greater data rates—and this without changing the installed VITA 66.1 connector modules.

Of particular interest to the VITA community may be the InfiniBand fourteen data rate (FDR) protocol. This protocol defines both an optical and electrical implementation, each supporting a bandwidth of 14.4 Gbaud. It is known that there are simultaneous development efforts within the VITA community to support both an optical and an electrical implementation of the InfiniBand protocol. The electrical implementation of InfiniBand FDR may represent the maximum possible bandwidth for the existing VPX copper backplane technology. However, InfiniBand FDR is clearly just the beginning of the optical bandwidth capability.

One of the first OpenVPX systems to incorporate VITA 66.1 backplane optical I/O at the InfiniBand FDR rate is CSPI’s TeraXP Embedded Server. This card, shown in Figure 2, incorporates an InfiniBand FDR, 56 Gbit/s Host Channel Adapter with failover capabilities via a 56 Gbit/s QSFP transceiver on the front panel or the VITA 66.1 optical interconnect to the backplane.

Several other aspects of fiber optic I/O will be realized with the adoption of the VITA 66.1 standard. In addition to the supercharged bandwidth, optical signaling is also much lighter than copper cabling. This weight savings could be a significant advantage for vetronic- and aerospace-mounted VPX systems using optical I/O. In addition to weighing less, optical fibers are also immune to EMI-RFI interference or snooping. This means that once a decision is made to implement fiber I/O through the backplane, the end customer benefits immediately by reduced cable weight and improved noise immunity.

Another looming technical hurdle facing VPX systems that may be resolved with the implementation of VITA 66.1 fiber optic I/O is related to the limitations of copper I/O cabling. Copper cabling can carry 1000Base-T Gigabit signals for 1000 meters. At InfiniBand FDR data rates, the practical distance for copper I/O cabling drops sharply to 7 meters. It should be clear that copper cabling is running out of steam for high-performance devices at InfiniBand FDR data rates, and that optical cabling will be a natural solution for I/O at faster data rates.

The higher speeds supported by optical interconnects along with the necessary copper connections for power, control plane and system management should fit the basic requirements for high-performance computing.

Another architecture that is often considered for super computing applications is the n-dimensional hypercube. It turns out that the 6U VPX architecture lends itself to some interesting implementations of n-dimensional hypercubes. With a choice between the VITA 46 electrical interface and the VITA 66 optical interface, either copper or optical hypercube topologies are quite possible.

Because hops in a network of processing nodes are directly related to system latency, meshes arranged as hypercubes are of great interest because of their inherent low latency. And because InfiniBand is an architecture with especially low latency, InfiniBand, hypercube meshes and VPX would seem to be meant for each other.

First some math relating to n-dimensional hypercubes, Table 1 lists some topological features of various hypercube implementations based on four nodes per slot. The column labeled “exits per card” is the number of bi-directional links required for each implementation. This number is the number of fiber channels required from each card to the backplane.

Let’s start with a simple but practical example. Figure 3 illustrates a 4-dimension hypercube. In such a topology, there are a total of 16 nodes. These nodes can be thought of as vertices of two nested cubes or as processors in a super computer. In either case, a 16-node configuration requires only four hops to travel between any two nodes.

If you locate four processors on each plug-in card in a 6U VPX system, you will only require four cards to support a 16-node computer. With this arrangement, each slot will only require eight I/O channels to the backplane, because some of the paths between nodes are entirely within the bounds of a given card. The 16-node, 4-dimensional hypercube topology is illustrated in two different ways. Figure 3 illustrates the hypercube as two nested cubes. Figure 4 illustrates the 16-node hypercube as four cards each containing four nodes. The numbering and positions of the nodes in relation to each other is the same in both illustrations.

Figure 3
Topologically, a 4-dimensional hypercube can be represented as two nested cubes requiring only four hops between any two nodes.

Figure 4
A more physically oriented diagram of the connections in a 16-node, 4-dimensional hypercube.

For a second example, let’s consider a 6-dimension hypercube that would contain 64 nodes. If arranged as before with four nodes located on each card in a 6U VPX system, you would need a total of 16 cards. This is exactly how many cards a typical backplane would accommodate with the cards on 1-inch centers. In this arrangement with 16 slots, you would also need 16 I/O channels from each card to the backplane. This is very practical for either optical or electrical connections from a 6U VPX card.

In an electrical implementation, if each I/O channel is comprised of one Fat Pipe (x4 bi-directional link), only four of the six available modules would be occupied by the data-plane on each slot, leaving two modules for a control plane and some user I/O.

The optical implementation would be even more efficient because the 64-node hypercube would only require 16 optical channels (32 fibers) and VITA 66 defines a module with as many as 48 fibers in a single module. Of course you could utilize optical channels, each compromised of six bi-directional links (six fibers each for a total of 192 fibers) that would occupy the same four modules in the electrical example and have a much greater bandwidth.

The 64-node structure is illustrated in Figure 5. However to show it in a flat 2-D format, you must imagine that each link exiting at the top of the illustration tunnels through space and enters at the bottom of the illustration. The same thing must be imagined for the links exiting at the left edge of the illustration; they connect directly at the far right edge of the illustration. If the flat 2-D illustration were stretched and pulled to bring the top and bottom edges together and at the same time stretched to bring the right and left edges together, the resulting hypercube would look like a donut.

Table 2 shows what the above structure would provide at the various data rates currently offered within VPX and planned for the near future. This same chart shows the control, data and expansion data rates as well as expected optical data rates. Here we will only consider the electrical and optical data planes. At the present point in time, the InfiniBand Trade Association has not released full electrical specifications for InfiniBand FDR. However, silicon for electrical devices and for optical devices has been available on the market for nearly a year.

Some companies are already advertising InfiniBand EDR optical devices, however many believe that electrical devices beyond FDR are not going to be practical for use with the VPX architecture unless there is a new or significantly modified connector. Table 2 shows proposed data rates for 2013 and a guess at the future beyond 2013. Note that data rates beyond 14.4 baud are only projected for optical devices for VPX.

Figure 6 illustrates three OpenVPX slot profiles. Profile A represents the slot profile needed to support the 64-node hypercube with copper Fat Pipe channels. B and C represent the same in optical where slot profile B would support an optical channel comprised of a minimal two fibers each, and slot profile C represents an optical profile where the 64 nodes are connected by channels comprised of six fibers each.

The bandwidth that could be achieved for each of the three slot profiles at various future data rates are shown in Table 3. This would be a typical implementation of a 64-node hypercube with four nodes per card and 16 cards per system.

A typical payload card today with four Fat Pipes (x4) allocated to the data plane in a backplane that supports the fastest data rate currently defined in ANSI-VITA 65 R2012 would support four 25 Gbaud channels per slot. Compare that to a similar slot profile with a single VITA 66.1 optical module with 32 fibers, driven with InfiniBand enhanced data rate (EDR) silicon. The latter card would have an expected bandwidth of 416 Gbaud!

It is still not clear when the first VPX optical hypercube mesh architectures are going to appear. However, with the release of ANSI-VITA 66.0 and 66.1 it is clear that the components are in place to support a variety of optical backplane I/O applications. With sensor arrays approaching PCIe Gen 3 speeds, optical I/O may be the only practical interconnect to support such devices. 

Elma Electronic
Fremont, CA.
(510) 656-3400
www.elma.com

CSPI
Bellerica, MA.
(978) 663-7598
www.cspi.com