RTC Magazine

Those with a long memory will recall that InfiniBand (IB) was once touted as the replacement for PCI, destined to bring a new high-performance, high-availability I/O model to the enterprise. The advantages of a switched serial interconnect seemed overwhelming compared to the limitations of PCI’s parallel bus structure and the PCI tree topology. As it turned out, the weight of PCI’s infrastructure and its software legacy made it too heavy to unseat. The need for a complete new infrastructure, both software and hardware, proved to be an insurmountable barrier. After suffering through a long gestation period, the IB camp suffered attrition coincident with the bursting of the tech bubble. Just when we had written it off, signs began to appear that IB had secured a niche. Now the surviving IB players are enjoying its success as a cluster interconnect in high-performance computing.

When it became clear that IB would not replace PCI, the need remained to re-invent PCI. The lessons of the recent past were applied to this task (by the usual suspects), and out of the ashes of the original IB vision arose PCI Express (PCIe). PCIe uses a PHY very similar to IB but retains its PCI heritage. PCIe is configured just like PCI; its topology is a tree of PCI-to-PCI bridges in which routing is performed by address and device ID (bus, device, function number). With PCI-to-PCIe bridges, legacy PCI devices can be employed with good performance using unmodified legacy software. PCIe was driven to success by implementation on Northbridges and graphics processors, followed quickly by storage and network adapters for the enterprise. While systems still ship with a mix of PCI and PCIe slots, it is abundantly clear that PCIe is replacing PCI and AGP, and will soon be as ubiquitous as PCI once was.

PCIe was initially set on a parallel technological track with IB but pursuing an entirely different set of applications and market segments. PCIe provides cost-optimized but still high-performance I/O for the desktop and enterprise limited to a single root complex, while IB provides performance-optimized cluster interconnects scaling beyond 10,000 processing nodes.

The embedded market has a history of taking enterprise and desktop technology and adapting it to its needs. The economies of scale thus leveraged are compelling. The storage and Ethernet controller chips developed for the enterprise and the desktop are used almost everywhere such interfaces are needed in the embedded space. X86 processors compete for tasks formerly performed by embedded and communications processors. When PLX applied non-transparent bridging, which is standard in PCI, to PCIe switches, it allowed PCIe to expand beyond its nominal single-host limit to support multiple hosts, failover and systems with redundant fabrics. In addition to non-transparent bridging, two PCIe specification development activities at the PCI-SIG—I/O Virtualization (IOV) and the PCIe Cable Standard—will allow PCIe to encroach further into IB’s interconnect space. However, it’s important to look at some of the key elements of these developing specifications to see just how far the new capabilities extend.

I/O Virtualization

For the past 18 months, the PCI-SIG’s IOV work group has been developing a standard that will allow the sharing of I/O adapters by multiple hosts, as well as by multiple system images running on a single host. IOV, whose specification the PCI-SIG expects to complete by the end of 2006, is part of a larger trend toward virtualization throughout the enterprise. Servers themselves are being virtualized to reduce maintenance costs, increase system resiliency and make better use of multi-core processors. When a server’s workload is divided among applications with each running under its own guest operating system, then fault and error-side effects can be constrained to just a single application. IOV supports that trend by allowing separate virtual I/O devices, within a single physical I/O component, to be assigned to each guest OS, thus limiting the scope of I/O errors to a single guest OS or system image.

I/O sharing and IOV are two sides of the same coin. I/O sharing is enabled when a single I/O device is made to look like many, primarily by giving each virtual instance of the I/O function its own set of control and status registers (CSRs). The ability to share I/O devices among blades provides a compelling advantage for PCIe as the backplane interconnect for blade servers. I/O can be removed from the blades and direct connections made from the root complexes to the backplane switch, saving both cost and latency. Throughput is increased by stepping up from 1 Gbit/s to x4 or x8 PCIe. The cost of 10 Gbit/s I/O adapters is amortized over all the compute blades in the backplane. The IOV standard minimizes the software impact of an otherwise revolutionary advance.

To share virtual instances of an I/O function among multiple hosts, one must create a multiple-host-aware fabric. The direction chosen by the PCI-SIG is to extend packet headers with a host ID field, which are added/removed at the host ports of switches to allow the use of legacy root complexes. Multi-host-aware switches implement a separate CSR space, including address and ID routing information, for each host using the host ID to select the routing information for each packet as it passes through the switch. Multi-host-aware I/O controllers use the host ID as part of the address for incoming packets and attach it to outgoing packets to allow them to be routed upstream. The result is that each host sees a virtual PCI hierarchy fully compatible with that of a standard, single-host PCIe system. The multi-host fabric contains multiple virtual PCI hierarchies overlaid within a single physical switch or fabric, as illustrated via the color coding of Figure 1.