RTC Magazine

Following the debut of PCI Express-based graphics applications in 2004, a wide array of product categories have adopted the technology. Server and storage systems, for example, have taken advantage of PCI Express (PCIe) bandwidth and scalability. But while PCIe technology had already been expected to expand to other market segments, no one anticipated just how widespread its deployment would be in areas such as communications, embedded systems, home entertainment and consumer electronics, for which a steady stream of products are now rolling out. Driving this rapid adoption across product categories is the ever-increasing demand for bandwidth, designers’ preference for pain-free migration from the conventional PCI architecture, and an abundance of various PCIe devices becoming available.

PCIe technology was created back in 2002 when major system OEMs realized that existing bus-based interconnect technologies such as conventional PCI and PCI-X had reached their peak. What resulted was a 2.5 Gbit/s serial interface with several advantages over conventional bus-based interconnect technologies. Table 1 outlines these advantages.

Since releasing that first PCIe specification in 2002, the PCI Special Interest Group (PCI-SIG) has introduced several enhancements to the PCIe base specifications to propel its use in a surprisingly wide range of additional applications. The following is a brief description of these enhancements:

1. Gen2 PHY Layer: A new physical interface for 5 Gbit/s links (Rev. 0.9 released). This allows high-performance computing to scale to the next level of performance. The systems equipped with Gen2 devices are expected to be released by mid-2007.

2. Cable Specification: The specification to connect systems through PCIe cable (Rev 0.9 released). This allows systems with PCIe ports to be connected together within the range of nine meters. Pre-standard cables have been shipping for almost six months.

3. I/O Virtualization (IOV) Spec: The specification for implementation of PCIe in multi-host and shared-I/O environments such as blade servers (Rev 0.5 released). This will reduce the cost of ownership and management for blade servers as the I/O’s resources will be shared amongst different CPUs. The spec is expected to be completed in the second half of 2007.

From servers to video/graphics applications to embedded systems, PCIe technology fits the needs of an ever-increasing number of industry segments—some highly anticipated, while others less so. In these applications, it provides scalable bandwidth between the CPU and I/Os in servers; matches the SAS/SATA and Fibre Channel (FC) data rates in storage; provides high-speed links to control or packet processors in communication; links real-time audio/video processors in consumer applications; and facilitates high-bandwidth serial links in many embedded applications.

Rackmount and Blade Servers

Most servers would fall in one of two classes—I/O servers and compute servers. I/O servers tend to have one to two CPUs and many I/O slots, or devices that connect to I/O resources, such as storage and communication devices. On the other hand, the compute servers have a lot of processing power (four or more CPUs) with fewer I/O resources. Typically, servers come in a 19-inch-wide 1-4U high rackmount casing and a comparatively new form-factor called the “blade server.” Blade servers are becoming increasingly popular as they offer cost savings over rackmount servers as well as lower power consumption, smaller form-factor and ease of management.

Servers started the transition to PCIe to take advantage of the bandwidth, scalability, broad ecosystems and ubiquity of this technology. The servers shipping today offer some PCIe slots, and while the majority of the slots are still PCI-X, we’re seeing a steady trend toward less PCI-X and more PCIe as chipset vendors eliminate PCI-X interfaces from the chipset.

Most chipsets on the market today offer a fixed number (three or four) of PCIe ports. However, servers used in I/O-intensive applications such as storage require more ports. PCIe ports on server motherboards can be increased by using off-the-shelf PCIe switches, with various lane and port counts, from vendors such as PLX Technology. Figure 1 illustrates the use of a PCIe switch to create more high-speed slots in a server application.

Typically, blade servers provide several interfaces such as SAS/SATA, Fibre Channel and Gigabit Ethernet (GE) to connect to network and storage devices that are not shared by all the blades. As mentioned above, the new PCIe protocol enhancement IOV will allow reduction of these interconnects and sharing of I/O resources. This will significantly reduce cost of server procurement, maintenance, support and management.

PC Graphics

PC Graphics is the key driver of PCIe technology, as it has brought economies of scale to reduce the cost of components. A large majority of high-end PCs manufactured in 2006 were built to support a x16 PCIe slot enabling 3D, high-resolution graphics.

PC-based video gaming has been growing by leaps and bounds, and graphics chip vendors are providing cutting-edge performance through high-resolution graphics-processing units (GPUs). High-end graphics is also becoming important for scientific, entertainment and engineering communities, as more and more applications take advantage of this graphics revolution. The GPU manufacturers are pushing the envelope for gaming enthusiasts by creating two x16 PCIe slots to install two GPUs to drive a single monitor for the ultimate gaming experience (Figure 2).

Video Distribution

Multiple-monitor computing is in the early stage of its growth curve and is about to take off as a major trend. The factors that are playing key roles in this development are PCIe technology, newer operating systems, lower memory prices, enhancements in LCD technology, monitor prices and enhanced GPU devices.

Traditionally, these systems were used by the financial industry and graphics professionals, but due to a recent decline in the costs, a new generation of users and applications has emerged. These include working between multiple applications and their interfaces in addition to addressing the need to efficiently and simultaneously view and process a multitude of information sources. In utilizing multiple monitors, users can move and size a variety of information and images across any or all screens to increase efficiency.

Many modern systems depend on high-speed connectivity between chips, modules and systems. In video and graphics applications, the PCIe interface provides the connectivity between the GPUs and the processor (or chipset). The switches are being used to expand the PCIe port on the host processor or the chipset to connect to multiple GPUs. This allows motherboard and card manufacturers to create more graphics ports (slots) for multi-monitor systems. The scalable bandwidth of PCIe allows for matching the I/O bandwidth with the performance requirements of the end systems.

The applications taking advantage of multiple monitors include spreadsheet analysis, desktop publishing, tool palette storage, CAD, CAM, CAID, project tracking, Web design, gaming, game development, model design, trade show presentations, presentation systems, financial analysis, stock trading, software development, simulation, videoconferencing, animation, video/audio editing, technical research and video editing.

Storage Systems and Routers

A typical storage system depends on high-speed connectivity between CPU, memories, I/O chips, modules and storage devices. In many of today’s systems, PCIe provides the connectivity between the storage interfaces, such as FC, SCSI and SATA, and the processors that control or manage the storage system.

Fibre Channel host bus adapters (HBAs) are dominant in enterprise storage systems. Traditionally, the HBAs with FC, SCSI, SATA and other interfaces have used conventional PCI or PCI-X to connect to the host bus; the HBA would connect to the host through chipsets (for x86 architecture) or directly (for RISC processors). The task of designing a system with conventional buses like PCI/PCI-X becomes more challenging as the bus width and clock speed increase to support the increased CPU speed and storage interface data rates. Furthermore, to increase the number of slots, additional PCI/PCI-X bridges are needed. These bridges bring additional cost, noise, complexity, board space and latencies. This creates room for a serial switching technology like PCIe.

PCIe switches are being used to expand the PCIe port on the storage system’s host board to connect to multiple components or ASICs on the HBA. Most FC HBA vendors are migrating to PCIe, as it offers serial interface for better board design and higher and scalable bandwidth that matches the FC line rate requirements of 1 Gbit/s, 2 Gbits/s, 4 Gbits/s and 8 Gbits/s. There are many FC HBA and SATA cards available from multiple vendors today.

Internet connectivity depends on the router’s ability to move information (packets) between users, computers and remote systems (Web surfing, email, ftp, etc.). High-end routers process millions of packets per second in order to support an ever-increasing demand for speed and real-time response by modern applications. These routers perform deep packet processing for authentication, security, quality of service, reliability, route optimization and network management. Several application-specific processors or custom ASICs are involved in packet processing. These processors and ASICs need to be interconnected through efficient high-speed, chip-to-chip or board-to-board interconnects. Typically, routers consist of line cards, router modules and control modules. An example of a line card with PCIe as an interconnect technology is shown in Figure 3.

Industrial and Embedded

PCIe technology is being adopted by many standards bodies in the industrial and embedded spaces, such as the Advanced Telecom, MicroTCA system and AMC specifications. The AMC family of specifications defines a form-factor that utilizes high-speed serial interface technologies such as PCIe. This form-factor can be utilized to support AMC modules in MicroTCA systems, carrier module for AdvancedTCA systems, or custom chassis. The AMC modules are being used in many embedded applications such as communication systems, medical equipment, cellular base stations, and imaging systems.

The PCIe interface provides this connectivity on chipsets used with x86-based architecture, MIPS processors and PowerPC processors. PCIe switches are being used to expand the limited number of ports available on the chipsets or the processors. Some systems are taking advantage of the switches’ peer-to-peer capabilities to develop a system backplane or switch fabric in order to connect to multiple I/O devices.

Most router vendors have used conventional PCI to connect to subsystems of the router for management and control. Recent bandwidth and processing demands on the management modules are forcing designers to look for faster interconnect technologies. PCIe technology, thanks to its software compatibility with PCI, provides this capability without a forklift upgrade of the network operating systems. PCIe switches are being used to connect the embedded and network processors, as most of them offer PCIe interfaces.

Traditionally, security systems have been limited in their scope and functionalities, such as simply detecting open doors and windows. In the last few years, these systems have grown in functionality, providing video surveillance at many points of the secured area through wired or wireless cameras. With numerous high-resolution cameras comes an extensive need for increased bandwidth and throughput in the systems. For example, a frame-grabber board takes video images from the cameras, processes them locally and feeds the info to the host for analysis and action. This requires multiple high-speed ports to aggregate to the CPU or host. PCIe switches fit this application perfectly, as the technology provides high-speed, point-to-point connections to end-points with aggregation to the host (Figure 4).

PLX Technology
Sunnyvale, CA.
(408) 328-3500.
[www.plxtech.com].