PCI Express Extensions Beyond the Motherboard Chipset

Fractal Realms series. Backdrop of fractal elements, grids and symbols on the subject of education, science and technology

Though there have been some initial delays in the widespread introduction of
PCI Express, the high-speed interface is well on its way to becoming one of
the dominant serial interfaces used in semiconductor devices. The vast majority
of devices using this low-cost, high-bandwidth interface will be concentrated
in the PC and server motherboard markets, driven by Intel’s push to make
this the dominant data path interface used in the computing industry.

As one might expect, many companies are looking to leverage this low-cost interface
across a wide variety of applications that require high-speed serial connectivity
outside of the desktop computing space. Though the price point of some of these
other applications dictates the use of embedded links, several high-volume applications
can benefit greatly from using a flexible, low-cost programmable external PCI
Express solution to deliver targeted benefits for specific applications.

PCI Express is a third-generation PCI I/O interconnect serial interface with a
signaling rate of 2.5 Gbits/s (data rate of 2 Gbits/s) per link that offers system
architects and designers opportunities for higher performance and smaller, lower
cost systems, while maintaining a software model that supports legacy PCI drivers.
This PCI-compatible software model enables PCI Express-based systems to boot existing
PCI-based operating systems and support existing PCI I/O device drivers without
any modifications. This is a key aspect that enables rapid PCI Express adoption
in servers and workstations along with desktop and mobile computers. Thus, PCI
Express can quickly be adapted in the hardware solution to offer several technical
advances over the legacy PCI interconnect. These advances include:

  • Bandwidth scaling by aggregating 2, 4, 8, 12, 16 or 32 lanes to
    achieve a maximum data rate of 80 Gbits/s
  • Elimination of trace matching requirements, which reduces cost by
    simplifying board design
  • Enabling box-to-box interconnects via a connector or cables
  • Support for quality-of-service (QoS) differentiated services
  • Support for multiple power management levels
  • Hot-plug and hot-swap support without any requirements for sideband
  • Support for peer-to-peer communications messaging across large fabrics
  • Support for end-to-end data integrity to support high-availability
  • Support for advanced error-handling and reporting

In addition to embedded computing applications, PCI Express is also targeted for
use in communications, storage and embedded communication systems applications.
As mentioned earlier, the majority of PCI Express ports will be shipped in the
core logic chipsets of PC and server motherboards. This is the result of Intel’s
technology roadmap that started with the introduction of PCI Express technology
in its high-end server and workstation chipsets.

These chipsets, first publicly demonstrated at the September 2003 Intel Developers
Forum (IDF), used PCI Express to double the maximum performance of the graphics
interface (from 2 Gbits/s using AGP8X to 4 Gbits/s using PCI Express) to support
high-bandwidth applications such as multimedia, computer-aided design and digital
video editing. This demonstration laid the groundwork for the introduction of
PCI Express into the

desktop and mobile platform chipsets by mid-2004, which will be the main drivers
for the 2 billion PCI Express ports expected to ship by 2008 (note—the legacy
PCI bus has shipped over 1 billion ports since its introduction in 1992).

PCI Express on the Motherboard

Inside a PC, notebook computer, server or workstation, the motherboard contains
the main semiconductor components and connects to the peripheral components
attached to the unit through a variety of different interconnects. The processor
is the main brain of the motherboard and is connected to a North Bridge that
serves as an interface bridge between the processor, the main memory and the
graphics chip (Figure 1). Both the processor and the graphics chip need low
latency access to memory and use the North Bridge device as the low latency
interface to the main memory.

This North Bridge device also connects to a South Bridge, which serves as an interface
bridge to different components that either plug into slots on the motherboard
or are connected to external peripherals. Because there are many different types
of devices that can connect to the South Bridge—such as integrated hard
drives and CD-ROM drives, external printers through USB ports and LANs through
Ethernet ports—the South Bridge device is sometimes called an I/O bridge.

The basic architecture for connecting the CPU to the main memory, the graphics
chip and other peripherals using the North Bridge and South Bridge (sometimes
these two chips are referred to as the chipset) represents the general makeup
of the major semiconductor content of the motherboard. The interconnects between
the CPU and the North Bridge and the North Bridge and the South Bridge devices
are proprietary. This is due to the fact that these devices are soldered to the
board and are not intended to be replaced or upgraded after the motherboard has
been manufactured. As seen in Figure 1, the CPU is connected to the North Bridge
through the Frontside Bus interface and the North Bridge is connected to the South
Bridge through another proprietary interface, sometimes called the Direct Media
Interface (DMI).

Outside of these proprietary interfaces, the other motherboard interconnections
need to be standardized as the memory, graphics, disk drive, CD-ROM drive and
external peripherals will all be manufactured by other vendors and will thus come
with industry standardized interfaces. The memory interface interconnects will
be some flavor of DDR (DDR266, DDD333, etc.), which is the industry standard double
data rate memory interface. The graphics device has traditionally been connected
using the Accelerated Graphics Port (AGP).

However, with the introduction of the Intel 915 (formerly code named Grantsdale)
and the Intel 925X Express (formerly code named Alderwood) PC chipsets, the
connection to the graphics chip is now comprised of a 16-lane PCI Express channel
(Figure 2). In addition, PCI Express has also been integrated on the South Bridge,
which has two PCI Express lanes connecting to two motherboard slots. Because
of the aggressive pricing of PCs and notebooks, these chipsets are both offered
well under $50 in high volumes. Given the standardized architecture of the motherboard
and the challenge to meet aggressive cost targets in this space, flexibility
is not needed on the motherboard and thus the PCI Express links need to be embedded
in both the North Bridge and the South Bridge to meet the cost targets in this

PCI Express in Other Applications

Many companies are leveraging the adoption of PCI Express in the computing space
to introduce this low-cost serial interconnect in communications, storage, test
and medical equipment applications. For some of these applications, such as host
bus adapters connecting a server to large storage arrays, the protocol translation
and interconnect are defined in such a way that an embedded PCI Express solution
makes sense. For example, connecting a server to an array of Serial ATA drives
can make use of host bus adapters that translate PCI Express to Serial ATA using
controller devices from companies such as Intel, Marvell or Broadcom.

In some of these applications, companies will be looking to connect PCI Express-based
devices to devices using legacy interconnects. For these systems, an embedded
PCI Express solution may not be possible, requiring a bridging solution instead.
The most flexible bridging solution will be a programmable bridge that can implement
the required connectivity along with any protocol conversions that may be required.
This programmable bridge can be implemented using an FPGA device and external
PCI Express PHY device that address the required bandwidth, jitter and power requirements.

Though this two-chip solution may appear to be quite expensive, proven solutions
exist that meet key price points to enable support for these applications. For
example, a simple two-chip bridging solution from a single-lane PCI Express connection
can be obtained for under $15 in high volume. An extremely flexible two-chip solution
that is able to support integration of additional functionality in addition to
the bridging functionality is available for under $20 in high volume.

Beyond choosing a programmable solution that meets the required price points,
there are some additional technical considerations that need to be investigated
with a two-chip solution. In addition to the usual jitter, return loss and power
considerations that come when selecting a PCI Express PHY device, one must ensure
that the connectivity between the PHY device and the FPGA device is robust. In
many discrete PCI Express PHY devices, the slow-speed parallel (host) interface
is based on the PHY Interface for PCI Express (PIPE) interface defined by Intel.
This interface was defined mainly for internal connectivity and has some potential
technical issues when used as an external interconnect. For example, all transmit
and receive data transfers across the parallel interface are synchronized to the
parallel interface data clock (PCLK in Figure 3). This makes the off-chip clocking
requirements quite stringent, requiring a round-trip clocking time of either 8
ns (for a 16-bit interface running at 125 MHz) or 4 ns (for an 8-bit interface
running at 250 MHz).

There are some solutions available today that address these challenges. One such
proven solution is shown in Figure 4. Here, an FPGA is connected to an external
PHY device using the 8-bit 250 MHz SSTL-2 interface. The TXCKL clock used in this
solution is a source synchronous transmit clock, which addresses the challenges
that synchronous clocking faces when attempting to match the clock distribution
paths between the two devices. Using one additional clock for the transmit path
enables clock generation in the same physical location as the transmit data, creating
a tight timing relationship between the transmit clock and data. In addition,
the FPGA device implements the necessary logic to address the requirements for
the protocol handling as well as provide the programmable flexibility to address
many different PHY signaling levels. Such programmable two-chip PCI Express endpoint
solutions have been successfully demonstrated at industry tradeshows and are also
able to support the extremely aggressive cost requirements in the applications
seeking to leverage low-cost PCI Express solutions.

The programmability of these solutions also enables new developments to support
a multitude of interconnects and functionality. First, a programmable two-chip
PCI Express solution enables companies to eliminate development costs and accelerate
product delivery to the marketplace. In addition, the programmable combination
lengthens the time that products are in the marketplace as companies can adapt
to changing product and customer requirements by simply reconfiguring the programmable
solution. Also, companies are able to employ design reuse techniques to reduce
required inventory levels because the programmable flexibility of the FPGA-based
solution enables a single two-chip solution to address many different end product

With the helping hand of Intel, PCI Express will become the predominant graphics
and host bus interconnect used in the computing space. Designers and manufacturers
of communications, storage, test and medical equipment applications will look
to leverage the ubiquity and low cost nature of this interconnect to introduce
products based on PCI Express in their domains. Their efforts will be greatly
assisted by the flexibility offered by a two-chip, low-cost programmable PHY solution
consisting of an FPGA device and an external PHY device.

These devices can address the aggressive price points required of applications
using PCI Express while offering the benefits of programmable flexibility to aid
in development, cost reduction, accelerated product delivery, and the opportunity
for extending product lifetimes and reducing inventory levels by leveraging the
benefits of programmability. These benefits are actually available today, as low-cost
programmable PCI Express PHY solutions have been demonstrated to successfully
overcome the technical issues associated with a two-chip solution to meet the
necessary technical requirements and also have been priced low enough to meet
the required price points.

San Jose, CA.
(408) 559-7778.