ATCA E-Keying Aids Fabric Management

Designed for long life cycles, AdvancedTCA boasts future-proofing capabilities such as E-keying. E-keying lets existing shelf managers handle new fabrics and proprietary links as they emerge and evolve.


  • Page 1 of 1
    Bookmark and Share

AdvancedTCA is a blade architecture that provides tremendous flexibility for high bandwidth communication within a shelf, as befits its emphasis on telecom applications. An AdvancedTCA backplane is fabric-agnostic, so different communicating pairs of fabric interfaces on the blades may use the backplane links very differently. With this flexibility comes a challenge: ensuring that the fabric interfaces on each end of a backplane link are compatible with each other. For the simplest case of a fixed function interface on each end, a simple compatibility match is required. In the general case, each end of a link may be configurable (perhaps via dynamically loaded FPGAs) to support one of a range of fabric types; that possibility needs to be supported also.

In AdvancedTCA, this challenge is addressed by the E-Keying subsystem of platform management—the foundation management layer that inventories the Field Replaceable Units (FRUs) in the shelves of a system, monitors their basic health and manages their power, cooling and interconnect resources. PICMG has chosen to leverage the widely used Intelligent Platform Management Interface (IPMI) infrastructure, adding advanced extensions that address the special platform management needs of communication systems. PICMG 3.0, the AdvancedTCA specification, defines the necessary extensions to IPMI. Figure 1 shows the management-related logical elements of an example AdvancedTCA shelf.

Intelligent Platform Management in ATCA

AdvancedTCA brings with it some terminology that differs from traditional embedded computer architectures. For instance, “blades” are referred to as “boards”, and “shelf” replaces the term “chassis” that is traditional in some circles. An AdvancedTCA Shelf Manager communicates inside the shelf with IPM Controllers, each of which is responsible for local management of one or more Field Replaceable Units (FRUs), such as boards, fan trays or power entry modules.

Management communication within a shelf occurs primarily over the Intelligent Platform Management Bus (IPMB), which is implemented on a dual-redundant basis as IPMB-0 in AdvancedTCA. FRU Information (accessed via the IPM Controller responsible for each FRU) provides data about the FRU, including inventory information such as manufacturer and model identification. Shelf FRU Information provides similar information for the shelf, itself, including the integrated backplane.

The Shelf Manager has two main responsibilities. First, it manages/tracks the FRU population and common infrastructure of a shelf, especially the power, cooling and interconnect resources and their usage. Within the shelf, this management/tracking primarily occurs through interactions between the Shelf Manager and the IPM Controllers over IPMB-0. The other duty of the Shelf Manager is to enable the overall System Manager to join in that management/tracking through the System Manager Interface, which is typically implemented over Ethernet.

Electronic Keying in AdvancedTCA

The IPMC’s non-volatile storage includes FRU Information (with E-Keying data) for the board. Figure 2 shows the management aspects of an ATCA board that are relevant to E-Keying. The IPMC manages the point-to-point E-Keying controls for the Base, Fabric and Update Channel Interfaces that are implemented on a particular board. The board Payload, itself, manages E-Keying controls for its use of the Synchronization Clock Interface and Metallic Test buses. AdvancedTCA’s provisions for E-Keying these bused resources are very different from the point-to-point E-Keying provisions. The remainder of this article focuses on the point-to-point E-Keying architecture.

The first step in point-to-point (P2P) E-Keying happens when a shelf is first powered up. The Shelf Manager gets the backplane P2P connectivity data from the Shelf FRU Information. This data specifies the P2P interconnections that the backplane provides between specific slots and specific channels on each slot.

Next, for each board in the shelf (whether present at power up or added later), the Shelf Manager determines from board level FRU Information the P2P connections to the backplane that each board implements and the potential logical links to other boards that each of those connections can support. Each such link has a link descriptor that identifies:

  • a P2P Interface on the backplane (Fabric, Base or Update Channel) and a channel number within that interface;
  • the ports (or sets of differential signal pairs) on that channel that are involved in the link; and
  • the link type (which identifies the governing PICMG 3.x subsidiary specification—say, PICMG 3.4 PCI Express—and selects among any variants defined by that specification).

Using the backplane connectivity information, the Shelf Manager then identifies both ends of the P2P connections involving a board and searches for compatible link descriptors at each end of the connections. If compatible links (e.g., both PCI Express Advanced Switching with 4x width) are found, the links are enabled using “Set Port State (Enable)” commands over IPMB-0. Otherwise, “Set Port State (Disable)” commands ensure that incompatible connections are not turned on.

The link type can be within a special range that indicates OEM-defined, rather than specification-defined, link characteristics. This range selects a 128-bit Globally Universal Identifier (GUID) from up to 15 that can be defined for the board. In this case the link descriptor comparison step above includes the GUIDs at each end of a link in the comparison. Given a complete match, the link descriptors are considered compatible.

FPGAs Play a Role

A given backplane connection on a board may have multiple link descriptors defining a range of protocols that the connection supports. In fact, some board architectures may use an FPGA as the main interface to the backplane; in this case, the “Set Port State (Enable)” commands may have the effect of loading the appropriate backplane interface protocol into the FPGA.

Figure 3 shows another example ATCA shelf (focusing on just the ATCA boards for simplicity). Unlike Figure 1, in this configuration the Shelf Manager function is integrated with dual redundant hub boards that provide switching resources for a PICMG 3.1-compliant Fabric Interface. As the example shows, along with the hub boards, four additional boards provide additional processing and are fully interconnected via a PICMG 3.4 PCI Express Advanced Switching (Ex AS) fabric. These four boards are considered to be PICMG 3.4-“mesh-enabled” since each board has a connection to the other three. In the example, these boards are assumed to operate in redundant pairs, using an Update Channel Interface to coordinate within each pair.

Figure 4 shows the likely set of enabled links among these boards, identifying the enabled link types with the same colors used in Table 1. Figure 4 also includes one of the bused backplane resources in ATCA, the Synchronization Clock Interface.

E-Keying complements the fabric-agnostic nature of an ATCA backplane. The same backplane traces in shelves like the example in Figure 4 may be used for Gbit Ethernet and FibreChannel or Gbit Ethernet alone. Another option would be to use PCI Express Advanced Switching with 4x, 2x or 1x widths. A third choice could include proprietary protocols defined entirely by a specific OEM, such as provisions for Shelf Manager coordination.

A specific P2P backplane connection on a board can support multiple link types; matching types will be enabled in a specific configuration. The hub boards can support node boards that either do or don’t support FibreChannel in addition to Gigabit Ethernet. The Shelf Manager automatically enables the appropriate combinations.

Fabric Independent

The E-Keying mechanism doesn’t depend on fabric-specific knowledge. Consider, for instance, the PCI Express AS links. PCI Express can auto negotiate the width of a link, so a 4x link can successfully communicate with a 2x link. But E-Keying compatibility assessments are based on simple binary comparisons of link types, with no knowledge of whether a given link type supports auto negotiation on link widths. Therefore a 4x link always has link descriptors for 1x and 2x widths as well.

OEM-specific link types (such as the Shelf Manager coordination use of the Update Channel Interface on the hub boards or arbitrary Update Channel use between redundant boards such as those in slots 3/4 and 5/6) can be defined by any OEM, entirely without outside coordination. As long as the boards at each end of a P2P backplane connection cite an identical 128-bit GUID, the link can be enabled with high confidence that the boards are able to communicate. Two incompatible boards that use the Update Channel Interface differently will have that link between them automatically disabled.

The P2P E-Keying architecture is future-proofed in the sense that the Shelf Manager simply compares binary fields of the link descriptors looking for a match. When new link types are specified by PICMG, or new OEM GUIDs are defined for proprietary link types, existing Shelf Managers can handle them without change.

The Pigeon Point Systems (PPS) IPM Sentry Shelf Manager fully supports point-to-point E-Keying, and is the first shelf manager to ship on a production basis for AdvancedTCA, with many hundred already shipped. The corresponding facilities in an IPM Controller are covered by the off-the-shelf IPM Sentry board management reference designs and corresponding software, which have already been incorporated in at least fifteen AdvancedTCA boards as of this writing. Interoperability of the IPM Sentry implementations with numerous other implementations at the board and shelf level has been successfully demonstrated in PICMG’s AdvancedTCA Interoperability Workshops.

Pigeon Point Systems
Scotts Valley, CA.
(831) 438-1565.