Power Management

Removing ATCA’s Architected Single Point of Failure

Although a key strength of the AdvancedTCA architecture is its shelf manager, the connections between shelf manager FRUs and Ethernet hub boards have limited overall effectiveness. PICMG’s recently adopted ECN 3.0-2.0-001 amends the formal ATCA specification to fix this limitation, while preserving backward compatibility with pre-existing equipment.


  • Page 1 of 1
    Bookmark and Share

The AdvancedTCA (ATCA) architecture is rapidly gaining acceptance among telecom equipment providers and users of that equipment. This architecture’s mandatory shelf manager, typically implemented in a dual-redundant configuration by a pair of boards, is a crucial strength of ATCA. Many ATCA shelves implement shelf managers as dedicated field replaceable units (FRUs) connected to the in-shelf Ethernet network. These are also typically implemented in a dual-redundant configuration by a pair of Ethernet hub boards.

Unfortunately, the original AdvancedTCA specification only defines a standard way for each shelf manager to connect to one of these Ethernet hub boards, not to both of them. This limitation can reduce the effectiveness of both the dual-redundant shelf managers and the dual-redundant hub boards, since a failure in just one of each pair may require both pairs to switch over in order to preserve shelf manager connectivity to the in-shelf network. It is more difficult, and much less desirable, to arrange such a coordinated switchover of two separate types of FRUs. In fact, the single shelf manager link to these Ethernet hub boards is the only known architectural single point of failure in ATCA. In contrast, all normal node boards in a shelf have the ability to connect with both hub boards.

Recently, PICMG adopted an Engineering Change Notice (ECN) to PICMG 3.0 R2.0—the formal specification defining the AdvancedTCA architecture—that fixes this limitation, while preserving backward compatibility with pre-existing equipment.

Before ECN: Each ShMC Links to One Hub

In the original AdvancedTCA specification, through revision 2.0 adopted in March 2005, each Ethernet “Base Interface” hub board is connected by a 10/100/1000 Mbit/s link to one of the dedicated shelf manager FRUs, also called Shelf Management Controllers (ShMCs) (Figure 1). This link is provided on the backplane and reaches the hub boards via their Zone 2 high-speed signal connectors. Other external or auxiliary connections may also be available to each ShMC and to the hub boards. Each 10/100/1000 Mbit/s link requires four differential signal pairs or a total of eight wires.

At the architectural level, there are several consequences of linking each ShMC to one hub. One is that a failure in a ShMC may cause the hub boards to switch over if the interconnects between the two hubs are not used. Another consequence is that a failure in a hub board will definitely cause the ShMCs to switch over. In general, the original architecture reduces the effectiveness of dual redundancy in ShMCs and hub boards due to these effects. AdvancedTCA vendors have developed various ad hoc approaches to remove this single point of failure in their shelf products. However, there has been no generic solution that all vendors can use that preserves interoperability so that a hub board can be installed in any shelf and still implement a solution to these problems.

After ECN: Each ShMC Links to Both Hubs

The recently adopted ECN 3.0-2.0-001 provides an optional way to remove this single point of failure. It defines dual 2-pair 10/100 Mbit/s ShMC cross-connects that can connect each ShMC to both hubs, as well as map these dual 2-pair links into the single 4-pair connection (the ShMC port) that was originally defined in the AdvancedTCA specification on a hub board for use as an ShMC link. Furthermore, ECN 3.0-2.0-001 adds this option while allowing full backward compatibility for products that conform to the pre-ECN specification. Each 10/100 Mbit/s link requires only two differential signal pairs or four wires.

Figure 2 shows the connections enabled by ECN 3.0-2.0-001, including how the two 2-pair ShMC cross-connects are mapped into a single 4-pair ShMC port on each hub board. When two 2-pair connections—which can each support a 10/100 Mbit/s link—are mapped into a 4-pair connection—which could otherwise support a single 10/100/1000 Mbit/s link—the tradeoff is a reduced data rate for improved availability. However, few, if any, existing shelf manager products support or need a 1000 Mbit/s link to the outside world. For most ATCA users, the availability improvement enabled by the ShMC cross-connects is much more important than the higher data rate.

E-Keying Self-Identifies ShMC Cross-Connect Support

ECN 3.0-2.0-001 also defines new constructs in AdvancedTCA’s Electronic Keying facility to allow the three participating shelf component types to self-identify their support for ShMC cross-connects. E-Keying information is represented as records in non-volatile FRU information storage associated with the key participating system components: in this case, the shelf, the dedicated ShMCs and the hub boards. The format of these records is defined in ATCA so that independently implemented system components can self-identify their capabilities, enabling automatic interoperability adjustments.

The shelf FRU information stored with the backplane can show that two ShMC cross-connects are routed on the backplane between each ShMC site and the two hubs. For a legacy shelf, this information indicates just one implemented link from each ShMC site to its corresponding hub. The board FRU information stored on each dedicated ShMC FRU reveals whether they can connect to both of the hubs, if the hubs support that, or to just one of the hubs. The board FRU information stored on the hub boards shows which of the following three possibilities is supported: dual 10/100 Mbit/s ShMC cross-connects, a single 10/100/1000 Mbit/s ShMC link or either of the first two options, dynamically selected when the hub is configured via E-Keying by the shelf manager.

When the shelf comes on, the active shelf manager retrieves the FRU information from each of the above components and enables the appropriate links, based on their capabilities. For instance, the hub boards may be capable of either two 10/100 Mbit/s ShMC links, or a single 10/100/1000 Mbit/s link. Depending on the capabilities of the shelf and the ShMCs, the shelf manager enables one of these two configurations on the hubs.

The Table displays the different configurations of the three critical shelf ingredients for ShMC cross-connects, along with the compatibility properties for each configuration. ShMC cross-connect B is the extra link from a ShMC to the opposite hub that is new with the ECN. In some of the configurations, such as numbers 2 through 5, that link must be disabled by E-Keying on the ShMCs, the hubs or both. Configuration 6 results in unpredictable behavior of generic Ethernet-compliant 10/100/1000 Mbit/s PHYs and must be avoided by system integrators.

Dynamically Supporting One or Two ShMC Links on a Hub

Given the capability descriptions in the E-Keying records, the shelf manager can notify a hub whether to configure itself for two 10/100 Mbit/s links, or for a single 10/100/1000 Mbit/s link. Such configurability is an optional feature for hub boards that are compliant with ECN 3.0-2.0-001.

Figure 3 shows how such configurability could be implemented in a hub. On the right side of the figure is the 4-pair ShMC port on the backplane connector of the hub board. On the left side are two 10/100/1000 Mbit/s PHY interfaces. Depending on how the analog multiplexers AM 1 and AM 2 are set, one of two possible connections is made between the switch fabric of the hub and the ShMC port. In the first, the top 10/100/1000 Mbit/s PHY is connected via four pairs to the ShMC port. This is the normal pre-ECN 3.0-2.0-001 configuration. In the second, each of the two PHYs is connected with a 2-pair (10/100 Mbit/s) connection to the ShMC port. This is the normal ShMC cross-connect-enabled configuration.

With the implementation shown, the hub can be configured by the shelf manager to support ShMC cross-connects or not, depending on the capabilities of the backplane and dedicated ShMCs.

Providing ShMC Cross-Connect Support

Pigeon Point Systems instigated the PICMG initiative for ShMC cross-connect support in AdvancedTCA and chaired the PICMG subcommittee that developed ECN 3.0-2.0-001.

Prior to the new ECN, some shelf manager platforms, such as the company’s popular IPM Sentry ShMM-500 Shelf Management Mezzanine, supported just two Ethernet interfaces: one was available for communication with one of the hubs and the other was used to communicate redundancy state updates between active/backup shelf manager pairs. For such shelf manager products, the advent of the ECN raised the question of how two Ethernet interfaces could be made available for hub connections.

This question was especially challenging for the compact, 67.60 mm x 50.80 mm SO-DIMM-sized ShMM-500, for example, because it would be difficult to add a third Ethernet interface while still maintaining the board’s compact size. The solution for the ShMM-500 was to use a pair of USB links for the redundancy state updates, freeing up both Ethernets for hub communication duty. Serendipitously, the four signals needed for such USB links could easily be routed across the same backplane traces that had carried the previous Ethernet-based redundancy state updates.

This approach enabled the quick availability of ShMC cross-connect support within one month after the enabling ECN was adopted by PICMG, allowing users to eliminate the architected single point of failure in AdvancedTCA systems.

Pigeon Point Systems
Scotts Valley, CA.
(831) 438-4709.