Fault Management

Interoperable Firmware Upgrades for PICMG Management Controllers

A specification-based upgrade process for hardware platform management means that firmware upgrades can be done in systems across components from multiple vendors.


  • Page 1 of 1
    Bookmark and Share

A recently adopted PICMG specification addresses an important gap that had existed in the PICMG hardware platform management (HPM) architecture: there was no spec-defined way to upgrade the firmware on management controllers. As a result, AdvancedTCA (ATCA) and MicroTCA systems built from a mix of components from different vendors could need a mix of incompatible firmware upgrade tools.

HPM.1, the Intelligent Platform Management Controller (IPMC) firmware upgrade specification, closes that gap for all the board and module level management controller types defined in the ATCA, MicroTCA and AdvancedMC (AMC) architectures. This specification is the first in what will likely be a series of PICMG specifications covering hardware platform management aspects of all three architectures. Even though this specification was developed in the PICMG 3.0 subcommittee, it is immediately applicable to the other two architectures as well.

Figure 1 shows the overall vision for HPM.1 firmware upgrades, which includes the board- and module-level controllers. Collectively referenced as IPM controllers in HPM.1, these also include AMC carrier IPMCs and module management controllers (MMCs), plus MicroTCA carrier management controllers (MCMCs) and enhanced MMCs (EMMCs). It also covers the ATCA and MicroTCA shelf managers that provide access to their respective types of shelves. HPM.1 was not engineered to cover upgrading of shelf manager firmware, which is usually substantially larger than board-level controller firmware. An upgrade agent applies HPM.1 upgrade images to the appropriate controllers by communicating with those controllers, often via the shelf managers and the intervening in-shelf buses, such as IPMB-0 and/or IPMB-L.

If all the IPM controllers in a shelf are HPM.1-compliant, a single HPM.1-compliant upgrade agent can upgrade the firmware in all the controllers, even if each element of the system is implemented by a different vendor, including the shelf managers and the upgrade agent, itself.

HPM.1 IPM controllers must support upgrades over IPMB-0 or IPMB-L, but can also support any other IPMI interface for this purpose. One popular option is the payload interface that links the IPM controller to the main processor(s) on a board. In this scenario, an upgrade agent could execute directly on a payload processor and communicate directly with an IPM controller over the payload interface. Figure 1 shows this option for the two MMCs, as an example.

Each upgrade image identifies the type (including the implementer and compatible version range) of IPM controllers it can be applied to. IPM controllers can identify themselves in these same dimensions. As a result, an independently implemented upgrade agent can apply firmware images of different types (for example, types A and B in Figure 1) to appropriate IPM controllers (for example, type A IPMCs and type B MMCs in Figure 1).

In addition to a descriptive header, each upgrade image includes a sequence of upgrade action records, some of which may include data, which is typically the new firmware to be uploaded. An upgrade agent communicates via HPM.1-defined IPMI commands with an IPM controller that is a candidate for upgrade and, for a compatible upgrade image and controller pair, proceeds to affect the upgrade actions on the controller.

Key Requirements

In addition to multi-vendor interoperability, the developers of HPM.1 in PICMG identified and achieved the following requirements for the firmware upgrades supported by this new facility:

• Reliable: an IPM controller can store redundant firmware images and automatically fall back to the prior image if it detects that a newly uploaded image has a serious error or loses connection to the upgrade agent. HPM.1 encourages and provides infrastructure for self-tests that the IPM controller can execute to validate a newly uploaded image.

• On-line: an IPM controller can accept a new firmware image while continuing to perform its normal management functions, executing a previously installed image.

• Deferrable: new firmware can be uploaded, but not activated until some later time, perhaps after all needed downloads are done and the system is in a low load period.

• Flagged payload effects: IPM controllers and upgrade images indicate whether an upgrade can affect the payload of the parent field replaceable unit; any such impact should be the rare exception, but may not be entirely avoidable. It may be critical for the upgrade agent and/or its operator to know whether an upgrade operation could require, for example, a reset of the payload.

• Multi-component: an IPM controller can apply upgrades to other low-level elements of a board, such as an FPGA.

• Flexible implementation: an HPM.1-compliant IPM controller may not implement any of the above features, perhaps because it is a very low-cost implementation that doesn’t require such features or because it was initially implemented before the advent of HPM.1 and is being retrofitted for HPM.1 compliance.

• Self-describing: IPM controllers and upgrade images identify the features they support, so that an upgrade agent and its operator can adjust upgrade scenarios to fit.

Upgrade Process

An HPM.1 upgrade for a particular upgrade image and a particular IPM controller includes three stages: preparation, upgrade and activation. Figures 2, 3 and 4 show these stages. Note that activation can be deferred (if supported by the IPM controller), and therefore may occur long after the corresponding upgrade stage.

The identification check step in the preparation stage confirms that the target controller and upgrade image have the same manufacturer and device type, as well as suitability of the current firmware revision for upgrading by this image.

The subsequent capabilities check step can cover compatibility topics (for instance, that both image and controller support manual rollback if one of them supports only that recovery option) as well as upgrade policy topics (such as an operations policy that controllers will only be upgraded if they support deferred activations). In addition, the upgrade agent collects data that will guide its later upgrade actions, such as the IPM controller’s estimates of the maximum time it needs for key operations.

Finally in the preparation stage, similar checks are made for compatibility at the component level (for example, the new FPGA load, versus the new microcontroller firmware). These checks include confirming that the number and identity of the components in the image are compatible with the components supported by the IPM controller. The upgrade process next enters the upgrade stage, as shown in Figure 3.

In the upgrade stage, the upgrade agent proceeds sequentially through the upgrade action records in the upgrade image. The IPM controller developer provides the image and takes responsibility for ensuring that the upgrade actions are appropriate for the target controller. If necessary, the image includes explicit steps to backup an existing image or perform implementation-specific preparation for the upgrade. Then, the upgrade agent uploads the blocks of upgrade data in an upgrade action record to the IPM controller. The upgrade data is opaque to the upgrade agent, so the detailed structure of that data can be completely different between different IPM controller vendors without affecting HPM.1 upgrade interoperability.

After the upgrade agent completes the uploading process, it activates the new firmware, as shown in Figure 4. For IPM controllers that support deferred upgrades, the upgrade agent may delay activation until some later time of its choosing. The activation step is not triggered by an upgrade action record; the upgrade agent or its operator decides when to activate an upgraded controller.

IPM controllers optionally include a self-test function that is invoked automatically as the final step of activating new firmware, independent of whether the activation is deferred or not. This function can include, for instance, checks of the controller’s memory and its key peripherals. The upgrade agent knows from its earlier capabilities checks whether the IPM controller implements a self-test function and a maximum for the time the self-test will take. If the self-test passes, the IPM controller proceeds into operation with the new firmware. If the self-test fails, the IPM controller can automatically roll back to the previous firmware or the upgrade agent can initiate a manual rollback. In either case, the IPM controller proceeds with operation on the previous firmware.

Implementation Experience

Though HPM.1 was just adopted in May 2007, several independent hardware platform management subsystem developers have already done initial implementations. As part of their work with HPM.1, Kontron took the initiative to add HPM.1 upgrade agent support to the popular open source IPMItool utility ( At least one Tier 1 TEM is already integrating upgrade agent functionality into their system management application.

Pigeon Point’s several years of experience with supplying IPM controller upgrade facilities was a strong input into the development of HPM.1. With HPM.1 compliance, these facilities are now even stronger, including several important new features. Upgrades can now be accomplished over any IPMI interface of the IPM controller, including the payload interface. Online upgrades allow normal operation of the IPM controller to continue while new firmware is uploaded. Deferred upgrades allow system operators to decouple uploading of new firmware and activating it, so that both can be done at an appropriate time. The bootloader, a special firmware module that executes when a controller comes out of reset, can now be upgraded as an HPM.1 component.

Pigeon Point Systems
Scotts Valley, CA.