The increase in energy density of MicroTCA’s smaller form-factors makes thermal design an important consideration in systems based on this architecture.
STUART JAMIESON, EMERSON NETWORK POWER
In 2003, the PCI Industrial Computer Manufacturers Group (PICMG), in conjunction with telco operators, defined the Advanced Telecommunications Compute Architecture (ATCA). AdvancedTCA addresses the needs of modern telecommunications systems by offering features such as a high-performance, switched-fabric serial backplane operating at speeds of up to 10 Gbits/s per link and built-in redundancy for essential system structures such as backplane channels and system management, as well as a native ability to allow hot-swap of circuit cards. In 2005, PICMG provided finer design granularity by releasing the Advanced Mezzanine Card (AMC) specification, defining modules that can plug into ATCA carrier cards to foster design reuse as well as allowing the system to offer hot-swap at the module level.
While these specifications addressed all of the features needed for modern telco systems design, they originally targeted only the needs of large installations such as central offices and major PBX (private branch exchange) systems. The 8U carrier cards used in ATCA prevent the technology from being used in smaller systems such as wireless base stations and customer premise equipment. To address the needs of these smaller systems while still using the software and AMC module hardware of ATCA, PICMG created MicroTCA.
MicroTCA leverages hardware and software developed for full-sized AdvancedTCA systems, but scales to more modest performance and much smaller form-factors. One side effect of this scaling is an increase in energy density, making thermal design an important consideration when creating MicroTCA systems.
Essentially, MicroTCA is a backplane and chassis. AMC modules can be plugged into it directly, rather than as part of a carrier card. The system hardware acts as a virtual ATCA carrier card for AMC modules, providing all of the software and signal support that the carrier provides in ATCA. As with ATCA, the backplane supports serial communications channels in star, dual-star and mesh configurations, allowing a MicroTCA system to re-create in miniature whatever functions can be implemented in ATCA. A pair of central switches in the system supports up to five serial communications lanes per module with up to 12 modules in a subrack.
One key feature of ATCA systems, the built-in system management that supports user-implemented hot-swap and fault-tolerance functions, is replicated in MicroTCA (Figure 1). Card management functions, provided by the shelf management controller in ATCA, are handled by MicroTCA Carrier Hubs (MCH), which also plug into the backplane. These functions include monitoring and control of system-levelresources such as fans and power supplies through the I2C-based Integrated Peripheral Management Interface (IPMI).
The MicroTCA Application Space
Although the MicroTCA architecture is adaptable to many types of embedded systems, its initial targets are telecommunications and networking applications. Its protocol-agnostic, high-speed serial backplane is a natural fit for the needs of packet-switched communications. Within this telecom space, MicroTCA suits a wide range of functions, running from small, consumer devices to full-featured, mid-capacity gateways and servers (Figure 2). Because MicroTCA uses unmodified AMC cards developed for larger ATCA-based systems, MicroTCA provides developers with an opportunity to address these mid-range systems at relatively low cost. Reuse means that most of the work has already been done and therefore additional markets for the AMC modules will help drive down manufacturing costs.
More importantly, however, MicroTCA also addresses these applications by offering a compact form-factor. Standard shelves based on the MicroTCA architecture can range in size from two-tier, with potentially 24 modules, to pico, with only a few modules (Figure 3). Pico shelves can even be as small as one AMC (with the MCH on a motherboard), making them potentially suitable even for consumer devices.
Representative applications for MicroTCA include wireless base stations and customer premise equipment. Wireless base stations need high performance to handle the data, voice and media traffic that make up today’s and tomorrow’s mobile communications. At the same time, however, they need to be relatively compact and low power due to their remote installation sites. Customer premise equipment must be even smaller, since installation in a closet or small room is highly probable. The many choices of form-factor along with the functional capacity of AMC help developers optimize MicroTCA designs for both such systems.
Incompatibility Issues Arise
Developers seeking to work with MicroTCA should be aware that there is a limited but growing availability of AMC functions along with two important design issues to consider. One design issue is related to limited management functionality on some AMC cards. The second is related to power for the MicroTCA chassis and the cooling concerns that arise.
Because the AMC specification is still in its early adoption phase, the present crop of commercial modules provides the most common functions of telco design, such as T1/E1 telephony interfaces, DSPs, storage system interfaces, CPUs and network controllers. The rest of the system will probably be a custom design. Fortunately, modules based on FPGAs are also starting to appear. These modules leave large segments of programmable logic available to the developer and thus enable the implementation of custom functions without the compatibility risks of full custom designs.
Unfortunately, some of the available modules that are now commercially available were rushed to market in an attempt to capture market share. During that rush, some design teams made compromises by implementing only a part of the module’s system management functions or by violating module power restrictions. Therefore, developers now seeking to use such off-the-shelf AMC modules in MicroTCA designs need to be thorough in their evaluation of candidates.
One compromise that has become apparent in some first-generation AMC modules is a failure to implement the full spectrum of shelf management functions. Much of the potential for fault tolerance and high-availability system operation stem from the system management functions. Failure to implement the full management system can thus make such operations difficult or impossible to achieve, preventing the reuse of the design effort.
Addressing Power Concerns
A second compromise has been a violation of module power limits. The power scheme for AMC modules recommends a staggered power limit based on card size, ranging from 20W for compact single cards to 80W for full double cards. These recommendations are based in part on the need to control overall system thermal characteristics and prevent the formation of “hot spots,” and in part on the limits of forced-air cooling. Developers are sometimes tempted to ease or exceed these power recommendations when they have control over both module and shelf design and can therefore fully control system cooling.
The risks that arise from exceeding module power limits stem in part from the highly compact structure of a MicroTCA system. A shelf fully populated with modules can attain a high power density if all the modules are dissipating their full power allocation, but systems are limited in their air cooling capabilities. Forced-air cooling is the norm for MicroTCA systems; it is not a panacea for thermal excesses. Without careful planning of airflow in a chassis, airflow dead spots may develop within the shelf. In addition, careless placement of modules within the enclosure may result in the creation of hot spots that exceed the cooling ability of local airflow.
Developers creating custom shelves may be tempted to address such issues by simply increasing airflow through the system, but that approach can create other problems. In order to meet the requirements of the European Telecommunications Standards Institute (ETSI) and Network Equipment Building System (NEBS) standards for telco equipment, for instance, the acoustic noise of fans must be below specified limits. Similarly, applications such as consumer equipment and commercial installations are subject to sound level regulations from worker safety agencies. The airflow needed to accommodate careless thermal design may not be attainable without violating such restrictions.
Even when all of the AMC modules adhere to specified power limits, however, thermal issues may still be a concern. Fortunately, there are several steps that developers can take to help reduce the impact of these issues. These include proper handling of airflow, careful selection of power supplies and the use of intelligent fans.
Airflow in a MicroTCA system is specified from bottom to top but can use a push or pull type of configuration. Pushing air through the system has the advantage of helping to keep dust and other contaminants from entering the system through gaps in the housing. It also provides an opportunity to direct the airflow toward high-powered boards. Because air exhaust is at the rear of the enclosure, however, this approach requires that the fans be mounted at the front.
But front-mounted fans put their noise into the user environment, which may not be acceptable in some types of installations. Pull-type configurations allow designers to provide a more even flow of air through the system, while placing the fans at the back or top of the system provides better acoustics. With careful filter design the contaminants issues can be resolved.
Noise can be controlled somewhat by using fans with built-in thermal sensors and adjustable speeds. Together with IPMI shelf management functionality, such intelligent fans can be run at lower speeds when heating is not excessive to keep noise levels down. These fans can also allow the system to adequately handle fault conditions. NEBS requirements, for instance, call for a system to maintain operation in the event of a single fan failure until the fan can reasonably be replaced. Similarly, the system must be able to maintain operation long enough for an orderly shutdown when the entire fan tray fails. Having intelligent fans linked to shelf management can help meet such requirements where needed.
Other Airflow Choices
Beyond fan placement and selection, airflow choices that developers need to consider include the placement of boards in the system and placement of the shelf itself. Placing high-powered boards together in a shelf can create hot spots and may simultaneously restrict airflow if large heat sinks occupy much of the space between boards. Gaps in board placement can also cause a problem by creating an open channel for much of the airflow. Developers should consider separating high-powered boards and using blank panels for empty slots to help keep airflow uniform.
The placement of the shelf itself should also be considered. Many rack-mounted systems have forced air to the rack from which each shelf draws its forced air supply. The same care that goes into managing airflow within a chassis should be applied to the rack as a whole to prevent feeding a shelf with overheated air.
Finally, developers should evaluate the system power supply for its thermal and management functionality. Because the power supply itself is not 100% efficient in converting line power to system DC, it is also a source of heat. The more efficient the power supply, therefore, the less system heat will need to be managed. To meet fault tolerance requirements, the supply should also be capable of responding to IPMI management directives, such as switchover to a redundant supply.
When these types of system thermal and management issues are properly addressed, the MicroTCA design approach holds great promise for compact telecom systems. MicroTCA presents developers with an opportunity to leverage technology and software developed for large systems in the development of compact systems. Capitalizing on this opportunity will require careful thermal design and full adherence to system management specifications, but the payoff will be great performance with the cost benefits of design reuse.
Emerson Network Power, Embedded Computing