RTC Magazine

The topic of Highly Available (HA) Systems is a crucial one for the readers of RTC magazine. Vendors to key verticals such as telecommunications and manufacturing have spent years and millions of dollars investing in hardware and software technology focused on maintaining availability at 99.999% levels. These investments have been made for very good reasons. Downtime in certain key application areas is a make-or-break business factor for many companies. Telecommunications and manufacturing provide two of the best examples, but as Web services and enterprise applications continue their evolution, those market segments are also becoming more and more focused on high-availability services as well. Moving successfully into this new era requires novel ways of thinking about hardware and software.

The telecommunications industry is under enormous pressure to revamp its networks and applications to accommodate explosive growth and emerging technologies, and at the same time to ensure that the services delivered are available and dependable. A number of industry trends are contributing to this pressure. Traffic volume continues to grow, especially video and 4G wireless applications. Mobile traffic in general is doubling every 9 months. Competitive intensity due to reduced development cycles is increasing with the rapid shift to standardized platforms such as ATCA and carrier-grade Linux. For network equipment providers, differentiation will increasingly be based on application development, managed services and customer support. Equipment providers require that these new applications achieve the highest levels of availability and dependability while contending with constant pressure to reduce development costs.

This shift is not unique to the telecommunications market, as similar trends are evident in a variety of different markets such as aerospace & defense (A&D), manufacturing and other segments. The service providers and vendors offering highly available systems are increasingly looking to the industry to supply building blocks they traditionally have designed and developed internally. The aerospace and defense industry, particularly in the U.S., is providing a strong incentive to their prime contractors—both product suppliers and system integrators—to include as many commercial-off-the-shelf (COTS) building blocks as possible. Similar trends are also emerging in the manufacturing and financial markets. High availability is maturing and becoming something much more than a niche market serving the telecom industry.  

In-House vs. COTS Systems 

This pressure for COTS systems in the HA space is driven first and foremost by straightforward economics. The expense and investment for High-Availability systems has traditionally required a large in-house investment by systems vendors. While these traditional in-house systems were of solid value and solved real problems, the investment demands on these systems have outpaced even the most responsive and innovative vendors. Solving this challenge demands a level of cooperation among industry players. Key standards are needed that create a clear delineation between various functional layers of a highly available system. This standardization of layers—hardware, operating system, middleware and application services—is greatly facilitating the ability of systems developers to put together highly available application-ready systems using COTS building blocks. Figure 1 shows a simple before and after visual of this industry trend. The emergence of multiple COTS suppliers for each of the building blocks is creating a robust ecosystem of COTS component suppliers. This enables system developers to focus their precious, often shrinking resources on things that differentiate them from the competition—applications and services.  

Manufacturing and Industrial Applications

The evolving economics and increasing demand for HA systems can be seen rather dramatically in manufacturing and industrial applications. Recent changes in the adoption of systems and applications in the manufacturing sector have increased the need for HA capabilities. The nature of the operations in this segment is particularly illustrative of the demand and trend toward commercial open standards HA systems.

First and foremost, more and more of the factory floor itself is now based on automation of various types. While this automation takes a variety of forms (lasers, robots, etc.), all the various areas of automation end up stimulating demand and requirements for information technology to monitor, manage and control the various devices on the floor. These IT support systems have an HA requirement by their very nature. Examples include control systems on robots, machine tools and automation. In many application scenarios, the floor systems also need to communicate with each other in real time and share process data. Vendors of a variety of programmable logic controller (PLC) products are enhancing their products with Ethernet capability. This adoption of Ethernet on the factory floor will increase interdependence and risk. Designers, suppliers and management will also be working on shorter cycles, and this means that high availability will increase in importance as an application requirement.

The Risk – Downtime in Manufacturing

The cost of downtime in manufacturing, already well-known before the latest wave of IT adoption, continues apace as the above trends march their way through the industry. The increased automation certainly provides increased productivity and quality, but the related systems must be highly available, or the economic benefits erode rapidly. Systems integrators and suppliers that may not have had a need to ponder HA requirements in most of their projects will now need to do so. The downtime cost for their manufacturing customers has changed dramatically, and systems integrators need to respond with new approaches to high availability.  Figure 2 shows the relatively higher cost of downtime in the manufacturing industry.

Given all of the above changes and increased risks, it is critical to consider the integration of systems and an overall HA-oriented architecture at the earliest stage possible. Most experienced manufacturing directors and engineers have gone through the process of “creeping integration” where they have to plumb disparate elements into a coherent system. The degree to which that integration is successful and sustainable is directly proportional to the time devoted to component selection and architecture early in the process.

Implementing High-Availability Systems

The evolving economics of high-availability systems and the increased demand for HA capabilities in manufacturing and other industries leads naturally to the question of implementation. As you consider how to get started with high availability, there are some key points to keep in mind before you get too far into the process.

First, once you have decided to go the commercial-off-the-shelf (COTS) direction, hardware selection becomes the paramount consideration. In the telecom space, ATCA (Advanced Telecommunications Computing Architecture) has seen increased adoption in recent years. In particular, the “Commercial ATCA” trend shows the evolution of ATCA beyond telecommunications to other industry segments and applications. For ATCA hardware, designers should focus on: switching options, fabric capacity, configurability and thermal issues. ATCA systems offer many options for blades and also enable the use of advanced mezzanine cards (AMCs) that are hot swappable and have integrated system management functions (Figure 3).  

After hardware selection, there are four other key considerations as you think about the design of your HA system. The first is Platform Independence and Portability. System changes are inevitable and need to be planned for. Flexibility is important in the middleware and in applications as well as in the hardware. HA implementers are encouraged to review and adopt industry standards where possible, especially those from the Service Availability (SA) Forum. The SA Forum’s Hardware Platform Interface (HPI) specification and Application Interface Specification (AIS)  are the two key standards to review for more on how to achieve platform independence and portability. 

System developers must also provide failover budgets in their designs. All HA systems have critical performance requirements. It is important to define the various failover scenarios early in the process. Start by classifying failover scenarios into three categories: sub 100 milliseconds, 100 to 500 milliseconds, and greater than 500 milliseconds. The technology applied will vary among the three different cases.

Designs must also ensure that there is no single point of failure. Any system that is expected to provide uninterrupted service must eliminate single points of failure. This is the most critical requirement of any HA system. Redundancy of various types is needed, including: base and backplane interfaces, power/cooling, and the HPI interface to availability middleware. Your choice of availability middleware is also important in this context. GoAhead Software’s SAFfire solution is an example of availability middleware to consider as you assess this component of your architecture. The product is based on open standards and GoAhead has a long and successful track record in the HA middleware market.

Finally, the system must be upgradable. Your HA design needs to incorporate the requirement that an operational system can be upgraded as new versions of software or hardware become available—without impacting service. Designing for upgradability means that the service must be maintained during the upgrade process. Rolling back upgrades that go wrong must also be designed from the start.  

Overall, there is no question that the availability of COTS components helps vendors get systems to market sooner, but it also means that the above considerations must be evaluated as part of the move. Assessing these areas can help you take full advantage of the COTS trend and economics, whether your final application is a complex manufacturing scenario or another highly available application.

The traditional high-availability demands of the telecommunications industry have begun to permeate many other market and industry niches. In particular, applications in the manufacturing market are driving the need for more high availability because of the unique characteristics of the segment. Developers and vendors serving these rapidly evolving markets would do well to begin designing high-availability service into their applications.  

GoAhead Software
Bellevue, WA. 
(425) 453-1900.
[www.goahead.com].