RTC Magazine

The telecommunications industry gave birth to the catch phrase “Innovation at the speed of the Internet,” so it should come as no surprise that telecom equipment vendors themselves are facing increasing pressure to rapidly bring complex carrier-grade solutions to market. An evolving ecosystem of industry standards and off-the-shelf products is helping to drive this cycle and equipment vendors can increasingly develop a large portion of their systems with off-the-shelf hardware, operating systems, protocol stacks and carrier-grade middleware.

Carrier-grade middleware provides the glue that binds the various hardware and software components in a system (Figure 1). The simplest way of looking at it is that it provides everything above the operating system that is not specific to the “application” of a system. The primary focus areas for middleware are core infrastructure services such as intra-cluster messaging, system and platform management, and, of course, high availability. In addition to providing specific features, the middleware also differentiates a product by defining the system’s overall character. Well-designed middleware can enable the rapid scoping of system requirements, model component and system behavior before a product is built; simplify the development and re-usability of applications across a product family; drastically reduce integration effort and support iterative enhancements across multiple product releases.

The carrier-grade features of a network element have traditionally been viewed as major market differentiators but they are now considered the norm, and a product is only distinguished by its lack of such characteristics. The bar will only go up in the future. By selecting appropriate off-the-shelf carrier-grade hardware, operating systems and middleware, equipment manufacturers can focus limited resources on their own application-specific innovations, while significantly reducing both risk and time getting a product to market.

Middleware: Tying Together a Platform

The term middleware is often used loosely, so it makes sense to define it in the context of a carrier-grade system. The features provided by carrier-grade middleware can be roughly broken down into four mutually interdependent areas: system management, platform management, high availability and core infrastructure. Particular implementations will naturally provide more or less functionality in each area.

In terms of system management, middleware should provide a complete solution to model the various logical and physical entities in a system using an object model or information base. It provides the run-time framework to manage and monitor these entities as well as standard interfaces such as simple network management protocol (SNMP) or command line interface (CLI) to integrate with external element management systems. A number of third-party products in the system management space have traditionally provided these capabilities. Middleware must provide comparable functionality as well as tighter integration with application and availability management capabilities.

Middleware also provides the ability to manage a specific platform using industry standard interfaces such as the Service Availability Forum’s Hardware Platform Interface (SAF HPI). Often, the middleware is pre-integrated with standard platforms such as AdvancedTCA or IBM’s BladeCenter/T, enabling a plug-and-play experience. A system designer need only write drivers or board support packages for application-specific hardware such as specialized cards or chip sets. Middleware may further simplify the integration of these components into the rest of the framework, providing an additional level of assurance that they will work together reliably.

Actual higher availability (“five nines”) is best achieved when a system designer plans for it up front and designs a system taking various constraints and behaviors into account. Middleware can aid this task by providing a comprehensive modeling framework and a design discipline, which, in addition to modeling and validating the desired system characteristics, also gives a measure of predictability to system behavior. Eventually, a system failure is inevitable and a system that fails and recovers predictably is preferable to one that handles the various fault modes in an ad-hoc manner. The framework can also enable consistent application behavior from one platform footprint to another.

Hand in hand with the modeling framework, middleware must also provide the run-time infrastructure to manage the components’ entire life and fault cycle, from start-up to shutdown, and from detection and diagnosis to service recovery and repair. This infrastructure may include an availability management service for managing the life cycle and work assignments of redundant components, a fault service for implementing user-defined repair policies, a checkpoint service for enabling stateful application failovers, a messaging service with high-availability semantics, an upgrade framework for non-service impacting software upgrades and group membership services for enabling the formation of independent redundancy groups. Obviously, this is just a sampling of the services that middleware should provide. It might also provide common application domain services such as an extensible virtual IP framework.

A carrier-grade system typically consists of multiple hardware and software components that must work together to become, in essence, a system. As such, they require infrastructure services such as intra-cluster messaging protocols, name services and replicated in-memory databases, all of which go beyond the capabilities of the typical operating system. These services must be provided in a portable manner for easy use of heterogeneous operating systems and processors.

All of the characteristics and features provided by middleware must work together to provide a cohesive product experience to both the system designer and the end user. Traditionally, equipment vendors have built many of these features in-house or sourced them independently and integrated them in-house, often at great expense. Pre-integrated solutions that offer a large percentage of a product’s middleware needs are the next logical step in the evolutionary process. An example of some of the features and services that can be included in a middleware implementation are shown in Figure 2.

There are two distinct types of pre-integration possible with middleware. One is internal pre-integration among the discrete components that make up the middleware, with, ideally, the discrete components built on a common framework of concepts and modules. The other is external pre-integration with operating system distributions (kernel and libraries) and hardware platforms (chassis, CPUs, switch fabrics, line cards, etc.). Despite standardization, external pre-integration gives rise to a large number of permutations, and most middleware vendors would likely offer a preferred list of tested combinations with others supported on an as-needed basis.

A Question of Character

In addition to enabling the manageability and availability of applications, carrier-grade middleware also imparts design characteristics to a system that truly distinguish it beyond the point features easily visible to the users of the system. It goes without saying that the middleware components must themselves be manageable and highly available, but middleware also impacts the final solution in other ways.

A middleware’s range of modeling capabilities can vary from none, to modeling via a programmatic interface, to a comprehensive IDE framework that does everything from importing entity attributes via management information bases (MIBs) to generating code stubs, and perhaps even validating system behavior before the system is built. Modeling frameworks are commonly used in visual programming paradigms at a component design level. Their adaptation to system design can truly surpass traditional design approaches and enable rapid and precise development of a system. Models can be shared, reused and iteratively enhanced, thus promoting commonality across a family of products.

Despite such commonality, one size does not fit all. The carrier-grade market ranges from small footprint embedded boards to large multi-chassis systems whose resources (processors, memory, etc.) compare with high-end enterprise servers. It is necessary to customize middleware both in terms of the components that must run on a processor and in the system resources used by each component. This can be a minefield for middleware developers as they trade off conflicting size, speed, scalability and algorithm complexity requirements. A comprehensive modeling framework can include the ability to customize the middleware for various target profiles, additionally supporting portability.

A modular middleware architecture supports customization. It also supports high availability by helping to isolate faults at a module level and enabling the system to take module-specific recovery actions.

To ease the task of integration, middleware may impose a programming model on application components to enable manageability and availability. Many established equipment providers have legacy applications in their systems and this model needs to integrate well with those applications. Middleware can ease the burden of application adaptation by auto-generating code as well as providing default templates.

One very significant feature of middleware is that it enables portability. Traditionally, system designers have built platforms and applications together, with potential leakage of one into the other. By defining a clear separation between the application and the platform, middleware forces portability into the design. In addition, middleware can explicitly support portability by implementing appropriate adaptation layers that hide target OS, processor and link specifics where necessary. The modeling framework can also support portability by allowing the system designer to define target profiles with distinct characteristics and adapt the abstract application models to specific profiles when “building” code for the model.

Since middleware hides the platform from the applications, system designers may view any potential problem with the system as a middleware issue. Middleware can differentiate itself by the traceability built into its components and by its ability to audit overall system behavior.

The Role of Standards

A number of industry specifications are helping to shape the middleware space. These include the Service Availability Forum’s HPI and AIS (Application Interface Specifications), the PICMG (PCI Industrial Computer Manufacturers Group) AdvancedTCA set of specifications and the SMASH (Systems Management Architecture for Server Hardware) CLP (Command Line Protocol) specification from the DMTF (Distributed Management Task Force). These standards help to assure equipment manufacturers that they can achieve a certain level of functionality and mitigate risk by choosing a standards-compliant implementation.

At the same time, it is important to understand the limitations of the standardization process and the scope for innovation available to different middleware vendors. With a few exceptions, standards such as the SAF AIS typically specify the interfaces desired from compliant middleware; they do not (and cannot) specify the internal behavior. It is possible for two implementations to be compliant at the interface level but to provide a very different quality of service, giving a very different character to a system.

The standardization process also focuses on defining a common denominator of services that are of interest to the participants. It cannot be all encompassing. Both the middleware designer and the system developer must balance the need for standards with the desire to provide a fully integrated solution. Non-technical factors such as the support provided by a particular middleware vendor can also factor into the process. Does the vendor provide an “off-the-shelf but throw over the wall” solution or are they amicable to a close partnership, with new features developed in a collaborative environment?

A Range of Choices and Decisions

It is important that equipment designers considering the development of a new product or product family explore the middleware landscape and take advantage of standards and competitive market dynamics. At the same time, it is important that they do so with the proper expectations.

A middleware solution can reduce development time and cost by a significant percentage, both up front and over the product life cycle, but it is not a magic wand for solving the designer’s specific problems. A full understanding of system and application needs is still required: knowledge of the middleware model is critical and application development, integration and testing with the middleware framework is still the burden of the equipment vendor. A good middleware solution can force many of these issues to be addressed up front in the development cycle rather than later when the cost to fix them is much higher.

A middleware solution needs to cover a broad range of deployment possibilities and it will thus offer numerous features and configuration knobs for each component—typically far more than developed by an internal team. This can help to future-proof a product, and give the ability to enable new features and behaviors with just a few parameter changes. However, a typical application will only need a few of these capabilities and it is best to phase in new features as the development team gains confidence with the middleware’s capabilities and, undoubtedly, its eccentricities.

Regardless of whether equipment vendors view middleware as a differentiator, a value-add or a necessary overhead, they are betting the farm on it. A concern of “not invented here” is natural and due diligence is clearly required. In our experience, system designers fall into two broad categories: those who wish to view the middleware and platform as a black box, and those who view no middleware solution as “perfect” and thus, wish to enhance it with internally developed capabilities. In either case, a prototyping approach may help to gain understanding of the strengths and weaknesses of each solution.

It may also help for system designers to explore a collaborative development model with the prospective middleware vendor to allay some of their fears and doubts. Collaborative development can also help with the integration of legacy applications and infrastructure components that the equipment vendor wishes to retain in the new solution. It is important to keep in mind that the middleware itself is a complex, evolving ecosystem and that both systems designer and middleware vendor can benefit tremendously from an open approach to development.

OpenClovis
Petaluma, CA.
(707) 285-2852.
[www.openclovis.com].