RTC Magazine

One consequence of Moore’s law is that as processors increase in speed, an increasingly large part of an electronic system is most effectively implemented in software. This has clearly been true for products like enterprise servers and PCs. Today however, following this performance curve, we now have game consoles, smart phones, airliners and automobiles all running different operating systems and applications. In order to keep up with the increase in—and complexity of—today’s device software, a major transformation in complex system-level embedded design is currently underway.

Typically, the critical path for the delivery of a conventional working system is to finalize the hardware design and then finish the development of the software, followed by the final completion of the system integration. However, as the task of developing software increases in complexity and length, it comes to dominate the cost and risk of the overall system development project.

Two major challenges face the system team: First, software development can no longer wait for hardware development to be completed and simply must begin earlier. Secondly, the cost of the traditional hardware-based approach to testing software is becoming both prohibitively expensive and unwieldy. The best solution to easing the pain associated with these challenges is full system virtualization—leveraging the power of virtualization technology to simulate the entire embedded system in development, enabling a faster concurrent development cycle.

Virtual Platforms

To follow a software development approach that takes advantage of full system virtualization, a virtual development platform has to be created. Specifically, a software model of the system in question is built and run on an underlying simulation environment.

For example, a set-top-box containing a DVR (Tivo) might consist of a MIPS processor, a disk drive, cable modem and various types of memory, interrupt controllers and so forth. Using virtualization, each of these devices is replaced by a model. This assemblage of models is known as a virtual platform and can be run just like the real hardware—a set-top-box in this instance.

While producing a functional virtual platform is a significant step toward easing development pain, the most important aspects of the platform—fidelity and performance—have to be adequately addressed before the platform can present an effective advantage to the developer.

A model with high fidelity will be sufficiently accurate so that the binary code that will eventually run on the real set-top-box can run unchanged on the virtual platform. In particular, the unaltered binary of the real-time operating system (RTOS) can be used. At the same time, a high-performance virtual platform must be fast enough that software developers enthusiastically prefer to use it, looping quickly through the edit-compile-debug cycle, unlike co-verification environments. Unfortunately past attempts at virtualization failed to produce acceptable levels of performance or fidelity.

To be truly effective, the virtual platform must be able to simulate not only the system being designed, but enough of the environment surrounding it to model real-world use—in effect, a virtual test rack. In the previous example, a team is designing a set-top-box, but it is not enough to model just the set-top-box itself. It may be necessary to model several set-top-boxes, along with a central server delivering video-on-demand, and perhaps a PC used to administer the system and a server to handle billing.

When modeling peripheral devices—just as when modeling the processor— performance and fidelity are essential. Fidelity for most peripheral devices means that the device registers should contain the correct values when the processor reads or writes them, and that suitable simulated actions take place when the peripheral is activated. However, in order for the overall performance of the virtual platform to be good, the peripheral device models must also be high performance.

Modern virtualization approaches are capable of impressive performance—sometimes exceeding four billion instructions per second running CPU-intensive benchmarks on the latest PC hardware (summer 2004). This is faster than the clock-rate of the underlying computer running the simulation. In more realistic examples, such dramatic speeds are reduced by a factor of almost five, leading to performance approaching one billion instructions per second. This is often faster than the real hardware, which might be an embedded processor fabricated in a semiconductor process, a generation or two behind that used for the microprocessor in the host PC.

Today, because a simulator can see events occurring in the future, an idling system can be simulated several times faster than reality. What’s more, this happens while running the unchanged binary of the code that will eventually ship with the equipment, including device drivers and protocol stacks. Even low-level real-time operating system routines cannot detect that the simulator is accelerating time.

Through a mixture of technical improvements in virtualization techniques, coupled with the ready availability of inexpensive high-performance PCs, virtualization has amassed significant advantage when compared to traditional development models. Virtualization is supplanting hardware-based methods as the approach of choice.

Virtualization is Cheaper Than Real Hardware

The most significant advantage to leveraging a virtual approach to system development is that it is significantly more cost-effective than real hardware, creating an immediate savings in the amount of capital that needs to be dedicated to software development and testing. This is especially true for organizations with huge product development teams, since the usual economics of software come into play.

While the first instance of software may be expensive, subsequent copies are much lower in cost. Unlike real hardware test racks—where there are never enough to go around to the entire team—a virtual platform can be available to every engineer who needs access to it. This economical scalability leads immediately to significantly reduced capital requirements and operational expenses. More subtly, as the marginal cost of additional test systems decreases, additional systems can be deployed, making it easier to raise the bar on quality.

Additionally, virtualization is ideal for simulating a complete test environment, a virtual test rack, which can take the place of a physical test rack than can easily cost several hundred thousand dollars. These racks consume a great sum of capital to create, and continuous expense to maintain and configure (Figure 1). The virtual test rack saves money directly, by costing less for each copy, and also generates new value by having a low marginal cost, thereby solving the problem of a team not having enough test units.

Virtualization is Better Than Real Hardware

Another considerable advantage to leveraging a virtual approach to system development is performance. Because of its significant improvements in the ability to observe and control the real-time code, the virtual approach is simply more attractive to developers than using real hardware.

One benefit connected to the virtual platform approach is that a development platform can be available much earlier. Compared to the full hardware development process, utilizing a virtual model is much more straightforward. By having the virtual platform available early, software development can almost completely overlap with hardware development. The virtual model is faithful enough that even an operating system port can run the same day that the prototype chip is delivered. In fact, some high-profile commercial projects have had virtual hardware available for developers more than a year ahead of pre-production hardware (Figures 2 and 3).

Full system virtualization also enables powerful debugging and inspection abilities for software at the hardware/software interface, raising its performance advantages to a level unattainable with traditional hardware development methods. Many problems in today’s systems occur during interactions between different products or multiple instances of the same product. These are precisely the areas that are the most difficult to test with real hardware, which is notorious for “heisenbugs”, bugs that, like a quantum particle, change their behavior when they are observed and disappear when you try to take a closer look at them. Being fully deterministic, the virtual approach makes tracking down these problems much easier, enabling you to resimulate until just before the bug occurs and investigate the internals at any level of detail necessary.

As the virtual platform is deterministic in a way that real hardware can never be, these bugs can also be recreated with ease. If a router crashes on a certain test run, it will behave in exactly the same way if the test is re-run. Since a virtual system can be check-pointed and re-started, many aspects of system testing become more effective than with real hardware.

Quite simply, virtualization goes far beyond the limitations of real hardware. With actual hardware, it is impossible to halt a router and freeze all the packets in transit. But a virtualized router can be break-pointed and probed while all the other simulated boxes freeze and single-step in unison. Normally, each test of a box requires that it be booted each time, but a virtualized system can be check-pointed at any convenient point. A new simulation can easily be launched from the checkpoint without re-executing the boot sequence. Moreover, multiple versions of the checkpoint can be launched, each injected with a different fault or running a different test script.

Full-system virtualization is beginning to create a sea change in the way that systems are designed and tested. Thanks to dramatic advances in technology driven by the ever-present challenges in real-time embedded system development, it’s now possible to simulate millions of dollars of hardware on a few thousand dollars worth of personal computing power.

As these inexpensive workstations become ever faster, and as simulation techniques continue to improve, the advantages of a virtual approach to advanced embedded system development and testing will only increase. The economics in favor of making the switch to full-system virtualization are compelling. A virtual model runs cleanly on even a laptop computer. So the next major advance in real-time embedded systems may take place on a plane or around the corner at your local coffee house!

Virtutech
San Jose, CA.
(408) 392-9150.
[www.virtutech.com].