RTC Magazine

Virtual memory, virtual storage, virtual hardware, virtual I/O, virtual systems, virtual functions, virtual machine monitor… Are these devices part of a virtual reality where we live and work in virtual worlds with our virtual colleagues, friends and pets? That might depend on how you define your friends!

There are many ways to use the term “virtual” in the context of computing systems. The term is heavily overloaded, with many specific meanings, but all of them emanate from a sense of being “real without being actual, ideal without being abstract.” French philosopher Gilles Deleuze described the concept of virtual as “that [which] every object carries with it, which is neither its reality, nor merely what it could have been, but rather what it is imagined to be.” In other words, a virtual system is what we make of it. Thus is the challenge of applying virtualization technology to real-time embedded systems. Stray too far from reality and you compromise performance and determinism. Stay too close to home and you will never reap the benefits.

Virtualization for Servers

Much of the information available about virtualization revolves around the use of a hypervisor also known as a virtual machine manager (VMM) for consolidation of multiple physical servers onto a single hardware platform. The obvious benefit is cost reduction brought about by using less hardware and decreasing energy consumption. Other benefits include the ability to quickly reconfigure and redeploy virtual servers on different physical hardware platforms as a function of loading and performance.

In server applications the VMM presents a standard set of virtual hardware to each guest OS (the operating system running inside the virtual machine). For example, the network interface card (NIC) presented by a server VMM might always appear to be an NE2000 network card. The I/O is defined by the server VMM, regardless of whether your real hardware is a 3Com, Realtek, or Brand X NIC.

The server VMM translates all I/O requests inside your guest OS into real I/O access to real hardware. This virtual to real translation happens for sound cards, keyboards, mice, etc. Limiting the size and scope of the virtual hardware presented to the guest OS makes it easy to support a range of real hardware platforms and without constraining the amount of I/O needed to support common server applications.

Hypervisor technology is becoming part of the server operating systems in the form of VMware ESX Server, Citrix XenServer and Microsoft Hyper-V. If you’re consolidating multiple servers this is good news. But these hypervisors are not appropriate if your application is embedded and you require real-time services or access to special hardware.

Virtualization for Embedded Applications

The server VMM is targeted at solving problems for corporate IT networks by maximizing the use of server resources and simplifying deployment and maintenance. A server VMM can’t easily accommodate providing a variety of specialized hardware devices to the guest OS because it is trying to present the application with a standard “white box.” Unfortunately, this approach doesn’t work well for embedded applications, where predictable performance and the need to interact with unusual and special hardware are paramount.

Embedded developers need a different approach to virtual machine management to support their specific I/O hardware needs and to simultaneously provide the performance needed for a deterministic environment. Such an approach requires a VMM that assigns hardware directly to the applications and drivers that control them, as illustrated by Figure 1.

A key difference between the embedded VMM model and the server VMM model is how physical resources are allocated to each virtual machine. The embedded VMM model partitions CPU cycles, RAM and I/O between each guest OS rather than multiplexing those resources among the virtual machines, as is the case with a server VMM. This AMP (asymmetric multiprocessing) model of resource allocation is useful where determinism and performance are more important than equal access and maximum hardware utilization. The virtualization technology built into many of the multicore processors available today can be used to isolate resources for use by a specific virtual machine and its guest OS.

Even in the AMP model, which is the basis of the embedded VMM, not all I/O is required to be exclusive. Some may or should be shared, such as the hard disk, an enterprise Ethernet adapter and a console device. In these instances the VMM includes virtual devices to facilitate sharing hardware between virtual machines.

Hardware Required

Key to ensuring determinism in an embedded VMM environment is the use of multicore processors, which allows the hypervisor to dedicate CPU cores to specific guest operating systems. This removes the requirement to share or time slice CPU cycles between each virtual machine and has a dramatic impact on real-time performance metrics, such as interrupt latency and determinism, when compared to platforms that must share CPU cycles between each virtual machine. Dedicating individual cores of a multicore processor to a VMM is a simple and relatively inexpensive option, given the number of multicore processors now available.

In addition to the use of multicore processors, different levels of virtualization technology are available, mostly as a function of the processor, chipsets and I/O devices. Table 1 outlines those technologies as defined for Intel Architecture processors.

VT-x: Hardware virtualization assists the process of implementing an embedded hypervisor (or VMM). Many of the latest embedded Intel multicore processors include VT-x (Table 1), which is a fundamental component used to create deterministic virtual machines. VT-x is a collection of processor instructions, special traps and a privileged “root mode” that helps VMM software efficiently host multiple virtual machines on a single hardware platform.

Prior to the availability of VT-x, creating a virtual machine required that the hypervisor and each operating system share ring 0, or supervisor mode, of the CPU, where both have full access to system level instructions (applications generally run in ring 3, or user mode). Implementing virtualization without the aid of VT-x requires that the VMM or hypervisor be carefully sandwiched between the CPU system-level hardware and the OS, a complicated chore that incurs significant overhead on the part of the hypervisor.

Using VT-x, the hypervisor runs in the new overarching “root mode,” allowing each guest OS to run in ring 0 without modification and without the complicated sandwiching tricks required to share a CPU in ring 0. Instead, the hypervisor sits between each guest OS and the CPU system-level hardware. From this vantage point the hypervisor can emulate and isolate hardware devices. For example, a virtual NIC in each guest OS can be used to transparently provide communication services between guest operating systems.

VT-d: By remapping DMA (direct memory access) operations, VT-d allows bus-master devices to run at near full speed in a VMM environment. Without VT-d the VMM must emulate such hardware, resulting in significant execution overhead by the hypervisor. VT-d assigns I/O devices to domains by providing remapping hardware that restricts DMA transactions from a specific I/O device to operate only in the physical memory present in its virtual machine, avoiding costly device emulation.

The remapping hardware sits between the DMA peripheral I/O devices and physical RAM and is programmed by the hypervisor to restrict transactions to a specific memory range or domain. Each domain represents that subset of the physical memory corresponding to a virtual machine.

While VT-d is not explicitly required to implement a virtual machine, hypervisor overhead can be reduced by using VT-d because VMM operations are limited to only those required to facilitate access to protected resources by the guest OS (such as I/O configuration space, interrupt controllers, timer hardware, etc.).

VT-c: More widely known as “PCI-SIG IOV,” VT-c is a set of standards for sharing PCI Express devices between multiple independent entities. A PCI Express device that supports IOV includes facilities to negotiate and share its internal device functions. The IOV extensions effectively make a single PCI Express card appear to contain multiple virtual functions that can be discovered, configured and managed by multiple operating systems.

In a VMM environment the primary IOV mechanisms of interest are ATS or “Address Translation Services” and SR-IOV or “Single-Root I/O Virtualization.” Combined, these provide a means by which multiple operating systems, each running in a virtual environment on a single machine, can share a PCI Express device with minimal hypervisor overhead.

Like VT-d, VT-c is also not required to implement a virtual machine. It can be used, however, to further reduce the overhead associated with a VMM, thereby increasing the performance of each of the guest operating systems hosted by the hypervisor.

Working with Windows

While a segmented form of virtualization is possible on a 32-bit x86 architecture, for example without VT-x, it must be tailored to the specific operating systems in use. An embedded VMM can support a wide range of general-purpose and real-time operating systems and can emulate legacy hardware for migration of legacy operating systems to new hardware platforms and combine multi-platform systems into single-platform systems for significant cost savings.

To be of value, each guest OS must run unmodified in its own virtual machine, without losing determinism or access to specialized hardware. Thus, the primary role of an embedded VMM is to isolate and partition each guest OS with minimum overhead. This is the tactic implemented by the TenAsys eVM Platform for Windows, an embedded VMM designed to combine a variety of standard real-time operating systems with the Windows OS (Figure 2).

Granting exclusive access to essential I/O is necessary for real-time responsiveness because it means each virtual machine has real physical access to its key hardware. Without exclusive physical assignment of key I/O, you run the risk of waiting indeterminately for hardware. A server VMM emulates each I/O request from the virtual machine by translating guest OS accesses into real I/O accesses to the physical hardware, something an embedded VMM cannot afford to do in order to ensure deterministic behavior.

Exclusivity of I/O does not apply only to a real-time guest OS. For example, medical imaging applications need access to real hardware for maximum performance. A virtual frame buffer may be too slow and inadequate in features for it to render moving 3D images. In that case the Windows guest OS would benefit from direct access to the physical frame buffer and its control I/O.

Inter-OS Protocols

Virtual device drivers can be used to facilitate inter-OS communication and signaling protocols in a multi-OS virtual machine environment. For example, an inter-OS protocol has been implemented entirely within a virtual PCI hardware interface in the TenAsys eVM platform. The guest operating systems are configured to share an area of physical memory to which common data is posted. After a guest updates its data structure in the shared memory region, it signals the other guests of the update via a register in the virtual PCI interface.

In Figure 3, each virtual PCI device presents two memory ranges to each guest. The first memory range, pointed to by PCI configuration register BAR0, maps the shared memory buffer. The second range, pointed to by BAR1, presents an I/O address to each guest OS. When an application within the guest OS accesses the BAR1 I/O address, a trap is made into the virtual device driver hosted by the hypervisor. The virtual device driver then injects a virtual IRQ into the target guest OS, which responds by accessing the shared memory area for updated data.

With the help of an embedded VMM, legacy code can migrate from obsolete hardware to modern embedded platforms. Because I/O can be virtualized, it’s possible to simulate old hardware devices, minimizing rewrite of proven legacy code. For example, an obsolete ISA device could be simulated within the hypervisor by intercepting I/O requests and redirecting them to equivalent on-board PCI I/O devices.

Multicore processors easily support multiple operating systems and high-performance, low-latency, deterministic applications by dedicating a CPU core to each guest OS. The CPU instruction cycles of each core are available exclusively to its dedicated virtual machine. Virtualization technology can be used to remove I/O contentions by isolating and dedicating hardware for the exclusive use of an individual guest OS. I/O that can be shared, like disk and network interfaces, are presented as virtual devices for use by all virtual machines.

The net gains from the application of embedded VMM technology on x86 multicore processor platforms are the elimination of redundant computer and communication hardware, faster communication and coordination between guest operating systems, improved reliability and robustness, re-use of proven legacy applications, and simplified development and debugging. Significant cost savings can be achieved by condensing systems comprised of separate hardware platforms onto a single, multicore, hardware platform.

TenAsys
Beaverton, OR.
(503) 748-4720.
[www.tenasys.com].