RTC Magazine

Today Linux is the fastest growing embedded operating system in the world. Bringing Linux up on an embedded target and debugging it along with its associated applications presents new challenges for embedded systems programmers. These challenges, in turn, must be addressed by the designers and suppliers of tools for these programmers.

It is difficult to port Linux to a new platform without some method of accessing the system boot information console and the login console. On many headless targets there is either no good candidate for this functionality or such a candidate is not available at boot time. For example, cell phones simply do not have serial interfaces. In addition, the MMU-based memory model makes classic halt-mode debugging difficult or even impossible. Because all tasks share the same virtual address in Linux, classic breakpoint methods need modification. Debug register breakpoints (hardware breakpoints) are of little use.

A new debug topology combines halt-mode run control with run-mode debugging and DCC-based communications to effectively solve all of these issues. This method provides a test environment console that uses virtually no production resources. In addition, it combines the best features of halt-mode and run-mode debugging and provides the programmer with a feature-rich and effective way for him or her to get an embedded Linux-based project completed, on time and fully tested.

Linux in the Embedded World

Embedded designs are changing in nature and requirements. In the past, "embedded" usually referred to relatively small computer systems contained in a hardware product that showed almost no characteristics of a "computer". This hardware product contained tiny amounts of memory and, therefore, small and very efficient programs. These products usually had neither the means nor the requirement to communicate with other programmed products. If these machines contained operating systems at all, they were usually lean and not very feature-rich. "Small" and "fast" were the key drivers for these systems.

Today’s embedded projects are becoming more and more sophisticated, both in terms of features and underlying hardware. Most embedded systems include some type of connectivity, such as Ethernet or USB. Cheaper and faster processors and memory are increasingly becoming available for embedded systems. Many of today’s cell phones accept downloaded programs. Along with more software and connectivity comes the requirement for remote configuration and management. It is common to keep the phone book from a cell phone on a desktop PC.

With all of these software-based features being added to embedded products, software reuse is more necessary than ever. Developers must acquire operating systems, protocol stacks, http servers, remote management and communication software and many other software components in order to produce the required features. Writing these components can produce noncompetitive development costs. To be competitive, these components must be selected from those in general use. Fortunately, many of these components exist in the public domain and in the open source community.

In order to deploy the software components required, the design must be based on an operating system on which these components can run. That tends to limit the field to UNIX derivatives or WindowsCE. Many will argue that only a UNIX-conforming platform will get the job done. The most popular selection today is Linux.

Debug/Deployment Challenges

In order to use Linux, embedded engineers must first port it to their hardware. Then they must write and debug any custom drivers required for their application as well as write and debug the boot-up code. The final stage is to write and debug the applications that will run under Linux. This process offers several tool challenges that do not exist when using more classic operating systems such as Vxworks or other "flat address" environments.

Porting Linux to a new target requires a BSP or boot code just like any other solution. Once the boot code instantiates and attempts to launch the kernel, several new requirements surface. Linux is a large body of code; the first order of debugging a new port is to observe the kernel process while booting to determine what has succeeded, what has failed and where the process stopped. One common way of monitoring this process is to observe the system console messages. This is obvious, but not necessarily easy. Many embedded applications have no UART or other common port on which to attach a console. Even if the final product does have such a port, it may not be functional until new drivers are written and debugged. In this case, debug of the driver is going to be a serious challenge since the kernel may still have problems. Getting that first kernel message may be difficult.

Most JTAG emulators use a debug state called "halt mode." In this state the processor is either physically stopped or redirected to special software that makes it look like it is physically stopped. The firmware engineer then examines and makes changes to the system in an entirely static condition. This is the view that is offered by most embedded debugging tools. For Linux application debug, however, this may present problems.

Once the kernel is booting successfully, the process of introducing and debugging the embedded application begins. Here another real challenge is encountered. Unlike operating systems that use a flat memory model, Linux dynamically maps all applications to the same virtual address space. This renders debug register-based breakpoints virtually worthless in debugging application code. Debug register-based breakpoints (sometimes called hardware breakpoints), in most architectures, monitor the virtual address. This is true for all ARM architecture processors including Intel XScale processors. Because these comparators are attached to the virtual address bus (IVA or DVA), they will detect a match to a particular virtual address without regard to the physical address currently mapped. This will cause the same address in any application (say 0x8330) to cause a trigger.

Triggering on a virtual address means halt mode debugging in the application will not be viable. Software breakpoints are clearly the answer. In order to use them, the halt mode debugger must know where the code is physically at all times and be aware of any reloads caused by the kernel. Address aliasing between user tasks also presents a new kind of challenge for locating and displaying the source code related to the program being debugged. In addition to the code address aliasing (physical versus virtual), another factor is the need to use a console-based login to control and observe tasks and applications. This may present the same problems as the initial console.

Some of these challenges may be met via "kernel aware" halt mode debuggers. These debug solutions need to be kept "in-sync" with a rapidly changing code base for Linux. Other challenges, such as console access, have not been met by current tools at all.

A New Paradigm

A different approach can be used to solve the debug challenges encountered when working with embedded Linux. This represents a new paradigm that has two parts. The first is a "tunnel" into the processor that provides a "hardware port" that is independent of target system peripherals. The second part is a debug kernel that runs on top of Linux and communicates to the remote hosted debug session via this virtual port. This virtual port also supports the console facilities.

The virtual communications port provided by this new paradigm is based on the DCC registers that are part of both the ARM and XScale architectures. This virtual port provides both a console connection and debug communications through a JTAG-interfaced run control unit (Figure 1) .

In this way, the console is available from power-up without any other hardware dependencies except the processor core and the JTAG debug port. This port is almost always used during boot and kernel operations. Thus, the very first messages produced by the kernel are available to the embedded software engineer (Figure 2).

Once the kernel is up, all user space software debugging and program control is done via a debug agent running as a user process under Linux. The agent uses standard Linux APIs to control and interrogate user space programs. This allows the debugger to more easily stay "in-sync" with subsequent kernel versions. The agent runs during the entire process so all of the usual system information is available, either through Linux utilities (such as ps) or directly from the remotely hosted debugger. The embedded engineer has his or her choice of tools and methods for each problem.

All debugging can be accomplished via the source-level debugger, or the user can use GDB or any other tool via the console window (Figure 3). When using a debugger such as SourcePoint, the engineer may switch between run and halt mode debugging at any time; the interface directly into the kernel is visible in its entirety and rendered at source-level with all symbols handled. Breakpoints can be set in each task that break only when that task reaches that point. All of these features can be greatly enhanced with real-time trace capabilities in the same tools.

Run Mode Features

"Run mode" debugging must offer features that are not available or are, at best, marginal in halt mode. These features include:

• Full process "observability"
• Full console support, independent of target peripherals
• Target file system write (for such things as application introduction)
• Remotely hosted debugger using only the JTAG connection
• Seamless connection to halt mode
• Real-time trace

These features are in addition to the expected ones such as:

• Single stepping
• Running to breakpoints
• Viewing programs
• Viewing and changing data objects in their natural format

Consequently, in run mode, programming engineers can debug their applications with full knowledge of them and other processes. They can examine the state of each process and understand "what is waiting on what." The program and all of its data can be displayed using all available source-level information.

The Best of Both Worlds

While run mode debug is the clear answer for application debug in Linux, there are times when halt mode is still the answer. If the engineer is tracing a call down into the kernel, he or she will want to stop the kernel and look at what is going on. This follows through to kernel drivers. These areas are all fully visible only in halt mode (with the processor stopped). Emulator tools that support this allow setting of hardware breakpoints that cause the processor to enter halt mode during a run mode debug session. In this way a problem or code path can be followed in its entirety. When ARM’s ETM is available, full sequential breakpointing can be used to enter halt mode.

In addition to debugging across the application/kernel boundaries, halt mode is very powerful in scripted board initialization. In this way, the kernel can be launched and debugged long before the boot code is written or functional. Kernel and application work can proceed while the boot code is being written and debugged. This produces faster times-to-market.

Key features include:

• Kernel download
• Programming flash memory
• Real-time trace (including precise time stamping)
• Profiling

Using Linux in the embedded world will be much easier with the introduction of some new tools that tackle the tough obstacles in this job. The biggest obstacles can be classified into two areas. These are 1) the need for a virtual port into the target hardware system, and (2) the need to get clear display and control of applications in spite of the address aliasing issues. The ARM architecture contains a "DCC"-based communications facility that can supply this "virtual port." Software/hardware products that can use this functionality can be produced to cleanly solve both problems. These tools will save weeks in any Linux-based embedded project.

American Arium
Tustin, CA.
(877) 508-3970.
[www.arium.com].