Multicore System Development

What’s Going on in My Multicore System?

Real-time event analysis is critical for multicore software development as is the ability to analyze the processing burden on multiple cores to even the load and improve overall performance.


  • Page 1 of 1
    Bookmark and Share

Article Media

Real-time systems must react quickly to external and internal demands. When a system uses a multicore architecture, the speed and number of interactions rises sharply. While this improves the performance of a system, it complicates the real-time sequencing of application events since multicore system events can occur simultaneously over multiple independent processors instead of occurring sequentially as on a single-processor system.

For the multicore developer, the increased complexity of managing the number of events and their simultaneous nature represents an exponentially more challenging system to design. Diagnosing the cause of a system failure or inefficiency is much more difficult than with a single processor system. With few multicore-ready tools, developers have been left with primitive “print statement” techniques to leave “bread crumbs” throughout the operation of the system, indicating certain data about various events of interest that have occurred. The developer must gather and make sense of the crumbs and infer the system’s state—a process often requiring subsequent re-instrumentation to gain a finer degree of granularity and a repeat of the process.

To gain more efficiency in unraveling the intricate sequence of operations on a multicore system, developers need a tool that enables them to examine the system’s individual operations that immediately precede an area of interest. Much like an airliner’s “black box,” such a tool can be invaluable in shedding light on the critical events leading up to a certain point, or even a system crash. This article will show how this is possible using TraceX, a development tool that displays real-time events that occurred on a multicore system. As shown by the example in Figure 1, developers can see exactly what is going on in their multicore system across a particular period of time. A graphical analysis of all system events is displayed across a unified timescale, organized by application thread, and grouped by processor core.

Figure 1
Express Logic’s TraceX offers a graphical view of real-time events in a multicore system. In this example, you can see Core-0 and Core-1 simultaneously executing different threads.

The Traditional Approach to System-Event Analysis

Real-time programmers have long understood the importance of system behavior to the functionality and performance of their applications. The conventional approach addresses these issues by generating data on system behavior when the code reaches a certain stage by toggling an I/O pin, using printf, setting a variable, or writing a value to a file.

Inserting such responses requires a considerable amount of time, especially when you consider that the instrumentation code often doesn’t work exactly as expected the first time around and also needs to be debugged. Once that part of the application is verified, the instrumentation code needs to be removed and its removal also needs to be debugged. Since much of the instrumentation process is manual, the process is time-consuming and prone to additional errors.

Besides instrumenting the code, the developer also needs to find a way to interpret the data generated. The volume of information generated by the instrumentation code makes the task of determining what system events took place and in what sequence challenging. Modern debuggers can trace individual instruction execution, stop execution at a breakpoint, and show memory and register values at any point. But they lack the ability to show RTOS actions, like context switches, or semaphore gets, which can be valuable clues to system behavior.

New Approach Offers Advantages

In contrast, TraceX automatically analyzes and graphically depicts system and application events captured on the target system during run-time. Events such as thread context switches, preemptions, suspensions, terminations and system interrupts leave a trail of “bread crumbs” in a target-resident “trace buffer” that is uploaded, interpreted and displayed graphically on the host. The bread crumbs describe each event that just happened, which thread was involved, which core that thread was running on, when the event occurred and other relevant information.

The user also can log any desired application events using an application programming interface (API). Event information is stored (“logged”) in a circular buffer on the target system with buffer size determined by the application. A circular buffer enables the most recent “n” events to be stored at all times and to be available for inspection in the case of a system malfunction or other significant event.

Event logging can be dynamically stopped and started by the application program, such as when an area of interest is encountered. This avoids cluttering the database and using up target memory when the system is performing correctly. The event log may be uploaded to the host for analysis at any time, either when encountering a breakpoint, a system crash, or after the application has finished running. 

Once the event log is uploaded from target memory to the host, the events are displayed graphically on the horizontal axis which represents time (again, Figure 1). The various application threads and system routines related to events are listed along the vertical axis, and the events themselves are presented in the appropriate row. Events are represented by color-coded icons, located at the point of occurrence along the horizontal timeline as well as to the right of the relevant thread or system routine. Each event icon contains an abbreviation of the event itself, for example, “QS” is used to indicate a “Queue Send” operation. For multicore systems, the events are linked to their respective processor core and grouped together so that developers can easily see all the events for a particular core.

Figure 1
Express Logic’s TraceX offers a graphical view of real-time events in a multicore system. In this example, you can see Core-0 and Core-1 simultaneously executing different threads.

All events are also presented in the top “summary row,” regardless of core or thread. This provides developers with a handy way to obtain a complete picture of system events without scrolling down through all threads and cores. The axes may be expanded to show more event detail or collapsed to show more events. The timescale can be panned left (back) or right (ahead) to show any point in the trace buffer. When an individual event is selected, as in Figure 2, detailed information is provided for that event, including the core, context, event, thread pointer, new state, stack pointer and next thread point.

Figure 2
Individual event details can be displayed by clicking on an event icon.

Solving Priority Inversion Problems

One of the most challenging problems encountered in a real-time system is to resolve priority inversion situations. Priority inversions arise because RTOSs employ a priority-based preemptive scheduler that ensures the highest priority thread that is ready to run actually runs. The scheduler may preempt a lower-priority thread in mid-execution to meet this objective.

Problems can occur when high- and low-priority threads share resources, such as a memory buffer. If the lower-priority thread is using the shared resource when the higher-priority thread is ready to run, the higher-priority thread must wait for the lower-priority thread to finish. If the higher-priority thread must meet a critical deadline, then it becomes necessary to calculate the maximum time it might have to wait for all its shared resources in determining its worst-case performance.

Priority inversions occur when a high-priority thread is forced to wait while the CPU serves a lower-priority thread. Worse yet is the situation where a mid-priority thread interrupts the low-priority thread that currently holds the shared resource. In this case, the low-priority thread cannot continue, and hence, cannot finish its use of the shared resource. Thus, a high-priority thread needing that resource can be held up indefinitely if the mid-priority thread continues to run. This is unacceptable in a real-time system, since it prevents deterministic behavior.

Priority inversions are difficult to identify and correct. Their symptom is normally poor performance, but poor performance stems from many potential causes. Compounding the challenge of identifying the cause is the fact that priority inversion can also evade testing, only occurring infrequently and not in any test case that has been constructed for the system, which could mean that the system is non-deterministic.

With a systems event tool such as TraceX, it is possible to easily and automatically identify priority inversions. The trace buffer clearly identifies which thread is running at any point in time and records any change in a thread’s readiness. It is therefore easy to go back in time to determine whether a higher-priority thread is ready to run, but blocked by a lower-priority thread that holds a resource needed by the higher-priority thread. The priority inversion shown in Figure 3 is non-deterministic.

Figure 3
This display shows a non-deterministic priority inversion.

In Figure 3, we can see that Low_ thread holds a mutex (guarding a shared resource) when it is preempted by High_thread. High_thread then seeks the same mutex, but must wait for Low_thread to release it. However, Medium_thread has intervened and can run for an indeterminate length of time, delaying not only Low_thread, but also High_thread. Only when Medium_thread yields enough time to Low_thread for it to complete its processing and release the mutex can High_thread resume. Since there is no way to determine how long Medium_thread might continue to run, the system becomes non-deterministic.

Figure 3
This display shows a non-deterministic priority inversion.

Of course, there are other ways to avoid priority inversions of this type. For instance, “priority inheritance” for mutexes would prevent the inversion in this example. With priority inheritance, when a mutex, held by a thread, is needed by a higher-priority thread, the priority of the thread holding the mutex is temporarily raised to that of the requesting, high-priority thread. Thus, the low-priority thread now can run until it releases the mutex, and then its priority is restored to its original level. As a result, the high-priority thread then can get the mutex and continue its work.

Improving Application Performance

While most developers begin using tools to understand and correct problems, gaining an execution profile is a potentially broader benefit derived from using the tool to analyze and improve system-level application performance. Using an execution profile, developers see the amount of CPU time used by each thread and by system services (Figure 4). The developer can easily drill down on specific events for diagnostic purposes.

Figure 4
An execution profile, showing CPU time used by each thread.

Even more relevant to multicore system operation, balancing the processing load across all available cores can be very effective in achieving greater system throughput. If a system profile provides information about which cores have greater idle time, as is shown in Figure 4, it can give the developer a strong clue as to how to shift processing to an otherwise idle core.

Figure 4
An execution profile, showing CPU time used by each thread.

In conclusion, a tool such as TraceX paints a graphical picture of the system in a way that standard debuggers cannot. This enables developers to get a clear picture of interrupts, context switches and other system events that could otherwise only be detected through time-consuming instrumentation of code and tedious examination of the resulting data. The result is that developers can find and fix bugs and optimize application performance in substantially less time than would be required using standard debugging tools alone. With debugging taking up to 70% of application development, such tools offer the opportunity to significantly improve products while requiring less development time. 

Express Logic
San Diego, CA.
(858) 613-6640