RTC Magazine

Whether they realize it yet or not, test teams and their software development colleagues at embedded product companies are headed for trouble—or are knee-deep in it already. There is an increasing likelihood that they will discover serious defects in their products late in their software development cycles, or worse—after products have been shipped to customers. In a recent Wind River embedded industry survey, for example, 72% of the nearly 900 respondents reported that they had their product schedules disrupted by late-cycle quality surprises this year, which caused 58% of them to delay product shipment. 

These problems can cause significant and sometimes devastating damage to companies and careers. That’s why it’s time for executives, development managers and test team leaders to step up, recognize the problem and develop an action plan for dealing with it. 

The most obvious evidence of this growing problem is the recent news coverage of high-profile, software-related product failures. But a closer review of news outlets and technical journals turns up many more problems with embedded products stemming from their software components. Although most of these situations are handled outside of the media spotlight, they still cause significant damage to the companies involved. 

For leaders at embedded product companies, the first step back from trouble is to understand why these problems are happening. They need to see and appreciate how changes and trends in the industry are combining to make the job of producing software for embedded products to quality, on time and on budget much harder than it has ever been before. 

Product failures are actually the symptom. The “disease” is the fact that in many companies, software quality testing methods, tools and cultures have not kept up with the accelerating pace of change. At most embedded companies today, test and development teams are contending with skyrocketing software content, which is roughly doubling or increasing even more every two years. This brings with it increasing architectural complexity—usually involving 32- and 64-bit architectures, COTS operating systems, and often, multicore processors. 

In addition, evolving development models, such as Agile, Extreme and other iterative development environments, require early, often and continual testing throughout the entire cycle, not just at the end. But at the same time product development and test teams are being forced to deliver far more complex products in less time than ever before so delivery schedules remain tight, resulting in less thorough testing.

When Staying ‘Positive’ Doesn’t Pay

Most embedded companies today base their quality assurance strategy on “positive” testing of functional requirements. This means that the test group takes the user specification and exercises the device with manual and automated tests to see if all the features do what they were specified to do. Leading test teams actually map these positive tests to requirements to show traceability to tested features and results.

Positive testing is good—it’s a fundamental and necessary component of any QA or test program. But research shows that many, if not most, of the defects customers are seeing in the field relate to unusual conditions that the device was not tested to handle. 

In the aforementioned Wind River survey, the top two causes of shipped defects were “the product was used in unanticipated ways” and “edge conditions were not tested.”  Sources of these problems include unusual combinations of data inputs, overflows from long running operation, performance bottlenecks triggered by unanticipated sequences of events, and unrecoverable system conditions caused by hardware failures or other untested fault conditions. 

As a result, support personnel often hear statements from developers like, “Why are they using it like that?” or “We never tested the product in that failure mode.” That may be true, but it isn’t really acceptable. In the end, products are failing more often, and doing so in customers’ hands. In other words, development and test teams have been staying “positive”—and it isn’t getting the job done.  

Getting Negative with White Box Testing

To catch and address these issues before they blow up into major problems at customers’ sites, companies need to do much more “negative” testing. They need to run tests that try to break the system, validate fault and exception handlers or otherwise force the device under test into an unusual state or “edge condition.” 

Making this shift requires a mindset change among team members—a focus that expands beyond what should happen to what could happen. Teams need to start thinking more about how to anticipate and test for failure modes and other conditions that devices may encounter in the real world. Increasingly devices are no longer single function and are part of an ecosystem of interoperable devices. Therefore the potential modes of operation and  modes of failure are exponentially greater. 

This transition also presents an opportunity to institute more of a “design-for-testability” approach that starts up front in the development process. Product requirements should specify anticipated failure modes and should contain requirements such as “The device will have a fault handler to cover the following conditions and these shall be tested as specified…” Test planners need to translate these requirements into suitable test plans. Software developers must then work with test developers to provide appropriate “hooks” to allow automated tests to drive the software or device into failure conditions during testing. 

Given that many of these failure modes can be time-consuming and expensive to test using traditional manual methods, test planners need to look at new tools that can allow automated testing using “white box” techniques that let tests access device internals at runtime. This not only saves time and money, but more importantly, allows deep and thorough regression testing of edge conditions “early and often,” with each new software build. 

Leading unit test tools like IPL’s Cantata++ and modern device simulation systems like Wind River’s Simics provide white box features for developers to use. Now new embedded test optimization solutions like the Wind River Test Management system are providing white box testing capability for quality assurance professionals. 

These new test optimization systems use new dynamic instrumentation technology to allow white box access to the actual production software on the embedded device under test—at runtime. They allow injection of test data or code in synch with test cases without stopping or rebooting the device. This enables automation of normally hard-to-test scenarios by easily injecting faults, forcing failovers, or otherwise triggering exception conditions at runtime. 

In white box testing, developers working to the test plan create reusable dynamic instrumentation that can be used to manually or automatically input data, force states, read or log data at a given subsystem, or inject faults or failures in synch with test cases (Figure 1). Example uses of dynamic instrumentation for white box testing include:

• Injecting specific values into variables
• Forcing faults or conditional expressions to execute code paths, error handlers, or failover
• Logging internal device data generated at runtime for validation against expected parameters
• Altering the system’s environment (e.g., change values read from sensors, set CPU and peripheral controllers register values, modify streaming data)

White box testing features like this can now dramatically expand the range and depth of both positive and negative testing that can be accomplished on device software while still meeting product delivery schedules and staying within testing budgets. 

Focus on the ‘Deltas’ with Change-based Test Automation

Negative testing is far easier to accomplish when the testing group understands how the code is structured, what their test suites are actually testing, and what code is actually not being properly tested (e.g. error handlers). This information can be extremely difficult to obtain using only traditional tools. Fortunately new dynamic instrumentation technology and tools can now provide this visibility. 

Using the same white box techniques, auto-generated instrumentation can be deployed into the production software at runtime to gather test coverage information as test suites are run. This tells the user specifically what software has been tested and, more importantly, what software has not been tested. Test coverage reports provide pointers to areas of the code needing special attention. 

Test coverage information can then be leveraged to generate a traceability map that shows specifically what code is exercised by which test cases. This provides clues to the developer and tester as to specifically what tests verify what code and where to apply resources to enhance test suites to fill gaps.

As software changes over the life of the project, testers have to rerun their tests to guard against regressions. The traceability map can also be used to help testers then decide specifically what tests to rerun to verify this hard-to-test code. For example, an automated binary compare between software builds can be used to identify changes, and leveraging test coverage information and the traceability map, can determine precisely what tests need to be rerun to verify changed code (Figure 2). 

Each of these white box techniques works together to give the device testing professional better visibility and control of the device under test. By helping testers “see inside” embedded systems at runtime, these white box testing systems can help engineers and QA professionals test more thoroughly. They also help management get more accurate and thorough assessments of the quality of their software projects at any point in the development process, so they can focus their scarce time and resources more effectively.

It’s Time for Testers to Step Up

None of this will be possible, however, if QA and test teams and their management don’t step up to the challenge. More sophisticated devices require more sophisticated approaches to testing them. 

QA teams need to take a more active role in understanding how these devices work internally. They need to work much more closely with development teams, particularly when using new iterative and agile methodologies. They also need to provide an independent view to device operation and deployment that can project where additional failures could occur from operator error, component failure, or complex deployment configurations. They must anticipate how devices may fail in ways that were not anticipated by designers

Tools are available today that can help. Real advances have been made around leveraging real-time, white box input from devices under test and this can significantly help create more focused, more informed and more automated quality testing. These new advances enable teams to build on the positive testing components they already have in place, and make comprehensive negative testing an integral and cost-effective element of their overall programs.  

Wind River
Alameda, CA.
(510) 748-4100.
[www.windriver.com].