RTC Magazine

Whether they realize it yet or not, test teams and their software development colleagues at embedded product companies are headed for trouble—or are knee-deep in it already. There is an increasing likelihood that they will discover serious defects in their products late in their software development cycles, or worse—after products have been shipped to customers. In a recent Wind River embedded industry survey, for example, 72% of the nearly 900 respondents reported that they had their product schedules disrupted by late-cycle quality surprises this year, which caused 58% of them to delay product shipment. 

These problems can cause significant and sometimes devastating damage to companies and careers. That’s why it’s time for executives, development managers and test team leaders to step up, recognize the problem and develop an action plan for dealing with it. 

The most obvious evidence of this growing problem is the recent news coverage of high-profile, software-related product failures. But a closer review of news outlets and technical journals turns up many more problems with embedded products stemming from their software components. Although most of these situations are handled outside of the media spotlight, they still cause significant damage to the companies involved. 

For leaders at embedded product companies, the first step back from trouble is to understand why these problems are happening. They need to see and appreciate how changes and trends in the industry are combining to make the job of producing software for embedded products to quality, on time and on budget much harder than it has ever been before. 

Product failures are actually the symptom. The “disease” is the fact that in many companies, software quality testing methods, tools and cultures have not kept up with the accelerating pace of change. At most embedded companies today, test and development teams are contending with skyrocketing software content, which is roughly doubling or increasing even more every two years. This brings with it increasing architectural complexity—usually involving 32- and 64-bit architectures, COTS operating systems, and often, multicore processors. 

In addition, evolving development models, such as Agile, Extreme and other iterative development environments, require early, often and continual testing throughout the entire cycle, not just at the end. But at the same time product development and test teams are being forced to deliver far more complex products in less time than ever before so delivery schedules remain tight, resulting in less thorough testing.

When Staying ‘Positive’ Doesn’t Pay

Most embedded companies today base their quality assurance strategy on “positive” testing of functional requirements. This means that the test group takes the user specification and exercises the device with manual and automated tests to see if all the features do what they were specified to do. Leading test teams actually map these positive tests to requirements to show traceability to tested features and results.

Positive testing is good—it’s a fundamental and necessary component of any QA or test program. But research shows that many, if not most, of the defects customers are seeing in the field relate to unusual conditions that the device was not tested to handle. 

In the aforementioned Wind River survey, the top two causes of shipped defects were “the product was used in unanticipated ways” and “edge conditions were not tested.”  Sources of these problems include unusual combinations of data inputs, overflows from long running operation, performance bottlenecks triggered by unanticipated sequences of events, and unrecoverable system conditions caused by hardware failures or other untested fault conditions. 

As a result, support personnel often hear statements from developers like, “Why are they using it like that?” or “We never tested the product in that failure mode.” That may be true, but it isn’t really acceptable. In the end, products are failing more often, and doing so in customers’ hands. In other words, development and test teams have been staying “positive”—and it isn’t getting the job done.