Software testing is an industry in transition as we become increasingly dependent on products whose functionality is controlled by software. For example, the industry is learning to adapt to the effects of Technical Debt on IoT as more devices become interconnected.
by William McCaffrey, Co-founder and Chief Operating Officer of Vector Software
Software was once seen as the “holy grail;” since it is not mechanical, many organizations figured it could be written once and used many times without breaking down. But problems began to appear, ultimately caused by the continual development of software without the correct quality control processes, typically due to incredible business pressures to release new products.
This has resulted in legacy software applications carrying an enormous amount of technical debt, a metaphorical term for latent defects introduced during system architecture, design or development. The accumulated liability of technical debt created when organizations take design and test shortcuts in order to meet short-term goals eventually make software difficult to maintain. As technical debt increases, developers spend a majority of their time fixing bugs and struggling with fragile code, rather than building new features.
For this reason, legacy code bases are one of the biggest challenges in software today. Many organizations are finding that legacy software typically has a waning lifespan; after that, they are forced to decide whether to throw it away and start again from scratch or try to salvage it. In most cases, a substantial financial investment has been made in the code base, and there is tremendous pressure to re-use it.
Security Concerns Increasing in an IoT World
When software was held in proprietary systems or closed enterprise environments, technical debt issues could stay somewhat contained. However, with the Internet of Things (IoT) enabling the interconnection of the physical and virtual world based on interoperable communication technologies, technical debt becomes a more serious problem. Gartner predicted that there would be “4.9 billion connected things in 2015, up 30% from 2014, and…[could] reach 25 billion by 2020. Note 1”
In IoT, essentially every electronic device will have network connectivity. That means every manufacturer of electronic devices will also be in the software business. This redefines the need for security by expanding the scope of responsibility into new platforms and services. The demand for predictable behavior is paramount, especially if the safety of users or the environment is at risk. However, in a fiercely competitive industry such as IoT, first-to-market advantage is huge, and developers will be under intense pressure to get products released quickly.
It has been proven time and again in software development that this thinking sacrifices quality for speed. This tradeoff can be dangerous with regard to many IoT-enabled products such as smart cars, medical devices and home safety systems. Malfunction of these systems can put lives at risk. Stricter software quality measures and a focus on testing are going to be a top priority for many senior executives.
Characterizing the Behavior of Software
The key to reducing technical debt is to refactor components (the process of restructuring application components without changing its external behavior/API) over time, but developers are often hesitant to do so for fear of breaking existing functionality. One of the biggest impediments to refactoring is the lack of tests that formalize existing behavior, or a testing environment that proves the correct behavior of the component. Without refactoring, an application’s code becomes overly complicated and difficult to maintain. As new features and bug fixes are bolted onto existing functionality, the original design often loses its simplicity.
According to a Gartner study, “a lack of repeatable test cases limits an organization’s ability to demonstrate functional equivalence in an objective, measurable way. Note 2” This is a growing problem as there are many deployed/fielded applications based on legacy code bases that don’t have test cases to run against. The reason companies are facing an issue in the first place is often that they want to deploy software in a new platform/product and they have essentially broken the existing test frameworks needed to validate the software. There is a massive quality gap that needs to be addressed along with insurmountable work ahead – and companies usually don’t know where to start or don’t have the resources needed to address the problem.
The lack of sufficient tests typically means that a software application cannot be easily modified since, unfortunately, changes frequently break existing functionality. If some existing capability is broken, the developer needs to understand if it was because the original software was written incorrectly or missed a requirement that wasn’t adequately captured in the original implementation.
Baseline testing, also known as characterization testing, can be done to characterize the current behavior of software and is useful for legacy code bases that have inadequate test cases. Once the current behavior of the software is understood, it is then reasonable to look at the source code as a basis for defining test scenarios.
By using automatic test case generation to quickly provide this baseline set of tests that capture existing behavior, testing completeness of legacy applications is improved and refactoring can be done with confidence that application behavior has not regressed.
Approach for Paying Off Technical Debt
The baseline testing process will determine how much technical debt an application is carrying. To address it, an organization will usually attempt to go through an exercise of refactoring.
Once the behavior of the software has been characterized through baseline testing, a developer can start making updates and modifications to the code. Change-based testing can then be used to run only the minimum set of test cases needed to show what effect the code changes have on the integrity of the whole system. It is not uncommon for a company to take one to two weeks to run all of its test cases. With change-based testing, a developer can make a “one-line” change and get feedback on its impact to the entire application within a few minutes.
See Figure 1.
Figure 1 Caption: Approach to paying off technical debt. Once the behavior of software has been baselined, developers can make modifications to the code with confidence that application behavior has not regressed.
As a result, developers are able to make quick increments on the code while knowing that they have the test cases needed to capture the existing behavior of the software. They are also able to do further analysis if something is broken to work out if a bug has been introduced, a capability has been removed that actually should be there, or if there is a bug that should be addressed because it may have other ramifications.
IoT and Legacy Software: When Consumer-Grade Systems Become Safety-Critical
There has been an interesting shift driven by IoT that has resulted in a new generation of software that previously did not have safety-critical requirements but now does. For example, in the era of self-driving cars, the telematics system is becoming one of the most critical in-vehicle systems. Autonomous driving capabilities have shifted this software-driven system from a consumer-grade communication application to safety-critical as certain autonomous driving capabilities are reliant on telematics information.
This telematics information is enabling potentially life-saving applications. For example, if a vehicle five cars in front of yours brakes, instead of you having to wait for a visual cue from the vehicle directly in front, the vehicle five cars ahead will send a broadcast to your vehicle over the telematics interface that it is braking –and that your vehicle also needs to start braking now. In this case, faulty software has severe ramifications, so quality is no longer an option — it is a necessity.
Existing software usually has to be updated to accommodate these new capabilities and there often isn’t evidence to demonstrate how the software worked correctly in the past. Baseline testing can be used to characterize the behavior of the existing software and to build the test cases needed to enable developers to make needed updates with quality and confidence.
In this example, the telematics system could possibly have latent system defects (technical debt) that may not be critical in its current environment — but if the product were to be deployed in another environment, those errors could manifest themselves as critical. Baseline testing has two main benefits here: it will uncover the latent bugs, and when developers are doing a new revision in the software, they can more easily make those changes without worrying about new issues as a result.
William McCaffrey co-founded Vector Software in 1990. McCaffrey teamed with John Paliotta in 1994 to develop the initial VectorCAST® dynamic testing product. He is currently the COO overseeing worldwide sales and marketing operations and business development activities at Vector Software, and has over 30 years of experience in software development specializing in safety-critical embedded systems.