Presentation Title: Innovations for Testing
Company: Test and Verification Solutions
Country: United Kingdom
Concurrent software exploits the availability of multiple CPU
cores on today's modern hardware platforms in an effort to improve
performance. However, concurrency introduces new classes of bugs
(e.g. deadlock, race conditions) and increases the number of
different possible executions through the code. Research suggests
that 10% of the bugs in concurrent software are due to the parallel
nature of the execution. Unfortunately, our traditional testing
techniques for sequential code will not find these bugs.
In this talk we discuss the industry drivers for moving to
multicore hardware platforms and how this will impact how software
is developed. The talk will cover in detail the challenges of
testing concurrent software and discuss the innovations required in
our software testing processes required to meet the challenges: in
the types of tests we need; in the metrics we need to measure the
effectiveness of our
testing; and the methods we need to debug test failures. The talk
will offer possible solutions to these innovations. Such solutions
and metrics are vital to the effort to ensure the quality of
The industry drivers for the move to concurrent software The
unique challenges of testing concurrent software The innovations
needed to ensure concurrent software is properly tested.
Every few years a new technology comes along that changes the
way we develop software - for example the move from machine code to
assembler to 3GL's; relational databases; the internet and mobile
applications. We are now entering the era of multicore.
We are used to seeing performance double every 2 years (known as
Moore's law after Gordon Moore of Intel) through faster clock
speeds. Unfortunately power consumption is exponentially related to
clock speed and so such speed improvements are no longer acceptable
(battery life, electricity costs, green issues,...). So the
solution now coming from hardware designers is to get performance
improvements through multiple CPUs. Increased performance with
linear power - problem solved!
Well - solved from the hardware perspective. The problem is now
passed to the software community to make use of the extra CPUs. For
example, your next server or mobile phone could have 8 cores (or
CPUs) and your programmer needs to be able to write software that
distributes the processing load over those cores. This is hard -
So how does all of this affect me the tester? The answer is VERY
significantly. Parallel programming has been around for a number of
years (mainly as a niche activity) and research suggests that it
introduces new classes of bugs which add about 10% to the total bug
count. So, you may say, what is the fuss? It is just a 10% increase
in bugs - not particularly significant.
Well, the problem is that the new 10% are much harder to find.
The new classes of bugs include non-determinism. This is where the
precise order of program execution is no longer predictable
(remember - it is now parallel rather than linear) and different
executions can yield different results. Debugging such failures are
notoriously difficult as the bug disappears when you try to
re-create it - that is the nature of non-determinism! Other new bug
types are deadlock, race conditions, livelock,...
Multicore will also necessitate a change in our completion
metrics as the current ones do NOT take into account issues caused
by parallel execution and the new classes of bugs mentioned
This paper is not for the feint-hearted. It discusses
technically challenging issues related to changes in the way we
need to test to react to changes in the way we program for
If you want more detail on this then you can access a lot of
detail on this on the TVS company website at:
Dr Mike Bartley gained a PhD in Logic at Bristol University and an
MSc in Software Engineering and MBA with the Open University. More
recently he has added ISEB Practitioner Testing Qualifications in
both Test Management and Test Analysis. Mike has been involved in
software and hardware development for 20 years as both a tester and
test manager with responsibility for the signoff of a number of
complex products. Mike has a particular interest in multicore
software and how this will affect the software testing industry.
Mike regularly writes articles and delivers conference papers on
testing and verification.