Dave’s Archives

In my persuit of software engineering data, I've recently been poring over a 2002 report to the US Government on the annual costs of software  defects. The report is entitled "The Economic Impacts of Inadequate Infrastructure for Software Testing". Ultimately, it estimates that software defects cost the US economy $59.5 billion every year.

Modelling such economic impacts is an incredibly complex task, and I haven't read most of the report's 309 pages (because much of it isn't immediately relevant to my work). However, since trying to use some of the report's data for my own purposes, certain things have been bothering me.

For instance, the following (taken from the report):

This table summarises the consequences to users of software defects (where "users" are companies in the automotive and aerospace industries).

Strictly speaking, it shouldn't even be a table. The right-most column serves no purpose, and what remains is a collection of disparate pieces of information. There is nothing inherently "tabular" about the data being presented. Admittedly, for someone skimming through the document, the data is much easier to spot in a table form than as plain text.

The last number piqued my curiosity, and my frustration (since I need to use it). What kind of person considers a $4 million loss to be the result of a "minor" error? This seems to be well in excess of the cost of a "major" error. If we multiply it by the average number of minor errors for each company (70.2) we arrive at the ludicrous figure of $282 million. For minor errors. Per company. Each year.

If the $4 million figure is really the total cost of minor errors - which would place it more within the bounds of plausibility - why does it say "Costs per bug"?

The report includes a similar table for the financial services sector. There, the cost per minor error is apparently a mere $3,292.90, less than a thousandth of that in the automotive and aerospace industries. However, there the cost of major errors is similarly much lower, and still fails to exceed the cost of minor errors. Apparently.

What's more, the report seems to be very casual about its use of the words "bug" and "error", and uses them interchangeably (as you can see in the above table). The term "bug" is roughly equivalent to "defect". "Error" has a somewhat different meaning in software testing. Different definitions for these terms abound, but the report provides no definitions of its own (that I've found, anyway). This may be a moot point, because none of these terms accurately describe what the numbers are actually referring to - "failures".

A failure is the event in which the software does something it isn't supposed to do, or fails to do something it should. A defect, bug or fault is generally the underlying imperfection in the software that causes a failure. The distinction is important, because a single defect can result in an ongoing sequence of failures. The cost of a defect is the cost of all failures attributable to that defect, put together, as well as any costs associated with finding and removing it.

The casual use of the terms "bug" and "error" extends to the survey instrument - the questionnaire through which data was obtained - and this is where the real trouble lies. Here, potential respondants are asked about bugs, errors and failures with no suggestion of any difference in the meanings of those terms. It is not clear what interpretation a respondant would have taken. Failures are more visible than defects, but if you use a piece of buggy software for long enough, you will take note of the defects so that you can avoid them.

I'm not sure what effect this has on the final estimate given by the report, and I'm not suggesting that the $59.5 billion figure is substantially inaccurate. However, it worries me that such a comprehensive report on software testing is not more rigorous in its terminology and more careful in its data collection.

An "official communication" from early June demanded that all Engineering and Computing postgraduate students take part in the Curtin Engineering & Computing Research Colloquium. Those who didn't might be placed on "conditional status", the message warned.

A slightly rebellious instinct led me to think of ways to obey the letter but not the spirit of this new requirement. Particularly, the fact that previous colloquiums have been published online introduced some interesting possibilities:

• a randomly-generated talk;
• a discussion of some inventively embarrassing new kind of pseudo-science/quackery; or
• the recitation of a poem.

In the end I yielded, and on the day (August 25) I gave a reasonably serious and possibly even somewhat comprehensible talk on a controlled experiment I'd conducted on defect detection in software inspections.

A while afterwards, I received in the mail a certificate of participation, certifying that I had indeed given the talk I had given. It felt a little awkward. Giving a 15 minute talk isn't something I'd have thought deserving of a certificate. It might be useful for proving that I've done it, since it now appears to be a course requirement, but a simple note would have sufficed.

Interestingly, I later received another certificate, identical except that my thesis title had been substituted for the actual title of my talk. In essence, I now have a piece of paper, signed personally by the Dean of Engineering, certifying that I've given a talk that never happened.

The story of the end of Christian Rossiter has been in the news recently, and serves as another hook into the euthanasia debate. Euthanasia is one of those controversial subjects where the politics seems stubbornly opposed to what people generally regard as sensible.

I'm not unreservedly committed to the right to die. I consider myself a humanist, and as such I regard human life as being as close to sacred as anything can possibly be. However, on balance, in situations where there is no hope and where appropriate couselling is provided and informed consent given, the arguments against the right to die seem rather unconvincing.

One thing that does bother me, in this particular situation, is the following quote from Christian's lawyer (given in the ABC article above): "Death I suspect comes as quite a relief for Christian."

Those are rather poorly chosen words. For Christian, death cannot possibly provide relief, or indeed any emotion or physical sensation (unless there's an afterlife*). Death is the option chosen when relief is unattainable. Relief may be felt by those close to the individual, on account of the end of the suffering, but that's not quite the same thing.

This is not a happy ending, but merely an ending that could have been worse.

* This ought to be a somewhat redundant qualification. Clearly anything could happen if we suppose the existence of some hitherto unobserved and inexplicable magic.

I have a hypothesis on politics - a somewhat unfortunate hypothesis given its implications. Roughly speaking, it's this: the workability of democracy diminishes with large populations. I'm not talking about the logistics of holding elections, but about the ability of society to engage in meaningful debate.

My reasoning goes like this. Insofar as I can tell, in any given (relatively democratic) country, the media tends to focus predominantly on the national politics of that country. At the same time, there are of course a variety of political parties and interest groups seeking to alter public perception for their own ends. We can think of this in two parts:

1. the effort expended on politically-charged adverts, campaigns, editorials, etc.; and
2. the resulting effects on the public mindset.

Due to mass media (TV, radio and the Internet), a fixed amount of "effort" will probably yield the same result, independent of the population size. That is, the effectiveness of a single TV ad will not diminish simply because more people are viewing it.

However, countries with larger populations will naturally have a higher talent pool from which to draw people to promote particular causes. Thus, more effort will be expended on political advertising, campaigns, editorials, etc., and so the effect on the public mindset will be greater. (I also assume that the proportion of people employed to promote particular causes is independent of population size.)

Now, we might naïvely assume that all this political advertising "balances out", since there's always an array of competing interests. I say this is naïve, because all efforts to promote political causes have one thing in common - one thing that can't easily be balanced out: deception. I'm not only talking about outright lies (though it does come to that with tedious regularity), but also errors of omission, logical fallacies, appeals to emotion and any other psychological tricks used to blunt your critical thinking. They're not even necessarily deliberate.

Without wanting to generalise, there are certainly a subset of PR people, political strategists and so on who do seem to hold an "ends justifies the means" view. These are the people who really feed the political machine, who take things out of context, invent strawmen, engage in character assassination, and generally pollute the political debate with outrageous propaganda. The larger the population, the more of these people there will be, and so the louder, better organised, more pervasive and more inventive the disinformation.

The effect of disinformation is to disconnect public perception from reality. At at sufficient level this would cripple democracy, because democracy relies on the people having at least some understanding of government policy and its consequences.

I can't comment too much on India - the world's largest democracy - because I honestly know very little about it.

I don't claim much expertise on American politics either, but I suspect the US is suffering this affliction. To me, American politics now seems to languish in a state of heated anachronism. The political machine instantly suffocates any sign of meaningful debate with ignorant fear and rage. You're still perfectly able to exercise your rights to free speech and free expression, but it's not going to achieve anything. Meanwhile, in a desperate attempt to climb above the fray, the media sometimes treats political debate more like a sporting match than a tool of democracy. I'm sure there is an element of this in every democratic country, but in the US it seems to be boiling over.

It might pay to consider this if we intend to move towards a World Federation, as science fiction often proposes, and which appeals to me intuitively. Of course, a "One World Government" is the nightmare-fantasy shared by so many conspiracy theorists. However, the danger is not that the government will have too much control, but that even with our rights fully protected, democracy will nevertheless be pummelled to oblivion by global armies of political strategists and PR hacks.

Just a thought.