Bureaucracy, Safety and Software: a Potentially Lethal Cocktail

This position paper identifies a potential problem with the evolution of software controlled safety critical systems. It observes that the rapid growth of bureaucracy in society quickly spills over into rules for behaviour. Whether the need for the rules comes first or there is simple anticipation of the need for a rule by a bureaucrat is unclear in many cases. Many such rules lead to draconian restrictions and often make the existing situation worse due to the presence of unintended consequences as will be shown with a number of examples. In science and engineering, the effects of such bureaucracy are generally mitigated because the rules naturally devolve from the exercise of the scientific method whereby evidence leads to policy and lasting benefit. In the absence of the scientific method (which is usually the case in software systems development), policy flourishes like weeds without the constraints of reality. In software con-trolled systems, any consequent unintended side-effects could be lethal.