Open source and certified systems

Carlo Daffara has sent through this interesting post that looks at the myth that Open Source software isn’t appropriate for building systems that require certification…Thanks Carlo!

The popular idea that open source, as a collaboratively developed system, does not have the intrinsic quality or reliability to be part of life critical system is quite common. As an example a recent white paper, published by the Election Technology Council (an industry trade association representing providers for over 90% of the voting systems used in the United States), analyses the potential role of open source software in voting systems, and claims that the inherent process that creates OSS is unable to meet the quality standard necessary for a system that must meet strict certifications. The white paper does contains a critical flaw, the result of mixing the “legal” perspective of OSS (the license on which the software is released) with the “technical” aspects (the collaborative development model), arriving at some false conclusions that are unfortunately shared by many others. For this reason, I would like to add my perspective on the issue of “certified” source code and OSS:

First of all, there is no causal relation between the license aspect and the quality of the code or its certifiability. Licensing is orthogonal to development model, and most projects use the gated barrier model to check every contribution from the outside (an example is the Eclipse development process).

It can be observed that open source development models tend to have an higher improvement in quality within a specific time frame when compared to proprietary software systems under specific circumstances (like a healthy contributor community). Succi (Succi, Paulson, Eberlein. An Empirical Study of Open-Source and Closed-Source Software Products, IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, V.30/4, april 2004 ) for example found that “The hypothesis that open-source software fosters more creativity is supported by our analysis. The growing rate, or the number of functions added, was greater in the open-source projects than in the closed-source projects. This indicates that the open-source approach may be able to provide more features over time than by using the closed-source approach. Practitioners interested in capturing market share by providing additional features should look to the open-source methodology as a method to achieve this. In terms of defects, our analysis finds that the changing rate or the functions modified as a percentage of the total functions is higher in open-source projects than in closed- source projects. This supports the hypothesis that defects may be found and fixed more quickly in open-source projects than in closed-source projects and may be an added benefit for using the open-source development model.” This is consistent with results from vendors of software defect identification tools, such as Reasoning, that found that while the bug density ratio in initial project releases is on par with proprietary developments, it improves rapidly and for some projects defect densities are significantly lower than that of the average proprietary code; this was confirmed by other studies like the reports from Coverity.

It is possible to certify open source systems under the strictest certification rules, like the SABI “secret and below” certification, medical CCHIT, encryption FIPS standard, common criteria Evaluation Assurance Level EAL4+ (and in one case, meet or exceed EAL5), civil engineering (where the product is used for the stability computations for EDF nuclear plants designs), avionics and ground-based high-integrity systems, like air traffic control and railrway systems. The UK Health and Safety Executive, in a study from 2002 found that Linux was robust enough and that it could be certified up to SIL3 with limited effort. This would make it amenable for use in air traffic control displays, railways control systems and process plant control.

So, it is clear that OSS can effectively be as reliable as software developed under traditional proprietary proecesses. I have a personal opinion on why this happens, and is really related to two different phenomenons:the first aspect is related to code reuse: the general modularity and great reuse of components is in fact helping developers, because instead of recoding something (introducing new bugs) the reuse of an already debugged component reduces the overall defect density. This aspect was found in other research groups focusing on reuse; for example in a work by Mohagheghi, Conradi, Killi and Schwarz called “An Empirical Study of Software Reuse vs. Defect-Density and Stability” (available here) we can find that reuse introduces a similar degree of improvement in the bug density and the trouble report numbers of code than that measured in open source projects: As it can be observed from the graph, code originated from reuse has a significant higher quality compared to traditional code, and the gap between the two grows with the size (as expected from basic probabilistic models of defect generation and discovery). The second aspect is that the fact that bug data is public allows a “prioritization” and a better coordination of developers on triaging and in general fixing things. This explains why this faster improvement appears not only in code that is reused, but in newly generated code as well; the sum of the two effects explains the incredible difference in quality (50-150 times), higher than any previous effort like formal methods, automated code generation and so on. And this quality differential can only grow with time, leading to a long-term push for proprietary vendor to include more and more open source code inside of their own products to reduce the growing effort of bug isolation and fixing.