↓Skip to main content
  1. Posts/

Niall's naive list of unsolved problems in software reliability

·3 mins

This post is a part of Project Cinnte (KIN-TEH), an effort to look at some foundational questions in software reliability.

I participate in the Resilience in Software Foundation, the Slack for which contains a large number of people interested in questions that also obsess me - software, systems thinking, safety science, and so on.

A recent thread about metrics provoked this response from me, which one of the thread participants thought was helpful, so I reproduce it here. This is basically my Hilbert problem list, but I think it’s more appropriate to call it “Niall’s naive unsolved problems of the field” list:

  1. We have a lot of qualitative models and not many (perhaps no?) quantitative models. Is this necessarily true?
  2. We have a lot of impossibility theorems and tradeoff models, but not many constructive models. This is in the sense of “you can achieve X if you build it in way Y”) models. Is this a necessary outcome? What’s stopping us getting more positive constructive models?
  3. We have no real theory of software failure, other than Rice’s theorem. Can we do better?

We have well-founded intuitions about software failures, and a ton of experience about how they happen, but we can neither show they are infinite nor finite, not characterise their behaviour. At the other end of the spectrum, evidence-based software engineering papers are fairly hopeless when it comes to long-lasting fundamental results, and we basically have Brook’s Law and Conway’s Law.

I accept that socio-technical observations about wider organisational behaviour are of course not going to have six-sigma output, will suffer from reproducibility problems, et cetera. I know all that. I should be clear that in many cases we have valuable observations and occasionally even working models that can inform behaviour (as above, e.g. “do not add more humans to an already late software project”, “you should think about how your org structure is going to get reflected in your software, because it will” and so on). But the occasions, if I can put it that way, are few and far between, and I see, particularly in the LFI end of things, a lot of epistemic objections entirely at odds with management frameworks, both of them unfortunately rooted in the reality of their lived experience.

The current state of play is that a practitioner working in software - a sharp-end person speaking to a blunt-end person in the learning from incidents/software reliability context cannot explain:

  • On what basis we should believe incidents will happen
  • Predict what might happen with an individual future incident, or indeed any aggregate metric of all future incidents
  • Articulate suitable replacement metrics for understanding behaviour or improvement in the space
  • Understand or explain what is likely to happen to organisations turning to AI-inflected software development

Am I wrong about any of the above? Where am I missing nuances?

When I emphasise “necessary” above, what I mean is that “I don’t know if all of the above is a necessary result of the space, i.e. we have a bunch of limits theorems mostly because it’s impossible to do anything better, or is it merely an accidental result of us not looking in the right direction with the right tools”.

Or in other words, can we do something to make the situation better?