Software Reliability: what we believe #

This is part one of Project Cinnte (KIN-TEH), an effort to look at some foundational questions in software reliability.

Introduction #

Being professionally interested in questions of software reliability, we have often wondered whether or not we can find good models to help explain, characterise, or predict failure. The industry doesn’t have much in the way of such models today. What we have is practitioner’s intuition, and it’s often the best thing we do have, but it’s hard to know if it’s the best thing we could have.

In the act of reflecting on this question, we realised that there isn’t a good source that documents precisely what that intuition actually says. The rest of this document is an attempt to do that.

The standard doctrine #

Here is our take on the “standard doctrine” on bugs, outages, and how software development typically happens - happened? - in the pre-AI SWE era.

We label the following propositions for ease of reference, together with their commentary.

P0: Bugs happen as a function of the sustained intellectual effort underpinning writing software. The longer the effort or the more intense the effort required, the larger the chance of incorrect software being written.

Software engineers believe this because they model the act of writing software as having a problem to deal with, thinking of a way to solve that problem, and writing down the solution. The tougher the problem, the more chance you get it wrong; the larger the problem, the more chance you get it wrong.

P1: Other things being equal, the bigger or more complex the software, the more bugs there are.

P2: Other things being equal, the newer the software, the more bugs there are.

These look like simple corollaries of the above, though there is more complexity there when you tease it apart. When we say “bigger” above, that follows clearly from P0, but “more complex” also brings in scenarios where the problem might not require a large mass of software to solve, but might be algorithmically complex, have a very large number of business domain exceptions, have uncommon constraints (e.g. latency), or similar. P2 in particular introduces a new idea: newly made software has not encountered reality yet, and will have more bugs than software which has encountered reality.

P3: Bugs can occur for a variety of reasons, and on a number of different levels. The category of “bug” itself is quite wide, and includes at least typos, inaccurately writing what you want, accurately writing what you want but it turns out it’s not the right thing to want, incorrectly capturing your own requirements, incorrectly understanding other people’s requirements, working with third-party software or services with different models of the problem domain, compiler errors, microcode errors, firmware errors, or cosmic rays.

The current authors have had all of these happen to them at one stage or another. There is probably some kind of long-tail distribution of the frequency of occurrence, but the structure doesn’t matter for most of these observations.

A rather forceful description of the current experience of using GenAI at work. Prisoners are not taken.

There’s a lot that resonates with my narrow area of interest: the continuing necessity for human-in-the-loop (despite much pressure for that to be eliminated); senior-versus-junior productivity dichotomies; and the idea that “accomplishing the work itself used to teach the judgment” now going away means that there is no clear way to acquire judgement when you don’t have the feedback loop to acquire it. Which, I suppose, increases the pressure to remove the humans and use LLM judgement instead.

It’s hard to reconcile in the one model of the world both this piece and the like of the pieces that I scroll past every day, which are insanely enthused about the sheer possibilities of the tools and the huge rate at which they’re improving. Erdős problems, passable literary works, code (of course). And yet I don’t feel that either of those kinds of experience are wrong, or lies.

Came across this site the other day. Though the Changelog seems to suggest it was last updated in May 2016, which makes it a bit more likely to be slop, I think the overall message is not misplaced.

A great piece from James Shore on the maintenance cost of software, and some potential down-sides to AI SWE.

A great treatise on wifi. Ah, the good old days, etc.

A short while ago, an ex-colleague from an old team asked for some career advice. I did what I could (though I didn’t think it was very useful). He was kind enough to send me a card and a (very large) collection of chocolates in return. It was a great kindness that reminded me of a great team that was always under pressure.

As the tech industry is collectively doing a lot to erase every aspect of being human from working in it, and especially any question of the vulnerability that naturally attaches to it, it does some good to remind ourselves that there is virtue in being human.

A great article from the always authentic and enjoyable World’s Greatest Newsletter (ignore the name) from the Raw Signal Group.

“Businesspeople, we’ve entered a weird moment when caring about the organization and your craft is a liability. And when pressed for details on why caring less seems appealing, the answers are dark.”

I’ve been thinking for a while about how incident response is going to change, and how it has already changed since the pre-ML days. Todd Underwood did a great chapter in Reliable Machine Learning which tried to illustrate how IR changes in the modern world. In brief, it becomes harder to both investigate what’s going on, and also follow the standard troubleshooting approach of building a mental model in your head of what’s happened when you no longer have a causally strong relationship between actions and outcomes. It’s also going to involve a lot more coordination between different groups, as ML will typically pull in data from across the business to a previously unprecedented extent.

But I came across this today - thanks to Eric Dobbs in RISF - which talks about one likely feature of the future that hasn’t gotten much attention outside leading edge circles, and that’s the fact that as AI SRE systems hoover up the easier tasks, the harder tasks will be the only ones that are left: the “left behind” issue.

Most folks who look at this have pointed out that as the easier issues go away, it’s harder to train on what remains, and (modulo learning styles) I think that’s true; what I think is less explored is how IR changes when you actually can’t construct a model of how the system works by asking a sufficiently aware human. We will, in short, become dependent on the same tools that created the additional complexity to penetrate and resolve that complexity in real-time, every time there’s an incident.

We should bear that in mind when we think about how to staff, and what to pay for, in the domain of incident response. The stuff that’s left behind - the incident residue - is the stickiest of all.

The AI SRE space is, as of the time of writing, absolutely insane. At some point in 2025, I counted the number of players and the amount of money rushing into the space - it was 20+ and over a billion dollars, if you included all funding numbers I’d found plus the numbers of incumbents in e.g. Cloud talking about how much they were going to invest in the space. It may well turn out to be one of those situations where it’s easy to make a prima-facie argument that the problem space is big, almost everyone “suffers from it”, and that it’s easy to make progress (given the current state of agentic development, etc etc), but it’s quite hard to deliver something that actually makes a difference and more importantly that is not like everyone else’s three foundational models in a trenchcoat.

Earlier in my career there were very similar conversations about mobile phone providers (really operators), who quickly became seen as being essentially commodotised - everyone would pick from a similar set of network gear provided by a small set of manufacturers, the handsets were mostly commodotised etc, etc. Ultimately they did what a lot of businesses in similar positions did, which is to attempt to differentiate themselves on price, branding/marketing, or customer service. There may well be a similar effect playing out in this market too.

In unrelated events, I see that Komodor are organising an AI SRE summit and that looks like an interesting speaker list, though I wonder precisely how vendor neutral that’s going to be.

From college mate Ian’s time at a Cloudflare session, we learn that bot traffic is 50% of overall web traffic, and AI agent traffic is circa 7%.

It seems likely both of those numbers will go up.

The incomparable Ethan Ding on the disjunction that most of us are feeling right now. Claude speeds up certain things - quite a lot - but it also slows us down. We need to have a more accurate model of what’s happening to software, and this is an accessible primer on one possible scenario.

(I also loved his piece on levered beta, which I think played out essentially as he wrote.)

As an author, I strongly dislike O’ Reilly’s Early Release model, since my stuff gets poked at before it’s ready. As a reader, I strongly like O’ Reilly’s Early Release model, since I can poke at other people’s stuff before it’s ready!

O’ Reilly’s Safari platform is hosting the latest chapters on STPA and AI for SRE.

I strongly suspect that Zvi doesn’t need more inbound links, but his latest model card assessment (which is, as usual, very well written) has a couple of notable quotes:

So yeah, none of that sounds great. It all sounds like the types of thing that, if you caught a human doing them even once, that would be a very bad sign, and in several cases you would obviously have to fire them.

Check out the examples of Mythos Preview attempting (and in some cases succeeding, only to be caught by the human at the last moment) to escape containment.