Infinite Undo

Software As Narrative How-To Articles Devops Reading List CC Sharealike © 2025 by Noah Sussman

[infiniteundo]

How to see the connections between Testing, Observability And Devops (TOAD 🐸)

I think that if the mental model one is using is like the New View on Systems Safety, then the connections between the disciplines in TOAD (Testing, Observability And Devops 🐸) naturally become obvious as you work more and more with software systems.

Conversely for people rooted in the traditional / Cartesian view of software systems, it’s pretty much impossible to explain. (edited)

For instance

If I think that the history of a software is composed of discrete, finite frames of time that can be reconstructed after the fact then good fucking luck explaining to me why I need an observability framework that lets me respond quickly to unknowns.

Because if I think history is composed of discrete, comprehensible frames that can be reconstructed after the fact, then I have a lot to learn from the system out of that history — your observability and prediction techniques are nice but not necessary.

However

If I don’t accept that history is composed of discrete comprehensible events, then I have very little to learn from history and then the o11y framework and the culture of resiliency makes sense This is btw what is obliquely being gotten at in the Clay Shirkey quote:

Process is an embedded reaction to prior stupidity.

It means people with a lot of process are people with the old / Cartesian view of system safety. They have a lot of process because they think they can predict what sort of events will be important in the future.

Since formal process is reactive, it follows that in order to have a lot of formal process, we have to have a lot of events we can predict with great specificity.

People who operate mainly off of prediction just don’t think that way. We don’t come up with taxonomies of past failures and try to derive ontologies that address them.

[infiniteundo]

Feedback Engagement

Feedback engagement is a metric that describes how often and where developers engage with feedback from the CI system. Engagement rates can be calculated using all of the standard engagement measurement tools from production-facing systems: email open rate, the frequency with which developers respond to slack announcements, and most obviously: the rate at which failing tests are fixed or ignored.

[infiniteundo]

Phenotypic Conformance Analysis

Phenotypic conformance analysis describes the practice of establishing a static analysis ruleset based on the observed properties of the codebase rather than on an existing open source standard or a ruleset voted upon by the team.

Then the initial feedback loop revolves around How much new changesets resemble the code that is already in the codebase. This gets at the important properties that static analysis speaks to: consistency and the absence of syntax errors — without resorting to a normative standard that risks a bad fit with existing practices and can lead to bikeshedding.

[infiniteundo]

How do you know what you know?

Testing is just applied epistemology. — Brett Pettichord

[infiniteundo]

stuff you can test

A diagram.

[infiniteundo]

How Change Works In Large Organizations

The Kübler-Ross change curve and the Six Phases Of A Project are two time-tested ways of visualizing how organizations cope with change!

Here the Kübler-Ross curve and the Six Phases are together for the first time!

I hope this infographic helps to achieve every initiative in your portfolio!

[infiniteundo]

Software Engineering As Hypothesis Invalidation

If testing software and writing code feel very different to you, it’s only because you haven’t written enough code yet. That is only my own admittedly controversial opinion. I believe that everything we call “software testing” is a subset of the activity we call “programming.”

Implementation is a test of a hypothesis. To implement a pattern in code one must first form a narrative or if you prefer a hypothesis. Implementation is itself a test of whether the narrative holds up.

Corollary: Consider that the full specification for a program and the program itself are the same thing. This implies you can’t design computer programs by up-front, complete specification. You are constrained by “the laws of nature” to begin with an incomplete hypothesis and proceed by testing the implementation of said hypothesis, using the results of that test to decide how to go about modifying either the hypothesis or the implementation or both, then repeating that process.

At all levels of the stack, it is always the case that complete specification is functionally impossible since such a thing could only exist in the form of a complete implementation. The largest Web application and the derpiest hello world program both have this quality: that they cannot be implemented by complete specification but must be built iteratively via (in)validation of hypotheses.

For a much more thorough exploration of this idea you can read or refer back to Programming As Theory Building by Peter Naur (1985).

[infiniteundo]

Thanks everyone for upvoting my post on HN!

It's always exciting to see my work on the front page of Hacker News :)

Full list of Software As Narrative posts.

[infiniteundo]

Software Rot

Software rot

The software development life cycle is predictable in that any long-lived product will eventually outgrow some of its subsystems. For instance a Web service that begins life with a single monolithic database server will as it scales need the capacity increase that comes from a distributed database. A historical example can be found in Twitter’s original dependence on the ActiveRecord ORM, which over time was replaced with a variety of databases and services.

For historical reasons

In one sense this might be considered as a canonical definition of legacy systems: the system contains subsystems that are not optimally suited to day-to-day functioning despite the fact that at some point in the past those same subsystems did function optimally. There is a concept of Software Rot or Bit Rot that metaphorically encapsulates the life cycle phases that precede this sort of legacy system.

Frictionless yet it still wears out

The observed course of the software development life cycle is that features begin life in a “working” state — meaning that they satisfy the requirements agreed upon by an empowered group of stakeholders — but that inevitably the same features begin to exhibit bugs that are not in any way related to changes of code nor hosting environment. Software rot (as this phenomenon has come to be called) is caused because the requirements for features continue to change and evolve even after such features are in the hands of their users.

the system contains subsystems that are not optimally suited to day-to-day functioning

This is not a well-understood area of software production, nor does Computer Science have much to contribute by way of solutions. The problems are not algorithmic but environmental, social, aesthetic — in other words it is what programmers like to call a squishy problem because so much of the problem space is taken up not by software but by humans and their co-collaborators.

In programming, soft skill is hard

The idea of engaging with squishy problems is uncomfortable to a lot of programmers. I think this is because programmers currently do not have an opportunity to learn the heuristics that would allow them to distinguish good solutions from bad when it comes to human-and-social issues.

Taking software rot as an example, it is a well-known phenomenon long documented in the literature of software. Yet it has no commonly-agreed up on solution. Its management is not a topic of discussion in job interviews nor in performance evaluations (for the most part). The contraventions for software rot are not listed in general programming books nor taught in coding boot camps.

I do not believe that software rot is ignored as a topic because no one recognizes its importance. I’m pretty sure it’s ignored because no one feels comfortable giving advice about it, because almost no one has successfully dealt with the long-term requirements-changes and subsystem upgrades that go with solving software rot.

The knowledge about how this problem have been successfully dealt with are locked away in a couple of books. And those books are old, using server-side programming and Java as their example environment. That’s a hard sell to a junior engineer fresh out of a boot camp or undergrad program.

The recent movement toward systems thinking in software is a hopeful sign. But we need modern discourse that concerns how to deal with changing requirements over time. And so far such discourse hasn’t been forthcoming despite all the growth and hype about code over the last ten years.

[infiniteundo]

Suboptimization is THE reason for technical debt. But I have to go outside of programming world to find discussion of it.

Suboptimization is THE reason for technical debt. But I have to go outside #programming to find discussion of it. https://t.co/cgTN57MrT4 pic.twitter.com/SKjYKUVV4O

— Tentacular Devops 🐙 (@noahsussman) June 18, 2017

[infiniteundo]

The uncomfortable truth is that dev is a complex relationship with an increasingly intelligent other.

The uncomfortable truth is that dev is a complex relationship with an increasingly intelligent other.

The uncomfortable truth is that dev is a complex relationship with an increasingly intelligent other.

— Tentacular Devops 🐙 (@noahsussman) June 28, 2015

[infiniteundo]

--- An illustration of the "funnel" for FOSS developer engagement.

[infiniteundo]

Miller is like jq for CSV and other tabular data

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON.

Here's how I use Miller to pipe CSV data into jq

jq is currently my tool of choice when it comes to processing all sorts of data. Except XML and CSV. XML is pretty well handled by xmlstarlet but I have never found a CSV parsing tool that I liked. I just make do with using jq to work with CSV and to be honest I find the results I can achieve to be substandard.

Anyway, Miller solves all that and is easy to install even if you don't use homebrew and need to compile it from source.

mlr --c2j cat my_file.csv | jq .

It is that easy! Now my CSV is structured as JSON, which I have spent a lot of time learning to enjoy working with in jq.

[infiniteundo]

This is what concerns me about the "testing vs. checking" meme in software quality assurance circles today.

"Testing vs. Checking"

"Testing vs. Checking"

You will find this meme under discussion at all of the leading software quality assurance (SQA) conferences today including GTAC, STPCon, STAR, CAST and the various Selenium confs.

In short the discussion revolves around whether a Cartesian divide can be effected between the human and the computer's role in software quality assurance.

"Testing vs. Checking"

It is not the discussion itself that bothers me. Although Cartesian division is out-of-fashion with Devops Luminaries(tm) such as myself, I can certainly see the utility of a Cartesian point of view when establishing certain foundational local taxa --- as we Test Architects often must do!!!

Concern trolling

What jerks my fucking chain though is the god damned concern trolling.

The biggest problem by far is all the fucking language policing that goes on in the testing and in the agile community. Language policing is microagressions. Stop it and apologize.

The most common form of Language Policing (aka Tone Policing) that I've seen take place around "testing vs. checking" is "correcting" people who choose to identify as automators of test activities.

I've personally been interrupted while speaking at a conference at least three times only to have the interruptor explain that "checking isn't testing." Yeah motherfucker. I already knew that and I don't give a shit about it becasue from my perspective that's minutia that everyone takes as written in the first place.

Interrupting other professionals' to corret trivial language choices is uncool at best. Viewed in a dispassionate light, such behavior is elitist, immature and alienating. Most people won't put up with that, either because it makes them feel bad or because it makes them feel like they've wasted their time. Yet the behavior persists for some reason.

[infiniteundo]

A Modern Vision of Testing as Sympoietic Development Praxis

[infiniteundo]

Software testing considered as a series of band-pass filters

These are a pair of slides from a talk I am currently developing.

[infiniteundo]

Teleonomy and the Quality Without A Name (QWAN)

Good systems converge on quality. They snap back into shape when deformed. Autoscaling is an example of how Web systems regain their good state after some event has impacted functionality.

Bad systems act the same. when pushed toward a better state they gradually edge back toward the bad state. Take for example the familiar case of a legacy system that has been put under test but gradually edges back toward untestedness again.

Teleonomy confirms what we intuitively already know. That some systems are easy to collaborate with and that other systems seem adversarial in our interactions with them.

The principle of Teleonomy shows that intent is not required in order for a system to pursue a goal. Neither collaboration nor antagonism requires intent.

Alexander is very clear in defining QWAN: Good systems pursue the goal of inner consistency. Bad systems pursue a state of internal conflict. when humans pursue a goal of internal consistency then we are pursuing the same goal as a well-functioning complex system. That the goals are aligned is sufficient to say that human and computer are collaborating. Intent is not required: the observed state of alignment qualifies fully as collaboration.

The "snapping back into shape" of complex systems is a practical example of system memory, one of the defining characteristics of a complex System as opposed to a simple one. Memory is also a characteristic of cellular automata such as the Abelian sandpile model.

So computers can be seen (without resorting to metaphor) as collaborating agents that not only have goals for the future but also remember the past. This is the observed reality of working on the computer system: it is difficult not to eventually regard computer systems as having motivations and goals which do not always align with our own. Teleonomy and sympoiesis are lenses through which this observed reality makes sense.

Dijkstra once said that the metaphor of "the bug that crept in while the programmer wasn't looking" is intellectually dishonest; because any bug in the system must have originally been put there by a programmer. But Teleonomy implies sympoiesis between human and program and that in turn calls Dijkstra's statement into question.

Bugs (and features and everything in between) do NOT originate with a human agent. Instead the characteristics of the system are defined in the collaborative interaction BETWEEN human agents and computer agents. Bugs may creep in while we are not looking because our systems DO have agency, according to the principle of Teleonomy.