True in software, true in life. @tistil - Tumblr Blog

Composition over Inheritance, emotionally:

I have feelings, not I _am_ my feelings.

I can notice them, decide how to use them.

Kubernetes is like a dishwasher

I’m capable of washing dishes by hand. But I don’t enjoy doing that every time I eat. And I don’t appreciate when people get sloppy and there’s still food on the plates.

The dishwasher is a standard place to put the dirty dishes. Washing happens. I have to load it and unload it, but the washing is reliable, consistently thorough, with hotter water than I could use by hand and no dirty washcloths.

Kubernetes is like a dishwasher. Yes, I could run all my apps by hand, but I wouldn’t do as good a job, I can’t prescribe to other people how to do it with the same precision, and I might re-use infrastructure (dirty washcloths).

Kubernetes, once loaded, keeps the apps running until I unload them. It watches over them, so I don’t have to think about them every day.

It does suck when the dishwasher breaks down and I don’t know how to fix it. Glad there are companies who will do that for me.

Kubernetes is better than a dishwasher because it has an API; I can program computers to load and unload it, triggered on events like a successful build. I’m still working to program my kids to put their dishes in the dishwasher upon successful meal completion.

learning to learn

Francesco Cesarini remarked the other day that most of learning programming is learning how to learn.

As an adult, I do this consciously now. I’m learning piano, and when a teacher tells me: practice only that part that I struggle with. Practice it slowly. One hand at a time. When you get it right, keep going: practice it correctly five times before moving on.

I do this, and it helps. I learn the song, and I learn something about learning.

The other day, practicing a 10-minute talk, I stopped after the part I struggled with it. I practiced that one part, repeatedly, not until I got it right but until I got it right three times in a row. It helped.

So now you have a trick for learning something. The larger point is: look for you own learning tricks, and apply them more broadly, and get better at learning everything. This is the next level of humanity, this is the modern age, this is software.

shared intentions: leveling up at helping

Today a friend installed a sink in my basement. I know nothing about plumbing, so I hung around to watch. When it was clear I could help him with some action, I did: holding a pipe steady while he cut, sanding the second end after watching him do the first. It was tempting to pick up the installation instructions and try to contribute ideas, but I knew better.

This works with pairing on code, too: when someone else has all the context, I’m not gonna make them explain to me top-down what we’re doing, not at a level detailed enough that I could contribute. If I can participate in the action they’re currently taking, then I can help. As I gain context, I can contribute at a higher level: suggesting a different path when one gets hard, recognizing potential problems as we embed them. If I try to contribute at too high a level, then I force my partner to justify decisions they’ve made. Even if I’m right (unlikely), they don’t experience that as help.

Next time I join a new team, I’m going to remember this. I don’t have to have all the context to help with actions in progress. As I gain context, I can help at a higher level.

The ability to shared intentions is what makes us human. (from The Righteous Mind, by Haidt) This applies at many levels, from a mission statement down to a task. (seamaps illustrate this.) I can help at the levels that I understand, and gain understanding in the process.

a thousand small problems

There’s an anecdote in which Gorbachev (or Yeltsin) visits New York (or London, or Austin) in the waning years of the USSR. He is astonished by a grocery store, with its fully stocked shelves and no lines. He asks, “Who is in charge of the supply of bread to New York?”

because it is ingrained in him that serious work is best done by a single planning mind. Yet the evidence said otherwise: New York has plenty of bread, and Moscow did not.

No one is in charge of the supply of bread to New York. A thousand people are in charge of the supply of bread to one store. A thousand small problems, each solved in their own way, are tractable; one big problem, solved “efficiently,” is not.

In an enterprise, it sometimes feels efficient to put a person in charge of the supply of servers, or testing, or purchasing. Yet by lumping these small problems together by category, you’ve created a big problem that can never be solved well for everyone.

Teams need different servers, at different paces, so gathering guesses and then planning and then allocating is always going to be too slow and never accurate. Letting each team get their own is a small pain for each team, but a big pain for no one. Even better: offer a general solution to solve 80% of the problem (fill out this web form to get a company-standard cloud VM within five minutes), and then let teams handle their special needs themselves.

The stores in New York mostly buy bread from a few distributors, but they can bake their own bread if they choose.

When developing infrastructure and tooling for an enterprise, I don’t want to be the central planner, very efficient at slowing everyone down. I want to be a distributor. The internal market is large enough and specific enough to target. When I do my job well, many teams will use my products. A few will still bake their own, and I’m not in their way.

Why does it work?

"Why” is too vague a question. Read my post on medium for the true-in-life half of this post.

John Allspaw and the SNAFU catchers are studying incident retrospectives. People ask, “Why did this fail?” and you can learn a lot from that. In this talk my favorite bit is: ask “What made this not be worse?” Focus on the relative successes so that you can sustain those controls/tools/people in the future.

The useful question in a system “why” is “How does it stay true?”

We can look at a system in that perspective, we can ask, “it’s working. how does it stay working?” and the answer is, people. The people above the line are monitoring and adjusting as the external environment fluctuates and changes.

We can look at each part of the system and learn something.

“The database is up. how does it stay up?” and one answer is, “there’s a controller that starts new replicas when needed, and shuts down ones that aren’t responding.” In another case, a controller is watching and switches to failover when necessary, and another piece keeps the failover database in sync with the live one. Software below the line automates controls that people used to spend energy on.

Ask this question about various parts of your system, and you’ll learn about it. which will be useful in your next incident.

P.S.

Mike Nygard made a post along these lines a few days before I did. When the world is ready for an idea, it doesn’t come to just one person ;-)

smaller steps are the best steps

Opening a bag of lozenges: I want to rip the bag, it doesn’t tear. Maybe with my teeth? I want to open it in one step.

Slow down, Jess, humans are tool users. Get the scissors. Cut the bag open. Return the scissors.

It’s more steps, what fun is that? but each step is easier than the hardest step in the original plan. It’s the hardest step that is limiting, not the quantity.

I think about this in code, too: three pieces of complexity 2 are, in aggregate, less complex than one piece of complexity 4. It isn’t the sum, it’s the maximum, that limits what we can understand.

In this post, Kent Beck applies this to refactoring. “There is a tradeoff between the cost of adding steps to a workflow and the expected cost of mistakes.”

Adding steps feels more expensive than it is. Slow is smooth, smooth is fast, even for household tasks.

nope, can’t do everything

The other day in the kitchen, I was disgusted by the knife left in the sink. Knives are dangerous. Wash them, dry them, put them directly back into the knife block.

It’s just one thing in the kitchen. It’s one that I’ve picked to be important. Meanwhile I left bits of food on the floor, kicked them under the edge of the counter. Someone else would find that disgusting.

i can let go of my fury about the knives now. And laundry baskets on the couch (those things sit on the basement floor, ew). And ripped-open envelopes, those are so hideous I must throw them away immediately.

These are only my pet peeves. The other adults in my household are not deficient people for prioritizing something else instead. There are infinitely many useful things to do or clean. If they do choose to put the knives away or trash the envelopes for my sake, it is a gift.

It’s the same in code. I have my pet peeves. How important is it that the tests hit live services instead of faked ones? Yes it will be faster and better when we put in the work to change that, but there are tons of other things to work on too. It’s a balance. Sometimes we do a thing because it makes the other people on our team more relaxed. Christian doesn’t like abbreviations so I’ve started writing out “repository” more. These are gifts we can give each other to smooth our teamwork.

can’t spend all day on self care

My dentist tells me that taking great care of my teeth is essential for health. My chiropractor thinks my spine is the #1 thing I should keep in order. (Well, my former chiropractor; I didn't revisit that one after the guilt trip.) A personal trainer will say exercise is a top determiner, and a dietician that food is first.

They're all right; all these are important. But I can't do everything! I have work to do and a life to live!

It's a balance. At each moment one of these things matters most; if my back is wonked then chiropractic care will help. If my energy is super low, am I low on iron or need more exercise? Life isn't about doing everything right; it's about getting the right things a little more right. And "the right things" are extremely contextual.

In software, people exhort developers to be responsible. Write tests first. Make good abstractions. Create a universal language. Experiment and validate. Retrospect and follow up. Refactor. Read everything in Slack, and all the commits, and emails. Get the user docs beautiful. Add null checks and error messages. Unit test and property test and integration test and simulation test. Learn the business and the code and the tech. Alphabetize your imports.

If I wrote code Correctly, it'd be useless by the time I shipped it. (qualifier: I'm talking about business software that's unproven, not serious libraries.) Oh yeah, and ship constantly.

It's contradictory! So you know for sure it's contextual, that some of these rules don't matter in your situation.

There's several ways that my team is doing it "wrong." There are several dirty things in the pull request I'm about to submit. The question is, is it good enough? and which improvement will make a difference in this context?

We tend to improve the things that stand out to us, which may or may not matter. Code smells, we know how to spot those. But what's the leverage point, what's the constraint in the system? Maybe it is a few missing tests. More likely it isn't software at all, but communication among team members. Or my understanding of the business.

The magic solution for one team is a waste of time for the next. Consider your context carefully, and don't let people guilt you about the tasks you consciously chose not to do right now.

don’t make me think about style

“A good style guide is one of the keys to writing more and faster. Avoiding style decisions alone offers the same sort of speed boost as word processing.” - the Responsible Communication Guide

Programming is fractal decisionmaking. There’s a little bit of typing to express the results of our decisionmaking, and yet there’s a ton of decisionmaking even in that typing. what file do we put it in? what do we call the function? If we are also thinking about how to format the code, that’s yet more decisionmaking that we could have spent on important business decisions like which errors to handle and how.

Style guides are important. Even better is when they’re automated, so that I hit save and poof my code conforms to the style. Then I don’t have to memorize the style guide. This is particularly important when switching between languages or teams that have different style guides.

Automated stylers like elm-format and go-fmt remove obstacles from adding this language to your repertoire.

Clutter

This comic describes life in a typical heterosexual household: https://english.emmaclit.com/2017/05/20/you-shouldve-asked/

Consider the part about clearing the table. The protagonist goes to put something away, sees laundry needs moved, moves it; sees food to put away, puts it; sees mustard is missing, notes it. As she moves about the home, she’s doing stuff that needs doing, moving what needs moving, noting what needs done later. It’s 2 hours before she’s cleaned the table, but the house is in an overall better state.

Her partner clears the table in 10 minutes, but does nothing else to improve the house. He probably stacked shit somewhere else.

Which of them looks more productive?

If we evaluate our performance based on the time it takes us to do the named task at hand, we look slow unless we skip everything else we see.

If we’re evaluated at how smoothly the household operates -- or how smoothly our development team’s work goes long-term -- this looks very different. My team has mustard, and clean laundry, and the food is in the fridge. We aren’t gonna be in the “oh crap I have no clean shirts and no sandwiches” panic when a deadline approaches.

Individual productivity is a destructive thing to measure.

Think about generativity instead: the difference between your team’s work without you, vs your team’s work with you.

Who on your team would the work slowly go to crap without? (on mine it’s @ddgenome.) Say thanks to that person today, and tell their manager.

time-dependent rate phenomenon

Here is a fascinating article about biology: https://www.quantamagazine.org/20170314-time-dependent-rate-phenomenon-evolution-viruses/

It says that when you look at organisms over a few generations, you see lots of mutations. Yet when you look at those organisms over thousands of years, the rate looks much slower.

This means that we have lots of variation, and it circles back to where it started after a while.

If we consider this with our teams: say a team has a retro every 2 weeks, and they choose to try something different in their process for the next 2 weeks. They can keep doing this over and over, and at the end of the year, they’re likely to be quite close to where they started. A year-over-year retro (carefully recording norms each time, because we’ll forget by next year) would show that not much has changed. And _this is success_.

Consider this with our teams: if velocity is constant from sprint to sprint, we’re doing it wrong. We’re not changing, we’re not implementing anything new. Average velocity over 6 months should be pretty constant, but if week-to-week it’s the same, either we’re cheating somehow, or we’re not evolving at all.

If frequent changes stress you out, take a broader perspective. We’ll circle back, and be mostly the same but hopefully a little better. If the lack of change over time bores you, focus smaller, and try temporary changes to keep yourself learning, and very gradually the organization will learn too.

changing systems

Is there a word for acceptance without approval? intimacy without attachment? embracing without preserving?

To change a software system I need to become part of it, I need to understand how the bits fit together. For example: When we spin up a new AWS instance, we use code in one repository, which activates a system controlled by another repository, which includes a script that shells into another machine where it makes an HTTP call to localhost to a service in a third repository, which digs around in its history and links to artifacts created by code in a fourth repository to package up a fifth repository. Do I approve of this flow? It doesn't matter. I need to accept its existence, without accepting its inevitability. I need to love it enough to understand it, and only then can I confidently change it.

In social systems too, I am part of the system already. I need to understand its workings; I am part of them. It is from here I can exert force for changing them. Resentment, denial, judgement: these hurt me, and they change nothing. I will face my fear, and my pain, and let it pass through me. I will be not surprised, I will be not approving, I will not expect that because it is this way now, that it must always be. I am part of the system but I do not help it self-preserve, in ways that I am able to see because I no longer punish myself with resentment and terror for being able to see them.

Participation is not endorsement.

To understand a complex system, whether society of software, means immersing myself in it, while retaining perspective. It means observing each interaction without surprise, it means intimacy without losing my self. To understand deeply is to love -- to love without needing it to stay the same. There's a concept our culture doesn't have: love, intimacy, familiarity while encouraging change. Even with children we are like "I wish they didn't grow up so fast!" No. Let them grow, it is the changing that holds their greatest beauty.

When I can hold in my head both the way the system works now and the way I wish it to work soon, then I can find the stepping stones and the pressure points to move it from here to there.

Stepping Stones

(from a progressive political position) My husband and I shouted at each other over Obamacare this morning. “It’s progress!” I said. “It is not!” he felt. Later he calmed down and admitted that Obamacare has helped some people some. But what we need is single payer.

Even if everyone in the government agreed that single payer healthcare is better, the way to get there is not BAM! Change the rules. Healthcare companies, out of business. Doctor offices, change operations. We don’t get from here to there in one decree.

We need stepping stones. Where we are today matters. How the system works today determines where we can go in the near future. Obamacare is a small step. Change the system, let it adjust. Change a bit more, let people move. Provide transitional support at each step. Sometimes it takes generations for opinions and skillsets to shift.

Even in tech, when we talk about “disruption,” we don’t mean BAM! change everyone right now. We mean: add a new option to the system, which alters people’s preferences, and then the system changes itself.

In software systems, where we are today matters. Once software is in production, change is hard. It’s incremental, or it’s dangerous. Each step needs transitional support. Customers need time to adjust, APIs need backwards compatibility.

In developer tooling, where an organization is today matters. The latest new open-source deployment system looks cool, and could be less work than maintaining the one we have now, but getting from here to there is far more work than either. It means making the new tool support everything we currently do. It means keeping today’s UI working, running both systems for a while, as we train teams in the new way. It means migrating production services gradually.

Aysylyu Greenberg gave a great talk about implementing a new distributed build architecture at. The majority of the work went into transition.

Standing up the new system is easier than moving from the old to the new.

There is no “just rewrite it.”

Improving systems (software and people) can be orders of magnitude harder than building them from scratch. Yet when we’re not starting from zero, our work has guaranteed impact: everyone now participating in the system will benefit. I like this kind of work, but it takes a lot of patience. At least in software, we can work faster than government. Releases are faster than generations.

Haydn or Beethoven

Joseph Haydn wrote over a hundred symphonies. He pretty much defined and perfected the form of the symphony, as within it he wrote so many, so well loved. Beethoven, a generation later, wrote only nine symphonies. Few - and yet each one is different and unique, each one redefined the structure in a new way.

As a developer, are you a Haydn or a Beethoven? Excellent software, a process defined and followed to great productivity? Or excellent software, a process altered with great originality? Both are masters. Haydn was prolific in his output, while Beethoven was prolific in his innovation.

Which does your company need more of now? And how would we hire for each?

Source: How to listen to and understand great music, a course by Prof Greenberg. Lecture 29 or 30.

distributed systems in a band

Watching the symphony last night: the conductor provides common ground for the joint activity of playing the music. She is a single point of coordination, that canonical clock that everyone can see at the same time. There is no information delay from the conductor to any member of the orchestra. They are playing from the same script, they have situational awareness of each other, plus they have a common source of timing.

Maybe a jazz band, playing improvisationally, is more like a distributed system without a central coordination point. All listening to each other, all sharing enough knowledge of the music and each other to respond and build the song. They have common snippets of scripts, and lots of awareness of each other.

For distributed systems in computing, imagine a jazz band with information delay. They’re trying to improvise but they can only hear what the other musicians played a few measures ago. Worse: usually it’s one measure ago but sometimes it’s several measures ago and no one knows which. They have to decide what to play in realtime based on out-of-date information. Everything they play will affect their peers, but they don’t know when.

Yeesh.

Add a central point of coordination, and a strict script, and we’re back to being in an orchestra - except with information delay, they can’t play in sync. If it takes the conductor’s movements a different amount of time to reach each musician, they aren’t playing together anymore.

We want our software systems to sing. But we can’t expect them to play a perfect song, not with central coordination, not with peer-to-peer. I don’t know what the solution is, but more and more I appreciate that this is hard.

Good news, bad news

The big news stories are almost always about disasters and terrible things. The good news in life is not nearly so dramatic. The good parts are people helping each other, one-on-one. One building going up, one life improved by science. Each child who grows and learns for another day.

This is a property of complex systems. Improvement happens in “lots of tiny, measured steps in the same direction performed by many people over a long time.“ (@jackdanger) Catastrophic failures are sudden.

It is the same with software. Once critical systems are in production, and reach a level of scale and complexity that makes them useful and hard to change, improvements come in small incremental steps. Disaster comes suddenly. Success is in keeping them alive to grow and learn another day.

Trending Blogs

Recently Viewed Blogs

True in software, true in life.