#init system

[mentalisttraceur-software]

systemd won the Linux init system and service supervision game because a side-effect of being a monolith is being a "no assembly required" solution.

This is probably the most important lesson I took away from the unfortunate way that played out.

If you're making an s6, runit, or perp, and you want to win over a systemd, you need to understand that systemd is playing an entirely different game than you.

You are designing the best LEGO pieces for building a castle - you're thinking how to elegantly decompose service supervision, socket activation, cgroup management, and all the other things, to make the pieces that best enable a user like you to build exactly the castle you want. Systemd is shipping a premade castle - they're throwing it all into one project and showing users how easily they can do each of those things with a config line.

So design your LEGOs, sure. Wax poetic in your docs and blogs about how elegantly minimal and composable your LEGOs are to your heart's content. Bask in that subtly morally and intellectually elitist satisfaction of building a toolset which maximizes flexibility and freedom for the power user who wants to understand and control each piece.

Then take all your LEGOs, build an opinionated all-features-included castle out of them, build it so that it makes it easy to do all those things without grokking the elegance and vision of how all your LEGOs fit together. That competes with systemd. That's what you market to the mass of users.

Because when most system admins or developers come to you, they don't want to understand your LEGOs, how to put them together, which separately released pieces to install for those features, or how to bootstrap a system from them, either just because they don't care, or because they don't have the time. Or if they do, the rest of their team certainly doesn't.

Monoliths win the quick-and-dirty, good-enough, 20%-effort-for-80%-of-the-result, I-just-want-this-to-just-work usability game.

If you want to beat the monoliths with LEGOs, pre-assemble the LEGOs into a better monolith, then market that monolith the way monoliths market themselves - on features, efficiency, operational ease, and so on.

Save the UNIX philosophy for the philosophers.

[mentalisttraceur-software]

Actually, in principle, PID 1 in a UNIX-like system only needs to

fork and execute one child, then

reap any process exits that it receives.

Notably, init does not have to pass the PID and exit information of reaped orphans anywhere, like I said in a prior post. Not in the strictest sense. Not if you have some other satisfactory means of reacting to the deaths of processes that only init can reap.

These are the only things that absolutely cannot move out of init because:

Traditional user-space initialization in UNIX only starts the init process - nothing else runs in user-space unless init starts it, so you want more than one process, init has to fork once.

PID 1 has final responsibility for preventing a resource leak by reaping zombie process entries from each process that got orphaned by its parent - and subreapers if it had any.

The only way to have other functionality be implemented outside of the init binary - or outside of dynamic libraries that init loads - is to have init exec a different binary.

Put those three together and you get the minimal necessary feature set. Fork, parent moves on to the reaping loop where it mostly idles in the wait system call, child moves on to exec.

You could also split the fork-and-exec part and the reaping-loop part into separate binaries. Of course then you have to exec in the parent as well, and either hard-code one more path, or complicate the CLI interface, or both.

[mentalisttraceur-software]

If we want PID 1 to support upgrades without rebooting, it also must have some way of being told to execve another program.

So to add that feature, the executable that contains the PID 1 reaper loop must also have a code path for executing a command, and a code path for some kind of signaling mechanism.

The signaling mechanism could in principle be anything - native signals, reading from a pipe, polling the file system, abuse of the wait system call by creating an orphan process and having it exit with a very specific status or kill itself with a very specific signal, and so on.

That last one is probably a terrible idea, but I wanted to mention it because it's the only mechanism that doesn't require adding a different code path for interfacing with the outside world differently to an as-minimal-as-possible init.

But signal handler is probably the most appropriate thing to do here, even though a most minimal init doesn't strictly speaking need to have a signal handler - if you're doing something more complicated you might need to handle SIGCHLD, but if the init is doing nothing except waiting on process exits, SIGCHLD is redundant.

[mentalisttraceur-software]

In principle, PID 1 in a UNIX-like system only needs to run one process, then reap any orphaned process exits that come up, and somehow pass the PID and exit information to that one process.

The first variant of this in my mind is to spawn a single process, with a pipe from init into that process, then supervise it and restart it if needed. Orphan exit information gets written to the pipe. That process, in turn, could be the root of any other fancy features like service supervision and so on.

A more extreme version doesn't even need to supervise any process. The most minimal distillation can just execute a command once at start - then execute that same command again whenever it reaps a dead orphan, with the exit information as one or more extra arguments.

Of course this is probably maybe taking the minimalism too far. It's notable that s6, runit, and perp all put more logic than that into PID 1.

The obvious problem is that everything I just described adds overheads.

In all of these cases, the whole philosophy and benefit of this is undermined unless you pass the reaped orphan information in a human-friendly form, so there's some stringification and parsing overhead. Although... I think the stringified form ripples far in this design, without ever needing to be parsed, so that most code is just dealing with process identifiers as opaque byte strings, which just happen to be readable as integers when a human looks at them.

The first version needlessly takes a round trip through a pipe to handle any reaped orphans, which is an overhead that at a minimum you pay for the worst-case service supervision path - when the supervisor dies and then the supervised process dies. A reliable wills implementation - as reliable as you could get in userspace at least - would also have to go through this path in the worst case.

Of course a pipe between two processes could be easily and safely optimized to a futex and a shared memory mapping, but you still have to context switch between processes.

The second version seems at a glance like even more of a toy, with even worse overheads. It takes system call overhead to fork and execute a new process on each PID. It takes overhead to convert PIDs to and from a form compatible with command-line arguments. If you fire off a process each time you reap, then you pay overheads to save and load any state externally. Though there is a certain elegance and power to all this state being stored in the file system - imagine the manual debugging and introspection potential.

But a possible redemption I see here is that once you know what functionality you want to go to production with, even the second version could actually be optimized by rolling the functionality of both this most-minimal init and the command it invokes and the external state-saving into a larger binary which runs as one process, and if you want to keep the file system view of the state you can expose it as a virtual file system mount.

One last negative: forking and executing does create surface area for problems where none previously existed. By definition, those system calls can fail - maybe not in practice in this context, especially if you carefully set up process resource limits, but in principle. Then again, a basic function call compiled down to machine code can fail too, we just pretend it can't - for example if you overflow your stack, or because you ran it on an older CPU or from a corrupted binary and hit an illegal instruction, or the CPU overheats. So in all cases we could - and maybe for an init system should - ask ourselves what is the right behavior in such cases. Going through a fork and exec just forced us to see it.

I suspect there is something elegant and powerful in this design direction, and I know for sure it would be a great learning exercise to try to build a system that boots to a usable state based on something like this. This idea has been rattling in my head for some time... and maybe these bits of vision - state as a file system, ability to optimize it down to one binary and process later, and the analogy of fork+exec errors in this specific use-case to inescapable lower-level errors... that feels like maybe that's the last pieces I need to confidently start tinkering.