#bourne shell

[mentalisttraceur-software]

I think it's a really good sign of growth and healing that I'm finding myself increasingly repulsed by the kind of portability extremism that once compelled me.

One of the biggest and worst examples was shell scripts. /bin/sh was the Bourne shell in UNIXv7 (prior to that, there was the Thompson shell, and thankfully I managed to keep my mind cancer from metastasizing further backwards in time to try to achieve compatibility with that shell too). After the Bourne shell, every /bin/sh on every system was a Bourne-like shell, and if you thought that meant you could just write something that worked, take a glance at:

GNU Autoconf's Portable Shell documentation.

Sven Mascheck's various pages.

Paul Jarc's "lintsh" notes.

Ubuntu's "dash"-as-/bin/sh guide.

and others which you can find from there.

Now, a healthy person simply rejects this problem space. But for years, I was obsessed with writing shell scripts which would work on all /bin/sh still in production. It started as a growing annoyance with so many programs depending on bash - I was otherwise happily using a system with a more minimal shell at the time, and the limitations of my beloved Nokia N900 as a pocket Linux device gave me some real reason to prefer "reducing bloat" back then. Of course if it mattered to me, my compassion generalized it to everyone else in the same boat (everyone real or imagined... and in this case, mostly imagined). Then one day in the first year of my career as a software developer I got into a small argument with a coworker about them mandating #!/bin/bash instead of #!/bin/sh in our shell scripts - after he asserted that it was unreasonable to expect developers to remember what is or isn't a bashism, my maladaptive narcissistic cope reflexively kicked into full gear and now I had something to prove.

I still remember bits of that evening after work. It's... kinda horrifying looking back on it, because I was aware of what was happening in my mind. I was aware that I was basically starting to involuntarily, compulsively terraform my own preferences and values about shell scripts, from the modest and real and practical "I just want scripts to run on my N900s (BusyBox ash implementation for /bin/sh), and maybe also my Debian boxes (dash for /bin/sh)" to some perverse "principled" stance with poorly-defined scope which was divorced from any specific concrete goals. I had seen this runaway snowballing of artificial nitpicky values happen in my mind before, and I recognized that what I was doing in my head was feeding it, that it was happening again or that I was making it happen again, and I felt some conflict with that, I could see how it was bad... but back then I didn't know how to do anything about it. I didn't know how to diffuse those wants back then. I could in some technical sense, have chosen to not do it, but I couldn't stop wanting to, and I couldn't stop rationalizing it.

So I became the kind of guy that basically had every caveat mentioned on the above pages memorized. I even went as far as having a Solaris 10 VM, some old Android phones, and a PDP emulator running UNIXv7, so that I could test things not mentioned or not elaborated on those pages. But since it's really costly to remember so much trivia, I only remembered the caveats themselves, not necessarily which shells/systems they applied to. I could tell you off the top of my head "well you see, on some shells, 'set -e' will not affect the code inside functions", but I couldn't tell you which shells - I just had the caveats grouped by

"only matters on systems that no one runs anymore",

"only matters in situations you/we will never need to be compatible with (like Solaris 10's /bin/sh)",

"only matters if you want portability on Windows ports of UNIX-y shell stuff",

"only matters if you want portability beyond just Linux", and

"only matters if you want portability beyond just 'bash'".

I also used to have a little template for shell portability disclaimers that I would add to my shell scripts, deleting/re-adding lines as-needed:

# This script is compatible with Bourne and POSIX shells. # EXCEPT for the following exceptions (last verified on YYYY-MM-DD): # Comments (Appeared in 1981, still not universal around 1987) # Functions (First appeared in SVR2 Bourne shells in 1984) # `mkfifo` (First appeared sometime circa 1984, possibly earlier; unsure) # `test -p` (First appeared in SVR1 Bourne shell in 1983). # `wait` exit status (Missing in Almquist shell until 4.4BSD in 1993) # `hash` builtin (First appeared in SVR2 Bourne shells in 1984) # `type` builtin (First appeared in SVR2 Bourne shells in 1984) # $() is used instead of `` (not supported by some ancient Bourne shells) # `shift` when no positional parameters (broke some old MIPS RISC/os shells) # ${VAR%glob} substitution (Solaris (<= 10) /bin/sh does not support it) ...

That version of me looked at my old esceval.sh with pride, as if it was important or worthwhile. It tries to use modern-ish POSIX shell features but falls back to portable shell if it must. Basically every single line has at least one detail that is a deliberate portability choice. Almost every degree of freedom has been optimized for portability (and then some performance optimization within that) - change almost anything and it's probably less portable.

I revisited "esceval" for the first time in years this past week, and I noticed something really nice. I no longer have enough appetite for this portabiliy stuff. I'm too acutely aware, down to my motivating emotions, that it's a waste of my life. I'm once again in touch with actual concrete use-cases and benefits that have high odds of coming up in my life. I've re-learned to value myself and my goals more than this portability shit.

So I'm going to delete the portability fallback from "esceval.sh". I'm done trying to figure out what the portability fallback looks like for the other esceval pieces that I still want to finish. Unless I'm being compensated better than I can get elsewhere, I'm never again going to lift a finger to support Solaris 10 /bin/sh, or Android phones lobotomized to the point of not having a "printf" command in their shell, or anything else that isn't at least POSIX-compatible shell. And even then I'd suggest implementing that by writing a backpiler from modern shell to older. Maybe I'll answer portability questions if I still remember the answer and can say it off the top of my head - I enjoy helping people after all.

And it goes deeper than that. I'm very done giving Bourne-style shells nearly as much time and effort as I've given them so far. They're good DSLs for redirecting file descriptors and sorta okay DSLs for invoking and managing processes, and that's about it. As an unfortunate practical matter, Bourne-style shell is one of the most widely deployed programming language families, so if the goal is "I want to be able to give this tiny CLI to a coworker so they can run it on their machine with minimal human hassle", it can be nice to have a #!/bin/sh implementation (but so is having a couple statically compiled executables for the common platforms and a cross-compiler ready for the rest, or a Python script, or [...]).

It'll take me some time to figure out exactly where that balance is, and to fully unlearn the various hangups and compulsions that I've built up which motivate writing a /bin/sh script instead of something else, but what I've been doing so far definitely ain't that balance, ain't even close, and now I finally have a strong-enough hunger for breaking free and moving in the direction of that healthier balance.

[mentalisttraceur-software]

If we slap this

source_() { if sourced; then . "$@"; else _source "$@"; fi; }

_source() { sourced() { true; } && . "$@"; _source_return "$?"; }

_source_return() { sourced() { false; } && return "$1"; }

(PATH=; export PATH; sourced) 2>&- || sourced() { false; }

at the top of every shell script, and load it into all of our interactive shells, then every shell script sourced with

source_

can tell if it is currently being sourced with

if sourced

and this is forward compatible with a hypothetical beautiful future where shells just implement a `sourced` builtin themselves.

[mentalisttraceur-software]

I recently made two simple shell one-liners to practice memorizing and checking my passphrases - without ever storing them. Commented versions (note that this is using GnuPG 1 - you have to adjust the commands for GnuPG 2):

pwhash()

{

‍ ‍ ‍ # Password Hash

‍ ‍ ‍ #

‍ ‍ ‍ # Uses GnuPG to encrypt an empty file

‍ ‍ ‍ # with the password, thus storing a

‍ ‍ ‍ # secure salted hash of the password.

‍ ‍ ‍ : | gpg -‍-symmetric -‍-cipher-algo AES256 >"$1"

‍ ‍ ‍ # `:` is the no-op shell command,

‍ ‍ ‍ # and thus the pipe reads empty.

}

pwcheck()

{

‍ ‍ ‍ # Password Check

‍ ‍ ‍ #

‍ ‍ ‍ # Tries to use GnuPG to decrypt a given

‍ ‍ ‍ # file with the password, which fails

‍ ‍ ‍ # unless the password is correct.

‍ ‍ ‍ gpg -‍-decrypt -‍-quiet <"$1"

}

So as an example of usage, I might do

pwhash ~/memory/tumblr.com

then type my Tumblr password twice, as you'd expect from any proper password input where the correctness upon initial storage matters, and then when I want to practice or check it, I would do

pwcheck ~/memory/tumblr.com

You can of course use this for memorization of any sensitive information, not just passphrases. Pairs well with any automatic file synchronization or cloud storage solution. I'd like to connect this to a spaced repetition system somehow, but my forgetting sense is good enough now that I can do without.

[mentalisttraceur-software]

When it comes to implementing shell modules properly, two more pieces that need actual code besides SHPATH and `sourced` come to mind.

First, it seems like it would be really nice if some sort of recursion protection was possible. Circular imports are a problem in many programming languages, but they might actually be very manageable here. Bourne-like shells are very "late binding": names are not looked up until the command they are in is invoked. So if two files source each other, in the typical case where they both just define some functions that the other uses, then the only problem with the cyclical import is the recursive execution itself.

Second, the robustness and portability properties of the sourcing feature which the shell provides. It may actually be that a proper implementation instead has to reimplement the whole thing, parsing out and looping on each path in $SHPATH, trying to `cat` the file into a variable, and then using `eval` on the contents - or something like that. This might seem perverse, but it may prove to be the only way to get consistent and reliable behavior on all shells worth supporting.

I may back away from bothering with both of these, but they are the sort of thing that I really want to fully think through when I see them in the problem space or design space.

[mentalisttraceur-software]

One of the annoying things about Bourne-like shells is that they systematically conflate their own variables and environment variables. You cannot portably touch any variable in a Bourne-like shell without it possibly affecting an environment variable with the same name.

If you're really lucky, you're using a shell where `local` or `typeset` do not unset or override an environment variable of the same name if it exists. But in many shells, if you already have an environment variable called `FOO`, and you run

local FOO

FOO=bar

printenv FOO

it will either print nothing because the variable has also been unset in the environment, or it will print "bar" because it still remembers that a variable named "FOO" is exported to the environment.

[mentalisttraceur-software]

So, combining the `sourced` implementation with the SHPATH idea:

source_() { if sourced; then _source_ "$@"; else _source "$@"; fi; }

_source() { sourced() { true; } && _source_ "$@"; _source_return "$?"; }

_source_() { _source_PATH=$PATH && PATH=$SHPATH && . "$@"; }

_source_return() { sourced() { false; } && return "$1"; }

if (PATH=; export PATH; sourced) 2>&-; then PATH=$_source_PATH

else sourced() { false; }; fi

By the way I want to point out that the naming and formatting here is awful. In this case there is an exceptional goal of minimizing visual space and lines taken up, because it is boilerplate that would need to be pasted into all participating files. But don't take this as an example of what is generally appropriate.

[mentalisttraceur-software]

@vaguely-none replied to the SHPATH post:

shouldn’t that be _source_PATH="$PATH" (added doublequotes) etc?

The first order approximation answer is "yes, good!"

The pedantic technical answer is:

Regular variable assignment is one of the two special places in the syntax of Bourne-like shells where field splitting and globbing are not done on the result of expansions, so it is always safe to leave them unquoted.

Caveat: I said regular variable assignment. That does not apply to assignments that are syntactically just arguments to other commands, like `export` or `env`.

Caveat to the caveat: `local` is special and has all sorts of other portability caveats - in at least some shells, assignments that are "arguments" to the `local` "command" are also syntactically special and bypass field splitting and globbing.

True wisdom is probably to just quote them anyway, because coding for the human includes not doing things which are only correct due to invisible edge case reasons, because such things are liable to undergo nuance decay into broader wrong usage when people learn from example.

[mentalisttraceur-software]

SHPATH

Every shell descended from the original Bourne shell searches $PATH when sourcing a plain file name instead of an absolute or relative path name.

The downside of that is that using the same search path for both conflates sourceable shell script libraries with executable programs. This is bad because

if a file is only executable or only sourceable, having it available as a candidate file when looking for the thing it is not is at best inefficient and at worst might cause errors, and

if a file is both, then you cannot reimplement that executable in a language that the shell cannot source without breaking its usability as a sourceable.

The upside is that this might be a good way to implement a proper library loading feature in the shell. Suppose we have a separate shell library search path. I'm going to call it SHPATH for now.

So ideally, when sourcing, the shell would search $SHPATH, not $PATH. So how can we implement that using what the shell gives us? Well, to start, before sourcing we would need to do this:

_source_PATH=$PATH

PATH=$SHPATH

And then we source the file we want. But there's a problem: since that PATH value has to be set when the sourcing builtin is called, that is the PATH value the sourced script sees while it is being sourced. So we have to reset the PATH back to the right value at the very beginning of the sourced script, by doing:

PATH=$_source_PATH

Of course, a top-level script that is being executed instead of sourced should absolutely not do that, else it will lose its PATH. Good thing we have a very robust and portable mechanism which cooperating scripts can use to tell if they are being sourced.

Of course, if you don't mind the edge cases, you can just use the expansion of $_source_PATH or the comparison of $PATH to $SHPATH to decide if you are being sourced.