Prone adopts the Ninja build system
I've spent most of my technical energy on Matplotlib lately, and I still plan to finish the article I was writing on my effort there, but there's human parts of the process that are just going to go a bit slow, and I'd like to wait to write about that work until it's merged. In the meantime, I've found it refreshing to get a bit of stuff done on my own little hobby project.
So, what's new, and what's coming?
Prone is officially "built different"
As of yesterday, Prone still uses make as an outer wrapper for running commands, but the meat of compilation work is actually orchestrated in the Ninja build system.
"I have altered the build system. Pray I do not alter it further." - Dev Vader
While everybody's heard of GNU Make and various compat-alikes for it, Ninja is a (comparatively) newer kid on the block. The two have a lot in common, conceptually. They both use file modification times and a dependency graph to decide what needs to be rebuilt, and how to rebuild it. Where the two part ways, though, is that Makefiles are meant to be directly written by humans (and have features like wildcards, populating variables with shell executions, etc.), but *.ninja files are meant to be written by other programs, so that Ninja itself can be simple and fast. This is why Ninja is a popular backend for multiple human-friendly build frontends, like CMake and Meson.1
The main reason that most people use Ninja is speed, particularly on large projects, and the manual even says:
... if you’re happy with the edit-compile cycle time of your project already then Ninja won’t help.
As much as I care about the speed of incremental builds in Prone, I'm exactly the person in this picture! Builds already are fast, because Prone is small and organized for both compile time and runtime performance.
So, while speed was a requirement, I switched from Make for other reasons. Throughout the life of Prone, I've repeatedly run into challenges getting Make to do what I want, without surprising behavior. I've had to debug problems from implicit rules, problems from using generated files within a wildcard selection, and general messiness from in-tree builds being the path of least resistance. I really should have kept a tally of all the times I thought I had a good Makefile, until I later realized some gap between my mental model of the Makefile and the way that GNU Make actually interprets it. It also tends to be very noisy in terms of output, while missing some features you'd expect from a higher-level tool.
Ninja gets exactly these things right:
The config format is simple and unambiguous. Very readable and reliable.
The terminal output is much cleaner, both in the happy case, and in how it handles failures (with per-command buffering). Command descriptions are a killer feature.
When generating output files, their parent directories are automatically created for you.
You have total control over how you generate configs.
Every single one of these is a big win for me, but especially the last one, because I have ambitions to run Prone on embedded devices, and cross-compiling brings a wrecking ball to a lot of normal development assumptions. Being able to really micromanage my build graph gives me a lot more peace of mind that I can craft a cross-compile experience later that's actually enjoyable (possibly taking some inspiration from how BusyBox does it). I know I'm making an investment today that will save me a lot of floundering when I'm doing ESP32 stuff again in a few months.
So, Ninja's a good fit, and it was very easy to port to. I knew I was going to need to add a configure script soon, to smuggle information into the C compiler environment that isn't otherwise available, like "is this a chip that wants to store the type info of DVs in the high bits or the low ones?" So, rather than adopt some existing high-level build system like CMake or Meson, and lose some of the control benefits I mentioned earlier, I wrote a bash-based configure script that generates a build.ninja file, complete with comments, which can eventually populate some C defines with compiler flags. It's simple, clean, and straightforward. I've been a Make apologist for a long time, but it was a pretty revelatory experience to incrementally move more and more responsibility onto Ninja, and every step of the way, it either just worked exactly as I expected or failed for reasons that were immediately visible and understandable.2
It was also an opportunity to go back to static linking, because that's not a bottleneck for the test cycle anymore, and it makes a few things in the performance testing domain more straightforward. More code can be inlined and optimized3, the dynamic loader is doing less work on startup, and the generated binaries are a lot more portable - I'm still linking against libc, but nothing else, so I could compile a Prone-based program on my laptop and then just copy it to any other Linux+GLibc+AMD64 computer, and it would run. I could finally stop using LD_LIBRARY_PATH=./dist all over the place, too. Honestly, though, the biggest quality of life win here is that I use the perf command (especially with assembly annotation) a lot when I'm on one of my optimization benders, and perf tends to get a bit disoriented with every function call going through the Global Offset Table (the GOT). Studying performance traces just got way cleaner.
This also leads into some of the other stuff I plan to do soon!
Getting a header (or two) in life
Right now, Prone's approach to headers is mostly one-size-fits-all, with a few flags. The most extreme examples are examples/test.c and examples/bench.c.
// test.c
#define PRN_INCLUDE_TEST
#define PRN_TEST_INCLUDE_IMPL
#include "../src/prone.h"
// bench.c
#define PRN_INCLUDE_TEST
#define PRN_TEST_INCLUDE_IMPL
#define BENCH_CT 10000
#include "../src/prone.h"
In both cases, Prone's full test harness is available within the header if you ask for it. PRN_INCLUDE_TEST causes some of the basic test header stuff to be provided at all, while PRN_TEST_INCLUDE_IMPL brings in the source code for actually running the tests in a random order and reporting the results. These are just dials you can tweak to determine how much stuff you get (and which) when you #include "prone.h". The only difference between the two examples, is that bench.c customizes BENCH_CT, while test.c leaves it at the default value of 1.
This approach was semi-inspired by Tsoding's nob.h build system, and while it works, I'm not sure it's the best approach for Prone. I find myself struggling to remember the names of the dials, and I'm the one who made them! For big picture "only include what you need" type stuff, it makes a lot more sense to have multiple headers within a common directory, like #include <sys/types.h>.
With this in mind, I have a goal to move the headers out of the main source tree and actually set the compiler flags so that we consistently import them like #include <prone/types.h> (with those angle brackets) everywhere.
Currently, as Prone is outgrowing being a bare prototype, there's a haphazard mix of name conventions. Ideally, at the symbol table level, everything should start with a prn_ prefix, but it's also pretty unpleasant to use the prefix all the time for everything. While I'm usually glad to avoid C++, there are a few specific features that would be nice to work with, and namespaces are definitely on the list.
Right now, the thing that's worked the best is the following pattern, which you see a lot in the current runtime.h header, but not as consistently as it should be:
#define atom_repr prn_atom_repr
#define atom_wrepr prn_atom_wrepr
HText *atom_repr(LText atom);
HText *atom_wrepr(HText *s, LText atom);
With all the rearranging of headers, I have the opportunity to try other approaches, especially because I can rely on Ninja for some pre-processing. Consider the following code:
#define PRN_NS(name) prn_##name
HText *PRN_NS(atom_repr) (LText atom);
HText *PRN_NS(atom_wrepr) (HText *s, LText atom);
When run through the C preprocessor, this gets us the same result:
HText *prn_atom_repr(LText atom);
HText *prn_atom_wrepr(HText *s, LText atom);
However, we can also generate a <prone/strip.h> file by finding all instances of PRN_NS in the other headers, thus generating:
// prone/strip.h
#define atom_repr prn_atom_repr
#define atom_wrepr prn_atom_wrepr
Making for an optional header that allows internal development to be a lot easier.
Something else that's come up lately is that, even for people who don't use Prone as a language, Prone's type system has a lot to offer as a data interchange format between libraries in a single program. It's simple, has most of the concepts you could care about, and prevents some of the things that make data interchange a hazard.
If you want to share information between libraries that could translate nicely to JSON if you needed to, Prone's types are a very nice way to share that information between parts of a process. On the receiving end, you can actually receive complex objects that don't feel like a white elephant gift! Total support for reflection, no worrying about object "behavior."
This gets to an idea of Prone as kind of a layered onion of functionality that a person might need, rather than a single giant glob of functionality. The exact layering, I need to play around with, but I know that I could offer the ability to do reading and reference counting on data at the most fundamental layer, without even needing to specify allocation stuff. Then you have object construction, and beyond that, pretty much everything else!
One of the things I was struggling with for awhile is, in an embedded environment where even things as simple as print() may not have a sensible way to implement them4, how on earth do you come up with an abstraction for the current platform (which might be anything from Linux to WebAssembly to a PIC controller) that can somehow be a bedrock for the standard library?
The answer is that you can't, so rather than most of Prone depending on the platform, the platform depends on Prone Core - the stuff you can do in memory, no matter the shape of the outside world. Platforms extend the standard library to cover topics like IO. So, roughly speaking, using Prone as a REPL on Linux, the stack ends up looking something like this:
You -> REPL -> Prone Linux Platform -> Prone Core -> Prone Types
There's probably going to be some code sharing between platforms, but they're allowed to diverge from each other, whereas Core and Types will always be consistent everywhere, and the REPL will only have some narrow/trivial dependencies on the platform's IO.
This will also be an opportunity to revise the platform tests to rely on pure bash with no Python. Platform tests are small programs that are run in a tiny harness that reads standard output, standard error, and the exit code. These are then turned into a single text object that can be diffed against the expected result. The concept is still good, but being able to reduce a dependency (Python, in this case) helps ease the discomfort of adding a dependency (Ninja) in a project like Prone where minimal deps are considered a virtue.
Prone is a highly dynamic language, and sometimes you need a wildcard type that maps to the behind-the-scenes DV type. Python would call it typing.Any, and if you squint, C++ would call it auto. In Prone, that's going to be called dyn.
This is mainly a naming choice, but it also comes from some subtle consequences of me thinking about Prone in the "pure data interchange" use case, where typed arrays are more common. The existing list type is great, but I do want to support typed arrays eventually (like an array of 12 u16s), and really, heterogenous lists are just arrays over some kind of any-type. So some future work is going into making typed arrays in the future, and treating heterogenous lists as typed arrays where the type defaults to dyn.
None of this is likely to happen at an immediate pace, because I'm still prioritizing MPL, because contributing to other people's open-source software will (hopefully) improve my employability. Prone is going to stay my for-fun, let-off-some-steam project. But when I do get time to work on it, this is the vague technical roadmap laid out before me. This kinda meshes with the remaining type matrix work I need to finish, including key-value stores.
I personally got my first real exposure to Ninja through Matplotlib, which uses Meson to generate and manage .ninja files. That first impression went something like this: "Ugh, what the ---- is this extra layer of stuff I have to learn? ...Oh. Oh that's kind of nice, actually."
There's a reason I'm such a huge fan of complex processes being broken down into smaller steps with plain, visually inspectable data being the thing fed between steps. It makes for multiple checkpoints that you can tap into and check for sanity, quickly tracking down exactly what went wrong.
This actually forced me to fix some Valgrind suppression config. There's a few tests that intentionally "break the rules" to verify what misbehaving code will do under different allocation strategies. Before, I'd put all my known-illegal behavior into a tiny helper function, so I could just make that my known exception for Valgrind. That helper function was so aggressively inlined, I had to give up and add suppressions for each of the allocator tests instead.
Of course, embedded is likely to have features that you wouldn't want to try to support or expose on AMD64. The "hello world" of a lot of tiny chips is using the GPIO pins to make a light blink on and off. Your average laptop CPU isn't exactly known for the killer feature of GPIO blinkenlights! Checkmate, expensive superscalar hardware ;)
