#c static include

[mentalisttraceur-software]

One of my many "eventually maybe" project ideas - although I'm hoping other optimizations at other levels of the tech stack will make it mostly unnecessary - is an optimizing compiler turning shell scripts into single binaries with inlined implementations of all the shell commands it knows about.

That way for example canonical source code can look like `curl ... | sed 's/foo/bar/'` and the binary might use something like libcurl (ideally in a static includes way, although I suspect the world will keep insisting on the less efficient-to-implement approach of link-time optimization) and an inlined implementation of just the string-replacing logic of sed.

The unfortunate thing is that a realistic implementation would need a dictionary of heuristics and definitions - for a program with this name, parse the arguments like so, and based on that pick a pre-written implementation of it, and inline it. So if you have a non-standard or uncommon program, or are using a program in a very unusual or unexpected way, you might need to add/enhance/override the detection heuristics or program definition.

It would also probably never have full coverage over weird behavior edge cases. For example, if you are deliberately doing something weird like sending a SIGSTOP to one process in your shell script while another process piping into it is still running, then you can't optimize that pipeline into one process, at least not in the general case. (On Linux you can signal a specific thread instead of process, but then the sender has to be adjusted too.)

[mentalisttraceur-software]

C libraries supporting static includes cannot rely on pointers to incomplete types to do data hiding.

Luckily, this is pretty easy to fix. Instead of

struct foo

{

    /* private members */

};

we can use a dummy struct at the beginning of the private struct type as the public type:

struct foo { char unused; };

struct foo_

{

    struct foo public_handle;

    /* private members */

};

The public struct has to have something in it because C does not allow structs with no size or no named members. Some compilers allow it, but it is not standard. I made it a char to represent the fact that C does not allow any struct to be smaller than a char but we want it as small as possible. In practice the compiler will either add more padding for alignment reasons, or will optimize it to take zero space if it can prove that it never gets used.

Anyway, with that done, we can now change

struct foo * new_foo(/* ... */)

{

    struct foo * foo;

    foo = malloc(sizeof(struct foo));

    /* ... */

    return foo;

}

into something like this instead:

struct foo * new_foo(/* ... */)

{

    struct foo_ * foo;

    foo = malloc(sizeof(struct foo_));

    /* ... */

    return &foo->public_handle;

}

This works in a portable way because it is one of the few cases where C allows pointers of different types to alias each other: if the first member of a struct is a type, then pointers to that type can be converted into pointers to the struct type.

So by having a dummy public first member, we can pass around pointers to a type that never gives access to our private data, just like an incomplete type, and by making that dummy first member a struct we ensure that it is still a clearly unique type for both humans and compilers.

It might feel less hidden because when doing static includes the private struct definition is still visible in the translation unit, but hiding the data layout at the source level is not what data hiding is about. In either approach, it is easy enough to find out the struct layout or cast the pointer to poke at the data.

The most important part of data hiding is compiler protection against accidental or casual mutation of data our code relies on and accidental or casual reliance on data our code does not guarantee as part of its interface. This retains that.

[mentalisttraceur-software]

C Static #include

I think the C community should start treating compiling whole programs in a single translation unit as a first class use case. This is like doing static linking, except everything "link time optimization" is trying to achieve comes from the optimization the compiler already does very well.

For example, `#include "foo.c"` and even `#include <stdio.c>` should be possible and normalized. When I make a library, I put an include guard in the .c files for this reason. Or perhaps you still `#include "foo.h"` or `#include <stdio.h>` but a compile-time macro switches on including the full implementation source in the header. Another alternative is having a separate "foo-static.h" or "stdio-static.h".

This does force every top level name in a file to be manually prefixed. Data hiding has to look different. But the benefits are significant. SQLite does it. The way Go builds was deliberately designed to get these benefits.

Some of this idea is floating around under the term "header-only libraries", but that puts the focus on a different spot. The above examples give the library user a choice between regular and static includes.

It is best for choices like that to be exposed at build time. It is best if the choice can be made as both a catch-all default and on a per-library basis. This empowers users of the software the most. Coincidentally, if a piece of code is statically included, adding `static` to every function in it helps the compiler optimize even more. Finally, a library that wants to support both regular and static includes without code duplication will probably use a compile time preprocessor switch internally anyway, to include the implementation .c files.

So, I propose a simple static includes convention: libraries supporting static includes should

still use conventional separation into .h and .c files,

take the same care to prefix all file-scoped identifiers in the .c files as in the .h files,

include both .c and .h files in the development packages that normally include the .h files,

use a check like this in the .h files to conditionally include the .c files (where `FOO` is a prefix the library uses):

#ifndef FOO_STATIC_INCLUDES

#ifdef STATIC_INCLUDES

#define FOO_STATIC_INCLUDES STATIC_INCLUDES

#endif

#endif

#if FOO_STATIC_INCLUDES

/* #include one or more .c files */

#endif

[mentalisttraceur-software]

My idea of "static includes" for C code was largely motivated as a way to make it a lot easier for a C compiler to do whole-program optimizations.

Because it seemed really wasteful for C compilers to pour work into figuring out how to do link-time optimization after the source code has already been turned into object files.

But maybe the right vision is that instead of everyone writing libraries to support static includes, we really should just work on figuring out the best ways to enable compilers to do that link-time optimization. Imagine a really simple version: C compilers can trivially produce object files which include their entire C source code, and re-parse that code during linking to fully inform all their link-time optimizations. Perhaps as a string at a symbol name reserved by the implementation like `_C_source` or `__c_source`, and these can be deleted or catenated when the object files are linked. Is this the best way? Probably not, it's just an example of a really simple solution for the entire problem of losing any optimization-guiding information between C source files and intermediate linkable object files.

[mentalisttraceur-software]

One of the project ideas "I'll do eventually" for years now:

A libc implementation, except all of the implementation is in the header files. Like always statically linking, except every bit of the C library you are using is in the same translation unit so you get the full power of modern optimizing compilers.

Compilation on compilers that have it will use the modern inline assembly syntax that lets the compiler intelligently connect assembly registers to C variables, and of course the low level details like C runtime setup, the true entry point like `_start` which eventually calls `main`, etc, can all be included through the headers I think.

So the simple "hello world" case on a modern compiler on systems where the ABI between userspace and kernel is stable (read: not Windows) literally involves no actual linking of multiple object files: just one translation unit directly to finished binary. But I'd also want to support systems where the compiler can't do that, by having a single C file.

The underlying work will be reusable to any other "static include" base library we may wish to make instead of or in addition to the standard one.