All the political stuff has been getting me down, so here’s a tech post.
I love non-cryptographic hashes. It’s a hard problem with tons of research and objective results. Whereas cryptographic hashes have to worry about stuff like collision attacks and that’s no fun. In the crypto world, speed can be your enemy (e.g. password hashing). But in non-crypto it’s all about speed and how well you cover the keyspace.
Turns out that Yann Collet, author of xxhash and lz4 compression came out with a new thing, zstd (code) last year when I wasn’t looking. This may be old hat to some of you all out there, but I think it’s pretty cool. It’s in the same space as zlib (i.e. gzip), which is to say medium-fast, medium-space, vaguely symmetric cost for compress vs. decompress. It does all kinds of neat tricks to be a LOT faster than zlib at similar compression ratios. The linked article goes into more detail. Spoiler alert: it’s pretty rad if you are a massive geek.
It’s also got some cool dictionary building stuff, so under some loads you can get massive compression improvements, similar to what Facebook published before with their preview image optimization.
I got all hot and bothered over LZMA/7-Zip/xz. The performance implications of LZMA were a barrier to many of the more general use cases. Zstd is just the opposite and I dig it.