LLBBL.blog @llbbl - Tumblr Blog

Python and Rust Have the Same Supply Chain Problem as NPM

Last post I walked through the threat model for supply chain attacks and dug into the NPM ecosystem specifically: postinstall scripts, npm ci, pnpm’s release-age cooldown. The same structural problems exist in Python and Rust, but the failure modes are different and the tooling has evolved in some surprising directions. Worth understanding both, because if you write any backend code in 2026 you’re probably touching at least one of these ecosystems.

Python: setup.py Is a Remote Code Execution Primitive

The thing most Python developers don’t appreciate is that pip install runs arbitrary code by default. Not after install. During install. If a package ships a setup.py, that file is executed in a Python interpreter the moment pip resolves the dependency. Whatever the author wrote, including reading ~/.aws/credentials, scraping environment variables, or opening a reverse shell, runs as your user with full filesystem access.

This is the part that confuses people coming from other ecosystems: venv and virtualenv don’t help. They isolate Python package versions to avoid conflicts. They are not a security boundary. A package installed inside a virtualenv has the exact same privileges as the user who ran pip install. None of this is a bug, exactly. It’s just an artifact of setup.py being a regular Python script that pip has always been willing to execute.

The defense-in-depth stack for Python looks like this:

Stop using pip. I mean it. pip is the worst package manager in mainstream use today and it is the single biggest reason Python’s supply chain story is a disaster. It has no native lockfile. requirements.txt is a shopping list, not a lockfile; it tells pip what to fetch, not what you actually got last time. Run pip install -r requirements.txt twice on two different days and you can get two different dependency trees, because pip resolves transitive deps fresh every time against whatever happens to be on PyPI in that moment. Builds aren’t reproducible. Hashes aren’t verified by default. There’s no separation between “what I asked for” and “what was actually resolved.”

Every other ecosystem solved this a decade ago. npm has package-lock.json. Cargo has Cargo.lock. Bundler has Gemfile.lock. pip has vibes.

The --require-hashes flag exists, technically, but it’s duct tape on a broken design. You have to generate the hashes with a separate tool (pip-tools), maintain them by hand, and remember to pass the flag on every install. Nobody does this in practice. The Python Packaging Authority spent fifteen years insisting pip was fine while every other community built proper lockfile-based managers.

Use uv or Poetry. Both produce real lockfiles with SHA-256 hashes for every direct and transitive dependency, both make installs reproducible by default, both are dramatically faster than pip. uv in particular is the obvious default for new projects in 2026, it’s a drop-in replacement that’s roughly 10-100x faster and treats the lockfile as a first-class artifact instead of an afterthought. Hash verification isn’t a flag you have to remember. It’s how the tool works.

This doesn’t protect you from a malicious package you pinned on day one. But it does slam the door on silent registry tampering, makes “what’s actually deployed?” a question with an answer, and gets you out of the pip swamp.

pip-audit for known vulnerabilities. Scans your environment or requirements file against the OSV database, PyPA advisories, and GitHub advisories. Run it in CI. Combined with a real lockfile you get a tight loop: pin exact versions, scan those versions for CVEs, fail the build if anything critical shows up.

Trusted Publishing (OIDC). If you maintain a package on PyPI, get rid of your long-lived API token and switch to OIDC-based publishing. Your CI runner generates ephemeral, short-lived tokens scoped to a specific repository, branch, and workflow. Leaked PyPI tokens have been the source of multiple high-profile compromises. Trusted Publishing makes the credential effectively un-leakable because it doesn’t exist as a persistent secret.

The thing I’d actually call out, though, is that none of the Python tooling addresses the setup.py execution problem at install time. Hash pinning verifies you got the right bytes. It doesn’t tell you those bytes aren’t malicious. For that you’re back to either sandboxing the install (Docker, devcontainers) or trusting the registry’s malware detection, which lags by hours to days.

Rust: The Safety Guarantees Stop at the Compiler

Rust’s reputation for safety is real, but it’s a property of the compiled language, not the supply chain. The borrow checker doesn’t help you when the crate you’re depending on exfiltrates your SSH key during cargo build.

The mechanism is build.rs. Crates can include a build script that runs before the compiler, with full user privileges. Procedural macros do the same thing at compile time. In both cases, the code can read files, open network sockets, do whatever it wants. A malicious build.rs is effectively an unsandboxed unsafe block that bypasses code review because nobody reads build scripts. The Rust core team has been discussing sandboxing for years, but nothing has shipped.

This isn’t theoretical. Two examples from the last six months:

September 2025: faster_log and async_println were caught scraping Ethereum and Solana private keys at runtime and exfiltrating them to Cloudflare workers.

March 2026: chrono_anchor, dnp3times, and time-sync, all masquerading as time utilities, were transmitting .env file contents to threat actors.

Both clusters used compromised GitHub OAuth credentials to push under legitimate-looking namespaces. crates.io authenticates via GitHub, so a phished GitHub account is a phished crates.io account.

The defensive tooling is actually better than what most ecosystems have:

Tool What it does cargo-audit Scans Cargo.lock against the RustSec Advisory Database. Run in CI. cargo-deny Lints the dependency graph. Block specific crates, enforce license policies, restrict registries. cargo-crev Decentralized “web of trust” where developers cryptographically sign crate reviews. Elegant, but heavy lift in practice. cargo-vet Mozilla’s pragmatic answer to crev. Centralized audit records per org, with the ability to import audits from peer orgs (Google, Mozilla, Embark) instead of re-auditing every transitive dep yourself.

If you’re picking one to start with, cargo-audit is the easy baseline. It’s npm audit for Rust and you should be running it in CI yesterday. cargo-deny is the next step up. It lets you actually enforce policy, which is what you want once you’ve used cargo-audit long enough to be tired of triaging the same warnings.

cargo-vet is the interesting one for any team beyond about five engineers. The insight is that you don’t actually need to audit every crate. You just need to know that someone you trust did. By importing audit records from Mozilla and Google, a small team can effectively delegate the audit work for hundreds of common dependencies without running anything themselves. It’s the closest thing the Rust ecosystem has to a working trust network, and it works because the cryptographic overhead lives at the org level instead of being pushed onto individual developers.

The Pattern Across All Three Ecosystems

NPM, PyPI, and crates.io all share the same fundamental design flaw: package installation executes attacker-controlled code by default. NPM has postinstall. Python has setup.py. Rust has build.rs and proc macros. Different files, same problem.

The mitigations also rhyme. Lock your versions to specific hashes. Run an audit tool in CI. Where possible, prevent install-time execution entirely (--ignore-scripts, pre-built wheels, sandboxed build scripts when they finally land in Cargo). Where you can’t, isolate the install with devcontainers, ephemeral CI runners, anything that contains the blast radius when a dependency turns out to be hostile.

Next post I’ll get into the isolation side specifically: devcontainers, OrbStack, Landlock, and the practical question of how a solo developer with no security budget actually keeps their laptop from getting owned by an AI agent that just pip installed a hallucinated package name.

Sources

Securing Package Managers: Why NPM, PyPI, and Cargo Are High-Value Targets

Defense in Depth: A Practical Guide to Python Supply Chain

PyPI Security: How to Safely Install Python Packages

Rust Supply Chain Security — Managing crates.io Risk

crates.io: Malicious crates faster_log and async_println

Five Malicious Rust Crates and AI Bot Exploit CI/CD Pipelines

About RustSec Advisory Database

cargo-vet FAQ

Auditing Rust Crates Effectively (arXiv)

Explore sandboxed build scripts — Rust Project Goals

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

Your Software Is Mostly Strangers' Code

Modern applications aren’t really written anymore. They’re assembled. Seventy to ninety percent of a typical proprietary codebase is open-source code pulled from public registries, NPM, PyPI, crates.io, maintained by thousands of people you’ve never met. Every npm install is an act of implicit trust extended to strangers, and that trust model has quietly become the weakest link in most security architectures.

Attackers figured this out a long time ago. Compromising one popular package gives you a blast radius that phishing campaigns can only dream about: CI pipelines, developer laptops, production workloads, client devices, all simultaneously. SolarWinds. The XZ Utils backdoor. The Shai-Hulud worm, which self-propagated through 170+ npm and PyPI packages by hijacking GitHub Actions OIDC tokens and quietly minted new publish credentials as it spread. The ByBit developer compromise. These aren’t outliers anymore. They’re the shape of the threat.

I want to dig into the mechanics of how this actually happens, and then look at the ecosystem most developers touch every day: NPM.

How Supply Chain Attacks Actually Work

The first thing to understand is that supply chain attacks aren’t really “vulnerabilities” in the classic sense. A buffer overflow is an accidental weakness. A malicious package is intentional code, written to steal credentials, drop a reverse shell, or exfiltrate environment variables the moment it lands on your machine. Traditional appsec tools were built to find the former. They are largely blind to the latter.

The attack patterns cluster into a few categories.

Typosquatting. Publish axois and wait for someone to fat-finger axios. Sounds trivial, but it works constantly because developers install packages at high velocity and rarely double-check spelling.

Dependency confusion. If your company has an internal package called corp-auth, an attacker publishes a public package with the same name and a higher version number. Many package managers default to “highest version wins,” and your build pulls the public one instead of your internal one.

Maintainer hijacking. Compromise a real maintainer through phishing, credential stuffing, or a missing 2FA setup, and push a poisoned update to a package that already has millions of weekly downloads. The Axios compromise in March 2026 followed exactly this pattern. The XZ Utils backdoor was a slower variant. The attacker spent months building trust as a “helpful” co-maintainer before slipping a backdoor into the build.

The thing that makes all of this so effective is the automation downstream. Unpinned versions, auto-merging update bots, transitive dependencies five layers deep. Once a malicious version hits the registry, it propagates fast.

Why NPM Is the Highest-Stakes Ecosystem

NPM serves tens of billions of downloads a week. A typical JavaScript project today pulls in well over a thousand transitive dependencies. Ten years ago that number was in the dozens. The dependency graph is just structurally enormous, and it’s getting worse.

The specific architectural problem in NPM is the lifecycle script, specifically postinstall. NPM lets package authors define scripts in package.json that run automatically when the package is installed. This was designed for legitimate reasons: compiling native bindings, configuring environments. But it also means arbitrary shell commands execute on your machine the moment you type npm install. No code review. No second thought. Just immediate execution as your user.

There are a few practical mitigations, and they’re worth knowing whether you’re a solo developer or running platform security at a large org.

Disable lifecycle scripts. Either pass --ignore-scripts ad hoc, or set it globally:

npm config set ignore-scripts true

This breaks some legitimate packages (esbuild, bcrypt, anything compiling native code). To manage that, tools like can-i-ignore-scripts scan your node_modules and generate an allowlist of packages that genuinely need scripts to run. Frameworks like @lavamoat/allow-scripts formalize this with a deterministic config you can check into the repo.

Use npm ci in CI, not npm install. This is non-negotiable for production builds. npm install will happily resolve newer minor versions inside your semver ranges and rewrite package-lock.json. npm ci refuses to do that. If the lockfile doesn’t match exactly, the install fails. That’s the behavior you want when the question is “did anything change that I didn’t approve.”

Consider switching to pnpm. pnpm 10+ has been quietly building some of the best structural defenses in the ecosystem. Postinstall scripts are off by default and require an explicit allowBuilds list. blockExoticSubdeps prevents transitive deps from resolving via random Git URLs or tarballs.

The killer feature, though, is minimumReleaseAge. As of pnpm v11 (May 2026), the default is 1440 minutes, so pnpm simply refuses to resolve any package version less than 24 hours old. Most malicious packages get pulled from the registry within hours of being detected. A 24-hour cooldown turns the community into your early warning system, with no behavioral analysis or commercial tooling needed.

That last one is the single highest-leverage change you can make as an individual developer. It costs nothing, it doesn’t break your workflow, and it neutralizes most day-zero registry malware before it ever reaches you.

Next post I’ll dig into the Python and Rust sides of this. pip’s setup.py execution problem, Rust’s build.rs issue, and the surprisingly mature auditing toolchain the Rust community has built around cargo-audit, cargo-deny, and cargo-vet.

Sources

Securing Package Managers: Why NPM, PyPI, and Cargo Are High-Value Targets

Defending Against NPM Supply Chain Attacks: A Practical Guide

NPM Ignore Scripts Best Practices

Mitigating supply chain attacks

Get safe and remain productive with can-i-ignore-scripts

The Landscape of Malicious Open Source Packages: 2025 Mid-Year Threat Report

The Evolving Software Supply Chain Attack Surface

Introducing OpenSSF’s Malicious Packages Repository

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

Five Modern JavaScript Features That Make the Old Patterns Look Silly

I’ve been doing some reading on what JavaScript has been picking up over the last few releases, and the current batch is unusually good. Cleaner resource management, real Set math, lazy iterators, and a couple of small ergonomic wins that retire some genuinely tedious patterns. So this post is my attempt at summarizing five of the more interesting ones, what they replace, and where each one stands on browser support. I hope it helps if you’re trying to figure out what’s actually shipping versus what’s still a proposal.

1. Explicit Resource Management with using

If you’ve written C# or Python, this will feel familiar. The using keyword (and its async sibling await using) ensures a resource is cleaned up the moment the variable goes out of scope, even if your code throws. Under the hood it looks for Symbol.dispose or Symbol.asyncDispose on the object.

The old way meant remembering to wrap everything in try/finally:

async function fetchUser() { const db = new DatabaseConnection(); await db.connect(); try { return await db.query('SELECT * FROM users WHERE id = 1'); } finally { await db.close(); } }

The new way:

async function fetchUser() { await using db = new DatabaseConnection(); await db.connect(); return await db.query('SELECT * FROM users WHERE id = 1'); }

No finally, no forgetting to close the connection on the error path. The cleanup is guaranteed.

Browser/runtime support: Chrome 123+, Firefox 119+, Node 20.9+. Safari is still pending.

2. New Set Methods

For years, JavaScript’s Set was basically a deduplicated array with a fancy name. If you wanted actual set math, you were converting to arrays and looping. Now the operations are built in and run at engine speed.

const userRoles = new Set(['read', 'write', 'comment']); const adminRoles = new Set(['read', 'write', 'delete', 'ban', 'comment']); userRoles.intersection(adminRoles); // shared roles adminRoles.difference(userRoles); // what admin has that user doesn't userRoles.union(adminRoles); // everything, deduped userRoles.isSubsetOf(adminRoles); // true

That’s it. That’s the whole job. No more new Set([...a].filter(x => b.has(x))) incantations. The full method set also includes symmetricDifference, isSupersetOf, and isDisjointFrom.

These shipped as part of ES2024 and have reached Baseline. Available in Chrome 122+, Safari 17+, and recent Firefox.

3. Iterator Helpers

This one is genuinely a big deal. Until now, .map() and .filter() only worked on arrays, and arrays load everything into memory. If you’re streaming a 50GB log file through a generator, calling Array.from() on it will introduce you to your operating system’s OOM killer.

Iterator helpers bring those same methods to iterators, operating lazily, one item at a time.

The old way:

function* infiniteNumbers() { let i = 1; while (true) yield i++; } const evens = []; for (const num of infiniteNumbers()) { if (num % 2 === 0) { evens.push(num); if (evens.length === 3) break; } }

The new way:

const result = infiniteNumbers() .filter(n => n % 2 === 0) .take(3) .toArray(); // [2, 4, 6]

It only computes what take(3) needs. You can chain on an infinite sequence and it just works.

These are part of ES2025. Firefox has shipped them, Chrome is in the process of shipping in V8, and Safari’s implementation is roughly half done.

4. Map Upsert

The naming bounced around (early proposals called it emplace, then upsert), but the final landing is getOrInsert(key, default) and getOrInsertComputed(key, callback). The idea is simple: stop doing the three-step “check, default, fetch” dance every time you group data.

The old way:

const wordMap = new Map(); for (const word of words) { const key = word[0]; if (!wordMap.has(key)) { wordMap.set(key, []); } wordMap.get(key).push(word); }

The new way:

const wordMap = new Map(); for (const word of words) { wordMap.getOrInsert(word[0], []).push(word); }

This is the kind of thing every codebase has a groupBy helper for. The proposal reached Stage 4 in January 2026, so it’s officially in the spec, but engine implementations are still in progress as of this writing. Worth knowing about, not yet safe to ship without a polyfill.

5. Import Attributes

As ES Modules took over, importing JSON natively became a real need. The catch is that just letting import pull in a .json file is a security problem. If the server quietly serves JavaScript instead of JSON, the engine would happily execute it as code.

Import attributes fix that by making you declare the type explicitly. If the file isn’t what you said it was, the engine refuses.

import config from './config.json' with { type: 'json' }; console.log(config.databaseHost);

No more fs.readFileSync for config, no more require hacks in otherwise-modern codebases. Just an import that’s safe by default.

If you’ve seen the assert { type: 'json' } form in older articles, that was an earlier syntax that got renamed before shipping. The current keyword is with. Available in Chrome, Edge, Firefox, and Safari since April 2025, plus Node and Deno.

The Through-Line

What stands out across these five is that each one retires a pattern that’s been written into JavaScript codebases millions of times. The try/finally cleanup. The custom groupBy helper. The Lodash imports for set operations. The for loop with a manual counter because there was no .take() on generators. The fs.readFileSync for loading a config file in an otherwise-modern ESM project.

The language is quietly absorbing the utility belt, and the code that’s left looks a lot more like what we meant to write in the first place. Sign me up.

Sources

Explicit Resource Management — V8

JavaScript Set methods reach Baseline — web.dev

Iterator helpers — V8

Map.prototype.getOrInsert — MDN

Import attributes — MDN

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

Three CSS Features That Finally Let Us Delete the JavaScript

Every few years, CSS quietly absorbs something we used to need a JavaScript library for. This year, three big ones are landing at once: masonry layouts, scroll-driven animations, and styleable <select> elements. All three have a long history of clunky workarounds (Masonry.js, scroll event listeners, div-based fake dropdowns) and watching them become native CSS features is genuinely surprising progress for a single year.

Here’s what each one does.

Native Masonry Layouts

Pinterest-style grids, where items of varying heights pack together without awkward vertical gaps, used to require Masonry.js or one of its cousins. Those libraries measure every element, calculate positions, and use absolute positioning to slot things in. The result works but is expensive, fragile on resize, and hard to make truly responsive.

The native version is display: grid-lanes:

.masonry-grid { display: grid-lanes; grid-template-columns: repeat(auto-fill, minmax(200px, 1fr)); gap: 16px; }

Set up your columns the way you would for any responsive grid, switch the display value to grid-lanes, and the browser handles packing. There’s also a flow-tolerance property that controls how aggressively items can shift to fill gaps.

The naming has a history. The CSS Working Group debated for years between an earlier grid-template-rows: masonry proposal (which would have layered onto existing Grid) and a separate display: masonry value. The resolution, in early 2026, was a third option: a new grid-lanes display value that’s distinct from Grid but borrows its column/row sizing. So if you read older articles showing grid-template-rows: masonry, that syntax did not ship.

Safari was first to ship grid-lanes (Safari 26). Chrome and Firefox have it behind experimental flags as of early 2026.

Scroll-Driven Animations

Reading progress bars, fade-ins as elements enter the viewport, parallax effects. All of these used to require listening to the window’s scroll event and updating styles in JavaScript. Scroll events fire dozens of times per second, and if your main thread is busy doing literally anything else, the animation stutters. It’s the most common source of jank on the modern web.

animation-timeline: scroll() ties a standard CSS keyframe animation to scroll position instead of a time duration. The browser runs it on the compositor thread, completely independent of your JavaScript. Buttery smooth, even when the main thread is on fire.

Here’s a reading progress bar in pure CSS:

@keyframes grow-progress { from { transform: scaleX(0); } to { transform: scaleX(1); } } .progress-bar { position: fixed; top: 0; left: 0; width: 100%; height: 8px; background: linear-gradient(to right, #ff416c, #ff4b2b); transform-origin: 0 50%; animation: grow-progress linear; animation-timeline: scroll(root block); }

No event listeners. No requestAnimationFrame. No throttling logic. The animation is just bound to scroll position and the browser figures out the rest.

Customizable <select>

This is the one with the longest backstory. The native <select> element is rendered by the operating system, which means whatever the OS decides to give you is what you get. The dropdown picker, option styling, hover states inside the list, none of it has been styleable from CSS.

The workaround was div-based fake dropdowns built with JavaScript. Most of them broke keyboard navigation, screen reader support, or both. Accessibility regression in exchange for visual polish, every time.

appearance: base-select opts you out of the OS-rendered version and into a fully styleable structure with new pseudo-elements for the popup. To opt the popup itself in, you also apply appearance: base-select to ::picker(select):

.custom-dropdown { appearance: base-select; padding: 10px; border-radius: 8px; border: 1px solid #ccc; } .custom-dropdown::picker(select) { appearance: base-select; background-color: #1e1e1e; border-radius: 12px; box-shadow: 0 4px 12px rgba(0,0,0,0.2); padding: 8px; } .custom-dropdown option:hover { background-color: #333; cursor: pointer; }

The ::picker(select) pseudo-element is the headline feature. The popup container is the thing every custom dropdown library was reinventing from scratch, and now it’s a styleable part of a real <select>, so native keyboard and accessibility behavior comes along for free.

Browser support is Chromium-only as of early 2026 (Chrome, Edge, Opera 135+). Firefox and Safari haven’t shipped it yet.

A Note on Adoption

Support varies by feature. Scroll-driven animations are the most settled, with stable support in Chrome and Safari and a flag-gated implementation in Firefox. display: grid-lanes shipped first in Safari 26 and is behind flags in Chrome and Firefox. appearance: base-select is Chromium-only, with no Firefox or Safari implementation yet. For all three, a @supports block with a sane fallback is the right pattern. Check caniuse.com before you ship.

But the direction is clear. Three of the most common reasons we used to pull in JavaScript are becoming native CSS features. Better performance, less code, and accessibility you don’t have to rebuild from scratch.

I’ll take that trade.

Sources

Introducing CSS Grid Lanes — WebKit

Masonry layout — MDN

CSS scroll-driven animations — MDN

The <select> element can now be customized with CSS — Chrome for Developers

Customizable Select Element explainer — Open UI

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

CSS Finally Got Inline Conditionals

I’ve been digging into some of the newer features landing in CSS, and inline conditionals with the if() function jumped out as one of the more interesting ones. The idea of writing a condition right next to the property it affects, instead of opening a separate @media block ten lines away, sounds useful. So this post is my attempt at summarizing what I found. I hope it helps if you’re trying to figure out what if() actually does and where it fits.

What It Actually Does

The shape is condition, colon, value, with an optional else branch:

property: if(<condition>: <value>; else: <fallback>;);

You evaluate something, you get back a value, and the property uses that value. The else: branch is optional. If you leave it off and the condition fails, the property falls back to its inherited or initial value.

That’s it.

What makes it powerful is what you can put in that condition slot. Three kinds of queries are supported: media() for viewport and device conditions, style() for custom property values on the element, and supports() for feature detection. The same things you’ve always been able to ask CSS, but now you can ask them inline, right next to the property they affect.

Theming Without the Yo-Yo

Dark mode is the most obvious win.

.card { background-color: white; color: black; } @media (prefers-color-scheme: dark) { .card { background-color: #333; color: white; } }

And here’s the same thing with if():

.card { background-color: if(media(prefers-color-scheme: dark): #333; else: white;); color: if(media(prefers-color-scheme: dark): white; else: black;); }

The logic lives next to the property. You don’t have to hunt down a separate block to figure out what happens in dark mode. When you change the light color, the dark color is right there staring at you. That’s a real maintenance win, especially in a stylesheet that’s been touched by five different people.

Variants Without Modifier Classes

This is the example that makes the case best. Component libraries are drowning in modifier classes. .btn-primary, .btn-danger, .btn-success, on and on. Every one of them is just a different background color and maybe a border.

With if() and a custom property, you can collapse the whole thing:

.button { padding: 10px 20px; border-radius: 6px; color: white; background-color: if( style(--variant: danger): red; style(--variant: success): green; else: blue; ); }

<button class="button">Submit</button> <button class="button" style="--variant: danger;">Delete</button>

No JavaScript reading props and toggling class names. No BEM modifier soup. You set a variable, the CSS reacts. The component owns its own logic.

Feature Detection Inline

The same pattern works for graceful degradation when you want a newer feature with a fallback:

.element { display: if(supports(display: grid): grid; else: flex;); color: if( supports(color: lch(75% 0 0)): lch(75% 0 0); else: rgb(185 185 185); ); }

This used to require a dedicated @supports block with the property repeated inside. Now it doesn’t.

Observations

A few things are happening here that go deeper than syntactic sugar.

The first is bloat reduction. A component that used to need a base class plus four modifier classes plus a media query block can be one selector. Multiply that across a design system and the savings get real.

The second is encapsulation. A component’s stylesheet can carry its own logic without relying on a JavaScript framework to compute the right class name and shove it into the DOM. The CSS engine is doing the work, natively, and it’s a lot faster at this than your render loop is.

If the logic lives where the value lives, you don’t context-switch to figure out why a property has the value it does. You read the property, you read the condition, you understand it, you move on.

That’s the same reason inline error handling beats a separate try/catch block ten lines away. Co-location is a maintenance superpower.

Browser Support

Browser support is still narrow. Chrome shipped if() in version 137, but Firefox and Safari haven’t followed yet, so this isn’t something to drop into a production stylesheet without a fallback. The pattern is to declare a plain value first and then override with if(), so non-supporting browsers ignore the line they don’t understand:

.card { background-color: white; background-color: if(media(prefers-color-scheme: dark): #333; else: white;); }

The syntax has also shifted through CSS Working Group drafts, so older articles you find about if() may show a comma-ternary form that no longer works. Check MDN before copying anything.

But the direction is right. CSS has been quietly absorbing more and more of the work we used to push to JavaScript, and if() is one of the bigger steps in that direction. I’ll probably keep poking at it.

Sources

if() CSS function — MDN

CSS conditionals with the new if() function — Chrome for Developers

Lightly Poking at the CSS if() Function in Chrome 137 — CSS-Tricks

CSS if() function — Can I use

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

JavaScript Finally Gets a Real Date API

When has working with dates ever been easy? Every language has its own version of the same headaches: time zones, parsing, leap years, arithmetic that does weird things at month boundaries. JavaScript just had some quirks layered on top, extra cruft left over from when the language was first created. Third party libraries like Moment.js or date-fns had to fill in the gaps.

Those days are over. Now we have the Temporal API.

Why Date Was Broken

Let me give you the short version of why Date is the way it is. It was inspired by Java’s java.util.Date from the 90s, which Java itself eventually deprecated. JavaScript inherited the design and never let it go.

The problems are well-known at this point:

Mutability. Pass a Date to a function and that function can change it underneath you.

const myDate = new Date('2023-01-01'); function addDays(date, days) { date.setDate(date.getDate() + days); return date; } addDays(myDate, 5); console.log(myDate.toISOString().slice(0, 10)); // '2023-01-06'. Surprise, your original is gone.

Time zone confusion. Date stores milliseconds since the Unix epoch but formats itself in the user’s local time zone. Working in any other zone means reaching for moment-timezone or date-fns-tz.

Parsing roulette. new Date("2023-01-01") and new Date("Jan 1, 2023") can return different things depending on the browser and the assumed time zone.

Math that lies. Adding a month to January 31st? Date rolls it forward to March 3rd because February doesn’t have 31 days. That’s not a bug exactly. It’s just Date being honest that it doesn’t really understand calendars.

What Temporal Actually Fixes

Temporal is a new global object designed from the ground up to address all of this. The design choices are worth walking through because they’re opinionated in the right ways.

Everything is immutable

Every operation returns a new object. Your original data stays put.

const start = Temporal.PlainDate.from('2023-01-01'); const end = start.add({ days: 5 }); console.log(start.toString()); // '2023-01-01' console.log(end.toString()); // '2023-01-06'

Different types for different concepts

This is the part I find most interesting. Date tries to be everything, a timestamp, a calendar date, a wall clock time, all at once. Temporal splits these into distinct types and forces you to pick:

Temporal.PlainDate: a calendar date, no time, no zone. Birthdays, anniversaries.

Temporal.PlainTime: a wall-clock time, no date.

Temporal.PlainDateTime: date and time, no zone.

Temporal.ZonedDateTime: fully zone-aware and calendar-aware. The one for global apps.

Temporal.Instant: an exact point in time, like epoch milliseconds.

Temporal.Duration: a length of time.

Making you pick the right type up front is the whole game. Half the bugs in date code come from pretending a Date is one thing when it’s actually another.

Time zones and calendars built in

Temporal natively understands IANA time zones (America/New_York, Europe/Paris) and non-Gregorian calendars (Hebrew, Islamic, Japanese). No external library needed.

Math that respects the calendar

const t = Temporal.PlainDate.from('2023-01-31'); const nextMonth = t.add({ months: 1 }); console.log(nextMonth.toString()); // '2023-02-28'

It clamps to the end of the month instead of rolling over. That’s almost always what you actually wanted.

Comparisons and Diffs

A couple of quick ones, because these are the operations you do constantly.

Comparing two dates:

const t1 = Temporal.PlainDate.from('2023-01-01'); const t2 = Temporal.PlainDate.from('2023-01-01'); console.log(Temporal.PlainDate.compare(t1, t2) === 0); // true

No more getTime() dance to compare primitives. There’s an actual comparison function.

Finding the difference:

const start = Temporal.PlainDate.from('2023-01-01'); const end = Temporal.PlainDate.from('2023-12-31'); const diff = start.until(end, { largestUnit: 'days' }); console.log(diff.days); // 364

No more dividing milliseconds by 1000 * 60 * 60 * 24 and hoping DST doesn’t mess you up.

Should You Use It Yet?

Check your runtime. Browser and Node support has been landing, but you’ll want to verify Temporal is available where you’re shipping, or use the official polyfill while you wait.

For most date and time work, this replaces Moment.js and date-fns entirely. Moment has been in maintenance mode for years. Temporal gives you the good parts of those libraries as a standard, immutable, well-typed API.

Date will stick around forever for backwards compatibility. But for new code, use Temporal. The API is better, the semantics are saner, and less bug-prone.

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

Semantic Docs Spring Update: Astro 6, Auto-Releases, npm

The last two months on Semantic Docs have mostly been maintenance work, but a few things I wanted to talk about. I pushed through a major framework upgrade, swapped out a vendored library for a real published package, and finally automated the release pipeline. Five tagged releases later, here’s where we are.

The Headlines

Upgraded to Astro 6

Switched from a vendored logger to the published logan-logger npm package

Shipped an auto-release workflow driven by Conventional Commits

Three rounds of dependency updates plus a security-focused sweep

Five tagged releases, v1.3.3 through v1.5.0

Astro 6

The Astro 6 upgrade was easy. Semantic Docs runs a hybrid setup, static article pages plus a server-rendered search endpoint, and that part barely needed any attention. Most of the work was in the dependency layout, not the application code.

One note if you’re forking or syncing this theme: if you’re upgrading from v1.3.5 or earlier (anything pre-Astro-6, which landed in v1.4.0), delete your node_modules and your lockfile and do a clean install. Skip that step and you’ll get weird errors that look like your code is broken when it’s really just leftover state.

A Real npm Package Instead of a Vendored Logger

For a while, the project was using a logger I wrote to experiment with publishing to both npm and JSR. It was a useful exercise. I wanted to see what a clean foundational package looked like across both registries, and I think it turned out well.

But for this repo, I wanted consistency over experimentation. So I swapped the vendored copy for the published logan-logger npm package. Behavior is the same, the surface area is the same, it’s just back on the npm registry.

Automated Releases

I’ve liked using Conventional Commits to drive automated releases. When a PR merges to main, the workflow figures out the next version from the commit messages, tags it, and publishes a GitHub release with a generated changelog.

The commit type determines the version bump. feat: bumps the minor, fix: bumps the patch, breaking changes bump the major. The changelog falls out of the same metadata. More automation here the better.

If you’ve been on the fence about Conventional Commits, this is the use case that sold me.

What’s Next: Embedding Quality

The reference implementation uses TEI for search embeddings, and that’s been fine. But “fine” is not the same as “good,” and I want to actually compare quality across providers before I commit to anything long term.

Two I want to test:

Jina (now owned by Elastic)

Mistral, which has been putting out genuinely strong embedding models

The goal is to run the same corpus through each, evaluate the search results, and figure out which one earns a highlight. Whatever I learn from that work will get folded back into the open source Semantic Docs repo so anyone running their own instance can make an informed choice instead of just trusting my defaults.

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

SAST vs AI PR Review: Two Tools, Different Jobs

If you have worked in DevSecOps, you might be wondering if AI pull request review tools are going to replace traditional SAST scanners. Short answer: no. Longer answer: they’re solving different problems, and if you’re picking one over the other, you might be making a mistake.

Here is how I think about it.

SAST is the Compliance Gatekeeper

Static Application Security Testing tools, think Semgrep, SonarQube, Checkmarx, Fortify, parse your source code (usually into an Abstract Syntax Tree) and hunt for known vulnerability patterns. They don’t run the code. They just read it and “pattern-match” against rules.

The focus here is security, compliance, and strict rule enforcement. SAST is the automated gatekeeper that makes sure your code clears the OWASP Top 10 bar before it merges.

What SAST does well:

It’s deterministic. If a rule matches a pattern, the engine flags it every single time. Run it twice on the same code, get the same result.

It satisfies auditors. Frameworks like PCI-DSS, SOC 2, and HIPAA expect documented secure-development practices, and a formal SAST scanner is the easiest way to produce that evidence. AI agents don’t count here, at least not yet.

It can do real taint analysis. Enterprise tools can track untrusted input from the moment it enters your app to the moment it hits a dangerous sink.

Where SAST falls down:

The false positive rate is brutal. Rigid rules with no context means a lot of noise. Developer fatigue is real, and once your team starts ignoring scanner output, you’ve lost the game.

It can’t see your business logic. A SAST tool has no idea what your application is supposed to do, so it can’t tell you when the logic itself is broken.

Comprehensive scans are slow. Hours on large codebases isn’t unusual, though Semgrep has been doing good work on this front.

AI PR Agents are the Peer Reviewer

Tools like CodeRabbit, Qodo, Greptile, GitHub Copilot Code Review, Cursor Bugbot, and Claude Code (set up as a review skill) plug into your version control and read the PR diff with the surrounding code context. They behave less like a scanner and more like a colleague who actually read your changes.

The focus is developer productivity, code quality, logic bugs, and contextual feedback.

What they do well:

They understand intent. LLMs can reason about why the code is changing, not just whether it matches a rule. That’s a different category of feedback.

The signal-to-noise ratio is good. When an AI flags something, it usually comes with an explanation that makes sense. Less noise, more useful comments.

They suggest fixes. Not just “this is wrong” but “here’s a diff you can apply.” That’s huge for actually closing the loop on review feedback.

The scope is broader. Architecture, performance, style, security, all in one pass.

Where they fall down:

They’re non-deterministic. Same vulnerability, two PRs, two different outcomes. That’s not a bug, that’s how LLMs work, and it’s why auditors don’t trust them.

They don’t satisfy compliance. No auditor is going to accept “the AI looked at it” as a substitute for a formal scanner.

Hallucinations happen. Invented issues, misread intent, suggestions that refactor things that didn’t need refactoring. You still need a human filtering the output.

The Quick Comparison

Feature SAST AI PR Review Primary Goal Security & Compliance Code Quality & Productivity Analysis Method Deterministic rules & AST Non-deterministic LLMs Business Logic Blind Context-aware False Positives Often high Usually low Compliance Proof Accepted as evidence Not accepted Feedback Loop Dashboard / CI output PR comments / chat

The Lines Are Starting to Blur

The interesting thing happening right now is convergence from both directions.

On the SAST side, tools like DryRun Security are pitching themselves as “AI-native SAST,” trying to keep the deterministic backbone while using LLMs to filter out the false positives that make traditional scanners painful to live with.

On the AI agent side, CodeRabbit and Greptile keep getting better at catching real security vulnerabilities, not just style issues. They’re slowly creeping into territory that used to belong exclusively to SAST.

This is going somewhere, but it’s not there yet.

Where to Start Your Evaluation

Treat them as complementary, not competitive.

For SAST, evaluate against your audit footprint, the languages in your codebase, and how much false-positive triage your team can absorb. Semgrep, SonarQube, Checkmarx, and Fortify all sit in different price-and-friction zones, and the right one depends on what your business actually needs to prove.

For AI PR review, evaluate based on how it fits your existing review workflow, what languages and frameworks it understands well, and the signal-to-noise ratio in practice on your codebase. CodeRabbit, Qodo, Greptile, Copilot Code Review, Bugbot, and a Claude Code review skill all approach the problem differently.

If you pick one category and skip the other, you’re either passing compliance with mediocre code review, or getting great review feedback while failing your next audit. Neither is a win.

The AI tools aren’t replacing SAST. They’re filling in the gap SAST was never designed to cover.

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

npmx.dev Is the NPM Frontend We've Been Asking For

If you’ve spent any time on npmjs.com, you know the drill. You land on a package page, eyeball the tarball size, squint at the dependency list, then bounce out to bundlephobia, then to Are The Types Wrong, then to Socket.dev, just to figure out if this thing is safe to install. That’s not a workflow. That’s a scavenger hunt.

For years, the JavaScript community has been filing requests on the official npm tracker asking for this stuff to live in one place. As Andrew Nesbitt notes, native dark mode was the single most upvoted request on the tracker for something like five years before it shipped. Real install sizes (transitive, not just the tarball). UI warnings about compromised packages and hidden postinstall hooks. Concrete version resolution instead of squinting at semver ranges. The list is long, and it mostly went nowhere.

So the community built npmx.dev instead.

What It Actually Is

It’s important to note: npmx.dev is not a separate registry. It doesn’t mirror packages, and npm install still pulls from the official Microsoft/GitHub-owned registry. What npmx is, strictly, is an alternative frontend. It hits the official npm APIs in real time, caches the results at the edge, and renders a much better page.

The stack is Nuxt on top of the full VoidZero toolchain with Vite, Vitest, Rolldown, oxlint, oxfmt, and the other usual packages you’d expect. It also leans heavily on SSR and ISR for near-instant page loads. VoidZero (the company behind Vite) sponsors the project, which makes sense, since it’s open-source MIT and the operational costs are basically web traffic and compute, not package storage.

What You Actually Get

I really like the dependency view. Instead of a flat list of what the package declares, you get an expandable tree with recursive vulnerability tracking via OSV. You can drill into transitive dependencies and see which ones are flagged. That’s the thing every security-conscious developer has been doing manually for years.

A few other wins worth calling out:

Real install size. Not the tarball. The actual disk footprint with all dependencies resolved.

postinstall scripts surfaced. If a package is going to run arbitrary code on your machine at install time, npmx puts that on the page where you can see it. Not buried in the manifest.

Concrete resolved versions displayed alongside the semver range, so you know what you’re actually getting.

Banners suggesting lighter alternatives for bloated legacy packages. Opinionated, and I’m here for it.

A Quick Note on OSV

The vulnerability data comes from the OSV ecosystem, and it’s worth understanding what that is, because it’s one of the more important pieces of open-source infrastructure right now.

OSV is a centralized aggregator that pulls from GitHub Security Advisories, PyPA, RustSec, the Global Security Database, and many ecosystem-specific feeds. The schema is standardized JSON, machine-readable, and maps vulnerabilities to exact versions or commit hashes. That matters because it’s what lets automated scanners avoid the false-positive flood that makes most security tooling annoying to use.

It’s an OpenSSF project under the Linux Foundation. Google maintains the infrastructure; GitHub, the Rust Foundation, the PyPA, Red Hat, and a long list of ecosystem maintainers contribute data. Vendor-neutral, free, and the boring kind of governance you want for security data.

Making It Your Default

Here is how you can make it the actual default npm frontend. Since npmx.dev maintains URL parity with npmjs.com, you can redirect every npm link you click without thinking about it. A couple of options depending on your setup.

A few options:

Kagi users can add a global URL redirect rule:

^https://www.npmjs.com|https://npmx.dev

Click an npm link in your search results, land on npmx instead.

Browser extensions work for everyone else. There’s a dedicated npmx-redirect extension for Chrome and Firefox that does only this one thing. Or use Redirector with a regex rule:

Include: ^(?:https?://)?(?:www\.)?npmjs\.com/(.*)

Redirect to: https://npmx.dev/$1

Custom site search. In Chrome, Brave, Arc, or Safari, add a new site search with shortcut nx (or @npm) pointing at:

https://npmx.dev/search?q=%s

Now you type nx, space, zod, enter. Done. You never see npmjs.com again unless you want to.

Why This Matters

The npm registry is critical infrastructure for, conservatively, a huge chunk of the software running on the internet. The fact that the community had to build its own frontend to surface basic security and observability data tells you something about where the priorities have been.

I’ll be using npmx as my default going forward. I’m not going back.

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

Sources

Andrew Nesbitt, Features everyone should steal from npmx

VoidZero, VoidZero and npmx: Building Better Tools Together

OSV.dev data sources

Red Hat × OSV.dev collaboration (OpenSSF)

Your Data Lake's Vulnerability Problem Is Really an Identity Problem

I’ve been reading through the post-mortems on the last few years of data lake breaches, and the pattern is depressing. We keep blaming the platforms. We should be blaming ourselves.

Let me give you an example.

The Snowflake Breach Wasn’t a Snowflake Breach

In mid-2024, at least 165 organizations got hit through their Snowflake instances. AT&T lost over 50 billion call records. Ticketmaster, Santander, Advance Auto Parts. The headlines wrote themselves: Snowflake hacked.

Except Snowflake wasn’t hacked. Mandiant, CrowdStrike, and Snowflake all reached the same conclusion in their forensics. No zero-day. No flaw in the cryptographic platform. No internal compromise of Snowflake’s corporate network. No brute-force attacks against API limits.

What actually happened? UNC5537, a financially motivated group also tracked as Scattered Spider and ShinyHunters, walked through the front door with valid stolen credentials. Those credentials were harvested over years by commodity infostealer malware (VIDAR, LUMMA, REDLINE) running on the personal laptops of third-party contractors. The same laptops these contractors used for gaming and pirated software also held the keys to their clients' enterprise data lakes.

One contractor laptop. Multiple enterprise environments compromised. That’s the actual story.

79.7% of the accounts UNC5537 used had prior credential exposure. Some had been valid and un-rotated since November 2020.

The Two Doors They Walked Through

The first attack vector was the SSO side door. Plenty of victim organizations had a perfectly fine enterprise IdP enforcing strong passwords and MFA. They just forgot to make SSO mandatory. A local authentication pathway was left active alongside it. Attackers logged in directly with stolen local credentials, completely bypassing the IdP, and the MFA requirement never fired.

The second was credential stuffing against inactive, orphaned, and demo accounts belonging to former employees. Nobody audits those. Nobody enforces MFA on those. So they don’t get protected by the controls that exist on the production accounts.

Once inside, the kill chain was almost boring. SHOW TABLES to enumerate. CREATE TEMPORARY STAGE to make an ephemeral staging area that disappears when the session ends, erasing forensic evidence. COPY INTO with GZIP compression to keep the payload small enough that volumetric alarms didn’t trigger. GET to pull it down to a VPS in some offshore jurisdiction. Done.

No IP allowlisting was in place anywhere. The connections from Mullvad and PIA exit nodes were treated with the same trust as an employee on the corporate VPN.

The Bucket Problem Hasn’t Gone Away Either

Alongside the identity attacks, the boring stuff keeps working. Misconfigured S3 buckets are still the most reliable way to expose a data lake. In late 2024, an open bucket used as a shared network drive was found containing raw customer data, cryptographic keys, and secrets. In 2025, a US healthcare provider left millions of patient records readable for weeks before anyone noticed.

Then there’s Codefinger. In January 2025, that group used compromised AWS credentials to access S3 buckets and then weaponized AWS’s own Server-Side Encryption with Customer-Provided Keys (SSE-C) to ransomware the data in place. They didn’t even need to exfiltrate it. They just encrypted it with a key the victim didn’t have and demanded Bitcoin.

That’s a native cloud feature being turned against you because somebody granted too many permissions to a service account.

The Boring Conclusions Are the Important Ones

Identity is the perimeter now. The encryption-at-rest story we’ve been telling ourselves for a decade is irrelevant when the attacker authenticates as a real user. Stop treating SSO as optional. Stop leaving local auth paths open next to it. Enforce MFA on every account, including the demo and service accounts you forgot about.

Your data lake should not be reachable from the public internet. Route everything through PrivateLink or the equivalent in your cloud. Allowlist the IPs that should be touching analytical workloads, and don’t make exceptions for “just this one contractor.”

And as you start handing access to AI agents, remember that static roles aren’t going to cut it. Just-in-time entitlements and contextual access control are the only way you’re going to keep up with autonomous systems making queries on your behalf.

The data lake industry spent years arguing about table formats, vendor lock-in, and egress fees. Meanwhile, attackers were just collecting passwords from gaming laptops and walking in.

Fix the doors first.

Sources

UNC5537 Targets Snowflake Customer Instances (Mandiant / Google Cloud) — Forensic analysis, kill chain, infostealer attribution

Snowflake Data Breach: Lessons Learned (AppOmni) — SSO side door, MFA bypass mechanics

Major AWS S3 Bucket Breach Exposes Data (NHIMG) — Codefinger SSE-C ransomware tactic

Misconfigured Cloud Assets: How Attackers Find Them (CybelAngel) — Recent open-bucket exposure incidents

5 Key Lessons from the Snowflake Data Breach (Tanium) — Defensive posture summary

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

Your Data Lake Has a Permissions Problem

Consolidating every business unit’s data into one giant lakehouse sounds like a win until you realize the security model from your old data warehouse can’t scale to it. You took ten silos, each with their own access rules, and merged them into one location. Now everyone wants in, and your security team is the bottleneck.

Let me walk through three places where the cracks usually show up.

RBAC Falls Over Faster Than You Think

Role-Based Access Control is the model most teams start with. Permissions are tied to a job function. Sales reps get read access to sales tables, data engineers get write access to staging, and so on. It works fine when you have ten roles.

It does not work when you have a thousand.

Say your sales reps should only see accounts in their territory, and only accounts they personally manage. Under pure RBAC, you need a unique role for every territory-by-account-owner combination. That’s role explosion, and it’s how compliance audits become impossible and legitimate access slows to a crawl. The roles list grows faster than anyone can review it, which means stale permissions sit there forever.

The answer is Attribute-Based Access Control. Instead of asking “what role is this user in,” the system asks “what attributes does this user have, what attributes does this data have, and what’s the policy at this exact moment.” Tag a column as PII. Tag a schema as HR. Write one policy that says anyone outside the HR compliance group sees masked data when they touch a PII column. Done. That single policy replaces hundreds of bespoke roles.

This is what Unity Catalog and Starburst Galaxy are built around, and it’s the model that will scale with the data.

Column and Row Security Should Be Boring

Once you have ABAC and a real metadata catalog, column-level masking and row-level filtering become a non-event. You write a SQL expression that masks the first five digits of an SSN for lower-privileged roles. You write a row filter that silently appends WHERE region = 'user_region' to every executive’s SELECT *.

The key word is silently. The user doesn’t see a different table. They don’t have a sanitized copy. The policy is enforced at the catalog layer, so it works the same whether they’re querying through Spark, Trino, a BI dashboard, or a pipeline. One source of truth, one policy, every engine.

If you’re still maintaining separate “sanitized” copies of tables for different audiences, you’re doing it the 2015 way and you’re going to drift.

The IAM Default Problem

Most cloud services ship with default IAM roles, and a surprising number of those defaults attach AmazonS3FullAccess or something equally permissive.

SageMaker does it. The Ray autoscaler role does it. There are more.

Picture the failure mode. An attacker compromises some peripheral app, maybe a forgotten Jupyter notebook, maybe a misconfigured Lambda. That workload has an IAM role attached because that’s how cloud workloads talk to S3 without hardcoded credentials. The attacker inherits the role. And because the role has full S3 access, they’re not constrained to the bucket the application actually uses. They can enumerate every bucket in the entire account.

That’s how a single compromised container becomes a full data lake breach. Researchers call it a bucket monopoly attack. I call it the most predictable incident in the industry.

The fix is not glamorous. Stop using s3:* in any policy. Write resource-scoped policies that name the exact buckets and prefixes a workload needs. Audit the default roles every cloud service hands you and replace them. Use Security Lake or Detective to flag cross-service API calls that don’t match normal patterns. None of this is fun. All of it is necessary.

And Then There’s the Agent Problem

The new wrinkle is that humans are no longer the primary consumers of your data. Autonomous agents are. They issue more queries, hit more tables, and move faster than any human team.

Long-lived credentials and static roles don’t fit that workload. The pattern emerging is Just-In-Time entitlements, where an agent gets a narrow, ephemeral permission for the duration of a single execution thread, then loses it. Pair that with declarative policy metadata baked into the data assets themselves, so the agent knows what it’s allowed to do with a dataset before it ever runs the query.

We’re early on this. Most organizations are still working through the basics, and that’s fine. But if you’re designing access controls today, design them assuming the next thing hitting your lake isn’t a person.

What to Actually Do

If you’re auditing your own data lake security, the order I’d work in:

Find every IAM role with a wildcard permission. Replace them.

Move from RBAC to ABAC at the catalog layer. Stop creating new roles.

Pull your data lake off the public internet. PrivateLink, private endpoints, IP allowlists for the legacy stuff that can’t move.

Then start thinking about agents.

The lakehouse pitch is unification. The lakehouse reality is that unification multiplies the cost of every bad permission. Get the basics right before you bolt on anything fancy.

Sources

AWS Default IAM Roles Found to Enable Lateral Movement (The Hacker News) — SageMaker / Ray autoscaler default roles, bucket monopoly attacks

What Is Fine-Grained Data Access Control? (TrustLogix) — RBAC role explosion, ABAC fundamentals

Core concepts for ABAC (Databricks Unity Catalog docs) — Tag-driven policy enforcement

Top 12 Data Governance Predictions for 2026 (Hyperight) — Just-in-time entitlements, declarative policy metadata

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

The Real Cost of Your Data Lake (It's Not the Storage)

If you’re sketching out a data platform on a whiteboard right now, I want you to do something. Stop calculating storage costs. They’re not the bill.

I pulled the public pricing for AWS, Azure, GCP, Databricks, and Snowflake and stacked them next to each other. Storage is the cheap part. The expensive part is everything that moves the data, and the expensive part is the part you’re least likely to model correctly when you’re picking a vendor.

Let me walk through what actually shows up on the invoice.

Raw Object Storage Is Basically Free

For hot, frequently accessed data, the big three are within a rounding error of each other:

Azure Blob (LRS, Hot): $0.018 per GB/month

Google Cloud Standard: $0.020 per GB/month

AWS S3 Standard: $0.023 per GB/month (first 50 TB)

Drop into the cool tiers and AWS S3 takes the lead at $0.0125 per GB. Drop into deep archive and you’re paying $0.00099 per GB on either AWS Glacier Deep Archive or Azure Archive. That’s a tenth of a cent per gigabyte, per month, for data you almost never touch.

Good for you, but I think anyone leading with “per-GB storage cost” in a procurement deck is selling you a story. Storage capacity is roughly five percent of a typical Databricks bill. Five. The other 95% is the part nobody wants to talk about.

The Egress Trap

Ingress is free. Always. The cloud providers want your data in.

Getting it back out is where they collect.

Azure Blob: $0.087/GB external egress

AWS S3: $0.090/GB

Google Cloud: $0.120/GB (but free if you stay inside Google’s ecosystem, which is the whole point of that pricing)

Then layer on API operations. A million GET requests on S3 costs about $0.40. The same million GETs on Google Cloud Storage can run closer to $5.00 because they classify operations differently. If your analytics workload is hammering small files, those API calls add up faster than the storage they’re reading.

Storing 10 TB? Maybe $200 a month. Storing 500 TB? You’re at $10,000 a month before a single byte leaves the region or a single query fires.

Databricks: Two Bills, One Headache

Databricks uses what’s commonly called a Two-Bill Model. You get one invoice from your cloud provider for the actual VMs and storage, and a separate invoice from Databricks for the software, measured in DBUs (Databricks Units).

In a typical mid-sized deployment around $18,000/month, the breakdown looks like this:

VM compute from the cloud provider: ~55%

Databricks DBU fees: ~30%

Storage: ~5%

Network egress: ~5%

The DBU rate changes based on what you’re doing. Automated jobs start at $0.15/DBU. Interactive notebooks for analysts start at $0.40/DBU. That’s not an accident. Databricks wants you running production workloads on cheap job clusters, not on the expensive all-purpose clusters your data scientists love to leave running over a weekend.

If you’re not actively pushing teams toward job clusters and ARM-based instances, you’re leaving real money on the table.

Snowflake: The Hidden Storage Multiplier

Snowflake’s pricing pitch sounds clean. Pass-through storage at $40/TB/month on-demand, dropping to $23/TB/month with a capacity commitment. Compute as Credits. Done.

Except it isn’t done. Snowflake stores data in immutable 16MB micro-partitions. Immutable. You can’t change them in place. Update a single row in a 1 TB table and Snowflake writes a new file and keeps the old one around.

Why keep the old one? Two features:

Time Travel: query historical states of your data for up to 90 days

Fail-Safe: a 7-day disaster recovery window you cannot turn off

This is the part that gets people. A 1 TB table that’s getting updated multiple times a day can balloon to 25 TB of billed storage because Snowflake is retaining every prior version of every micro-partition you’ve touched. Your dashboard says “1 TB table.” Your invoice says otherwise.

And compute? Virtual Warehouses bill per second, but with a 60-second minimum every single time you resume or resize. Aggressive auto-suspend sounds like a cost optimization. It’s not. If you’re spinning a warehouse up and down every 30 seconds, you’re paying the 60-second minimum every time and quietly multiplying your bill.

What I’d Actually Do

A few things I’d put on the wall before signing anything:

Model egress, not storage. Run your worst-case query pattern through the calculator. Storage is noise.

Lifecycle everything. Cool tier and archive pricing are 10x to 100x cheaper. If your data is older than 90 days and nobody’s queried it, it shouldn’t be in hot storage.

For Databricks: push every recurring workload to job compute. Audit interactive cluster usage monthly.

For Snowflake: if you have high-frequency update patterns, profile your actual storage footprint, not your logical table size. The gap will surprise you.

For multi-cloud: don’t. Egress will eat the savings before you finish the architecture diagram.

The vendors all have a story about why their model is the cheap one. Read past the per-GB number on the slide. The bill is somewhere else.

Happy modeling.

Sources

Databricks Pricing Explained (Dawiso) — Two-Bill Model, DBU breakdown

Snowflake Pricing Explained (SELECT.dev) — Time Travel storage multiplier, micro-partition behavior

Cloud & AI Storage Pricing Comparison 2026 (Finout) — AWS / Azure / GCP per-GB and tier pricing

S3 vs GCS vs Azure Blob Storage (ai-infra-link) — Egress and API operation pricing

Snowflake Pricing in 2026 (CloudZero) — Virtual Warehouse 60-second minimum behavior

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

Happy Star Wars day. Here is your reminder to finish Andor and/or Mandalorian. 😄

The Data Lakehouse Won. Now Pick a Table Format.

If you’ve been ignoring the data infrastructure conversation for the last few years, here’s where we landed in 2026: the data lakehouse won. The data warehouse vendors will fight about it for another decade, but the architectural argument is over.

Let me back up.

The Quick History

At the bottom of every modern data stack is a cloud storage bucket. S3, Azure Blob, GCS. Pick your hyperscaler. A bucket is dumb on purpose. It stores files cheaply and durably and doesn’t care what’s in them. No schemas, no transactions, no relational anything. Just objects.

When you dump raw logs, IoT telemetry, and CSV exports into a bucket without any organizing layer, congratulations, you have a data lake. Cheap, flexible, and almost completely useless for analytics until someone builds a pipeline to make sense of it.

The traditional answer to that mess was a data warehouse. Snowflake, Redshift, BigQuery, the whole gang. You force your data through ETL, conform it to a strict schema, and pay a premium to keep it sitting in the vendor’s proprietary storage format. You get fast SQL, ACID transactions, and a vendor lock-in problem so severe that exporting your data becomes a major friction point.

The lakehouse is what happens when someone finally says: what if we kept the cheap object storage, but added the warehouse features as a layer on top?

What a Lakehouse Actually Is

The trick is decoupling. Storage stays in your bucket. Compute is whatever engine you point at it. Metadata lives in an open table format that turns a pile of Parquet files into something that behaves like a real database table.

One copy of the data. Multiple engines can query it. Schema evolution, time travel, ACID transactions, all without copying everything into a proprietary system. From what I’ve read, teams that move from a pure warehouse to a lakehouse are able to cut storage costs noticeably in the process, and they stop fighting their ML team about getting access to the same data.

That’s the pitch, and it’s a good one. The hard part is picking your table format.

The Four Formats Worth Knowing

Apache Iceberg

Iceberg is the one to bet on if you care about not getting locked in. It came out of Netflix and even Snowflake and Databricks have been forced to support it. The metadata is hierarchical, which sounds boring but matters: it lets query engines skip enormous chunks of irrelevant data without listing directories one by one. Iceberg also handles partition evolution gracefully, so you can change your partitioning strategy without rewriting petabytes of history.

If I’m starting a new lakehouse in 2026 and I don’t have a strong reason to pick something else, it’s Iceberg.

Delta Lake

Delta is what Databricks ships and what everyone using Spark already knows. It uses an append-only transaction log in a _delta_log directory, and it’s beautifully integrated with the Databricks platform. Z-Ordering, native Spark performance, the whole ecosystem.

If your team lives inside Databricks, Delta is the obvious answer. If you don’t, the calculus is harder, because Delta’s openness has improved a lot but it still feels most at home in the Databricks world.

Apache Hudi

Hudi came out of Uber and it was built for one thing: high-frequency upserts. If your problem is Change Data Capture, streaming ingestion, or constant record-level updates, Hudi is probably your answer. It gives you two storage modes. Copy-on-Write rewrites files on update so reads stay fast. Merge-on-Read writes deltas and reconciles at query time, which is what you want when writes are heavy and reads are flexible.

Hudi is the right pick when your pipeline is full of UPSERT and you can’t afford to rewrite large files every time something changes.

Apache Paimon

Paimon is the newest of the four and it’s worth keeping an eye on. It came from the Flink world and uses an LSM-tree style organization, which is what databases like RocksDB use under the hood. The whole point is unifying batch and streaming in a single format. If you’re doing real-time event-driven work and don’t want to maintain a separate streaming and batch stack, Paimon is interesting.

It’s not the safe choice yet, but it’s the one I’d watch most closely over the next two years.

So Which One?

Honestly, the answer depends less on the format and more on which ecosystem you’re already in.

Mostly Spark and Databricks? Delta.

Streaming-heavy with constant upserts? Hudi.

Real-time event-driven and willing to bet on newer tech? Paimon.

Anything else, or you want to keep your options open? Iceberg.

The format wars have mostly converged. Most major engines support multiple formats now, and the gap between them on raw query performance has shrunk. The choice is more about operational fit than performance ceilings.

The lakehouse pattern itself is the real story. The format is just plumbing.

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

AI Code Reviewers Won't Save You

Dropping an AI reviewer into your pull request pipeline is just a band-aid. Tools like CodeRabbit or Greptile are great for catching syntax errors or basic anti-patterns, but they can’t assess architectural intent or domain-specific business logic. They’re spell-checkers for code. Useful, sure. But nobody ever said “our codebase is solid because we run spell check.”

AI doesn’t change your engineering baseline. It just accelerates it. If your foundational guardrails are weak, agentic tools will help your team generate technical debt at unprecedented speeds. So the real question isn’t “how do we review AI code?” It’s “how do we build systems that prevent slop from ever reaching production?”

Shift Left, Hard

When engineers use agents to scaffold a new Go service or spin up a SvelteKit frontend, they’re inevitably pulling in generated dependencies or utilizing unfamiliar libraries. Models hallucinate packages. They suggest insecure patterns with total confidence.

Your CI pipeline needs to be ruthless before a human ever looks at the code. Aggressive SAST and SCA should automatically block PRs that introduce vulnerable dependencies or hardcoded secrets. If the agent generates slop, the pipeline rejects it instantly. No discussion.

Make the Agents Write the Tests

Agents are incredibly eager to generate feature code, but humans are historically lazy about writing the tests for it. The influx of AI-generated code means human reviewers can’t possibly step through every logic branch manually.

So flip the script. Use the agentic tools to build the guardrails themselves. Mandate that any generated feature code must be accompanied by generated, human-verified unit tests. If an agent writes a sprawling TypeScript function, the build should fail if the test coverage doesn’t meet a strict threshold. You’re already using AI to write the code. Use it to prove the code works, too.

Context Boundaries Matter

Bloated AI output often happens because the model is given too much context or allowed to generate too much at once. Heavyweight IDEs with aggressive multi-file auto-completion can easily create cascading messes across a codebase.

Define strict architectural boundaries and API contracts upfront. Agents should be tasked with solving small, well-defined, modular problems. “Write a function that parses this specific JSON schema” is a good prompt. “Build the backend” is not. The tighter the scope, the less room for generated nonsense.

Observability Is Your Safety Net

You can’t catch all generated slop at the PR level. Some of it only reveals itself under load. An agent might write a technically correct query that causes an N+1 database issue, or introduce a subtle memory leak that passes all unit tests.

Your ultimate safety net is what happens at runtime. You need an airtight observability stack to trust the velocity AI brings. Logs, distributed tracing, metrics, all feeding into dashboards your team actually watches. When generated code hits staging, you need the immediate telemetry to spot performance regressions before they reach production.

Redefine the Human Review

Because AI makes the “typing” part of coding trivial, the human code review needs to fundamentally shift. Reviewers should no longer be looking for missing semicolons. They should be asking: “Does this component fit our architecture?” and “Did the agent over-engineer this solution?”

Train your senior engineers to review for intent and systemic impact. That’s the stuff AI genuinely can’t do yet. Leave the syntax checking to the robots.

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

Leading Teams When AI Does the Typing

AI is changing how teams build things. That part is obvious. What’s less obvious is that it doesn’t change what makes people want to follow you.

If anything, as the technical execution gets more automated, the human stuff becomes more important. Empathy, vision, strategic judgment. This has been top of mind for me a lot lately. I think the leaders who are going to thrive aren’t the ones who adopt AI the fastest. They’re the ones who understand what AI can’t do.

Focus on Outcomes, Not Output

AI accelerates generation. Code, docs, data parsing. All of it gets faster. That means individual output spikes, and measuring your team’s success by raw productivity becomes a losing game.

Your job is to provide the why. AI can handle the how, but it cannot determine why you’re building something in the first place. Crystal-clear business context and architectural vision are what keep your team’s AI-augmented velocity pointed in the right direction.

Something worth thinking about: when engineers can generate solutions faster, system complexity increases. Fast. Leadership means making sure the team is building the right things, not just building things quickly. Speed without direction is just expensive chaos.

Build the Guardrails Before You Need Them

Before you hand a team tools that let them move at lightspeed, the brakes and steering are important to have in place.

Automate your compliance. Strong SAST/SCA tooling, foolproof secret management, the boring stuff that lets people experiment without risking the infrastructure. As AI assists with more logic and agents take on more autonomous tasks, these guardrails become non-negotiable.

Same goes for observability. You can’t manage what you can’t see. When AI agents are handling high-volume tasks, a solid observability stack is what lets you trust the automation and catch it when a model hallucinates or a system drifts.

Automate the Toil, Protect the Thinking

A good leader uses AI to elevate human effort, not replace it. Look for the most repetitive, low-joy tasks in your team’s workflow. Scaffolding test environments, parsing logs, summarizing meetings. Deploy AI to handle the toil so your people can do the work that actually requires a brain.

Then protect that time. With AI handling the busywork, guard your team’s calendar for deep architectural thinking, complex debugging, creative system design. The stuff AI still struggles with. That’s where your team’s real value lives.

The Human Skills Are the Whole Game Now

The most valuable skills in an AI-driven org are the ones algorithms can’t replicate.

Psychological safety. AI tools are changing fast. Your team needs to feel safe experimenting, failing, and learning. Punish well-intentioned experimentation and you’ll kill innovation before it starts.

Mentorship. AI can answer factual questions all day long. It cannot mentor a junior engineer through a crisis of confidence or help a senior navigate organizational politics. Put your energy into 1:1s, career mapping, and active listening.

Ethical judgment. As AI agents take on more responsibility in domains like finance, underwriting, or automated operations, you are the moral compass. You’re the one who needs to ask the hard questions about bias, fairness, and unintended consequences.

So What’s the Job Now?

The job is the same as it’s always been: create clarity, remove obstacles, and give a damn about your people. AI just raises the stakes on all three.

The leaders who get this right won’t be the ones with the best AI strategy deck. They’ll be the ones whose teams actually want to show up and build something together.

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

What Is a Runbook and Why Should You Care?

If you’ve ever been woken up at 3 AM by a pager and stared at your screen trying to remember how the database failover works, you already know why runbooks matter. You just might not have had one yet.

A runbook is a step-by-step guide for handling a specific operational scenario. Database goes down? There’s a runbook for that. Failed deployment needs a rollback? Runbook. Routine certificate rotation? You get the idea. They range from simple markdown files to fully automated scripts where a human only needs to click “approve.”

That’s the idea anyways. The impact of having good ones versus not having CAN be massive.

Why They Matter

When something breaks in production, your brain is not at its best. Adrenaline kicks in, Slack is blowing up, and suddenly you can’t remember if you’re supposed to restart the service first or check the connection pool. A runbook takes the thinking out of the equation. You follow the steps. You restore the service. You go back to sleep.

This directly lowers your Mean Time To Recovery (MTTR). Instead of spending twenty minutes in a group call debating what to try next, you open the runbook and start executing.

Runbooks also solve the consistency problem. If five different engineers respond to the same alert five different ways, you’re rolling the dice every time. One of those approaches might cause a secondary outage. A runbook ensures everyone follows the same diagnostic and remediation path, which means fewer surprises.

And then there’s the tribal knowledge issue. Every team has that one senior engineer who knows exactly how to fix the weird thing that happens once a quarter. What happens when they’re on vacation? Or they leave the company? A runbook gets that knowledge out of their head and into a document the whole team can use.

It also makes onboarding way faster. New engineers can start handling on-call rotations with confidence instead of hoping nothing breaks on their watch.

Treat Them Like Code

This is the part a lot of teams get wrong. Runbooks shouldn’t live in a random Confluence page that hasn’t been updated since 2023. They should live in version control. Sometimes they’re kept in the repo with the code. Other times they’re kept separate. It’s up to you. It’s up to the team on where to put it.

If a developer changes how a service authenticates or connects to a database, the associated runbook needs to be updated in the same pull request. An outdated runbook is worse than no runbook at all. It sends engineers down the wrong path during an outage, which burns time and trust.

Share Early, Share Often

A runbook sitting in someone’s private folder is doing exactly nothing for your team.

Start during the draft phase. Have someone who didn’t write the runbook try to follow it. If they get confused or stuck, the runbook needs work. This is the cheapest way to find gaps.

When a new service is heading to production, the runbook should be part of the readiness review. I’d argue a service shouldn’t go live without one. And after an incident, if the runbook was wrong or didn’t exist, creating or fixing it should be a mandatory action item from the post-mortem.

One more thing. Practice them. Run game days where the team actually walks through runbooks before a real emergency happens. The worst time to discover your runbook has a missing step is when production is on fire.

So Here We Are

Runbooks aren’t glamorous. Nobody’s giving a conference talk about the beautiful runbook they wrote last quarter. But they’re the difference between a calm, methodical incident response and a panicked Slack thread full of guesses. Write them, version them, share them, and practice them. Your future self will thank you.

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

Trending Blogs

Recently Viewed Blogs

LLBBL.blog