137Foundry Web & App Design

[137-foundry]

Why We Cut Onboarding Steps Before Adding Them

We have worked on enough SaaS onboarding redesigns over the last few years that a pattern has become hard to ignore. The instinct on every project is to add new screens, new tutorials, new tooltips, new prompts. The reality on every project is that the biggest activation lift comes from removing screens, not adding them.

This is a counterintuitive observation when you say it out loud. A team that has spent weeks discussing how to "improve" onboarding is rarely thinking about subtraction. The brief is usually some version of "make it better", which implicitly assumes additive change. The product manager pitches a guided tour. The designer mocks up a more sophisticated empty state. The growth lead wants a personalized welcome screen. Everyone is proposing additions.

Photo by Akshar Dave🌻 on Pexels

What the Data Actually Shows

When we instrument a SaaS onboarding funnel and look at where users drop off, the biggest losses are almost always at the screens that were added recently. The original signup-to-first-action flow was usually short and direct. The accumulated additions are usually the source of the drop-off.

Across the projects we have run at 137Foundry, the pattern is consistent:

Original flow at product launch: signup, password, dashboard, first action. Four screens.

Flow after 24 months of additions: signup, password, email verification, team setup, profile completion, plan selection, tutorial overlay, settings preview, integration prompts, dashboard, first action. Eleven screens.

Activation rate after 24 months of additions: roughly half what it was at launch.

The team adding each screen had a reasonable argument. The email verification was for security. The team setup was for collaboration. The profile completion was for personalization. Each addition, in isolation, was defensible.

In aggregate, they killed activation.

The Mental Model That Helps

The mental model we use is to ask, for every screen, "if a user closes the tab on this screen, what fraction of them will come back?" The answer is almost always lower than the team assumes. Most users who close the tab during onboarding never come back. The cost of every screen between signup and first useful action is the percentage of users who drop off at that screen, multiplied by the lifetime value of those users.

This makes additions much harder to justify. The team adding the email verification screen is implicitly accepting that some percentage of acquired users will be lost to it. The percentage is rarely measured. When we instrument it and show them the number, they are usually surprised.

When Additions Actually Help

We are not anti-addition. Some additions help, and some do not, and the difference is measurable rather than aesthetic.

Additions that help: sample data on first login, defaults on first-action forms, inline-first-action patterns that compress multiple steps into one, contextual prompts that surface only when the user has shown intent for the related action.

Additions that do not help: forced tutorial overlays, pre-task configuration wizards, email-gated unlocks, multi-step team setup wizards before the first useful action, mandatory profile fields that the product does not strictly need.

The pattern: additions that reduce the path to first useful action help. Additions that lengthen the path hurt. This sounds obvious. In practice, almost every addition lengthens the path, because the path-shortening additions tend to be invisible (a default value, an inline form, a sensible sample dataset) while the path-lengthening additions tend to look like progress (a new screen, a new feature, a new prompt).

The Practice We Use

For every onboarding addition we propose to a client, we ask three questions before we agree to ship it:

Does this addition shorten or lengthen the path from signup to first useful action?

What percentage of users do we expect to drop off because of this addition?

What value does the addition produce that justifies the drop-off cost?

If the addition lengthens the path, the answer to question 2 is non-zero. The question becomes whether the value in question 3 justifies the cost. Most of the time, the answer is no, and we propose a path-shortening alternative.

Email verification is the classic example. It lengthens the path. Some percentage of users drop off because of it. The value (preventing fake signups, enabling password reset, complying with anti-spam rules) is real. But the value can usually be captured without forcing verification on day one. The user can use the product immediately, and verification is required only when they take an action that needs it -- inviting a team member, configuring billing, contacting support.

The same logic applies to most other "required" additions. Configuration that matters for the user's third or fourth action can be deferred to the third or fourth action. The first action almost never needs more than sensible defaults.

What This Looks Like in Practice

At 137Foundry, our onboarding work for clients usually starts with a screen inventory. We list every screen between signup and first useful action and assign each one a cost (the percentage of users who drop off at it) and a value (the function it serves). The work that follows is mostly about cutting screens whose cost exceeds their value.

This is unglamorous work. The output is not a beautiful new design system or a clever new feature. The output is a shorter flow that the team built in the first place but bloated over time. The activation rate moves. The downstream metrics move with it. Nobody puts "removed 4 unnecessary onboarding screens" on their portfolio site.

For the framework we use, the 137Foundry guide on onboarding drop-off walks through the systematic redesign process. The services we offer include onboarding flow audits for SaaS and B2B products.

External design references we keep coming back to: the Nielsen Norman Group research on activation funnels, the Smashing Magazine editorial archive on UX case studies, and the broader W3C accessibility guidelines which intersect onboarding design in ways that most teams underweight.

The Mindset Shift

The hardest part of this work is the mindset shift. Teams who have been adding to the onboarding flow for two years find it counterintuitive to subtract. The screens that we propose cutting were added by people who are still on the team and remember the meeting where they argued for each addition.

The conversation about cutting them has to be framed against the activation data. "We are not saying your screen was a bad idea. We are saying the funnel data shows it is costing more than it is producing. Here is the number." When the conversation is framed this way, the team usually agrees to cut. When the conversation is framed as a design preference, the team usually defends the screen and the cut does not happen.

The Compounding Effect

The teams that get good at cutting onboarding screens tend to stay good at it. The discipline becomes part of how they ship product. Every new feature gets evaluated against the activation funnel. The accumulation of cruft slows. The activation rate stays stable or improves over time.

The teams that do not develop the discipline tend to drift in the other direction. The onboarding flow gets longer. The activation rate falls. The team blames upstream acquisition channels for the rising cost-per-activated-user, when the real problem is the flow that has accumulated 24 months of "improvements".

We have worked on both kinds of teams. The first kind is more fun to work with. The work is sharper and the results are easier to see. The second kind takes a few months to acclimate to the data-first mindset, but once they do, the same pattern emerges.

Cutting first is the right default. Adding after measurement is the discipline. Both produce better products than the additive instinct that most teams ship with.

[137-foundry]

5 Onboarding Patterns We See Killing SaaS Activation

Across the SaaS onboarding redesigns we have run at 137Foundry, five patterns come up over and over as the dominant sources of activation drop-off. Each of these is technically defensible in isolation. In aggregate, they explain most of the gap between signup rate and activation rate.

If a product team is wondering why activation is below industry benchmarks, the answer is usually one or more of these five.

Photo by Walls.io on Pexels

1. The Empty Dashboard

The user signs up, lands on the main dashboard, and sees nothing. No projects. No data. No model for what success looks like. The dashboard has a "create your first project" button at the top, but the button leads to a blank form that expects the user to invent something from scratch.

A meaningful percentage of users close the tab here. They did not sign up to invent a project; they signed up because someone told them the product would help them with work they were already doing. The empty dashboard is asking them to do work the product was supposed to do for them.

The fix is sample data on first login, defaults that get the user to a populated state in one click, or an inline first-task flow that does not require navigating through the empty dashboard. The Nielsen Norman Group has documented this pattern across many product categories.

2. The Configuration Wizard Before First Use

The user signs up to send invoices, and the product asks them to configure currency, tax settings, payment methods, business hours, notification preferences, and team structure before they can send anything. The configuration wizard has 8 to 12 steps. The user came to send an invoice, not to configure a SaaS product.

The configuration wizard is a leftover from the enterprise-software era when products assumed users would deploy them carefully over weeks before using them. The pattern translates poorly to self-serve SaaS, where the user has 5 minutes to decide whether the product is worth more of their time.

The fix is to defer every configuration question until the moment it actually matters for the task at hand. The user can send an invoice with defaults. If the defaults are wrong for this specific invoice, surface the configuration question inline on the invoice screen. The full settings page is for users who have already gotten value from the product, not for users who are deciding whether to.

3. The Forced Tutorial Overlay

The user lands in the product and immediately sees an overlay that walks them through every feature on the screen. Each feature has a callout box with explanatory text and a "next" button. The user has 7 to 12 callouts to dismiss before they can interact with the product.

The forced tutorial is one of the most reliably hated onboarding patterns. Users skip them, dismiss them, and -- in the worst cases -- close the product entirely rather than dismiss them. The information in the tutorial is rarely retained because the user has no context for why the features matter.

The fix is to make tutorials optional and contextual. A small "show me around" button that the user can click when they want a tour, paired with inline hints that appear only when the user is about to take a related action, outperforms forced tutorials on every metric we have measured.

Smashing Magazine has published case studies on tutorial overlay performance. The consistent finding is that optional tutorials with contextual triggers outperform forced overlays by 20 to 40 percentage points on retention metrics.

4. The Email Verification Wall

The user signs up, provides their email, and is told to check their inbox to verify before they can use the product. They do not check the inbox. They close the tab. They never come back.

Email verification has legitimate purposes -- preventing fake signups, enabling password reset, complying with anti-spam rules. The mistake is requiring verification before first use, which turns the verification step into a major drop-off point.

The fix is to allow immediate access to the product while making verification a prerequisite for actions that actually require it. Inviting team members, configuring billing, contacting support -- these can require verification. Using the product for the first time should not.

5. The Profile Completion Demand

The user lands in the product and is shown a profile completion form. Photo, job title, company size, role, phone number, country, time zone. The form is 8 fields. The user is told they cannot use the product until they complete it.

The profile completion form is almost always for the company's benefit, not the user's. The company wants demographic data for sales segmentation, ICP analysis, or marketing. The user wants to use the product. The mismatch produces drop-off.

The fix is to make profile fields optional and surface the questions over time as they become relevant. The product can get most of the demographic data through behavior rather than self-report. The fields that are genuinely necessary for the product to function (timezone for scheduling, role for permissions) can be defaulted to sensible values and surfaced only when the defaults break.

Photo by Kawê Rodrigues on Pexels

What All Five Have in Common

The pattern across all five: each one prioritizes something the team wants over something the user wants. Configuration, verification, demographic data, education, and structured setup are all team priorities. The user priority is "let me use the product".

When the team priorities are forced on the user before they can use the product, the user leaves. This is not a UX problem in the narrow sense; it is a prioritization problem. The team is signaling that their priorities matter more than the user's, and the user is responding by going elsewhere.

The fix in every case is to defer the team priority until the user has gotten value from the product. Once the user has experienced what the product can do, the team priority becomes negotiable. Configuration, verification, demographic data, education, and structured setup all become acceptable once the user has a reason to want the product to keep working.

How We Find These Patterns

The diagnostic process is straightforward. At 137Foundry, we run a free onboarding flow audit on SaaS products where we sign up from scratch and walk through every screen, take notes on where the friction appears, and produce a prioritized list of cuts and changes.

The pattern is consistent. Most products have 3 to 4 of the five patterns above. The audit identifies which ones are most expensive in terms of activation impact and proposes specific changes. The implementation work is usually 4 to 8 weeks, and the activation improvement is usually 10 to 30 percentage points.

For the framework behind the audit, the 137Foundry guide on onboarding drop-off walks through the redesign process. The services we offer include onboarding flow audits for SaaS and B2B products.

External design references we keep coming back to: the Nielsen Norman Group research on activation funnels, the Smashing Magazine editorial archive on UX case studies, the W3C accessibility guidelines on progressive disclosure, and the MDN UI patterns reference for the technical implementation patterns.

Closing Thought

The five patterns are not a comprehensive list of every onboarding problem we see, but they account for most of the activation drop-off we have measured across client projects. Teams that fix these five usually move activation by enough to make the redesign work pay for itself within a few quarters.

The harder part is institutionalizing the discipline. The empty dashboard, the configuration wizard, the forced tutorial, the email verification wall, and the profile completion demand all came from someone proposing them in a meeting and the team agreeing. Avoiding the next round of accumulation requires building the activation-first habit into how the team evaluates new product proposals.

That habit is the deeper output of an onboarding redesign. The shorter flow is the visible result. The improved decision-making is what keeps the result from regressing over the next 12 to 24 months.

[137-foundry]

Regex Tools We Actually Use in Production Data Pipelines

Writing regex without good tooling is like writing SQL without an EXPLAIN plan. You can do it, but you should not have to. Over years of building production validation and data ingestion layers at 137Foundry, a small toolkit of regex utilities has earned a permanent place in our workflow. This is a roundup of those tools, what each is good for, and which is wrong for which job.

The criteria for inclusion: actually used in our work, available for free (most have paid tiers but the free tiers are sufficient), and providing capabilities that meaningfully improve regex correctness or performance.

1. regex101

regex101 is the most-used regex tester in our toolkit. It supports PCRE, ECMAScript, Python, Java, and Go dialects, with side-by-side input testing, capture group highlighting, and a step-count metric that reveals catastrophic backtracking before it hits production.

The features that matter most for production work:

Step counter. Patterns with more than a few thousand steps on small inputs are likely vulnerable to ReDoS attacks. The step counter makes this visible before deployment.

Multi-dialect testing. The same pattern can behave differently across regex dialects. Being able to switch dialects and re-test catches subtle bugs.

Saved patterns. Patterns can be saved with URLs that include the test input, which makes them easy to share in code review or document in design docs.

Explanation panel. For complex patterns, the breakdown of what each token does is faster than reading the regex character by character.

The free tier is generous and covers nearly every realistic use case. Paid tier adds team features and more saved patterns.

2. regexr

regexr is an alternative regex tester with a slightly different feature set. It is faster for quick interactive testing (the UI is lighter) and includes a cheat sheet sidebar with quick references for common patterns.

Where it shines: rapid iteration on a regex while watching live highlighting of matches. The lighter UI makes it less distracting for short sessions.

Where regex101 wins: deeper analysis (step counting, dialect comparison), saved patterns with test input, and broader dialect support.

We use both. regexr for quick "does this pattern match what I think?" checks. regex101 for production-bound patterns and ReDoS analysis.

3. MDN Regex Reference

The MDN documentation on JavaScript regex is the canonical reference for ECMAScript regex behavior. It covers every token, every flag, browser-specific quirks, and the gradual evolution of the regex spec across ECMAScript versions.

For any regex destined to run in browser JavaScript or Node.js, MDN is the first stop when you are unsure of behavior. The lookbehind support in ES2018, the Unicode property escapes added in ES2018, and the dotAll flag are all documented there with examples.

For other languages, the analogous canonical references are: Python's re module docs, the PCRE documentation, and the Java regex Pattern class documentation.

4. Atom-Style Verbose Mode

Not a tool per se, but a practice worth highlighting. Python's re.VERBOSE flag (and PCRE's equivalent extended mode) let you write multi-line, commented regex:

EMAIL = re.compile(r""" ^ [A-Za-z0-9._%+-]+ # local part @ [A-Za-z0-9.-]+ # domain \. [A-Za-z]{2,} # TLD $ """, re.VERBOSE)

This is unrelated to a specific tool but transforms regex from a write-only language into a readable one. Languages that do not support verbose mode (JavaScript, sadly) get the same effect by composing the pattern from named string variables and constructing the final regex with new RegExp(parts.join('')).

5. Language-Specific Linters

Most modern linters can flag obvious regex problems before they reach production. ESLint has regex-specific rules. Bandit (a Python security linter) flags regex patterns with ReDoS risk. Spotbugs catches problematic regex in Java.

These rarely catch sophisticated bugs but consistently catch the obvious ones: unanchored patterns where anchors are expected, missing escapes, nested quantifiers in untrusted-input paths. Worth enabling.

6. A Test Corpus Approach

Not a tool but a discipline: maintain a test corpus of known-bad inputs alongside the regex patterns themselves. Every time a production incident reveals an input the regex should have rejected (or accepted), add the input to the corpus. The corpus becomes the institutional memory of "what we have learned about real input distributions."

The corpus is more valuable than any external tool because it captures the bugs that are specific to your domain. Generic regex testers catch generic problems. The corpus catches the problems that hit your users.

We track corpora as plain text files in the repos for each pipeline, with categories like valid_inputs.txt, invalid_inputs.txt, and historical_failures.txt. The free regex documentation by 137Foundry maintained in our internal docs includes example corpora for common input types, available on the 137Foundry site.

7. Compilation Profiling

For high-throughput regex (running thousands of times per second in a pipeline), compilation overhead matters. The general advice is "compile once, reuse the compiled pattern" in any language that supports it.

Profiling tools for the host language reveal whether you are actually doing this. In Python, cProfile shows time spent in re.compile() calls. If the same pattern is being compiled repeatedly, the fix is moving the compile out of the hot loop and into module-level or class-level constants.

In JavaScript, regex literals (/pattern/) are cached automatically. new RegExp(string) inside a loop is not. The fix is the same: build the regex once.

8. ReDoS Detection Tools

For regex patterns running on user-controlled input, ReDoS detection is worth automating. Several open-source tools analyze regex statically for backtracking risk:

rxxr2 for academic-grade analysis (slower but thorough)

safe-regex for Node.js codebases (npm-installable)

OWASP regex linter in various forms

These do not catch everything. They catch the patterns with obvious backtracking risk, which is the majority of ReDoS issues. For security-sensitive paths, running one of these against your regex inventory periodically is worth the time.

The OWASP ReDoS cheat sheet covers the underlying theory and the patterns most likely to be vulnerable. Worth reading once if you build regex into security-sensitive paths.

9. Live-Reloading Regex Test Suites

The fastest way to iterate on regex is a unit test file that runs on save. Add the test for the new input you want to handle, watch it fail, modify the regex, watch it pass, watch the previous tests still pass.

This pattern works in any language with a fast test runner: Jest in JavaScript, pytest in Python, JUnit with continuous testing in Java. The key is that the feedback loop is under a second; anything slower and you lose the interactive flow.

We pair this with the regex101 web tester for exploratory work and a unit test file for committed patterns. Exploratory: in the browser, fast iteration. Committed: in the test suite, permanent regression coverage.

10. Plain Old Documentation

The least-glamorous but most useful tool is plain documentation alongside the regex. Every pattern in our production code has a comment that includes:

What inputs the pattern accepts (examples)

What inputs the pattern rejects (examples)

What inputs it deliberately does not try to handle

Known assumptions about input format

This kind of documentation makes the next reader (often the original author six months later) able to understand what the pattern is doing without reverse-engineering it. The documentation is more valuable than the pattern itself when the assumptions need to change.

Putting the Toolkit Together

A practical workflow:

Sample real input. Inspect the variations.

Sketch a pattern in regex101. Watch it match.

Test against the boundary cases. Watch some of them fail.

Adjust. Test the step counter for backtracking risk.

Commit with documentation, test cases, and a corpus entry for any past failure mode.

Profile in production. Adjust compilation strategy if needed.

Periodically run static analysis on the regex inventory.

Each step uses one or two of the tools above. No single tool covers the whole flow. The combination is more useful than any single tool.

For the full reference on the regex patterns we use in production validation layers at 137Foundry, see Regex Code Snippets: Patterns for Common Validation and Parsing Problems. Our data integration service covers the architectural side of validation in production data pipelines, where these tools and patterns get most of their use.

[137-foundry]

Hot Take: Most Regex Bugs Are Not Regex Bugs

We see a lot of production data pipelines and validation code at 137Foundry. After enough postmortems, a pattern has emerged. The bugs everyone blames on regex are almost never actually regex bugs. They are assumption bugs. The regex did exactly what it was told to do. What it was told to do was based on a wrong understanding of what the input would look like.

This sounds pedantic, but it changes how you should think about regex in production. Here is what we mean.

The Pattern Behind the Pattern

A typical regex bug postmortem goes like this. A function is processing user input, validating it with a regex, and one day it crashes on a record that "should have been rejected." The team digs in. The regex is reviewed. Sure enough, the regex accepts the malformed input. Someone updates the regex. The bug is filed as a "regex bug."

But the regex was written against an assumption: that the input would always be ASCII, or always have at most three digits, or always end in .com. The assumption was wrong. The regex faithfully implemented the assumption. Calling it a regex bug is like calling a hammer a bad hammer because it does not drive screws.

The actual bug was upstream. Someone made an assumption about input distribution and codified it in a pattern. The assumption was incomplete. The fix is not just a better regex; the fix is a better process for noticing when assumptions about input do not hold.

How This Plays Out in Production

A few examples from the field, anonymized:

The phone number import. A regex validated US phone numbers in a customer onboarding flow. It worked for a year, then suddenly started rejecting one customer in a hundred. The customers were not malformed. The customers were Canadian (same format as US, but the regex accidentally required area codes starting with 2-9, which fails for some Canadian regions).

The regex was correct under the assumption "all customers are US-based." The assumption stopped being true. The "regex bug" was actually a product expansion bug.

The email validation incident. A regex rejected addresses with + characters in the local part, breaking signups for users on Gmail who used plus-tagging ([email protected]). The regex was wrong under any reasonable email validation rules. But the underlying problem was that nobody on the team had ever used Gmail plus addressing, so the assumption "valid emails do not contain plus signs" was unchecked.

The CSV parser explosion. A regex pattern was used to split CSV-like lines. It worked perfectly for two years. Then a customer uploaded a CSV from Excel on Windows that included a Unicode byte order mark () at the start of the file. The first field of the first row was now customer_id instead of customer_id, and every downstream lookup against the first field failed. The regex did not have a bug. The assumption "CSV files do not start with BOM characters" had a bug.

In each case, fixing the regex addressed the immediate problem. But the underlying issue was an assumption about input that had gone unverified.

What This Looks Like to Fix Properly

The fix that scales is not "write more elaborate regex." More elaborate regex codifies more elaborate assumptions, which means more elaborate failure modes when the assumptions are wrong. The fix that actually scales is a process for surfacing and testing assumptions about input.

A few concrete practices we use at https://137foundry.com:

Sample real input before writing regex. If the input is user-controlled, grab a sample of recent inputs from production and inspect them. If the input is from a partner system, request a representative file. The variations in real input are almost always wider than the variations you would have guessed.

Document what the regex assumes. A comment on the regex listing the input characteristics it assumes (ASCII only, no whitespace, US format, no fractional seconds) makes the assumption explicit. When the assumption stops holding, the comment lets the next reader see immediately what changed.

Test against pathological input. Empty strings, leading whitespace, trailing whitespace, mixed line endings, BOM characters, Unicode characters, very long strings. These are the inputs production will throw at you eventually. Test them now, not later.

Log rejections instead of silently dropping them. A regex that rejects a few percent of input over time tells you something about your assumptions. A regex that rejects silently lets the assumption drift be invisible until it causes a customer-visible issue. Log the rejected inputs with structured metadata. Periodically review the rejections to see if the assumptions still hold.

Test the assumption migration. If the regex's assumptions change (new country supported, new client integration with different format), update the test corpus first. Confirm the regex correctly handles the new shape before deploying. This catches more bugs than any amount of in-regex sophistication.

Why "Just Use a Library" Is Not Always the Answer

A common reflex when we make this argument is "well, then use a library instead of regex." This is sometimes right and sometimes wrong.

For email validation, parser libraries are generally not better than the pragmatic regex. They have the same fundamental limitation: format does not equal reachability. A library that returns "valid" for an unreachable address is no better than a regex that does the same.

For phone numbers, libraries (libphonenumber) are much better than regex because the rules are genuinely too complex for regex to encode correctly. The library handles every country's format quirks.

For URL parsing, the language's built-in parser is almost always better than regex. URL canonicalization, query parameter handling, and IDN decoding are too subtle for regex to get right.

The pattern: when the underlying rules are complex and well-specified, use a library. When the underlying rules are loose and just need a sanity check, regex is fine. The category mistake is using regex for genuinely complex parsing or using a heavy library for a one-line format check.

What This Means for Engineering Practice

If most regex bugs are actually assumption bugs, the engineering practice that prevents them is not "be better at regex." The practice is "be better at surfacing assumptions."

A few questions worth asking before shipping regex in any code path:

What input distribution does this pattern assume?

Where did the assumption come from (specification, intuition, "looked at one example")?

What changes in the upstream system would invalidate the assumption?

How would we notice if the assumption became wrong?

The pattern itself is a small part of the answer. The surrounding discipline is most of it.

Further Reading

We wrote a longer article covering the practical regex patterns we actually use in production validation layers at 137foundry.com, including the assumption notes that make them safe to deploy: Regex Code Snippets: Patterns for Common Validation and Parsing Problems. The patterns are useful. The discipline around the patterns is what makes them actually work.

For more on our approach to production data validation, the 137Foundry data integration service covers the architectural patterns. The general principle is the same one above: cheap format checks at the boundary, expensive verification later, structured logging of every rejection. The regex is one tool in the layer. The layer is the part that matters.

Useful tooling resources we point clients to: regex101.com for interactive testing and step counting, and MDN documentation for the canonical reference on JavaScript regex behavior across browsers. Both pair well with the kind of explicit-assumption practice described above.

Treat regex as an executable assumption. Document the assumption. Test what happens when the assumption is wrong. Most "regex bugs" stop happening once you do.

[137-foundry]

What We Have Learned Building Bi-Directional Sync Systems at 137Foundry

We have built bi-directional data syncs across a range of client contexts at 137Foundry: CRM-to-billing syncs for SaaS companies, inventory syncs between warehouse management systems and e-commerce platforms, and customer data syncs between support platforms and backend databases.

The first integration always seems straightforward. It is the one running six months later that reveals what we got wrong at the start.

The conflict resolution question is a business question first

Every team we work with wants to talk about the technical implementation of conflict resolution before they have answered the business question: when both systems update the same record simultaneously, which system should win?

We have learned to force this question early. If the CRM team and the billing team disagree about which system owns customer_status, the sync layer cannot resolve that disagreement -- it will just consistently pick the wrong answer for half the stakeholders.

We now require a field authority document before any code is written: a spreadsheet mapping every shared field to an owner and a conflict rule, signed off by both sides. The PostgreSQL logical replication documentation describes the mechanics of change capture, but it does not tell you which system is authoritative for which fields. Getting the business question answered in writing first saves enormous pain downstream.

Sync loops are invisible until they are catastrophic

On every new bi-directional sync integration, we add sync loop detection before we add anything else. The detection is simple: tag changes made by the sync layer with an application identifier, filter those events out of the CDC stream before propagating them.

The reason this is the first thing we add: a sync loop does not cause obvious errors. The sync runs, data looks correct, metrics look fine. What accumulates is API call volume and change event volume -- both growing exponentially with every sync cycle. We have seen sync loops run for 18 hours before anyone noticed, generating millions of change events and exhausting the API quota for three downstream integrations.

The fix once it is running is to stop the sync, clear the change log, and restart. The fix before it runs is one conditional check in the CDC consumer. We treat this as a non-negotiable day-one requirement for every integration we ship.

Replication slot lag is a disk space problem waiting to happen

For syncs using PostgreSQL WAL-based CDC, the replication slot holds WAL segments until the consumer acknowledges them. If the consumer stops processing -- a deployment, a bug, an overloaded consumer -- the slot accumulates unreprocessed WAL segments. PostgreSQL will not clean up WAL segments referenced by an active replication slot.

In a high-write production database, an unmonitored lagging slot can fill disk in hours. We add a monitoring check on pg_replication_slots for lag_bytes within 48 hours of any WAL-based CDC setup. Alert at 1 GB lag, page at 5 GB. The two times this alert has fired in production, it prevented a disk-full outage.

The dead-letter queue needs a human interface before it has human traffic

Every bi-directional sync we ship now includes a dead-letter queue and a lightweight admin interface for reviewing DLQ entries. We build the interface before we have any DLQ traffic, not after.

The reason: the first time an unresolvable conflict lands in the DLQ, you want someone to be able to inspect it and decide whether to retry, manually resolve, or dismiss it. If the first DLQ entry arrives and there is no UI to review it, it sits in Redis accumulating siblings until someone builds the interface under pressure.

The interface does not need to be sophisticated. A table showing the record ID, the two conflicting versions, the conflict reason, and buttons to apply version A, apply version B, or dismiss is sufficient. What matters is that it exists before it is needed, not after the first incident that makes it necessary.

We documented the architecture and code patterns for all of this in How to Build a Bi-Directional Data Sync Between Business Applications. That guide reflects what we actually deploy, not the idealized version.

What Changed After We Built the Infrastructure First

After establishing this foundation -- field authority document before code, sync loop detection on day one, replication slot monitoring, DLQ with UI before first traffic -- our subsequent bi-directional sync projects have been substantially less eventful operationally.

The first incident on a new integration used to happen within 2 to 3 weeks: either a sync loop, a replication slot growing unnoticed, or a conflict resolution bug discovered by a user. Now the first incident on a new integration typically happens at month 2 or 3, is detected by monitoring rather than user report, and resolves in minutes rather than hours.

The cost of building the foundation upfront is roughly 20 to 30 percent more initial implementation time. The cost of not building it is one 4am incident call, a Sunday afternoon cleanup, and a retro where the team agrees to "add better monitoring" -- which then happens under time pressure and is therefore incomplete.

We have documented all of this in the implementation guide at 137Foundry for the teams who want to build it themselves: How to Build a Bi-Directional Data Sync Between Business Applications. For teams who want help designing or implementing the integration, the 137Foundry data integration team is available for architecture reviews and implementation work.

Why This Matters for Production Reliability

The failure modes of bi-directional sync are almost always discovered in production, not in testing. Test environments rarely replicate the exact conditions that cause clock skew conflicts -- clock synchronization on development machines is generally better than on production infrastructure. Test environments rarely replicate the specific bulk operation patterns that create consistency gaps. And test environments rarely run long enough to reveal the slow drift that accumulates when a field authority map is not updated after a schema change.

This is not an argument against testing -- it is an argument for investing in observability alongside testing. The monitoring patterns in this guide (sync lag, DLQ depth, conflict rate, record count parity) give you visibility into problems that tests will not catch before they affect users.

For teams building a bi-directional sync for the first time, the practical recommendation is: build the operational baseline (DLQ, monitoring, idempotency, loop prevention) before the first production deployment, not after the first production incident. The upfront cost is modest. The incident prevention value is significant.

For technical implementation guidance, see 137Foundry and the data integration resources. For production architecture review and implementation support, the 137Foundry services team works with teams across the integration lifecycle.

Why This Matters for Production Reliability

The failure modes of bi-directional sync are almost always discovered in production, not in testing. Test environments rarely replicate the exact conditions that cause clock skew conflicts -- clock synchronization on development machines is generally better than on production infrastructure. Test environments rarely replicate the specific bulk operation patterns that create consistency gaps. And test environments rarely run long enough to reveal the slow drift that accumulates when a field authority map is not updated after a schema change.

This is not an argument against testing -- it is an argument for investing in observability alongside testing. The monitoring patterns in this guide (sync lag, DLQ depth, conflict rate, record count parity) give you visibility into problems that tests will not catch before they affect users.

For teams building a bi-directional sync for the first time, the practical recommendation is: build the operational baseline (DLQ, monitoring, idempotency, loop prevention) before the first production deployment, not after the first production incident. The upfront cost is modest. The incident prevention value is significant.

For technical implementation guidance, see 137Foundry and the data integration resources. For production architecture review and implementation support, the 137Foundry services team works with teams across the integration lifecycle.

[137-foundry]

How We Handle Conflict Resolution in Production Data Sync at 137Foundry

Conflict resolution is the part of bi-directional sync that most guides skip. "Use last-write-wins" sounds reasonable until you are debugging why a customer's payment status keeps flipping between systems. Here is the process we actually follow at 137Foundry.

Step 1: Map every shared field to an owner before writing code

We start every bi-directional sync project by building a field authority matrix: a spreadsheet with every field in the shared data model, its source system, its destination system, and which system wins on conflict.

This document must be reviewed and approved by stakeholders on both sides before any code is written. The reason: conflict resolution is a policy decision, not a technical one. If the CRM team and the billing team disagree about which system owns customer_status, the sync layer cannot resolve that disagreement -- it will just consistently pick the wrong answer for half the stakeholders.

Step 2: Implement conflict resolution as configuration, not code

We implement the field authority map as a YAML configuration file:

field_authority: name: owner: crm rule: crm_wins payment_status: owner: billing rule: billing_wins last_activity: owner: shared rule: last_write_wins_with_tolerance tolerance_ms: 500 notes: owner: shared rule: dlq_on_conflict

The conflict resolution engine reads this configuration at startup. Adding a new field to the sync just requires updating the YAML -- no code deploy required.

Step 3: Build the resolver with an audit log

Every conflict resolution decision gets written to an audit log: the record ID, both versions, the resolution rule applied, and the outcome. We use SQLAlchemy for the audit rows and a Redis sorted set for DLQ entries.

The audit log serves two purposes: diagnosing incorrect conflict resolutions after the fact, and building the operational case that the rules are working correctly over time. Without the audit log, you cannot distinguish "the resolver is working as designed" from "the resolver is silently making bad decisions."

Step 4: Test with the actual conflict scenarios your system will see

We ask both system owners to describe the most common simultaneous write scenarios they expect. "CRM bulk import while billing processes end-of-month invoices" is a concrete conflict scenario. "Both systems update at the same time" is not useful.

For each described scenario, we write a test that simulates the exact timestamps and field values involved. These tests become regression tests when the configuration is updated.

Step 5: Monitor conflict rate as a first-class metric

We add conflict rate -- conflicts per 1,000 sync events -- to the primary operational dashboard for every integration we ship. A stable, low conflict rate (often well under 1 percent) is normal. A spike means something changed: a new bulk import process, a new application writing to the same fields, or a misconfigured integration that is creating duplicate writes.

The conflict rate dashboard catches problems early. An integration that starts generating 5 percent conflict rates three weeks in almost always traces to a new process added on one side of the integration without coordination with the other side.

For the full architecture and code patterns behind our bi-directional sync approach, see How to Build a Bi-Directional Data Sync Between Business Applications. To discuss the architecture for your specific integration, reach out to the 137Foundry data integration team.

What This Process Produces

The output of this five-step process is not just a conflict resolver that works. It is a conflict resolution system that is auditable, configurable without code deploys, tested against real-world scenarios, and monitored for drift.

When a user reports that "my data looks wrong" six months after a bi-directional sync is deployed, the audit log tells us exactly which record, which conflict rule was applied, and what both versions looked like at the time of resolution. That kind of auditability is the difference between a 20-minute investigation and a 3-day debugging session.

We also use the field authority document as a living specification. When either system adds a new field, the first step is to update the field authority YAML and get sign-off from both system owners before updating the sync consumer code. This disciplines the schema change process and prevents the "we didn't know the sync would be affected" conversation that comes up when fields are added without coordination.

The data integration work at 137Foundry applies this process across CRM, billing, support, and ERP integration contexts. The underlying patterns are the same regardless of which specific platforms are being connected. For the technical implementation details, see How to Build a Bi-Directional Data Sync Between Business Applications.

Why This Matters for Production Reliability

The failure modes of bi-directional sync are almost always discovered in production, not in testing. Test environments rarely replicate the exact conditions that cause clock skew conflicts -- clock synchronization on development machines is generally better than on production infrastructure. Test environments rarely replicate the specific bulk operation patterns that create consistency gaps. And test environments rarely run long enough to reveal the slow drift that accumulates when a field authority map is not updated after a schema change.

This is not an argument against testing -- it is an argument for investing in observability alongside testing. The monitoring patterns in this guide (sync lag, DLQ depth, conflict rate, record count parity) give you visibility into problems that tests will not catch before they affect users.

For teams building a bi-directional sync for the first time, the practical recommendation is: build the operational baseline (DLQ, monitoring, idempotency, loop prevention) before the first production deployment, not after the first production incident. The upfront cost is modest. The incident prevention value is significant.

For technical implementation guidance, see 137Foundry and the data integration resources. For production architecture review and implementation support, the 137Foundry services team works with teams across the integration lifecycle.

Why This Matters for Production Reliability

The failure modes of bi-directional sync are almost always discovered in production, not in testing. Test environments rarely replicate the exact conditions that cause clock skew conflicts -- clock synchronization on development machines is generally better than on production infrastructure. Test environments rarely replicate the specific bulk operation patterns that create consistency gaps. And test environments rarely run long enough to reveal the slow drift that accumulates when a field authority map is not updated after a schema change.

This is not an argument against testing -- it is an argument for investing in observability alongside testing. The monitoring patterns in this guide (sync lag, DLQ depth, conflict rate, record count parity) give you visibility into problems that tests will not catch before they affect users.

For teams building a bi-directional sync for the first time, the practical recommendation is: build the operational baseline (DLQ, monitoring, idempotency, loop prevention) before the first production deployment, not after the first production incident. The upfront cost is modest. The incident prevention value is significant.

For technical implementation guidance, see 137Foundry and the data integration resources. For production architecture review and implementation support, the 137Foundry services team works with teams across the integration lifecycle.

The underlying message is that bi-directional sync failures are rarely random -- they follow predictable patterns, and the patterns have known mitigations. A team that learns these patterns before deploying a bi-directional sync ships a more reliable integration. A team that learns them after the first production incident ships a better second integration. The choice between the two paths is mostly a function of whether the team invests in understanding the failure modes at design time.

For technical implementation guidance including Python code examples, see 137Foundry data integration and the full guide to bi-directional data sync.

[137-foundry]

The CSS and Front-End Tools We Actually Use in Production at 137Foundry

We get asked regularly about our tooling stack. This is an honest answer: what we use, why we use it, and where we have moved away from tools we previously relied on. This is not a comprehensive list of everything available -- it is a specific list of what we reach for on real projects.

Native CSS Custom Properties for Theming

We use native CSS custom properties as the foundation for design tokens and theming on every project. We stopped using Sass variables as our primary theming mechanism because CSS custom properties solve the same problem with a significant advantage: they are live values that JavaScript can read and modify at runtime.

This matters specifically for dark mode. When theme switching depends on a JavaScript-toggled attribute, the CSS custom property layer handles the entire UI update with no JavaScript per element. Sass variables cannot do this -- they are resolved at compile time and baked into static values. For any application with user-controlled theming, runtime CSS variables are strictly more capable.

We still use PostCSS for certain things (nesting in older projects, autoprefixer for some target browser profiles), but the primary color and spacing token system is always CSS custom properties rather than preprocessor variables. The token layer also gives us a single place to audit design decisions -- every color and spacing value in the application is visible in one file rather than scattered across component stylesheets.

Vanilla JS for Theme Persistence

We do not use a library for localStorage-based theme persistence. The implementation is small enough that adding a dependency would increase complexity rather than reduce it:

function applyTheme(theme) { document.documentElement.setAttribute('data-theme', theme); } function getPreferredTheme() { return localStorage.getItem('theme') || (window.matchMedia('(prefers-color-scheme: dark)').matches ? 'dark' : 'light'); } applyTheme(getPreferredTheme());

This runs inline in the <head> before stylesheets paint, which prevents the flash of wrong theme. That placement is intentional and important -- the script blocks rendering briefly, but the blocking cost is negligible for a script this small, and it eliminates the visible theme flash that a deferred script produces. This is one of the few cases where a blocking script in the head is the correct choice.

The full implementation and context for this pattern is in our team's write-up: How to Add Dark Mode to a Web Application.

Stylelint for CSS Quality

Stylelint catches CSS issues that code review misses: unknown properties, selector specificity problems, use of deprecated values, and violations of team-defined conventions like requiring var() instead of hard-coded hex values.

We run Stylelint as part of CI on every project with a meaningful CSS surface area. The rule we use most is a custom rule that flags hard-coded hex values in component files -- any color value that is not accessed through a CSS custom property is a gap in the design token system, and Stylelint catches it before merge.

For teams starting with Stylelint, the standard config is a reasonable starting point. We layer custom rules on top based on the project's architecture decisions. The payoff is consistent: issues that previously surfaced in production or code review now surface at the lint stage, where they are cheapest to fix.

The W3C Contrast Checker for Accessibility Validation

We validate dark mode color palettes against WCAG contrast requirements before shipping. The minimum ratio for normal text is 4.5:1 against its background. This requirement does not relax in dark mode -- dark text on dark background failures are as common as they are in light mode, and they are easy to miss visually.

The most common failure we see in dark mode audits is secondary text -- labels, captions, placeholder text -- that falls below 4.5:1 because the muted text color was designed for a light background and was not recalibrated for dark. The fix is always to lighten the secondary text token for dark mode. We also avoid pure white on pure black combinations. Very high contrast between foreground text and dark background can cause halation on OLED displays -- a glow effect where bright text bleeds slightly into the surrounding dark area. A slightly off-white text value against a deep navy background reads more cleanly than maximum contrast extremes.

caniuse.com for Browser Support Decisions

caniuse.com is the reference we use for every browser support question. The questions we check most:

CSS custom properties -- supported across the board

prefers-color-scheme -- full modern browser support

The distinction between "supported in all modern browsers" and "supported in all browsers our specific client targets" matters. Enterprise applications targeting specific browser profiles often have different constraints than consumer web applications. caniuse gives us the data to make that judgment per project rather than applying a blanket rule.

What We Have Moved Away From

CSS-in-JS libraries for theming. We used CSS-in-JS for theming on early React projects when native CSS custom properties were less established. We have moved off it for new projects because the native CSS solution is faster (no runtime style injection), simpler (no library dependency), and works outside React components. The server-side rendering complexity of CSS-in-JS libraries was also a recurring source of friction that we no longer need to manage.

Global class-based dark mode toggles. The .dark-mode-enabled class on <body> approach requires overriding every component's styles in a matching .dark-mode-enabled .component selector block. This doubles the specificity of every dark mode rule and creates a maintenance burden proportional to the number of components. The data-theme attribute on <html> with CSS custom property overrides is cleaner and requires no per-component override rules.

Preprocessor variables as the primary theming mechanism. Already covered above -- compile-time variables cannot do what runtime custom properties do for theming. We keep PostCSS in the build pipeline for autoprefixer and occasionally for nesting in projects where native CSS nesting is not yet the target, but the token layer is always native custom properties.

How Tool Decisions Compound Over Time

Tooling decisions compound over the life of a project in both directions. A library that reduces friction in week one but requires major version migrations every 18 months creates a recurring cost that accumulates over a project's lifespan. A slightly more verbose native approach that is stable and maintainable without dependency upgrades often turns out to be cheaper over the same period when you account for the full cost of maintaining third-party dependencies.

Our general heuristic: if the native platform solves a problem adequately, use the native solution. Add a dependency only when there is a specific, named capability gap that the platform cannot fill. This is why we use vanilla JavaScript for theme persistence rather than a theming library, and why we use CSS custom properties rather than a CSS-in-JS runtime for token management. The resulting codebases are smaller, faster to onboard new developers onto, and cheaper to maintain over multi-year timelines.

"Tooling decisions compound over the life of a project. The tools we choose in week one are still there in year two, either serving us well or creating friction in every sprint. We are conservative about adding dependencies and aggressive about removing them when the platform catches up." -- Dennis Traina, founder of 137Foundry (view services)

The front-end work at 137Foundry starts with choosing the right tools for the project's actual requirements rather than the tools that are currently popular. For teams navigating these decisions on a new project or a front-end refactor, our services page has context on the kind of front-end architecture work we take on.

[137-foundry]

What We Have Learned About CSS Architecture After Building Dozens of Production Applications

At 137Foundry, we build and maintain web applications across a range of industries and complexity levels. After doing this for long enough, we have developed strong opinions about CSS architecture -- specifically about what works at scale and what creates maintenance problems that compound over time.

This is not a theoretical take. These are the decisions we make repeatedly on production codebases because they have consistently produced better outcomes than the alternatives.

Start With a Token System, Not Component Styles

The single most impactful structural decision in any web application's CSS is establishing a design token layer before writing any component styles. A token system is a set of named CSS custom properties that represent every design decision in the application: colors, spacing, border radii, font sizes, shadow values.

Without this layer, teams write component styles with hard-coded values. The button is #3b82f6. The card has padding: 16px. The heading is 1.125rem. These values are correct when they are written. Over time, the design evolves, requirements change, and someone needs to find every instance of #3b82f6 across two hundred CSS files to update the brand color. That search-and-replace exercise is the direct cost of skipping the abstraction layer.

With a token system, the brand color is var(--color-accent). Changing the brand color is a one-line edit. The same applies to dark mode: a semantic token set that maps cleanly to a dark variant makes theme switching a configuration change rather than a CSS surgery project.

We have shipped this architecture on every new project for the past three years. We have also retrofitted it onto existing codebases that did not have it, and the retrofit is always more expensive than starting with it. Start with the token layer.

The Cascade Is a Feature, Not a Bug

We see teams fight the CSS cascade constantly -- adding !important declarations, using inline styles to override stylesheet values, increasing specificity with deeply nested selectors. This is almost always a symptom of not trusting the cascade rather than a genuine limitation of it.

The cascade works correctly when selectors are written at the right specificity level and values are not overridden at the wrong layer. A component's base styles belong at low specificity. State variations (hover, focus, active) and modifier classes belong at the same specificity level as the base or slightly higher. The !important annotation should be genuinely rare in a well-architected stylesheet.

When we audit a codebase that has cascading issues, the most common cause is that component styles were written without considering the document structure they would appear in. A component written to work in isolation often has specificity problems when it is placed inside a more complex layout. The fix is usually not to add more specificity -- it is to remove the accidental specificity from the base component style.

Theming Is Infrastructure, Not a Feature

We treat dark mode and theme switching as infrastructure decisions rather than feature work. This distinction matters for timing: infrastructure decisions need to be made before component development begins, not added afterward.

When dark mode is retrofitted onto an existing stylesheet, the work is proportional to how many hard-coded color values were written before the retrofit. We have seen this process take weeks on mature codebases. When dark mode is part of the initial architecture, it costs almost nothing to implement -- the token layer handles it.

Our dark mode implementation guide, How to Add Dark Mode to a Web Application, reflects this infrastructure mindset. The article covers the CSS custom property architecture, the system preference detection, and the JavaScript toggle pattern that we use as a starting point on new projects.

We are not advocating for building dark mode before you know users want it. We are advocating for building the CSS architecture in a way that makes adding dark mode a configuration task rather than a refactoring project. The cost difference is significant.

Component-Level CSS Should Know Nothing About the Page Layout

A card component should not have margin or positioning rules. A button should not have margin-top or position: absolute. These layout properties belong in the parent element or in a composition layer that controls how components are arranged in the page.

When components include their own layout and spacing rules, they become fragile in new contexts. The card that assumes it is always followed by a sibling assumes a context that does not always hold. The button with a margin-top: 24px hardcoded into its stylesheet breaks every layout where that spacing is wrong.

We call this the layout-vs-component boundary, and we enforce it explicitly in code review. Components define their internal appearance -- colors, typography, borders, internal padding. Layout parents define how components are positioned relative to each other. Composition layers (using CSS Grid or Flexbox in a container element) define how multiple components are arranged in a larger structure. Each layer knows about its own concerns and nothing about the others.

This boundary is what makes components genuinely reusable. A card that has no layout assumptions about its context works correctly whether it is placed in a full-width content area, a narrow sidebar, a modal, or a grid of three.

Use New CSS Features When They Solve Real Problems

We adopted CSS custom properties for design tokens before they were universally recommended because they solved a genuine problem. We adopted container queries when they solved the "component that breaks in a narrow sidebar" problem that viewport media queries cannot address. We adopted native CSS nesting when it allowed us to remove a preprocessor dependency from projects that did not need other preprocessor features.

The pattern is consistent: adopt when the feature solves a real problem we were previously working around, not because it is new. We are currently evaluating prefers-color-scheme as the baseline for dark mode in client projects that previously used JavaScript-only approaches, because the media query approach eliminates the flash-of-wrong-theme issue at zero additional JavaScript cost.

CSS is improving fast enough that keeping a working knowledge of what the platform can do natively matters more than it did five years ago. The answer to "should we use a CSS-in-JS library for theming?" is now "almost certainly not" for most projects, because the CSS platform handles theming natively and with better performance characteristics.

"Our CSS architecture stance is simple: use the platform where it is capable, add tooling only for specific gaps, and design for the three-year maintenance window rather than the two-week shipping deadline. The teams that have the cleanest codebases two years later are the ones that resisted the temptation to add abstraction layers before understanding the problem they were solving." -- Dennis Traina, founder of 137Foundry (view services)

What This Means for Projects We Take On

When we start a new front-end engagement, the CSS architecture conversation happens before the first component is written. We establish the token layer, define the component-vs-layout boundary, and set up the theming infrastructure whether or not dark mode is a stated requirement. These decisions cost almost nothing to make at the beginning and a lot to retrofit later.

If you are starting a new web application or reaching a point where your existing CSS is creating friction, 137Foundry's web development team has done this front-end architecture work across enough projects to know where the common mistakes are and how to avoid them from the start. The about page has context on our approach if you want to understand how we work before reaching out.

CSS architecture is not glamorous work. It is also the kind of work where getting it right early pays dividends for the entire life of a project, and getting it wrong creates compounding friction that is visible in every sprint review. We take it seriously because our clients feel the consequences if we do not.

[137-foundry]

Why We Always Start Search Features With Debounce

We have built search into somewhere north of fifty client web applications at this point. Product catalogs, admin tools, documentation search, command palette interfaces, SaaS dashboards with global search, CMS record lookups. The backends vary enormously: Algolia, Typesense, PostgreSQL full-text, Fuse.js for client-side, Elasticsearch for a few of the larger projects, and more than a few SQL LIKE queries that were supposed to be replaced and have not been.

One thing is consistent across every single one of these: the first thing we implement, before we write a single line of backend code, is debounce on the search input.

Not because it is the most exciting part of building search. Because it is the part that fails silently if you skip it, and the failure costs more to diagnose after launch than it would have cost to build correctly in the first place.

The Problem That Hides Until Traffic Increases

Here is what happens without debounce. A user types "wireless keyboard" into the search box. That is sixteen characters. The search input fires sixteen separate API requests: one when they type "w", one for "wi", one for "wir", and so on through the full string.

At low traffic and with fast server responses, this looks fine in the browser. Results update smoothly as the user types. The backend team does not see a problem in the logs because the queries are fast and the traffic is low enough that sixteen requests per search interaction is not obviously visible as unusual.

Then traffic increases. A sale event, a feature launch, a product getting featured somewhere. Database query times stretch from 50ms to 300ms. Now the requests are in-flight simultaneously rather than completing before the next one starts. Whichever response arrives last overwrites the UI, regardless of whether it matches what the user actually typed.

A user who typed "wireless keyboard" might see the results for "wireles" if that query happened to be the last response to arrive. They typed nine more characters and the results got worse. This is confusing enough that users assume the search is broken -- because from their perspective, it is.

We have diagnosed this bug in client projects that had been live for months. The conversations are always the same: "search worked fine in testing." It did. The load was too low for the race condition to manifest. The only thing that changed was traffic, not the code.

Debounce eliminates this category of bug before it can happen.

What Debounce Actually Does

Debounce is a function wrapper that delays execution until after a pause. The classic use case is the search input: instead of firing a search request on every keystroke, debounce waits until the user stops typing -- for 250-350 milliseconds, typically -- and then fires exactly one request.

The user who types "wireless keyboard" still sees their results as intended. The search fires when they finish the word and pause. But instead of sixteen requests, the server receives one.

The implementation in vanilla JavaScript is small:

function debounce(fn, delay) { let timer; return (...args) => { clearTimeout(timer); timer = setTimeout(() => fn(...args), delay); }; } const handleSearch = debounce((query) => { fetchResults(query); }, 300); searchInput.addEventListener('input', (e) => { handleSearch(e.target.value); });

This is the entire pattern. No dependencies. No framework required. It composes with any search function: an Algolia API call, a Typesense request, a fetch to a Rails search endpoint, a Fuse.js local search. The function it wraps does not change. Only the timing changes.

For projects using React, we typically use the useMemo or useCallback hooks to memoize the debounced function so it is not recreated on every render. The underlying logic is identical.

Why 300ms and Not 150ms or 500ms

The 300ms default is calibrated to human typing speed, not picked arbitrarily. Average adult typing speed is 40-50 words per minute, which translates to roughly 200-250ms between keystrokes for normal words. A 300ms debounce fires when the user pauses -- between words, at the end of a term, when they hesitate mid-search. That pause is the signal that they have finished typing a meaningful unit and are ready for results.

At 150ms, debounce barely filters anything for users who type quickly. Two fast keystrokes land within 150ms of each other frequently at normal typing speed. The request volume reduction is minimal.

At 500ms, users who type quickly and then stop notice the delay. They have finished typing "keyboard" and wait half a second for anything to happen. At modern network speeds with a well-indexed database, 300ms is enough buffer to eliminate unnecessary requests without introducing visible latency.

The one exception is client-side search -- Fuse.js, Lunr.js, FlexSearch running entirely in the browser. When the search itself is synchronous and runs in under 2ms, there is no network to wait for. The debounce delay can drop to 150-200ms, because the goal is only preventing unnecessary DOM updates and the search result does not carry any latency penalty.

The Race Condition That Survives Debounce

Debounce reduces request volume. It does not eliminate all race conditions.

Even with a 300ms debounce, two requests can be in flight simultaneously: the user makes a query, pauses, makes a different query. Both are debounced, both fire. If the first query takes longer to complete than the second, its response arrives after the second query's results have already rendered. The stale results overwrite the current ones.

This is where AbortController comes in. Before each new request, cancel the previous one:

let controller; const handleSearch = debounce(async (query) => { if (controller) controller.abort(); controller = new AbortController(); try { const results = await fetchResults(query, { signal: controller.signal }); renderResults(results); } catch (err) { if (err.name === 'AbortError') return; throw err; } }, 300);

AbortErrors must be caught and treated as non-failures. They are intentional. The request was canceled because a newer query superseded it. That is correct behavior.

We use both patterns on every server-based search we build at 137Foundry: debounce to reduce request volume, AbortController to guarantee that stale responses never overwrite current results. They solve different problems, both are required.

The Backend Choice Comes Third

After debounce and AbortController, the third consideration is the loading state. After that, the backend architecture. We sequence it this way deliberately.

The frontend patterns -- debounce, race condition handling, loading feedback, empty state messaging -- are the same regardless of whether the search backend is Algolia or a PostgreSQL LIKE query. They determine whether the search feels right to users. A perfectly tuned search index behind a broken frontend UX does not feel like good search. A mediocre LIKE query behind a well-debounced, well-instrumented input feels surprisingly good.

When we scope search work with new clients, we scope the frontend work and the backend work separately. The frontend work is faster and its improvements are immediately visible. The backend work is slower and often requires real user search data before relevance tuning can be meaningfully done. We can ship a fast, well-behaved search input while the backend indexing strategy is still being finalized, and the users who encounter it first will not know the backend is not perfect yet.

The full implementation -- debounce, AbortController, loading states, minimum character thresholds, and how they connect to different backend approaches -- is covered in the 137Foundry guide on building search-as-you-type. 137Foundry builds the complete search stack for web applications where search quality and user experience are non-negotiable requirements.

[137-foundry]

How We Handle In-Flight Request Cancellation in Search Features

One of the quietest bugs in search-as-you-type implementations is the stale response problem. A user types a query, the search fires, the user keeps typing before the first response arrives, a second search fires -- and then the first response arrives after the second. The UI renders results for the wrong query. The user sees results that do not match what they typed.

This bug is invisible in normal development conditions because the server is fast and local network latency is low. It surfaces in production under real-world latency, database load, or slower connections. Users report that search is "showing wrong results" or "flickering." The bug is real, it is predictable, and it is completely preventable.

At 137Foundry, we use AbortController to cancel in-flight requests before every new search fires. Here is how we implement it and what to watch out for.

Step 1: Understand the Race Condition

Before reaching for AbortController, it helps to be precise about what the race condition is.

Search-as-you-type fires a request as the user types. If the user types fast, or if network latency is variable, two requests can be in flight simultaneously. The responses arrive in an unpredictable order -- whichever backend query happens to complete first wins. If the first query (for "keyboa") completes after the second query (for "keyboard"), the UI ends up showing results for "keyboa" even though the user finished typing "keyboard."

This is the classic stale closure problem applied to asynchronous network requests. The solution is to cancel any in-flight request before starting a new one, so only one request is ever active at a time.

Step 2: Create the AbortController

The Web API for canceling fetch requests is AbortController. Each fetch call can be passed an AbortSignal. When the controller is aborted, any fetch using that signal is canceled immediately.

The setup is two lines:

const controller = new AbortController(); const { signal } = controller; const response = await fetch('/api/search?q=query', { signal });

Calling controller.abort() cancels the fetch. The promise rejects with a DOMException named AbortError.

Step 3: Wire It to the Search Input

In practice, we keep the controller in a module-level variable so we can cancel it from the next search call. The pattern looks like this:

let activeController = null; async function fetchResults(query) { if (activeController) { activeController.abort(); } activeController = new AbortController(); const response = await fetch(`/api/search?q=${encodeURIComponent(query)}`, { signal: activeController.signal, }); const data = await response.json(); return data.results; }

Every call to fetchResults cancels the previous request before starting a new one. Only the most recent request ever completes.

Step 4: Catch AbortErrors Without Treating Them as Failures

AbortErrors are intentional. Aborting a request is not an error in the application sense -- it is correct behavior. If you let AbortErrors propagate to your error handler, you will see false error logs, incorrect "search failed" states in the UI, and potentially broken analytics.

Catch AbortErrors separately and return silently:

async function handleSearch(query) { try { const results = await fetchResults(query); renderResults(results); } catch (err) { if (err.name === 'AbortError') return; showErrorState(); console.error('Search failed:', err); } }

The AbortError branch just returns. No UI update, no error log, no retry. The canceled request never had its results used, so there is nothing to clean up.

Step 5: Combine With Debounce

AbortController and debounce are complementary. They solve different problems.

Debounce reduces the number of requests fired. With a 300ms debounce, a user typing "keyboard" fires one request instead of eight. Debounce eliminates most of the race condition risk by simply not firing intermediate requests.

AbortController handles the race condition that survives debounce. Even with debounce, two requests can be in flight -- the debounce delay passes twice if the user types, pauses, then types again. AbortController ensures the first one is canceled when the second fires.

Together:

function debounce(fn, delay) { let timer; return (...args) => { clearTimeout(timer); timer = setTimeout(() => fn(...args), delay); }; } let activeController = null; async function search(query) { if (activeController) activeController.abort(); activeController = new AbortController(); try { const res = await fetch(`/api/search?q=${encodeURIComponent(query)}`, { signal: activeController.signal, }); const data = await res.json(); renderResults(data.results); } catch (err) { if (err.name === 'AbortError') return; showError(); } } const debouncedSearch = debounce(search, 300); searchInput.addEventListener('input', (e) => { const query = e.target.value.trim(); if (query.length < 2) { clearResults(); return; } debouncedSearch(query); });

This is the pattern we use for all server-side search implementations at 137Foundry. It is small, dependency-free, and provably correct for the race condition case.

Step 6: Test on a Slow Network

AbortController bugs and race conditions are invisible on fast local networks. To test that your implementation handles them correctly, use Chrome DevTools to simulate a slow connection.

Open DevTools, go to the Network tab, and change the throttling dropdown from "No throttling" to "Slow 3G" or a custom profile with 300-500ms latency. Then type a search query quickly. You will see the canceled requests appear in the network log with a red canceled status. Only the final request should resolve successfully and update the UI.

If the wrong results appear, or if the UI updates multiple times with different results as responses arrive, AbortController is not correctly wired up.

This test takes two minutes and makes the bug visible immediately. We run it on every search implementation before shipping.

When AbortController Is Not Needed

AbortController matters for server-based search. It is not needed for client-side search using Fuse.js, Lunr.js, or FlexSearch.

Client-side search runs synchronously in the browser. fuse.search(query) returns results immediately -- there is no network request, no in-flight state, and no possibility of a race condition. The debounce is still useful to avoid unnecessary DOM updates during fast typing, but AbortController adds nothing. If the dataset fits comfortably in browser memory, client-side search is often the right architecture: zero latency, no server load, and no concurrent request management needed.

If you switch from client-side search to a server-backed implementation later (because the dataset grew or you needed better relevance), adding AbortController at that point is the right time. It is a small addition and easy to retrofit.

What This Looks Like in a React Hook

For React projects, we encapsulate this in a custom hook:

function useSearch(fetchFn, delay = 300) { const [results, setResults] = useState([]); const controllerRef = useRef(null); const search = useCallback( debounce(async (query) => { if (controllerRef.current) controllerRef.current.abort(); controllerRef.current = new AbortController(); try { const data = await fetchFn(query, controllerRef.current.signal); setResults(data); } catch (err) { if (err.name === 'AbortError') return; setResults([]); } }, delay), [fetchFn, delay] ); return { results, search }; }

The hook manages the controller ref, handles cleanup, and exposes a search function that is already debounced and abort-safe. Components receive results and a search trigger without touching any of the concurrency logic.

The complete implementation walkthrough -- including how to connect this to different search backends and tune the debounce timing for different use cases -- is covered in the 137Foundry guide on building search-as-you-type. 137Foundry builds web applications with search as a first-class feature, from initial architecture through production relevance tuning.

[137-foundry]

Python Error Recovery Patterns for Long-Running Scripts

Long-running scripts fail differently than web requests. A web request fails in milliseconds and the next request starts fresh. A script that processes 50,000 records fails hours in, and the question is not just what went wrong but how much work was lost and how much needs to be redone.

At 137Foundry, we work on data pipelines and automation scripts that process large batches of records. The patterns here come from building those systems and watching what breaks in production.

The Problem With Failing Fast in Long-Running Jobs

For short operations, fail fast is the correct default. If an API call fails at the start of a web request, raise immediately and let the framework return an error response. Clean and predictable.

For long-running batch jobs, fail-fast means losing all progress on a partially-completed batch. If you are processing 10,000 records and record 8,743 causes an unhandled exception, you have lost the results for records 1 through 8,742. You also have no idea which records failed and which succeeded.

The alternative is record-level error isolation: catch exceptions at the per-record level, log them with enough context to identify the problem record, and continue processing the remaining records.

results = {"success": [], "failed": []} for record in batch: try: output = transform(record) results["success"].append(output) except Exception: log.exception("Transform failed on record %s", record.id) results["failed"].append(record.id) log.info("Batch complete: %d succeeded, %d failed", len(results["success"]), len(results["failed"]))

This is a deliberate trade-off. You are choosing to continue processing over failing loudly. The correctness condition is that each record's failure or success is fully captured before moving to the next one.

Distinguishing Retryable from Fatal Failures

Not all failures are the same. A database timeout at record 4,000 might succeed on retry. A record with a malformed date field will fail every time. Treating both identically -- either always retry or always skip -- is wrong.

A custom exception hierarchy makes the distinction explicit:

class RetryableError(Exception): """The operation failed but may succeed on retry.""" pass class FatalError(Exception): """The operation will not succeed on retry. Escalate.""" pass

The service layer raises the appropriate type. The batch processor catches them differently:

for record in batch: for attempt in range(MAX_RETRIES): try: output = process(record) results["success"].append(output) break except RetryableError as err: if attempt == MAX_RETRIES - 1: log.error("Max retries reached for %s: %s", record.id, err) results["failed"].append(record.id) else: time.sleep(2 ** attempt) # Exponential backoff except FatalError: log.exception("Fatal failure on record %s, skipping", record.id) results["failed"].append(record.id) break

The Python documentation covers the exception hierarchy and how custom exceptions should be defined. The Python Enhancement Proposals on peps.python.org -- specifically PEP 3134 -- describe exception chaining, which is how you preserve the original cause when wrapping exceptions in your service layer.

Checkpointing to Avoid Reprocessing Completed Work

For very long jobs, error isolation is not enough. If the entire process crashes after 8 hours of work, you still lose all progress. Checkpointing lets you resume from the last successful point rather than starting over.

The simplest checkpoint is a file or database table that records which records have been processed:

import json CHECKPOINT_FILE = "progress.json" def load_checkpoint(): try: with open(CHECKPOINT_FILE) as f: return set(json.load(f)["completed"]) except FileNotFoundError: return set() def save_checkpoint(completed_ids): with open(CHECKPOINT_FILE, "w") as f: json.dump({"completed": list(completed_ids)}, f) completed = load_checkpoint() for record in batch: if record.id in completed: continue # Already processed on a previous run try: output = process(record) completed.add(record.id) save_checkpoint(completed) except Exception: log.exception("Failed on record %s", record.id)

The checkpoint write frequency is a trade-off: write after every record for maximum resume granularity, write less often for better throughput. For most batch jobs, writing every 100 records is a reasonable balance.

Structured Failure Reports

After a batch completes, the failure records are only useful if they are in a format that makes remediation possible. Logging exception messages is a minimum. Persisting the failed record IDs to a file or table is better, because it enables re-running just the failed subset without modifying the main script.

if results["failed"]: with open("failed_records.txt", "w") as f: for record_id in results["failed"]: f.write(f"{record_id}\n") log.warning("%d records failed. See failed_records.txt", len(results["failed"]))

For re-runs, the batch loading code accepts an optional filter:

def load_batch(rerun_ids=None): if rerun_ids: return [r for r in all_records if r.id in rerun_ids] return all_records

This pattern -- isolate failures, record them, enable targeted reruns -- is one of the consistent elements in the data pipeline work we do at 137Foundry. The same structure applies whether the batch is 1,000 records or 10 million.

When Record-Level Isolation Is Not Appropriate

Record-level isolation is not always the right pattern. It is specifically appropriate when:

Each record can be processed independently, with no side effects between records.

A failure on one record leaves the system in a clean state for the next record.

Partial completion is acceptable and resumable.

It is not appropriate when records have dependencies. If record B references the output of record A, and record A failed, record B may appear to succeed but produce corrupt output based on a missing or stale dependency. In these cases, the dependency graph needs explicit representation in the batch design, and failures need to propagate to dependent records.

Similarly, record-level isolation does not protect against failures in setup or teardown code outside the loop. If the database connection drops mid-batch and you are catching DatabaseConnectionError inside the loop, every remaining record will fail with the same error. The isolation masks what is actually a systemic failure. A connection health check before the loop and a circuit breaker pattern to abort when failure rate spikes prevents the isolation from hiding a systemic outage.

Production Monitoring Considerations

Error isolation means individual failures do not crash the job, but it also means failures can accumulate silently. A batch that completes with 200 failed records looks like a success from the job runner's perspective if you are only checking the exit code.

Three additions address this:

Exit with a non-zero code if failure rate exceeds a threshold. A 2% failure rate might be acceptable; a 40% failure rate indicates a systemic problem.

failure_rate = len(results["failed"]) / len(batch) if failure_rate > 0.05: log.error("Failure rate %.1f%% exceeds threshold", failure_rate * 100) sys.exit(1)

Send an alert when failure counts exceed a threshold. The Sentry error monitoring platform integrates with Python logging to aggregate error counts and alert when error rates spike above configured thresholds.

Log a structured summary at job end. A final log entry with total records, success count, failure count, and elapsed time makes it trivial to build dashboards and detect regressions across job runs without parsing individual record-level entries.

log.info( "Batch %s complete: total=%d success=%d failed=%d elapsed=%.1fs", batch.id, len(batch), len(results["success"]), len(results["failed"]), time.time() - start_time )

The full reference for Python exception handling patterns -- including exception chaining, service boundary design, and testing failure paths with pytest -- is in the Python Error Handling article on the 137Foundry blog.

The patterns described here -- record-level isolation, retryable vs. fatal error distinction, checkpointing, and structured failure reporting -- are composable. A long-running job that is not important enough to checkpoint may still benefit from record-level isolation and a structured failure report. Choose the patterns that match the job's requirements for recovery, auditability, and rerunnability, rather than applying all of them uniformly.

[137-foundry]

How We Handle Errors in 137Foundry Data Projects

Error handling is one of those things that looks like plumbing -- boring and invisible until it breaks. At 137Foundry, we have a consistent approach to exception handling that we apply to every Python data pipeline and web application we build. This post describes what that approach looks like and why we settled on it.

We Start With a Custom Exception Hierarchy

Every project that has a service layer gets a custom exception hierarchy in the first commit. Not at refactor time, not after the first incident -- in the initial setup.

The hierarchy follows a predictable shape:

class ProjectError(Exception): """Base for all application-layer failures in this project.""" pass class RetryableError(ProjectError): """The operation failed but may succeed on retry.""" pass class PermanentError(ProjectError): """The operation will not succeed on retry. Escalate.""" pass class ValidationError(PermanentError): """Input data failed validation rules.""" def __init__(self, message, field=None, value=None): super().__init__(message) self.field = field self.value = value

The motivation for starting here is that it forces a conversation early about which failure modes exist and which require different responses. A RetryableError and a ValidationError need completely different handling. If you do not make the distinction in the type system, you make it in scattered if/else logic that is harder to maintain and easier to get wrong.

We Use log.exception() at Service Boundaries

Every service boundary in our projects -- API handlers, background job entry points, CLI commands -- has a consistent logging pattern for caught exceptions:

try: result = service.process(request_data) except ProjectError: log.exception("Service operation failed for request %s", request.id) raise

log.exception() captures the full traceback automatically. We do not use log.error("Failed: %s", err) at boundaries because the error message alone rarely contains enough information to diagnose the problem. The traceback tells us the exact call chain and the exact line. Without it, an incident investigation starts from scratch.

We re-raise after logging so that callers at higher levels get the exception and can respond to it with their own logic -- returning an HTTP 500, scheduling a retry, or escalating to an alert.

We Preserve Tracebacks With Chaining

When we catch a low-level exception and raise a domain exception, we always use raise X from Y:

try: raw = db.query(sql) except DatabaseConnectionError as err: raise RetryableError("Database temporarily unavailable") from err

The from err clause preserves the original DatabaseConnectionError in the traceback. When a developer reads the log, they see both exceptions and can trace the failure back to its source. Without chaining, the original error disappears and the traceback shows only the RetryableError, which is less useful for diagnosis.

The Python documentation at docs.python.org covers exception chaining, and PEP 3134 on peps.python.org explains the design intent behind __cause__ and __context__.

We Write Tests for Failure Paths First

In data projects, failure path tests are often the ones that prevent the most expensive bugs. A test that verifies the service raises RetryableError on database timeout ensures that the retry logic in the background job processor actually fires on timeout. Without that test, a change that accidentally swallows the timeout and returns None instead would not be caught until timeout caused silent data loss in production.

Our standard pattern for failure path tests uses pytest and mocker:

def test_raises_retryable_on_db_timeout(mocker): mocker.patch("project.db.query", side_effect=TimeoutError) with pytest.raises(RetryableError): service.fetch_records(batch_id) def test_validation_error_carries_field_name(): with pytest.raises(ValidationError) as exc_info: service.process_record({"amount": "not_a_number"}) assert exc_info.value.field == "amount"

The pytest documentation at docs.pytest.org covers pytest.raises, the match parameter for asserting on exception messages, and mocker.patch for simulating failures.

How We Handle Failures in Background Jobs

Background job processors have different error handling requirements than API endpoints. A web request must respond within a timeout; a job can fail and be retried. But the failure must be recorded and the retry must actually happen -- silent failures are just as dangerous in a job queue as in an API.

Our background job processors follow a consistent structure:

The job function catches RetryableError and marks the job for retry with appropriate backoff.

The job function catches PermanentError and marks the job as permanently failed, triggering an alert.

Any unexpected exception propagates to the job runner's outer handler, which marks it as failed and sends an immediate alert.

def process_job(job): try: result = service.process(job.data) job.mark_complete(result) except RetryableError as err: delay = min(2 ** job.attempt_count, 3600) job.schedule_retry(delay=delay) log.warning("Retryable failure on job %s, retry in %ds: %s", job.id, delay, err) except PermanentError: log.exception("Permanent failure on job %s", job.id) job.mark_failed() alerts.send(f"Job {job.id} permanently failed")

The except PermanentError handler does not catch RetryableError because it is not a subclass of PermanentError. Adding a new failure mode to the hierarchy does not require changing this handler unless the new type needs a distinct response.

The Value of Consistent Patterns Across a Codebase

Exception handling inconsistency compounds as a codebase grows. When some services raise ValueError, others raise Exception("message"), some return None and log an error, and some return (None, error_message) tuples, adding a new feature requires understanding the error contract for every function you call, because none of them are the same.

The consistent approach described here is not complex. Each piece is a few lines. The value accumulates because every developer working on the codebase -- including developers joining the project later -- can read any service and immediately understand its failure modes. The exception type tells them whether to retry, escalate, or discard. The logging tells them where to look. The test tells them the retry logic actually fires.

We have found this consistency to be particularly valuable during handoffs. When we finish a project and transfer it to a client's internal team, the new developers spend far less time understanding the error handling because it follows the same pattern everywhere. That is the return on the investment in establishing the convention at project start.

We Apply the Same Pattern to All Project Types

The same exception hierarchy and logging pattern applies whether we are building a REST API, a batch data pipeline, or a scheduled automation script. The boundary changes -- the API handler is the boundary in a web app, the job entry function is the boundary in a pipeline -- but the design is the same.

This consistency has a practical benefit: when we hand a project off to a client's internal team, the error handling is predictable throughout. Every exception type communicates a specific failure mode. Every boundary logs with log.exception(). Every service-to-service exception is chained. Developers who are new to the codebase can read the exception type and know whether to retry, escalate, or discard.

The full reference collection of Python error handling code snippets -- covering try-except-else-finally, exception chaining, contextlib.suppress, and testing patterns -- is on the 137Foundry blog. It is the reference we send to developers starting on a new project.

One thing we have consistently observed: the conversations that exception handling forces -- which failure modes exist, which require retries, which require escalation -- are valuable design conversations in their own right. Starting with the exception hierarchy is often the clearest way to make implicit failure assumptions explicit before writing the rest of the service logic. The exception types become documentation of what the service can fail with, and that documentation lives in the code where it remains accurate as the code changes.

[137-foundry]

What We've Learned Running Event-Driven Pipelines in Production

We've built and operated event-driven data pipelines at 137Foundry across a range of client contexts: SaaS platforms that need to sync records as they're created, e-commerce backends that process order events in real time, and analytics pipelines that need to reflect data changes in near real-time rather than overnight.

Most of the hard-won lessons aren't in the initial implementation -- they're in what breaks after the first month of production load. Here's the honest version.

The thing that bites you first: consumers crashing silently

The first event-driven pipeline we deployed had a consumer that occasionally died without a trace. No error in the logs. No alert. Just a growing queue depth we noticed only because we happened to look at the Redis dashboard for an unrelated reason.

The root cause: an unhandled exception in a code path that was only reachable with a specific message format we hadn't tested. The consumer process exited, the supervisor restarted it, but the messages it was processing at the time were gone -- Redis had already popped them from the queue before the crash.

We added confirmed delivery (BRPOPLPUSH processing list pattern for Redis consumers), explicit exception handling with structured logging, and queue depth monitoring that alerts when depth exceeds a threshold for more than 5 minutes. None of this was in the initial implementation because none of it seemed necessary until it was.

The lesson: event-driven pipelines fail differently than scheduled jobs. A scheduled job that crashes is visible in the cron log immediately. A consumer that crashes is invisible until you notice the queue is growing.

The dead-letter queue gap

Our second major production incident came from a pipeline that was processing webhook events for a client. One category of event had a schema change that wasn't communicated to us -- the webhook payload added a required field that our schema validation code didn't expect, causing every event of that type to fail. Celery retried three times, hit max retries, marked the tasks as failed, and moved on.

We discovered this 48 hours later when the client noticed that three days of events weren't reflected in their data. The fix was straightforward once we knew about it, but the 48-hour gap was a problem.

The change we made: all Celery tasks now route failure entries to a DLQ Redis list on on_failure. We have a lightweight monitoring script that checks DLQ depth every 5 minutes and pages us if it exceeds 10 entries. We also have a requeue script that inspects DLQ entries and re-dispatches them after we've fixed the underlying issue.

The DLQ setup takes about two hours to implement properly. The operational visibility it gives you is worth significantly more than that.

Message volume assumptions that don't survive production

In development, we test with small message volumes: 10 events, 100 events. Production systems don't send 10 events. They send 10,000 events in a burst when something triggers a batch operation -- an import, a migration, a scheduled sync.

The first time we hit a burst like this, our single-consumer pipeline fell 6 hours behind real time before we noticed. The queue had backed up to 50,000 messages. Adding consumers helped, but the burst had already happened and the catch-up process was slow.

We now size consumer pools based on peak volume, not average volume. If the maximum realistic burst is 10,000 messages in 10 minutes, we configure enough consumers to handle that rate. This means running more workers than necessary most of the time, which costs money, but it's cheaper than a 6-hour lag when a client is watching.

We've also added circuit breakers on downstream dependencies (databases, APIs) so that a slow downstream doesn't cause the queue to back up indefinitely -- it causes the consumers to pause and retry, which is a different (and more manageable) failure mode.

The idempotency debt we paid later

Early in our event-driven pipeline development, we didn't implement idempotent processing because our broker (Celery with Redis) was delivering messages exactly once in practice. No duplicates, no issue.

Then we migrated one pipeline to RabbitMQ for durability. RabbitMQ's at-least-once delivery guarantee is not a theoretical concern -- it's actual behavior. Consumer acknowledgment after processing means that if the consumer crashes between processing and acknowledgment, the message is redelivered. We hadn't designed for this.

We spent a weekend adding deduplication to three pipelines. Each one required a "seen message IDs" set in Redis and a check at consumer entry. Not complex, but the refactor touched a lot of code paths we thought were stable.

The right approach is idempotent consumers from day one, regardless of which broker you're using. The cost of adding it at the start is much lower than retrofitting it.

The monitoring gap we kept running into

Each incident above taught us something about visibility. After the silent consumer crash, we added queue depth monitoring. After the DLQ incident, we added DLQ depth alerting. After the burst volume incident, we added consumer count dashboards. After the idempotency weekend, we added deduplication coverage to every new pipeline from day one.

What's notable in hindsight: every piece of monitoring we added came after a specific failure. None of it was in place before the first incident that made it necessary. That's a pattern we eventually broke by treating monitoring as a day-one requirement rather than a response to incidents.

Here is the baseline we now deploy with every pipeline before the first production traffic:

Queue depth with trend alerting. We scrape queue depth every 30 seconds. Alert when depth exceeds a threshold AND hasn't decreased in 5 minutes. Spikes that drain naturally are fine. Depth that grows and stays grown means something is wrong.

DLQ depth with escalating alerts. A script checks DLQ length every 5 minutes. Alert at 10 entries, page at 50. The threshold review happens every quarter -- pipelines that stabilize usually get tighter thresholds over time.

Consumer heartbeat monitoring. Celery Flower gives us a real-time view of worker status and last heartbeat time. A worker that hasn't checked in for twice the expected interval gets flagged automatically.

Structured consumer logs. We log task start, task complete, and task error as separate structured events, with task ID and event type in every entry. This makes "tasks that started but never completed" a queryable condition rather than something you discover by noticing the queue isn't draining three hours later.

Process supervisor alerts. Workers run under Supervisor or systemd. Crash loops -- more than three restarts in a short window -- generate alerts to the same channel as DLQ depth. The restart log tells us when the loop started and how frequently it happened.

None of this setup is complex. The overhead is a few hours per pipeline. The value is finding problems when they're small rather than after they've been accumulating unnoticed for hours.

What we'd do differently from the start

If we were starting a new event-driven pipeline today, the decisions we'd make differently:

DLQ from day one. Not after the first incident. Every task in Celery gets a base class with on_failure that routes to a DLQ Redis list.

Queue depth alerting in week one. Not as an afterthought after we've shipped the pipeline.

Idempotent consumers from the start. The at-most-once vs. at-least-once assumption is a time bomb when the broker or configuration changes.

Burst capacity planning. Size for peak, not average.

Schema validation at consumer entry. Malformed messages should fail fast with a clear error, not propagate through processing logic.

The technical implementation for these patterns is what we documented in "How to Build an Event-Driven Data Pipeline With Python and Message Queues" -- that guide reflects what we actually do, not what looked good in the initial design.

The hardest part of event-driven architecture isn't the initial implementation. It's building in the operational visibility and failure handling that makes the difference between a pipeline that works and one that works reliably. We got to "works reliably" the slow way; hopefully this helps you skip some of those incidents.

[137-foundry]

How We Monitor Queue Health in Our Data Automation Stack

Queue monitoring is the thing you build after your first production incident, not before it. We learned this the hard way -- a consumer died silently, the queue backed up for two hours, and we found out from a client asking why their data wasn't updating. After that, monitoring went into every pipeline from day one.

Here is the exact setup we use.

Step 1: Track Queue Depth as Your Primary Metric

Queue depth is the most important health indicator for any event-driven pipeline. It's the number of messages waiting for a consumer. A healthy pipeline keeps depth near zero. Depth that grows without recovering means consumers cannot keep pace with producers.

For Redis-backed pipelines, get queue depth directly:

redis-cli LLEN your_queue_name

For Celery, active and reserved task counts come from the inspect API:

from celery.app.control import Inspect i = app.control.inspect() active = i.active() # tasks currently executing reserved = i.reserved() # tasks waiting to execute

We push these values to our metrics backend every 30 seconds. For paid client projects we use Datadog, which has a Redis integration that tracks list length natively. For self-hosted setups we use Grafana with the Prometheus Redis Exporter -- it scrapes Redis metrics and exposes them as Prometheus time series.

Both tools let you set threshold alerts: when queue depth exceeds N for more than X minutes, send a notification. We use a 5-minute window to reduce alert fatigue from short spikes that drain naturally.

The important threshold to set: not just "depth above N" but also "depth has not decreased in X minutes." A queue that has been at 5,000 entries for 20 minutes is more concerning than one that briefly hit 5,000 and is draining. The second alert -- no decrease over time -- catches stuck consumers that aren't crashing but also aren't making progress.

Step 2: Monitor DLQ Depth With Alerting

Every pipeline we build routes failed tasks to a dead-letter queue: a Redis list called celery_dlq where tasks end up after exhausting all retries. Monitoring that list is as important as monitoring the main queue.

Our DLQ depth check runs on a 5-minute interval:

import redis r = redis.Redis(host='your_redis_host', port=6379, db=0) def check_dlq(): depth = r.llen("celery_dlq") if depth > 10: severity = "critical" if depth >= 50 else "warning" send_alert( title=f"DLQ alert ({severity})", message=f"celery_dlq depth is {depth}", severity=severity ) return depth

The send_alert() implementation varies by client: Slack webhook, PagerDuty API, or email. The specifics don't matter. What matters is that a real person, who is empowered to fix the underlying issue, sees the alert and can act on it quickly.

We alert at 10 entries and escalate at 50. Ten entries usually means a category of messages is failing consistently -- a schema change, a dependency outage, a new code path that wasn't tested with production data. Fifty entries usually means the initial alert was missed or the issue has been going on long enough to accumulate significant failures.

A DLQ that is never checked is only slightly better than no DLQ. The monitoring has to loop back to a human who can identify the root cause, fix it, and requeue the affected messages.

Step 3: Check Consumer Health Beyond Queue Depth

An empty queue can mean two completely different things: all messages were processed (healthy), or consumers are down and no new messages are arriving (broken). Queue depth alone doesn't distinguish between them.

To detect "consumers are down," we track three things:

Consumer heartbeats. Celery workers emit heartbeats on a configurable interval. A worker that hasn't emitted a heartbeat in twice the expected interval is probably down. Flower, Celery's built-in monitoring interface, displays worker status and last-seen time. We run Flower on all production deployments and include it in our standard pipeline setup checklist.

Task completion rate. Even when queue depth is low, a drop in the rate at which tasks complete is worth investigating. If the completion rate drops to zero, something is wrong even if the queue looks healthy. Datadog's Celery integration tracks task events including success and failure rates by task name.

Process supervision. We run Celery workers under Supervisor or as systemd services. Both handle crash detection and automatic restarts. We configure Supervisor to alert after three consecutive restart attempts in a short window:

[program:celery_worker] command=celery -A myapp worker -l info -Q default autostart=true autorestart=true startretries=3 redirect_stderr=true stdout_logfile=/var/log/celery/worker.log

A worker in STOPPED state after hitting startretries is what generates the page-out. The restart log entries tell us when the crash loop started and how frequently it happened, which is usually enough to identify the cause.

Step 4: Add Structured Logging at Consumer Entry and Exit

Queue metrics tell you what is happening at the infrastructure level. Consumer logs tell you what is happening at the individual task level. Both matter, and they're measuring different things.

We add structured logging at three points in every consumer task:

import logging import time logger = logging.getLogger(__name__) @app.task(bind=True, max_retries=3) def process_event(self, event_data: dict): task_id = self.request.id start = time.monotonic() logger.info("task_start", extra={ "task_id": task_id, "event_type": event_data.get("type") }) try: result = do_processing(event_data) duration_ms = int((time.monotonic() - start) * 1000) logger.info("task_complete", extra={ "task_id": task_id, "duration_ms": duration_ms }) return result except Exception as exc: logger.exception("task_error", extra={ "task_id": task_id, "exception": str(exc) }) raise

This produces queryable log lines. You can find tasks that started but never completed -- which indicates a crash between entry and success. You can find tasks taking longer than expected, which indicates a slow downstream dependency. You can filter by event_type to identify which categories of events are failing.

We aggregate logs to Datadog Logs for paid client projects and to a self-hosted ELK stack (Elasticsearch, Logstash, Kibana) for lower-budget setups. The aggregation platform doesn't matter as much as the structured format: unstructured logs are much harder to query at 3am when you're debugging a production incident.

Step 5: Trim the DLQ Before It Grows Unbounded

A DLQ that grows indefinitely will eventually exhaust Redis memory for high-volume pipelines. We cap DLQ size with an LTRIM call after every write:

MAX_DLQ_ENTRIES = 1000 @task_failure.connect def handle_task_failure(sender=None, task_id=None, exception=None, args=None, kwargs=None, **kw): entry = build_dlq_entry(sender, task_id, exception, args, kwargs) r.rpush("celery_dlq", json.dumps(entry)) r.ltrim("celery_dlq", -MAX_DLQ_ENTRIES, -1)

LTRIM with -MAX_DLQ_ENTRIES, -1 keeps only the 1,000 most recent entries and discards older ones. If more than 1,000 tasks have failed without anyone reviewing the DLQ, the operational problem is the lack of response to alerts, not the DLQ capacity. The most recent entries are the most useful for diagnosing the current failure condition.

When you fix the underlying issue and want to requeue the DLQ contents, you inspect the entries and re-dispatch them as Celery tasks before clearing the list. The details of that requeue script are in the DLQ setup guide.

What This Monitoring Catches (and What It Doesn't)

The five checks above cover the most common production failures: consumer crashes, consumers falling behind burst volume, dead-letter buildup from data issues or service outages. In our experience these account for the majority of pipeline incidents.

What this monitoring does not catch: silent data corruption where the consumer completes successfully but writes incorrect values, producer failures that stop emitting events entirely, and downstream errors that are caught and swallowed quietly in application code rather than propagated. For those failure modes you need application-layer monitoring: data freshness checks that verify output is being produced at the expected rate, and integration tests that trace a synthetic event through the full pipeline end to end.

Queue monitoring is necessary but not sufficient. The setup described here is what 137Foundry's data automation team includes as a default on every new pipeline deployment and retrofits onto existing ones after a production incident or infrastructure migration. The full context for how these monitoring patterns connect to producer-consumer architecture decisions is in "How to Build an Event-Driven Data Pipeline With Python and Message Queues".

Good monitoring is a handful of checks, run on a schedule, alerting to the right people. The overhead is low. Finding a problem at 10 DLQ entries rather than at 10,000 is the entire point.

[137-foundry]

Why We Built Our Feature Flag System Instead of Paying for LaunchDarkly

At 137Foundry, we have evaluated a lot of tools in the feature management category over the years. LaunchDarkly, Unleash, Flagsmith, Split -- we have used several of these with clients and built with them internally. After running both the managed-platform path and the DIY path on multiple production projects, we consistently land in the same place for most applications: the custom implementation wins until the team hits 30 to 50 engineers actively using flags.

Here is our reasoning, and the specific cases where we change the answer.

The Managed Platform Pitch Is Real But Oversold

The value proposition for platforms like LaunchDarkly is genuine. You get a management dashboard, audit logs, per-user targeting, multi-environment support, SDK integrations for every major language, and a reliable API. For an enterprise team shipping to production 20 times a day, that operational tooling has clear value.

The pitch oversells, though, when it is directed at small and mid-size teams with straightforward rollout needs. Most applications need three things from a feature flag system: turn new code on and off, roll out to a percentage of users, and do both without a redeploy. Those three things take one database table and 50 lines of application code. They do not require a SaaS contract.

The real cost of a managed platform is not just the subscription fee -- it is the dependency. When you adopt LaunchDarkly's SDK, flag evaluation happens inside their library and their network requests. The flag system stops being your infrastructure and becomes an external service your application depends on. For flags that control critical application paths, that dependency matters.

What We Built and Why It Was the Right Call

For a recent client project we built at 137Foundry -- a mid-size SaaS application with a team of 8 engineers -- we implemented a custom feature flag system as part of the core application infrastructure. The full implementation: one PostgreSQL table, a 60-line Node.js evaluator with hash-based percentage bucketing, a Redis cache layer, and a React hook for client-side flag consumption.

Total implementation time: about 6 hours including tests and code review.

Total ongoing maintenance burden over 12 months: low. The system works, flags are created and retired on a schedule, and the whole thing is visible in the application codebase rather than in an external dashboard.

The complete implementation we use for these projects is documented on the 137Foundry blog. It covers the data model, the evaluator pattern (including the deterministic hash that prevents inconsistent per-user flag state), and the React integration. If you are building a new application or adding flags to an existing one, that guide is the starting point we give our own engineers.

What 12 Months of Custom Flag Operations Actually Looked Like

After shipping the custom flag system for that mid-size SaaS client project, the team used it in production for over a year. Here is what the operational reality looked like.

Flag volume stayed manageable. At the 12-month mark, the flags table had 23 rows. Fourteen were actively used, six had been permanently enabled and were candidates for cleanup, and three were operational toggles that would stay indefinitely. Managing 23 rows in a database is trivially easy. A managed platform would not have changed this.

Flag creation took two minutes. A SQL INSERT, a comment in the Slack deploy channel, and a calendar reminder for cleanup. The absence of a UI dashboard was not friction at this team size.

One rollback saved significant user impact. In month eight, a flag controlled a new billing calculation. At 5 percent rollout, the team noticed an edge case producing incorrect invoice totals for monthly-to-annual subscription upgrades. Rolling back was a single SQL UPDATE. The whole incident -- detection to resolution -- took eleven minutes. Without the flag, that bug would have hit all users simultaneously and required an emergency deploy to fix.

The only real limitation appeared at month ten. A new engineer on the project wanted to enable a flag for testing in their development environment without touching the shared staging database. The custom system had no per-environment flag isolation -- flags were global across all environments sharing the same database. The workaround was a development-only flag override in a local .env file. It worked, but it was friction that a managed platform would have solved natively.

That one limitation -- per-environment flag isolation -- is the most common trigger we see for teams moving to Flagsmith or LaunchDarkly. It is a real operational need, and the workaround is less clean than a proper solution. For teams that hit it, Flagsmith self-hosted is the most efficient path to resolving it without committing to a SaaS subscription.

The Hash-Bucketing Detail That Most Tutorials Skip

One thing we see wrong in almost every DIY feature flag tutorial is the percentage rollout implementation. The common mistake: generate a random number between 0 and 99 on each evaluation and check if it falls below the rollout percentage. This means the same user might see a feature on page load and not see it on the next API call -- fundamentally broken behavior.

The correct approach is to hash a stable input (flag key + user ID) to produce a deterministic bucket. Same inputs, same output, same flag state -- every time, for every user.

function stableHash(input) { let h = 0; for (let i = 0; i < input.length; i++) h = (h * 31 + input.charCodeAt(i)) >>> 0; return h % 100; }

This is a small implementation detail that has a large user-experience impact. Without it, 10 percent rollouts produce inconsistent behavior that looks like a bug. We have debugged this exact issue on client projects that used naive random-number implementations before coming to us.

When the Calculus Changes

We change our recommendation from "build" to "buy" in two scenarios:

The team is large and ships frequently. At 30+ engineers shipping to production multiple times a day, the absence of a management dashboard becomes a real operational friction. Engineers need to create and modify flags without database access, flag history needs to be auditable, and per-user targeting requirements tend to become more sophisticated. That is when LaunchDarkly or Flagsmith (self-hosted, for cost reasons) earns its place.

Experimentation is a primary use case. If the team is running A/B tests with statistical significance measurement as part of their growth workflow, a purpose-built experimentation platform like Split.io is meaningfully better than a custom flags table. The experiment analysis tools alone justify the cost at that point.

For everyone else -- and that is most development teams we work with -- the custom approach is the right default. Start there, understand your system, and reach for a managed platform when the operational complexity genuinely justifies it.

The Recommendation

If you are building a new web application or adding feature flags to an existing one, implement the custom approach first. Use the build guide on our site as the starting point. Set a review date at 6 months. If the flag system is causing friction, evaluate managed options at that point.

If you are inheriting an existing application with no flag system and you are under time pressure, Flagsmith self-hosted is the fastest path to a working management UI with reasonable self-hosting costs. LaunchDarkly is the right call if the team is large, compliance requirements are strict, and the subscription cost is not a constraint.

We are happy to dig into the specifics for any application architecture -- reach out through the app development services at 137Foundry if you are weighing these options.

[137-foundry]

How We Roll Out Features to a Percentage of Users Before Full Launch

One of the most valuable deployment practices we use at 137Foundry is percentage-based feature rollouts. Instead of flipping a new feature on for all users at once, we start with 5 to 10 percent, watch for errors and user behavior changes, and expand gradually over a few days. This approach has caught several bugs in production before they affected the majority of users -- and it has given us confidence to ship more frequently.

Here is the exact process we use, including how we set up the rollout, what we watch for, and how we decide when to expand.

The Foundation: Hash-Based Bucketing

Percentage rollouts sound simple but have a common implementation pitfall. The naive approach is to generate a random number on each flag evaluation and check if it falls under the target percentage. This produces inconsistent behavior -- the same user sees the feature on some requests and not others -- which creates confusing UI states and makes bugs hard to reproduce.

We use hash-based bucketing instead. We hash the combination of the flag key and the user ID to produce a deterministic integer in the 0 to 99 range. The same user always falls in the same bucket for the same flag, so their experience is consistent across every request.

function stableHash(input) { let h = 0; for (let i = 0; i < input.length; i++) h = (h * 31 + input.charCodeAt(i)) >>> 0; return h % 100; } // User is in the rollout if their bucket falls below the rollout_pct const bucket = stableHash(`${flagKey}:${userId}`); const isActive = is_enabled && (rollout_pct >= 100 || bucket < rollout_pct);

Using the flag key as part of the hash input ensures that different flags distribute users independently. If you hash only the user ID, every flag rolls out to the same 10 percent of users -- which means the same users are always in every experiment and rollout, which is not representative sampling.

The full implementation guide on our site covers this evaluator in context alongside the database schema and React hook.

Our Rollout Schedule

For most new features, we follow a tiered schedule:

Day 1: 5 percent. This catches obvious breaking bugs before they affect a significant user base. We watch error rates and any feature-specific metrics for the first 24 hours.

Day 3 to 5: 25 percent. If no issues surface at 5 percent, we expand. At 25 percent we have a meaningful sample for behavioral data -- do users interact with the new feature the way we expected?

Day 7 to 10: 100 percent. If metrics look healthy and no regressions appear, full rollout. The flag stays in place for another two to four weeks while we monitor, then we remove the flag and the branching code together.

For higher-risk changes -- database schema changes, payment flow modifications, authentication changes -- we start at 1 percent and hold longer at each tier.

What We Watch During Rollout

Percentage rollouts are only useful if you are actively monitoring the right signals. We watch three categories:

Error rates by flag bucket. Sentry (or your error tracking tool) should be segmented to show errors from flag-enabled users separately from control users. A spike in errors for the enabled group that does not appear in the control group is a clear signal to pause the rollout.

Performance metrics. New features sometimes have performance regressions that are not obvious in development -- additional database queries, synchronous operations that should be async, rendering overhead in client-side components. We check response time distributions for flagged endpoints before expanding to higher percentages.

User behavior changes. For UI changes, we watch session recordings and funnel conversion rates for the enabled group. If users are getting lost or abandoning flows at higher rates, that is a signal to revisit the UX before expanding.

Redis with a short TTL on flag values makes it easy to adjust rollout percentages quickly when you need to pause a rollout or expand faster than scheduled. A database update plus a cache flush propagates the change within 30 seconds without a redeploy.

When to Pause or Rollback

Rollback on any of the following:

Error rate for enabled users exceeds the control group error rate by more than 0.5 percentage points

P95 response time for affected endpoints increases by more than 20 percent

Any critical user path (checkout, authentication, billing) shows conversion rate degradation in the enabled group

Rollback is a database update to set is_enabled = false or rollout_pct = 0 for the flag. No code deploy, no coordination required. The flag state propagates in under a minute with caching in place.

The ability to rollback in under a minute without a code deploy is one of the main reasons we use feature flags on every significant deployment at 137Foundry. It changes the risk calculus for shipping: if something goes wrong, the blast radius is limited to the rollout percentage, and recovery is immediate.

Removing the Flag After Full Launch

When you reach 100 percent and the metrics look good, set a calendar reminder for 2 to 4 weeks out to remove the flag. A flag that stays in the codebase past its useful life becomes technical debt -- the conditional logic stays in the code review history and confuses future developers, and the database row lingers in the flags table with no clear purpose.

The cleanup process: remove the conditional code in the application, delete the database row, update any monitoring rules that were segmented by flag bucket. Clean code is one of the operating principles we emphasize in the development services at 137Foundry -- flags included.

For teams evaluating managed flag platforms, Flagsmith has a built-in flag lifecycle tracker that helps automate the cleanup reminder step. LaunchDarkly has similar tooling. Both are worth evaluating when the flag management overhead of a custom system starts to exceed the value of the DIY approach.

Coordinating Rollouts Across Multiple Services

One complexity that appears in more mature applications: flag-controlled features that span multiple services. When a new checkout flow involves both a frontend component and a backend API endpoint, both services need to evaluate the flag the same way for the same user.

The approach we use: evaluate the flag on the server and pass the result explicitly to all layers, rather than having each layer evaluate independently. The request handler evaluates isEnabled('new_checkout_v2', userId) once, and the result flows through as a standard boolean through the response context or as a query parameter in service-to-service calls. This ensures consistency and makes the flag evaluation auditable at a single point.

When services share the same Redis cache layer, we configure both to read from the same flag cache namespace. Changes to the flag state propagate simultaneously across both services within one cache TTL window rather than potentially resolving at different times.

For teams using microservices where shared Redis is not practical, passing flag state explicitly in request context headers is the cleanest option. This prevents the same user from seeing inconsistent behavior across services and keeps flag evaluation logic isolated to the service that handles user sessions.

The Bottom Line

Percentage rollouts catch bugs before they affect your full user base, give you behavioral data on new features before committing to them fully, and provide a quick rollback path if something goes wrong. The implementation is not complex -- a deterministic hash, a database update, and a monitoring setup.

We use this pattern on almost every significant feature deployment at 137Foundry. The investment in the tooling is small; the operational confidence it provides is significant.

[137-foundry]

Hello from 137Foundry

We're 137Foundry — a small studio that builds web applications, AI integrations, and the kind of internal automation that makes a business actually run.

This Tumblr is the team blog. The main site at 137foundry.com covers our services and case studies in their finished form; this is the looser space — half-shaped ideas, working notes, problems we hit in client work and how we ended up solving them. The kind of thing that doesn't quite fit on the main blog yet but is too interesting not to write down.

A rough idea of what to expect here:

Engineering notes — odd bugs, unexpected platform behaviors, things we learned from a postmortem

AI and automation perspectives — what we're actually shipping in client work, what's overhyped, what's genuinely useful

Hot takes — opinions on web dev tooling, framework drama, the state of the developer ecosystem

Behind-the-scenes — process posts about how we scope projects, write proposals, structure engagements

We post when we have something to share, not on a schedule. If you're a founder or operator thinking about a build, the front door is the main site or our about page — happy to talk.

If you're a fellow practitioner, follow along and argue with us. We learn faster that way.