seen from Australia
seen from Sweden
seen from Russia
seen from China
seen from United States
seen from United States
seen from United States
seen from United States
seen from United States
seen from Russia
seen from Japan
seen from China
seen from China
seen from United States
seen from New Zealand
seen from Netherlands
seen from United States
seen from United States
seen from United States
seen from Macao SAR China
Interface Engineer En & EW
Job title: Interface Engineer En & EW Company: Mactech Energy Group Job description: Interface Engineer – En & EW 1210ADS Based in our London office with hybrid working available Closing date: 8th… on from the success of Hinkley Point C (HPC), the SZC Project is a nuclear new build project in Suffolk, which has obtained… Expected salary: £620 per day Location: London Job date: Fri, 27 Jun 2025…
The Future of Software Engineering: What 2025 Holds
The landscape of software engineering is rapidly evolving. It is shaping up to be as exciting as it is transformative. Gone are the days when coding was solely the domain of dedicated programmers tucked away in dimly lit offices. Today, we’re entering a new era. Artificial intelligence is changing how we approach software development. Real-time computing is also a key factor. Additionally, the…
Python Error Recovery Patterns for Long-Running Scripts
Long-running scripts fail differently than web requests. A web request fails in milliseconds and the next request starts fresh. A script that processes 50,000 records fails hours in, and the question is not just what went wrong but how much work was lost and how much needs to be redone.
At 137Foundry, we work on data pipelines and automation scripts that process large batches of records. The patterns here come from building those systems and watching what breaks in production.
The Problem With Failing Fast in Long-Running Jobs
For short operations, fail fast is the correct default. If an API call fails at the start of a web request, raise immediately and let the framework return an error response. Clean and predictable.
For long-running batch jobs, fail-fast means losing all progress on a partially-completed batch. If you are processing 10,000 records and record 8,743 causes an unhandled exception, you have lost the results for records 1 through 8,742. You also have no idea which records failed and which succeeded.
The alternative is record-level error isolation: catch exceptions at the per-record level, log them with enough context to identify the problem record, and continue processing the remaining records.
results = {"success": [], "failed": []} for record in batch: try: output = transform(record) results["success"].append(output) except Exception: log.exception("Transform failed on record %s", record.id) results["failed"].append(record.id) log.info("Batch complete: %d succeeded, %d failed", len(results["success"]), len(results["failed"]))
This is a deliberate trade-off. You are choosing to continue processing over failing loudly. The correctness condition is that each record's failure or success is fully captured before moving to the next one.
Distinguishing Retryable from Fatal Failures
Not all failures are the same. A database timeout at record 4,000 might succeed on retry. A record with a malformed date field will fail every time. Treating both identically -- either always retry or always skip -- is wrong.
A custom exception hierarchy makes the distinction explicit:
class RetryableError(Exception): """The operation failed but may succeed on retry.""" pass class FatalError(Exception): """The operation will not succeed on retry. Escalate.""" pass
The service layer raises the appropriate type. The batch processor catches them differently:
for record in batch: for attempt in range(MAX_RETRIES): try: output = process(record) results["success"].append(output) break except RetryableError as err: if attempt == MAX_RETRIES - 1: log.error("Max retries reached for %s: %s", record.id, err) results["failed"].append(record.id) else: time.sleep(2 ** attempt) # Exponential backoff except FatalError: log.exception("Fatal failure on record %s, skipping", record.id) results["failed"].append(record.id) break
The Python documentation covers the exception hierarchy and how custom exceptions should be defined. The Python Enhancement Proposals on peps.python.org -- specifically PEP 3134 -- describe exception chaining, which is how you preserve the original cause when wrapping exceptions in your service layer.
Checkpointing to Avoid Reprocessing Completed Work
For very long jobs, error isolation is not enough. If the entire process crashes after 8 hours of work, you still lose all progress. Checkpointing lets you resume from the last successful point rather than starting over.
The simplest checkpoint is a file or database table that records which records have been processed:
import json CHECKPOINT_FILE = "progress.json" def load_checkpoint(): try: with open(CHECKPOINT_FILE) as f: return set(json.load(f)["completed"]) except FileNotFoundError: return set() def save_checkpoint(completed_ids): with open(CHECKPOINT_FILE, "w") as f: json.dump({"completed": list(completed_ids)}, f) completed = load_checkpoint() for record in batch: if record.id in completed: continue # Already processed on a previous run try: output = process(record) completed.add(record.id) save_checkpoint(completed) except Exception: log.exception("Failed on record %s", record.id)
The checkpoint write frequency is a trade-off: write after every record for maximum resume granularity, write less often for better throughput. For most batch jobs, writing every 100 records is a reasonable balance.
Structured Failure Reports
After a batch completes, the failure records are only useful if they are in a format that makes remediation possible. Logging exception messages is a minimum. Persisting the failed record IDs to a file or table is better, because it enables re-running just the failed subset without modifying the main script.
if results["failed"]: with open("failed_records.txt", "w") as f: for record_id in results["failed"]: f.write(f"{record_id}\n") log.warning("%d records failed. See failed_records.txt", len(results["failed"]))
For re-runs, the batch loading code accepts an optional filter:
def load_batch(rerun_ids=None): if rerun_ids: return [r for r in all_records if r.id in rerun_ids] return all_records
This pattern -- isolate failures, record them, enable targeted reruns -- is one of the consistent elements in the data pipeline work we do at 137Foundry. The same structure applies whether the batch is 1,000 records or 10 million.
When Record-Level Isolation Is Not Appropriate
Record-level isolation is not always the right pattern. It is specifically appropriate when:
Each record can be processed independently, with no side effects between records.
A failure on one record leaves the system in a clean state for the next record.
Partial completion is acceptable and resumable.
It is not appropriate when records have dependencies. If record B references the output of record A, and record A failed, record B may appear to succeed but produce corrupt output based on a missing or stale dependency. In these cases, the dependency graph needs explicit representation in the batch design, and failures need to propagate to dependent records.
Similarly, record-level isolation does not protect against failures in setup or teardown code outside the loop. If the database connection drops mid-batch and you are catching DatabaseConnectionError inside the loop, every remaining record will fail with the same error. The isolation masks what is actually a systemic failure. A connection health check before the loop and a circuit breaker pattern to abort when failure rate spikes prevents the isolation from hiding a systemic outage.
Production Monitoring Considerations
Error isolation means individual failures do not crash the job, but it also means failures can accumulate silently. A batch that completes with 200 failed records looks like a success from the job runner's perspective if you are only checking the exit code.
Three additions address this:
Exit with a non-zero code if failure rate exceeds a threshold. A 2% failure rate might be acceptable; a 40% failure rate indicates a systemic problem.
failure_rate = len(results["failed"]) / len(batch) if failure_rate > 0.05: log.error("Failure rate %.1f%% exceeds threshold", failure_rate * 100) sys.exit(1)
Send an alert when failure counts exceed a threshold. The Sentry error monitoring platform integrates with Python logging to aggregate error counts and alert when error rates spike above configured thresholds.
Log a structured summary at job end. A final log entry with total records, success count, failure count, and elapsed time makes it trivial to build dashboards and detect regressions across job runs without parsing individual record-level entries.
log.info( "Batch %s complete: total=%d success=%d failed=%d elapsed=%.1fs", batch.id, len(batch), len(results["success"]), len(results["failed"]), time.time() - start_time )
The full reference for Python exception handling patterns -- including exception chaining, service boundary design, and testing failure paths with pytest -- is in the Python Error Handling article on the 137Foundry blog.
The patterns described here -- record-level isolation, retryable vs. fatal error distinction, checkpointing, and structured failure reporting -- are composable. A long-running job that is not important enough to checkpoint may still benefit from record-level isolation and a structured failure report. Choose the patterns that match the job's requirements for recovery, auditability, and rerunnability, rather than applying all of them uniformly.
How We Handle Errors in 137Foundry Data Projects
Error handling is one of those things that looks like plumbing -- boring and invisible until it breaks. At 137Foundry, we have a consistent approach to exception handling that we apply to every Python data pipeline and web application we build. This post describes what that approach looks like and why we settled on it.
We Start With a Custom Exception Hierarchy
Every project that has a service layer gets a custom exception hierarchy in the first commit. Not at refactor time, not after the first incident -- in the initial setup.
The hierarchy follows a predictable shape:
class ProjectError(Exception): """Base for all application-layer failures in this project.""" pass class RetryableError(ProjectError): """The operation failed but may succeed on retry.""" pass class PermanentError(ProjectError): """The operation will not succeed on retry. Escalate.""" pass class ValidationError(PermanentError): """Input data failed validation rules.""" def __init__(self, message, field=None, value=None): super().__init__(message) self.field = field self.value = value
The motivation for starting here is that it forces a conversation early about which failure modes exist and which require different responses. A RetryableError and a ValidationError need completely different handling. If you do not make the distinction in the type system, you make it in scattered if/else logic that is harder to maintain and easier to get wrong.
We Use log.exception() at Service Boundaries
Every service boundary in our projects -- API handlers, background job entry points, CLI commands -- has a consistent logging pattern for caught exceptions:
try: result = service.process(request_data) except ProjectError: log.exception("Service operation failed for request %s", request.id) raise
log.exception() captures the full traceback automatically. We do not use log.error("Failed: %s", err) at boundaries because the error message alone rarely contains enough information to diagnose the problem. The traceback tells us the exact call chain and the exact line. Without it, an incident investigation starts from scratch.
We re-raise after logging so that callers at higher levels get the exception and can respond to it with their own logic -- returning an HTTP 500, scheduling a retry, or escalating to an alert.
We Preserve Tracebacks With Chaining
When we catch a low-level exception and raise a domain exception, we always use raise X from Y:
try: raw = db.query(sql) except DatabaseConnectionError as err: raise RetryableError("Database temporarily unavailable") from err
The from err clause preserves the original DatabaseConnectionError in the traceback. When a developer reads the log, they see both exceptions and can trace the failure back to its source. Without chaining, the original error disappears and the traceback shows only the RetryableError, which is less useful for diagnosis.
The Python documentation at docs.python.org covers exception chaining, and PEP 3134 on peps.python.org explains the design intent behind __cause__ and __context__.
We Write Tests for Failure Paths First
In data projects, failure path tests are often the ones that prevent the most expensive bugs. A test that verifies the service raises RetryableError on database timeout ensures that the retry logic in the background job processor actually fires on timeout. Without that test, a change that accidentally swallows the timeout and returns None instead would not be caught until timeout caused silent data loss in production.
Our standard pattern for failure path tests uses pytest and mocker:
def test_raises_retryable_on_db_timeout(mocker): mocker.patch("project.db.query", side_effect=TimeoutError) with pytest.raises(RetryableError): service.fetch_records(batch_id) def test_validation_error_carries_field_name(): with pytest.raises(ValidationError) as exc_info: service.process_record({"amount": "not_a_number"}) assert exc_info.value.field == "amount"
The pytest documentation at docs.pytest.org covers pytest.raises, the match parameter for asserting on exception messages, and mocker.patch for simulating failures.
How We Handle Failures in Background Jobs
Background job processors have different error handling requirements than API endpoints. A web request must respond within a timeout; a job can fail and be retried. But the failure must be recorded and the retry must actually happen -- silent failures are just as dangerous in a job queue as in an API.
Our background job processors follow a consistent structure:
The job function catches RetryableError and marks the job for retry with appropriate backoff.
The job function catches PermanentError and marks the job as permanently failed, triggering an alert.
Any unexpected exception propagates to the job runner's outer handler, which marks it as failed and sends an immediate alert.
def process_job(job): try: result = service.process(job.data) job.mark_complete(result) except RetryableError as err: delay = min(2 ** job.attempt_count, 3600) job.schedule_retry(delay=delay) log.warning("Retryable failure on job %s, retry in %ds: %s", job.id, delay, err) except PermanentError: log.exception("Permanent failure on job %s", job.id) job.mark_failed() alerts.send(f"Job {job.id} permanently failed")
The except PermanentError handler does not catch RetryableError because it is not a subclass of PermanentError. Adding a new failure mode to the hierarchy does not require changing this handler unless the new type needs a distinct response.
The Value of Consistent Patterns Across a Codebase
Exception handling inconsistency compounds as a codebase grows. When some services raise ValueError, others raise Exception("message"), some return None and log an error, and some return (None, error_message) tuples, adding a new feature requires understanding the error contract for every function you call, because none of them are the same.
The consistent approach described here is not complex. Each piece is a few lines. The value accumulates because every developer working on the codebase -- including developers joining the project later -- can read any service and immediately understand its failure modes. The exception type tells them whether to retry, escalate, or discard. The logging tells them where to look. The test tells them the retry logic actually fires.
We have found this consistency to be particularly valuable during handoffs. When we finish a project and transfer it to a client's internal team, the new developers spend far less time understanding the error handling because it follows the same pattern everywhere. That is the return on the investment in establishing the convention at project start.
We Apply the Same Pattern to All Project Types
The same exception hierarchy and logging pattern applies whether we are building a REST API, a batch data pipeline, or a scheduled automation script. The boundary changes -- the API handler is the boundary in a web app, the job entry function is the boundary in a pipeline -- but the design is the same.
This consistency has a practical benefit: when we hand a project off to a client's internal team, the error handling is predictable throughout. Every exception type communicates a specific failure mode. Every boundary logs with log.exception(). Every service-to-service exception is chained. Developers who are new to the codebase can read the exception type and know whether to retry, escalate, or discard.
The full reference collection of Python error handling code snippets -- covering try-except-else-finally, exception chaining, contextlib.suppress, and testing patterns -- is on the 137Foundry blog. It is the reference we send to developers starting on a new project.
One thing we have consistently observed: the conversations that exception handling forces -- which failure modes exist, which require retries, which require escalation -- are valuable design conversations in their own right. Starting with the exception hierarchy is often the clearest way to make implicit failure assumptions explicit before writing the rest of the service logic. The exception types become documentation of what the service can fail with, and that documentation lives in the code where it remains accurate as the code changes.
I'm excited to officially join the Nvidia Developer Program. This milestone will empower Amoateng Technologies Limited to accelerate our AI solutions using next-generation GPU technology. I look forward to driving innovation with my team and delivering high-impact results for our clients.
Build reliable, scalable software with our Software Development Services. We deliver custom solutions to drive efficiency and growth.
Why Software Teams Start with a Proof of Concept
When a software team has an idea, one of the first and smartest moves is to create a Proof of Concept (PoC) in the development process. A PoC is a compact demonstration that shows whether the planned system is technically viable and meaningful for the business. Read more:- Proof of Concept
