Site Reliability Engineering (SRE): A Practical Guide with Node.js Examples
From SRE Fundamentals to Reliable Financial Workflows in Pace Equity
Introduction
When we build a backend application, we usually focus on implementing business requirements, writing APIs, connecting databases, and returning the correct response to the frontend.
But building a working application is only one part of software engineering.
A production application must also handle unexpected failures, recover from infrastructure problems, process duplicate requests safely, maintain data consistency, and provide enough visibility for engineers to understand what is happening.
Consider a financial application such as Pace Equity.
A borrower might have a $5 million commitment but receive the money in several installments through a feature called Delayed Draw.
When the borrower requests another funding installment, the backend may need to create a funding record, update balances, generate amortization and payment schedules, and trigger accounting operations.
What happens if the database connection fails halfway through this process?
What happens if the API succeeds but the response never reaches the client?
What happens if a queue delivers the same message twice?
What happens if the funding record exists but the required amortization schedule is missing?
These are reliability problems, not merely coding problems.
This is where Site Reliability Engineering (SRE) becomes useful.
SRE applies engineering practices to the challenge of operating software systems reliably. It helps us define measurable reliability goals, detect failures, recover from incidents, automate repetitive work, and learn from production problems.
In this article, we will start with the fundamentals of SRE, explore practical Node.js examples, and then apply these concepts to a production-style Delayed Draw funding workflow.
By the end, you should understand not only what SRE means, but also how to design a backend system that can recover safely when things go wrong.
Part 1: Understanding Site Reliability Engineering
1. What Is Site Reliability Engineering?
Site Reliability Engineering is an engineering discipline focused on making software systems reliable, scalable, observable, and maintainable in production.
Traditional development asks:
Does the API work?
Is the business logic correct?
Does the database contain the expected records?
Do the tests pass?
SRE asks additional questions:
How often does the API fail?
How long does a request take?
Can the system recover from temporary failures?
How do we detect problems before users report them?
What happens when traffic increases?
Can we safely retry an operation?
Can we recover incomplete financial workflows?
How do we learn from incidents and prevent them from recurring?
Imagine an API that works correctly in development but fails 2% of the time in production because of database timeouts.
The business logic may be correct, but the service is not reliable enough.
SRE encourages us to measure reliability, establish service objectives, automate recovery, reduce repetitive operational work, and continuously improve the system.
Core SRE principles
| Principle | Meaning | Example |
|---|---|---|
| Service Level Indicator (SLI) | A measurement of service behavior | Funding API success rate |
| Service Level Objective (SLO) | A target for a reliability measurement | 99.9% successful eligible requests |
| Service Level Agreement (SLA) | A formal commitment to customers | Contractual availability commitment |
| Error budget | The unreliability permitted by an SLO | Maximum failure allowance over a measurement period |
| Monitoring | Collecting information about service health | Tracking API latency |
| Alerting | Notifying engineers when action is required | Alerting on stuck funding workflows |
| Automation | Replacing repetitive manual operations | Automatically retrying transient failures |
| Incident management | Responding to production failures | Investigating a failed funding release |
| Blameless postmortem | Learning from incidents without focusing on personal blame | Improving recovery after a duplicate event |
| Toil reduction | Reducing repetitive operational work | Automating schedule reconciliation |
The goal is not to eliminate every possible failure. That is generally impossible.
The goal is to design systems that fail safely, detect problems quickly, and recover predictably.
2. Reliability Is More Than Availability
Availability is an important part of reliability, but it is not the whole story.
Consider two financial APIs.
API A is available 99.99% of the time, but occasionally creates duplicate funding records.
API B is available 99.9% of the time, but correctly protects financial data and recovers safely after temporary failures.
For a financial application, API A has a serious correctness problem even though its availability is higher.
Reliability can include several dimensions:
Availability: Can users access the service?
Correctness: Does the service produce the right result?
Consistency: Do related records agree?
Durability: Does committed data survive failures?
Latency: Does the service respond within an acceptable time?
Recoverability: Can incomplete work be resumed safely?
Security: Does the service protect data and operations?
For Pace Equity, preventing duplicate funding and maintaining accurate balances are just as important as keeping the API available.
A reliable financial system must protect business invariants even when infrastructure fails.
Part 2: SLIs, SLOs, SLAs, and Error Budgets
3. What Is a Service Level Indicator (SLI)?
An SLI is a metric that measures the actual behavior of a service.
For example, suppose a funding API receives 1,000 eligible requests and successfully commits 995 of them.
The success rate is:
[ \text{Success Rate} = \frac{\text{Successful Requests}}{\text{Eligible Requests}} \times 100 ]
[ \text{Success Rate} = \frac{995}{1000} \times 100 = 99.5% ]
That measurement is an SLI.
Useful SLIs include:
| SLI | What it measures |
|---|---|
| Request success rate | Percentage of eligible requests that succeed |
| API latency | Time required to respond to a request |
| Workflow completion time | Time from accepting work until required processing finishes |
| Error rate | Frequency of unexpected failures |
| Queue backlog | Amount of work waiting to be processed |
| Recovery time | Time required to restore normal service after an incident |
| Data integrity discrepancies | Number of detected inconsistencies |
Example: Measuring API latency
Suppose the funding API has these response times:
| Request | Response time |
|---|---|
| 1 | 100 ms |
| 2 | 150 ms |
| 3 | 200 ms |
| 4 | 250 ms |
| 5 | 2,000 ms |
The average response time is:
[ \frac{100+150+200+250+2000}{5}=540\text{ ms} ]
But the average does not tell the entire story. Most requests were fast, while one request was much slower.
This is why production monitoring often tracks percentiles such as p50, p95, and p99.
p50: Half of requests complete within this time.
p95: 95% of requests complete within this time.
p99: 99% of requests complete within this time.
Percentiles help identify slow requests that an average can hide.
4. What Is a Service Level Objective (SLO)?
An SLO is a target for an SLI.
For example:
99.9% of eligible funding API requests should be accepted successfully over a rolling 30-day period.
95% of accepted workflows should complete within 30 seconds.
No critical workflow should remain unresolved beyond its operational threshold without an alert.
These are example targets, not established Pace Equity service commitments. A team should choose targets based on business requirements, historical data, and operational capabilities.
It is also important to distinguish API acceptance from workflow completion.
If an API returns HTTP 202, it means the request was accepted for processing. It does not necessarily mean that the amortization schedule, payment schedule, and accounting operations have finished.
Therefore, measure API reliability and end-to-end workflow reliability separately.
5. What Is a Service Level Agreement (SLA)?
An SLA is a formal commitment made to customers or another party.
For example, an organization might contractually commit to a particular availability percentage over a defined period.
The difference is:
| Concept | Purpose |
|---|---|
| SLI | Measures what actually happened |
| SLO | Defines the reliability target |
| SLA | Defines a formal commitment, potentially with consequences if it is missed |
An internal SLO does not automatically become a customer-facing SLA.
For financial workflows, an internal processing-time objective may be different from the external API availability commitment.
6. What Is an Error Budget?
An error budget represents the amount of unreliability allowed by an SLO.
Suppose the SLO is 99.9% success.
The error budget is:
[ 100%-99.9%=0.1% ]
If the system processes 10,000 eligible requests during the measurement window, the budget is:
[ 10{,}000 \times 0.001=10 ]
That means 10 unsuccessful requests fit within the numerical budget.
If the system exceeds its error budget, the team may need to prioritize reliability improvements over risky feature releases.
However, error budgets do not make financial integrity failures acceptable. A duplicate funding release or incorrect balance requires investigation regardless of the availability target.
Part 3: Monitoring and the Four Golden Signals
7. The Four Golden Signals
A useful SRE monitoring framework focuses on four categories:
Latency
Traffic
Errors
Saturation
Together, these signals help engineers understand how a service is behaving and whether its resources are approaching their limits.
7.1 Latency
Latency measures how long an operation takes.
For a Node.js application, measure:
HTTP request duration.
Database query duration.
Database transaction duration.
Queue waiting time.
Background job duration.
End-to-end workflow completion time.
For example, the API might respond in 100 milliseconds while a background schedule-generation workflow takes 20 seconds.
Both measurements matter.
7.2 Traffic
Traffic measures how much work the system receives.
Examples include:
Requests per second.
Funding requests per minute.
Queue messages published per second.
Number of schedule-generation jobs.
Database query volume.
Traffic metrics help explain whether a performance problem is associated with a sudden increase in demand.
7.3 Errors
Errors measure failed operations.
Examples include:
HTTP 5xx responses.
Database connection failures.
Queue publication failures.
Schedule-generation failures.
Downstream service errors.
Workflows that exhaust their retry policies.
Separate expected business rejections from unexpected service failures.
For example, rejecting a request because it exceeds the remaining commitment is normal business behavior. It should not necessarily count as a service reliability failure.
7.4 Saturation
Saturation measures how close a resource is to its capacity.
Examples include:
CPU utilization.
Memory utilization.
Database connection pool usage.
Database lock waits.
Queue backlog.
Worker concurrency.
Node.js event-loop delay.
If the queue backlog grows continuously, the application may be receiving work faster than workers can process it.
Adding more API servers may not solve the problem. The bottleneck could be the database, the worker pool, or a downstream dependency.
Example monitoring plan
| Signal | Metric | Example alert |
|---|---|---|
| Latency | p95 API response time | p95 exceeds the defined threshold |
| Traffic | Requests per minute | Unexpected traffic increase |
| Errors | Unexpected failure rate | Error budget is being consumed too quickly |
| Saturation | Oldest pending job age | Queue backlog exceeds the operational threshold |
| Integrity | Unresolved reconciliation discrepancies | Critical discrepancy detected |
Thresholds should be established using real production behavior and business requirements.
8. Implement Prometheus Metrics in Node.js
Prometheus is commonly used to collect metrics, and Grafana can visualize those metrics in dashboards.
Install the Node.js client:
npm install prom-client
Create a metrics module:
const client = require("prom-client");
const register = new client.Registry();
client.collectDefaultMetrics({
register,
});
const httpRequests = new client.Counter({
name: "http_requests_total",
help: "Total HTTP requests",
labelNames: ["method", "route", "status"],
registers: [register],
});
const httpDuration = new client.Histogram({
name: "http_request_duration_seconds",
help: "HTTP request duration in seconds",
labelNames: ["method", "route"],
buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2, 5],
registers: [register],
});
module.exports = {
register,
httpRequests,
httpDuration,
};
Record request metrics in your API:
const {
httpRequests,
httpDuration,
} = require("./metrics");
async function exampleHandler(req, res) {
const endTimer = httpDuration.startTimer({
method: req.method,
route: "/api/fundings",
});
try {
const result = {
success: true,
};
httpRequests.inc({
method: req.method,
route: "/api/fundings",
status: "200",
});
res.status(200).json(result);
} catch (error) {
httpRequests.inc({
method: req.method,
route: "/api/fundings",
status: "500",
});
throw error;
} finally {
endTimer();
}
}
Expose a metrics endpoint in an Express application:
const express = require("express");
const { register } = require("./metrics");
const app = express();
app.get("/metrics", async (req, res) => {
res.set("Content-Type", register.contentType);
res.end(await register.metrics());
});
This example illustrates instrumentation. In production, ensure the middleware records the actual response status and that errors are handled by the application's normal error-handling mechanism.
Avoid high-cardinality labels
Do not use funding IDs, assessment IDs, customer IDs, or raw error messages as Prometheus labels.
Bad example:
labelNames: ["fundingId", "assessmentId"]
Every unique label combination can create another time series.
Prefer bounded labels such as:
labelNames: ["result", "step"]
Use structured logs and traces to investigate individual funding operations.
Part 4: Retries, Timeouts, and Resilience
9. Why Retrying Is Not Enough
Temporary failures are common in production:
Database connections time out.
A downstream API returns HTTP 503.
A network connection resets.
A service applies rate limiting.
A queue becomes temporarily unavailable.
Retrying may help, but aggressive retries can make an outage worse.
Imagine a downstream service is overloaded. If every failed request is retried immediately, it receives even more traffic while already struggling.
A better approach combines timeouts, retry policies, exponential backoff, jitter, and idempotency.
10. Exponential Backoff with Jitter
Exponential backoff increases the waiting time after each failed attempt.
For example:
First failure: wait approximately 1 second.
Second failure: wait approximately 2 seconds.
Third failure: wait approximately 4 seconds.
Fourth failure: wait approximately 8 seconds.
A common formula is:
[ \text{Delay}_n = \min(\text{Maximum Delay}, \text{Base Delay}\times 2^n) ]
Jitter adds randomness to reduce the likelihood that many workers retry simultaneously.
Node.js implementation
function sleep(ms) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
async function withRetry(
operation,
{
maxAttempts = 5,
baseDelayMs = 500,
maxDelayMs = 30000,
shouldRetry = () => true,
} = {}
) {
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
try {
return await operation();
} catch (error) {
const isLastAttempt = attempt === maxAttempts;
if (isLastAttempt || !shouldRetry(error)) {
throw error;
}
const exponentialDelay = Math.min(
maxDelayMs,
baseDelayMs * 2 ** (attempt - 1)
);
const delay = Math.random() * exponentialDelay;
await sleep(delay);
}
}
}
Example:
const result = await withRetry(
() => downstreamService.generateSchedule(fundingId),
{
maxAttempts: 5,
shouldRetry: (error) => {
return [
"ECONNRESET",
"ETIMEDOUT",
"SERVICE_UNAVAILABLE",
].includes(error.code);
},
}
);
This is appropriate only if repeating the operation is safe. Idempotency must be implemented at the appropriate layer.
For HTTP 429 responses, respect the server's Retry-After guidance where applicable.
What should we retry?
Usually retryable:
Temporary network failures.
Transient database connection failures.
Rate limiting.
Temporary downstream service unavailability.
Usually not retryable without changing the request or state:
Invalid funding amount.
Assessment not found.
Authorization failure.
Funding exceeding the remaining commitment.
Permanent business-rule violations.
Not every database error is transient, and not every server error is safe to retry.
11. Timeouts and Dependency Failures
Every network or external dependency call should have an appropriate timeout.
Without timeouts, a worker may wait indefinitely for an unresponsive dependency and consume resources that other work needs.
Example using Node.js fetch:
async function fetchScheduleService(url) {
const response = await fetch(url, {
signal: AbortSignal.timeout(5000),
});
if (!response.ok) {
throw new Error(
`Schedule service failed: HTTP ${response.status}`
);
}
return response.json();
}
The five-second timeout is illustrative. Choose a value that fits the dependency's expected latency and the overall workflow deadline.
Timeouts do not guarantee that a remote operation did not complete. The remote service may have processed the request before the connection timed out.
That is another reason idempotency is important.
12. Circuit Breakers
A circuit breaker prevents an application from continuously calling a dependency that is failing.
It usually has three states:
Closed: Requests are allowed.
Open: Requests fail fast without calling the dependency.
Half-open: A limited number of test requests determine whether the dependency has recovered.
For example, if an accounting service is consistently unavailable, the funding worker can temporarily stop sending requests to it instead of repeatedly consuming resources.
A circuit breaker does not replace retries or durable queues. It works alongside them.
If the circuit is open, the workflow should remain in a recoverable state, and the system should retry later according to its policy.
For financial workflows, do not treat a failed accounting call as proof that the accounting operation did not happen. Reconcile or query the downstream state before issuing a potentially duplicate financial effect.
Part 5: Idempotency and Reliable Database Operations
13. What Is Idempotency?
An operation is idempotent when repeating the same logical request does not create additional unintended effects.
Suppose a client sends a request to release $1.5 million.
The server commits the transaction, but the response is lost.
The client retries.
Without idempotency, the system might release another $1.5 million.
With idempotency, the server recognizes the retry and returns the result of the original operation.
Example: Idempotency key
POST /api/fundings/release
Idempotency-Key: assessment-42-funding-2-v1
Content-Type: application/json
Request body:
{
"assessmentId": 42,
"amount": "1500000.00",
"fundingReference": "FUNDING-2"
}
The client should reuse the same key for retries of the same logical request.
A genuinely new funding operation needs a new key.
Database table
CREATE TABLE idempotency_keys (
id BIGSERIAL PRIMARY KEY,
idempotency_key TEXT NOT NULL UNIQUE,
request_hash TEXT NOT NULL,
resource_id BIGINT,
status TEXT NOT NULL,
response_body JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
The request hash helps detect reuse of the same key with a different payload.
For example, if the original request asks for $1.5 million but the next request uses the same key for $2 million, the API should reject the conflict.
Important implementation rules
Enforce uniqueness in the database.
Handle concurrent requests using the same key.
Compare request hashes.
Return the original result for a completed request.
Do not create a second financial effect during a retry.
Make queue consumers idempotent too.
A simple lookup followed by an insert is not sufficient under concurrency. Two requests may both observe that the key does not exist.
The unique constraint must be the final authority, and the application must handle the resulting conflict safely.
Part 6: Incident Management, Toil, and Continuous Improvement
14. What Is an Incident?
An incident is an event that causes or threatens a meaningful degradation of service or business operations.
Examples include:
Funding requests are failing.
Queue processing has stopped.
Database connections are exhausted.
Schedule generation is consistently failing.
A reconciliation job finds duplicate funding records.
Accounting records are not being created for completed funding operations.
An incident is not limited to a complete service outage. A financial integrity problem can be a serious incident even if the API remains available.
A practical incident response process
Detect: Monitoring or an alert identifies a problem.
Triage: Determine the impact and scope.
Mitigate: Stop the problem from causing additional damage.
Investigate: Find the technical and operational causes.
Recover: Restore correct operation.
Verify: Confirm that the system is healthy and the data is correct.
Learn: Create follow-up actions that reduce the chance or impact of recurrence.
For example, if schedule generation begins failing after a deployment, the team may temporarily pause new background processing, identify the failing code path, and roll back the release if that is the safest mitigation.
After the service is restored, the team must still identify and recover any funding workflows left incomplete during the incident.
15. Blameless Postmortems
A postmortem documents what happened, how it affected users or business operations, why the system allowed the failure to occur, and what should change.
A blameless postmortem focuses on system behavior and contributing factors rather than personal blame.
For example:
Incident: Some funding workflows remained incomplete after a deployment.
Impact: Funding records were committed, but required schedules were missing.
Contributing factors:
The schedule-generation step was not idempotent.
The worker had no reliable recovery checkpoint.
No alert existed for workflows stuck in
PROCESSING.The deployment process did not verify background-worker compatibility.
Corrective actions:
Make schedule generation idempotent.
Add workflow-state monitoring.
Implement stuck-workflow reconciliation.
Add deployment verification for worker processing.
Add tests that simulate a worker crash after schedule generation.
The goal is to improve the system so that the same class of failure becomes less likely or less damaging.
16. What Is Toil?
Toil is repetitive operational work that can often be automated.
Examples include:
Manually rerunning failed jobs.
Manually identifying missing schedules.
Checking the same dashboards repeatedly.
Fixing stale workflow statuses.
Manually publishing missing events.
Repeating the same deployment steps.
Some operational work is necessary, but repeated manual intervention is a signal that automation may be valuable.
For example, if engineers regularly run SQL queries to find funding records missing amortization schedules, create a reconciliation job that identifies those records, checks their state, and safely queues the required recovery work.
The recovery process should be idempotent and auditable.
Automation should not blindly modify financial data. It should verify the business state before taking action and record what it did.
Part 7: Capacity Planning, Scaling, and Deployment Safety
17. Capacity Planning
Capacity planning estimates whether the system can handle expected traffic and workload.
Suppose a funding service normally receives 10 requests per second but experiences bursts of 100 requests per second during business hours.
The team should understand whether the API, database, queue, and workers can handle those bursts.
Useful capacity indicators include:
Requests per second.
Database query latency.
Database connection pool usage.
Queue backlog growth.
Worker throughput.
Memory and CPU utilization.
Event-loop delay.
Queue backlog example
Suppose a queue receives 100 jobs per minute, while workers process only 80 jobs per minute.
The backlog grows by:
[ 100-80=20\text{ jobs per minute} ]
After 30 minutes, the backlog could grow by approximately:
[ 20 \times 30=600\text{ jobs} ]
Increasing worker concurrency may help, but only if the database and downstream services have sufficient capacity.
Scaling workers without considering database connections can make the system less reliable.
Backpressure
Backpressure is the practice of controlling how much work enters a system when downstream components cannot keep up.
Examples include:
Limiting concurrent workers.
Applying request-rate limits.
Setting queue capacity or admission policies.
Pausing nonessential processing during dependency incidents.
Rejecting or delaying work before the system becomes overloaded.
For financial workflows, accepted work must remain durable. Backpressure should not silently discard funding requests that the API has already promised to process.
18. Safe Deployments
A deployment can introduce failures even when the code passes tests.
Examples include:
A database migration incompatible with the previous application version.
A worker processing a message with an outdated schema.
A new schedule-generation rule producing unexpected results.
A rollback that cannot read records written by the new version.
Useful deployment practices include:
Automated unit and integration tests.
Backward-compatible database migrations.
Staged or canary deployments where appropriate.
Health checks and readiness checks.
Monitoring during rollout.
Rollback procedures.
Verification of background workers and scheduled jobs.
Reconciliation after releases that change financial processing.
For database changes, an expand-and-contract approach is often useful:
Add the new schema in a backward-compatible way.
Deploy code that can work with both old and new structures.
Backfill or migrate data.
Switch the application to the new structure.
Remove obsolete fields only after old code no longer depends on them.
This reduces the risk that an application rollback will fail because the database schema has moved too far ahead.
Part 8: Applying SRE to Pace Equity Delayed Draw Funding
19. Understand the Business Workflow
Now let us apply the SRE principles to a realistic financial workflow.
In Pace Equity, a project may receive funding in multiple installments through Delayed Draw.
Suppose an assessment has the following configuration:
| Funding | Amount | Status |
|---|---|---|
| Funding 1 | $1,000,000 | Completed |
| Funding 2 | $1,500,000 | Pending |
| Funding 3 | $1,000,000 | Pending |
| Funding 4 | $1,500,000 | Pending |
| Total commitment | $5,000,000 |
Funding 1 has already been released.
The remaining commitment is:
[ 5{,}000{,}000-1{,}000{,}000=4{,}000{,}000 ]
When Funding 2 is released:
[ \text{New funded amount} =1{,}000{,}000+1{,}500{,}000 ]
[ \text{Remaining commitment} =5{,}000{,}000-2{,}500{,}000 ]
The remaining commitment becomes $2,500,000.
Depending on the business rules, the release operation may also need to generate or update:
Funding records.
Assessment balances.
Amortization schedules.
Payment schedules.
Accrual schedules.
Accounting journal entries.
Audit records.
Downstream notifications.
The exact workflow must follow the application's actual financial rules.
What can go wrong?
Imagine this sequence:
The client requests Funding 2.
The API creates the funding record.
The system updates the assessment balance.
Schedule generation fails.
The client retries because it never received a response.
Without a reliable design, the system may create duplicate funding or leave an incomplete schedule.
We need to protect against both problems.
20. Define Reliability Requirements for Delayed Draw
The system should satisfy five fundamental requirements.
Financial correctness
The system must not release more funding than the permitted commitment.
Duplicate requests must not create duplicate financial effects.
Data consistency
Related database operations must be committed atomically where appropriate.
Availability
Temporary infrastructure failures should not permanently prevent valid operations from completing.
Recoverability
Incomplete workflows must be detectable and resumable.
Observability
Engineers must be able to identify failed requests, stuck jobs, missing schedules, and unresolved accounting operations.
These requirements lead to the architecture below.
Part 9: Production Architecture
21. Recommended Architecture
A reliable Delayed Draw workflow can use the following components:
Node.js API.
PostgreSQL.
Sequelize.
Database transactions.
Transactional outbox.
Durable message queue.
Background workers.
Workflow state machine.
Prometheus metrics.
Structured logging.
Reconciliation jobs.
Architecture diagram
flowchart TD
A[Client Application] --> B[Node.js Funding API]
B --> C[Validate Request]
C --> D[PostgreSQL Transaction]
D --> E[Funding Record]
D --> F[Balance Update]
D --> G[Workflow Record]
D --> H[Outbox Event]
H --> I[Outbox Publisher]
I --> J[Durable Queue]
J --> K[Funding Worker]
K --> L[Amortization Schedule]
K --> M[Payment Schedule]
K --> N[Accounting Operations]
L --> O[Update Workflow]
M --> O
N --> O
O --> P[Monitoring and Alerts]
P --> Q[Reconciliation Job]
Why separate these responsibilities?
The funding API should not need to keep an HTTP request open while every downstream operation completes.
The database transaction protects core financial changes.
The outbox ensures that the system durably records the need to publish an event.
The queue distributes work to workers.
The workflow state machine records processing progress.
Monitoring detects abnormal behavior.
Reconciliation finds incomplete work that escaped normal processing.
The database remains the source of truth for the financial state. The queue is a delivery mechanism, not the authoritative record of whether funding has occurred.
22. Why Not Perform Everything in One API Request?
A simple implementation might look like this:
async function releaseFunding(req, res) {
const funding = await createFunding(req.body);
await updateBalance(funding);
await generateAmortizationSchedule(funding);
await generatePaymentSchedule(funding);
await createJournalEntry(funding);
return res.status(200).json({
success: true,
fundingId: funding.id,
});
}
This implementation has several risks.
Long-running requests: The API remains open while all operations execute.
Partial failure: The funding record might be committed before schedule generation fails.
Duplicate requests: A client retry might create another funding record.
Tight coupling: A temporary accounting-service failure can prevent the API from completing its response.
A better API commits the core operation and durable processing instruction before returning.
async function releaseFunding(req, res) {
const result = await fundingService.release(req.body);
return res.status(202).json({
success: true,
fundingId: result.fundingId,
workflowId: result.workflowId,
status: result.status,
});
}
HTTP 202 indicates that the request has been accepted for processing.
The service should return this response only after the required database changes have been durably committed.
The client can then query a workflow-status endpoint.
Part 10: Database Transactions and Financial Integrity
23. Use Transactions for Related Database Changes
Suppose a funding release requires:
Creating the funding record.
Updating the funded balance.
Creating a workflow record.
Creating an outbox event.
If these operations use the same PostgreSQL database, they should be committed in one transaction when they represent a single required unit of work.
Sequelize example
The following example illustrates the transaction boundary. Adapt the model names, fields, and status values to the actual schema.
const {
sequelize,
Funding,
Assessment,
FundingWorkflow,
OutboxEvent,
} = require("../models");
async function createFundingTransaction({
assessmentId,
amount,
idempotencyKey,
}) {
return sequelize.transaction(async (transaction) => {
const assessment = await Assessment.findByPk(
assessmentId,
{
transaction,
lock: transaction.LOCK.UPDATE,
}
);
if (!assessment) {
throw new Error("Assessment not found");
}
const remainingCommitment =
Number(assessment.totalCommitment) -
Number(assessment.totalFunded);
if (amount <= 0) {
throw new Error("Funding amount must be positive");
}
if (amount > remainingCommitment) {
throw new Error(
"Funding exceeds remaining commitment"
);
}
const funding = await Funding.create(
{
assessmentId,
amount,
status: "COMMITTED",
idempotencyKey,
},
{ transaction }
);
assessment.totalFunded =
Number(assessment.totalFunded) + amount;
await assessment.save({ transaction });
const workflow = await FundingWorkflow.create(
{
fundingId: funding.id,
status: "PENDING",
},
{ transaction }
);
await OutboxEvent.create(
{
eventType: "FUNDING_RELEASED",
aggregateId: String(funding.id),
payload: {
fundingId: funding.id,
assessmentId,
workflowId: workflow.id,
},
status: "PENDING",
},
{ transaction }
);
return {
fundingId: funding.id,
workflowId: workflow.id,
status: workflow.status,
};
});
}
What does this protect us from?
Suppose the outbox event cannot be inserted.
The transaction rolls back the funding record, balance update, and workflow record as well.
The database does not commit only part of the required operation.
Concurrency control
Suppose the remaining commitment is $1 million.
Two requests arrive at almost the same time:
Request A wants $700,000.
Request B wants $600,000.
If both requests read the same original balance without concurrency control, both may pass validation.
Together, they would exceed the remaining commitment.
A row lock, an appropriately designed conditional update, or another correct concurrency-control strategy must prevent this.
Database constraints should provide additional protection where possible.
Financial data types
Avoid relying on ordinary JavaScript floating-point arithmetic for exact monetary calculations.
Use integer minor units when appropriate, or use a suitable decimal library together with PostgreSQL NUMERIC columns.
Also consider whether the funded balance should be derived from immutable funding records rather than maintained as a separate source of truth.
Part 11: Idempotency in Financial Workflows
24. Prevent Duplicate Funding
Imagine the client submits a funding request.
The database transaction succeeds, but the HTTP response is lost.
The client cannot tell whether the request succeeded, so it retries.
The backend must recognize the second request as a retry of the same logical operation.
Example request
POST /api/fundings/release
Idempotency-Key: assessment-42-funding-2-v1
Content-Type: application/json
{
"assessmentId": 42,
"amount": "1500000.00",
"fundingReference": "FUNDING-2"
}
The client reuses the same key when retrying the same request.
Idempotency table
CREATE TABLE idempotency_keys (
id BIGSERIAL PRIMARY KEY,
idempotency_key TEXT NOT NULL UNIQUE,
request_hash TEXT NOT NULL,
resource_id BIGINT,
status TEXT NOT NULL,
response_body JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
If the key already exists:
Compare the request hash.
Reject reuse with a different payload.
Return the original result if the operation completed.
Handle concurrent requests safely.
The unique database constraint is essential. A lookup followed by an insert is not sufficient when requests can arrive concurrently.
Idempotency at the worker level
HTTP idempotency alone does not solve duplicate queue deliveries.
A worker may complete a schedule-generation operation and crash before acknowledging the queue message. The message may be delivered again.
The worker must recognize that the schedule already exists and avoid creating another one.
For example:
CREATE UNIQUE INDEX unique_funding_schedule
ON funding_schedules (funding_id, schedule_type);
This is an illustrative constraint. Choose the unique key according to the actual business identity of a schedule.
The goal is that retrying the same logical operation does not create another financial effect.
Part 12: The Transactional Outbox
25. Why Direct Event Publication Is Risky
Consider:
await Funding.create(fundingData);
await queue.publish("FUNDING_RELEASED", {
fundingId: funding.id,
});
Two failure scenarios exist.
Scenario A: The funding record is committed, but queue publication fails.
Scenario B: The queue receives the event, but the database transaction fails.
Both situations can leave the system inconsistent.
The transactional outbox pattern addresses this by storing the event in the same database transaction as the funding record.
26. Create the Outbox Table
CREATE TABLE outbox_events (
id BIGSERIAL PRIMARY KEY,
event_type TEXT NOT NULL,
aggregate_id TEXT NOT NULL,
payload JSONB NOT NULL,
status TEXT NOT NULL DEFAULT 'PENDING',
attempts INTEGER NOT NULL DEFAULT 0,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
published_at TIMESTAMPTZ
);
CREATE INDEX outbox_pending_events_idx
ON outbox_events (created_at)
WHERE status = 'PENDING';
The transaction inserts the funding record and outbox event together.
A separate publisher reads pending events and sends them to the queue.
Simplified publisher
async function publishPendingEvents() {
const events = await OutboxEvent.findAll({
where: { status: "PENDING" },
order: [["createdAt", "ASC"]],
limit: 100,
});
for (const event of events) {
try {
await queue.publish(event.eventType, event.payload);
await event.update({
status: "PUBLISHED",
publishedAt: new Date(),
});
} catch (error) {
await event.increment("attempts");
console.error("Failed to publish event", {
eventId: event.id,
eventType: event.eventType,
error: error.message,
});
}
}
}
This code demonstrates the idea, but production publishers must safely claim events so concurrent instances do not race.
They must also handle the case where publication succeeds but updating the event status fails.
That scenario may cause duplicate publication.
Therefore, consumers must be idempotent.
The outbox provides reliable at-least-once publication when implemented with suitable recovery mechanisms. It does not guarantee exactly-once execution across separate systems.
Part 13: Durable Queues and Background Workers
27. Process Funding Asynchronously
After the funding transaction commits, a durable queue can distribute downstream work to background workers.
Possible technologies include BullMQ with Redis, RabbitMQ, or another queue that meets the system's durability requirements.
The right choice depends on the existing infrastructure, message-delivery guarantees, operational requirements, and team experience.
Example worker
const { Worker } = require("bullmq");
const worker = new Worker(
"funding-processing",
async (job) => {
const {
eventId,
fundingId,
workflowId,
} = job.data;
await processFundingWorkflow({
eventId,
fundingId,
workflowId,
});
},
{
connection: redisConnection,
concurrency: 5,
}
);
worker.on("completed", (job) => {
console.log("Funding job completed", {
jobId: job.id,
});
});
worker.on("failed", (job, error) => {
console.error("Funding job failed", {
jobId: job?.id,
error: error.message,
});
});
The processor must be idempotent, and the workflow status must be stored durably.
The queue tells the system which work to attempt. The database records the actual financial state and workflow progress.
Why is that distinction important?
A worker may generate a schedule successfully and then crash before acknowledging the message.
The queue may deliver the message again.
The second execution must verify the schedule's existing state instead of creating a duplicate.
Part 14: Model the Workflow as a State Machine
28. Track Workflow Progress Explicitly
A workflow state machine makes it easier to understand what has happened and what should happen next.
For example:
PENDING
|
v
PROCESSING
|
+------> COMPLETED
|
v
RETRY_PENDING
|
v
PROCESSING
|
+------> COMPLETED
|
+------> FAILED
A workflow table might look like this:
CREATE TABLE funding_workflows (
id BIGSERIAL PRIMARY KEY,
funding_id BIGINT NOT NULL UNIQUE,
status TEXT NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
next_attempt_at TIMESTAMPTZ,
last_error TEXT,
started_at TIMESTAMPTZ,
completed_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
In production, consider foreign keys, valid-status constraints, leases, audit history, and step-level state.
Example workflow processor
async function processFundingWorkflow({
eventId,
fundingId,
workflowId,
}) {
const workflow = await FundingWorkflow.findByPk(workflowId);
if (!workflow) {
throw new Error("Funding workflow not found");
}
if (workflow.status === "COMPLETED") {
return;
}
await workflow.update({
status: "PROCESSING",
startedAt: workflow.startedAt || new Date(),
});
try {
await ensureAmortizationSchedule(fundingId);
await ensurePaymentSchedule(fundingId);
await ensureAccountingProcessing(fundingId);
await sequelize.transaction(async (transaction) => {
const currentWorkflow =
await FundingWorkflow.findByPk(workflowId, {
transaction,
lock: transaction.LOCK.UPDATE,
});
if (currentWorkflow.status !== "COMPLETED") {
await currentWorkflow.update(
{
status: "COMPLETED",
completedAt: new Date(),
lastError: null,
},
{ transaction }
);
}
});
} catch (error) {
await workflow.update({
status: "RETRY_PENDING",
attempts: workflow.attempts + 1,
lastError: error.message,
});
throw error;
}
}
This example illustrates the workflow but is not a complete production implementation.
A production implementation should:
Claim workflows safely.
Prevent concurrent processing of the same workflow.
Make each processing step idempotent.
Distinguish transient errors from permanent business failures.
Record step progress durably where necessary.
Recover workflows abandoned by crashed workers.
Define the exact conditions required for completion.
If schedule generation succeeds but payment schedule generation fails, the next attempt should safely reuse or verify the existing amortization schedule and continue.
A long-running workflow should not be treated as one giant database transaction. Instead, each step should have a clearly defined transaction boundary and recovery strategy.
Part 15: Monitoring Delayed Draw Workflows
29. Define Useful SLIs and SLOs
For Delayed Draw, useful measurements include:
| Metric | Meaning |
|---|---|
| Funding request success rate | Percentage of eligible requests committed successfully |
| API response latency | Time required to accept or reject a request |
| Workflow completion duration | Time required to finish required processing |
| Schedule generation failure rate | Frequency of schedule-generation failures |
| Outbox backlog | Events waiting to be published |
| Queue backlog age | How long the oldest pending job has waited |
| Retry count | Number of processing retries |
| Stuck workflow count | Workflows that exceed expected processing time |
| Reconciliation discrepancy count | Inconsistencies detected by reconciliation |
An internal SLO might require most accepted workflows to complete within a defined period.
The exact target should be selected based on actual processing behavior and business requirements.
Example Prometheus metrics
const client = require("prom-client");
const register = new client.Registry();
client.collectDefaultMetrics({ register });
const fundingRequests = new client.Counter({
name: "funding_requests_total",
help: "Total number of funding requests",
labelNames: ["result"],
registers: [register],
});
const workflowDuration = new client.Histogram({
name: "funding_workflow_duration_seconds",
help: "Funding workflow duration",
labelNames: ["result"],
buckets: [0.1, 0.5, 1, 2, 5, 10, 30, 60, 120],
registers: [register],
});
const workflowFailures = new client.Counter({
name: "funding_workflow_failures_total",
help: "Funding workflow failures",
labelNames: ["step"],
registers: [register],
});
module.exports = {
register,
fundingRequests,
workflowDuration,
workflowFailures,
};
Use these metrics to build dashboards and alerts.
Do not include funding IDs or assessment IDs as Prometheus labels because they create high-cardinality metrics. Use structured logs for individual operations.
30. Structured Logging and Correlation IDs
A single funding operation may travel through the API, outbox publisher, queue, worker, and downstream services.
Consistent identifiers help engineers follow that operation across the system.
Example:
logger.info("Funding workflow started", {
eventId,
fundingId,
workflowId,
correlationId,
});
If schedule generation fails:
logger.error("Amortization schedule generation failed", {
eventId,
fundingId,
workflowId,
correlationId,
errorCode: error.code,
message: error.message,
});
Use the correlation ID to connect related operations.
Avoid logging unnecessary customer data, credentials, authentication tokens, or sensitive financial information.
Logs should be access-controlled and retained according to policy.
Part 16: Reconciliation and Financial Integrity
31. Why Reconciliation Is Necessary
Transactions, queues, retries, and monitoring greatly improve reliability, but production systems still need reconciliation.
Reconciliation compares the current system state with the state required by business rules.
For Delayed Draw, reconciliation can identify:
Funding records with no workflow.
Workflows stuck in
PROCESSING.Workflows that have exceeded their retry policy.
Funding records missing required amortization schedules.
Funding records missing required payment schedules.
Required accounting operations that have not completed.
Incorrect balances or duplicate business references.
The reconciliation job should detect discrepancies and initiate safe recovery where possible.
32. Find Funding Records Without Workflows
Assuming the relevant tables and relationships exist:
SELECT
f.id AS funding_id,
f.assessment_id,
f.amount,
f.created_at
FROM fundings f
LEFT JOIN funding_workflows w
ON w.funding_id = f.id
WHERE w.id IS NULL
AND f.status = 'COMMITTED';
This query identifies committed funding records without a workflow.
If both records are required to be created in the same database transaction, this condition should normally be impossible.
A result could indicate historical data problems, a legacy code path, or an incorrectly implemented transaction boundary.
33. Find Stuck Workflows
SELECT
id,
funding_id,
status,
attempts,
updated_at,
last_error
FROM funding_workflows
WHERE status IN (
'PENDING',
'PROCESSING',
'RETRY_PENDING'
)
AND updated_at < NOW() - INTERVAL '10 minutes';
The ten-minute threshold is illustrative and should reflect expected processing times.
A workflow may appear stuck because a worker crashed, a message was lost, a dependency is unavailable, or a status update failed.
The reconciliation process should inspect the actual state before retrying the work.
34. Find Missing Amortization Schedules
SELECT
f.id AS funding_id,
f.assessment_id
FROM fundings f
LEFT JOIN amortization_schedules s
ON s.funding_id = f.id
WHERE f.status = 'COMMITTED'
AND s.id IS NULL;
This query assumes every committed funding operation requires an amortization schedule and that the schedule table has the specified relationship.
Adapt it to the actual business rules and schema.
35. Recover Safely
Finding a discrepancy is only the first step.
A recovery job should:
Identify the affected funding operation.
Verify the funding record and current workflow state.
Determine which required processing steps have already completed.
Check whether the missing operation is safe to retry.
Requeue or resume the incomplete work.
Record the recovery action.
Verify that the expected state has been reached.
Do not blindly regenerate schedules or create journal entries. The recovery logic must first check whether the operation already succeeded.
For accounting operations, use the appropriate unique business references and idempotency controls to avoid duplicate journal entries.
Part 17: Financial Invariants and Accounting Reliability
36. Define Business Invariants
A financial invariant is a rule that must remain true even when the application experiences failures.
Examples include:
Funding commitment
[ \text{Total Funded Amount} \leq \text{Total Commitment} ]
Funding amount
[ \text{Funding Amount} > 0 ]
Journal entry balance
[ \text{Total Debits}
\text{Total Credits} ]
Workflow completeness
A workflow cannot be marked COMPLETED until all required business operations have reached their required successful states.
These rules should be enforced at the appropriate layers.
Use application validation, database constraints, transaction boundaries, and reconciliation together.
Application validation alone is not enough when concurrent requests can modify the same financial records.
37. Accounting Must Be Idempotent Too
Suppose a funding worker triggers journal-entry creation.
The accounting operation succeeds, but the worker crashes before updating the workflow status.
The message is delivered again.
If the worker blindly creates another journal entry, the accounting records may be duplicated.
The accounting integration should therefore use a stable business reference, a unique constraint, or an equivalent idempotency mechanism.
For example, an accounting operation might use a unique source reference tied to the funding ID and accounting category.
The exact uniqueness rules depend on whether one funding operation can legitimately generate multiple journal entries.
A retry should return or recognize the original accounting result rather than create an unintended duplicate.
If accounting is performed by a separate service or database, PostgreSQL transactions in the funding service cannot atomically commit the external accounting operation. Use a durable workflow, an outbox, idempotent consumers, and reconciliation to manage that boundary.
Part 18: Testing for Reliability
38. Test Failure Scenarios, Not Only Successful Paths
A reliable backend needs more than unit tests for business calculations.
Tests should cover what happens when individual operations fail.
Useful scenarios include:
The database transaction rolls back.
The client sends the same idempotency key twice.
Two concurrent requests attempt to exceed the commitment.
Queue publication fails.
The outbox event is published twice.
A worker crashes after generating a schedule.
Schedule generation succeeds but payment schedule generation fails.
Accounting processing times out.
A workflow exceeds its retry policy.
Reconciliation finds a missing schedule.
A retry attempts to create an already-existing journal entry.
39. Example: Test Idempotency
A simplified Jest example:
describe("releaseFunding", () => {
it("does not create duplicate funding for the same key", async () => {
const input = {
idempotencyKey: "assessment-42-funding-2-v1",
assessmentId: 42,
amount: "1500000.00",
};
const first = await fundingService.release(input);
const second = await fundingService.release(input);
expect(second.fundingId).toBe(first.fundingId);
const count = await Funding.count({
where: {
id: first.fundingId,
},
});
expect(count).toBe(1);
});
});
This is a conceptual test. Adapt the service contract and fixtures to the actual implementation.
Also test concurrent requests, because sequential tests alone do not prove concurrency safety.
40. Example: Test a Failed Schedule Step
describe("processFundingWorkflow", () => {
it("keeps the workflow recoverable when schedule generation fails", async () => {
jest
.spyOn(scheduleService, "generateAmortization")
.mockRejectedValue(new Error("Temporary failure"));
await expect(
processFundingWorkflow({
eventId: "event-1",
fundingId: 100,
workflowId: 200,
})
).rejects.toThrow("Temporary failure");
const workflow = await FundingWorkflow.findByPk(200);
expect(workflow.status).toBe("RETRY_PENDING");
});
});
This test assumes that processFundingWorkflow() records the retryable state and then rethrows the error for the queue's retry mechanism.
In production, verify that status updates are durable and that the worker can safely resume the workflow.
41. Add Integration and Recovery Tests
Unit tests validate isolated logic.
Integration tests validate the interaction between the API, database, queue, and worker.
Recovery tests should simulate failures between steps rather than only testing exceptions returned by individual functions.
For example:
Commit the funding transaction.
Publish the outbox event.
Generate the amortization schedule.
Simulate a worker crash before the next step.
Redeliver the event.
Verify that the existing schedule is reused and the remaining work completes.
These tests help demonstrate that the architecture works under realistic failure conditions.
Part 19: A Practical SRE Implementation Roadmap
42. Implement Reliability in Stages
A team does not need to introduce every SRE mechanism simultaneously.
A staged approach makes implementation easier to review and validate.
Stage 1: Establish visibility
Add structured logs.
Add request and workflow metrics.
Measure API latency and error rate.
Track queue backlog and processing duration.
Create dashboards for critical workflows.
Stage 2: Protect financial correctness
Define business invariants.
Add appropriate database constraints.
Implement transaction boundaries.
Add idempotency keys.
Protect concurrent funding operations.
Stage 3: Make processing durable
Introduce workflow state.
Implement the transactional outbox.
Use a durable queue.
Make workers idempotent.
Add bounded retry policies.
Stage 4: Add recovery
Implement reconciliation queries.
Detect stuck workflows.
Build safe replay or resume operations.
Record recovery actions.
Alert on unresolved discrepancies.
Stage 5: Improve operations
Define SLOs.
Establish error-budget policies.
Test deployment and rollback procedures.
Conduct blameless postmortems.
Automate repetitive operational tasks.
The stages can overlap, and the exact order should reflect the application's existing architecture and most serious risks.
Part 20: Production Readiness Checklist
43. Before Releasing a Financial Workflow
Use this checklist when reviewing a production implementation.
Business correctness
[ ] Funding amounts are validated.
[ ] Remaining commitment is checked safely.
[ ] Concurrent funding requests cannot overfund the assessment.
[ ] Financial calculations use appropriate numeric representations.
[ ] Accounting entries satisfy the required debit-credit rules.
Database integrity
[ ] Required database changes use appropriate transactions.
[ ] Unique constraints protect critical business references.
[ ] Workflow records are durable.
[ ] Database migrations are backward-compatible where required.
[ ] Failed transactions leave no partial committed state.
Idempotency
[ ] Client retries reuse the same idempotency key.
[ ] Reuse of a key with a different payload is rejected.
[ ] Queue consumers tolerate duplicate deliveries.
[ ] Schedule generation is safe to retry.
[ ] Accounting operations cannot create unintended duplicate entries.
Background processing
[ ] Outbox events are published reliably.
[ ] Concurrent publishers safely claim events.
[ ] Worker failures are recorded.
[ ] Retry policies are bounded.
[ ] Permanent errors do not retry forever.
[ ] Stuck workflows can be identified and recovered.
Observability
[ ] API latency and errors are measured.
[ ] Workflow duration is measured.
[ ] Queue backlog and oldest-message age are monitored.
[ ] Structured logs include correlation identifiers.
[ ] Alerts are actionable.
[ ] Sensitive information is excluded from logs.
Recovery and operations
[ ] Reconciliation detects missing required records.
[ ] Recovery actions are idempotent and auditable.
[ ] Deployment rollback procedures are documented.
[ ] Failure scenarios are covered by tests.
[ ] Incident response and postmortem procedures are established.
Conclusion
Site Reliability Engineering changes the way we think about backend development.
Instead of asking only whether an API works, we ask whether the entire service behaves correctly under real production conditions.
We define measurable reliability targets, monitor the four golden signals, use timeouts and bounded retries, protect financial operations with idempotency, and make background processing recoverable.
We also recognize that production reliability depends on more than code. It requires safe deployments, incident management, reconciliation, automated recovery, and continuous learning.
For a financial workflow such as Pace Equity's Delayed Draw funding, these practices are particularly important.
A funding operation may involve several related activities: creating the funding record, updating balances, generating schedules, and triggering accounting operations. A failure in one step must not leave the system permanently inconsistent or cause duplicate financial effects when the operation is retried.
A practical design combines:
PostgreSQL transactions for atomic database changes.
Idempotency for safe retries.
A transactional outbox for reliable event publication.
A durable queue for background work.
A workflow state machine for tracking progress.
Prometheus metrics and structured logs for observability.
Reconciliation for detecting and recovering incomplete work.
Automated tests for failure scenarios.
SRE practices for learning from incidents and improving reliability.
The most important principle is simple:
A reliable system is not one that never fails. It is one that detects failures, protects data, recovers safely, and makes it possible for engineers to understand and improve what happened.
When these principles are built into the architecture from the beginning, we create backend systems that are easier to operate, safer to change, and more dependable for the people and businesses that rely on them.
Comments
Join the discussionNo comments yet. Be the first to comment.