When a national payments client came to us, their nightly reconciliation batch was taking long enough to threaten the next business day's settlement window. Volume had tripled in eighteen months; the architecture hadn't changed at all. This is the story of how we rebuilt it - and what we'd do differently if we started today.
The problem wasn't throughput. It was contention.
Our first instinct was to assume the reconciliation job just needed more compute. It didn't. Profiling showed the batch was spending most of its time waiting on row-level locks in a single Postgres table that every transaction type wrote to, regardless of channel. ACH credits, RPPS debits, and manual adjustments were all fighting for the same rows at the same time.
The fix wasn't a bigger database. It was giving each transaction type room to breathe.
We split the monolithic ledger table into channel-partitioned tables behind a single reconciliation service interface, and moved settlement matching into an event-driven pipeline built on Kafka. Each channel now reconciles independently and merges into a unified settlement report at the end of the run.
Partitioning strategy
We partitioned by settlement_date and channel_type, which matched how the bank's downstream systems actually consumed the data. This sounds obvious in hindsight, but the original schema had been partitioned by account_id - a decision that made sense for the transactional side of the system but actively worked against the batch reconciliation workload.
The result: what used to be a single 6-hour sequential batch became five parallel channel jobs, each finishing in under 90 minutes, with zero cross-channel lock contention.
Getting to zero-downtime settlement
Throughput solved the speed problem, but the client's real requirement was availability - the reconciliation service couldn't go down during a settlement window, even for a deploy. We addressed this with blue-green deployment at the service level and idempotent event replay, so a mid-batch restart never produced duplicate or missing settlements.
- Every settlement event carries an idempotency key derived from transaction ID + channel + date
- Failed or interrupted batches replay from the last committed offset, not from zero
- Health checks gate traffic shifting during blue-green cutover, so a bad deploy never touches live settlement traffic
What we'd tell other teams facing this
If your reconciliation job is slowing down as volume grows, look at contention before you look at compute. Adding more CPU to a system that's fighting over the same locks just makes the fight faster, not shorter. Partition around how the data is actually consumed downstream, not how it was originally modeled upstream - and design for restart-safety from day one, because the deploy that takes your reconciliation service down will always land during a settlement window.