Proof Pack · Case study
Refund execution against the bank
Problem & context
One money path served two products: a B2C online travel agency and a B2B travel platform for agencies.
Refund execution against a bank is a distributed transaction with no rollback. Three properties have to hold, and the code held none of them. An errored call must be treated as an unknown — this one was read as a failure. The record of what the gateway was asked to do must exist before the asking — it was written after, leaving a window. And two concurrent paths must not reach the same credit — the lock was keyed on the queue identifier rather than on the money, so two paths with different identifiers for one credit did not contend.
This is the class of failure the design had to close.
Constraints
Four are on the record.
- Finance approval had to survive automation. Refunds could become fast; they could not become unsupervised.
- The provider abstraction is pre-existing and not my design. I integrated four bank providers into an interface I did not own.
- Bank settlement time is downstream and not mine. The system can guarantee when it hands off, never when the money lands.
- The platform is live and carries real money. Every change shipped against production traffic.
Two constraints I am deliberately not documenting publicly: the specific retry bound, and the compatibility window the change had to hold for in-flight refunds.
My role & ownership
Top author on the gateway-refund paths by a wide margin, and second only to one colleague across payment and gateway work overall, from December 2019 onward, still active as of August 2026.
De-facto owner of the money path since January 2020, when the refund system began. Informal — no title, no reports, no org-chart authority.
Core changes are dated: credit-keyed locking, June 2026; retry cap, serialization lock, immediate persistence and bank re-inquiry, August 2026.
Architecture
Described in words; no diagram is embedded.
Components by role, not by name: a refund request store holding state and actor attribution; a worker running the execution; a distributed lock keyed on the credit; a gateway adapter behind the pre-existing provider-agnostic abstraction; an inquiry path back to the bank for indeterminate outcomes; and an operator surface — a daily stuck-refund report plus an admin gateway report.
Shape: those five in a line, with inquiry returning from the bank to the worker and a report leaving the store for operators. The lock encloses the worker.
Key decisions & tradeoffs
I keyed the lock on the credit rather than the queue identifier, because what needs protecting is the money and not the message. I made execution re-inquire with the bank rather than treat an errored call as failed: an error is an unknown, not a "no". I moved gateway-state persistence ahead of the side effect, so the record exists before the request. I bounded the retry at a fixed cap rather than retrying until success, and I hold a partial outcome open rather than resolving it one way. I put a second serialization lock behind a runtime-toggleable parameter, enabled without a deploy.
One decision, as an ADR:
Context. Concurrent paths could reach the same credit carrying different queue identifiers, so a lock keyed on the identifier did not make them contend.
Decision. Key the lock on the credit.
Consequences. The races closed, and no recurrence-shaped commit appears in history since. What it cost is not measured: whether keying on the credit increases contention on a hot credit was never instrumented, and no alternative keying is on record as having been considered. I would want that number before defending the choice as optimal rather than merely correct.
The other five are recorded as choices rather than trade studies; their alternatives belong in a conversation.
Implementation approach
Redis holds the distributed locks. Hangfire runs the bounded retry. Refund and gateway state live in SQL Server through Entity Framework Core, written on interaction rather than on completion, with MassTransit over RabbitMQ between the B2B and B2C sides and Serilog across the gateway paths. The customer gets an SMS when retries exhaust, and admins can inquire directly against the bank for a stuck refund.
Outcomes
Every figure is labelled with its source.
- The failure branch holds under the conditions that used to break it: a retry, a bank that goes quiet, and two requests reaching the same credit. No recurrence-shaped commit appears anywhere in history; the clean window at measurement was roughly ten weeks for the earliest fix and days for the latest. Measured from private repository history, so not independently verifiable. A clean window is not proof of absence.
- Refund turnaround: about a week, manual across three teams, to about a day, automated, finance approval retained — air and hotel. Automation's outcome, not the reliability work's. Recollected operational durations, not instrumentation.
- Authorship: top author on these paths. Proxy, not publicly verifiable — the repositories are private. A live page proves a feature, never who wrote it.
What this study does not prove
The claim is "closed", not "eliminated". The clean window is weeks, not years. The second serialization lock is confirmed enabled in production, but how long it has been enabled is not established, so the window cannot be extended backwards. And authorship of all of it is provable only from a private repository.