Payment Distribution Engine

Problem

Credit-card payment distribution ran as a single long batch, taking 30–40 minutes per execution. That is long enough to collide with downstream windows, and long enough that a mid-run failure is expensive: a retry meant paying the whole cost again.

Three things were wrong, and only one of them was obvious.

1. Row-at-a-time database access. The dominant cost was not computation, it was round-trips. Each record pulled its own context.
2. No natural work boundary. Because the batch was one indivisible unit, it could not be parallelised and it could not be resumed.
3. Related records processed apart. Records sharing a hierarchy repeatedly re-fetched and re-derived the same parent state.

Architecture

                    ┌──────────────┐
                  │ Payment Data │
                  └──────┬───────┘
                         │
                  ┌──────▼───────┐
                  │ Distribution │
                  │    Engine    │
                  └──────┬───────┘
                         │
          ┌──────────────┼──────────────┐
          ▼              ▼              ▼
   Redistribution     Reversal       Disputes
          │              │              │
          └──────────────┼──────────────┘
                         ▼
                   Distribution

What I changed

Range-based processing. Work is claimed in key ranges rather than scanned as one set. A range is a unit that can be sized, retried and, critically, run beside its siblings. It also turns an index scan into a bounded seek.

Card-hierarchy grouping. Records that share a hierarchy are grouped so parent context is resolved once per group instead of once per record. This was the single largest win, because it removed work rather than moving it.

Controlled batching. Reads and writes are batched to a size tuned against the database, not to a round number. Too small and round-trips dominate; too large and locks are held long enough to hurt everything else.

Bounded parallelism. Ranges execute across a fixed-size pool. Deliberately a pool, not a parallel stream — the work is I/O-bound and sharing the common fork-join pool with unrelated code is a production hazard, not a shortcut.

Query shape. Access paths were rewritten to match existing indexes rather than adding indexes to match the queries, keeping write cost flat.

Result

Execution time went from 30–40 minutes to 6–8 minutes — roughly a five-fold improvement — and the cycle gained two properties it did not have before. It is resumable, because a failed range is the only thing that needs retrying, and it is tunable, because range size and pool width are configuration rather than code.

A later extension added un-adjustment logic to CCPD and bucket distribution, moving the same pipeline closer to near real-time processing rather than a purely scheduled batch.

What I would revisit

Range assignment is currently static. Skewed hierarchies mean some ranges finish early while others hold the tail — a work-stealing claim model would flatten that. It has not been worth the added coordination complexity yet, but it is the next obvious move if volume doubles again.