← Engineering decisions

Why Kafka for cache invalidation rather than Redis pub/sub?

Ordering per key, replay for instances that were down, and an audit trail of what invalidated when.

CachingKafkaOperability

Redis pub/sub is fire-and-forget. An instance that is restarting during a publish never learns it should evict — and there is no way, afterwards, to find out that it did not.

Kafka gives three things that matter when you are debugging a stale read a day later:

  • Ordering per key, so two rapid changes to the same entity cannot be applied out of order.
  • Replay from offset, so an instance that missed messages catches up on start rather than serving stale data indefinitely.
  • A log, so “was the evict published?” is a question with an answer.

Short local TTLs still back this up. Invalidation you cannot verify should never be the only thing standing between you and a wrong value.

Next decisionWhy group batch work by card hierarchy instead of by arbitrary chunks?