High-throughput event processing with zero duplicate side effects.
A production-grade transactional event pipeline engineered with Java 21, Spring Boot 3, and Apache Kafka (KRaft). Features atomic Redis idempotency (SETNX), Dead Letter Topic (DLT) fault isolation, and PostgreSQL transactional state guarantees.
In high-throughput distributed systems, network blips and consumer rebalances inevitably cause duplicated message deliveries and out-of-order execution, risking catastrophic double charges or inconsistent state.
Core Architectural Problems Solved
- At-least-once delivery duplicates: Overcoming network retries triggering double credit card debits or duplicate inventory allocation.
- Poison pills: Corrupted payloads causing endless consumer group crash loops.
- Zookeeper latency overhead: Modernizing cluster coordination via Kafka native KRaft consensus protocol.
Engineered with KRaft-based Apache Kafka and an atomic Redis SETNX idempotency filter, the engine validates message uniqueness before execution. Transient failures route through an exponential backoff retry mechanism into a Dead Letter Topic (DLT), while final state commits atomically to PostgreSQL.
Engineering decisions
01. Atomic Redis SETNX lock with 24h TTL prevents duplicate transactions even under concurrent rebalances.
02. Non-blocking Dead Letter Topic (DLT) isolates poison pills without halting partition throughput.
Production Stack
• Runtime & Framework: Java 21 (Virtual Threads ready), Spring Boot 3.3
• Messaging: Apache Kafka 3.7 (KRaft Consensus Mode, Spring Kafka)
• Caching & Locks: Redis 7 (Lettuce client, atomic SETNX)
• Persistence: PostgreSQL 16, Spring Data JPA / Hibernate, HikariCP
• Infrastructure & Telemetry: Docker Compose, Spring Boot Actuator, Micrometer, Prometheus
Inspect the complete codebase on GitHub
Full documentation, Docker Compose environment, integration test suites, and Kafka producer/consumer configurations are available in the public repository.
Explore architecture & code ↗