ENGINEERING IMPLEMENTATION · 2026
Job Intake Service — Durable Asynchronous Processing
A Java and PostgreSQL reference service demonstrating transactional idempotency, bounded intake, crash-recoverable workers, and opt-in connection pooling.
View GitHub ↗Engineering problem
Clients must be able to retry ambiguous submissions without duplicate jobs. Workers can crash, compete for work or lose their leases while the database remains the source of truth.
My contribution
Built a personal Java job-processing service and its reliability experiments, from the HTTP acceptance and PostgreSQL transactions through leased worker recovery, failure-injection checks and bounded connection-pool experiments.
Technical approach
- Used PostgreSQL transactions, a unique idempotency key and a configuration-row lock for atomic acceptance
- Implemented lease ownership, retry scheduling, stale-token fencing and durable worker state
- Added bounded HTTP handlers, readiness and liveness endpoints, process-local metrics and checksum-verified migrations
- Measured JDBC setup and lock contention through isolated CI instrumentation and direct-store comparisons
- Added explicitly enabled HikariCP pooling, multi-instance correctness tests and backend-failure recovery checks
Key decision and trade-off
Decision
Keep a simple serialized PostgreSQL admission transaction and make connection pooling opt-in.
Why
Correctness across simultaneous API instances is more important than an unverified throughput improvement.
Trade-off
A shared lock limits concurrent submissions and each API replica needs its own database-connection budget.
Outcome
A local job-processing reference with passing PostgreSQL, worker, pooled-API, independent-JVM and recovery regression suites. It is not a deployed multi-tenant job platform.
Show technical details ↓Hide technical details ↑Constraints, alternatives, edge cases, ownership breakdown, lessons.
Constraints
- Duplicate requests must return the original job identifier
- Conflicting idempotency-key reuse must be rejected
- Concurrent API instances must honor one database-wide capacity
- A stale worker must never overwrite a newer execution attempt
- No production throughput claim should be inferred from short CI benchmarks
Failure cases and edge cases
- Client retries after commit acknowledgement is lost
- Twenty simultaneous requests with one idempotency key
- A worker crashes after claiming a job
- A stale lease owner tries to complete newer work
- Pool exhaustion or terminated PostgreSQL connection
- Capacity is full but the original key must remain replayable
Full technology list
- Java 21
- PostgreSQL
- JDBC
- HikariCP
- HTTP
- GitHub Actions
What I learned
Connection setup and database locking both have measurable costs; isolated performance tests need failure invariants and clear methodology.
What I would improve
Separate active capacity from retention and add real downstream idempotency before introducing more complex job types.