Data engineering
Streaming backbone
A Kafka and structured-streaming spine with exactly-once semantics, checkpointing and replay, built so a late record has a defined fate.
8–14 weeksTypical duration
The problem this solves
Your streaming pipelines mostly work, and when they do not, the recovery is to reprocess everything and hope the duplicates do not matter.
What you receive
Artefacts you can hold, and that you can accept or refuse — never a list of activities.
- A streaming topology with the delivery semantics stated per pipeline and enforced
- Checkpointing, offset management and a tested replay procedure
- An idempotent write path, so a reprocess does not duplicate
- A defined policy for late and out-of-order records, per stream
Also in data engineering
Lakehouse foundation
An open-table-format estate with the catalogue, the layout, the compaction and the retention decided rather than defaulted.
Batch-to-streaming conversion
An existing nightly estate turned incremental without a rewrite, one pipeline at a time.
Data contracts & schema governance
A schema registry and an enforcement point, so a producer cannot break a consumer silently.