CDC with Debezium: core data onto the lakehouse
Change data capture gives you near real-time synchronisation without running heavy queries against the transactional system. It also brings its own traps.
In most assessments, the first question a company asks is not “which architecture do we need” but “why are our reports still slow after investing in a warehouse”. The answer usually is that the chosen architecture does not match the data types and access patterns in play.
1. Data warehouse — optimised for known questions {#warehouse}
A traditional warehouse requires a schema before loading. That gives excellent query performance and consistent numbers — ideal for financial reporting and stable metrics.
Limits appear with unstructured data — images, audio, free text, logs — or when volume grows fast enough that license and storage cost scale exponentially.
2. Data lake — flexible, and easy to turn into a swamp {#lake}
A lake stores everything raw and applies structure on read. Storage is cheap and any data type is welcome. The risk is governance: after 18–24 months nobody knows which dataset is correct and which is stale.
Across the inherited systems BigAI has been asked to review, the most common cause of lake failure was never technology. It was the absence of a catalogue and of a named owner per dataset.
3. Lakehouse — combining both strengths {#lakehouse}
A lakehouse keeps the cheap storage and flexibility of a lake while adding what only warehouses used to offer: ACID transactions, data versioning, schema evolution and query optimisation. The technical basis is an open table format — Apache Iceberg, Delta Lake or Apache Hudi.
4. Six-criteria comparison {#comparison}
| Criterion | Warehouse | Lake | Lakehouse |
|---|---|---|---|
| Data types | Structured | Any | Any |
| Storage cost | High | Low | Low |
| BI query performance | Excellent | Poor | Good |
| ML support | Limited | Good | Good |
| Governance and quality | Strong | Weak without investment | Strong |
| Vendor lock-in risk | High | Low | Low |
5. Decision framework {#decision}
- Under 1 TB, mostly structured, reporting-led: a lean warehouse is enough.
- 1–50 TB with ML or unstructured data: a lakehouse is the best balance of cost and capability.
- Above 50 TB or heavy real-time volume: lakehouse plus a streaming tier, with serious governance investment from day one.
- A warehouse already working well: keep it and add a lakehouse for what it cannot handle.