Data architecture

CDC with Debezium: core data onto the lakehouse

Change data capture gives you near real-time synchronisation without running heavy queries against the transactional system. It also brings its own traps.

BigAI data engineering team
· 1 min read · Updated

1. Why CDC {#why-cdc}

Traditional batch ingestion runs full-table scans, usually overnight. Against a core banking system or ERP, those queries compete with the business workload and cap how often you can synchronise.

CDC reads the database transaction log instead of querying tables, so it puts almost no load on the source.

2. Configuration in practice {#config}

On our most recent banking project, the single most important setting was limiting which tables are captured. Enabling CDC across an entire database is a common mistake: the log grows fast and most of those tables are never used downstream.

The rule we apply: capture only tables that actually feed a report or a model, and review that list every quarter.

3. Three common mistakes {#mistakes}

  • Unannounced schema changes. You need automated checks that detect structural change and alert before the pipeline breaks.
  • Ignoring deletes. Teams frequently forget to handle delete events, so the lakehouse ends up holding more records than the source actually has.
  • Event ordering. For tables with composite keys, make sure all events for a given key land in the same Kafka partition, or you will apply them out of order.

Related content

How RAG works and when to fine-tune instead

The two approaches are often framed as alternatives. RAG solves a knowledge problem; fine-tuning solves a behaviour problem. Confusing them is a common way to overspend.

Cutting Spark cluster cost by 60%

Most Spark spend is wasted in places that are easy to fix: the wrong storage format, partitions that are too small, avoidable shuffles, and clusters idling overnight.

Start with a free 60-minute data assessment

A BigAI solution engineer will review your current data estate with you, identify the highest-value problem to solve and sketch a realistic roadmap. No commitment.

  • Data maturity assessment
  • 2–3 use cases with clear ROI
  • Budget and timeline estimate

By submitting this form you agree to the BigAI Privacy Policy.

Hotline Free consultation