Model drift: catch it before the business does
Models rarely fail abruptly. They decay — and without monitoring, the first person to notice is usually a customer.
This is drawn from a cost-reduction engagement for a customer running roughly 340 Spark jobs a day.
1. Measure before optimising {#measure}
The first step is always a per-job cost dashboard. On this project, 12 jobs accounted for 68% of total spend — optimising exactly those twelve produced nearly all the value.
Without that measurement, teams usually optimise whichever job they happen to be looking at.
2. Seven techniques applied {#techniques}
- Move to a columnar format. Several jobs still read JSON and CSV. Switching to Parquet immediately cut bytes read by 40–60%.
- Fix partition sizing. Thousands of small files meant scheduling overhead exceeded actual compute time.
- Remove avoidable shuffles. Reordering joins and using broadcast joins for small tables.
- Enable Adaptive Query Execution. Let Spark adjust partition counts against real data volumes.
- Right-size the instance type. Many jobs were I/O bound, not CPU bound, so high-core machines were pure waste.
- Shut down idle clusters. Development clusters were running overnight, unnoticed.
- Filter and project early. Push predicates and column selection as close to the source as possible.
3. Results {#results}
Total infrastructure cost fell 60% in six weeks, with no architecture change and no rewrite of business logic.
Worth noting: the first four techniques delivered 80% of the saving. The remaining three were worth doing but would not have justified the project on their own.