Cutting Spark cluster cost by 60%
Most Spark spend is wasted in places that are easy to fix: the wrong storage format, partitions that are too small, avoidable shuffles, and clusters idling overnight.
1. Two kinds of drift {#two-kinds}
Data drift is when the distribution of inputs changes — for example the customer base expands into a new segment. Concept drift is when the relationship between inputs and outputs changes — for example buying behaviour shifts after a major event.
The response differs: the first usually needs retraining, the second may require redesigning features.
2. Signals to monitor {#signals}
- Distribution of each input feature against the training set
- Distribution of predictions over time
- Latency and error rate of the inference service
- Actual quality once labels arrive, which is typically days or weeks later
That last one is the ground truth, but it is also the slowest — which is exactly why the first three matter.
3. Setting thresholds that survive {#thresholds}
Thresholds that are too sensitive generate false alarms, and an operations team that gets false alarms starts ignoring the channel entirely.
Our approach is to start wide, observe natural variation for four to six weeks, and only then tighten.