Research · Quant ML · 2026

Train Often, Deploy Selectively

A sole-author paper (under submission to ACM ICAIF 2026) that treats model replacement as an authorization decision rather than a retraining schedule: Shadow Before Swap warm-refits a challenger off the serving path, waits a week for real labels to mature, and promotes it only when it beats the incumbent that has kept learning in the meantime.

Three panels showing relative NLL reductions for Shadow Before Swap against calendar replacement, blind promotion, and continuous maintenance, replicated across episodes, seeds, and trial lengths. — Selective replacement replicates: the gate's advantage over all three comparators stays positive across both episodes, all three seeds, and one-, two-, and three-week forward trial budgets.

Problem

Production forecasters are maintained systems, not static checkpoints. Between scheduled rebuilds a serving model keeps refreshing its normalizer, consuming labels as they mature, and adapting its head. So the real question at release time is not when to retrain but whether a newly trained state should replace a state that is still improving on its own. Offline benchmarks miss this: they compare a fresh fit against the stale checkpoint the experiment started from, not against the incumbent as it actually exists at release. And because the promoted state becomes the parent of every later refit, one bad swap propagates.

Approach

Shadow Before Swap (SBS) runs the comparison as a causal shadow trial. At each scheduled boundary the incumbent is deep-copied, the clone is warm-refit on recent history, and both branches advance independently on the same market stream with separate normalizers and delayed-label queues. Neither can use a label before it has matured, so the gate only fires once the trial week's labels are real. Promotion requires a paired negative-log-likelihood advantage above a fixed deadband (τ = 10⁻⁴); otherwise the incumbent is retained and the challenger is discarded in full. Four complete recursive replays — maintenance, calendar replacement, schedule-matched blind promotion, and SBS — separate the value of waiting from the value of authorizing.

Binned density of one-week paired trial gain against three-week future challenger value, plus rank correlation, sign agreement, and regret diagnostics for one-, two-, and three-week trials. — What the forward trial actually measures: a one-week paired gain ranks future challenger value at ρ = 0.524 with 74% sign agreement — imperfect ranking is enough, because the gate only has to screen out damaging state transitions.

Outcome

Across 48 UTC weeks spanning two nonoverlapping Binance episodes, three seeds, eight underlyings, and USD-M and COIN-M perpetual contracts, SBS reduces NLL by 0.1472% relative to calendar replacement, 0.0755% relative to schedule-matched blind promotion, and 0.0428% relative to continuous maintenance — the last comparison ruling out the trivial explanation that never refitting is simply safest. It promotes 114 of 528 challengers, cutting deployed-state turnover by 78.4% while still improving the serving trajectory, and avoiding 414 unnecessary model changes. The effect holds across seeds, promotion margins, trial budgets, an earlier 20-asset panel, a Coinbase transport screen, and a topology-matched supervised objective. A deliberately misspecified Temporal-CNN stress test makes the case sharpest: the gate rejects 236 of 240 proposals and retains 99.98% of maintenance performance, where automatic replacement collapses.