Training the model is ten percent of the work

There is a particular meeting that happens on most machine learning projects. A model has been trained, the validation numbers are good, and the room agrees it is nearly done. The remaining work is described as deployment, and estimated in weeks.

That estimate is usually wrong by a factor of five, not because anyone is careless, but because the work that remains is a different kind of work than the work that has been done. The notebook answered whether the problem is learnable. Everything after it answers whether the answer can be relied upon by other software, every day, without supervision.

The features in production are not the features you trained on

In the notebook, features are computed over a static dataframe with the whole history available. In production they are computed per request, from whatever is known at that instant. These two computations are written by different people, at different times, in different languages, and they diverge.

The divergence is rarely dramatic. A rolling average computed over calendar days in training and over trailing 24 hours in serving. A category encoded from a lookup that has since gained three new values. A null handled as zero in pandas and as a missing key downstream.

The structural fix is to compute features in exactly one place and read them from that place both times. Whether that is a feature store or a shared library matters less than the property: one implementation, used by both paths.

@feature(entity="account", version=3)
def payments_trailing_30d(ctx: Context) -> float:
    """Sum of settled payments in the 30 days before ctx.as_of.

    as_of is supplied by the caller: the training job passes each row's
    label timestamp, the serving path passes now(). Neither can see past
    its own cut-off, so backfills cannot leak the future into a label.
    """
    window = ctx.as_of - timedelta(days=30)
    return ctx.query(
        Payment.amount.sum(),
        where=(Payment.account_id == ctx.entity_id)
        & (Payment.settled_at.between(window, ctx.as_of)),
    ) or 0.0

Monitor the inputs, not just the outputs

Accuracy in production is usually unmeasurable in real time, because labels arrive late or never. A fraud model learns whether it was right when a chargeback lands sixty days later. A churn model learns in a quarter.

What is measurable immediately is the distribution of what goes in and what comes out. Those move first, and they move before the business metric does.

• Feature distributions against the training baseline, per feature, checked daily.
• Null and default rates — an upstream schema change usually shows up here first.
• Prediction distribution, which drifts before accuracy does.
• Segment coverage, so a model quietly failing for one country is visible before it is reported.
• Latency at the tail, because a p99 that doubles is often a feature lookup falling back to a slow path.

You need to be able to go back

A model is a deployable artefact and deserves the same discipline as any other. That means a version, an immutable record of the data and code that produced it, and a path back to the previous one that does not involve retraining.

Retraining to roll back is not a rollback. It is a rebuild, it takes hours at best, and it will not reproduce the previous model exactly unless the data snapshot and every hyperparameter were captured. During an incident, hours is not an available budget.

Ship it in shadow first

Run the new model against live traffic without acting on its output. Log what it would have decided next to what the current model did decide. A week of that answers questions no offline evaluation can: how it behaves on the traffic that actually arrives, how it handles the malformed inputs the training set never contained, and what it costs to serve at real volume.

1. Shadow deploy — full traffic, no effect, decisions logged side by side.
2. Compare against the incumbent on live data for at least one full weekly cycle.
3. Canary to a small share of traffic, with the proxy metrics on a dashboard someone is watching.
4. Ramp gradually, holding a control group so the effect stays measurable.
5. Keep the previous version deployable for as long as labels take to arrive.