Serving a Model: Endpoint, Batch, or Embedded?

In Part 1, I argued that a model in a notebook is a demo, and production is a promise. This part is about the first plank of keeping that promise: serving. Where does the model actually live, and how does anything reach it?

This sounds like a plumbing question. It is actually the most consequential design decision you will make, because it is decided by a single fact about your problem, and if you get that fact wrong, everything downstream, your cost, your latency, your reliability, is wrong too. The good news is that there are only really three answers, and choosing between them is easier than the tooling makes it look.

The one question that decides everything

Before you touch a serving framework, answer this: who consumes the prediction, and how quickly do they need it?

That is the whole decision. Not "what is the best architecture," not "what does the tutorial use." Who is waiting, and how long can they wait. Everything else follows from the answer.

The three ways to serve a model

1. The real-time endpoint. The model sits behind an address that other systems call, and it answers each request in the moment. A fraud check while a payment is happening. A recommendation as a page loads. This is what most people picture when they think "deploy a model," and it is the right choice when a human or a system is genuinely waiting on each individual answer.

2. The batch job. The model runs on a schedule, scores a whole pile of inputs at once, and writes the results somewhere for later. Nightly risk scores. A weekly churn list. A monthly forecast. Nobody is waiting in the moment, so you can run it cheaply, when you like, and simply store the output. An enormous share of real-world models should be batch jobs, and are not, because their builders assumed "deploy" meant "endpoint."

3. Embedded, or on-device. The model ships inside the application itself and runs locally, with no server call at all. A model on a phone, in a browser, on a piece of field equipment. You reach for this when the network is unreliable or absent, when latency must be near zero, or when the data should never leave the device.

A worked example: the crop model is a batch job

My Sierra Leone crop-yield model is a clean case, because the naive instinct is exactly wrong.

The exciting-sounding version is a real-time endpoint: an API a farmer or ministry could query any second. But think about the actual problem. The prediction depends on rainfall data that arrives roughly monthly. The decision it informs, when to plant, changes on the scale of weeks, not seconds. Nobody needs a fresh answer at 11

and a different one at 11.

So the right serving pattern is a batch job: when the new rainfall data lands each month, a scheduled process runs the model over it and produces an updated early-warning signal, which is stored and put in front of whoever needs to act. No always-on server. No endpoint to keep alive and pay for around the clock. No 2am outage risk. The unglamorous choice is the correct one, and it is correct precisely because I answered the one question honestly: nobody is waiting in the moment.

The hidden cost of defaulting to the endpoint

The reason this matters is that the endpoint pattern quietly signs you up for a long list of obligations the notebook never mentioned. An always-on endpoint has to be always on, which is a bill that runs whether or not anyone calls it. It has to scale when traffic spikes and not fall over. It needs its own monitoring, its own versioning, its own failover. Each of those is a real piece of engineering, and you have taken on all of them the moment you chose "endpoint," often without deciding to.

Batch asks for almost none of that. Embedded trades server cost for the discipline of shipping and updating a model on someone else's device. Every pattern has a price. The mistake is paying the endpoint's price without checking whether the problem required it.

Why this matters most on a constrained budget

Everything above is doubly true when compute is metered and power is not guaranteed. An always-on endpoint is the single easiest way to burn money you do not have on a model nobody is calling at 3am, and it is the pattern most exposed to an unreliable connection or a power cut, because it has to be up to be useful.

Batch is the frugal, robust default: it runs when you have power and compute, it costs nothing between runs, and a missed run is a delay, not an outage. Embedded is the answer when connectivity itself is the constraint, putting the model on the device so it keeps working when the network does not. In this environment, choosing the serving pattern well is not a nicety. It is the difference between a system you can afford to keep running and one you cannot.

What would make me wrong

The three patterns are a map, not a law, and the borders blur. Streaming sits between real-time and batch, for data that arrives continuously. Serverless functions can give you endpoint behaviour that only costs money when called, softening the always-on bill. Some systems legitimately need more than one pattern at once. And the opposite error is real too: do not turn a genuinely interactive product into a batch job just to save money, because a recommendation that arrives a day late is not a cheaper recommendation, it is a useless one. The point is not that batch is always right. The point is that the pattern should be a decision you make from the one question, not a default you inherit from a tutorial.

Key takeaways

1. Serving is decided by one question: who needs the answer, and how fast?
2. Three patterns: real-time endpoint (per-request), batch (bulk, scheduled), embedded (on-device, offline).
3. Default to batch when unsure. It is cheaper, simpler, and a failure is a delay, not an outage.
4. The endpoint pattern hides real obligations: always-on cost, scaling, versioning, monitoring, failover.
5. "Real-time" is usually a want, not a need. Ask what actually breaks if the answer were an hour old.

Next in the series: once your model is serving predictions, how do you know when the world has moved and those predictions have quietly gone wrong? That is drift, and it is where good models die.

---

This is Part 2 of From Notebook to Production. Part 1 covered the gap between a model that runs once and one that runs for real.