Why compute cost is a product metric, not an ops footnote
The dependency graph of an AI product starts not with features or UX, but with the price of a single inference. Dollars per request, milliseconds per token—that number determines which models are viable, which features ship, and which get cut in the next sprint. Treating compute as a backend concern is a fast path to a product that loses money on every interaction.
Consider a real-time document summarizer. A 7B-parameter model on a single T4 GPU can stay fast enough for a user’s attention span, with cost per request at fractions of a cent. A 70B model on the same hardware? Multiple seconds, higher cost, and a product that feels broken before it finishes its first sentence. The compute constraint didn’t just affect the backend—it defined the product’s shape entirely.
How latency budgets shape model selection and feature scope
Every user-facing AI feature has a latency budget. That budget is not a technical target; it is a product constraint that narrows the model search space. If your app needs quick responses for a chat interface, you are not running a large parameter model. You are quantizing, pruning, or distilling down to something that fits the window.
The MLCommons Inference Datacenter benchmark exists precisely because latency and throughput are not abstract numbers—they are the difference between a feature that feels native and one that feels like waiting for a page to load in 1998. The benchmark measures real serving conditions, not theoretical peak flops. Product teams that ignore these numbers end up with features that work in demos and fail in production.
The hidden cost of batch inference and cold starts
Batch inference looks efficient on a cost-per-token basis. But batching introduces latency that kills interactive features. A recommendation engine that batches requests quickly feels snappy. One that batches slowly feels like the system is thinking too hard.
Cold starts compound the problem. Serverless GPU endpoints that spin down between requests save money but add noticeable cold-start latency. That works for background jobs. It kills user-facing features. The product decision is not “can we serve this model?” but “can we serve this model within the user’s patience window at a cost we can sustain?”
When to optimize for throughput vs. latency
A simple decision framework: if the user is waiting, optimize for latency. If the user is not waiting, optimize for throughput.
- User-facing chat, search, summarization: latency-first. Accept higher per-request cost to keep response times under a second. Quantize models, use smaller architectures, or run on dedicated endpoints.
- Background data pipelines, batch processing, nightly reports: throughput-first. Maximize tokens per dollar. Batch aggressively, use larger models, tolerate cold starts.
The mistake is applying throughput logic to latency-sensitive features. That is how you ship a product that costs too much and feels too slow—the worst of both worlds.
Serving infrastructure as a product lever
The choice between edge, cloud, and on-prem is not an ops decision. It is a product lever that affects latency, privacy, and cost structure.
Edge inference keeps data local and latency low, but limits model size and requires hardware deployment. Cloud inference scales elastically but introduces network latency and data-transfer costs. On-prem gives full control but demands capital expenditure and capacity planning.
Each option changes the product’s value proposition. A medical-imaging tool that processes on-device can claim zero data egress. A customer-service chatbot running on cloud endpoints can scale to millions of conversations but must account for inference cost in per-seat pricing. The infrastructure decision is the product decision.
Practical steps to audit compute sensitivity
- Measure per-request cost in production, not in a notebook. Use real traffic patterns, not synthetic benchmarks.
- Set a latency budget per feature before model selection. If the budget is tight, do not evaluate models that cannot meet it.
- Profile cold-start behavior for serverless endpoints. If cold starts exceed user patience, pre-warm or switch to dedicated instances.
- Quantize or distill before scaling. A quantized smaller model often matches a larger model on latency and cost while retaining acceptable quality.
- Model serving cost as a COGS line item, not a cloud bill line. If inference cost exceeds the feature’s revenue, the feature is not viable.
Compute constraints are not a backend problem to solve later. They are the first product constraint, and ignoring them means building on a dependency graph that will not hold.
