How we modelled the inference market

Ask a simple question: what does it cost to run Llama 3.3 70B?

One provider will tell you $0.40 per million output tokens. Another quotes $2.99 per GPU-hour for an H100 you rent and operate yourself. A third offers a dedicated endpoint billed by the hour whether you use it or not. A fourth sells per-second serverless compute that scales to zero between requests — and bills your GPU, CPU cores, and RAM as three separate meters. A fifth, if you're an enterprise, will sell you “provisioned throughput units” that guarantee capacity without telling you what hardware sits underneath.

Every one of those is a legitimate answer to the question. None of them is denominated in the same unit. And that is the actual problem nfer exists to solve — not scraping prices, which is table stakes, but making prices comparable without lying about them.

The market sells incomparable things on purpose#

The inference market did not converge on a standard unit of sale, and it isn't going to. The shapes in the wild today, all straight off public pricing pages:

Product shape	How it's billed	Examples from public rate cards
Per-token API	Per million input and output tokens, with discount tiers for cached input and batch	OpenAI, Anthropic, Mistral; Anthropic also prices cache writes, at two time-to-live tiers
Hourly hardware rental	Dollars per GPU-hour or per instance-hour	Lambda, CoreWeave; AWS at $98.32/hour for a p5.48xlarge — eight H100s in a trench coat
Monthly rental	Flat monthly fee for a dedicated server	Hetzner GPU servers, priced in euros
Per-second serverless	Per second of compute, scale-to-zero; sometimes one meter, sometimes three	Replicate bills whole-bundle seconds; Modal bills GPU-seconds, CPU-core-seconds and GiB-of-RAM-seconds separately, at millionths of a dollar
Provisioned throughput	Per reserved capacity unit, with commitment terms from one hour to years	Azure PTUs (tokens/min, split by direction); Vertex GSUs (a single tokens/sec figure); Bedrock PTUs (sizing not published at all)
Everything else	Per image, per search query, per request, per “neuron”	Cloudflare Workers AI's neurons — a compute unit that exists nowhere else in the universe

The naive comparison site handles this by picking one column — usually $/1M tokens — and either dropping everything that doesn't fit or force-converting it with assumptions it doesn't disclose. That produces a tidy table and a wrong answer. A team deciding between renting H100s and buying a dedicated endpoint isn't asking a per-token question; an enterprise sizing PTUs against GSUs isn't asking an hourly one. Collapse the market into one price column and you've built a ranking of things that aren't alternatives to each other.

Beneath the branding, every offering answers the same questions#

Here's the insight the whole product rests on. Strip away the marketing names — “serverless”, “dedicated”, “provisioned”, “managed” — and every inference product on the market is fully described by where it sits on five dimensions:

Dimension	Why it changes the comparison	Illustrative values — examples, not the list
Infrastructure ownership — who owns the compute?	Decides whether you're buying a service or running an estate	The provider's datacentre, or your own hardware in a colocation cage
Tenancy — do you share it?	Shared capacity bills what you use; capacity set aside for you bills whether you use it or not	Shared with strangers, or a replica that's yours
Serving-stack operation — who runs inference?	Operations are a real cost that never appears on the rate card	The provider operates the model for you, or you do — on call at 3 a.m.
Abstraction — what are you handed?	Sets what you can change, and everything you must now maintain	A model behind an API, or a place to put your container
Metering — how does the meter charge?	Determines which unit “cheap” is even measured in	Metered usage, or reserved capacity

Place any product on those five dimensions and it lands at a point in a shared coordinate system. And something satisfying happens: the categories everyone argues about fall out as consequences. “API”, “dedicated endpoint”, “IaaS”, “self-hosted” stop being labels a provider chose for its pricing page and become derived facts — the same coordinates always produce the same category, no matter what the marketing says. When Modal calls itself serverless and Replicate calls itself a model API, our model doesn't care what they call themselves; it cares where they sit.

This is also what makes the model future-proof. Provisioned throughput didn't exist as a mainstream product a few years ago; per-second disaggregated billing is newer still. Both slotted into the coordinate system without remodelling, because they're new answers to the old questions, not new questions. We expect the same of whatever the market invents next.

The dimensions themselves we're happy to name — and the values above are examples, not the vocabulary. The full coordinate system — every value each dimension can take, and which combinations can actually exist in the market — is the part of nfer's model we deliberately keep to ourselves: it took months of iteration against real catalogues, and validating every offering against it is where the engineering lives.

When are two prices the same offer?#

The coordinate system tells you what kind of thing an offering is. It doesn't yet tell you when two observed prices belong to the same thing — and that question sounds pedantic right up until you try to build price history, or answer “cheapest”, without a firm answer to it.

Two prices describe the same offer only when everything a buyer would consider part of the deal agrees. These are the attributes that split offerings apart:

Identity attribute	Why it splits offerings	Concrete example
The provider	Two companies' H100s are never one product	Lambda's H100 and CoreWeave's H100 are different offers even at the same hourly rate
The model being served (when there is one)	A raw GPU rental bundles no model; an endpoint does	Llama 3.3 70B and Qwen behind the same provider's API are different offers
The hardware shape, and how many (when disclosed)	Bundle size changes both the price and what you're buying	An 8× H100 box, a 1× H100 VM and a quarter-H100 slice are three offers, not one “H100 price”
Where it runs	Sovereignty and price both differ by region	An H100 in Paris is not the same offer as the same H100 in Virginia — one passes an EU procurement review, and they rarely cost the same
The commercial tier	Commitment, spot, and product line change the deal itself	A 3-year commitment is not on-demand; a spot instance is not a reservation; a provider's pods and its serverless on the identical GPU are different products and must never merge
The served context-window tier	Providers price context tiers as separate products	The same model served at 200k context and at 1M context carries different per-token rates — two offers

Get this wrong in either direction and the comparison lies. Merge too eagerly and a provider's serverless price contaminates its pods' history, or a Paris price masquerades as a Virginia one. Split too eagerly and every collection run mints a fresh duplicate, the catalogue doubles weekly, and “history” means nothing because no offer lives longer than a day.

Get it right and two things follow. First, price history becomes real: a fresh observation lands on the same offer it priced last week, so a falling rate shows up as one line moving, not two unrelated dots. Second, “cheapest” becomes honest: every row in a ranking is a genuinely distinct deal — no duplicates padding the list, no incompatible tiers averaged together, no phantom option that's really the same product counted twice.

Rules that keep the numbers trustworthy#

A good ontology with sloppy data handling is still a bad comparison site. Four disciplines are non-negotiable in how we store prices.

Prices are stored exactly as published, in the provider's own currency. Hetzner prices in euros; we store euros. Conversion to your display currency happens at the moment you look, at a current rate. The alternative — converting at scrape time and baking that day's FX rate into the record forever — silently corrupts every non-USD price as exchange rates drift. It's a subtle failure mode, and most aggregators have it.

Price history is append-only. We never overwrite a price. Every collection run records a new observation, so when a provider cuts a rate — and in this market they cut them constantly — we can show you the move, not just the destination. Spot markets get the same treatment: successive observations preserve the curve. If you're timing a reservation, the trend matters as much as the current number.

Every offering keeps the stable identity described above. Today's collection run recognises the offer it saw yesterday and updates it rather than minting a duplicate — even when a scraper restarts mid-run, which is exactly when duplicates love to appear.

Freshness is judged at the level of whole successful collection runs. A product that disappears from a provider's catalogue disappears from ours. But a scraper that fails halfway — a moved page, a redesigned table, a 301 to somewhere new, all of which happen monthly — cannot wipe out a provider's listings. Only a complete, successful run gets to declare what still exists. Anyone who has scraped cloud pricing pages knows why this rule earned its place.

We wrote the buyers before we wrote the model#

How do you know a market model is right? Our answer: we wrote the buyers first. Before implementation, we drafted a set of concrete personas — real purchasing situations, not marketing archetypes — and turned their questions into executable acceptance tests, each with an expected answer computed by hand on paper, never by running the code first. The model ships only when every persona gets the number a careful human would have got.

Three of them, briefly:

The enterprise capacity planner needs guaranteed throughput for a frontier model and is comparing Azure PTUs, Vertex GSUs and Bedrock PTUs. Her test: “cheapest reserved capacity sustaining 10,000 output tokens per minute” must size each vendor's alien unit correctly — split tokens-per-minute here, mixed tokens-per-second there — and, crucially, must flag Bedrock's unpublished sizing as unsizeable rather than fabricating a number. If any vendor's quote can't honestly be computed, the honest answer is “contact sales”, and the model has to be able to say so.

The FinOps analyst tracks H100 rates weekly and normalises everything to dollars per GPU-hour. Her test ranks an eight-GPU box, a single-GPU VM, and a quarter-GPU serverless slice on one axis — and demands that on-demand, one-year, three-year and spot quotes for the same machine surface as distinct options rather than being averaged into mush.

The serverless developer ships a bursty image feature and buys per-second compute. His test caught a real bug: costing a Modal-style workload from the GPU meter alone understates the bill, because CPU and RAM are billed too. The hand-computed monthly total includes all three meters or the test fails. Another persona test caught rates so small — fractions of a thousandth of a cent per second — that a careless precision choice would have quietly rounded them into fiction.

These personas aren't documentation; they're a contract. When the market grows a new shape, the persona and its hand-computed answer come first, then the model earns the right to store it.

What this buys you#

Because every offering sits in one coordinate system with disciplined data behind it, we can do the things naive tables can't: compare a per-token API against a rented GPU against a provisioned-throughput commitment for your workload, in your currency, without hidden conversion assumptions; show you how a price has moved, not just where it is; and tell you plainly when a vendor's pricing is genuinely opaque instead of decorating it with an invented number.

The comparison is live at nfercost.com, with per-model deployment breakdowns under nfercost.com/host and our methodology written up at nfercost.com/methodology. Bring an awkward workload — that's what the model was built for.