System Design · Issue 5

How to Price Serverless GPUs

One ratio decides whether a reserved GPU contract or pay-as-you-go is cheaper. Here is the math Modal uses.

1 August 2026 A finance team and an engineering team disagreeing over a GPU spending chart

The disagreement

Paying for peak, earning on average

A finance team and an engineering team disagreeing over a GPU spending chart

Modal's pricing post opens with a line from Marc Brooker about the gap between paying for peak and earning on average. That gap is the whole story. Buy enough GPUs to cover your busiest second, and you pay for that capacity every second, including the many seconds you barely use it.

The real problem is organizational. Engineering teams want to ship and reach for whatever infra is fastest. Procurement and finance teams want to control spend and reach for long reservations. They argue past each other because neither side has the other's context: engineers lack the market rates, finance lacks the workload shape.

The question is not which GPU is cheaper per hour. It is which one is cheaper across a demand curve that moves all day.

Two ways to pay

The serverless bill and the reserved bill

Serverless capacity tracking demand versus reserved capacity fixed at peak

Modal builds the comparison from two months of real customer usage, a demand(t) curve that is spiky and uneven. Then it prices both options against that same curve using B200 rates on each side: Modal serverless versus a 3-year hyperscaler reservation.

Serverless: pay for what you use

C_s = R_s \times \sum_t \text{demand}(t)

Assuming allocation is instant, you pay the serverless rate for exactly the demand at each instant.

Reserved: pay for the peak, all the time

C_r = R_r \times T \times \max_t \text{demand}(t)

A reservation must cover peak demand for the full contract duration T.

Both formulas are written charitably. Serverless is assumed to allocate instantly, and the reservation is assumed to predict demand perfectly and size exactly to peak. Each assumption flatters one side, which matters later.

The derivation

Two numbers decide everything

Divide the reserved bill by the serverless bill and the contract duration T drops out, because the sum of demand over T is just average demand times T. What survives is startlingly small.

\frac{C_r}{C_s} = \frac{R_r}{R_s} \times \frac{\max \text{demand}}{\text{avg demand}}

Cost ratio = reservation discount rate × peak-to-average ratio.

That is the entire model. The reservation's per-hour discount pulls its cost down, and the workload's peak-to-average ratio pushes it back up because you paid for the peak but only used the average. Whichever number is bigger wins.

If reserved is 10x cheaper per hour but your workload peaks at 10:1, the two bills are identical.

The decision rule

Why serverless usually wins the math

A scale tipping because peak-to-average outweighs the reservation discount

Now plug in the observed ranges. Hyperscaler 3-year reservations run about 2x to 5x cheaper per hour than serverless. AI workloads, inference, training, and agentic dev, tend to have peak-to-average ratios of 5x to 10x, sometimes 10:1.

Discount 2–5x

how much cheaper reserved is per hour

Peak/avg 5–10x

how much you overpay covering the peak

Because the peak-to-average ratio usually beats the discount, serverless comes out cheaper on raw compute in the common case. For burst-only capacity layered on a reserved baseline, ratios climb to 10x, 100x, which is where serverless becomes hardest to argue against.

The same tradeoff, flipped

Read it as a utilization threshold

A utilization gauge marking the break-even point around 30 percent

Peak-to-average and utilization are the same idea inverted. A workload that peaks 10x above its average keeps its reserved fleet busy only a tenth of the time. Third-party guides land on a rule of thumb: serverless wins below roughly 30% utilization, reserved wins above it.

\text{break-even util} = \frac{\text{reserved hourly cost}}{\text{serverless per-sec cost} \times 3600}

The utilization above which a reservation starts to pay off.

That threshold is where the argument bites, because Modal cites industry surveys putting real reserved GPU utilization under 30%, often under 10%. If that number is right, most reservations sit idle most of the time, and the peak-to-average ratio is brutal by default. Worth flagging that Modal gives no study name for it, and it carries a lot of the argument's weight.

The load-bearing assumption

Making instant allocation real

Modal's scheduler allocating GPUs across many providers using snapshots

The serverless bill only holds if allocation really is near-instant. Modal's whole infra exists to make that assumption defensible: a linear-programming scheduler placing instances, health-checking across over a dozen cloud providers, and access that scales to hundreds or thousands of GPUs on the enterprise tier.

The hard part is cold starts. A real inference server like vLLM does not blink into existence. Modal's answer is container memory snapshotting: capture the process after the model is loaded and warmed, then restore from that image instead of booting from scratch.

cold boot from scratch      restore from snapshot
  load weights   ~460s        wake from image   ~70s
  jit + cuda graphs            (compile, cuda graphs,
  warmup requests               spec-decoding all
     |                          captured in snapshot)
     v                              |
  serving                           v
                                 serving

An independent reproduction took vLLM cold start from ~460s to ~70s with GPU memory snapshot + sleep mode.

The snapshot trick matters more than a plain speedup. The restored process keeps its production optimizations, torch.compile, CUDA graphs, speculative decoding, because they were captured in the image rather than disabled to boot faster. That is what lets Modal claim complex servers start in seconds, which is what keeps buffers and warm pools out of the serverless bill.

The two hooks that make it work

In Modal's primitives, @modal.enter(snap=True) runs once before capture: it loads the model, fires warmup requests to force JIT and CUDA-graph compilation, then moves weights to CPU RAM via vLLM's sleep mode. @modal.enter(snap=False) runs on every restore, waking the model back onto the GPU. The snapshot needs 3 to 5 cold boots to stabilize before it is trusted on the restore path.

Where the model bends

The assumptions that flatter each side

A clean formula cracking at four assumption points

Modal is unusually honest that its own model is rigged in two directions. Perfect prediction flatters the reservation: real forecasts miss, and both misses hurt. Under-provision and you get SLO violations, queues, and an emergency procurement renegotiation under duress. Over-provision and you get wasted spend and a slow, after-the-fact budget review.

Instant allocation flatters serverless. When cold starts are too slow, you keep warm buffers, and that cost creeps back into the serverless bill. At micro scale it is the same trap: keeping an H100 warm across a 5-minute window to serve 20 seconds of real work bills roughly 15x the GPU-time you actually used.

Perfect prediction (helps reserved)Instant allocation (helps serverless)

Each charitable assumption tilts the result toward one option.

Two more gaps. The model is all-or-nothing: a hybrid of reserved baseline plus serverless burst can win on unit economics but adds the operational cost of running two systems and two vendors, and Modal says a combined model is still only something it is exploring. And it treats GPU cost as the only factor, ignoring that fast alloc/dealloc lets you develop against prod-identical infra, cutting defects and dev time in ways the pure cost math never sees.

The bottom line

One ratio, honestly caveated

Why it matters

GPU spend has become a boardroom fight, and Modal's contribution is to compress it into one comparison a finance lead and an engineer can both read: the reservation's hourly discount against the workload's peak-to-average ratio. Because AI demand is spiky and reserved fleets sit mostly idle, that ratio usually favors pay-as-you-go, but only if cold starts really are near-instant. The whole argument rests on Modal's snapshotting engineering holding up in production; where it does not, buffers quietly restore the reservation's case.

The short version

The cost ratio reduces to **reservation discount rate × peak-to-average ratio**; the bigger factor wins.
Typical discounts run **2–5x** while AI workloads peak **5–10x** above average, so serverless usually wins on raw compute.
The same tradeoff reads as a **~30% utilization** break-even; reserved fleets often run under 10%.
The math assumes **instant allocation**, which Modal backs with container snapshotting that restores vLLM in ~70s with full optimizations intact.
The model ignores hybrid strategies, warm-pool costs, and developer-velocity gains, and its key utilization stat is an unattributed survey claim.