I have spent twenty years building cloud infrastructure, and I have never seen a category evolve as fast as GPU compute. Three years ago, the conversation was “should we buy H100s or rent them.” Two years ago it was “should we use managed endpoints or run our own.” Today the question is: “why are we managing inference infrastructure at all when serverless GPU platforms can do it better and cheaper for most workloads?”
The answer, for a growing number of teams, is: you probably should not be managing it. But the serverless GPU landscape is genuinely confusing, the marketing is thick, and the wrong choice costs real money. Modal and RunPod are both “serverless GPU” but they solve different problems. Together.ai and Fireworks.ai look like the same product until you try to deploy a fine-tuned model. Baseten is excellent until it is not.
This is my attempt to cut through that noise with something actually useful: when each platform makes sense, where each one breaks down, and how I think about the decision for different team sizes and workload types.
Why You Are Looking at This Category
The traditional paths for GPU inference were:
Managed API endpoints (OpenAI, Anthropic, Google): no infrastructure work, but you are locked to their model catalog and their pricing. The moment you fine-tune a model or need something not in their catalog, you are stuck.
Cloud provider GPU instances (AWS p3/p4/p5, GCP A100s, Azure NDv5): you control everything, but you are paying for idle time, you manage node failures yourself, and your ops team gets paged at 3am when a driver update corrupts your model serving container. I have been that ops team.
Self-managed Kubernetes with GPU nodes: the “right” architectural answer that involves Karpenter, node provisioners, Kubernetes DRA for GPU scheduling, custom resource management, and a dedicated platform team to keep it running. Excellent at scale. Overkill for most teams under $10M ARR.
Serverless GPU platforms try to give you the control of option 2 without the operational burden, at pricing somewhere between the extremes. The key innovation is genuine scale-to-zero: you pay only when requests are executing, measured in per-second or per-millisecond increments.
This changes the economics fundamentally. If your inference workload runs at 30% utilization, you are wasting 70% of your budget on a dedicated instance. With scale-to-zero, you pay for roughly 30% of what you would on a dedicated node, minus the platform markup. For bursty workloads, the math gets even better.
The catch is cold starts. A scale-to-zero system that takes 60 seconds to cold start is unusable for interactive applications. How each platform handles this problem is the most important thing to understand.
The Main Players
Modal
Modal is the platform I recommend most often to teams building custom inference pipelines. The developer experience is genuinely excellent: you decorate a Python function with @app.function(gpu="H100"), run modal deploy, and you have a serverless endpoint. No YAML, no Kubernetes manifests, no Dockerfile debugging.
What makes Modal technically interesting is their container image snapshotting approach to cold starts. They snapshot the memory state of your container after your model loads, so a cold start restores from that snapshot rather than re-executing your initialization code. For vLLM serving a 70B model, this can take a cold start from 120 seconds down to 10-15 seconds. For smaller models, you can hit sub-5-second cold starts.
The pricing is straightforward: you pay per second of GPU time, plus a small charge for data transfer. H100 SXM runs around $4/hour-equivalent (billed per second), which is competitive with on-demand cloud pricing once you factor in the zero-overhead operations model.
Where Modal gets complicated is at production scale. The platform does not give you the same level of control over networking, multi-region routing, or traffic shaping that you get with self-managed infrastructure. If you need strict data residency, Modal’s US-only infrastructure (as of mid-2026) may be a problem. Their burst limits can also bite you during traffic spikes if you have not pre-negotiated capacity with their team.
The team skews toward developers building AI applications rather than infrastructure engineers. That is a feature if you want fast iteration; it can feel like a limitation if you want detailed controls over scheduling and placement.
RunPod Serverless
RunPod built their serverless product on top of their community GPU marketplace, which gives them a different cost structure than Modal. You can configure your endpoints to use community GPUs (much cheaper, ~40-50% cheaper than Modal for equivalent hardware), reserved fleet GPUs, or a mix.
The FlashBoot feature is RunPod’s answer to the cold start problem. They pre-warm containers and use their own snapshot mechanism, targeting sub-200ms cold starts for compatible workloads. In practice, I have seen cold starts ranging from 1-30 seconds depending on container size and model loading approach. Not as consistently fast as Modal’s snapshotting, but the pricing differential on community GPUs can justify it for batch or near-real-time workloads.
RunPod’s strengths are cost and flexibility. You can deploy any Docker container, use any model, configure worker counts and scaling policies in detail, and get access to a wider range of GPU types (including older V100s and A10Gs at very low prices that are fine for smaller models).
The weakness is that community GPU reliability is genuinely variable. I have seen workers go offline mid-batch. For production inference where reliability matters more than cost, I steer toward RunPod’s dedicated fleet or Modal rather than community workers.
Together.ai and Fireworks.ai
These two platforms occupy a different niche: they provide API-based access to open-source models with inference performance tuned at the platform level. You are not deploying your own container; you are calling an API for Llama 3.3, Mistral, DeepSeek R1, and dozens of other models.
Together.ai and Fireworks.ai are the answer when you want to move off OpenAI/Anthropic but your model needs are met by the open-source ecosystem. Pricing is typically 50-80% cheaper than equivalent OpenAI pricing, and throughput is often better for latency-sensitive applications because they invest heavily in inference optimization (custom CUDA kernels, speculative decoding, continuous batching at scale).
The limitation is obvious: you cannot deploy a custom or fine-tuned model. If your competitive advantage is a fine-tuned model on proprietary data, these platforms are not the right fit. They are also not useful for workloads that require privacy guarantees around the model itself, since you are sending data to a shared inference cluster.
Fireworks.ai differentiates on their serverless fine-tuning feature: you can upload a fine-tuned adapter and deploy it alongside base models. This is genuinely useful for teams that have fine-tuned models but do not want to build inference infrastructure. The pricing for fine-tuned adapter hosting is reasonable.
Together.ai’s strength is their commitment to open-source model support. They typically have new flagship models available within days of release, and their API compatibility with OpenAI’s format makes migration simple.
Baseten
Baseten is the enterprise-oriented option. Their pitch is “managed Triton inference server plus your model,” and it holds up well in practice. You define your model with their Truss framework (a structured approach to model packaging), push it to their platform, and they handle scaling, load balancing, and serving optimization.
Where Baseten shines is for teams that have already standardized on Triton or NVIDIA’s inference stack and want managed hosting without rebuilding everything. The enterprise tier includes SLAs, dedicated infrastructure, private deployments, and the kind of support relationship that large companies require.
The trade-off is developer experience: Baseten is more opinionated and has more ceremony than Modal. If you are iterating quickly on model architectures or serving code, the Truss packaging layer adds friction. For stable, production-grade deployments that need enterprise support contracts, it is worth it.
The Cold Start Problem in Depth
Cold starts are the tax you pay for scale-to-zero economics, and they are the most common source of production incidents on serverless GPU platforms. Understanding what causes them helps you design around them.
A cold start has several phases: container image pull, container startup, Python runtime initialization, library imports (torch alone can take 3-5 seconds), and model weight loading (a 70B model in float16 is 140GB; loading from network storage at 10 GB/s takes 14 seconds). Add these up and you can easily hit 60-120 seconds for large models.
The mitigation strategies:
Memory snapshotting (Modal’s approach): Snapshot the container state after model load, restore from snapshot on cold start. Eliminates model loading time, reduces cold start to container restore time plus any remaining initialization. Most effective approach available today.
Pre-warming: Keep a minimum number of warm workers running at all times. Eliminates cold starts entirely at the cost of paying for idle compute. RunPod and Baseten both support this. Use it for latency-critical paths where you can predict minimum load.
Quantization: Smaller models load faster. A 7B model quantized to INT4 with AWQ or GPTQ might be 4GB; loading time becomes negligible. If your accuracy requirements allow it, quantization is the highest-leverage cold start optimization.
Container optimization: Minimize image layers, use multi-stage builds, move model weights to fast network storage. These are table-stakes optimizations that all platforms benefit from.
The design pattern I use in production: for interactive endpoints that serve user-facing requests, pre-warm at least one worker at all times and use snapshotting to minimize the time for additional workers to start. For batch or background inference, accept cold starts and design your job submission system to tolerate 30-60 second delays.
Pricing Realities
Every serverless GPU platform claims cost savings, and the true picture requires you to model your actual usage pattern. Here is my framework:
High-utilization workloads (consistently above 60% usage): Dedicated GPU instances or reserved capacity from any of these platforms will beat serverless pricing. Serverless pricing includes a platform margin on top of raw compute cost. If you are going to use 90% of a GPU, buy the GPU.
Medium-utilization workloads (20-60% usage): This is where serverless starts winning. The savings from scale-to-zero offset the per-second premium. Modal or RunPod are typically cost-effective here.
Low-utilization or bursty workloads (under 20% average but with spikes): Serverless wins decisively. Paying for compute only when processing requests can reduce costs 5x or more compared to maintaining an idle instance. This is the sweet spot for AI features in applications that are not primarily AI products.
Token-per-dollar comparison for API platforms: Together.ai and Fireworks.ai are consistently cheaper than OpenAI/Anthropic for equivalent model quality at comparable parameter counts. The gap has narrowed as the hyperscalers have cut prices, but open-source serving platforms still have a meaningful advantage for high-volume inference.
One hidden cost: data transfer. Several platforms charge for output tokens or data egress. For applications with long outputs or heavy throughput, these charges can materially affect your effective per-request cost. Model it explicitly.
The AI FinOps discipline applies here: tag every inference request with the feature and customer tier that triggered it, measure cost per outcome rather than cost per GPU-hour, and set up budget alerts before you deploy production workloads.
When to Use Which Platform
Here is my decision framework after working with all of these in production:
Use Modal when: You are a development-velocity-focused team deploying custom inference code. You want the fastest path from Python function to production endpoint. You are building new AI features and expect to iterate frequently on serving logic. Cold start performance is important.
Use RunPod when: Cost is the primary constraint and you can tolerate some reliability variability. You need a wider range of older or cheaper GPU types. You are running batch inference where cold starts are acceptable and per-second pricing matters more than latency.
Use Together.ai or Fireworks.ai when: You are using open-source models from the standard catalog without fine-tuning. You want the simplest possible migration from closed-source API providers. You need high throughput at low cost and your model needs are met by the available catalog.
Use Baseten when: You are in an enterprise environment that requires SLAs, dedicated infrastructure, and formal support relationships. You are already standardized on NVIDIA’s inference stack. Your team has Triton expertise and prefers a structured model packaging approach.
Use self-managed Kubernetes + Karpenter when: You are at a scale where the platform margin on serverless costs more than a dedicated platform team. You need multi-region active-active inference routing. You have compliance requirements that prohibit shared infrastructure. You are building a product where inference infrastructure is a core competitive differentiator.
Production Patterns
A few patterns that have served me well:
The Fallback Chain: Do not single-source your inference. For critical paths, configure a primary provider and a fallback. Modal primary, RunPod fallback. Together.ai primary, Fireworks.ai fallback. The cost of multi-provider complexity is low; the cost of a single-provider outage in a user-facing AI feature is high. The AI gateway pattern applies here: route through a proxy layer that handles provider selection and fallback logic rather than baking it into application code.
Async Inference for Non-Interactive Paths: For use cases that are not blocking a user response (document summarization, batch embeddings, background processing), use async invocation with a queue. This lets your serverless workers process at their own rate without cold start latency being a user-facing problem. Combined with agentic AI workload patterns, async inference can dramatically reduce your cost basis.
Warm Pool Sizing: For Modal specifically, the minimum workers setting is your primary lever for trading off cost versus latency. Start with zero warm workers in development, one in staging, and two-to-four in production for interactive paths. Revisit this as your traffic patterns become clearer.
Observability: Serverless GPU platforms provide limited built-in observability. Instrument your inference code to emit latency, token counts, and error rates to your own observability stack. The LLM observability practices from self-hosted deployments apply equally here. Treat the serverless platform as a black box for compute, but maintain full visibility into what your code is doing on top of it.
Secret Management: Never hardcode model weights’ storage credentials or API keys in your container images. Use your cloud provider’s secrets management or HashiCorp Vault, inject at runtime. This is table stakes, but I have seen teams get sloppy about it when moving fast with new platforms.
The Honest Assessment
The serverless GPU platforms have reached a maturity level where they are genuinely viable for production AI workloads, not just prototypes. Modal in particular has done impressive engineering work on cold start reduction that changes the economics of scale-to-zero for interactive inference.
The category is not a replacement for all self-managed inference. If you are at Netflix-scale or if inference infrastructure is your actual product, you need the control that only self-managed gives you. The LLM inference engines comparison and the GPU cluster networking deep dive are still relevant for teams at that scale.
But for the large middle ground of companies building AI-powered products, the serverless platforms let you move faster and spend less on operations. I have seen teams go from “we need to hire a GPU infra engineer” to “we deployed this in a week using Modal” too many times to dismiss the category.
The real risk is lock-in. Modal’s deployment format is not portable. RunPod Serverless workers are specific to their platform. If you need to migrate, you are re-implementing your serving code. Design your inference layer with a clean interface boundary: your application calls an inference endpoint, the specifics of what is behind that endpoint should be an implementation detail that you can swap.
Pick the platform that fits your team’s skills and workload characteristics, keep the interface clean, and revisit the decision as your scale changes. That is the same advice I give for every infrastructure decision, and it holds here too.
For teams just getting started: Modal for custom models, Together.ai for catalog models. Get your product working and revisit infrastructure when the costs and requirements become clearer. Do not optimize this too early.
If you are evaluating whether to move AI workloads to these platforms, start with spot instances and preemptible VMs as a baseline for cost comparison, then model the serverless economics against your actual traffic pattern. The numbers will tell you where the break-even point is.
Get Cloud Architecture Insights
Practical deep dives on infrastructure, security, and scaling. No spam, no fluff.
By subscribing, you agree to receive emails. Unsubscribe anytime.