Fractional GPUs in Kubernetes: MIG, Time-Slicing, and MPS Explained for Production AI Workloads

Here is a number that should make you wince: the average GPU utilization in production Kubernetes clusters is 5%. I have seen the Cast AI 2026 State of Kubernetes Optimization Report, and I have seen similar numbers internally on every platform I have audited. Five percent. You are paying for 100 units of compute and using five.

I spent six months in 2023 helping a mid-sized AI startup get their GPU bills under control. They were running H100s on AWS p5 instances, one workload per node, GPU sitting idle between inference requests, nobody questioning it because “that’s just how GPU workloads work.” By the time we were done, we had tripled effective GPU throughput on the same hardware budget through a combination of NVIDIA Multi-Instance GPU partitioning and request-level time-slicing. The bill dropped 60%. The workloads ran fine.

The tools exist. Most engineers just haven’t learned them yet because GPU scheduling is one of those topics that lives in NVIDIA documentation, Kubernetes operator docs, and scattered blog posts rather than anywhere coherent. This article is my attempt to fix that.

The Problem: GPUs Are Expensive and Most Workloads Don’t Need a Whole One

A single NVIDIA H100 SXM5 80GB costs around $30,000 to buy outright, or roughly $3 to $4 per hour in cloud GPU marketplaces. If your inference service is handling 10 requests per second with an average latency of 200ms, the GPU is active roughly 2% of the time. The other 98% it sits idle, burning money.

The naive solution is to run more services per GPU. The problem is that Kubernetes resource management only understands whole GPUs by default. When you request nvidia.com/gpu: 1, you get an entire physical GPU to yourself. Every other pod is locked out. This is sensible for isolation but catastrophic for utilization.

NVIDIA has three main mechanisms for sharing GPUs across workloads, and they have meaningfully different properties:

1. Time-Slicing: Multiple CUDA contexts take turns on the same physical GPU using context switching. No memory isolation between tenants.
2. Multi-Instance GPU (MIG): Hardware-level partitioning of the GPU into fully isolated instances with dedicated memory and compute slices. Supported on A100, H100, and H200.
3. Multi-Process Service (MPS): A CUDA server that multiplexes multiple processes onto one GPU with shared memory space, reducing context-switch overhead.

Each mechanism suits different workloads. Getting this wrong means either poor isolation (bad in multi-tenant environments) or leaving performance on the table.

Time-Slicing: The Easy Button

Time-slicing is how most teams start. It is simple to configure, works on any NVIDIA GPU (not just the newest datacenter cards), and requires no special hardware support.

The mechanism is exactly what it sounds like. When multiple CUDA contexts are active on a GPU, NVIDIA’s kernel driver switches between them on a time-shared basis. Each context gets a slice of GPU time. This is similar to how a CPU runs multiple processes, except GPU context switches are more expensive because there is more state to save and restore.

From a Kubernetes perspective, time-slicing lets you define a “replica count” on a device. If you set replicas to 10 on a single A10G, Kubernetes will advertise 10 nvidia.com/gpu resources on that node. Ten different pods can each request nvidia.com/gpu: 1 and get scheduled there.

The NVIDIA GPU Operator handles this via a ConfigMap. A basic time-slicing configuration looks like this in your cluster:

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  any: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        replicas: 10    

Apply this, patch your ClusterPolicy to reference it, and the GPU Operator reconfigures the node. Pods requesting nvidia.com/gpu: 1 start getting scheduled 10-per-GPU instead of 1-per-GPU.

What time-slicing does not give you: memory isolation. All tenants share the full GPU memory space. If one pod allocates 60GB of VRAM on an 80GB H100, there is only 20GB left for everyone else, and nothing stops any pod from over-allocating and causing out-of-memory errors for its neighbors. You also get no compute isolation guarantees. One noisy neighbor doing heavy batch processing can starve your latency-sensitive inference service.

Time-slicing works well in these scenarios:

• Development and experimentation clusters where isolation is not a concern
• Workloads with naturally low and bursty GPU utilization (think interactive notebooks)
• Older GPU hardware (V100, A10G) that does not support MIG
• Situations where you need maximum flexibility in pod packing

I would not run production inference for paying customers on time-slicing without additional admission controls to limit VRAM per pod and careful bin-packing of workloads that have complementary utilization curves.

Multi-Instance GPU: Real Isolation at the Hardware Level

MIG changes the game entirely. Instead of sharing a physical GPU through context switching, MIG partitions it into independent hardware slices at the silicon level. Each MIG instance gets dedicated streaming multiprocessors (SMs), dedicated L2 cache, dedicated HBM memory bandwidth, and its own memory partition. A fault in one MIG instance cannot affect another.

MIG is supported on A100 (40GB and 80GB), A30, H100, and H200. On an H100 SXM5 80GB, you can create up to 7 MIG instances of a specific profile. The profiles determine the size of each instance.

The available profiles on an H100 80GB follow naming conventions like 1g.10gb, 2g.20gb, 3g.40gb, 4g.40gb, and 7g.80gb. The number before g is the GPU slice count (out of 7 total), and the number after is the memory allocation.

Common partitioning strategies for an H100 80GB:

• 7x 1g.10gb: Maximum density, seven isolated 10GB instances. Good for small models or batch jobs.
• 1x 3g.40gb + 2x 2g.20gb: Balanced for mixed workloads.
• 2x 3g.40gb + 1x 1g.10gb: Two medium inference instances plus a small dev instance.
• 1x 7g.80gb: No partitioning, full GPU, useful when one workload needs the whole card.

The key insight is that you configure MIG profiles per node based on your actual workload mix, not on some fixed global setting.

Here is where it gets operationally interesting. The NVIDIA GPU Operator manages MIG configuration through a ConfigMap and a MigStrategy setting. You choose either single strategy (every MIG instance on a node exposes the same resource name) or mixed strategy (different profiles get different resource names, allowing fine-grained pod scheduling).

With mixed MIG strategy, Kubernetes nodes advertise resources like:

• nvidia.com/mig-1g.10gb
• nvidia.com/mig-3g.40gb
• nvidia.com/mig-7g.80gb

Your pods request specific MIG profiles. A small inference pod requests nvidia.com/mig-1g.10gb: 1. A large training job requests nvidia.com/mig-3g.40gb: 1. The scheduler matches them to appropriate partitions. This is proper multi-tenancy on a single GPU card.

The operational catch with MIG: you cannot change profiles on the fly without disrupting running workloads. When you reconfigure MIG partitions, existing GPU contexts are destroyed. Plan your partition strategy around your longest-running workloads and build your node selection strategy around stable profiles. I have seen teams try to dynamically repartition nodes as workloads change, and it gets messy quickly. Static profiles per node class (inference nodes, training nodes, dev nodes) are much more manageable.

For teams already using Kubernetes Dynamic Resource Allocation (DRA) for GPU scheduling, MIG integrates naturally because DRA treats resources as structured claims rather than simple integer counts. Read more about how DRA changes GPU scheduling for AI workloads.

MPS: When You Want Throughput, Not Isolation

Multi-Process Service (MPS) is the third option, and it is the least commonly understood.

MPS works by running a single CUDA context that all GPU clients share. Instead of each process having its own context that takes turns on the hardware, all processes submit work to the MPS daemon, which funnels everything through a single context. This eliminates the overhead of context switching and allows multiple processes to run CUDA kernels simultaneously without the cost of full context saves and restores.

The performance gains are real. On batch inference workloads where many small processes are submitting work concurrently, MPS can improve throughput by 20-40% compared to naive time-slicing. CUDA MPS also supports memory limits per client (a feature added in CUDA 11.x), which gives you some protection against one process consuming all VRAM.

The downsides: MPS does not provide fault isolation. If the MPS daemon crashes, all clients crash. If one client triggers a GPU fault (kernel timeout, memory corruption), every client sharing that MPS instance can be affected. Also, MPS is not compatible with MIG. You pick one or the other per device.

MPS makes sense for:

• Homogeneous batch inference workloads on a single large model
• Scenarios where you control all tenants on the node (no untrusted code)
• Maximizing throughput on inference servers that handle many small requests

I have used MPS successfully with vLLM when running multiple LoRA adapter variants concurrently on the same model. The base model weights live in shared VRAM, LoRA weights are per-tenant, and MPS keeps the GPU busy across concurrent requests. For more on how inference engines like vLLM handle this, see LLM inference engines compared.

Choosing the Right Approach

Here is how I frame the decision for teams I advise:

Use MIG when: you have A100, H100, or H200 hardware; you need strict isolation between tenants; you have a relatively stable workload profile that maps cleanly to available MIG slice sizes; or you are running production inference for multiple internal teams or customers who should not be able to interfere with each other.

Use Time-Slicing when: you need to support older GPU hardware (V100, A10, A10G); your workloads are primarily development/experimentation; utilization is naturally low and bursty; or you need the flexibility to run workloads of arbitrary sizes without being constrained to fixed slice profiles.

Use MPS when: you want maximum throughput on a homogeneous workload; you control all processes sharing the GPU; you are comfortable with the fault blast radius; and you need to squeeze performance out of a GPU beyond what time-slicing can deliver.

Combine MIG and Time-Slicing: This is increasingly common. You can enable time-slicing on MIG instances. A 1g.10gb MIG instance can itself be time-sliced with a replica count of 5, effectively creating 5 Kubernetes resources from a single small hardware partition. This layering gives you isolation at the MIG boundary and density at the time-slicing layer.

Configuring the NVIDIA GPU Operator for Fractional GPUs

The NVIDIA GPU Operator is the recommended path for managing GPU nodes in Kubernetes. It handles driver installation, the device plugin, MIG configuration, MPS, DCGM exporter (GPU metrics), and Node Feature Discovery.

Here is a minimal GPU Operator deployment via Helm that enables MIG support:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --set driver.enabled=false \  # if driver is pre-installed
  --set mig.strategy=mixed \
  --set devicePlugin.config.name=time-slicing-config

For MIG, you configure the partition layout by labeling nodes and providing a MIG configuration ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: mig-parted-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    mig-configs:
      all-1g.10gb:
        - devices: all
          mig-enabled: true
          mig-devices:
            "1g.10gb": 7
      mixed-inference-training:
        - devices: all
          mig-enabled: true
          mig-devices:
            "3g.40gb": 1
            "2g.20gb": 2
            "1g.10gb": 1    

Then label each node with the desired config:

kubectl label node gpu-node-01 nvidia.com/mig.config=all-1g.10gb
kubectl label node gpu-node-02 nvidia.com/mig.config=mixed-inference-training

The gpu-operator-mig-manager DaemonSet picks up the label and applies the partition configuration. If running pods are using the GPU, MIG reconfiguration will drain them first (or fail gracefully, depending on your configuration).

For cost visibility across your GPU fleet, you can combine this with OpenCost or Kubecost to track per-MIG-instance cost allocation and showback reports.

Monitoring GPU Utilization Per Slice

DCGM (Data Center GPU Manager) exports per-MIG-instance metrics when enabled. Key metrics to watch:

• DCGM_FI_PROF_GR_ENGINE_ACTIVE: GPU engine active time (utilization proxy)
• DCGM_FI_DEV_FB_USED: Framebuffer (VRAM) used per device/instance
• DCGM_FI_DEV_GPU_TEMP: Temperature per GPU
• DCGM_FI_PROF_DRAM_ACTIVE: Memory bandwidth active rate

With DCGM Exporter running in the GPU Operator, these metrics flow into Prometheus and render in Grafana dashboards. If you are running the Prometheus/Grafana observability stack, GPU metrics integrate cleanly via the DCGM ServiceMonitor.

Alert on:

• DCGM_FI_DEV_FB_USED approaching 90% of instance memory (OOM risk)
• DCGM_FI_PROF_GR_ENGINE_ACTIVE below 10% sustained (underutilization, candidate for time-slicing)
• High temperature sustained above 85C (thermal throttling affects performance)

Karpenter and Fractional GPU Node Provisioning

If you are using Karpenter for node provisioning, fractional GPU configurations require some thought. Karpenter provisions entire nodes and lets Kubernetes schedule pods onto them. For MIG, you need to ensure node provisioning templates request GPU instance types that support MIG (A100, H100, H200) and that the MIG configuration labels are applied automatically as part of node bootstrap.

A common pattern is to use EC2NodeClass userData to apply MIG node labels at boot time, and configure Karpenter NodePools to target specific GPU instance families:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-mig-h100
spec:
  template:
    metadata:
      labels:
        nvidia.com/mig.config: all-1g.10gb
    spec:
      nodeClassRef:
        apiVersion: karpenter.k8s.aws/v1
        kind: EC2NodeClass
        name: gpu-h100-class
      requirements:
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["p5.48xlarge"]

This pairs well with Kueue for batch AI job scheduling, where jobs queue until the right MIG profile is available, preventing over-scheduling.

Production War Stories: What I Have Learned the Hard Way

Story 1: The MIG reconfiguration incident.

A team I worked with configured MIG mixed mode on their H100 nodes and set up an automated script to repartition nodes as workload profiles changed throughout the day. In theory, small inference jobs in the morning would reclaim GPU slices for large training jobs overnight. In practice, the reconfiguration script fired during an inference burst, killed in-flight requests, and triggered a cascade of errors upstream. The on-call team spent three hours debugging what they thought was an application bug. It was the MIG manager draining the node.

Lesson: treat MIG reconfiguration as a disruptive maintenance event. Do not automate it without explicit drain, cordon, and validation steps. Static profiles per node class are boring and reliable.

Story 2: Time-slicing and VRAM exhaustion.

Another team deployed 20 replicas of a time-sliced H100 (setting replicas to 20 seemed generous, right?). Each pod loaded a 7B parameter model in float16 (roughly 14GB VRAM). 20 x 14GB = 280GB, on an 80GB card. The first five pods loaded fine. The sixth triggered CUDA out-of-memory errors on pods 1-5 as the kernel started swapping VRAM. The entire node became unstable within minutes.

Lesson: time-slicing does not enforce VRAM limits. You must enforce this at the application level or via LimitRange. At minimum, document your expected VRAM per workload and do the math before setting replica counts.

Story 3: MPS and fault blast radius.

I ran MPS in production briefly for a batch inference cluster. A bug in a customer’s custom CUDA kernel triggered a GPU fault. MPS propagated that fault to all other inference processes sharing the daemon. 50 concurrent inference jobs failed simultaneously. The blast radius was unacceptable for our SLA.

Lesson: MPS is powerful but the shared fate model is real. If you use it in production, design for rapid restart of the MPS daemon and ensure your inference service handles reconnection gracefully. Or just use MIG instead if your hardware supports it.

What Is Coming: CNCF Ownership of the NVIDIA DRA Driver

At KubeCon Europe 2026, NVIDIA announced it is donating the NVIDIA DRA (Dynamic Resource Allocation) Driver for GPUs to the CNCF. This is a significant shift. It moves GPU resource management from vendor-controlled software toward community-governed tooling under the Kubernetes project umbrella.

What this means practically: the DRA GPU driver, which handles structured resource claims for MIG instances, time-slicing replicas, and MPS configurations, will eventually be maintained under CNCF governance with broader community input. Expect better integration with cluster autoscalers, more interoperable resource models across vendors (AMD ROCm, Intel GPU), and slower feature velocity in the short term while CNCF governance processes mature.

For teams already using DRA for GPU workloads, the transition will be gradual. NVIDIA will maintain the driver through the CNCF transition period. For teams not yet on DRA, this is a good signal that the DRA API is the direction the ecosystem is moving.

The goal for AI infrastructure teams: run fractional GPUs through the GPU Operator today, adopt DRA resource claims as they stabilize, and position your platform to benefit from the cross-vendor GPU resource model that the CNCF governance model enables. For context on why AI FinOps and GPU cost control matter here, fractional GPU sharing is one of the highest-leverage levers you have.

Putting It Together: A Practical Starting Point

Here is my recommended starting point for most teams:

1. Audit your current GPU utilization with DCGM Exporter and Grafana. If sustained engine active time is below 30%, you have a utilization problem.
2. Identify your GPU hardware: A100, H100, H200 support MIG. Everything else falls back to time-slicing.
3. For production inference workloads on supported hardware: enable MIG in mixed mode. Start with 2g.20gb profiles for most 7B-13B models and 3g.40gb for larger ones. Adjust based on actual VRAM usage.
4. For development clusters or older hardware: enable time-slicing with conservative replica counts (4-8 per GPU), and enforce VRAM limits at the application level.
5. For batch throughput on homogeneous workloads: evaluate MPS, but test your fault recovery story first.
6. Wire up DCGM metrics and set up utilization alerts. Target above 40% GPU engine active time as a baseline, 60%+ as healthy.

The 5% average GPU utilization number is not destiny. It is a symptom of teams deploying GPU workloads the same way they deploy CPU workloads (one job, one resource, move on). GPU infrastructure rewards intentional resource design. MIG, time-slicing, and MPS are the tools that turn idle silicon into real capacity.

The hardware is expensive enough that getting this right pays for itself quickly. I have watched teams cut GPU bills in half without reducing throughput. The optimization is there. Go claim it.