Kubernetes Native Sidecar Containers: How KEP-753 Finally Fixes Job Completion, Startup Ordering, and Log Flushing for Production Clusters

For twenty years I have been running distributed systems, and few Kubernetes foot-guns have cost as much engineer time as the old sidecar pattern. You attach a log-shipper to a batch Job, the Job finishes, the log-shipper keeps running, and now you have hundreds of pods stuck in Running state forever because nobody told the sidecar that the work was done. I have debugged this exact issue more times than I care to count. It is not a bug in your application. It is a structural limitation of how Kubernetes modeled container lifecycles before Kubernetes Enhancement Proposal 753 fixed it.

KEP-753 introduced native sidecar containers, which reached alpha in Kubernetes 1.28 (August 2023), became beta and enabled by default in 1.29, and graduated to stable (GA) in Kubernetes 1.33 (April 2025). If you are running Kubernetes 1.29 or later and still deploying sidecars as regular containers in spec.containers, you are leaving a meaningful quality-of-life improvement on the table. This article explains what changed, which production workloads benefit most, and how to migrate without breaking your existing deployments.

Why Old-Style Sidecars Were a Hack

Before we get into the fix, it helps to understand exactly what was broken. In Kubernetes, a Pod is a group of containers that share a network namespace and, optionally, volumes. The original design treated every container in spec.containers as a peer: they start in an unspecified order, run concurrently, and the Pod is considered Completed only when all of them exit.

This worked fine for long-running Deployments. It fell apart in three common scenarios.

The Job completion problem. A batch Job Pod runs an ETL script. You attach a Fluent Bit sidecar to ship logs to your observability backend. The ETL script finishes and exits with code 0. Fluent Bit keeps running. Kubernetes sees that not all containers in the Pod have exited, so the Job never reaches Succeeded. Your CI pipeline hangs. Your on-call engineer wakes up at 2 AM to kubectl delete pod dozens of zombie pods.

The startup ordering problem. Your application needs the Vault agent sidecar to write secrets to a shared volume before it starts. The old workaround was using init containers for the initial secret fetch, plus a separate sidecar for rotation. But init containers stop before app containers start, so you end up with a fragile two-container scheme that cannot handle mid-life secret rotation without application-level retry logic.

The graceful shutdown problem. Your application receives SIGTERM and begins draining connections. But your service mesh proxy, also receiving SIGTERM simultaneously, may close the sidecar before the application finishes processing in-flight requests. This is why Istio users spent years manually configuring terminationDrainDuration and adding preStop sleep hooks. These workarounds worked but were brittle and required careful synchronization across every Deployment in the cluster.

What KEP-753 Actually Introduced

The implementation is simpler than you might expect. A native sidecar container is an init container with restartPolicy: Always. That single field change tells the kubelet to treat this init container differently: start it before regular containers, let it run for the full Pod lifetime, restart it independently if it crashes, and send it SIGTERM only after all regular containers have exited.

Here is what a native sidecar logging configuration looks like:

spec:
  initContainers:
  - name: fluent-bit
    image: fluent/fluent-bit:3.1
    restartPolicy: Always
    resources:
      requests:
        memory: "64Mi"
        cpu: "50m"
      limits:
        memory: "128Mi"
        cpu: "200m"
    volumeMounts:
    - name: varlog
      mountPath: /var/log
    startupProbe:
      httpGet:
        path: /api/v1/health
        port: 2020
      failureThreshold: 30
      periodSeconds: 2
  containers:
  - name: app
    image: myapp:latest

The critical behavior change: the kubelet starts fluent-bit, waits for its startupProbe to pass (or for it to enter Running state if no probe is defined), and only then starts app. When app exits, the kubelet sends SIGTERM to fluent-bit. Fluent Bit flushes its buffers and exits. The Pod completes cleanly.

One subtlety worth understanding: if you have multiple native sidecars, they start in the order they appear in initContainers, just like regular init containers. So you can guarantee that your secret-fetching sidecar runs before your proxy sidecar, which runs before your app. This deterministic ordering is something the old-style sidecar pattern simply could not provide.

The Three Production Scenarios That Change

Logging and Observability

This is the biggest win for most teams. Log-shippers like Fluent Bit, the OpenTelemetry Collector, and Logstash have been the canonical sidecar use case since the early Kubernetes days. With native sidecars, you get three improvements:

First, the startup guarantee means your log-shipper is running before your app writes its first line. With old-style sidecars, there was a race: if your app started logging during the brief window before the log-shipper was ready, those early-boot messages were lost. This mattered more than most teams realized, because startup errors, which are often the most important ones, frequently happened in exactly that window.

Second, the Job completion fix eliminates the zombie-pod problem entirely. Fluent Bit 3.0 and later explicitly support the native sidecar lifecycle model. When the main container exits, Kubernetes signals Fluent Bit, which flushes its in-memory buffers and exits cleanly.

Third, native sidecars support livenessProbe and readinessProbe, just like regular containers. If your log-shipper crashes and stays down, the liveness probe restarts it independently without disrupting the application container. With old-style sidecars in spec.containers, a crashed log-shipper either restarted the whole Pod (if restartPolicy: Always) or stayed dead and silently dropped logs.

For distributed tracing with OpenTelemetry, the same pattern applies: run the OTel Collector as a native sidecar to guarantee it is available before your app starts emitting spans, and guarantee it flushes its export queue before the Pod terminates.

Secrets Management

The Vault agent injector has been a popular pattern for delivering secrets to Pods without hardcoding credentials. The old approach used an init container for the initial fetch and a separate sidecar for rotation, connected through a shared emptyDir volume.

With native sidecars, you can collapse this into a single container that handles both the initial fetch and ongoing rotation, because native sidecars start before the app and run for the Pod’s full lifetime. You declare the Vault agent sidecar as a native init container, give it a startupProbe that passes only after the secrets file has been written to the shared volume, and your app container will never start until its credentials are in place.

spec:
  initContainers:
  - name: vault-agent
    image: hashicorp/vault:1.16
    restartPolicy: Always
    args:
    - agent
    - -config=/vault/config/agent.hcl
    startupProbe:
      exec:
        command: ["test", "-f", "/vault/secrets/db-creds.json"]
      failureThreshold: 30
      periodSeconds: 2
    volumeMounts:
    - name: vault-secrets
      mountPath: /vault/secrets
    - name: vault-config
      mountPath: /vault/config
  containers:
  - name: app
    image: myapp:latest
    volumeMounts:
    - name: vault-secrets
      mountPath: /app/secrets
      readOnly: true

This is significantly cleaner than the two-stage approach and eliminates the fragile ordering between an init container (that exits) and a sidecar (that starts after). For teams using External Secrets Operator or Sealed Secrets, native sidecars are less directly applicable, since ESO syncs secrets into Kubernetes Secret objects rather than running in-pod agents. But for teams running the Vault agent injector, the native sidecar model is the right migration target.

Service Meshes

This is where things get interesting. The sidecarless service mesh trend, covered extensively in Istio Ambient Mesh and Cilium, removes the data-plane proxy from the Pod entirely. But for teams that are not ready to migrate to ambient mode, native sidecars offer meaningful improvements to the traditional sidecar model.

Istio 1.27 (released mid-2025) enabled native sidecar injection by default for eligible pods: the istio-proxy is now injected as an init container with restartPolicy: Always rather than a regular container. If you are running Istio 1.24, 1.25, or 1.26 and want to opt in early, set the ENABLE_NATIVE_SIDECARS Helm value in your Istio compatibility profile. With native injection, the Envoy proxy starts before your application container, which solves the long-standing problem of application traffic failing during startup because the proxy was not yet ready. It also solves the graceful shutdown race: when your application finishes draining, Kubernetes terminates Envoy after the app exits, not simultaneously with it.

For teams choosing between traditional sidecars and Istio Ambient Mesh, native sidecars are a reasonable middle ground: you keep the existing per-pod Envoy model (with its per-pod mTLS enforcement and detailed per-service observability) while eliminating the lifecycle ordering bugs. If you are operating a large fleet of services with existing Envoy-based traffic policies, migrating to native sidecar injection is a much smaller change than migrating to ambient mode.

The service mesh comparison remains relevant here: Linkerd’s lightweight Rust proxy also supports the native sidecar injection model as of mid-2026.

Resource Management and Probes

With old-style sidecars declared in spec.containers, resource accounting was straightforward: sum all containers’ requests and limits. Native sidecars in initContainers follow a slightly different accounting model. For init containers without restartPolicy: Always, Kubernetes uses the maximum resource request across all init containers (since they run sequentially). But native sidecars run concurrently with the main containers, so their resources are added to the sum, not the maximum.

This means adding a native sidecar increases your Pod’s effective resource request. If you migrate a DaemonSet log-shipper to a native sidecar pattern, every Node needs enough capacity to run both the sidecar and the main workload simultaneously. This was true of old-style sidecars too, but the counting path through the scheduler API is different, and some capacity planning tools were not updated to handle it correctly until Kubernetes 1.31 or so.

For CPU and memory limits, the same best practices apply as for any container: set requests conservatively (what you normally need) and limits tightly (what you must never exceed). OOM kills in sidecars behave the same way they do in regular containers: the kubelet restarts the crashed sidecar independently.

Probes deserve attention. You can and should configure startupProbe on your native sidecars to give the kubelet a reliable signal that the sidecar is truly ready before the main container starts. Without a startupProbe, the kubelet advances to the next container as soon as the sidecar transitions to Running, which is the point when the process started, not the point when it is ready to serve. For a log-shipper this is usually fine; for a secret-writing agent it is not.

Migration Playbook

Migrating existing workloads to native sidecars requires care, especially if you are using a mutating admission webhook (like Istio’s or Vault’s) that injects sidecars automatically.

For manually managed sidecars: Move the sidecar’s spec from spec.containers to spec.initContainers and add restartPolicy: Always. Add a startupProbe appropriate to the sidecar. Test in a non-production namespace. Watch for any tooling (Helm charts, Kustomize overlays, operator configs) that constructs pod specs dynamically, because those will need updates.

For injected sidecars: Check whether the injecting tool supports native sidecar injection. Istio has used native injection by default since 1.27, and earlier releases can opt in via ENABLE_NATIVE_SIDECARS. For the HashiCorp Vault agent injector, native sidecar injection from the injector webhook was still an open feature request as of early 2024; teams that need native sidecar semantics for Vault agent today typically manage the pod spec directly (as in the example above) rather than relying on the injector to add restartPolicy: Always automatically. If your injector does not support native sidecars, you will need to manage the pod spec manually until the tooling catches up.

For existing Jobs with sidecar problems: This is the most impactful migration. If you have batch jobs that hang because of logging or proxy sidecars, a migration to native sidecars fixes the problem cleanly. The tricky part is that a running Job’s Pod spec cannot be updated in place. You need to let current job Pods complete (or force-delete them) and then deploy the updated PodTemplateSpec.

One gotcha: if you are running Kubernetes Operators that synthesize Pod specs programmatically, you need to ensure they add restartPolicy: Always to the correct containers. A few poorly maintained operators have hard-coded assumptions that all init containers exit before the main app starts, and adding a native sidecar would break their pod-state logic.

Operational Gotchas

Termination grace period. When all main containers in a Pod exit, Kubernetes sends SIGTERM to native sidecars and starts the termination grace period countdown. If your sidecar takes longer to flush and exit than terminationGracePeriodSeconds allows, Kubernetes will SIGKILL it, potentially causing log loss. Monitor your sidecars’ shutdown time and set terminationGracePeriodSeconds accordingly. For Fluent Bit this is usually under 10 seconds; for OTel Collectors with slow export backends, it can be longer.

DaemonSet behavior. When a DaemonSet pod is evicted or the node is drained, Kubernetes terminates native sidecars after main containers, same as everywhere else. No special handling needed. But if you are running a per-node log-shipper as a DaemonSet with a native sidecar for some auxiliary purpose, the lifecycle semantics are exactly what you would expect.

Debug containers. If you use kubectl debug to attach an ephemeral container to a Pod, the ephemeral container is not a native sidecar and cannot be given restartPolicy: Always. This is expected behavior; ephemeral containers have their own lifecycle contract.

PodDisruptionBudgets. PDBs protect against simultaneous evictions but do not understand the distinction between native sidecars and main containers. A pod is either available or not. This has not changed from before native sidecars.

Istio ambient and native sidecars together. If you are on a cluster where some namespaces use Istio ambient mode and others use traditional sidecar injection (a common migration configuration), native sidecar injection is orthogonal to ambient mode. Pods in non-ambient namespaces get native sidecar Envoy proxies. Pods enrolled in ambient mode get no in-pod proxies at all. The cluster handles both simultaneously without conflict.

When Not to Use Native Sidecars

Native sidecars are not always the right tool. If you need a container to run once and exit before the main app starts (for example, a database migration runner or a configuration renderer), you want a classic init container without restartPolicy: Always. The point of native sidecars is long-running auxiliary processes; one-shot setup tasks should remain classic init containers.

If you are running a high-density workload where adding per-pod resource overhead is prohibitive, consider whether a node-level DaemonSet log-shipper is a better fit than a per-pod sidecar. Per-pod sidecars consume resources on every Pod; a node-level DaemonSet amortizes the cost across all pods on the node. This trade-off existed before native sidecars and remains unchanged.

For service mesh functionality specifically, if you are starting fresh and your team does not have a strong reason to stay in per-pod proxy mode, Istio Ambient Mesh or Cilium’s sidecarless mode avoids the sidecar complexity entirely. Native sidecars improve the traditional model significantly, but the fundamental complexity of a per-pod proxy (resource overhead, debug complexity, certificate management) remains. For eBPF-based networking, the sidecarless approach is increasingly where new deployments land.

The Practical Verdict

After running Kubernetes clusters in production for most of my career, I think native sidecar containers are one of the most important quality-of-life improvements the project has shipped in recent years. The change is small from a spec perspective (one field: restartPolicy: Always) but the operational impact is large.

The Job completion fix alone is worth the migration for any team running batch workloads with logging sidecars. The startup ordering guarantee is worth it for any team injecting secrets or proxies. The graceful shutdown fix is worth it for any team that has ever added a preStop: sleep 5 hack to work around simultaneous SIGTERM delivery.

Start with your most problematic workloads: batch Jobs with log-shippers and Deployments where startup ordering between the proxy and the application has caused production incidents. Move those first. Then roll out native sidecar injection for Istio or Vault across your fleet during the next maintenance window. The rollout is low-risk: old Pods keep running until you do a rolling restart, and new Pods get the native sidecar lifecycle from that point forward.

KEP-753 is one of those infrastructure changes that sounds boring when you read the KEP title and becomes obviously important the moment you hit one of the problems it solves.