The instrumentation tax has always been one of the quiet, persistent costs of running microservices. You want distributed traces. Your observability team wants distributed traces. The on-call engineer at 2 AM desperately wants distributed traces. But actually getting them means touching every service: adding the OpenTelemetry SDK, configuring exporters, managing context propagation, bumping package versions when a CVE drops, arguing with the Python team about whether auto-instrumentation breaks their framework’s internals. In a monorepo with twenty services written by fifteen engineers, this is a multi-quarter project that inevitably deprioritizes itself.
I have been through that cycle more times than I care to count over twenty years in this industry. We would get 60% coverage and call it good enough, knowing that the 40% dark zones were exactly where the interesting failures happened.
OpenTelemetry eBPF Instrumentation, which most people still call by its original name Beyla, changes the economics of that problem. It hooks into the Linux kernel via eBPF and captures distributed trace data from your applications without a single line of code change, a sidecar, or a restart. Splunk launched it in beta at KubeCon EU 2026, and the CNCF Observability TAG reported that 67% of production Kubernetes clusters were already running at least one eBPF-based observability tool by Q1 2026. That adoption number should tell you something about how significant this shift is.
This article is about how OBI works technically, how it compares to SDK-based and sidecar-based instrumentation, what the real trade-offs are, and how to deploy it in a production Kubernetes cluster. I am not going to pretend it solves everything, because it does not. But for a large class of services, it is the right call.
The Instrumentation Problem
Before getting into eBPF, it is worth being precise about what problem we are actually solving.
Traditional distributed tracing has three approaches, each with a different cost profile.
The first is SDK instrumentation: you add the OpenTelemetry SDK to each service, manually instrument critical paths, configure an exporter pointed at your collector, and manage context propagation via HTTP headers or gRPC metadata. This is the most complete and accurate approach. It can capture business-level spans, add semantic attributes, and trace through message queue consumers where network sniffing cannot reach. The downside is exactly what you expect: it requires touching every service, it adds latency to the hot path (usually under a millisecond, but measurable), and it creates a maintenance burden. When you upgrade OTel SDK, you do it in thirty repositories.
The second is sidecar injection: a service mesh like Istio intercepts all traffic via a sidecar proxy and generates L7 spans. You get coverage with minimal per-service work, but you pay in memory and CPU for every pod running a proxy. In our service mesh article, I covered the resource overhead this introduces, and it is not trivial on clusters with hundreds of pods.
The third is eBPF-based instrumentation, which is what OBI does. A single DaemonSet runs on each node and attaches eBPF probes to kernel functions. It intercepts system calls, TLS handshakes at the uprobe level, and HTTP/gRPC traffic without modifying the application process at all.
One DaemonSet per node instead of a sidecar per pod. No SDK changes. No restarts. You get coverage in minutes.
How eBPF Instrumentation Actually Works
If you have read our eBPF explainer, you know the basics: eBPF programs run in a sandboxed virtual machine inside the Linux kernel, attach to kernel hooks via probes, and the verifier ensures they cannot crash the system. The overhead is measured in microseconds.
OBI uses three types of probes to capture HTTP and gRPC traffic.
Kprobes attach to kernel functions and can observe system calls like read(), write(), send(), and recv(). When your Go HTTP server calls write() to send a response, OBI’s kprobe fires and captures the file descriptor, the data being written, and the process ID. This is how it correlates traffic to specific pods.
Uprobes attach to user-space function entry and exit points in compiled binaries. For Go, OBI attaches to net/http.(*Transport).roundTrip and net/http.(*Server).ServeHTTP. For Python, it attaches to urllib3. This is how OBI can extract trace context even from TLS-encrypted traffic: it hooks into the HTTP layer before encryption happens in the application process. No need to intercept TLS at the kernel level.
Socket filters observe raw network packets for unencrypted HTTP/1.1 traffic and gRPC over plaintext. This covers legacy services and internal east-west traffic that skips TLS.
The instrumentation agent reads the HTTP method, URL, status code, and headers from these probes. When it sees a traceparent header in an incoming request, it extracts the trace ID and span ID for correlation. When there is no existing trace context, it generates a new trace ID. All of this happens in the kernel, with the data shipped via a ring buffer to a user-space component that formats it into OTLP spans and sends it to your OpenTelemetry Collector.
The result is distributed traces with accurate timing, correct parent-child span relationships, and RED metrics (Rate, Errors, Duration) for every service on the node.
Language and Protocol Support
This is where OBI’s uprobe strategy becomes important to understand, because the support matrix varies by language.
Go gets the best coverage. OBI attaches uprobes to the compiled binary, which means it works regardless of which framework you use, whether that is chi, gin, echo, or raw net/http. Go compiles to native binaries with predictable symbol names, which makes uprobe attachment reliable.
Java, Python, Node.js, .NET, and Ruby are supported via a combination of uprobes for common runtimes and socket filters for HTTP. The uprobe attachment is less precise for interpreted languages because the JVM and CPython have their own internal goroutine and thread models that require additional adaptation layers.
Rust and C/C++ work via socket filters and kprobes, similar to any native application.
For protocols, OBI handles HTTP/1.1, HTTP/2 (including gRPC), and DNS. Kafka, AMQP, and other message queue protocols are explicitly not in scope for the current release, which is the correct architectural decision: you genuinely cannot reconstruct a Kafka consumer’s trace context from packet-level observation alone, because the correlation lives in message headers that require application-level understanding.
This is a critical limitation to understand before you get rid of your SDK instrumentation entirely. If you have services that produce or consume from Kafka, SQS, or any async messaging system, OBI will give you excellent traces for the synchronous HTTP paths but the async paths remain dark. You need SDK instrumentation for those.
Deployment on Kubernetes
OBI ships as a DaemonSet. You install it cluster-wide and it instruments everything, or you can use namespace and pod label selectors to target specific workloads. The configuration is a Kubernetes resource, and GitOps workflows pick it up naturally.
The basic installation via Helm is straightforward:
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install obi open-telemetry/opentelemetry-ebpf-instrumentation \
--namespace obi --create-namespace \
--set config.otelExporterOtlpEndpoint=http://otel-collector.observability:4317
The DaemonSet needs specific kernel capabilities: CAP_BPF, CAP_PERFMON, and CAP_NET_ADMIN. It also needs to run in privileged mode or with the appropriate seccomp profile to load eBPF programs. This is the cost of kernel-level access, and it is worth an honest conversation with your security team before you deploy to production. The good news is that the eBPF verifier prevents the agent from doing anything outside its declared scope. The kernel will reject any program that could crash the system or access arbitrary memory.
For the node-level visibility OBI requires, the kernel minimum is Linux 5.10. Most cloud-managed Kubernetes offerings run 5.15+ by default in 2026, so this is rarely a constraint anymore.
The DaemonSet approach has a meaningful resource advantage over sidecars. In a cluster with 100 pods across 10 nodes, you run 10 OBI instances instead of 100 sidecars. Each OBI instance uses roughly 200MB of memory and 50m CPU in our testing under moderate load. A comparable Envoy sidecar uses 50-100MB per pod, which adds up to 5-10GB for the same cluster. The math on large clusters is compelling.
Integration with Your Existing OTel Stack
OBI exports standard OTLP, which means it drops into any existing OpenTelemetry infrastructure without changes. The spans it generates carry the W3C trace context format, so if some of your services have SDK instrumentation and some are covered by OBI only, the traces compose correctly. A request that starts in a manually instrumented gateway service, passes through an OBI-only backend, and writes to a database where you have SDK instrumentation will produce a single coherent trace with no gaps.
This composability is what makes the hybrid approach practical. You do not have to choose between OBI and SDK instrumentation. The right model for most organizations is to use OBI for broad coverage across all services, and layer SDK instrumentation for specific high-value services where you want business attributes, queue-based traces, and deeper internal spans.
When you pair OBI with the OpenTelemetry Collector, you can use the collector’s processing pipeline to add resource attributes, filter out noisy health check spans, and route to multiple backends. The OBI agent is intentionally minimal: it captures and exports. All the enrichment and routing logic lives in the collector, where it belongs.
On the metrics side, OBI emits RED metrics as OTLP metrics, which flow naturally into Prometheus and Grafana. Service maps in Grafana’s Application Observability view are automatically populated from OBI’s span data. In my experience, this is the feature that converts skeptical engineering managers: you go from no service map to a complete topology diagram of your entire platform in about an hour, with no code changes.
The Real-World Case for Zero-Code Instrumentation
In 2023, I was brought in to help a financial services company whose incident response was broken. They had thirty-some microservices, complete Prometheus metrics coverage, and almost no distributed tracing because the SDK rollout had stalled at about eight services. When something went wrong, the on-call engineer could see which service had elevated error rates but could not follow a request through the system to find the root cause. Mean time to diagnosis was genuinely painful.
We did not have a tool like OBI at the time. We spent two quarters getting SDK coverage to 70% of the services, which was enough to make incident response tolerable. What I keep thinking about when I look at OBI is that we could have had that coverage in a week.
The failure mode I see most often is instrumentation debt. Teams know they need tracing, they add it to the backlog, it keeps getting deprioritized because it does not ship features, and then the service that has never had instrumentation is exactly the service that fails at the worst possible moment. OBI eliminates that debt at the point of cluster deployment rather than service deployment.
The second use case is polyglot environments. If you have a cluster running Go, Python, Java, and Ruby services, you need four different SDK packages, four different auto-instrumentation libraries, and four different upgrade cycles. With OBI, the instrumentation layer is outside the application entirely. Language is irrelevant.
The third case is third-party or vendor-supplied workloads. You cannot add an SDK to a database operator or a service mesh control plane, but OBI can instrument the HTTP traffic in and out of those components. For the first time, you get visibility into the observability of your observability stack.
What OBI Does Not Give You
I want to be specific here because I have seen people read the marketing materials and conclude that SDK instrumentation is obsolete. It is not.
OBI cannot see inside an application process. If you have a Go service that processes a request, does ten internal method calls, makes three database queries, and then responds, OBI sees one span: the inbound HTTP request. The internal structure is invisible. SDK instrumentation with proper manual span creation is the only way to get that granularity.
OBI cannot capture custom attributes. If you want to attach a user_id, tenant_id, or transaction_type to a span for business-level analysis, you need SDK instrumentation. These are the attributes that turn observability from a debugging tool into a business intelligence tool.
OBI cannot trace through message queues, as I mentioned earlier. The Kafka consumer problem is fundamental: there is no network event for OBI to observe when a consumer processes a message. The correlation has to happen at the application level.
Finally, OBI’s Go uprobe support depends on specific function signatures in the standard library and popular frameworks. If you use an unusual HTTP framework or compile with unusual flags, the probes may not attach correctly. The fallback is socket-level observation, which misses TLS context.
None of these limitations mean OBI is the wrong choice. They mean it is an excellent complement to SDK instrumentation, not a replacement for it in all cases.
Practical Deployment Recommendations
After watching several teams adopt OBI over the past months, here is the approach that works best.
Start with OBI cluster-wide. Get the DaemonSet running and the traces flowing into your collector. Within a day you will have a service map and RED metrics for every service. Use this baseline to identify the highest-traffic services and the ones with the most complex internal logic.
Then add SDK instrumentation to those high-value services selectively. If your checkout service handles the most revenue and has the most internal complexity, it deserves the full SDK treatment with custom spans and business attributes. The remaining services can stay on OBI-only coverage indefinitely.
For Kubernetes deployment, integrate OBI into your cluster bootstrap rather than treating it as an afterthought. If you are using Cluster API or a similar provisioning framework, the OBI DaemonSet should be part of your default addon set alongside CoreDNS and your CNI. Any new cluster gets observability from day one.
On the security side, review the required capabilities carefully with your security team. In environments with strict Pod Security Standards, you may need a specific namespace-level exception for the OBI DaemonSet. The capabilities it needs (CAP_BPF, CAP_PERFMON) are well-scoped but not in the restricted baseline. This is a reasonable exception for an observability component that runs at node level, but document it in your compliance records. Our Kubernetes security hardening guide covers how to handle these exceptions systematically.
For cost control, the OBI DaemonSet will generate substantially more spans than you were collecting before, because it is instrumenting services that previously had no instrumentation at all. Set up sampling at the collector level before you enable OBI in production. A head-based sampler at 10% for high-traffic services, combined with tail-based sampling to always capture errors and slow requests, is a reasonable starting configuration. The observability pipeline patterns I covered previously apply directly here.
Comparing OBI to Grafana Beyla
You will see both names in the wild and it helps to understand the relationship. Beyla was Grafana Labs’ eBPF instrumentation agent, developed internally and open-sourced in 2023. In early 2025, Grafana Labs donated the codebase to the OpenTelemetry project, where it became the foundation for the OpenTelemetry eBPF Instrumentation (OBI) SIG.
The projects are closely related but not identical. Beyla continues to exist as a Grafana Labs project with features optimized for the Grafana ecosystem (native integration with Grafana Cloud, Mimir metrics, Tempo traces). OBI is the vendor-neutral version co-developed by Grafana Labs, Splunk, Coralogix, Odigos, and others, designed to work with any OTLP-compatible backend.
If you are committed to the Grafana observability stack, Beyla may offer tighter integration. If you want a vendor-neutral foundation that you can route to any backend, OBI is the right choice. For most organizations, OBI is the correct default. You can always add a Grafana backend without changing the instrumentation layer.
Watching a Maturing eBPF Ecosystem
OBI is part of a broader trend toward moving observability and security logic into the kernel via eBPF. The same approach powers Cilium’s networking and security, Tetragon’s runtime security policies, and Parca’s continuous profiling. The kernel is becoming the platform for cloud-native infrastructure tooling, and eBPF is the mechanism.
What this means for your architecture is that eBPF expertise is increasingly valuable. Understanding how probes attach, how ring buffers work, what the verifier allows, and how to read kernel flamegraphs is moving from kernel developer knowledge to infrastructure engineer knowledge. I spent two years trying to understand eBPF well enough to evaluate tools that use it before I felt like I actually understood what I was accepting when I deployed one. If you are planning to rely heavily on this stack, invest in that understanding now.
The other trend worth watching is the convergence of observability and security on the same eBPF infrastructure. Tetragon can send security events alongside OBI’s trace data through the same collector pipeline. You start to get a unified signal where a suspicious file access event correlates directly with the distributed trace that triggered it. That kind of cross-signal analysis is genuinely new and the tooling to exploit it is still early.
The Bottom Line
If you are running a Kubernetes cluster with more than a handful of services and your distributed tracing coverage is incomplete, OBI is the most practical path to comprehensive visibility in 2026. The deployment is a DaemonSet and a Helm chart. The integration with existing OpenTelemetry infrastructure is seamless. The overhead is a fraction of what sidecars cost.
The limitations are real: no visibility inside application processes, no custom business attributes, no async queue tracing. Build your instrumentation strategy around those limitations: OBI for broad HTTP coverage everywhere, SDK instrumentation for the services where internal visibility and business context matter most.
The era of “we will get to instrumentation eventually” is over. There is no longer any excuse for running uninstrumented services in production when a single DaemonSet will cover the entire cluster. And if you are a principal engineer trying to make the case internally, the argument is straightforward: the total engineering cost of OBI adoption is an afternoon, compared to a quarter for a full SDK rollout. The coverage you get on day one is comparable. You can spend the rest of the quarter adding depth where it matters rather than breadth everywhere.
That is a trade-off I would make every time.
Get Cloud Architecture Insights
Practical deep dives on infrastructure, security, and scaling. No spam, no fluff.
By subscribing, you agree to receive emails. Unsubscribe anytime.