Real-Time OLAP at Scale: Apache Druid, Apache Pinot, and When ClickHouse Is Not Enough

There is a category of analytics problem that neither your data warehouse nor ClickHouse handles well, and most teams do not discover this until they are already in production with angry users. The problem is this: you need sub-second query latency on data that is seconds old, at thousands of concurrent queries per second, from an application serving end users or operational dashboards that cannot tolerate spikes. You need your analytics to behave like a database under OLTP-style concurrency, but on OLAP-scale data volumes.

In twenty years building data infrastructure, I have watched teams try to solve this with every tool imaginable. They throw Redis caches in front of Spark queries. They try pre-aggregating everything into PostgreSQL summary tables. They reach for ClickHouse, which helps enormously but starts sagging under concurrency pressure past a few hundred simultaneous queries. Eventually, the teams that get this right land on Apache Druid or Apache Pinot, and they tend to stay there.

These are not household names the way Kafka or Spark are, but they power the analytics infrastructure at some of the largest internet companies in the world. Druid runs the real-time dashboards at Airbnb, Netflix, and Lyft. Pinot was built at LinkedIn and powers its user analytics, and Apache Pinot is now used at Uber, Stripe, and Microsoft. If you are building anything that looks like user-facing analytics, operational dashboards, or interactive exploration of streaming data at meaningful concurrency, you need to understand what these systems do and where they differ from ClickHouse.

The Use Case That Makes Druid and Pinot Necessary

Before getting into architecture, it helps to understand the exact problem these databases solve, because the design decisions only make sense in that context.

Imagine you are building an analytics dashboard for a marketplace platform. Each merchant can log in and see their order volume, revenue by product, conversion funnel metrics, return rates, and customer acquisition costs, all updated in real time as orders come in. The data volume is in the hundreds of billions of events total, with millions of new events per minute. There are 200,000 merchants, each running queries simultaneously during peak hours. The queries are ad hoc: different merchants slice by different dimensions, different time ranges, different filters. And the data cannot be stale: if an order came in 30 seconds ago, it needs to show up in the merchant’s dashboard.

This is the “user-facing analytics” or “embedded analytics” use case, and it is harder than it looks. Your cloud data warehouse (Snowflake, BigQuery, Redshift) can handle the queries but at 2-10 second latency and $5-20 per query in compute costs. That is fine for internal analysts running reports but catastrophic for an interactive user-facing product. ClickHouse handles the query latency well (sub-second on a well-tuned cluster), but it was not designed for thousands of concurrent queries from heterogeneous workloads. At high concurrency, ClickHouse’s memory usage per query starts competing with itself. Pinot and Druid were purpose-built for exactly this concurrency pattern.

The secondary use case is operational analytics: internal dashboards that need real-time visibility into business metrics during incidents or operational decisions. Marketing dashboards. Fraud detection exploration. Capacity planning. These workloads share the same requirement: freshness plus concurrency plus interactivity.

Apache Druid: The Lambda Architecture as a Database

Apache Druid was built at MetaMarkets in 2011 (before that company pivoted and open-sourced the project) with a specific design philosophy: time is the primary dimension of all analytics data, so optimize storage and querying around it.

Druid stores data in immutable segments, each covering a specific time interval. A segment for the hour 2026-08-27T14:00:00 to 2026-08-27T15:00:00 is a standalone unit of storage containing all events in that window, compressed and indexed. Once a segment is finalized (when the time window closes), it never changes. This immutability is central to Druid’s performance: segments can be cached aggressively on historical nodes, and queries against finalized segments never deal with write contention.

For fresh data (events from the last few minutes to hours), Druid uses a different code path. The MiddleManager process (or the newer Peon workers) ingests streaming data from Kafka in real time, building in-memory indexing structures called “realtime segments.” These realtime segments are queryable immediately but are mutable, which is why they live on separate nodes with different performance characteristics. When a realtime segment’s time window closes, Druid publishes it to deep storage (typically S3, GCS, or Azure Blob Storage) as an immutable historical segment, and the historical nodes load it for long-term query serving.

This gives Druid what amounts to a lambda architecture baked into the storage layer itself. You query through Druid’s Broker node, which fans out to both realtime nodes (for fresh data) and historical nodes (for historical data), then merges the results. You do not write this lambda architecture yourself; you just configure Druid.

The internal segment format is where Druid’s query performance comes from. Each segment stores data in three structures: a forward index (column values for each row), an inverted index (which rows contain each value of a dimension, stored as compressed bitmaps), and a timestamp index. For a query like “show me order volume by product category for merchant ID 12345 over the last 7 days,” Druid uses the inverted index to find the rows matching merchant ID 12345 in microseconds, then scans only those rows for the product category and order value columns.

Druid’s query language is SQL, backed by a distributed planner. The Broker node receives a SQL query, parses it, generates a physical plan, and routes sub-queries to the appropriate historical and realtime nodes. Historical nodes execute locally on their segments and return partial results, which the broker merges into the final answer. This fan-out-and-merge architecture is what enables Druid to serve thousands of concurrent queries: each query only touches the segments relevant to its time range, and segments are distributed across nodes based on workload.

The operational model for Druid has historically been complex. A production Druid cluster involves six distinct node types: Coordinator (manages segment assignment to historical nodes), Overlord (manages ingestion tasks), Broker (routes queries), Router (optional load balancer in front of brokers), Historical (serves finalized segments), and MiddleManager (handles realtime ingestion). Apache Druid 31+ has simplified this considerably with the integrated process model, which can combine roles. But you still need ZooKeeper for cluster coordination, a metadata database (typically PostgreSQL or MySQL), and deep storage. Kubernetes deployments using the official Druid operator have made this more tractable, but Druid is not a one-pod deployment.

Apache Pinot: Built for LinkedIn’s Concurrency Problem

Apache Pinot has a similar origin story and similar architecture, but the design priorities are subtly different in ways that matter for real-world deployments.

Pinot was built at LinkedIn specifically to serve user-facing analytics. LinkedIn’s “Who viewed my profile,” “Company analytics,” and “Campaign analytics” products all needed the same thing: sub-second query latency for individual users running against their own slices of massive datasets, at hundreds of thousands of concurrent queries during business hours. This drove design choices that show up throughout Pinot’s architecture.

Pinot’s cluster consists of Controller, Broker, and Server nodes. Servers are either offline servers (serving historical batch segments) or realtime servers (handling live ingestion from Kafka). The Controller manages cluster metadata and orchestrates segment transitions from realtime to offline. The Broker handles query routing and result merging.

The segment structure looks similar to Druid, but Pinot has invested more heavily in its indexing capabilities. Pinot supports a wider variety of index types per column than Druid: inverted indexes (like Druid’s), sorted indexes (which double as both a clustered column structure and an index), range indexes for numeric range queries, text indexes for full-text search using Lucene, JSON indexes for semi-structured data, and most distinctively, the StarTree index.

The StarTree index is Pinot’s answer to the pre-aggregation problem. Rather than materializing a fixed set of aggregations, the StarTree index computes and stores aggregates in a tree structure that covers all possible combinations of a configured set of dimensions. For a table with dimensions (merchant, product, region, channel), a StarTree index pre-computes sums, counts, and other aggregates for all 16 possible dimension combinations (merchant alone, product alone, merchant+product, merchant+region, all four, and so on). At query time, Pinot’s query planner recognizes when a query can be answered from the StarTree tree nodes rather than scanning raw data, often resulting in orders-of-magnitude speedups for GROUP BY queries with filters.

Pinot also has a more mature upsert story than Druid for high-cardinality primary keys. If your streaming data contains updates to existing records (think: order status changes, inventory updates), Pinot’s full upsert mode maintains a primary key mapping across realtime segments and serves the latest version of each record at query time. Druid has added upsert support but Pinot’s implementation handles it more cleanly, especially at scale. This makes Pinot the stronger choice for event streams that are not strictly append-only.

Pinot’s operational model is simpler than Druid’s: you need Controller, Broker, and Server nodes, plus ZooKeeper for coordination and deep storage. No separate Overlord or Coordinator equivalent. The Apache Iceberg integration means Pinot’s offline segments can be sourced directly from an Iceberg table, enabling a clean division where your data lakehouse serves as the historical source and Pinot handles the realtime layer.

Pinot’s query language is a subset of SQL with some extensions for specialized functions (approximate distinct counts, percentiles, funnel queries). Most standard SQL analytics patterns work without modification.

ClickHouse in the Same Landscape

Since most teams arrive at Druid or Pinot after evaluating ClickHouse first, it is worth being precise about where ClickHouse excels and where it struggles.

ClickHouse wins when:

• Query concurrency is moderate: under 200-300 simultaneous queries
• Workload is primarily exploratory (internal analysts, not end users)
• Data arrives in batches or with low-latency Kafka ingestion via the ClickHouse Kafka engine
• The team wants operational simplicity: ClickHouse can run as a single node or a small cluster without the complexity of separate node types
• Query patterns are diverse and hard to optimize upfront

ClickHouse’s MergeTree engine is purpose-built for scan-heavy aggregation queries on append-only data. For a team running a few dozen analysts against hundreds of billions of rows, ClickHouse is hard to beat. The operational overhead is lower, the SQL compatibility is better, and the materialized view system handles continuous aggregation elegantly.

The problems emerge at concurrency scale. ClickHouse’s memory model allocates per-query memory from a shared pool. At high concurrency, queries compete for memory, and the cluster either throttles or experiences memory pressure. ClickHouse does not have the architectural separation of concerns that lets Druid and Pinot serve thousands of simultaneous queries without interference.

The Kafka ingestion story in ClickHouse also introduces replication lag. The ClickHouse Kafka engine pulls from Kafka asynchronously, and the data is not immediately queryable after insertion the way Druid and Pinot’s realtime segments are. For use cases where you need seconds-fresh data (not minutes-fresh), ClickHouse creates awkward workarounds.

This is not a condemnation of ClickHouse. It is the right tool for an enormous range of analytics use cases. But when you are building something user-facing with strict latency requirements and high concurrency, the architectural choices in Druid and Pinot matter.

Connecting Kafka to Druid and Pinot

Both systems have first-class Kafka ingestion, which is how most production deployments pull in real-time data. If you are evaluating Kafka alternatives like Redpanda or AutoMQ as your streaming backbone, both work with Druid and Pinot because they expose the standard Kafka API.

For Druid, you configure a “supervisor” that manages Kafka ingestion tasks. The supervisor monitors consumer group offsets, creates MiddleManager tasks to consume from assigned partitions, and handles task failure and recovery. The ingestion spec defines schema mapping (Kafka message JSON to Druid columns), rollup rules (whether to pre-aggregate at ingestion or store raw events), partition-to-task assignment, and segment granularity.

Druid’s rollup feature is worth calling out specifically. If your raw event stream contains individual clicks or impressions but you only query by hour, Druid can pre-aggregate at ingestion time: instead of storing 10 million individual click events per hour, it stores a segment with one row per (user_segment, product, region, hour) combination with summed metrics. This dramatically reduces storage and query time at the cost of flexibility. For fixed query patterns (common in user-facing analytics), this is a major win.

For Pinot, the ingestion configuration is similar but structured around “table” definitions with separate realtime and offline configurations. The realtime table pulls from Kafka, and the offline table serves batch-loaded historical segments. A “hybrid” table combines both, with the broker routing queries across them transparently. One thing I have seen trip teams up is the “pause-and-resume” behavior in Pinot’s realtime segment completion protocol: when a realtime segment fills up and gets committed to deep storage, there is a brief window where the segment is transitioning. Pinot handles this correctly but it requires understanding the segment lifecycle to debug properly.

For upstream stream processing, if you need joins, enrichment, or complex aggregations before data lands in Druid or Pinot, Apache Flink is the standard choice. Flink handles the stateful processing, Druid or Pinot handle the serving. The combination is significantly more powerful than either alone, though it adds operational complexity.

Production Deployment on Kubernetes

Both systems run well on Kubernetes in 2026. The Druid community maintains a Kubernetes operator, and StarTree (the commercial Pinot vendor) has contributed significantly to Pinot’s Kubernetes deployment story.

For Druid on Kubernetes, I recommend starting with the Helm chart from the Apache Druid project and adjusting for your workload. A minimal production cluster for moderate workloads (100-200 queries per second, 100 billion rows) runs:

• 1-3 Coordinator+Overlord pods (combined mode), 8 GB memory
• 3 Broker pods, 16-32 GB memory (query memory comes from here)
• 3-6 Historical pods, 64-128 GB memory each (segment caching scales with memory)
• 2-4 MiddleManager pods, 32-64 GB memory (depends on ingestion parallelism)

Deep storage on S3 or GCS handles the actual segment data. The metadata database (PostgreSQL in production) stores segment manifests, task status, and supervisor state. ZooKeeper or Druid’s native ZK-free mode (in newer versions) handles cluster coordination.

For Pinot, the topology is similar in resource requirements but simpler in node types:

• 1-3 Controller pods, 8 GB memory
• 3 Broker pods, 16-32 GB memory
• 3-6 Server pods running both offline and realtime modes, 64-128 GB memory each

Resource management for both systems centers on historical/server node memory: these nodes cache hot segments in memory and serve queries from that cache. Undersizing memory means constant segment loading from deep storage, which tanks query latency. I have seen teams underprovision here more than anywhere else, then wonder why their p99 latencies are inconsistent.

Druid and Pinot both integrate naturally with Kubernetes resource management patterns including requests/limits configuration. Set memory limits conservatively on historical and server nodes to prevent OOM kills, but do not over-restrict CPU as query processing is CPU-bound during scans.

The Decision Framework: Which Tool to Pick

After building production systems with all three, here is how I would structure the decision:

Choose ClickHouse if:

• Your peak query concurrency is under 200-300 simultaneous queries
• You have internal analysts rather than end-user products
• Operational simplicity matters more than maximum concurrency
• You need good SQL compatibility and frequent schema evolution
• Your data is primarily append-only with batch or low-volume streaming

Choose Apache Druid if:

• Query concurrency will reach hundreds to thousands of simultaneous queries
• Your data is strictly time-series (events with timestamps)
• You need Druid’s rollup feature for pre-aggregation at ingestion
• You want the most mature operational tooling in this category
• You are comfortable with a more complex cluster topology

Choose Apache Pinot if:

• You need upserts or late-arriving data handled cleanly at high cardinality
• You want the StarTree index for accelerating specific GROUP BY query patterns
• You are building on a lakehouse where the Iceberg integration for offline segment sourcing is valuable
• You prefer a slightly simpler cluster topology than Druid
• You want strong commercial support (StarTree offers an enterprise distribution)

There is also the category of streaming databases like RisingWave and Materialize to consider. These serve a different use case: maintaining materialized views that update incrementally as new data arrives, then serving those views as tables. They are excellent for a fixed set of known aggregations. If your query patterns are predictable and you want the simplest possible architecture, streaming databases are worth evaluating. If you need ad hoc exploration across arbitrary dimension combinations at scale, Druid and Pinot remain the better fit.

Schema Design Considerations

Both Druid and Pinot reward careful schema design in ways that relational databases do not. A few principles I have learned from production deployments:

Dimension cardinality matters enormously. High-cardinality string columns (user IDs, order IDs, session IDs with millions of distinct values) are expensive in inverted index storage and slow for GROUP BY queries. If you need to filter on user ID but do not need to GROUP BY it, consider converting it to a numeric hash. If you do not need to filter on it at all, store it only in the forward index, not the inverted index.

Time granularity is a fundamental schema decision in both systems. Choosing hourly segment granularity means each hour’s data lives in one segment. Day-level granularity means larger segments but fewer total segments. Finer granularity gives better time-range pruning for queries on short windows; coarser granularity reduces metadata overhead. For user-facing analytics dashboards where queries mostly cover the last 24-48 hours, hour-level granularity works well.

Null handling in both systems has gotchas. Druid treats null as the default value for numerics (zero for counts and sums), which can produce surprising aggregation results. Pinot has improved null handling in recent versions but still requires explicit configuration. Test your null behavior early.

Where This Fits in the Broader Data Stack

In a mature data platform, Druid and Pinot sit in a specific layer: the serving layer for fresh, high-concurrency analytics. They consume from your streaming backbone (Kafka or a compatible system), while your data pipeline orchestration tools manage the batch path for loading historical data into offline segments. Your cloud data warehouse sits alongside for ad hoc exploration by analysts who care more about flexibility than latency. Your data lake or Iceberg lakehouse provides the cold storage tier and the authoritative historical record.

The teams that do this well build a clear contract around which layer serves which query type. OLAP systems for user-facing interactive queries, Druid or Pinot for sub-second freshness at high concurrency, and data warehouses for complex analytical SQL at lower concurrency and higher cost. The mistake is trying to use one tool for all three use cases, which is how teams end up with either a data warehouse billing disaster or a ClickHouse cluster buckling under concurrent user traffic.

The tooling ecosystem around both Druid and Pinot has also matured considerably. Apache Superset connects to both as a data source and works well for operational dashboards. Grafana supports both through community plugins. Startree Data Manager simplifies Pinot administration. For the teams I have worked with who made the investment in understanding these systems, the payoff has been dashboards that feel instantaneous to end users, analytics infrastructure that does not become the bottleneck during business-critical periods, and substantially lower query costs than running everything through a cloud data warehouse.

If your analytics data is growing into the hundreds of billions of rows, your users are demanding real-time visibility, and your current solution is either too slow or too expensive under concurrent load, it is time to seriously evaluate Druid and Pinot. They are not simple systems. But neither is the problem they solve.