Streaming Databases in Production: RisingWave, Materialize, ksqlDB, and Why Your Flink Pipeline Might Be Overkill

Three years ago I was staring at a Flink job that had started its life as a twenty-line SQL statement. By the time I found it, it was a four-thousand-line Java project. The original author was gone. The checkpoint configuration was tuned for a cluster that no longer existed, and the RocksDB state backend was periodically running the JVM out of heap in ways that violated everything I thought I knew about garbage collection. The pipeline produced a single real-time aggregation that a product dashboard needed every five seconds.

That experience is what made me pay close attention when streaming databases started appearing. The premise, to let you write CREATE MATERIALIZED VIEW in SQL and have the system figure out the incremental computation, sounded almost too clean. After twenty years of watching every “SQL over streaming data” promise collapse under real workloads, I was skeptical. But the category has matured enough now that I have strong opinions about when these systems belong in your architecture, and when they do not.

What a Streaming Database Actually Does

A streaming database is not a traditional database with a Kafka connector bolted on. The defining characteristic is that queries do not run on demand against historical data. Instead, you define a query once, and the system continuously maintains the result as new data arrives. When you read from a materialized view, you are reading pre-computed results, not triggering a scan.

The closest analogy from the traditional database world is an indexed view, but applied continuously to an append-mostly event stream. When a Kafka topic receives a new event, the streaming database identifies which materialized views reference that topic, computes the delta, and applies it to the stored result. A COUNT(*) grouped by user ID does not recount everything; it increments the count for the specific user ID that just appeared.

This incremental view maintenance model is what separates streaming databases from tools like ClickHouse or Druid. Those are excellent for fast analytics against stored data, but they do not continuously update query results as events land. They execute queries. Streaming databases maintain answers.

It also separates them from Flink and Kafka Streams. Those frameworks give you a general-purpose computation engine: powerful, flexible, and operationally complex. A streaming database is a more constrained system that gives you SQL semantics and hides the execution model, at the cost of some flexibility.

The Three Main Players

The streaming database landscape in 2026 has effectively consolidated around three systems. Each makes different tradeoffs worth understanding before you commit.

RisingWave

RisingWave is the option I recommend first for most teams. It is written in Rust, it speaks the PostgreSQL wire protocol, and it is open source under the Apache 2.0 license. The Apache 2.0 licensing matters because it means you can run it in production without worrying about future commercial restrictions.

The architectural decision that sets RisingWave apart from Flink-style systems is compute-storage separation. State lives in S3-compatible object storage, not in JVM heap or local RocksDB instances. This eliminates the most common production failure mode I see with Flink: running out of local state capacity and cascading into checkpoint failures. When a RisingWave compute node dies, it reads state back from S3 and resumes. You can also scale compute nodes independently from storage, which matters when your materialized views grow large.

RisingWave connects natively to Kafka, Kinesis, and Pulsar as source connectors. It also has first-class CDC support, which I consider a requirement for production use. The ability to ingest PostgreSQL and MySQL CDC streams without a separate Debezium deployment simplifies the change data capture architecture considerably. Sinks include Kafka, PostgreSQL-compatible targets, Apache Iceberg, and Delta Lake.

The PostgreSQL wire protocol compatibility is not just a convenience feature. It means your existing BI tools, Grafana dashboards, and application code can query RisingWave materialized views with no driver changes. I have seen teams set up real-time operational dashboards by pointing their existing Postgres query layer at RisingWave views.

The limitation is that RisingWave is designed for eventually consistent streaming analytics. It is not suitable for workloads that require strict transactional guarantees or low-latency point-lookups on individual rows.

Materialize

Materialize is the system to reach for when consistency is non-negotiable. It uses a technique called differential dataflow internally, which gives it a mathematical foundation for maintaining incrementally updated views with strict-serializable semantics. Every query you run against a Materialize view reflects a consistent point-in-time snapshot across all sources.

That guarantee matters for use cases like fraud detection, financial reconciliation, and operational systems where two queries running a millisecond apart must never see an inconsistent state. RisingWave will eventually deliver the right answer, but Materialize will give you the right answer with provable consistency boundaries.

The tradeoff is that Materialize is a managed SaaS service only. There is no self-hosted option. Pricing starts around one dollar per cluster-hour for the smallest configuration, which adds up quickly if you have multiple environments or want to run it for development and staging. For workloads that genuinely need its consistency properties, the cost is justified. For analytics pipelines where eventual consistency is fine, it is probably not.

Materialize ingests from Kafka topics and PostgreSQL (via logical replication). Its source support is narrower than RisingWave, which reflects its focus on operational rather than analytics use cases.

ksqlDB

ksqlDB is Confluent’s SQL layer over Apache Kafka, and it occupies a different position than the other two. Rather than being a standalone streaming database, it is deeply integrated with the Kafka ecosystem. Tables in ksqlDB are backed by Kafka topics and Kafka Streams state stores. You can think of it as making Kafka look like a relational database from the outside.

The benefit is tight coupling with Kafka operations. If you are already running a full Confluent Platform deployment and your team knows Kafka well, ksqlDB requires less additional infrastructure to operate. Queries translate directly into Kafka Streams topologies, so the execution model is familiar to anyone who has worked with Kafka Streams.

The drawbacks are significant in practice. ksqlDB is tied to Confluent’s ecosystem in ways that make it awkward to use with other Kafka distributions like Redpanda or AutoMQ. The SQL dialect is non-standard in ways that matter, and complex queries involving multiple stream-stream joins are difficult to express correctly. The stateful query semantics are also subtly different from what SQL developers expect, which produces bugs that are hard to diagnose.

I use ksqlDB primarily when a team is already on Confluent Platform, the queries are straightforward aggregations or filters, and the operational overhead of standing up a separate system outweighs the SQL ergonomics of the alternatives. For new greenfield streaming pipelines, I start with RisingWave.

When to Use a Streaming Database Instead of Flink

This is the question I get most often, and the answer is less about feature sets and more about what kind of team is operating the system.

Apache Flink is a general-purpose distributed stream processing framework. It can express computations that no SQL system can represent cleanly, including custom windowing strategies, stateful event-time processing with late arrival handling, and complex joins across streams with heterogeneous schemas. If you are building a fraud detection model that needs to compute graph-based features across a sliding window of transaction events with custom serialization, Flink is probably what you need.

The operational cost of Flink in production is real. You need to understand checkpoint configurations, state backend sizing (RocksDB memory vs. file system), savepoint management for job upgrades, backpressure dynamics, and watermark strategies. A well-run Flink deployment requires engineers who have internalized these concepts. A poorly-run Flink deployment, which is what I see more often, produces the kind of situation I described at the opening.

Streaming databases earn their place when the computation you need to express fits in SQL. Aggregations grouped by dimensions, joins between streams and slowly-changing dimension tables, simple window functions, alert conditions computed from running averages: these are all things that streaming databases handle better than Flink because there is no code to maintain, no JVM to tune, and no job topology to reason about when something goes wrong.

The decision framework I use: if a data engineer who knows SQL but not distributed systems could write the logic, use a streaming database. If the logic requires custom operators, complex event processing patterns, or sub-millisecond latency requirements with precise watermark control, use Flink.

For most data engineering teams I work with, the streaming database path covers sixty to seventy percent of what they were using Flink for, and eliminates the majority of their on-call burden for those pipelines.

Production Architecture Patterns

Let me walk through the architecture patterns I actually see in production.

The Real-Time Operational Dashboard

The most common use case is a dashboard that needs to show aggregated metrics updated within seconds. A user wants to see orders per minute by region, broken down by product category, refreshed continuously.

The traditional implementation: Kafka consumer group writing events to ClickHouse, with a query running every five seconds. Works fine until you need consistency across dimensions or the query gets complex.

The streaming database implementation: a CREATE MATERIALIZED VIEW in RisingWave that joins the orders stream against a product catalog table (itself populated from a CDC source), groups by region and category, and updates incrementally. The dashboard queries the view directly over the PostgreSQL protocol. The complexity of data pipeline orchestration is replaced by a single SQL definition.

The view definition stays simple. The operational burden stays low. The latency is sub-second.

CDC-Driven Materialized Views

This pattern is where streaming databases shine most clearly. You have a PostgreSQL operational database, you want to maintain a denormalized read model for fast queries, and you want that read model to update within seconds of writes to the source.

With Flink, this typically means Debezium, a Kafka topic, Flink SQL or a Java job, and a sink. Four separate systems to configure and operate. With RisingWave, you connect it directly to the PostgreSQL logical replication slot, define your materialized views, and point your application at the RisingWave endpoint. Two systems.

This is also where the change data capture foundation matters. RisingWave’s native CDC support handles the replication offset tracking, schema evolution events, and tombstone handling that you would otherwise need to build into your Debezium + Kafka pipeline manually.

Multi-Source Stream Joins

Joining two Kafka topics in real time, a classic stream processing use case, is painful in Flink because you need to reason carefully about watermarks, join semantics, and state growth. In RisingWave, a stream-stream join is a SQL join. The semantics are standard, and the incremental maintenance is handled by the engine.

The catch is that temporal joins (joining a stream against a point-in-time snapshot of another stream) require understanding RisingWave’s temporal join syntax, which differs from Flink’s. I recommend reading the documentation carefully before assuming SQL semantics translate directly. The concepts map, but the syntax has quirks.

Production Pitfalls

I want to give you the failures I have seen, not just the success stories.

Unbounded state growth. Materialized views that group by a high-cardinality key without a time window will grow indefinitely. A view that groups by user ID accumulates a row for every user who has ever appeared in the stream. Over months, this consumes significant storage. Add WHERE event_time > NOW() - INTERVAL '30 days' or use tumbling window semantics to bound your state. This sounds obvious but I have seen it bite teams repeatedly.

CDC source ordering. When you ingest from a CDC source, events arrive ordered by commit timestamp within a single table. If your materialized view joins two tables, you may see temporary inconsistencies during high write throughput because updates to the two tables are ingested on different schedules. Materialize handles this with its consistency model; RisingWave handles it eventually. Know which you need before you build on it.

Schema evolution is harder than you expect. Adding a column to a source Kafka topic requires updating the schema registry, updating the source definition in the streaming database, and potentially rebuilding materialized views that reference the old schema. The Kafka Schema Registry integration works, but rolling schema changes in production requires a plan. I have never seen a team get this right on the first try.

Cascading view complexity. RisingWave supports cascading materialized views, where one view is defined on top of another. This is powerful for building layered transformations, but it creates dependency chains that are hard to reason about during incidents. If a base view gets behind on ingestion, every downstream view gets behind too. Keep your view dependency depth shallow in production, two or three levels maximum.

Testing is underdeveloped. Unlike Flink, which has a test harness and rich documentation on testing streaming jobs, streaming databases have limited tooling for automated testing. I write integration tests that spin up a local instance against a test Kafka broker, but there is no equivalent of Flink’s StreamTestHarness. This is an area where the ecosystem needs more investment.

Cost and Sizing

One thing streaming databases have in common: they are cheaper to operate than equivalent Flink infrastructure, once you account for engineering time.

A Flink cluster handling a few hundred thousand events per second typically needs dedicated engineering attention several hours per week for tuning, checkpoint management, and handling occasional job failures. That engineering time is expensive.

A RisingWave cluster handling the same workload, once set up, runs largely unattended. The S3-backed state model eliminates most failure modes. The compute nodes can be smaller because there is no JVM overhead and no large RocksDB state in heap. For the workloads that fit the streaming database model, I consistently see teams getting the same business outcome with less infrastructure spend and less operational overhead.

Materialize is the exception. Its pricing model makes it expensive compared to self-hosted alternatives, and I would only use it when the consistency guarantees justify the cost.

For RisingWave sizing, start with two compute nodes of four cores each and scale from there based on observed lag between ingestion and view staleness. The S3 costs for state storage are typically low relative to compute.

Connecting to the Broader Data Platform

Streaming databases do not live in isolation. They sit between your event sources and your downstream consumers, and the integration points matter.

On the input side, you are typically ingesting from Kafka (ideally via the Apache Kafka ecosystem) or from database CDC. RisingWave handles both natively.

On the output side, streaming database materialized views typically feed into three places: application queries over the wire protocol, exports to Apache Iceberg tables for long-term analytical storage, and Kafka output topics for downstream consumers. The ability to write materialized view results back to Kafka is important for integrating with systems that cannot query a database directly.

For teams building a complete data platform, streaming databases fit alongside stream processing tools rather than replacing them. Put the complex stateful computation and custom operators in Flink. Put the SQL aggregations, dashboards, and read models in a streaming database. The two categories complement each other.

Data governance becomes interesting when streaming databases are in the mix. Materialized views need to appear in your data catalog with their lineage documented. OpenMetadata and DataHub both support custom connectors, and the RisingWave community maintains integrations for both. This is not automatic setup, but it is straightforward once you have the connector configured.

What I Would Actually Deploy

If I were starting a new real-time data platform today and needed to cover the streaming analytics use case, I would deploy RisingWave. Self-hosted, S3-backed, PostgreSQL-compatible. For teams already on Kubernetes, it runs well as a Helm deployment with persistent volume claims for the local compute cache and S3 for durable state.

I would use Materialize only if the use case required strict serializable consistency and the budget supported managed SaaS pricing. For most analytics workloads, the eventual consistency of RisingWave is fine and the cost difference is significant.

I would avoid ksqlDB for new projects. The ecosystem dependency on Confluent is a long-term risk, and the SQL dialect limitations create friction that compounds over time.

Flink stays in the picture for workloads that need it: complex custom operators, sub-millisecond latency, or processing patterns that cannot be expressed in SQL. But I no longer reach for Flink by default. After twenty years of watching teams underestimate the operational cost of distributed stream processing frameworks, I appreciate systems that hide the execution model behind a SQL interface and let engineers focus on what the pipeline actually needs to compute.

The streaming database category is not a replacement for everything that came before it. It is a better tool for a specific class of problem, and knowing where that class boundary sits will save you months of Flink debugging.