Apache Arrow in Production: How the Columnar Format and Arrow Flight Protocol Became the Nervous System of Modern Data Engineering

For the first fifteen years of my career, moving data between systems was a constant tax. You serialized into JSON, deserialized at the other end, paid the allocation costs twice, and hoped nobody had introduced a floating-point precision problem in between. When you needed speed, you dropped to CSV, which is somehow both more efficient and more broken. JDBC cursors fetched rows one at a time, or maybe a batch of a thousand, and you tuned fetch size like an artisan working a kiln. The waste was massive and everyone accepted it as the price of interoperability.

Apache Arrow changed that bargain. Over the past four years, it has quietly become the connective tissue of the modern data stack, and most engineers working with Polars, DuckDB, Spark, or any modern data warehouse are using it without knowing it. Understanding Arrow directly changes how you design data pipelines, data APIs, and the hand-off points between systems. This article covers the actual architecture, not just the marketing.

What Apache Arrow Actually Is

Arrow is not a database, not a query engine, and not a file format (though it does have one). It is primarily a specification for an in-memory columnar data layout, plus a set of official libraries implementing that specification in C++, Java, Python, Go, Rust, and other languages. The Apache Arrow project also defines the IPC format for persisting or streaming Arrow data, and Arrow Flight, the high-performance transport protocol built on gRPC.

The spec was originally proposed by Wes McKinney (creator of Pandas) and Jacques Nadeau in 2016. The design goal was a single canonical in-memory representation that any analytics tool could adopt so that handing data from one tool to another required zero serialization. That goal is largely achieved now. When you call to_arrow() on a Polars DataFrame or when DuckDB hands off a result to Python, there is no copy, no conversion, just a pointer to a chunk of memory organized the way Arrow specifies.

The key insight is that columnar layout is not just a storage optimization. It is the right shape for analytical computation in RAM.

Why Columnar Layout Changes Everything

Imagine a table with a million rows and twenty columns. In a row-oriented layout (what your PostgreSQL heap or a Python list-of-dicts gives you), each row is stored contiguously. To sum a single column, the CPU must skip across all twenty column values per row, dragging sixty-three irrelevant columns through the cache along the way.

In columnar layout, all values for a given column sit contiguously in memory. The CPU fetches a 64-byte cache line and gets eight 8-byte doubles, all for the same column. Modern CPUs with SIMD instructions can then sum sixteen 32-bit integers per clock cycle, or eight 64-bit doubles, with no scalar loop. Arrow’s specification mandates 64-byte alignment and dictates that buffers are padded to multiples of 64 bytes, which is not an accident. It is designed so that SIMD vectorization works without conditional branches or misaligned loads.

The practical result: operations like sum, min, max, filter, group by, and join on Arrow buffers are five to twenty times faster than on equivalent Pandas DataFrames backed by NumPy. This is why Polars, built natively on Arrow, is so much faster than Pandas for most analytical workloads. Arrow is the reason, not just Rust.

Arrow also defines a rich type system: integers of various widths, floats, decimals with specified precision and scale, dates, timestamps with timezone info, durations, binary data, strings, lists, structs, maps, and dictionary-encoded types. The dictionary encoding is important: it gives you categorical variables without paying the string copy cost on every value. A billion-row dataset with a “country” column stores the country strings once in a dictionary and the column as an array of 16-bit integer indices.

The Zero-Copy Promise

The deepest value of Arrow is not just speed within a single system. It is the zero-copy transfer between systems. Before Arrow, handing data from a Spark DataFrame to a Python function required serializing the Spark object, sending it over the network or through Py4J, deserializing on the Python side, and probably converting to Pandas. That round-trip could cost hundreds of milliseconds per operation.

With Arrow, if two processes share memory (via a shared memory segment or simply by running in the same process), passing an Arrow RecordBatch is literally handing a pointer. There are no allocations, no copies, no serialization. The receiving process views the same bytes the producing process wrote. This is what pyarrow.plasma (now deprecated) and more modern shared memory implementations in ADBC and DataFusion exploit.

Even when you cannot share memory, Arrow’s IPC format is designed for minimal-overhead streaming. An Arrow IPC stream is a sequence of RecordBatches, each self-describing with a schema and a compact metadata header. Deserializing an IPC message is just computing offsets into a buffer you already received, not parsing fields one by one.

I spent several months in 2023 replacing a system that exchanged data between a Flink job and a Python scoring service via Protobuf. The scoring service was the bottleneck. We were spending forty percent of CPU time in deserialization. We switched to Arrow IPC over a Unix socket. Throughput tripled and CPU dropped by thirty percent. The only code change was the serialization layer.

Arrow IPC vs Parquet: Not the Same Thing

A common confusion is conflating Arrow IPC format with Parquet. They are both columnar, they are both produced by the same project, but they serve different purposes.

Parquet is designed for on-disk storage and long-term retention. It applies heavy compression (Snappy, Zstd, LZ4, Gzip), uses dictionary encoding, delta encoding, and run-length encoding at the byte level, and its metadata is designed for predicate pushdown when scanning files. Reading a Parquet file into memory is expensive because of all that decompression and decoding. But Parquet files are small and cheap to store, which is why they dominate the Apache Iceberg data lakehouse ecosystem.

Arrow IPC is designed for in-memory speed and streaming transfers. It does not compress (by default), it does not apply complex encodings, and it deserializes to the Arrow in-memory format with near-zero work. The tradeoff is size: an Arrow IPC file might be two to five times larger than the equivalent Parquet file.

The right pattern is: store data in Parquet for the lake, read it into Arrow format for processing, exchange it between services as Arrow IPC, and write query results as Arrow IPC before returning them to callers. This is exactly what DuckDB does internally: it reads Parquet, processes in Arrow-layout buffers, and returns Arrow RecordBatches.

Arrow Flight: Data Transfer at Wire Speed

Arrow Flight is the protocol that takes Arrow from an in-process format to a network-capable data API. It is a gRPC-based protocol where the payload is Arrow IPC streams, not JSON or Protobuf. The design goal was simple: move as many Arrow RecordBatches per second as possible over a TCP connection, with backpressure and flow control.

A Flight server exposes a small set of verbs: ListFlights for discovery, GetFlightInfo for metadata about a specific dataset, DoGet for retrieving data, DoPut for uploading data, and DoAction for arbitrary RPCs. The actual data transfer in DoGet and DoPut is a stream of Arrow IPC messages, which means the client can start processing the first batch while the server is still producing the last one.

The benchmark numbers are striking. Arrow Flight over a 100 GbE network can sustain transfers above 20 GB/s. JDBC over the same network, even with batched fetches, tops out around 2-3 GB/s because of the per-row serialization overhead and the blocking cursor model. For data-intensive workflows where you need to pull a billion rows from a query engine into a training pipeline, that difference is the difference between a five-minute operation and a forty-minute one.

Arrow Flight also supports parallel transfers: a FlightInfo response can return multiple FlightEndpoint objects, each pointing to a different server holding a partition of the data. Clients can fetch these partitions in parallel, fully saturating multiple network links simultaneously. This is how distributed systems like Dremio and Ballista implement high-throughput query result delivery.

Arrow Flight SQL: Queries Over Arrow

Arrow Flight SQL extends Flight with a standard set of actions for running SQL queries, fetching prepared statement results, and introspecting catalog metadata. It is designed to replace JDBC and ODBC as the primary interface for analytical databases.

The pitch is compelling. Instead of a JDBC driver that fetches rows into Java objects that Spark then converts to its internal representation, an Arrow Flight SQL driver returns Arrow RecordBatches directly. You skip two serialization round-trips. DuckDB ships an Arrow Flight SQL server in its ADBC driver. Dremio, DataFusion, and several others implement it natively. Snowflake has been adding Arrow-based result delivery to its own drivers for the same reason.

For Apache Spark, the Arrow integration is particularly impactful. When you use PySpark and call toPandas(), Spark uses Arrow to transfer the data from the JVM to Python, avoiding the Py4J row-by-row serialization that once made this operation impractically slow for large datasets. The config is spark.sql.execution.arrow.pyspark.enabled = true. Enable it and you will notice.

ADBC: Replacing JDBC and ODBC

Arrow Database Connectivity (ADBC) is the unified driver interface that lets you talk to any Arrow Flight SQL or Arrow-native database through a single API. Think JDBC but without the object-relational impedance mismatch, and without the implicit serialization penalty.

An ADBC driver returns Arrow RecordBatches from queries. Your application code works with those directly, passing them to Polars, Pandas 2.x, or DuckDB without any conversion. The ADBC spec defines a C interface so that drivers can be written in any language and linked without JVM or Python overhead.

This matters for two reasons. First, it simplifies your stack. Instead of a JDBC driver, a Pandas read function, and a conversion to Arrow, you have one driver call and an Arrow RecordBatch. Second, it enables a new class of data applications: services that accept Arrow IPC from upstream, compute over it, and return Arrow IPC to downstream consumers, all without ever touching a row-oriented representation.

I have been building a small internal query API for a client using DataFusion as the execution engine. The server accepts Arrow Flight calls, runs DataFusion queries over Parquet in S3, and returns Arrow IPC streams. The calling services use ADBC drivers. The whole pipeline, from query dispatch to result available in memory, happens in Arrow-native format. We are seeing latency two to three times lower than the JDBC-based predecessor for the same queries on the same data volume.

The Arrow Ecosystem: More Than You Think

Most engineers encounter Arrow through a library but do not realize how deep the ecosystem goes.

Apache DataFusion is a query engine written in Rust that uses Arrow as its internal representation for everything: plans, intermediate results, and final output. It is embedded in DuckDB’s Arrow extension, in delta-rs (the Rust Delta Lake library), and in dozens of other tools. If you have used DuckDB for embedded analytics, you have run Arrow-native compute.

Apache Substrait is a cross-language specification for query plans that pairs naturally with Arrow. A system can serialize its query plan as Substrait, send it to another system via Arrow Flight, and that system can execute it natively. This is the foundation of a genuinely federated query ecosystem where the computation moves to the data rather than the data moving to the computation.

Lance is a new columnar storage format optimized for multimodal AI data: it stores vector embeddings alongside tabular columns and images in a single Arrow-compatible format. LanceDB, a vector database built on Lance, is natively Arrow-compatible. As vector search becomes part of standard RAG architectures, Arrow becomes the bridge between the vector layer and the tabular analytics layer.

Velox, Meta’s open-source vectorized execution engine that powers Presto on Meta’s infrastructure, uses Arrow as its memory format. When you run a Presto or Trino query on Meta-scale infrastructure, Arrow is handling the in-memory representation.

Production Patterns: How to Actually Use This

Understanding the theory is one thing. Here is how Arrow shows up in production data stacks.

The result API pattern: Instead of a REST API returning JSON arrays, expose data results as Arrow IPC streams over HTTP or gRPC. Clients that can consume Arrow (Python, Java, Go, Rust, C++) read results with no parsing overhead. Clients that cannot (browsers, legacy systems) get a JSON fallback. The performance difference for data-heavy endpoints is an order of magnitude.

The inter-service hand-off: If you have a service that produces data and a service that consumes it within the same cluster, Arrow over Unix sockets or shared memory eliminates serialization entirely. This is increasingly practical as data pipelines run as sidecars or colocated microservices.

The training data pipeline: Between feature generation (often Spark or Flink) and model training (Python/PyTorch), the serialization layer is frequently the bottleneck. Arrow IPC files written to local NVMe, or Arrow Flight streams from a DataFusion service, eliminate the Parquet-read overhead at training time. Training pipelines I have seen move from Parquet files to Arrow IPC files for the “hot” data tier that feeds GPU workers, keeping Parquet only for the cold archive.

The ML serving layer: Tabular ML models (gradient boosting, neural networks for structured data) need feature vectors assembled at inference time. An Arrow-native feature store can compute features in columnar layout, apply batch inference across a RecordBatch, and return predictions as Arrow. Frameworks like XGBoost and LightGBM have native Arrow support for exactly this pattern.

When Arrow Is Not the Right Tool

Arrow optimizes for analytical, columnar workloads with large batches and sequential access. It is the wrong representation for:

Transactional, row-oriented access. If your access pattern is “fetch row by primary key, update two columns, commit,” Arrow adds no value and significant complexity. PostgreSQL and your row-oriented OLTP databases already have this solved. Arrow is not trying to replace them.

Small data. The setup overhead of Arrow buffers, schema definitions, and Flight connections does not pay off below roughly 100K rows or 1 MB of data. For small responses, JSON or Protobuf is simpler and fast enough.

Streaming row events. A Kafka topic of individual event records has no natural fit with Arrow’s batch-first model. Kafka with Avro or Protobuf schemas is the right tool. Arrow Flight with micro-batches can work if the consumer is analytically oriented, but it is engineering overhead for marginal gain.

Unstructured data. Arrow is a typed, schematized format. If you are moving binary blobs, images, or free-form text around, object storage or gRPC streaming with Protobuf is simpler.

The pattern I use: Arrow for any data path where the consumer will perform aggregate, filter, or join operations over the data; Protobuf or Avro for individual event records and API responses; Parquet for archival storage. These three cover ninety percent of what I encounter.

What This Means for Your Architecture

The Apache Arrow ecosystem changes the economics of data architecture in two concrete ways.

First, it makes co-located compute-with-data practical. When the cost of moving data between systems is nearly zero (because Arrow IPC is close to memcpy speed), you can afford smaller, specialized compute nodes that hand off Arrow streams to each other rather than one giant shared-memory cluster. This has implications for how you size your data warehouse and data lakehouse infrastructure.

Second, it decouples the compute engine from the query interface. A DataFusion engine behind an Arrow Flight SQL server can be hot-swapped for DuckDB or Velox as long as they speak the same protocol. This is a genuinely new kind of interoperability. In twenty years of building data platforms, I have spent enormous effort on ETL glue code that existed solely because systems could not agree on a data representation. Arrow makes that glue code unnecessary for the analytical layer.

The practical advice: if you are designing a new data service, data API, or processing pipeline, evaluate whether your components support Arrow natively. DuckDB, Polars, and DataFusion do. Pandas 2.x has improved Arrow support. Apache Spark with Arrow enabled behaves like a different (much better) system. Snowflake’s result delivery increasingly returns Arrow. BigQuery’s Storage Read API uses Arrow. The ecosystem has reached the point where building Arrow-native is the path of least resistance, not the advanced path.

Arrow is not a buzzword or a new product to buy. It is a specification that the whole ecosystem has converged on, quietly, over the past few years. Recognizing it means recognizing why your DuckDB queries feel so fast, why Polars handles hundred-million-row DataFrames without complaint, and why the toPandas() call in Spark suddenly stopped being the bottleneck it used to be. The format did the work. The least you can do is understand it.