Apache Arrow

Apache Arrow is a universal columnar memory format and multi-language toolbox designed for fast data interchange and efficient in-memory analytics across different programming languages and systems. It defines a language-independent columnar memory format for both flat and nested data structures, optimized for efficient analytical operations on modern hardware including CPUs and GPUs. The Arrow format supports zero-copy data sharing between processes, eliminating the serialization and deserialization overhead that typically dominates data transfer costs in analytics pipelines. This zero-copy capability enables data to be passed between different languages such as Python, R, Java, C++, Rust, Go, and JavaScript without any conversion cost, making it foundational infrastructure for modern data engineering and data science workflows. The Arrow ecosystem includes several key subprojects: Arrow Flight for high-performance data transport with gRPC-based RPC framework, Arrow Flight SQL for database query over Flight, Parquet for columnar on-disk storage format, Gandiva for LLVM-based expression evaluation, and Arrow Dataset for reading partitioned datasets. Arrow is the underlying technology powering major data platforms including Apache Spark, Pandas (via PyArrow), Polars, DuckDB, Dremio, InfluxDB, and many others. Major cloud providers have adopted Arrow as the standard for columnar data exchange, with Amazon, Google, and Microsoft all contributing to the project. As an Apache Software Foundation top-level project with over 15,000 GitHub stars, Arrow represents a fundamental building block of the modern data engineering stack, enabling interoperability between diverse data tools and dramatically reducing the computational overhead of data processing.