User Guide

• High-Level Overview
  • The physical layer
  • The one-dimensional layer
  • The two-dimensional layer
  • The compute layer
  • The IO layer
  • The Inter-Process Communication (IPC) layer
  • The file formats layer
  • The devices layer
  • The filesystem layer
• Memory Management
  • Buffers
    • Accessing Buffer Memory
    • Slicing
    • Allocating a Buffer
    • Building a Buffer
  • Memory Pools
    • Default Memory Pool
    • Overriding the Default Memory Pool
    • STL Integration
  • Devices
    • Device-Agnostic Programming
  • Memory Profiling
• Arrays
  • Building an array
    • Available strategies
    • Using ArrayBuilder and its subclasses
    • Performance
  • Size Limitations and Recommendations
  • Chunked Arrays
  • Slicing
• Data Types
  • Creating data types
  • Type Traits
  • Visitor Pattern
• Tabular Data
  • Fields
  • Schemas
  • Tables
  • Record Batches
• Compute Functions
  • The generic Compute API
    • Functions and function registry
    • Input shapes
    • Invoking functions
  • Implicit casts
    • Common numeric type
  • Available functions
    • Type categories
    • Aggregations
      • Grouped Aggregations (“group by”)
    • Element-wise (“scalar”) functions
      • Arithmetic functions
      • Bit-wise functions
      • Rounding functions
      • Logarithmic functions
      • Trigonometric functions
      • Comparisons
      • Logical functions
      • String predicates
      • String transforms
      • String padding
      • String trimming
      • String splitting
      • String component extraction
      • String joining
      • String Slicing
      • Containment tests
      • Categorizations
      • Selecting / multiplexing
      • Structural transforms
      • Conversions
      • Temporal component extraction
      • Temporal difference
      • Timezone handling
      • Random number generation
    • Array-wise (“vector”) functions
      • Cumulative Functions
      • Associative transforms
      • Selections
      • Containment tests
      • Sorts and partitions
      • Structural transforms
      • Replace functions
      • Pairwise functions
• The Gandiva Expression Compiler
  • Expression, Projector and Filter
  • External Functions Development
    • Gandiva Expression, Projector, and Filter
      • Building Expressions
      • Projectors and Filters
        • Evaluating projections
        • Evaluating filters
        • Evaluating projections and filters
    • Gandiva External Functions Development Guide
      • Introduction
      • Overview of External Function Types in Gandiva
        • Choosing the Right Type of External Function for Your Needs
      • External function registration
        • Metadata Registration Using the NativeFunction Class
        • External C functions
          • C Function Signature
            • Signature Mapping
            • Handling arrow::StringType (utf8 type) and arrow::BinaryType
        • External C function registration APIs
        • External IR functions
          • IR function implementation
            • Examples and Tools for Compilation
            • Consistency in Parameter and Return Types
          • Registering External IR Functions in Gandiva
      • Conclusion
• Acero: A C++ streaming execution engine
  • Acero Overview
    • What is Acero?
      • Acero is not…
        • A Library for Data Scientists
        • A Database
        • An Optimizer
        • Distributed
      • Acero vs…
        • Arrow Compute
        • Arrow Datasets
        • Substrait
        • Datafusion / DuckDb / Velox / Etc.
    • Relation to Arrow C++
    • Core Concepts
      • ExecNode
      • ExecBatch
      • ExecPlan
      • Declaration
  • Acero User’s Guide
    • Using Acero
    • Creating a Plan
      • Using Substrait
      • Programmatic Plan Creation
    • Executing a Plan
      • DeclarationToTable
      • DeclarationToReader
      • DeclarationToStatus
      • Running a Plan Directly
    • Providing Input
    • Available ExecNode Implementations
      • Sources
      • Compute Nodes
      • Arrangement Nodes
      • Sink Nodes
    • Examples
      • source
      • table_source
      • filter
      • project
      • aggregate
      • sink
      • consuming_sink
      • order_by_sink
      • select_k_sink
      • table_sink
      • scan
      • write
      • union
      • hash_join
    • Summary
  • Using Acero with Substrait
    • Substrait Conformance
      • Plans
      • Extensions
      • Relations (in general)
      • Read Relations
      • Filter Relations
      • Project Relations
      • Join Relations
      • Aggregate Relations
      • Expressions (general)
      • Literals
      • Types
      • Functions
  • Developer’s Guide
    • Understanding ExecNode
      • ExecNode::StartProducing()
        • Examples
      • ExecNode::InputReceived()
        • Examples
      • ExecNode::InputFinished()
        • Examples
      • ExecNode::PauseProducing() / ExecNode::ResumeProducing()
        • Examples
      • ExecNode::StopProducing()
        • Examples
      • Initialization / Construction / Destruction
      • Summary
    • Extending Acero
    • Scheduling and Parallelism
      • Parallel Execution of Plans
      • Locally Distributed Plans
      • Pipeline Parallelism
      • Asynchronicity
      • Task per Pipeline (and sometimes beyond)
      • Thread Pools and Schedulers
      • Intra-node Parallelism
      • All Work Happens in Tasks
    • Ordered Execution
    • Partitioned Execution
    • Spillover
    • Distributed Execution
    • Profiling & Tracing
    • Benchmarking
    • Bindings
      • Public API
      • R (dplyr)
      • Python
      • Java
    • Design Philosophies
      • Engine Independent Compute
      • Make Tasks not Threads
      • Don’t Block on CPU Threads
      • Don’t Reinvent the Wheel
      • Avoid Query Optimization
    • Performance Guidelines
      • Batch Size
    • Ongoing & Deprecated Work
      • Scanner v2
      • OrderBySink and SelectKSink
    • Upstreaming Changes
• Input / output and filesystems
  • Reading binary data
  • Writing binary data
  • Filesystems
  • Defining new filesystems
• Reading and writing the Arrow IPC format
  • Reading IPC streams and files
    • Synchronous reading
    • Event-driven reading
  • Writing IPC streams and files
  • Configuring
• Reading and Writing ORC files
  • Supported ORC features
    • Data types
    • Compression
  • Reading ORC Files
    • ORCFileReader
  • Writing ORC Files
    • ORCFileWriter
• Reading and writing Parquet files
  • Reading Parquet files
    • FileReader
      • Performance and Memory Efficiency
    • StreamReader
  • Writing Parquet files
    • WriteTable
    • StreamWriter
    • Writer properties
  • Supported Parquet features
    • Page types
    • Compression
    • Encodings
    • Types
      • Physical types
      • Logical types
      • Converted types
      • Special cases
      • Roundtripping Arrow types and schema
        • Parquet field id
        • Serialization details
      • Limitations
    • Encryption
    • Miscellaneous
• Reading and Writing CSV files
  • Reading CSV files
    • TableReader
    • StreamingReader
    • Tradeoffs
  • Writing CSV files
  • Column names
  • Column selection
    • Interaction with column names
  • Data types
    • Dictionary inference
    • Timestamp inference/parsing
    • Nulls
    • Character encoding
  • Write Options
  • Performance
• Reading JSON files
  • TableReader
  • StreamingReader
  • Data types
• Tabular Datasets
  • Reading Datasets
    • Dataset discovery
    • Reading different file formats
    • Customizing file formats
  • Filtering data
  • Projecting columns
  • Reading and writing partitioned data
    • Different partitioning schemes
    • Partitioning performance considerations
  • Reading from other data sources
    • Reading in-memory data
    • Reading from cloud storage
  • A note on transactions & ACID guarantees
  • Full Example
• Arrow Flight RPC
  • Writing a Flight Service
  • Using the Flight Client
  • Cancellation and Timeouts
  • Enabling TLS
  • Enabling Authentication
  • Custom Middleware
  • Best practices
    • gRPC
    • Re-use clients whenever possible
    • Don’t round-robin load balance
    • Debugging connection issues
    • Memory management
    • Excessive traffic
    • Limiting DoPut Batch Size
    • Closing unresponsive connections
  • Alternative Transports
    • UCX Transport
• Debugging code using Arrow
  • GDB extension for Arrow C++
    • Manual loading
    • Automatic loading
    • Supported classes
• Thread Management
  • Thread Pools
    • CPU vs. I/O
    • Serial Execution
    • Jemalloc Background Threads
  • Asynchronous Utilities
    • Future
• OpenTelemetry
  • Creating a Build with OpenTelemetry Tracing Enabled
  • Exporting Tracing Information
  • Visualizing Traces with Jaeger UI
• Environment Variables