Arroyo is a Rust stream processor that turns SQL into fault-tolerant pipelines

Arroyo is a distributed stream processing engine written in Rust. The pitch in its own README is direct: it lets you "ask complex questions of high-volume real-time data with subsecond results," operating on both bounded and unbounded sources and emitting results as soon as they are available, rather than waiting for batch boundaries.

The defining choice is that pipelines are defined in SQL. You do not write operators by hand; the engine compiles a query into an execution graph of stateful operators — windows, joins, aggregations — and runs them across a cluster with checkpointing so jobs can recover from failures. It is dual-licensed under Apache-2.0 OR MIT.

What's in the repository

The repo is a Cargo workspace of roughly two dozen crates plus webui/, k8s/, and docker/. The crate names map cleanly onto the system's parts:

• arroyo-planner turns SQL into an executable program, built on DataFusion. It registers functions (including JSON and window functions), parses with a custom ArroyoDialect, and produces a LogicalProgram.
• arroyo-datastream holds that logical representation: the operator graph, edge types (Forward, Shuffle, joins), and window types such as tumbling, sliding, session, and instant.
• arroyo-controller is the job lifecycle state machine. It tracks jobs through states like Created and schedules them across workers.
• arroyo-worker executes a dataflow. Its engine.rs builds the operator graph from the logical program, wiring constructors for windows, joins, projections, watermark generation, and connector sources and sinks into SubtaskNodes that run in parallel.
• arroyo-state and arroyo-state-protocol implement checkpointing: epochs, generations, manifests, and state tables.
• arroyo-api is the REST API; webui/ is a React front end served at port 5115.

The planner parses SQL with sqlparser's custom ArroyoDialect and uses DataFusion (pinned to version 54.1.0) for analysis and optimization, then rewrites the logical plan into Arroyo's own graph via rewriters such as SinkInputRewriter, SourceMetadataVisitor, and a chaining optimizer that fuses operators.

One binary, many roles

Arroyo ships as a single binary called arroyo. The CLI in crates/arroyo/src/main.rs dispatches to several subcommands:

• run — run a query as a local pipeline cluster
• cluster — start the full control plane (API, controller, compiler) plus workers
• api, controller, worker, compiler, node — start individual roles for distributed deployments
• migrate — run database migrations on Postgres
• visualize — render a query plan

For local use the README suggests starting everything with one command:

$ arroyo cluster

or in Docker, mapping port 5115 so the web UI is reachable at http://localhost:5115. The control plane persists metadata to either Postgres or SQLite; main.rs shows both paths, including a migration runner that creates and upgrades the schema.

How a pipeline runs

The flow from query to running job passes through several crates:

1. Plan. A SQL string is parsed with sqlparser's ArroyoDialect, analyzed by DataFusion, and rewritten into an Arroyo logical program — a directed graph of nodes like ConnectorSource, TumblingWindowAggregate, Join, and ConnectorSink.
2. Schedule. The controller takes the compiled program and assigns it to workers as jobs with parallelism (each node fans out into subtasks).
3. Execute. Each worker's engine instantiates operators from constructors — IncrementalAggregatingConstructor, TumblingAggregateWindowConstructor, JoinWithExpirationConstructor, and others in crates/arroyo-worker/src/engine.rs — and runs them as a dataflow, shuffling records across subtasks where the graph requires it.
4. Checkpoint. State is periodically snapshotted through the state protocol so a failed job can resume from its last consistent epoch.

The window types in crates/arroyo-datastream/src/lib.rs — tumbling, sliding, session, and instant windows with configurable widths and gaps — are where much of the "stateful" work lives. Time-oriented processing follows the Dataflow model cited in the README.

Connectors and deployment

Connectors live in crates/arroyo-connectors, which registers sources and sinks including Kafka and Iceberg, plus filesystem connectors for object storage reads and writes. The engine also supports user-defined functions: there is a Rust UDF host with dynamic-library loading (dlopen2), Python UDFs via arroyo-udf-python, and a compiler service that builds UDF code on demand.

For production, the repository ships Helm charts under k8s/arroyo for Kubernetes, alongside Docker images. The README also points to Cloudflare Pipelines as a managed offering built on Arroyo, currently supporting stateless pipelines ingesting into R2.

Who it's for

Arroyo is aimed at teams that want real-time streaming without becoming Flink or Spark experts — the README explicitly lists Apache Flink, Spark Streaming, and Kafka Streams as existing alternatives and argues its case with serverless-style operations, first-class SQL performance, and a clean separation between pipeline APIs and internals. Use cases it names include fraud detection, real-time product analytics, ingestion into data warehouses or lakes, and ML feature generation.

The project is actively developed — the latest release tag at this snapshot is v0.15.0 with recent commits through November 2026 — and welcomes contributions via CONTRIBUTING.md, a developer setup guide in the docs, and an active Discord community.