Broadway
Concurrent and multi-stage data ingestion and processing with Elixir
Broadway is an open-source Elixir library for building concurrent and multi-stage data ingestion and processing pipelines. It leverages the Erlang VM's supervision trees for robust failure handling and graceful shutdowns, with built-in integrations for popular message queues.
3K
GitHub Stars
medium
Learning Curve
4.0
DX Score
Pricing
Model
free
Free Tier
Completely free and open-source.
Features
- ✓ Concurrent data ingestion with batching
- ✓ RabbitMQ integration
- ✓ Amazon SQS integration
- ✓ Google PubSub integration
- ✓ Apache Kafka integration
- ✓ Back pressure via GenStage
- ✓ Built-in rate limiting
- ✓ Broadway Dashboard for monitoring
- ✓ Graceful shutdown handling
Pros
- + Robust failure handling through Erlang VM
- + Production-proven by Discord, Change.org
- + Extensive documentation
- + Built-in monitoring dashboard
- + Prevents system overload with back pressure
Cons
- - Limited to Elixir ecosystem
- - Requires understanding of GenStage concepts
- - Smaller community than Kafka Streams/Flink
- - Learning curve for non-Elixir developers
Best For
indie hacker startup
elixir erlang beam data-pipeline streaming queue kafka sqs rabbitmq