• Stars
    star
    2,414
  • Rank 19,056 (Top 0.4 %)
  • Language
    Elixir
  • License
    Apache License 2.0
  • Created about 6 years ago
  • Updated 4 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Concurrent and multi-stage data ingestion and data processing with Elixir

Broadway

CI

Build concurrent and multi-stage data ingestion and data processing pipelines with Elixir. Broadway allows developers to consume data efficiently from different sources, known as producers, such as Amazon SQS, Apache Kafka, Google Cloud PubSub, RabbitMQ, and others. Broadway pipelines are long-lived, concurrent, and robust, thanks to the Erlang VM and its actors.

Broadway takes its name from the famous Broadway street in New York City, renowned for its stages, actors, and producers.

To learn more and get started, check out our official website and our guides and docs.

Broadway Logo

Built-in features

Broadway takes the burden of defining concurrent GenStage topologies and provide a simple configuration API that automatically defines concurrent producers, concurrent processing, batch handling, and more, leading to both time and cost efficient ingestion and processing of data. It features:

  • Back-pressure
  • Automatic acknowledgements at the end of the pipeline
  • Batching
  • Fault tolerance
  • Graceful shutdown
  • Built-in testing
  • Custom failure handling
  • Ordering and partitioning
  • Rate-limiting
  • Metrics

Producers

There are several producers that you can use to integrate with existing services and technologies. See the docs for detailed how-tos and supported producers.

Installation

Add :broadway to the list of dependencies in mix.exs:

def deps do
  [
    {:broadway, "~> 1.0"}
  ]
end

A quick example: SQS integration

Assuming you have added broadway_sqs as a dependency and configured your SQS credentials accordingly, you can consume Amazon SQS events in only 20 LOCs:

defmodule MyBroadway do
  use Broadway

  alias Broadway.Message

  def start_link(_opts) do
    Broadway.start_link(__MODULE__,
      name: __MODULE__,
      producer: [
        module: {BroadwaySQS.Producer, queue_url: "https://us-east-2.queue.amazonaws.com/100000000001/my_queue"}
      ],
      processors: [
        default: [concurrency: 50]
      ],
      batchers: [
        s3: [concurrency: 5, batch_size: 10, batch_timeout: 1000]
      ]
    )
  end

  def handle_message(_processor_name, message, _context) do
    message
    |> Message.update_data(&process_data/1)
    |> Message.put_batcher(:s3)
  end

  def handle_batch(:s3, messages, _batch_info, _context) do
    # Send batch of messages to S3
  end

  defp process_data(data) do
    # Do some calculations, generate a JSON representation, process images.
  end
end

Once your Broadway module is defined, you just need to add it as a child of your application supervision tree as {MyBroadway, []}.

Comparison to Flow

You may also be interested in Flow by Dashbit. Both Broadway and Flow are built on top of GenStage. Flow is a more general abstraction than Broadway that focuses on data as a whole, providing features like aggregation, joins, windows, etc. Broadway focuses on events and on operational features, such as metrics, automatic acknowledgements, failure handling, and so on. Broadway is recommended for continuous, long-running pipelines. Flow works with short- and long-lived data processing.

License

Copyright 2019 Plataformatec
Copyright 2020 Dashbit

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

More Repositories

1

flow

Computational parallel flows on top of GenStage
Elixir
1,544
star
2

mox

Mocks and explicit contracts in Elixir
Elixir
1,343
star
3

nimble_parsec

A simple and fast library for text-based parser combinators
Elixir
807
star
4

nimble_csv

A simple and fast CSV parsing and dumping library for Elixir
Elixir
771
star
5

nimble_options

A tiny library for validating and documenting high-level options. 💽
Elixir
505
star
6

nimble_publisher

A minimal filesystem-based publishing engine with Markdown support and code highlighting
Elixir
467
star
7

nimble_totp

A tiny Elixir library for time-based one time passwords (TOTP)
Elixir
396
star
8

nimble_pool

A tiny resource-pool implementation for Elixir
Elixir
344
star
9

bytepack_archive

Archive of bytepack.io
Elixir
328
star
10

broadway_kafka

A Broadway connector for Kafka
Elixir
232
star
11

broadway_dashboard

Keep track of your Broadway pipelines from Phoenix LiveDashboard
Elixir
218
star
12

mix_phx_gen_auth_demo

Example repository for mix phx.gen.auth
Elixir
211
star
13

broadway_rabbitmq

A Broadway producer for RabbitMQ
Elixir
192
star
14

broadway_sqs

A Broadway producer for Amazon SQS
Elixir
92
star
15

broadway_cloud_pub_sub

A Broadway producer for Google Cloud Pub/Sub
Elixir
70
star
16

nimble_ownership

Elixir
55
star
17

broadway_bike_sharing_rabbitmq_example

An example of a Broadway pipeline for a bike sharing app with RabbitMQ and PostgreSQL
Elixir
50
star
18

nimble_strftime

A simple and fast strftime-based datetime formatter
Elixir
37
star
19

unpickler

A library for loading data in the Python's pickle format
Elixir
14
star
20

broadway_website

Landing page for the Elixir Broadway library.
HTML
6
star