A gem for creating workflows that "enhance" entities with extra attributes. At a high level, attr-gather provides a process to fetch information from many data sources (such as third party APIs, legacy databases, etc.) in a fully parallelized fashion.
# define a workflow
class EnhanceProfile
include Attr::Gather::Workflow
# contains all the task implementations
container TasksContainer
# filter out invalid data using a Dry::Validation::Contract
# anything that doesn't match this schema will be filtered out
filter_with_contract do
params do
required(:user_id).filled(:integer)
optional(:user).hash do
optional(:name).filled(:string)
optional(:email).filled(:string)
optional(:gravatar).filled(:string)
optional(:email_info).hash do
optional(:deliverable).filled(:bool?)
optional(:free).filled(:bool?)
end
end
end
end
# each task returns a hash of data that will be merged into the result
task :fetch_post do |t|
t.depends_on = []
end
# will run in parallel
task :fetch_user do |t|
t.depends_on = [:fetch_post]
end
# will run in parallel
task :fetch_email_info do |t|
t.depends_on = [:fetch_user]
end
endclass PostFetcher
def call(attrs)
res = HTTP.get("https://jsonplaceholder.typicode.com/posts/#{attrs[:id]}")
post = JSON.parse(res.to_s, symbolize_names: true)
{ title: post[:title], user_id: post[:userId], body: post[:body] }
end
endclass UserFetcher
# will have access to the PostFetcher attributes here
def call(attrs)
res = HTTP.get("https://jsonplaceholder.typicode.com/users/#{attrs[:user_id]}")
user = JSON.parse(res.to_s, symbolize_names: true)
{ user: { name: user[:name], email: user[:email] } }
end
endclass EmailInfoFetcher
# will have access to the PostFetcher attributes here
def call(user:)
res = HTTP.timeout(3).get("https://api.trumail.io/v2/lookups/json?email=#{user[:email]}")
info = JSON.parse(res.to_s, symbolize_names: true)
# will deep merge with the final result
{ user: { email_info: { deliverable: info[:deliverable], free: info[:free] } } }
end
endclass MyContainer
extend Dry::Container::Mixin
register :fetch_post, PostFetcher
register :fetch_user, UserFetcher
register :fetch_email_info, EmailInfoFetcher
endenhancer = EnhanceUserProfile.new
enhancer.call(id: 12).value!And this is the result...
{
:id => 12,
:user_id => 2,
:user => {
:email => "[email protected]",
:name => "Ervin Howell",
:email_info => { :deliverable => true, :free => true },
:gravatar => "https://www.gravatar.com/avatar/241af7d19a0a7438794aef21e4e19b79"
}
}You can even preview it as an SVG!
enhancer.to_dot(preview: true) # requires graphviz (brew install graphviz)- Offers DSL for defining workflows and merging the results from each task
- Execution engine optimally parallelizes the execution of the workflow dependency graph using concurrent-ruby Promises
- Very easy to unit test
- Ability to filter out bad/junky data using dry-validation contracts
- Operates on a single entity rather than a list, so easily adoptable in existing systems
- Focuses on the "fetching" and filtering of data solely, and not transformation or storage
- Focuses on having a clean PORO interface to make testing simple
- Provides a declarative interface for merging results from many sources (APIs, legacy databases, etc.) which allows for prioritization
 |
|---|
| Example of workflow that enhances a blog post |
Add this line to your application's Gemfile:
gem 'attr-gather'And then execute:
$ bundle
Or install it yourself as:
$ gem install attr-gather
After checking out the repo, run bin/setup to install dependencies. Then, run
rake spec to run the tests. You can also run bin/console for an interactive
prompt that will allow you to experiment.
To install this gem onto your local machine, run bundle exec rake install. To
release a new version, update the version number in version.rb, and then run
bundle exec rake release, which will create a git tag for the version, push
git commits and tags, and push the .gem file to
rubygems.org.
Bug reports and pull requests are welcome on GitHub at https://github.com/ianks/attr-gather. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.
The gem is available as open source under the terms of the MIT License.
Everyone interacting in the Attr::Gather project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.