A pure ruby implemenation of the boilerpipe algorithm.
This is a text extraction utility first written by Christian Kohlshutter - presentation
I went directly to the original author's github https://github.com/kohlschutter/boilerpipe and forked that code base here https://github.com/gregors/boilerpipe.
I saw other gems making use of boilerpipe via the free api but depending on time of day the api goes down due to exceeding the hosting plan. I also checked out some gems making use of Jruby but I had all kinds of dependency and bug issues. So I made some tweaks on my fork and created a new jruby-boilerpipe gem.
This solution works great if you're using Jruby but I wanted a pure ruby solution to use on MRI. Open vim - start coding...
Here's a high level diagram of how the system works.
Just use either ArticleExtractor, DefaultExtractor or KeepEverythingExtractor - try out the others when you feel like experimenting...
Presently the follow Extractors are implemented
- ArticleExtractor
- ArticleSentenceExtractor
- CanolaExtractor
- DefaultExtractor
- KeepEverythingExtractor
- KeepEverythingWithMinKWordsExtractor
- LargestContentExtractor
- NumWordsRulesExtractor
Add this line to your application's Gemfile:
gem 'boilerpipe-ruby', require: 'boilerpipe'
And then execute:
$ bundle
Or install it yourself as:
$ gem install boilerpipe-ruby
gregors$ irb
> require 'boilerpipe'
=> true
> require 'open-uri'
=> true
> content = open('https://blog.carbonfive.com/2017/08/28/always-squash-and-rebase-your-git-commits/').read; true;
> Boilerpipe::Extractors::ArticleExtractor.text(content).slice(0..40)
=> "Always Squash and Rebase your Git Commits"
> Boilerpipe::Extractors::DefaultExtractor.text(content).slice(0..40)
=> "Posted on\nWhat is the squash rebase workf"
> Boilerpipe::Extractors::LargestContentExtractor.text(content).slice(0, 40)
=> "git push origin master\nWhy should you ad"
> Boilerpipe::Extractors::KeepEverythingExtractor.text(content).slice(0..40)
=> "Toggle Navigation\nCarbon Five\nAbout\nWork\n"
After checking out the repo, run bin/setup
to install dependencies. Then, run rake spec
to run the tests. You can also run bin/console
for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run bundle exec rake install
.
The default run command will run the tests
docker build -t boilerpipe .
docker run -it --rm boilerpipe
Bug reports and pull requests are welcome on GitHub at https://github.com/gregors/boilerpipe-ruby.