Similarity
Overview
A Ruby library for calculating the similarity between pieces of text using a Term Frequency-Inverse Document Frequency method.
A bag of words model is used. Terms in the source documents are downcased and punctuation is removed, but stemming is not currently implemented.
This library was written to facilitate the creation of diagrams talked
about by Jonathan Stray in his
full-text
visualization of the Iraq War Logs post. An example of how to
generate a Gephi compatible file including labelling of nodes with key
words is included in the examples
directory.
The library depends on the GNU Scientific Library, and the gsl ruby gem but does not use sparse matrix representations to speed up the calculations, since there is no support for them in the GSL. I am currently looking into fixing this, and would appreciate any help!
Dependencies
Similarity depends on the GNU Scientific Library, and the gsl ruby gem. On OSX with https://github.com/mxcl/homebrew the GSL can be installed with
brew install gsl
The gsl
gem should then install normally. For other platforms,
please add the information to the wiki and Iโll add them to this
readme.
Usage
First we load some documents into the corpus
require 'similarity'
:
corpus = Corpus.new
:
doc1 = Document.new(:content => "A document with a lot of additional words some of which are about chunky bacon") doc2 = Document.new(:content => "Another longer document with many words and again about chunky bacon") doc3 = Document.new(:content => "Some text that has nothing to do with pork products")
:
[doc1, doc2, doc3].each { |doc| corpus << doc }
Then to compare documents we can use the similar_documents
method
corpus.similar_documents(doc1).each do |doc, similarity| puts "Similarity between doc #{doc1.id} and doc #{doc.id} is #{similarity}" end
:
#=> Similarity between doc 70137042580340 and doc 70137042580340 is 0.9999999999999997 Similarity between doc 70137042580340 and doc 70137042580240 is 0.06068602112714361 Similarity between doc 70137042580340 and doc 70137042580160 is 0.04882114791611661
The cross-similarity matrix (useful for creating graphs) is also available
similarity_matrix = corpus.similarity_matrix
For more examples, see the examples
directory.
Todo
- Performance improvements
- Switch to storing document vector spaces in sparse form, using linalg or csparse?
- (Optional) stemming of source terms
Contributing
- Fork the project
- Send a pull request
- Donโt touch the .gemspec, Iโll do that when I release a new version
Author
Chris Lowis - BBC R&D