diasks2/pragmatic_tokenizer diasks2/pragmatic_tokenizer

  • Stars
    star
    90
  • Rank 369,088 (Top 8 %)
  • Language
    Ruby
  • License
    MIT License
  • Created almost 9 years ago
  • Updated 3 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
diasks2/pragmatic_tokenizer
github

diasks2

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A multilingual tokenizer to split a string into tokens

More Repositories

1

ruby-nlp

A collection of links to Ruby Natural Language Processing (NLP) libraries, tools and software
1,269
star
2

pragmatic_segmenter

Pragmatic Segmenter is a rule-based sentence boundary detection gem that works out-of-the-box across many languages.
Ruby
546
star
3

chat_correct

A Ruby gem that shows the errors and error types when a correct English sentence is diffed with an incorrect English sentence.
Ruby
43
star
4

heroku-buildpack-mecab

This is a buildpack that enables using the mecab gem on Heroku Cedar.
Ruby
25
star
5

word_count_analyzer

Word Count Analyzer is a Ruby gem that analyzes a string for potential areas of the text that might cause word count discrepancies depending on the tool used. It also provides comprehensive configuration options so you can easily customize how different gray areas should be counted and find the right word count for your purposes.
Ruby
20
star
6

confidential_info_redactor

Ruby gem to semi-automatically redact confidential information from a text
Ruby
14
star
7

surveyor_example

extended NUBIC/surveyor example (tied to a user model)
Ruby
7
star
8

confidential_info_redactor_lite

The lite version of https://github.com/diasks2/confidential_info_redactor - include your own language packs
Ruby
3
star
9

amcharts_example

Ruby on Rails tutorial describing how to link an amCharts JavaScript chart to the data in your database
Ruby
3
star
10

proz

ProZ is a Ruby wrapper for the ProZ.com API
Ruby
2
star
11

sdltm_importer

Import the content of a .sdltm translation memory file
Ruby
2
star
12

unicode_case_converter

A pure Ruby implementation to upcase and downcase unicode strings
Ruby
2
star
13

pretty_strings

Take strings that have been abused in the wild and clean them up (for translation tools)
Ruby
2
star
14

xlf_importer

XLIFF / XLF file importer
Ruby
1
star
15

tbx_importer

TBX (TermBase eXchange) file importer
Ruby
1
star
16

tmx_importer

TMX translation memory file importer
Ruby
1
star
17

jvn_segmenter

jRuby bindings for JVnSegmenter: A Java-based Vietnamese Word Segmentation Tool
Ruby
1
star
18

era_835_parser

Electronic Remittance Advice (ERA) 835 parser
Ruby
1
star
19

finance

Personal Finance Tracker
Ruby
1
star
20

scheduler

Hospital Shift Scheduler
Ruby
1
star
21

txt_tm_importer

Import the content of a .txt translation memory file
Ruby
1
star