goru001/inltk

Natural Language Toolkit for Indic Languages (iNLTK)

iNLTK aims to provide out of the box support for various NLP tasks that an application developer might need for Indic languages. Paper for iNLTK library has been accepted at EMNLP-2020's NLP-OSS workshop. Here's the link to the paper

Documentation

Checkout detailed docs along with Installation instructions at https://inltk.readthedocs.io

Supported languages

Native languages

Code Mixed languages

Repositories containing models used in iNLTK

Note: English model has been directly taken from fast.ai

Effect of using Transfer Learning + Paraphrases from iNLTK

For more details around implementation or to reproduce results, checkout respective repositories.

Contributing

Add a new language support

If you would like to add support for language of your own choice to iNLTK, please start with checking/raising a issue here

Please checkout the steps I'd mentioned here for Telugu to begin with. They should be almost similar for other languages as well.

Improving models/using models for your own research

If you would like to take iNLTK's models and refine them with your own dataset or build your own custom models on top of it, please check out the repositories in the above table for the language of your choice. The repositories above contain links to datasets, pretrained models, classifiers and all of the code for that.

Add new functionality

If you wish for a particular functionality in iNLTK - Start by checking/raising a issue here

What's next

..and being worked upon

Shout out if you want to help :)

• Add Maithili support

..and NOT being worked upon

Shout out if you want to lead :)

• Add NER support for all languages
• Add Textual Entailment support for all languages
• Work on a unified model for all the languages
• POS support in iNLTK
• Add translations - to and from languages in iNLTK + English

iNLTK's Appreciation

• By Jeremy Howard on Twitter
• By Sebastian Ruder on Twitter
• By Vincent Boucher, By Philip Vollet, By Steve Nouri on LinkedIn
• By Kanimozhi, By Soham, By Imaad on LinkedIn
• iNLTK was trending on GitHub in May 2019

Citation

If you use this library in your research, please consider citing:

@inproceedings{arora-2020-inltk,
    title = "i{NLTK}: Natural Language Toolkit for Indic Languages",
    author = "Arora, Gaurav",
    booktitle = "Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.nlposs-1.10",
    doi = "10.18653/v1/2020.nlposs-1.10",
    pages = "66--71",
    abstract = "We present iNLTK, an open-source NLP library consisting of pre-trained language models and out-of-the-box support for Data Augmentation, Textual Similarity, Sentence Embeddings, Word Embeddings, Tokenization and Text Generation in 13 Indic Languages. By using pre-trained models from iNLTK for text classification on publicly available datasets, we significantly outperform previously reported results. On these datasets, we also show that by using pre-trained models and data augmentation from iNLTK, we can achieve more than 95{\%} of the previous best performance by using less than 10{\%} of the training data. iNLTK is already being widely used by the community and has 40,000+ downloads, 600+ stars and 100+ forks on GitHub. The library is available at https://github.com/goru001/inltk.",
}