• Stars
    star
    161
  • Rank 229,411 (Top 5 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created over 6 years ago
  • Updated over 5 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Lightning Fast Language Prediction 🚀

whatthelang

Build Status PyPI

Lightning Fast Language Prediction 🚀

Dependencies

The dependencies can be installed using the requirements.txt file:

$ pip install -r requirements.txt

Install

$ pip install whatthelang

Basic Usage

Predicting Language using whatthelang

>>> from whatthelang import WhatTheLang
>>> wtl = WhatTheLang()
>>> wtl.predict_lang("Mother")
'en'
>>> wtl.predict_lang("தாய்")
'ta'
>>> wtl.predict_lang("അമ്മ")
'ml'
>>> wtl.predict_lang("पिता")
'hi'

Batch Prediction is also supported

>>>wtl.predict_lang(["അമ്മ","पिता","teacher"])
['ml','hi','en']

Supported Languages

Supports 176 languages . The ISO codes for the corresponding languages are as below.

af als am an ar arz as ast av az azb ba bar bcl be bg bh bn bo bpy br bs bxr ca cbk
ce ceb ckb co cs cv cy da de diq dsb dty dv el eml en eo es et eu fa fi fr frr fy ga
gd gl gn gom gu gv he hi hif hr hsb ht hu hy ia id ie ilo io is it ja jbo jv ka kk km
kn ko krc ku kv kw ky la lb lez li lmo lo lrc lt lv mai mg mhr min mk ml mn mr mrj ms
mt mwl my myv mzn nah nap nds ne new nl nn no oc or os pa pam pfl pl pms pnb ps pt qu
rm ro ru rue sa sah sc scn sco sd sh si sk sl so sq sr su sv sw ta te tg th tk tl tr
tt tyv ug uk ur uz vec vep vi vls vo wa war wuu xal xmf yi yo yue zh

Model Training Details

Quantized model built using Fasttext. More details present in the fasttext blog

Reference

whatthelang is powered by FastText

Enriching Word Vectors with Subword Information

[1] P. Bojanowski*, E. Grave*, A. Joulin, T. Mikolov, Enriching Word Vectors with Subword Information

@article{bojanowski2016enriching,
  title={Enriching Word Vectors with Subword Information},
  author={Bojanowski, Piotr and Grave, Edouard and Joulin, Armand and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1607.04606},
  year={2016}
}

Bag of Tricks for Efficient Text Classification

[2] A. Joulin, E. Grave, P. Bojanowski, T. Mikolov, Bag of Tricks for Efficient Text Classification

@article{joulin2016bag,
  title={Bag of Tricks for Efficient Text Classification},
  author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1607.01759},
  year={2016}
}

FastText.zip: Compressing text classification models

[3] A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, T. Mikolov, FastText.zip: Compressing text classification models

@article{joulin2016fasttext,
  title={FastText.zip: Compressing text classification models},
  author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Douze, Matthijs and J{\'e}gou, H{\'e}rve and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1612.03651},
  year={2016}
}

More Repositories

1

aws-maintenance-lambda

A lambda function to send alerts (to Slack, HipChat) on AWS maintenance events.
JavaScript
133
star
2

schemer

Schema registry for CSV, TSV, JSON, AVRO and Parquet schema. Supports schema inference and GraphQL API.
Scala
112
star
3

web-auto-extractor

Automatically extracts structured information from webpages
JavaScript
108
star
4

matsya

Place ASGs on the right Spot Market
Scala
39
star
5

gocd-s3-artifacts

Set of GoCD plugins to publish and fetch artifacts from Amazon S3
Java
36
star
6

formland

A simple, super-flexible, extensible config based form generator for React.
TypeScript
33
star
7

mlflow-gocd

GoCD plugins to work with MLFlow as model repository in a CD flow
Java
29
star
8

sparkplug

Spark package to "plug" holes in data using SQL based rules ⚡️ 🔌
Scala
28
star
9

css-optimum-selector

Helps to extract shortest optimal css-selector and multi-selector.
CSS
26
star
10

gocd-mesos

Autoscale GOCD agents on top of a mesos cluster
Scala
16
star
11

kafkajs-lz4

🗜 LZ4 compression codec for KafkaJS
TypeScript
14
star
12

javascript-easy-object

Now easily access or modify an object in javascript with javascript-easy-object.
JavaScript
13
star
13

vasuki

Scale GoCD Agents on demand with Docker
Go
13
star
14

rocks

RocksDB Ops CLI
Go
11
star
15

indix-radar

Indix Tech Radar
JavaScript
10
star
16

vamana

Autoscaling toolkit based on custom Application Metrics
Scala
9
star
17

terraform-aws-maintenance-lambda

Terraform module to deploy aws-maintenance-lambda - A lambda function to send alerts (to Slack, HipChat) on AWS maintenance events.
HCL
7
star
18

indix.github.io

Indix Open Source website
HTML
6
star
19

utils

Scala utils for anything and everything
Scala
5
star
20

indix-api-java

Indix API Java client
Java
4
star
21

indix-api-nodejs

Indix API NodeJS Client
JavaScript
4
star
22

bubblewrap

Asynchronous crawler utils
HTML
3
star
23

ml2npy

Export spark ml SparseVectors as numpy csr matrix
Scala
3
star
24

marathon-logger

Push marathon app logs to local syslog daemon
Go
2
star
25

indix-api-ruby

Ruby client for indix API
Ruby
2
star
26

crawler4j

crawler4j fork from Google code
Java
2
star
27

the-vision

Reusable react components
HTML
2
star
28

auto-tag-s3-bucket

Automatically tag S3 buckets with tags from a Google Spreadsheet
Python
2
star
29

openvpn-ops

This repo can be used to create a openvpn server.
Shell
1
star
30

abelwatch

Alerting tool on top of Abel
Go
1
star
31

hadoop-as-publisher

Hadoop Autoscaling Metric Publisher
Python
1
star
32

ansible-ruby

Ansible role to install rvm and ruby
1
star
33

ansible-telegraf

Ansible role to install telegraf
1
star
34

mod_evasive

git mirror of mod_evasive apache module from http://www.zdziarski.com/blog/wp-content/uploads/2010/02/mod_evasive_1.10.1.tar.gz
C
1
star
35

abel

Business stats collection/aggregation
Scala
1
star
36

ansible-monit

Ansible role for monit
1
star
37

rocksdb-io

hadoop formats, cascading tap and scalding sources for RocksDB
Scala
1
star
38

Mobile-Product-Search

This app is a representation of how Indix API can be used to leverage information on stores, brands, products which forms the skeleton of Retail Industry.
Objective-C
1
star