AlasdairF/Tokenize

Stars
13
Rank 1,512,713 (Top 30 %)
Language
Go
Created over 10 years ago
Updated almost 9 years ago

AlasdairF/Tokenize

AlasdairF

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

All-in-one text tokenizer for Go. Super-fast. Lots of features.

Classifier

Text classifier for Go, aka document categorization.

BinSearch

Super-efficient, in-memory key/value data store

Sort

Faster than builtin sorting using primitives.

NormalizeText

Normalizes casing, spacing & punctuation in a paragraph of text

Titlecase

Unicode, multi-language, production-quality capitalization.

Custom

Super-efficient reading, writing, buffering

Hash

Fastest 32-bit & 64-bit hashing functions.

FileMime

Returns mime type for any filename or extension.

Deaccent

Normalizes and removes accents from UTF8 text

Conv

More efficient replacement for strconv on ints only, inc. padding feature.

Execute

Helper functions extending os/exec

Group

Structure for IDs that need to be grouped together

Keywords

Multi-language text keyword extraction, inc. stopword removal.

File

Wrapper for some simple OS functions: file exists, filesize, etc.

ReadAll

Replacement for ioutil.ReadAll for when you know the length.

Random

Fastest random string generator

Unleak

Easy functions for copying slices to avoid memory leaks

Daily

A crude but efficient daily function executer

Post

POST data to a URL

Binary

Simple binary encoding and decoding of unsigned integers.

Stemmer

Snowball string stemmer for Go in multiple languages.

Average

mean, median, mode, min, max, standard deviation, etc.

HTTPWrite

Wraps http.ResponseWriter for more efficient writing

Runes

Port of Go package 'bytes' to runes