GkSeg: yet another Chinese word segmentation package
GkSeg is a Chinese word segmentation package shipped by Guokr.com. It is based on character-based tagging heuristics and CRF algorithm.
Currently it only support Linux platform.
Features
- Precise: > 94%
- Scope: modern Chinese text, and even classic Chinese text(文言文)
- Terms auto-extraction: It can extract important terms from the text
- No dictionaries: See the section for character-based tagging heuristics
- Performance is good: 4 times slower than mmseg, but we support more features
- Training tool for the CRF model is also shipped in the same package
Character-based tagging heuristics
Character-based tagging heuristics is invented by N. Xue and others, and published at SIGHAN 2002 [Xue et al., 2002]
The basic idea is to mark each character in a sentence with its kind:
- b: begining character of a word
- m: middle character of a word
- e: end character of a word
- s: single character to form a word
And then using the marked corpus to train the segmentation program.
At conceptual level, we can treat its ability for segmenting from the inner pattern of Chinese language.
Interestingly, when we use the tool to segment classic Chinese text, it achieved a good performance. That is to say, the inner pattern of Chinese language is not vary greatly during the time.
CRF algorithm
Conditional random fields ( from http://en.wikipedia.org/wiki/Conditional_random_field )
Conditional random fields (CRFs) are a class of statistical modelling method often applied in pattern recognition and machine learning, where they are used for structured prediction. Whereas an ordinary classifier predicts a label for a single sample without regard to "neighboring" samples, a CRF can take context into account; e.g., the linear chain CRF popular in natural language processing predicts sequences of labels for sequences of input samples.
We use wapiti package from LIMSI-CNRS, it is a very neat CRF package ( http://wapiti.limsi.fr/ )
We changed wapiti package a little by our requirements.
Installation
Please follow below steps:
git clone git://github.com/guokr/gkseg.git gkseg
cd gkseg/wapiti
make
Now it is ready, you can use the tools provided by this package directly.
Usage for the tools
All the tools located under the bin directory
gkseg: segment a text into words
- gkseg <text>
gksegd: start a webserver to segment words by restful api
- gksegd
gksegt: trainning the tool
- gksegt add <basedir> <aspect> <trainfile>
- gksegt train <trainfile> <modelfile>
Using the API
Before using the API, you should intialize the program first, and then perform the segmentation, and finally destroy the program.
import gkseg
text = '话说天下大势，分久必合，合久必分'.decode('utf-8')
gkseg.init()
print gkseg.seg(text) #segment the sentence into a list of words
print gkseg.term(text) #extract the important words from the sentence
print gkseg.label(text) #label the sentence
gkseg.destory()
The training process
Step 1: prepare the training input
- gksegt add <basedir> <aspect> <trainfile>
Here we have
- <basedir>: The base path of the training corpus
- <aspect>: A specified aspect of the training corpus, see below corpus section
- <trainfile>: The target training file
Step 2: training the input file to get the model
- gksegt train <trainfile> <modelfile>
Here we have
- <trainfile>: The training file as input
- <modelfile>: The model file as output
The format of training corpus
In logic, a corpus is a set of files organized in several aspect. And in physics, a training corpus must be organized into the following way:
- A top folder with an index.txt file, in the index file it gives all the aspects and filename list in the corpus.
- An aspect is a subfolder contains all the files.
You can check the example at https://github.com/guokr/corpus/tree/master/zhxs
The python module - gkcrp - in this package can be used to deal with this corpus format.
Just as showed in the demo at https://github.com/guokr/corpus/tree/master/zhxs , we have two aspect - original and labeled. in labeled folder, we give all the articles labeled by the mark "m" to hightlight the important keywords.
Contributors
- Mingli Yuan (mountain at github)
- Rui Wang (isnowfy at github)
License
- MIT license for the main part of the project
- wapiti is under its own license
- uthash is under BSD license