This is an archival repository for Java BookNLP (ca. 2014); for the current Python version, see the booknlp repo.
BookNLP is a natural language processing pipeline that scales to books and other long documents (in English), including:
- Part-of-speech tagging (Stanford)
- Dependency parsing (MaltParser)
- Named entity recognition (Stanford)
- Character name clustering (e.g., "Tom", "Tom Sawyer", "Mr. Sawyer", "Thomas Sawyer" -> TOM_SAWYER)
- Quotation speaker identification
- Pronominal coreference resolution
- Supersense tagging (e.g., "animal", "artifact", "body", "cognition", etc.)
This pipeline is described in the following paper; please cite if you write a research paper using this software:
David Bamman, Ted Underwood and Noah Smith, "A Bayesian Mixed Effects Model of Literary Character," ACL 2014.
How To Run
Preliminaries
Download external jars (which are sadly too big for GitHub's 100MB file size limit)
- Download and unzip http://nlp.stanford.edu/software/stanford-corenlp-4.1.0.zip
- copy stanford-corenlp-4.1.0/stanford-corenlp-4.1.0-models.jar to the lib/ folder in the current working directory
Example
From the command line, run the following:
./runjava novels/BookNLP -doc data/originalTexts/dickens.oliver.pg730.txt -printHTML -p data/output/dickens -tok data/tokens/dickens.oliver.tokens -f
(On a 2.6 GHz MBP, this takes about 3.5 minutes)
This runs the bookNLP pipeline on "Oliver Twist" in the data/originalTexts directory and writes the processed document to data/tokens/dickens.oliver.tokens, along with diagnostic info to data/output/dickens. To run on your own texts, change the following:
- data/originalTexts/dickens.oliver.pg730.txt -> the path to the input book you want to process.
- data/tokens/dickens.oliver.tokens -> the path to the file where you want the processed text to be stored.
- data/output/dickens -> the path to the output directory you want to write any other diagnostics to.
Flags
Required
-doc : original text to process
-tok : file path to save processed tokens to (or read them from, if it already exists)
-p : the directory to write all diagnostic files to. Creates the directory if it does not already exist.
Optional
-id : a unique book ID for this book (output files include this in the filename)
-printHTML : also print the text as an HTML file with character aliases, coref and speaker ID annotated
-f : force the (slower) syntactic processing of the original text file, even if the in the -tok flag exists (if the -tok exists, the process that would parse the original text to create it is skipped)
Output
The main output here is data/tokens/dickens.oliver.tokens, which contains the original book, one token per line, with part of speech, syntax, NER, coreference and other annotations. The (tab-separated) format is:
- Paragraph id
- Sentence id
- Token id
- Byte start
- Byte end
- Whitespace following the token (useful for pretty-printing the original text)
- Syntactic head id (-1 for the sentence root)
- Original token
- Normalized token (for quotes etc.)
- Lemma
- Penn Treebank POS tag
- NER tag (PERSON, NUMBER, DATE, DURATION, MISC, TIME, LOCATION, ORDINAL, MONEY, ORGANIZATION, SET, O)
- Stanford basic dependency label
- Quotation label (begin quote, inside quote, outside quote)
- Character id (all coreferent tokens share the same character id)
- Supersense tag (https://wordnet.princeton.edu/documentation/lexnames5wn)
The data/output/dickens folder will now contain:
- dickens.oliver.twist.html (described above)
- dickens.oliver.twist.book (a representation of all of the characters' features, in JSON)
Modifying the code
With apache ant installed, running ant compiles everything.
Training coreference
Coreference only needs to be trained when there's new training data (or new feature ideas: current features are based on syntactic tree distance, linear distance, POS identity, gender matching, quotation scope and salience).
Data
Coreference annotated data is located in the coref/ directory.
annotatedData.txt contains coreference annotations, in the (tab-separated) format:
- book ID
- anaphor token ID
- antecendent token ID
bookIDs are mapped to their respective token files in docPaths.txt. All of these token files are located in finalTokenData/. These tokens files are all read-only -- since the annotations are keyed to specific token IDs in those files, we want to make sure they stay permanent.
Training a model
Given the coref/ folder above, train new coreference weights with:
./runjava novels.training/TrainCoref -training coref/annotatedData.txt -o coref/weights.txt
-training specifies the input training file
-o specifies the output file to write the trained weights to
Two parameters control the amount of regularization in the model (higher regularization dampens the impact of any single feature, and L1 regularization removes features from the model; both help prevent overfitting to training data.)
-l1 specifies the L1 regularization parameter (higher = more weights end up driven to 0). Default = 2
-l2 specifies the L2 regularization parameter (higher = weights shrink faster). Default = .1
To use the newly trained weights in the pipeline above, copy them to files/coref.weights or specify them on the novels.BookNLP command line with the -w flag.