Document QA
This repo contains code for our paper Simple and Effective Multi-Paragraph Reading Comprehension. It can be used to train neural question answering models in tensorflow, and in particular for the case when we want to run the model over multiple paragraphs for each question. Code is included to train on the TriviaQA and SQuAD datasets.
A demo of this work can be found at documentqa.allenai.org
Small forewarning, this is still much more of a research codebase then a library. we anticipate porting this work in allennlp where it will enjoy a cleaner implementation and more stable support.
Setup
Dependencies
We require python >= 3.5, tensorflow 1.3, and a handful of other supporting libraries. Tensorflow should be installed separately following the docs. To install the other dependencies use
pip install -r requirements.txt
The stopword corpus and punkt sentence tokenizer for nltk are needed and can be fetched with:
python -m nltk.downloader punkt stopwords
The easiest way to run this code is to use:
export PYTHONPATH=${PYTHONPATH}:`pwd`
Data
By default, we expect source data to be stored in "~/data" and preprocessed data to be stored in "./data". The expected file locations can be changed by altering config.py.
Word Vectors
The models we train use the common crawl 840 billion token GloVe word vectors from here. They are expected to exist in "~/data/glove/glove.840B.300d.txt" or "~/data/glove/glove.840B.300d.txt.gz".
For example:
mkdir -p ~/data
mkdir -p ~/data/glove
cd ~/data/glove
wget http://nlp.stanford.edu/data/glove.840B.300d.zip
unzip glove.840B.300d.zip
rm glove.840B.300d.zip
SQuAD Data
Training or testing on SQuAD requires downloading the SQuAD train/dev files into ~/data/squad. This can be done as follows:
mkdir -p ~/data/squad
cd ~/data/squad
wget https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json
wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json
then running:
python docqa/squad/build_squad_dataset.py
This builds pkl files of the tokenized data in "./data/squad"
TriviaQA Data
The raw TriviaQA data is expected to be unzipped in "~/data/triviaqa". Training or testing in the unfiltered setting requires the unfiltered data to be download to "~/data/triviaqa-unfiltered".
mkdir -p ~/data/triviaqa
cd ~/data/triviaqa
wget http://nlp.cs.washington.edu/triviaqa/data/triviaqa-rc.tar.gz
tar xf triviaqa-rc.tar.gz
rm triviaqa-rc.tar.gz
cd ~/data
wget http://nlp.cs.washington.edu/triviaqa/data/triviaqa-unfiltered.tar.gz
tar xf triviaqa-unfiltered.tar.gz
rm triviaqa-unfiltered.tar.gz
To use TriviaQA we need to tokenize the evidence documents, which can be done by
python docqa/triviaqa/evidence_corpus.py
This can be slow, we support multi-processing
python docqa/triviaqa/evidence_corpus.py --n_processes 8
This builds evidence files in "./data/triviaqa/evidence" that are split into paragraphs, sentences, and tokens. Then we need to tokenize the questions and locate the relevant answers spans in each document. Run
python docqa/triviaqa/build_span_corpus.py {web|wiki|open} --n_processes 8
to build the desired set. This builds pkl files "./data/triviaqa/{web|wiki|open}"
Training
Once the data is in place our models can be trained by
python docqa/scripts/ablate_{triviaqa|squad|triviaqa_wiki|triviaqa_unfiltered}.py
See the help menu for these scripts for more details. Note that since we use the Cudnn RNN implementations, these models can only be trained on a GPU. We do provide a script for converting the (trained) models to CPU versions:
python docqa/scripts/convert_to_cpu.py
Modifying the hyper-parameters beyond the ablations requires building your own train script.
Testing
SQuAD
Use "docqa/eval/squad_eval.py" to evaluate on paragraph-level (i.e., standard) SQuAD. For example:
python docqa/eval/squad_eval.py -o output.json -c dev /path/to/model/directory
"output.json" can be used with the official evaluation script, for example:
python docqa/squad/squad_official_evaluation.py ~/data/squad/dev-v1.1.json output.json
Use "docqa/eval/squad_full_document_eval.py" to evaluate on the document-level. For example
python docqa/eval/squad_full_document_eval.py -c dev /path/to/model/directory output.csv
This will store the per-paragraph results in output.csv, we can then run:
python docqa/eval/ranked_scores.py output.csv
to get ranked scores as more paragraphs are used.
TriviaQA
Use "docqa/eval/triviaqa_full_document_eval.py" to evaluate on TriviaQA datasets, like:
python docqa/eval/triviaqa_full_document_eval.py --n_processes 8 -c web-dev --tokens 800 -o question-output.json -p paragraph-output.csv /path/to/model/directory
Then the "question-output.json" can be used with the standard triviaqa evaluation script, the "paragraph-output.csv" contains per-paragraph output, we can run
python docqa/eval/ranked_scores.py paragraph-output.csv
to get ranked scores as more paragraphs as used for each question, or
python docqa/eval/ranked_scores.py --per_doc paragraph-output.csv
to get ranked scores as more paragraphs as used for each (question, document) pair, as should be done for TrivaQA web.
User Input
"docqa/scripts/run_on_user_documents.py" serves as a heavily commented example of how to run our models and pre-processing pipeline on other kinds of text. For example:
python docqa/scripts/run_on_user_documents.py /path/to/model/directory "Who wrote the satirical essay 'A Modest Proposal'?" ~/data/triviaqa/evidence/wikipedia/A_Modest_Proposal.txt ~/data/triviaqa/evidence/wikipedia/Jonathan_Swift.txt
Pre-Trained Models
We have four pre-trained models
-
"squad" Our model trained on the standard SQuAD dataset, this model is listed on the SQuAD leaderboard as BiDAF + Self Attention
-
"squad-shared-norm" Our model trained on document-level SQuAD using the shared-norm approach.
-
"triviaqa-web-shared-norm" Our model trained on TriviaQA web with the shared-norm approach. This is the model we used to submit scores to the TriviaQA leader board.
-
"triviaqa-unfiltered-shared-norm" Our model trained on TriviaQA unfiltered with the shared-norm approach. This is the model that powers our demo.
The models can be downloaded here
The models use the cuDNN implementation of GRUs by default, which means they can only be run on the GPU. We also have slower, but CPU compatible, versions here.