Improving Relation Extraction by Pre-trained Language Representations

This repository contains the code of our paper:
Improving Relation Extraction by Pre-trained Language Representations.
Christoph Alt*, Marc Hübner*, Leonhard Hennig

We fine-tune the pre-trained OpenAI GPT [1] to the task of relation extraction and show that it achieves state-of-the-art results on SemEval 2010 Task 8 and TACRED relation extraction datasets.

Our code depends on huggingface's PyTorch reimplementation of the OpenAI GPT [2] - so thanks to them.

Installation

First, clone the repository to your machine and install the requirements with the following command:

pip install -r requirements.txt

We also need the weights of the pre-trained Transformer, which can be downloaded with the following command:

./download-model.sh

The english spacy model is required for sentence segmentation:

python -m spacy download en

Prepare the data

We evaluate our model on SemEval 2010 Task 8 and TACRED, which is available through LDC.

Our model expects the input dataset to be in JSONL format. To convert a dataset run the following command:

python dataset_converter.py <DATASET DIR> <CONVERTED DATASET DIR> --dataset=<DATASET NAME>

Training

E.g. for training on the TACRED dataset, run the following command:

CUDA_VISIBLE_DEVICES=0 python relation_extraction.py train \
  --write-model True \
  --masking-mode grammar_and_ner \
  --batch-size 8 \
  --max-epochs 3 \
  --lm-coef 0.5 \
  --learning-rate 5.25e-5 \
  --learning-rate-warmup 0.002 \
  --clf-pdrop 0.1 \
  --attn-pdrop 0.1 \
  --word-pdrop 0.0 \
  --dataset tacred \
  --data-dir <CONVERTED DATASET DIR> \
  --seed=0 \
  --log-dir ./logs/

Evaluation

CUDA_VISIBLE_DEVICES=0 python relation_extraction.py evaluate \
  --dataset tacred \
  --masking_mode grammar_and_ner \
  --test_file ./data/tacred/test.jsonl \
  --save_dir ./logs/ \
  --model_file <MODEL FILE (e.g. model_epoch...)> \
  --batch_size 8 \
  --log_dir ./logs/

Trained Models

The models we trained on SemEval and TACRED to produce our paper results can be found here:

Download and extract model files

First, download the archive corresponding to the model you want to evaluate (links in the table above).

wget --content-disposition <DOWNLOAD URL>

Extract the model archive containing model.pt, text_encoder.pkl, and label_encoder.pkl.

tar -xvzf <MODEL ARCHIVE>

Run evaluation

• dataset: dataset to evaluate, can be one of "semeval" or "tacred".
• test-file: path to the JSONL test file used during evaluation
• log-dir: directory to store the evaluation results and predictions
• save-dir: directory containing the downloaded model files (model.pt, text_encoder.pkl, and label_encoder.pkl)
• masking-mode: masking mode to use during evaluation, can be one of "None", "grammar_and_ner", "grammar", "ner" or "unk" (caution: must match the mode for training)

For example, to evaluate the TACRED model with "grammar_and_ner" masking, run the following command:

CUDA_VISIBLE_DEVICES=0 python relation_extraction.py evaluate \
      --dataset tacred \
      --test-file ./<CONVERTED DATASET DIR>/test.jsonl \
      --log-dir <RESULTS DIR> \
      --save-dir <MODEL DIR> \
      --masking_mode grammar_and_ner

Citations

If you use our code in your research or find our repository useful, please consider citing our work.

@InProceedings{alt_improving_2019,
  author = {Alt, Christoph and H\"{u}bner, Marc and Hennig, Leonhard},
  title = {Improving Relation Extraction by Pre-trained Language Representations},
  booktitle = {Proceedings of AKBC 2019},
  year = {2019},
  url = {https://openreview.net/forum?id=BJgrxbqp67},
}

License

lm-transformer-re is released under the MIT license. See LICENSE for additional details.

References

1. Improving language understanding by generative pre-training. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.
2. PyTorch implementation of OpenAI's Finetuned Transformer Language Model