UKPLab/elmo-bilstm-cnn-crf UKPLab/elmo-bilstm-cnn-crf

  • Stars
    star
    388
  • Rank 110,734 (Top 3 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created over 6 years ago
  • Updated about 2 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
UKPLab/elmo-bilstm-cnn-crf
github

UKPLab

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

BiLSTM-CNN-CRF architecture for sequence tagging using ELMo representations.

BiLSTM-CNN-CRF with ELMo-Representations for Sequence Tagging

This repository is an extension of my BiLSTM-CNN-CRF implementation.

It integrates the ELMo representations from the publication Deep contextualized word representations (Peters et al., 2018) into the BiLSTM-CNN-CRF architecture and can improve the performance significantly for different sequence tagging tasks.

The system is easy to use, optimized for high performance, and highly configurable.

Requirements:

  • Python 3.6 - lower versions of Python do not work
  • AllenNLP 0.5.1 - to compute the ELMo representations
  • Keras 2.2.0 - For the creation of BiLSTM-CNN-CRF architecture
  • Tensorflow 1.8.0 - As backend for Keras (other backends are untested.

Note: This implementation might be incompatible with different (e.g. more recent) versions of the frameworks. See docker/requirements.txt for a full list of all Python package requirements.

Documentation

For an IPython Notebook with a simple example how to use ELMo representations for sentence classification, see: Keras_ELMo_Tutorial.ipynb.

This code is an extension of the emnlp2017-bilstm-cnn-crf implementation. Most examples can be used with only slight adaptation. Also please see that repository for an explanation about the definition of the datasets, the configuration of the hyperparameters, how to use it for multi-task learning, or how to create custom features.

Most aspects from emnlp2017-bilstm-cnn-crf work the same in this implementation.

Citation

This repository contains experimental software and is under active development. If you find the implementation useful, please cite the following paper: Alternative Weighting Schemes for ELMo Embedding

@article{Reimers:2019,
  author    = {Reimers, Nils, and Gurevych, Iryna},
  title     = {{Alternative Weighting Schemes for ELMo Embeddings}},
  journal   = {CoRR},
  volume    = {abs/1904.02954},
  year      = {2019},
  url       = {https://arxiv.org/abs/1904.02954}
}

Contact person: Nils Reimers, [email protected]

https://www.ukp.tu-darmstadt.de/ https://www.tu-darmstadt.de/

Don't hesitate to send us an e-mail or report an issue, if something is broken (and it shouldn't be) or if you have further questions.

First two layer of the ELMo Model

In my publication Alternative Weighting Schemes for ELMo Embedding, I show that it is often sufficient to use only the first to layers of ELMo. The third layers led for various tasks to no significant improvement. Reducing the ELMo model from three to two layers increases the training speed up to 50%.

You can download the reduced, pre-trained models from here:

This reduced ELMo model is also compatible with the models from AllenNLP, just replace the options_file / weight_file in your config with the provided URLs.

Setup

In order to run the code, Python 3.6 or higher is required. The code is based on Keras 2.2.0 and as backend I recommend Tensorflow 1.8.0. I cannot ensure that the code works with different versions for Keras / Tensorflow or with different backends for Keras.

Installation using conda / virtualenv

To get the ELMo representations, AllenNLP is required. The AllenNLP installation instructions describe a nice way how to setup a virtual enviromnent with the correct Python version.

Conda can be used set up a virtual environment with the version of Python required (3.6).

  1. Download and install Conda.

  2. Create a Conda environment with Python 3.6

    conda create -n elmobilstm python=3.6

  3. Activate the Conda environment. You will need to activate the Conda environment in each terminal in which you want to this code.

    source activate elmobilstm

Installing the dependencies with pip

You can use pip to install the dependencies.

pip install allennlp==0.5.1 tensorflow==1.8.0 Keras==2.2.0

In docker/requirements.txt) you find a full list of all used packages. You can install it via:

pip install -r docker/requirements.txt

Installation using docker

The docker-folder contains an example how to create a Docker image that contains all required dependencies. It can be used to run your code within that container. See the docker-folder for more details.

Test the installation

If the installation was successful, you can test the code by running:

python Train_Chunking.py

This trains the ELMo-BiLSTM-CRF architecture on the CoNLL 2000 chunking dataset.

Training

See Train_Chunking.py for an example how to train and evaluate this implementation. The code assumes a CoNLL formatted dataset like the CoNLL 2000 dataset for chunking.

For training, you specify the datasets you want to train on:

datasets = {
    'conll2000_chunking':                                   #Name of the dataset
        {'columns': {0:'tokens', 1:'POS', 2:'chunk_BIO'},   #CoNLL format for the input data. Column 0 contains tokens, column 1 contains POS and column 2 contains chunk information using BIO encoding
         'label': 'chunk_BIO',                              #Which column we like to predict
         'evaluate': True,                                  #Should we evaluate on this task? Set true always for single task setups
         'commentSymbol': None}                             #Lines in the input data starting with this string will be skipped. Can be used to skip comments
}

For more details, see the emnlp2017-bilstm-cnn-crf implementation.

ELMo Word Representations Computation

The ELMoWordEmbeddings-class provides methods for the efficient computation of ELMo representations. It has the following parameters:

The ELMoWordEmbeddings provides methods for the efficient computation of ELMo representations. It has the following parameters:

  • embeddings_file: The ELMo paper concatenates traditional word embeddings, like GloVe, with the context dependent embeddings. With embeddings_file you can pass a path to a pre-trained word embeddings file. You can set it to none if you don't want to use traditional word embeddings.
  • elmo_options_file and elmo_weight_file: AllenNLP provides different pretrained ELMo models.
  • elmo_mode: Set to average if you want all 3 layers to be averaged. Set to last if you want to use only the final layer of the ELMo language model.
  • elmo_cuda_device: Can be set to the ID of the GPU which should compute the ELMo embeddings. Set to -1 to run ELMo on the CPU. Using a GPU drastically improves the computational time.

Caching of ELMo representations

The computation of ELMo representations is computationally expensive. A CNN is used to map the characters of a token to a dense vectors. These dense vectors are then fed through two BiLSTMs. The representation of each token and the two outputs of the BiLSTMs are used to form the final context-dependent word embedding.

In order speed-up the training, the context dependent word embeddings can be cached. Then, those embeddings must only be computed for the first epoch. For consecutive epochs, the embeddings are used from the cache.

To enable the caching, you must set embLookup.cache_computed_elmo_embeddings to True:

embLookup = ELMoWordEmbeddings(embeddings_file, elmo_options_file, elmo_weight_file, elmo_mode, elmo_cuda_device)
#...
embLookup.cache_computed_elmo_embeddings = True

This method requires about 12 KB memory per token. For large datasets, you will need a few gigabyte of RAM.

Pre-compute ELMo embeddings once

The ELMoWordEmbeddings class implements a caching mechanism for a quick lookup of sentences => context dependent word representations for all tokens in the sentence.

You can run Create_ELMo_Cache.py to iterate through all you sentences in your dataset and create the ELMo embeddings for those. It stores these embeddings in the file embeddings/elmo_cache_[DatasetName].pkl.

Once you create such a cache, you can load those in your experiments:

embLookup = ELMoWordEmbeddings(embeddings_file, elmo_options_file, elmo_weight_file, elmo_mode, elmo_cuda_device)
embLookup.loadCache('embeddings/elmo_cache_conll2000_chunking.pkl')

If a sentence is in the cache, the cached representations for all tokens in that sentence are used. This requires the computation of the ELMo embeddings for a dataset must only be done once.

Note: The cache file can become rather large, as 3*1024 float numbers per token must be stored. The cache file requires about 3.7 GB for the CoNLL 2000 dataset on chunking with about 13.000 sentences.

Issues, Feedback, Future Development

This repository is under active development as I'm currently running several experiments that involve ELMo embeddings.

If you have questions, feedback or find bugs, please send an email to me: [email protected]

More Repositories

1

sentence-transformers

State-of-the-Art Text Embeddings
Python
15,213
star
2

EasyNMT

Easy to use, state-of-the-art Neural Machine Translation for 100+ languages
Python
1,164
star
3

emnlp2017-bilstm-cnn-crf

BiLSTM-CNN-CRF architecture for sequence tagging
Python
823
star
4

deeplearning4nlp-tutorial

Hands-on tutorial on deep learning with a special focus on Natural Language Processing (NLP)
Python
634
star
5

gpl

Powerful unsupervised domain adaptation method for dense retrieval. Requires only unlabeled corpus and yields massive improvement: "GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval" https://arxiv.org/abs/2112.07577
Python
322
star
6

emnlp2017-relation-extraction

Context-Aware Representations for Knowledge Base Relation Extraction
Python
287
star
7

arxiv2018-xling-sentence-embeddings

Concatenated Power Mean Embeddings as Universal Cross-Lingual Sentence Representations
JavaScript
185
star
8

coling2018-graph-neural-networks-question-answering

Accompanying code for our COLING 2018 paper "Modeling Semantics with Gated Graph Neural Networks for Knowledge Base Question Answering"
Python
173
star
9

plms-graph2text

Investigating Pretrained Language Models for Graph-to-Text Generation
Python
143
star
10

MMT-Retrieval

Python
129
star
11

kg2text

Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs (authors' implementation for the TACL20 paper)
Python
94
star
12

acl2019-BERT-argument-classification-and-clustering

Python
83
star
13

argument-reasoning-comprehension-task

The Argument Reasoning Comprehension Task: Source codes & Datasets
Java
71
star
14

pytorch-bertflow

Pytorch-version BERT-flow: One can apply BERT-flow to any PLM within Pytorch framework.
Python
69
star
15

acl2017-non-factoid-qa

Code for paper "End-to-End Non-Factoid Question Answering with an Interactive Visualization of Neural Attention Weights"
Python
67
star
16

acl2017-neural_end2end_am

Accompanying code for our ACL-2017 publication on Neural End-to-End Learning for Computational Argumentation Mining
Python
60
star
17

starsem2018-entity-linking

Accompanying code for our *SEM 2018 @ NAACL 2018 paper "Mixing Context Granularities for Improved Entity Linking on Question Answering Data across Entity Categories"
Python
59
star
18

fever-2018-team-athene

Python
46
star
19

nessie

Automatically detect errors in annotated corpora.
Python
46
star
20

mdl-stance-robustness

Multi-dataset stance detection and robustness experiments
Python
42
star
21

naacl18-multitask_argument_mining

Code for the paper "Multi-Task Learning for Argumentation Mining in Low-Resource Settings"
Python
40
star
22

semeval2017-scienceie

Code for keyphrase classification systems submitted to the SemEval 2017 shared task ScienceIE.
Python
36
star
23

starsem18-multimodalKB

Python
35
star
24

on-emergence

Codes and files for the paper Are Emergent Abilities in Large Language Models just In-Context Learning
Python
34
star
25

acl2020-interactive-entity-linking

Python
33
star
26

useb

Heterogenous, Task- and Domain-Specific Benchmark for Unsupervised Sentence Embeddings used in the TSDAE paper: https://arxiv.org/abs/2104.06979.
Python
32
star
27

5pils

Code associated with the EMNLP 2024 Main paper: "Image, tell me your story!" Predicting the original meta-context of visual misinformation.
Python
31
star
28

emnlp2017-graphdocexplore

Accompanying code for our EMNLP 2017 publication "GraphDocExplore: A Framework for the Experimental Comparison of Graph-based Document Exploration Techniques"
JavaScript
29
star
29

StructAdapt

Structural Adapters in Pretrained Language Models for AMR-to-Text Generation (EMNLP 2021)
Python
29
star
30

coling2018_fake-news-challenge

Python
28
star
31

iwcs2017-answer-selection

Repository for the IWCS 2017 paper "Representation Learning for Answer Selection with LSTM-Based Importance Weighting"
Python
28
star
32

controlled-argument-generation

Controlling Argument Generation via topic, stance, and aspect
Python
28
star
33

acl2016-convincing-arguments

Code and data for ACL2016 article "Which argument is more convincing? Analyzing and predicting convincingness of Web arguments using bidirectional LSTM" by Ivan Habernal and Iryna Gurevych"
Java
28
star
34

lagonn

Source code and data for Like a Good Nearest Neighbor
Python
28
star
35

arxiv2018-bayesian-ensembles

Python
26
star
36

naacl2019-like-humans-visual-attacks

Python
26
star
37

refresh2018-predicting-trends-from-arxiv

Python
26
star
38

emnlp2018-activation-functions

Shell
26
star
39

emnlp2020-debiasing-unknown

Python
25
star
40

naacl2019-does-my-rebuttal-matter

Ruby
25
star
41

acl2024-dapr

Python
25
star
42

acl2017-interactive_summarizer

A general framework for Interactive Multi-Document Summarization
Python
24
star
43

adaptable-adapters

Python
23
star
44

e2e-nlg-challenge-2017

E2E NLG Challenge submission
Python
23
star
45

acl2020-confidence-regularization

Python
23
star
46

linspector

Python
23
star
47

MetaQA

MetaQA: Combining Expert Agents for Multi-Skill Question Answering
Python
22
star
48

emnlp2019-dualgraph

Enhancing AMR-to-Text Generation with Dual Graph Representations (implementation for the EMNLP-IJCNLP-2019 paper)
Python
22
star
49

arxiv2024-divergent-cot

Code for the 2024 arXiv publication "Fine-Tuning with Divergent Chains of Thought Boosts Reasoning Through Self-Correction in Language Models"
Python
21
star
50

iclr2024-model-merging

This is the repository for "Model Merging by Uncertainty-Based Gradient Matching", ICLR 2024.
Python
20
star
51

aaai2019-coala-cqa-answer-selection

Python
20
star
52

tacl2017-event-time-extraction

Event Time Extraction with a Decision Tree of Neural Classifiers
Python
19
star
53

tac2015-event-detection

Files for Event Nugget Detection systems submitted to TAC 2015 shared task on Event Nugget Detection
Java
19
star
54

coling2018-xling_argument_mining

Erlang
16
star
55

CARE

Project CARE
Vue
16
star
56

llm-roleplay

LLM Roleplay: Simulating Human-Chatbot Interaction
Python
16
star
57

eacl2017-oodFrameNetSRL

Implementation of a simple frame identification approach (SimpleFrameId) described in the paper "Out-of-domain FrameNet Semantic Role Labeling"
Python
15
star
58

SciGen

Python
15
star
59

acl2020-dialogue-coherence-assessment

Python
15
star
60

TWEAC-qa-agent-selection

Python
14
star
61

emnlp2020-multicqa

MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale
Python
14
star
62

emnlp2024-code-prompting

Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs. EMNLP 2024
Python
14
star
63

acl2024-triple-encoders

triple-encoders is a library for contextualizing distributed Sentence Transformers representations.
Python
14
star
64

lrec2018-live-blog-corpus

Python
13
star
65

EACL21-personalized-conversational-system

Python
13
star
66

emnlp2017-claim-identification

Source code repository for our EMNLP paper on cross-domain claim identification
Java
13
star
67

emnlp2018-question-answering-interface

Accompanying code for our EMNLP 2018 Demo paper "Interactive Instance-based Evaluation of Knowledge Base Question Answering"
JavaScript
13
star
68

emnlp2016-empirical-convincingness

Code and data for EMNLP2016 article "What makes a convincing argument? Empirical analysis and detecting attributes of convincingness in Web argumentation" by Ivan Habernal and Iryna Gurevych
Java
13
star
69

germeval2017-sentiment-detection

Sentence Embeddings used in the GermEval-2017 Submission
Python
13
star
70

emnlp2018-april

Python
13
star
71

naacl2018-before-name-calling-habernal-et-al

Code and data for NAACL 2018 article "Before Name-calling: Dynamics and Triggers of Ad Hominem Fallacies in Web Argumentation" by Habernal et al.
Jupyter Notebook
13
star
72

tacl2018-preference-convincing

Experimental code for the paper 'Finding Convincing Arguments Using Scalable Bayesian Preference Learning'
TeX
12
star
73

emnlp2017-cmapsum-corpus

Accompanying code for our EMNLP 2017 publication "Bringing Structure into Summaries: Crowdsourcing a Benchmark Corpus of Concept Maps"
Java
12
star
74

nlpeer

Code associated with NLPeer: A unified resource for the study of peer review
Python
12
star
75

acl2019-GPPL-humour-metaphor

Python
12
star
76

AdaSent

This repository contains the code for the EMNLP'23 paper "AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification"
Python
12
star
77

incorporating-relevance

Code for "Incorporating Relevance Feedback for Information-Seeking Retrieval using Few-Shot Document Re-Ranking" (https://arxiv.org/abs/2210.10695).
Python
12
star
78

coling2016-pcrf-seq2seq

An adaptation of MarMot morphological tagger for generic sequence-to-sequence tasks
Python
11
star
79

lsdsem2017-story-cloze

Files for the system submitted to the LSDSem2017 Workshop Story Cloze Test Challenge
Python
11
star
80

acl2024-ircoder

Data creation, training and eval scripts for the IRCoder paper
Python
11
star
81

acl2021-metaphor-generation-conceptual

This repository is for the paper Metaphor Generation with Conceptual Mappings (ACL 2021).
Python
10
star
82

acl2022-impli

10
star
83

argotario

Argotario: a multi-lingual serious game to tackle fallacious argumentation
JavaScript
10
star
84

framenet-tools

Annotate text with FrameNet frames and arguments.
Jupyter Notebook
10
star
85

intertext-graph

A general-purpose library for cross-document NLP modelling and analysis
Jupyter Notebook
10
star
86

coling2016-genetic-swarm-MDS

A general framework for Multi-Document Summarization based on Genetic Algorithm and Swarm Intelligence
Python
10
star
87

emnlp2021-prompt-ft-heuristics

Python
10
star
88

ijcnlp2017-cmaps

Repository for the IJCNLP 2017 paper "Concept-Map-Based Multi-Document Summarization using Concept Co-Reference Resolution and Global Importance Optimization"
Java
10
star
89

acl2016-supersense-embeddings

Source code, data, and supplementary materials for our ACL 2016 article
Python
10
star
90

maps

Multicultural Proverbs and Sayings
Python
9
star
91

mdswriter

A software for manually creating multi-document summarization corpora and a platform for developing complex annotation tasks spanning multiple steps.
Java
9
star
92

acl2016-optimizing-rouge

Code for our optimizer which takes scored sentences and extract the best summary according to the ROUGE approximation.
Python
9
star
93

emnlp2021-hypercoref-cdcr

Python
9
star
94

cdcr-beyond-corpus-tailored

πŸ“„πŸ•ΈοΈ Generalizing Cross-Document Event Coreference Resolution Across Multiple Corpora
Python
9
star
95

emnlp2022-missing-counter-evidence

Source code and data of our paper "Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation" (https://arxiv.org/abs/2210.13865, to appear at EMNLP 2022).
Python
9
star
96

codeclarqa

Asking Clarification Questions for Code Generation in General-Purpose Programming Language
Python
9
star
97

thesis2018-tk_mtl_sequence_tagging

Python
9
star
98

emnlp2018-novel-metaphors

Annotations and code for the EMNLP 2018 paper 'Weeding out Conventionalized Metaphors: A Corpus of Novel Metaphor Annotations'
Python
9
star
99

emnlp2018-argmin-commonsense-knowledge

Accompanying code for our paper "Frame- and Entity-Based Knowledge for Common-Sense Argumentative Reasoning" at the 5th Workshop on Argument Mining @ EMNLP 2018.
Python
9
star
100

acl2020-empowering-active-learning

Python
8
star