UKPLab/gpl UKPLab/gpl

  • Stars
    star
    322
  • Rank 130,398 (Top 3 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created almost 3 years ago
  • Updated over 1 year ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
UKPLab/gpl
github

UKPLab

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Powerful unsupervised domain adaptation method for dense retrieval. Requires only unlabeled corpus and yields massive improvement: "GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval" https://arxiv.org/abs/2112.07577

Generative Pseudo Labeling (GPL)

GPL is an unsupervised domain adaptation method for training dense retrievers. It is based on query generation and pseudo labeling with powerful cross-encoders. To train a domain-adapted model, it needs only the unlabeled target corpus and can achieve significant improvement over zero-shot models.

For more information, checkout our publication:

For reproduction, please refer to this snapshot branch.

Installation

One can either install GPL via pip

pip install gpl

or via git clone

git clone https://github.com/UKPLab/gpl.git && cd gpl
pip install -e .

Meanwhile, please make sure the correct version of PyTorch has been installed according to your CUDA version.

Usage

GPL accepts data in the BeIR-format. For example, we can download the FiQA dataset hosted by BeIR:

wget https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/fiqa.zip
unzip fiqa.zip
head -n 2 fiqa/corpus.jsonl  # One can check this data format. Actually GPL only need this `corpus.jsonl` as data input for training.

Then we can either use the python -m function to run GPL training directly:

export dataset="fiqa"
python -m gpl.train \
    --path_to_generated_data "generated/$dataset" \
    --base_ckpt "distilbert-base-uncased" \
    --gpl_score_function "dot" \
    --batch_size_gpl 32 \
    --gpl_steps 140000 \
    --new_size -1 \
    --queries_per_passage -1 \
    --output_dir "output/$dataset" \
    --evaluation_data "./$dataset" \
    --evaluation_output "evaluation/$dataset" \
    --generator "BeIR/query-gen-msmarco-t5-base-v1" \
    --retrievers "msmarco-distilbert-base-v3" "msmarco-MiniLM-L-6-v3" \
    --retriever_score_functions "cos_sim" "cos_sim" \
    --cross_encoder "cross-encoder/ms-marco-MiniLM-L-6-v2" \
    --qgen_prefix "qgen" \
    --do_evaluation \
    # --use_amp   # Use this for efficient training if the machine supports AMP

# One can run `python -m gpl.train --help` for the information of all the arguments
# To reproduce the experiments in the paper, set `base_ckpt` to "GPL/msmarco-distilbert-margin-mse" (https://huggingface.co/GPL/msmarco-distilbert-margin-mse)

or import GPL's trainining method in a python script:

import gpl

dataset = 'fiqa'
gpl.train(
    path_to_generated_data=f"generated/{dataset}",
    base_ckpt="distilbert-base-uncased",  
    # base_ckpt='GPL/msmarco-distilbert-margin-mse',  
    # The starting checkpoint of the experiments in the paper
    gpl_score_function="dot",
    # Note that GPL uses MarginMSE loss, which works with dot-product
    batch_size_gpl=32,
    gpl_steps=140000,
    new_size=-1,
    # Resize the corpus to `new_size` (|corpus|) if needed. When set to None (by default), the |corpus| will be the full size. When set to -1, the |corpus| will be set automatically: If QPP * |corpus| <= 250K, |corpus| will be the full size; else QPP will be set 3 and |corpus| will be set to 250K / 3
    queries_per_passage=-1,
    # Number of Queries Per Passage (QPP) in the query generation step. When set to -1 (by default), the QPP will be chosen automatically: If QPP * |corpus| <= 250K, then QPP will be set to 250K / |corpus|; else QPP will be set 3 and |corpus| will be set to 250K / 3
    output_dir=f"output/{dataset}",
    evaluation_data=f"./{dataset}",
    evaluation_output=f"evaluation/{dataset}",
    generator="BeIR/query-gen-msmarco-t5-base-v1",
    retrievers=["msmarco-distilbert-base-v3", "msmarco-MiniLM-L-6-v3"],
    retriever_score_functions=["cos_sim", "cos_sim"],
    # Note that these two retriever model work with cosine-similarity
    cross_encoder="cross-encoder/ms-marco-MiniLM-L-6-v2",
    qgen_prefix="qgen",
    # This prefix will appear as part of the (folder/file) names for query-generation results: For example, we will have "qgen-qrels/" and "qgen-queries.jsonl" by default.
    do_evaluation=True,
    # use_amp=True   # One can use this flag for enabling the efficient float16 precision
)

One can also refer to this toy example on Google Colab for better understanding how the code works.

How does GPL work?

The workflow of GPL is shown as follows:

  1. GPL first use a seq2seq (we use BeIR/query-gen-msmarco-t5-base-v1 by default) model to generate queries_per_passage queries for each passage in the unlabeled corpus. The query-passage pairs are viewed as positive examples for training.

    Result files (under path $path_to_generated_data): (1) ${qgen}-qrels/train.tsv, (2) ${qgen}-queries.jsonl and also (3) corpus.jsonl (copied from $evaluation_data/);

  2. Then, it runs negative mining with the generated queries as input on the target corpus. The mined passages will be viewed as negative examples for training. One can specify any dense retrievers (SBERT or Huggingface/transformers checkpoints, we use msmarco-distilbert-base-v3 + msmarco-MiniLM-L-6-v3 by default) or BM25 to the argument retrievers as the negative miner.

    Result file (under path $path_to_generated_data): hard-negatives.jsonl;

  3. Finally, it does pseudo labeling with the powerful cross-encoders (we use cross-encoder/ms-marco-MiniLM-L-6-v2 by default.) on the query-passage pairs that we have so far (for both positive and negative examples).

    Result file (under path $path_to_generated_data): gpl-training-data.tsv. It contains (gpl_steps * batch_size_gpl) tuples in total.

Up to now, we have the actual training data ready. One can look at sample-data/generated/fiqa for a quick example about the data format. The very last step is to apply the MarginMSE loss to teach the student retriever to mimic the margin scores, CE(query, positive) - CE(query, negative) labeled by the teacher model (Cross-Encoder, CE). And of course, the MarginMSE step is included in GPL and will be done automatically:). Note that MarginMSE works with dot-product and thus the final models trained with GPL works with dot-product.

PS: The --retrievers are for negative mining. They can be any dense retrievers trained on the general domain (e.g. MS MARCO) and do not need to be strong for the target task/domain. Please refer to the paper for more details (cf. Table 7).

Customized data

One can also replace/put the customized data for any intermediate step under the path $path_to_generated_data with the same name fashion. GPL will skip the intermediate steps by using these provided data.

As a typical workflow, one might only have the (English) unlabeld corpus and want a good model performing well for this corpus. To run GPL training under such condition, one just needs these steps:

  1. Prepare your corpus in the same format as the data sample;
  2. Put your corpus.jsonl under a folder, e.g. named as "generated" for data loading and data generation by GPL;
  3. Call gpl.train with the folder path as an input argument: (other arguments work as usual)
python -m gpl.train \
    --path_to_generated_data "generated" \
    --output_dir "output" \
    --new_size -1 \
    --queries_per_passage -1

Pre-trained checkpoints and generated data

Pre-trained checkpoints

We now release the pre-trained GPL models via the https://huggingface.co/GPL. There are currently five types of models:

  1. GPL/${dataset}-msmarco-distilbert-gpl: Model with training order of (1) MarginMSE on MSMARCO -> (2) GPL on ${dataset};
  2. GPL/${dataset}-tsdae-msmarco-distilbert-gpl: Model with training order of (1) TSDAE on ${dataset} -> (2) MarginMSE on MSMARCO -> (3) GPL on ${dataset};
  3. GPL/msmarco-distilbert-margin-mse: Model trained on MSMARCO with MarginMSE;
  4. GPL/${dataset}-tsdae-msmarco-distilbert-margin-mse: Model with training order of (1) TSDAE on ${dataset} -> (2) MarginMSE on MSMARCO;
  5. GPL/${dataset}-distilbert-tas-b-gpl-self_miner: Starting from the tas-b model, the models were trained with GPL on the target corpus ${dataset} with the base model itself as the negative miner (here noted as "self_miner").

Models 1. and 2. were actually trained on top of models 3. and 4. resp. All GPL models were trained the automatic setting of new_size and queries_per_passage (by setting them to -1). This automatic setting can keep the performance while being efficient. For more details, please refer to the section 4.1 in the paper.

Among these models, GPL/${dataset}-distilbert-tas-b-gpl-self_miner ones works the best on the BeIR benchmark:

For reproducing the results with the same package versions used in the experiments, please refer to the conda environment file, environment.yml.

Generated data

We now release the generated data used in the experiments of the GPL paper:

  1. The generated data for the main experiments on the 6 BeIR datasets: https://public.ukp.informatik.tu-darmstadt.de/kwang/gpl/generated-data/main/;
  2. The generated data for the experiments on the full 18 BeIR datasets: https://public.ukp.informatik.tu-darmstadt.de/kwang/gpl/generated-data/beir.

Please note that the 4 datasets of bioasq, robust04, trec-news and signal1m are only available after registration with the original official authorities. We only release the document IDs for these corpora with the file name corpus.doc_ids.txt. For more details, please refer to the BeIR repository.

Citation

If you use the code for evaluation, feel free to cite our publication GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval:

@article{wang2021gpl,
    title = "GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval",
    author = "Kexin Wang and Nandan Thakur and Nils Reimers and Iryna Gurevych", 
    journal= "arXiv preprint arXiv:2112.07577",
    month = "4",
    year = "2021",
    url = "https://arxiv.org/abs/2112.07577",
}

Contact person and main contributor: Kexin Wang, [email protected]

https://www.ukp.tu-darmstadt.de/

https://www.tu-darmstadt.de/

Don't hesitate to send us an e-mail or report an issue, if something is broken (and it shouldn't be) or if you have further questions.

This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.

More Repositories

1

sentence-transformers

State-of-the-Art Text Embeddings
Python
15,213
star
2

EasyNMT

Easy to use, state-of-the-art Neural Machine Translation for 100+ languages
Python
1,164
star
3

emnlp2017-bilstm-cnn-crf

BiLSTM-CNN-CRF architecture for sequence tagging
Python
823
star
4

deeplearning4nlp-tutorial

Hands-on tutorial on deep learning with a special focus on Natural Language Processing (NLP)
Python
634
star
5

elmo-bilstm-cnn-crf

BiLSTM-CNN-CRF architecture for sequence tagging using ELMo representations.
Python
388
star
6

emnlp2017-relation-extraction

Context-Aware Representations for Knowledge Base Relation Extraction
Python
287
star
7

arxiv2018-xling-sentence-embeddings

Concatenated Power Mean Embeddings as Universal Cross-Lingual Sentence Representations
JavaScript
185
star
8

coling2018-graph-neural-networks-question-answering

Accompanying code for our COLING 2018 paper "Modeling Semantics with Gated Graph Neural Networks for Knowledge Base Question Answering"
Python
173
star
9

plms-graph2text

Investigating Pretrained Language Models for Graph-to-Text Generation
Python
143
star
10

MMT-Retrieval

Python
129
star
11

kg2text

Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs (authors' implementation for the TACL20 paper)
Python
94
star
12

acl2019-BERT-argument-classification-and-clustering

Python
83
star
13

argument-reasoning-comprehension-task

The Argument Reasoning Comprehension Task: Source codes & Datasets
Java
71
star
14

pytorch-bertflow

Pytorch-version BERT-flow: One can apply BERT-flow to any PLM within Pytorch framework.
Python
69
star
15

acl2017-non-factoid-qa

Code for paper "End-to-End Non-Factoid Question Answering with an Interactive Visualization of Neural Attention Weights"
Python
67
star
16

acl2017-neural_end2end_am

Accompanying code for our ACL-2017 publication on Neural End-to-End Learning for Computational Argumentation Mining
Python
60
star
17

starsem2018-entity-linking

Accompanying code for our *SEM 2018 @ NAACL 2018 paper "Mixing Context Granularities for Improved Entity Linking on Question Answering Data across Entity Categories"
Python
59
star
18

fever-2018-team-athene

Python
46
star
19

nessie

Automatically detect errors in annotated corpora.
Python
46
star
20

mdl-stance-robustness

Multi-dataset stance detection and robustness experiments
Python
42
star
21

naacl18-multitask_argument_mining

Code for the paper "Multi-Task Learning for Argumentation Mining in Low-Resource Settings"
Python
40
star
22

semeval2017-scienceie

Code for keyphrase classification systems submitted to the SemEval 2017 shared task ScienceIE.
Python
36
star
23

starsem18-multimodalKB

Python
35
star
24

on-emergence

Codes and files for the paper Are Emergent Abilities in Large Language Models just In-Context Learning
Python
34
star
25

acl2020-interactive-entity-linking

Python
33
star
26

useb

Heterogenous, Task- and Domain-Specific Benchmark for Unsupervised Sentence Embeddings used in the TSDAE paper: https://arxiv.org/abs/2104.06979.
Python
32
star
27

5pils

Code associated with the EMNLP 2024 Main paper: "Image, tell me your story!" Predicting the original meta-context of visual misinformation.
Python
31
star
28

emnlp2017-graphdocexplore

Accompanying code for our EMNLP 2017 publication "GraphDocExplore: A Framework for the Experimental Comparison of Graph-based Document Exploration Techniques"
JavaScript
29
star
29

StructAdapt

Structural Adapters in Pretrained Language Models for AMR-to-Text Generation (EMNLP 2021)
Python
29
star
30

coling2018_fake-news-challenge

Python
28
star
31

iwcs2017-answer-selection

Repository for the IWCS 2017 paper "Representation Learning for Answer Selection with LSTM-Based Importance Weighting"
Python
28
star
32

controlled-argument-generation

Controlling Argument Generation via topic, stance, and aspect
Python
28
star
33

acl2016-convincing-arguments

Code and data for ACL2016 article "Which argument is more convincing? Analyzing and predicting convincingness of Web arguments using bidirectional LSTM" by Ivan Habernal and Iryna Gurevych"
Java
28
star
34

lagonn

Source code and data for Like a Good Nearest Neighbor
Python
28
star
35

arxiv2018-bayesian-ensembles

Python
26
star
36

naacl2019-like-humans-visual-attacks

Python
26
star
37

refresh2018-predicting-trends-from-arxiv

Python
26
star
38

emnlp2018-activation-functions

Shell
26
star
39

emnlp2020-debiasing-unknown

Python
25
star
40

naacl2019-does-my-rebuttal-matter

Ruby
25
star
41

acl2024-dapr

Python
25
star
42

acl2017-interactive_summarizer

A general framework for Interactive Multi-Document Summarization
Python
24
star
43

adaptable-adapters

Python
23
star
44

e2e-nlg-challenge-2017

E2E NLG Challenge submission
Python
23
star
45

acl2020-confidence-regularization

Python
23
star
46

linspector

Python
23
star
47

MetaQA

MetaQA: Combining Expert Agents for Multi-Skill Question Answering
Python
22
star
48

emnlp2019-dualgraph

Enhancing AMR-to-Text Generation with Dual Graph Representations (implementation for the EMNLP-IJCNLP-2019 paper)
Python
22
star
49

arxiv2024-divergent-cot

Code for the 2024 arXiv publication "Fine-Tuning with Divergent Chains of Thought Boosts Reasoning Through Self-Correction in Language Models"
Python
21
star
50

iclr2024-model-merging

This is the repository for "Model Merging by Uncertainty-Based Gradient Matching", ICLR 2024.
Python
20
star
51

aaai2019-coala-cqa-answer-selection

Python
20
star
52

tacl2017-event-time-extraction

Event Time Extraction with a Decision Tree of Neural Classifiers
Python
19
star
53

tac2015-event-detection

Files for Event Nugget Detection systems submitted to TAC 2015 shared task on Event Nugget Detection
Java
19
star
54

coling2018-xling_argument_mining

Erlang
16
star
55

CARE

Project CARE
Vue
16
star
56

llm-roleplay

LLM Roleplay: Simulating Human-Chatbot Interaction
Python
16
star
57

eacl2017-oodFrameNetSRL

Implementation of a simple frame identification approach (SimpleFrameId) described in the paper "Out-of-domain FrameNet Semantic Role Labeling"
Python
15
star
58

SciGen

Python
15
star
59

acl2020-dialogue-coherence-assessment

Python
15
star
60

TWEAC-qa-agent-selection

Python
14
star
61

emnlp2020-multicqa

MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale
Python
14
star
62

emnlp2024-code-prompting

Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs. EMNLP 2024
Python
14
star
63

acl2024-triple-encoders

triple-encoders is a library for contextualizing distributed Sentence Transformers representations.
Python
14
star
64

lrec2018-live-blog-corpus

Python
13
star
65

EACL21-personalized-conversational-system

Python
13
star
66

emnlp2017-claim-identification

Source code repository for our EMNLP paper on cross-domain claim identification
Java
13
star
67

emnlp2018-question-answering-interface

Accompanying code for our EMNLP 2018 Demo paper "Interactive Instance-based Evaluation of Knowledge Base Question Answering"
JavaScript
13
star
68

emnlp2016-empirical-convincingness

Code and data for EMNLP2016 article "What makes a convincing argument? Empirical analysis and detecting attributes of convincingness in Web argumentation" by Ivan Habernal and Iryna Gurevych
Java
13
star
69

germeval2017-sentiment-detection

Sentence Embeddings used in the GermEval-2017 Submission
Python
13
star
70

emnlp2018-april

Python
13
star
71

naacl2018-before-name-calling-habernal-et-al

Code and data for NAACL 2018 article "Before Name-calling: Dynamics and Triggers of Ad Hominem Fallacies in Web Argumentation" by Habernal et al.
Jupyter Notebook
13
star
72

tacl2018-preference-convincing

Experimental code for the paper 'Finding Convincing Arguments Using Scalable Bayesian Preference Learning'
TeX
12
star
73

emnlp2017-cmapsum-corpus

Accompanying code for our EMNLP 2017 publication "Bringing Structure into Summaries: Crowdsourcing a Benchmark Corpus of Concept Maps"
Java
12
star
74

nlpeer

Code associated with NLPeer: A unified resource for the study of peer review
Python
12
star
75

acl2019-GPPL-humour-metaphor

Python
12
star
76

AdaSent

This repository contains the code for the EMNLP'23 paper "AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification"
Python
12
star
77

incorporating-relevance

Code for "Incorporating Relevance Feedback for Information-Seeking Retrieval using Few-Shot Document Re-Ranking" (https://arxiv.org/abs/2210.10695).
Python
12
star
78

coling2016-pcrf-seq2seq

An adaptation of MarMot morphological tagger for generic sequence-to-sequence tasks
Python
11
star
79

lsdsem2017-story-cloze

Files for the system submitted to the LSDSem2017 Workshop Story Cloze Test Challenge
Python
11
star
80

acl2024-ircoder

Data creation, training and eval scripts for the IRCoder paper
Python
11
star
81

acl2021-metaphor-generation-conceptual

This repository is for the paper Metaphor Generation with Conceptual Mappings (ACL 2021).
Python
10
star
82

acl2022-impli

10
star
83

argotario

Argotario: a multi-lingual serious game to tackle fallacious argumentation
JavaScript
10
star
84

framenet-tools

Annotate text with FrameNet frames and arguments.
Jupyter Notebook
10
star
85

intertext-graph

A general-purpose library for cross-document NLP modelling and analysis
Jupyter Notebook
10
star
86

coling2016-genetic-swarm-MDS

A general framework for Multi-Document Summarization based on Genetic Algorithm and Swarm Intelligence
Python
10
star
87

emnlp2021-prompt-ft-heuristics

Python
10
star
88

ijcnlp2017-cmaps

Repository for the IJCNLP 2017 paper "Concept-Map-Based Multi-Document Summarization using Concept Co-Reference Resolution and Global Importance Optimization"
Java
10
star
89

acl2016-supersense-embeddings

Source code, data, and supplementary materials for our ACL 2016 article
Python
10
star
90

maps

Multicultural Proverbs and Sayings
Python
9
star
91

mdswriter

A software for manually creating multi-document summarization corpora and a platform for developing complex annotation tasks spanning multiple steps.
Java
9
star
92

acl2016-optimizing-rouge

Code for our optimizer which takes scored sentences and extract the best summary according to the ROUGE approximation.
Python
9
star
93

emnlp2021-hypercoref-cdcr

Python
9
star
94

cdcr-beyond-corpus-tailored

πŸ“„πŸ•ΈοΈ Generalizing Cross-Document Event Coreference Resolution Across Multiple Corpora
Python
9
star
95

emnlp2022-missing-counter-evidence

Source code and data of our paper "Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation" (https://arxiv.org/abs/2210.13865, to appear at EMNLP 2022).
Python
9
star
96

codeclarqa

Asking Clarification Questions for Code Generation in General-Purpose Programming Language
Python
9
star
97

thesis2018-tk_mtl_sequence_tagging

Python
9
star
98

emnlp2018-novel-metaphors

Annotations and code for the EMNLP 2018 paper 'Weeding out Conventionalized Metaphors: A Corpus of Novel Metaphor Annotations'
Python
9
star
99

emnlp2018-argmin-commonsense-knowledge

Accompanying code for our paper "Frame- and Entity-Based Knowledge for Common-Sense Argumentative Reasoning" at the 5th Workshop on Argument Mining @ EMNLP 2018.
Python
9
star
100

acl2020-empowering-active-learning

Python
8
star