• Stars
    star
    384
  • Rank 107,666 (Top 3 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created about 4 years ago
  • Updated 10 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

UnifiedQA: Crossing Format Boundaries With a Single QA System

UnifiedQA

You may want to check out:

Update (Feb '22): UnifiedQA-v2

Using the models in PyTorch/HuggingFace

You can very easily load the models with Transformers >=3.1, instead of downloading them manually. The models are listed on this page. Here is a list of model these model names hosted on HuggingFace model hub:

Model Name Huggingface ID (s)
UnifiedQA (T5) - small allenai/unifiedqa-t5-small
UnifiedQA (T5) - base allenai/unifiedqa-t5-base
UnifiedQA (T5) - large allenai/unifiedqa-t5-large
UnifiedQA (T5) - 3B allenai/unifiedqa-t5-3b
UnifiedQA (T5) - 11B allenai/unifiedqa-t5-11b
UnifiedQA-v2 (T5) - small allenai/unifiedqa-v2-t5-small-[ckpt]
UnifiedQA-v2 (T5) - base allenai/unifiedqa-v2-t5-base-[ckpt]
UnifiedQA-v2 (T5) - large allenai/unifiedqa-v2-t5-large-[ckpt]
UnifiedQA-v2 (T5) - 3B allenai/unifiedqa-v2-t5-3b-[ckpt]
UnifiedQA-v2 (T5) - 11B allenai/unifiedqa-v2-t5-11b-[ckpt]

Where [ckpt] can be either 1251000 or 1363200. The numbers in the paper are reported based on 1251000 checkpoints.

Here is an examples:

from transformers import T5Tokenizer, T5ForConditionalGeneration

model_name = "allenai/unifiedqa-t5-small" # you can specify the model size here
tokenizer = T5Tokenizer.from_pretrained(model_name)
model = T5ForConditionalGeneration.from_pretrained(model_name)

def run_model(input_string, **generator_args):
    input_ids = tokenizer.encode(input_string, return_tensors="pt")
    res = model.generate(input_ids, **generator_args)
    return tokenizer.batch_decode(res, skip_special_tokens=True)

For instance, here is how you can use it to answer a multiple-choice question:

run_model("which is best conductor? \\n (a) iron (b) feather")

which gives: ['iron']

run_model("scott filled a tray with juice and put it in a freezer. the next day, scott opened the freezer. how did the juice most likely change? \\n (a) it condensed. (b) it evaporated. (c) it became a gas. (d) it became a solid.")

which produces: ['it condensed.'].

Note that you can also pass in the arguments for text generation to the run_model(.) function:

run_model("which is best conductor? \\n (a) iron (b) feather (c) wood (d) plastic",
         temperature=0.9, num_return_sequences=4, num_beams=20)

Feeding data into UnifiedQA

Datasets should be converted into a textin/text-out format.

  • Question always comes first.
  • We use \n separators between different parts of the input. This ensures having a humanlike encoding while not making it overly-specific to a certain format. Note that this separator isn't the newline character (which it looks suspiciously like), but rather backslash-n.
  • Make sure the whole input is correctly pre-processed (e.g., lower-cased)

Here are several examples:

Dataset SQuAD 1.1 (extractive QA)
Encoded Input At what speed did the turbine operate? \n (Nikola_Tesla) On his 50th birthday in 1906, Tesla demonstrated his 200 horsepower (150 kilowatts) 16,000 rpm bladeless turbine. ...
Encoded Output 16,000 rpm
Dataset NarrativeQA (Abstractive QA)
Encoded Input What does a drink from narcissus's spring cause the drinker to do? \n Mercury has awakened Echo, who weeps for Narcissus, and states that a drink from Narcissus's spring causes the drinkers to ''Grow dotingly enamored of themselves.'' ...
Encoded Output fall in love with themselves
Dataset ARC-challenge (Multiple-choice QA)
Encoded Input What does photosynthesis produce that helps plants grow? \n (A) water (B) oxygen (C) protein (D) sugar
Encoded Output sugar
Dataset MCTest (Multiple-choice QA)
Encoded Input Who was Billy? \n (A) The skinny kid (B) A teacher (C) A little kid (D) The big kid \n Billy was like a king on the school yard. A king without a queen. He was the biggest kid in our grade, so he made all the rules during recess. ...
Encoded Output The big kid
Dataset BoolQ (Yes-no QA)
Encoded Input Was America the first country to have a president? \n (President) The first usage of the word president to denote the highest official in a government was during the Commonwealth of England ...
Encoded Output no

If you wanna see how this encoding is done on our datasets, check out this script.

The datasets/tasks used in the experiments

While the datasets we used are all public, it could be a bit time-confusing to convert them all into text-to-text format. We're releasing the already-proccessed text-to-text datasets based on the encoding used in this work. Files are included in this Google Cloud bucket. Here is the script we used in order to convert each dataset into text-in-text-out format.

Prediction files

Reach out to DanielK if you want them! :)

Released Model Checkpoints

If you intend to create a QA system, you can use our QA-specialized models for your purpose:

T5 models

Note: In the experiments reported in our paper we always used the checkpoint closest to 100k steps (it usually corresponds to checkpoint 1100500)

You can use these in two ways:

  • If you don't have any training data, you can use them for the evaluation.
  • If you training data, you can use them as your initial models and fine-tune on them.

For more details see the T5 repository.

BART models

The BART models are downloaded from this link (3.6G). For detailed instructions on running the code (training/finetuning/testing), please refer to here. The uncased models usually gave us better and more robust results.

v2 T5 models

Note: In the experiments reported in our paper we always used the checkpoint closest to 250k steps.

FAQ

I am not getting the expected results. An common issue with using UnifiedQA is making sure you use the separator (\n) when encoding encoding your inputs. See the earlier section where we delineate how to encode the inputs.

Help! I am getting the following error! See this discussion if you're getting the following error:

ValueError: Configurable 'make_layer_stack' doesn't have a parameter named 'use_universal_transformer'.
  In file "gs://danielk-files/t5-models/union_mixture/11B/operative_config.gin", line 83

How to cite

If you extend or use this work, please cite the relevant papers:

@inproceedings{2020unifiedqa,
    title={UnifiedQA: Crossing Format Boundaries With a Single QA System},
    author={D. Khashabi and S. Min and T. Khot and A. Sabhwaral and O. Tafjord and P. Clark and H. Hajishirzi},
    journal={EMNLP - findings},
    year={2020}
}
@article{khashabi2022unifiedqa,
    title={UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Training},
    author={Khashabi, Daniel and Kordi, Yeganeh and Hajishirzi, Hannaneh},
    journal={arXiv preprint arXiv:2202.12359},
    year={2022}
}

More Repositories

1

allennlp

An open-source NLP research library, built on PyTorch.
Python
11,691
star
2

OLMo

Modeling, training, eval, and inference code for OLMo
Python
3,949
star
3

RL4LMs

A modular RL library to fine-tune language models to human preferences
Python
2,020
star
4

longformer

Longformer: The Long-Document Transformer
Python
1,955
star
5

bilm-tf

Tensorflow implementation of contextualized word representations from bi-directional language models
Python
1,621
star
6

scispacy

A full spaCy pipeline and models for scientific/biomedical documents.
Python
1,566
star
7

bi-att-flow

Bi-directional Attention Flow (BiDAF) network is a multi-stage hierarchical process that represents context at different levels of granularity and uses a bi-directional attention flow mechanism to achieve a query-aware context representation without early summarization.
Python
1,524
star
8

scibert

A BERT model for scientific text.
Python
1,432
star
9

ai2thor

An open-source platform for Visual AI.
C#
1,010
star
10

open-instruct

Python
932
star
11

XNOR-Net

ImageNet classification using binary Convolutional Neural Networks
Lua
839
star
12

mmc4

MultimodalC4 is a multimodal extension of c4 that interleaves millions of images with text.
Python
793
star
13

s2orc

S2ORC: The Semantic Scholar Open Research Corpus: https://www.aclweb.org/anthology/2020.acl-main.447/
Python
745
star
14

scitldr

Python
734
star
15

natural-instructions

Expanding natural instructions
Python
690
star
16

dolma

Data and tools for generating and inspecting OLMo pre-training data.
Python
678
star
17

visprog

Official code for VisProg (CVPR 2023 Best Paper!)
Python
642
star
18

papermage

library supporting NLP and CV research on scientific papers
Python
605
star
19

science-parse

Science Parse parses scientific papers (in PDF form) and returns them in structured form.
Java
566
star
20

writing-code-for-nlp-research-emnlp2018

A companion repository for the "Writing code for NLP Research" Tutorial at EMNLP 2018
Python
558
star
21

pdffigures2

Given a scholarly PDF, extract figures, tables, captions, and section titles.
Scala
514
star
22

allennlp-models

Officially supported AllenNLP models
Python
512
star
23

tango

Organize your experiments into discrete steps that can be cached and reused throughout the lifetime of your research project.
Python
507
star
24

objaverse-xl

πŸͺ Objaverse-XL is a Universe of 10M+ 3D Objects. Contains API Scripts for Downloading and Processing!
Python
490
star
25

dont-stop-pretraining

Code associated with the Don't Stop Pretraining ACL 2020 paper
Python
488
star
26

specter

SPECTER: Document-level Representation Learning using Citation-informed Transformers
Python
485
star
27

unified-io-2

Python
471
star
28

macaw

Multi-angle c(q)uestion answering
Python
451
star
29

document-qa

Python
420
star
30

scholarphi

An interactive PDF reader.
Python
410
star
31

deep_qa

A deep NLP library, based on Keras / tf, focused on question answering (but useful for other NLP too)
Python
405
star
32

acl2018-semantic-parsing-tutorial

Materials from the ACL 2018 tutorial on neural semantic parsing
402
star
33

kb

KnowBert -- Knowledge Enhanced Contextual Word Representations
Python
359
star
34

pawls

Software that makes labeling PDFs easy.
Python
356
star
35

PeerRead

Data and code for Kang et al., NAACL 2018's paper titled "A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications"
Python
354
star
36

naacl2021-longdoc-tutorial

Python
343
star
37

openie-standalone

Quality information extraction at web scale. Edit
Scala
329
star
38

python-package-template

A template repo for Python packages
Python
318
star
39

acl2022-zerofewshot-tutorial

293
star
40

allenact

An open source framework for research in Embodied-AI from AI2.
Python
293
star
41

ir_datasets

Provides a common interface to many IR ranking datasets.
Python
291
star
42

s2orc-doc2json

Parsers for scientific papers (PDF2JSON, TEX2JSON, JATS2JSON)
Python
290
star
43

beaker-cli

A collaborative platform for rapid and reproducible research.
Go
230
star
44

Holodeck

CVPR 2024: Language Guided Generation of 3D Embodied AI Environments.
Python
220
star
45

procthor

🏘️ Scaling Embodied AI by Procedurally Generating Interactive 3D Houses
Python
214
star
46

comet-atomic-2020

Python
212
star
47

FineGrainedRLHF

Python
209
star
48

fm-cheatsheet

Website for hosting the Open Foundation Models Cheat Sheet.
Python
207
star
49

spv2

Science-parse version 2
Python
206
star
50

scifact

Data and models for the SciFact verification task.
Python
206
star
51

OLMo-Eval

Evaluation suite for LLMs
Python
200
star
52

unified-io-inference

Jupyter Notebook
196
star
53

allennlp-demo

Code for the AllenNLP demo.
TypeScript
191
star
54

lumos

Code and data for "Lumos: Learning Agents with Unified Data, Modular Design, and Open-Source LLMs"
Python
190
star
55

citeomatic

A citation recommendation system that allows users to find relevant citations for their paper drafts. The tool is backed by Semantic Scholar's OpenCorpus dataset.
Jupyter Notebook
182
star
56

cartography

Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics
Jupyter Notebook
180
star
57

savn

Learning to Learn how to Learn: Self-Adaptive Visual Navigation using Meta-Learning (https://arxiv.org/abs/1812.00971)
Python
175
star
58

vampire

Variational Methods for Pretraining in Resource-limited Environments
Python
173
star
59

objaverse-rendering

πŸ“· Scripts for rendering Objaverse
Python
169
star
60

hidden-networks

Python
164
star
61

ScienceWorld

ScienceWorld is a text-based virtual environment centered around accomplishing tasks from the standardized elementary science curriculum.
Scala
156
star
62

vila

Incorporating VIsual LAyout Structures for Scientific Text Classification
Python
155
star
63

mmda

multimodal document analysis
Jupyter Notebook
154
star
64

cord19

Get started with CORD-19
149
star
65

PRIMER

The official code for PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Python
145
star
66

dnw

Discovering Neural Wirings (https://arxiv.org/abs/1906.00586)
Python
139
star
67

tpu_pretrain

LM Pretraining with PyTorch/TPU
Python
129
star
68

deepfigures-open

Companion code to the paper "Extracting Scientific Figures with Distantly Supervised Neural Networks" πŸ€–
Python
129
star
69

catwalk

This project studies the performance and robustness of language models and task-adaptation methods.
Python
129
star
70

allentune

Hyperparameter Search for AllenNLP
Python
128
star
71

lm-explorer

interactive explorer for language models
Python
127
star
72

pdffigures

Command line tool to extract figures, tables, and captions from scholarly documents in PDF form.
C++
125
star
73

SciREX

Data/Code Repository for https://api.semanticscholar.org/CorpusID:218470122
Python
125
star
74

s2-folks

Public space for the user community of Semantic Scholar APIs to share scripts, report issues, and make suggestions.
125
star
75

scidocs

Dataset accompanying the SPECTER model
Python
124
star
76

gooaq

Question-answers, collected from Google
Python
116
star
77

OpenBookQA

Code for experiments on OpenBookQA from the EMNLP 2018 paper "Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering"
Python
113
star
78

allennlp-as-a-library-example

A simple example for how to build your own model using AllenNLP as a dependency.
Python
113
star
79

alexafsm

With alexafsm, developers can model dialog agents with first-class concepts such as states, attributes, transition, and actions. alexafsm also provides visualization and other tools to help understand, test, debug, and maintain complex FSM conversations.
Python
108
star
80

allennlp-semparse

A framework for building semantic parsers (including neural module networks) with AllenNLP, built by the authors of AllenNLP
Python
107
star
81

scicite

Repository for NAACL 2019 paper on Citation Intent prediction
Python
106
star
82

peS2o

Pretraining Efficiently on S2ORC!
105
star
83

multimodalqa

Python
102
star
84

commonsense-kg-completion

Python
102
star
85

real-toxicity-prompts

Jupyter Notebook
101
star
86

ai2thor-rearrangement

πŸ”€ Visual Room Rearrangement
Python
97
star
87

embodied-clip

Official codebase for EmbCLIP
Python
97
star
88

aristo-mini

Aristo mini is a light-weight question answering system that can quickly evaluate Aristo science questions with an evaluation web server and the provided baseline solvers.
Python
96
star
89

s2search

The Semantic Scholar Search Reranker
Python
93
star
90

elastic

Python
91
star
91

reward-bench

RewardBench: the first evaluation tool for reward models.
Python
90
star
92

flex

Few-shot NLP benchmark for unified, rigorous eval
Python
89
star
93

gpv-1

A task-agnostic vision-language architecture as a step towards General Purpose Vision
Jupyter Notebook
89
star
94

manipulathor

ManipulaTHOR, a framework that facilitates visual manipulation of objects using a robotic arm
Jupyter Notebook
86
star
95

medicat

Dataset of medical images, captions, subfigure-subcaption annotations, and inline textual references
Python
85
star
96

propara

ProPara (Process Paragraph Comprehension) dataset and models
Python
82
star
97

allennlp-guide

Code and material for the AllenNLP Guide
Python
81
star
98

hierplane

A tool for visualizing trees, tailored specifically to the analysis of parse trees.
JavaScript
81
star
99

S2AND

Semantic Scholar's Author Disambiguation Algorithm & Evaluation Suite
Python
78
star
100

ARC-Solvers

ARC Question Solvers
Python
78
star