• Stars
    star
    273
  • Rank 150,780 (Top 3 %)
  • Language
  • License
    MIT License
  • Created over 2 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Must-read Papers of Parameter-Efficient Tuning (Delta Tuning) Methods on Pre-trained Models.

DeltaPapers

Must-read papers on parameter-efficient tuning methods (Delta Tuning) for pre-trained models.

Content

Why Parameter Efficient?

Increasingly larger pre-trained models bring both the blessing of effectiveness on existing and unknown tasks, and the curse of prohibitive adaptation cost. In this context, parameter-efficient tuning methods (delta tuning) are developed and demonstrate a promising way to stimulate colossal models with only a small portion of tunable parameters, thereby dramatically reducing the computational and storage costs of model adaptation. In addition to the obvious practical value, delta tuning seems to imply that the particular adaptation of pre-trained models may be a very simple process, which may usher in intriguing theoretical issues that are worth exploring.

Keywords Convention

We follow the general idea of PromptPapers to label the papers.

The abbreviation of the work.

The main explored task of the work.

The main explored property of delta tuning methods in the work.

Other important information of the work.

Papers

Overview

  • Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Model, Preprint 2022.

    Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Hai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, Maosong Sun. [pdf], [OpenDelta]

Methodology

  • Parameter-Efficient Transfer Learning for NLP, ICML 2019.

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, Sylvain Gelly. [pdf], [Project]

  • BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task, ICML 2019.

    Asa Cooper Stickland, Iain Murray. [pdf], [Project]

  • Masking as an Efficient Alternative to Finetuning for Pretrained Language Models, EMNLP 2020.

    Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, Hinrich Schütze. [pdf], [Project]

  • Prefix-Tuning: Optimizing Continuous Prompts for Generation, ACL 2021.

    Xiang Lisa Li, Percy Liang. [pdf], [Project]

  • Parameter-Efficient Transfer Learning with Diff Pruning, ACL 2021.

    Demi Guo, Alexander M. Rush, Yoon Kim. [pdf], [Project]

  • The Power of Scale for Parameter-Efficient Prompt Tuning, EMNLP 2021.

    Brian Lester, Rami Al-Rfou, Noah Constant. [pdf], [OpenPrompt Implementation]

  • COMPACTER: Efficient Low-Rank Hypercomplex Adapter Layers, Neurips 2021.

    Rabeeh Karimi Mahabadi, James Henderson, Sebastian Ruder. [pdf], [Project]

  • BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models, ACL 2022.

    Elad Ben Zaken, Shauli Ravfogel, Yoav Goldberg. [pdf], [Project]

  • LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022.

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen. [pdf], [Project]

  • Fast Model Editing at Scale, ICLR 2022.

    Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, Christopher D. Manning. [pdf], [Project]

  • Editing Factual Knowledge in Language Models, EMNLP 2021.

    Nicola De Cao, Wilker Aziz, Ivan Titov. [pdf], [Project]

  • Training Neural Networks with Fixed Sparse Masks, Neurips 2021.

    Yi-Lin Sung, Varun Nair, Colin Raffeln. [pdf], [Project]

  • GPT Understands, Too, Preprint 2020.

    Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, Jie Tang. [pdf], [Project]

  • LiST: Lite Self-training Makes Efficient Few-shot Learners, Preprint 2021.

    Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, Jianfeng Gao. [pdf], [Project]

  • Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks, ACL 2021.

    Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, James Henderson. [pdf], [Project]

Analysis

  • Towards a Unified View of Parameter-Efficient Transfer Learning, ICLR 2022.

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, Graham Neubig. [pdf], [Project]

  • AdapterBias: Parameter-efficient Token-dependent Embedding Shift for Adapters in NLP Tasks, Preprint 2021.

    Anonymous. [pdf]

  • AdapterFusion: Non-Destructive Task Composition for Transfer Learning, Preprint 2020.

    Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, Iryna Gurevych. [pdf], [Project]

  • On the Effectiveness Adapter-based Tuning for Pretrained Language Model Adaptation, ACL 2021.

    Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jia-Wei Low, Lidong Bing, Luo Si. [pdf]

  • UNIPELT: A Unified Framework for Parameter-Efficient Language Model Tuning, Preprint 2021.

    Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen-tau Yih, Madian Khabsa. [pdf], [Project]

  • Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data, ICLR 2021.

    Jonathan Pilault, Amine Elhattami, Christopher Pal. [pdf], [Project]

  • Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning, EMNLP 2021.

    Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, Fei Huang. [pdf], [Project]

  • Exploring Low-dimensional Intrinsic Task Subspace via Prompt Tuning, Preprint 2021.

    Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, Jie Zhou. [pdf], [Project]

  • Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning, ACL 2021.

    Armen Aghajanyan, Luke Zettlemoyer, Sonal Gupta. [pdf]

  • Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators, ACL 2021.

    Peiyu Liu, Ze-Feng Gao, Wayne Xin Zhao, Z.Y. Xie, Zhong-Yi Lu, Ji-Rong Wen. [pdf], [Project]

  • Movement Pruning: Adaptive Sparsity by Fine-Tuning, Neurips 2020.

    Victor Sanh, Thomas Wolf, Alexander M. Rush. [pdf], [Project]

  • Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameters, ICLR 2021.

    Aston Zhang, Yi Tay, Shuai Zhang, Alvin Chan, Anh Tuan Luu, Siu Cheung Hui, Jie Fu. [pdf]

  • Shapeshifter: a Parameter-efficient Transformer using Factorized Reshaped Matrices, Neurips 2021.

    Aliakbar Panahi, Seyran Saeedi, Tom Arodz. [pdf], [Project]

  • Adapterdrop: On the efficiency of adapters in transformers, EMNLP 2021.

    Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, Iryna Gurevych. [pdf]

Applications & Tools

  • OpenDelta: A Flexible Plug-in Tool for Delta Tuning. [Project]

  • Lightweight Adapter Tuning for Multilingual Speech Translation, ACL 2021.

    Hang Le, Juan Pino, Changhan Wang, Jiatao Gu, Didier Schwab, Laurent Besacier. [pdf], [Project]

  • VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks, CVPR 2022.

    Yi-Lin Sung, Jaemin Cho, Mohit Bansal. [pdf], [Project]

  • AdapterHub: A Framework for Adapting Transformers, EMNLP demo 2020.

    Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, Iryna Gurevych. [pdf], [Project]

Blogs

  • Parameter-efficient transfer learning for NLP, Overview Blog Post 2022, [Blog]

Contribution

Contributors

Contributing to this paper list

  • First, think about which category the work should belong to.
  • Second, use the same format as the others to describe the work. Note that there should be an empty line between the title and the author's list, and take care of the indentation.
  • Then, add keywords tags. Add the pdf link of the paper. If it is an arxiv publication, we prefer /abs/ format to /pdf/ format.

More Repositories

1

GNNPapers

Must-read papers on graph neural networks (GNN)
15,490
star
2

WantWords

An open-source online reverse dictionary.
JavaScript
6,933
star
3

OpenPrompt

An Open-Source Framework for Prompt-Learning.
Python
4,323
star
4

OpenNRE

An Open-Source Package for Neural Relation Extraction (NRE)
Python
4,322
star
5

PromptPapers

Must-read papers on prompt-based tuning for pre-trained language models.
4,059
star
6

OpenKE

An Open-Source Package for Knowledge Embedding (KE)
Python
3,813
star
7

PLMpapers

Must-read Papers on pre-trained language models.
3,161
star
8

NRLPapers

Must-read papers on network representation learning (NRL) / network embedding (NE)
TeX
2,524
star
9

UltraChat

Large-scale, Informative, and Diverse Multi-round Chat Data (and Models)
Python
2,225
star
10

THULAC-Python

An Efficient Lexical Analyzer for Chinese
Python
1,997
star
11

OpenNE

An Open-Source Package for Network Embedding (NE)
Python
1,683
star
12

KRLPapers

Must-read papers on knowledge representation learning (KRL) / knowledge embedding (KE)
TeX
1,532
star
13

TAADpapers

Must-read Papers on Textual Adversarial Attack and Defense
Python
1,505
star
14

ERNIE

Source code and dataset for ACL 2019 paper "ERNIE: Enhanced Language Representation with Informative Entities"
Python
1,408
star
15

KB2E

Knowledge Graph Embeddings including TransE, TransH, TransR and PTransE
C++
1,360
star
16

NREPapers

Must-read papers on neural relation extraction (NRE)
TeX
1,028
star
17

OpenDelta

A plug-and-play library for parameter-efficient-tuning (Delta Tuning)
Python
991
star
18

WebCPM

Official codes for ACL 2023 paper "WebCPM: Interactive Web Search for Chinese Long-form Question Answering"
HTML
977
star
19

OpenCLaP

Open Chinese Language Pre-trained Model Zoo
977
star
20

RCPapers

Must-read papers on Machine Reading Comprehension
890
star
21

ToolLearningPapers

865
star
22

NRE

Neural Relation Extraction, including CNN, PCNN, CNN+ATT, PCNN+ATT
C++
812
star
23

THULAC

An Efficient Lexical Analyzer for Chinese
C++
790
star
24

FewRel

A Large-Scale Few-Shot Relation Extraction Dataset
Python
727
star
25

THUOCL

THUOCL(THU Open Chinese Lexicon)中文词库
697
star
26

Chinese_Rumor_Dataset

中文谣言数据
693
star
27

OpenAttack

An Open-Source Package for Textual Adversarial Attack.
Python
682
star
28

DocRED

Dataset and codes for ACL 2019 DocRED: A Large-Scale Document-Level Relation Extraction Dataset.
Python
609
star
29

OpenHowNet

Core Data of HowNet and OpenHowNet Python API
Python
608
star
30

TensorFlow-TransX

An implementation of TransE and its extended models for Knowledge Representation Learning on TensorFlow
Python
514
star
31

LegalPapers

Must-read Papers on Legal Intelligence
465
star
32

CAIL

Chinese AI & Law Challenge
449
star
33

OpenMatch

An Open-Source Package for Information Retrieval.
Python
447
star
34

BERT-KPE

Python
443
star
35

Fast-TransX

An Efficient implementation of TransE and its extended models for Knowledge Representation Learning
C++
401
star
36

TensorFlow-Summarization

Python
390
star
37

Few-NERD

Code and data of ACL 2021 paper "Few-NERD: A Few-shot Named Entity Recognition Dataset"
Python
385
star
38

SOS4NLP

Survey of Surveys for Natural Language Processing (SOS4NLP)
327
star
39

THULAC-Java

An Efficient Lexical Analyzer for Chinese
Java
325
star
40

BMCourse

The repo for Tsinghua summer course: Interdisciplinary Seminar on Big Models
Python
321
star
41

InfLLM

The code of our paper "InfLLM: Unveiling the Intrinsic Capacity of LLMs for Understanding Extremely Long Sequences with Training-Free Memory"
Python
287
star
42

NSC

Neural Sentiment Classification
Python
286
star
43

LLaVA-UHD

LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images
Python
276
star
44

Chinese_NRE

Source code for ACL 2019 paper "Chinese Relation Extraction with Multi-Grained Information and External Linguistic Knowledge"
Python
268
star
45

PL-Marker

Source code for "Packed Levitated Marker for Entity and Relation Extraction"
Python
255
star
46

LEGENT

Open Platform for Embodied Agents
Python
250
star
47

SE-WRL

Improved Word Representation Learning with Sememes
C
197
star
48

SCPapers

Must-read Papers on Sememe Computation
196
star
49

THUCTC

An Efficient Chinese Text Classifier
Java
196
star
50

KnowledgeablePromptTuning

kpt code
Python
192
star
51

CANE

Source code and datasets of "CANE: Context-Aware Network Embedding for Relation Modeling"
Python
191
star
52

JointNRE

Joint Neural Relation Extraction with Text and KGs
Python
187
star
53

HATT-Proto

Code and dataset of AAAI2019 paper Hybrid Attention-Based Prototypical Networks for Noisy Few-Shot Relation Classification
Python
185
star
54

LegalPLMs

Source code and checkpoints for legal pre-trained language models.
Python
169
star
55

NLP-THU

NLP Course Material & QA
168
star
56

KernelGAT

The source codes for Fine-grained Fact Verification with Kernel Graph Attention Network.
Python
161
star
57

PTR

Prompt Tuning with Rules
Python
155
star
58

EntityDuetNeuralRanking

Entity-Duet Neural Ranking Model
Python
153
star
59

OOP-THU

OOP Course Material & QA
149
star
60

OpenBackdoor

An open-source toolkit for textual backdoor attack and defense (NeurIPS 2022 D&B, Spotlight)
Python
148
star
61

Auto_CLIWC

Code for Chinese LIWC Lexicon Expansion via Hierarchical Classification of Word Embeddings with Sememe Attention (AAAI18)
Python
142
star
62

attribute_charge

The source code of our COLING'18 paper "Few-Shot Charge Prediction with Discriminative Legal Attributes".
Python
128
star
63

ConceptFlow

Python
119
star
64

THUCKE

THU Chinese Keyphrase Extraction Toolkit
C++
118
star
65

CAIL2018

Python
112
star
66

Neural-Snowball

Code and dataset of AAAI2020 Paper Neural Snowball for Few-Shot Relation Learning
Python
112
star
67

KR-EAR

Knowledge Representation Learning with Entities, Attributes and Relations
C++
111
star
68

ChatEval

Codes for our paper "ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate"
Python
109
star
69

MultiRD

Code and data of the AAAI-20 paper "Multi-channel Reverse Dictionary Model"
Python
106
star
70

TransNet

Source code and datasets of IJCAI2017 paper "TransNet: Translation-Based Network Representation Learning for Social Relation Extraction".
Jupyter Notebook
103
star
71

RE-Context-or-Names

Bert-based models(BERT, MTB, CP) for relation extraction.
Python
101
star
72

AGE

Source code and dataset for KDD 2020 paper "Adaptive Graph Encoder for Attributed Graph Embedding"
Python
99
star
73

TopJudge

Python
97
star
74

Prompt-Transferability

On Transferability of Prompt Tuning for Natural Language Processing
Python
97
star
75

GEAR

Source code for ACL 2019 paper "GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification"
Python
95
star
76

HNRE

Hierarchical Neural Relation Extraction
Python
95
star
77

LEVEN

Source code and dataset for ACL2022 Findings Paper "LEVEN: A Large-Scale Chinese Legal Event Detection dataset"
Python
94
star
78

SememePSO-Attack

Code and data of the ACL 2020 paper "Word-level Textual Adversarial Attacking as Combinatorial Optimization"
Python
86
star
79

HMEAE

Source code for EMNLP-IJCNLP 2019 paper "HMEAE: Hierarchical Modular Event Argument Extraction".
Python
85
star
80

XQA

Dataset and baseline for ACL 2019 paper "XQA: A Cross-lingual Open-domain Question Answering Dataset"
Python
84
star
81

ERICA

Source code for ACL 2021 paper "ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning"
Python
83
star
82

CLAIM

78
star
83

TKRL

Representation Learning of Knowledge Graphs with Hierarchical Types (IJCAI-2016)
C++
76
star
84

TLNN

Source code for EMNLP-IJCNLP 2019 paper "Event Detection with Trigger-Aware Lattice Neural Network".
Python
75
star
85

NeuIRPapers

Must-read Papers on Neural Information Retrieval
72
star
86

MMDW

Max-margin DeepWalk
Java
71
star
87

KV-PLM

Source code for "A Deep-learning System Bridging Molecule Structure and Biomedical Text with Comprehension Comparable to Human Professionals"
Python
71
star
88

KNET

Neural Entity Typing with Knowledge Attention
Python
69
star
89

SelectiveMasking

Source code for "Train No Evil: Selective Masking for Task-Guided Pre-Training"
Python
68
star
90

MoEfication

Python
66
star
91

Adv-ED

Source code and dataset for NAACL 2019 paper "Adversarial Training for Weakly Supervised Event Detection".
Python
66
star
92

CorefBERT

Source code for EMNLP 2020 paper "Coreferential Reasoning Learning for Language Representation"
Python
65
star
93

ConversationQueryRewriter

Code and Data for SIGIR 2020 Paper "Few-Shot Generative Conversational Query Rewriting"
Roff
63
star
94

Ouroboros

Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)
Python
62
star
95

MuGNN

Source code for ACL2019 paper "Multi-Channel Graph Neural Network for Entity Alignment".
Python
61
star
96

sememe_prediction

Codes for Lexical Sememe Prediction via Word Embeddings and Matrix Factorization (IJCAI 2017).
Python
60
star
97

DIAG-NRE

Source code for ACL 2019 paper "DIAG-NRE: A Neural Pattern Diagnosis Framework for Distantly Supervised Neural Relation Extraction".
Python
59
star
98

topical_word_embeddings

Topical Word Embeddings
Python
57
star
99

QuoteR

Official code and data of the ACL 2022 paper "QuoteR: A Benchmark of Quote Recommendation for Writing"
Python
57
star
100

paragraph2vec

Paragraph Vector Implementation
Python
56
star