declare-lab/dialogue-understanding declare-lab/dialogue-understanding

  • Stars
    star
    123
  • Rank 290,145 (Top 6 %)
  • Language
    Python
  • License
    MIT License
  • Created over 4 years ago
  • Updated almost 2 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
declare-lab/dialogue-understanding
github

declare-lab

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

This repository contains PyTorch implementation for the baseline models from the paper Utterance-level Dialogue Understanding: An Empirical Study

Utterance-level Dialogue Understanding

This repository contains pytorch implementations of the models from the paper Utterance-level Dialogue Understanding: An Empirical Study, and Exploring the Role of Context in Utterance-level Emotion, Act and Intent Classification in Conversations: An Empirical Study

Alt text

Task Definition

Given the transcript of a conversation along with speaker information of each constituent utterance, the utterance-level dialogue understanding (utterance-level dialogue understanding) task aims to identify the label of each utterance from a set of pre-defined labels that can be either a set of emotions, dialogue acts, intents etc. The figures above and below illustrate such conversations between two people, where each utterance is labeled by the underlying emotion and intent. Formally, given the input sequence of N number of utterances [(u1, p1), (u2,p2),...., (uN,pN)], where each utterance ui=[ui,1,ui,2,.....,ui,T] consists of T words ui,j and spoken by party pi, the task is to predict the label ei of each utterance ui. In this process, the classifier can also make use of the conversational context. There are also cases where not all the utterances in a dialogue have corresponding labels.

Emotion Intent

Data Format

The models are all trained in an end-to-end fashion. The utterances, labels, loss masks, and speaker-specific information are thus read directly from tab separated text files. All data files follow the common format:

Utterances: Each line contains tab separated dialogue id and the utterances of the dialogue.

train_1    How do you like the pizza here?    Perfect. It really hits the spot.
train_2    Do you have any trains into Cambridge today?    There are many trains, when would you like to depart?    I would like to leave from Kings Lynn after 9:30.

Lables: Each line contains tab separated dialogue id and the encoded emotion/intent/act/strategy labels of the dialogue.

train_1    0    1
train_2    2    0    2

Loss masks: Each line contains tab separated dialogue id and the loss mask information of the utterances of the dialogue. For contextual models, the whole sequence of utterance is passed as input, but utterances having loss mask of 0 are not considered for the calculation of loss and the calcualtion of classification metrics. This is required for tasks where we want to use the full contextual information as input, but don't want to classify a subset of utterances in the output. For example, in MultiWOZ intent classification, we pass the full dialogue sequence as input but don't classify the utterances coming from the system side.

train_1    1    1
train_2    1    0    1

Speakers: Each line contains tab separated dialogue id and the speakers of the dialogue.

train_1    0    1
train_2    0    1    0

Datasets

The original datasets can be found in the glove-end-to-end/datasets/_original and roberta-end-to-end/datasets/_original directories. The following datasets are included in this project:

 IEMOCAP (Emotion Recognition in Conversations)
 DailyDialog (Emotion Recognition in Conversations)
 DailyDialog (Act Classification)
 MultiWOZ (Intent Recognition)
 Persuasion for Good (Persuader's act classification)
 Persuasion for Good (Persuadee's act classification)

Utterance-level minibatch vs Dialogue-level minibatch

The original datasets are experimented using two distinct minibatch formation techniques. They can be found in these directories glove-end-to-end/datasets/dialogue_level_minibatch, glove-end-to-end/datasets/utterance_level_minibatch, and roberta-end-to-end/datasets/dialogue_level_minibatch. Please refer to the paper for more information.

glove-end-to-end/datasets/dialogue_level_minibatch : This folder contains the dataset that can be used to prepare minibatches where all the utterances in the context having a valid label are classified.

glove-end-to-end/datasets/utterance_level_minibatch : This folder contains the dataset that can be used to prepare minibatches where only one utterance in the context is classified and rest of the utterances are used as context.

Alt text

Context Perturbations of the Original Data

Along with the original datasets above, we also provide different variations of these datasets with context perturbations such as style transfer, paraphrasing, label augmentation, etc. Readers can find context perturbed versions of the original datasets in the following folders and refer to the paper to find relevant information.

glove-end-to-end/datasets/inter_speaker
glove-end-to-end/datasets/label_augmentation
glove-end-to-end/datasets/paraphrase_attack
glove-end-to-end/datasets/spelling_attack
glove-end-to-end/datasets/style_transfer

Models

We provide implementations for end-to-end without context classifier, bcLSTM and DialogueRNN models. For bcLSTM and DialogueRNN, we also provide training argument which lets you specify whether to use residual connections or not.

RoBERTa Based Models

Navigate to roberta-end-to-end. We also provide training arguments with which you can alternately use BERT or Sentence Transformers models as feature extractors.

GloVe Based Models

Navigate to glove-end-to-end. We have also released scripts with which you can run different analysis experiments that we report in the paper.

Alt text

Execution

Once navigate to roberta-end-to-end or glove-end-to-end directories to use RoBERTa or GloVe based feature extractors for the models, run the following commands to execute different models explained in the paper. Note that some of the models present in the glove-end-to-end folder are not available in the roberta-end-to-end folder. However, it should not be difficult to adapt these models to use RoBERTa embeddings.

Main Model (Dialogue-Level Minibatch)

To train and evaluate the without context classifier model and the bcLSTM/DialogueRNN model with full context and residual connections:

python train.py --dataset [iemocap|dailydialog|multiwoz|persuasion] --classify [emotion|act|intent|er|ee] --cls-model [logreg|lstm|dialogrnn] --residual

The --cls-model logreg corresponds to the without context classifier.

Main Model (Utterance-level Minibatch)

python train_utt_level.py --dataset [iemocap|dailydialog|multiwoz|persuasion] --classify [emotion|act|intent|er|ee] --cls-model [logreg|lstm|dialogrnn] --residual

Speaker Level Models

w/o inter : Trained at dialogue-level minibatch. To train and evaluate bcLSTM model in this setting i.e. only with context from the same speaker:

python train_intra_speaker.py --dataset [iemocap|dailydialog|multiwoz|persuasion] --classify [emotion|act|intent|er|ee] --residual

w/o intra : Trained at utterance-level minibatch. To train and evaluate bcLSTM model in this setting i.e. only with context from the other speaker:

python train_inter_speaker.py --dataset [iemocap|dailydialog|multiwoz|persuasion] --classify [emotion|act|intent|er|ee] --residual

Shuffled Context and Shuffled Context with Order Prediction Models

Trained at dialogue-level minibatch. To train and evaluate bcLSTM model with various shuffling strategies in train, val, test:

python train_shuffled_context.py --dataset [iemocap|dailydialog|multiwoz|persuasion] --classify [emotion|act|intent|er|ee] --residual --shuffle [0|1|2]

--shuffle 0 : Shuffled context in train, val, test.

--shuffle 1 : Shuffled context in train, val; original context in test.

--shuffle 2 : Original context in train, val; shuffled context in test.

Context Control Models

Trained at utterance-level minibatch. The script is train_context_control.py. You can specify training arguments to determine how to control the context.

Baseline Results

Note

If you are running GloVe-based end-to-end models, please run the scripts multiple times and average the test scores of those runs.

Citation

Utterance-level Dialogue Understanding: An Empirical Study. Deepanway Ghosal, Navonil Majumder, Rada Mihalcea, Soujanya Poria. arXiv preprint arXiv:2009.13902 (2020).

Exploring the Role of Context in Utterance-level Emotion, Act and IntentClassification in Conversations: An Empirical Study. Deepanway Ghosal, Navonil Majumder, Rada Mihalcea, Soujanya Poria. Findings of ACL (2021).

More Repositories

1

conv-emotion

This repo contains implementation of different architectures for emotion recognition in conversations.
Python
1,336
star
2

MELD

MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversation
Python
793
star
3

tango

Codes and Model of the paper "Text-to-Audio Generation using Instruction Tuned LLM and Latent Diffusion Model"
Python
754
star
4

multimodal-deep-learning

This repository contains various models targetting multimodal representation learning, multimodal fusion for downstream tasks such as multimodal sentiment analysis.
OpenEdge ABL
730
star
5

awesome-sentiment-analysis

Reading list for Awesome Sentiment Analysis papers
517
star
6

instruct-eval

This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks.
Python
500
star
7

flan-alpaca

This repository contains code for extending the Stanford Alpaca synthetic instruction tuning to existing instruction-tuned models such as Flan-T5.
Python
343
star
8

awesome-emotion-recognition-in-conversations

A comprehensive reading list for Emotion Recognition in Conversations
252
star
9

MISA

MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Python
192
star
10

RECCON

This repository contains the dataset and the PyTorch implementations of the models from the paper Recognizing Emotion Cause in Conversations.
Python
169
star
11

Multimodal-Infomax

This repository contains the official implementation code of the paper Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis, accepted at EMNLP 2021.
Python
154
star
12

RelationPrompt

This repository implements our ACL Findings 2022 research paper RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extraction. The goal of Zero-Shot Relation Triplet Extraction (ZeroRTE) is to extract relation triplets of the format (head entity, tail entity, relation), despite not having annotated data for the test relation labels.
Python
123
star
13

contextual-utterance-level-multimodal-sentiment-analysis

Context-Dependent Sentiment Analysis in User-Generated Videos
Python
123
star
14

flacuna

Flacuna was developed by fine-tuning Vicuna on Flan-mini, a comprehensive instruction collection encompassing various tasks. Vicuna is already an excellent writing assistant, and the intention behind Flacuna was to enhance Vicuna's problem-solving capabilities. To achieve this, we curated a dedicated instruction dataset called Flan-mini.
Python
106
star
15

CASCADE

This repo contains code to detect sarcasm from text in discussion forum using deep learning
Python
86
star
16

red-instruct

Codes and datasets of the paper Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment
Python
75
star
17

BBFN

This repository contains the implementation of the paper -- Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis
Python
62
star
18

CICERO

The purpose of this repository is to introduce new dialogue-level commonsense inference datasets and tasks. We chose dialogues as the data source because dialogues are known to be complex and rich in commonsense.
Python
60
star
19

dialog-HGAT

Dialogue Relation Extraction with Document-level Heterogeneous Graph Attention Networks
Python
57
star
20

kingdom

Domain Adaptation using External Knowledge for Sentiment Analysis
Python
55
star
21

hfusion

Multimodal sentiment analysis using hierarchical fusion with context modeling
Python
44
star
22

MIME

This repository contains PyTorch implementations of the models from the paper An Empirical Study MIME: MIMicking Emotions for Empathetic Response Generation.
Python
43
star
23

speech-adapters

Codes and datasets for our ICASSP2023 paper, Evaluating parameter-efficient transfer learning approaches on SURE benchmark for speech understanding
Python
40
star
24

LLM-PuzzleTest

This repository is maintained to release dataset and models for multimodal puzzle reasoning.
Python
33
star
25

HyperTTS

Python
33
star
26

MSA-Robustness

NAACL 2022 paper on Analyzing Modality Robustness in Multimodal Sentiment Analysis
Python
31
star
27

CIDER

This repository contains the dataset and the pytorch implementations of the models from the paper CIDER: Commonsense Inference for Dialogue Explanation and Reasoning. CIDER has been accepted to appear at SIGDIAL 2021.
Python
28
star
28

sentence-ordering

This repository contains the PyTorch implementation of the paper STaCK: Sentence Ordering with Temporal Commonsense Knowledge appearing at EMNLP 2021.
Python
28
star
29

resta

Restore safety in fine-tuned language models through task arithmetic
Python
25
star
30

HyperRED

This repository implements our EMNLP 2022 research paper A Dataset for Hyper-Relational Extraction and a Cube-Filling Approach.
Python
25
star
31

MM-InstructEval

This repository contains code to evaluate various multimodal large language models using different instructions across multiple multimodal content comprehension tasks.
Python
24
star
32

MM-Align

[EMNLP 2022] This repository contains the official implementation of the paper "MM-Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequences"
Python
24
star
33

identifiable-transformers

Python
22
star
34

exemplary-empathy

This repository contains the source codes of the paper -- Exemplars-guided Empathetic Response Generation Controlled by the Elements of Human Communication
Python
22
star
35

della

DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling
Python
22
star
36

TEAM

Our EMNLP 2022 paper on MCQA
Python
21
star
37

DoubleMix

Code for the COLING 2022 paper "DoubleMix: Simple Interpolation-Based Data Augmentation for Text Classification"
Python
20
star
38

adapter-mix

Python
15
star
39

M2H2-dataset

This repository contains the dataset and baselines explained in the paper: M2H2: A Multimodal Multiparty Hindi Dataset For HumorRecognition in Conversations
Python
15
star
40

ASTE-RL

This repository contains the source codes for the paper: "Aspect Sentiment Triplet Extraction using Reinforcement Learning" published at CIKM 2021.
Python
14
star
41

SAT

Code for the EMNLP 2022 Findings short paper "SAT: Improving Semi-Supervised Text Classification with Simple Instance-Adaptive Self-Training"
Jupyter Notebook
13
star
42

KNOT

This repository contains the implementation of the paper -- KNOT: Knowledge Distillation using Optimal Transport for Solving NLP Tasks
Python
13
star
43

WikiDes

A Wikipedia-based summarization dataset
Python
13
star
44

Sealing

[NAACL 2024] Official Implementation of paper "Self-Adaptive Sampling for Efficient Video Question Answering on Image--Text Models"
Python
9
star
45

VIP

Our EMNLP 2022 paper on VIP-Based Prompting for Parameter-Efficient Learning
Python
8
star
46

domadapter

Code for EACL'23 paper "Udapter: Efficient Domain Adaptation Using Adapters"
Python
8
star
47

RobustMIFT

[Arxiv 2024] Official Implementation of the paper: "Towards Robust Instruction Tuning on Multimodal Large Language Models"
Jupyter Notebook
8
star
48

ferret

Ferret: Faster and Effective Automated Red Teaming with Reward-Based Scoring Technique
Python
8
star
49

SANCL

[COLING 2022] This repository contains the code of the paper SANCL: Multimodal Review Helpfulness Prediction with Selective Attention and Natural Contrastive Learning.
Python
7
star
50

segue

Codes and Checkpoints of the Interspeech paper "Sentence Embedder Guided Utterance Encoder (SEGUE) for Spoken Language Understanding"
Python
6
star
51

llm_robustness

Python
3
star
52

NLP-OT

1
star