declare-lab/contextual-utterance-level-multimodal-sentiment-analysis declare-lab/contextual-utterance-level-multimodal-sentiment-analysis

  • Stars
    star
    123
  • Rank 290,145 (Top 6 %)
  • Language
    Python
  • Created over 7 years ago
  • Updated almost 2 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
declare-lab/contextual-utterance-level-multimodal-sentiment-analysis
github

declare-lab

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Context-Dependent Sentiment Analysis in User-Generated Videos

πŸ”₯ πŸ”₯ πŸ”₯ Notice: This repository will no longer be maintained. Instead, we are are moving all our multimodal works to this new centralized repository: https://github.com/declare-lab/multimodal-deep-learning.

Context-Dependent Sentiment Analysis in User-Generated Videos

Code for the paper Context-Dependent Sentiment Analysis in User-Generated Videos (ACL 2017).

NOTE: Here is the updated version of the code - https://github.com/soujanyaporia/multimodal-sentiment-analysis

Requirements

Code is written in Python (2.7) and requires Keras (2.0.6) with Theano backend.

Description

In this paper, we propose a LSTM-based model that enables utterances to capture contextual information from their surroundings in the same video, thus aiding the classification process in multimodal sentiment analysis.

Alt text

This repository contains the code for the mentioned paper. Each contextual LSTM (Figure 2 in the paper) is implemented as shown in above figure. For more details, please refer to the paper.
Note: Unlike the paper, we haven't used an SVM on the penultimate layer. This is in effort to keep the whole network differentiable at some performance cost.

Dataset

We provide results on the MOSI dataset
Please cite the creators

Preprocessing

As data is typically present in utterance format, we combine all the utterances belonging to a video using the following code

python create_data.py

Note: This will create speaker independent train and test splits

Running sc-lstm

Sample command:

python lstm.py --unimodal True
python lstm.py --unimodal False

Note: Keeping the unimodal flag as True (default False) shall train all unimodal lstms first (level 1 of the network mentioned in the paper)

Citation

If using this code, please cite our work using :

@inproceedings{soujanyaacl17,
  title={Context-dependent sentiment analysis in user-generated videos},
  author={Poria, Soujanya  and Cambria, Erik and Hazarika, Devamanyu and Mazumder, Navonil and Zadeh, Amir and Morency, Louis-Philippe},
  booktitle={Association for Computational Linguistics},
  year={2017}
}

Credits

Devamanyu Hazarika, Soujanya Poria

More Repositories

1

conv-emotion

This repo contains implementation of different architectures for emotion recognition in conversations.
Python
1,336
star
2

MELD

MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversation
Python
793
star
3

tango

Codes and Model of the paper "Text-to-Audio Generation using Instruction Tuned LLM and Latent Diffusion Model"
Python
754
star
4

multimodal-deep-learning

This repository contains various models targetting multimodal representation learning, multimodal fusion for downstream tasks such as multimodal sentiment analysis.
OpenEdge ABL
730
star
5

awesome-sentiment-analysis

Reading list for Awesome Sentiment Analysis papers
517
star
6

instruct-eval

This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks.
Python
500
star
7

flan-alpaca

This repository contains code for extending the Stanford Alpaca synthetic instruction tuning to existing instruction-tuned models such as Flan-T5.
Python
343
star
8

awesome-emotion-recognition-in-conversations

A comprehensive reading list for Emotion Recognition in Conversations
252
star
9

MISA

MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Python
192
star
10

RECCON

This repository contains the dataset and the PyTorch implementations of the models from the paper Recognizing Emotion Cause in Conversations.
Python
169
star
11

Multimodal-Infomax

This repository contains the official implementation code of the paper Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis, accepted at EMNLP 2021.
Python
154
star
12

dialogue-understanding

This repository contains PyTorch implementation for the baseline models from the paper Utterance-level Dialogue Understanding: An Empirical Study
Python
123
star
13

RelationPrompt

This repository implements our ACL Findings 2022 research paper RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extraction. The goal of Zero-Shot Relation Triplet Extraction (ZeroRTE) is to extract relation triplets of the format (head entity, tail entity, relation), despite not having annotated data for the test relation labels.
Python
123
star
14

flacuna

Flacuna was developed by fine-tuning Vicuna on Flan-mini, a comprehensive instruction collection encompassing various tasks. Vicuna is already an excellent writing assistant, and the intention behind Flacuna was to enhance Vicuna's problem-solving capabilities. To achieve this, we curated a dedicated instruction dataset called Flan-mini.
Python
106
star
15

CASCADE

This repo contains code to detect sarcasm from text in discussion forum using deep learning
Python
86
star
16

red-instruct

Codes and datasets of the paper Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment
Python
75
star
17

BBFN

This repository contains the implementation of the paper -- Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis
Python
62
star
18

CICERO

The purpose of this repository is to introduce new dialogue-level commonsense inference datasets and tasks. We chose dialogues as the data source because dialogues are known to be complex and rich in commonsense.
Python
60
star
19

dialog-HGAT

Dialogue Relation Extraction with Document-level Heterogeneous Graph Attention Networks
Python
57
star
20

kingdom

Domain Adaptation using External Knowledge for Sentiment Analysis
Python
55
star
21

hfusion

Multimodal sentiment analysis using hierarchical fusion with context modeling
Python
44
star
22

MIME

This repository contains PyTorch implementations of the models from the paper An Empirical Study MIME: MIMicking Emotions for Empathetic Response Generation.
Python
43
star
23

speech-adapters

Codes and datasets for our ICASSP2023 paper, Evaluating parameter-efficient transfer learning approaches on SURE benchmark for speech understanding
Python
40
star
24

LLM-PuzzleTest

This repository is maintained to release dataset and models for multimodal puzzle reasoning.
Python
33
star
25

HyperTTS

Python
33
star
26

MSA-Robustness

NAACL 2022 paper on Analyzing Modality Robustness in Multimodal Sentiment Analysis
Python
31
star
27

CIDER

This repository contains the dataset and the pytorch implementations of the models from the paper CIDER: Commonsense Inference for Dialogue Explanation and Reasoning. CIDER has been accepted to appear at SIGDIAL 2021.
Python
28
star
28

sentence-ordering

This repository contains the PyTorch implementation of the paper STaCK: Sentence Ordering with Temporal Commonsense Knowledge appearing at EMNLP 2021.
Python
28
star
29

resta

Restore safety in fine-tuned language models through task arithmetic
Python
25
star
30

HyperRED

This repository implements our EMNLP 2022 research paper A Dataset for Hyper-Relational Extraction and a Cube-Filling Approach.
Python
25
star
31

MM-InstructEval

This repository contains code to evaluate various multimodal large language models using different instructions across multiple multimodal content comprehension tasks.
Python
24
star
32

MM-Align

[EMNLP 2022] This repository contains the official implementation of the paper "MM-Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequences"
Python
24
star
33

identifiable-transformers

Python
22
star
34

exemplary-empathy

This repository contains the source codes of the paper -- Exemplars-guided Empathetic Response Generation Controlled by the Elements of Human Communication
Python
22
star
35

della

DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling
Python
22
star
36

TEAM

Our EMNLP 2022 paper on MCQA
Python
21
star
37

DoubleMix

Code for the COLING 2022 paper "DoubleMix: Simple Interpolation-Based Data Augmentation for Text Classification"
Python
20
star
38

adapter-mix

Python
15
star
39

M2H2-dataset

This repository contains the dataset and baselines explained in the paper: M2H2: A Multimodal Multiparty Hindi Dataset For HumorRecognition in Conversations
Python
15
star
40

ASTE-RL

This repository contains the source codes for the paper: "Aspect Sentiment Triplet Extraction using Reinforcement Learning" published at CIKM 2021.
Python
14
star
41

SAT

Code for the EMNLP 2022 Findings short paper "SAT: Improving Semi-Supervised Text Classification with Simple Instance-Adaptive Self-Training"
Jupyter Notebook
13
star
42

KNOT

This repository contains the implementation of the paper -- KNOT: Knowledge Distillation using Optimal Transport for Solving NLP Tasks
Python
13
star
43

WikiDes

A Wikipedia-based summarization dataset
Python
13
star
44

Sealing

[NAACL 2024] Official Implementation of paper "Self-Adaptive Sampling for Efficient Video Question Answering on Image--Text Models"
Python
9
star
45

VIP

Our EMNLP 2022 paper on VIP-Based Prompting for Parameter-Efficient Learning
Python
8
star
46

domadapter

Code for EACL'23 paper "Udapter: Efficient Domain Adaptation Using Adapters"
Python
8
star
47

RobustMIFT

[Arxiv 2024] Official Implementation of the paper: "Towards Robust Instruction Tuning on Multimodal Large Language Models"
Jupyter Notebook
8
star
48

ferret

Ferret: Faster and Effective Automated Red Teaming with Reward-Based Scoring Technique
Python
8
star
49

SANCL

[COLING 2022] This repository contains the code of the paper SANCL: Multimodal Review Helpfulness Prediction with Selective Attention and Natural Contrastive Learning.
Python
7
star
50

segue

Codes and Checkpoints of the Interspeech paper "Sentence Embedder Guided Utterance Encoder (SEGUE) for Spoken Language Understanding"
Python
6
star
51

llm_robustness

Python
3
star
52

NLP-OT

1
star