• Stars
    star
    184
  • Rank 209,144 (Top 5 %)
  • Language
    Python
  • License
    MIT License
  • Created almost 4 years ago
  • Updated over 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Codes for paper "Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis"

Python 3.6

SELF-MM

Pytorch implementation for codes in Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis (AAAI2021). Please see our another repo MMSA for more details, which is a scalable framework for MSA.

Model

model

Usage

  1. Datasets and pre-trained berts

Download dataset features and pre-trained berts from the following links.

For all features, you can use SHA-1 Hash Value to check the consistency.

MOSI/unaligned_50.pkl: 5da0b8440fc5a7c3a457859af27458beb993e088
MOSI/aligned_50.pkl: 5c62b896619a334a7104c8bef05d82b05272c71c
MOSEI/unaligned_50.pkl: db3e2cff4d706a88ee156981c2100975513d4610
MOSEI/aligned_50.pkl: ef49589349bc1c2bc252ccc0d4657a755c92a056
SIMS/unaligned_39.pkl: a00c73e92f66896403c09dbad63e242d5af756f8

Due to the size limitations, the MOSEI features and SIMS raw videos are available in Baidu Cloud Drive only. All dataset features are organized as:

{
    "train": {
        "raw_text": [],
        "audio": [],
        "vision": [],
        "id": [], # [video_id$_$clip_id, ..., ...]
        "text": [],
        "text_bert": [],
        "audio_lengths": [],
        "vision_lengths": [],
        "annotations": [],
        "classification_labels": [], # Negative(< 0), Neutral(0), Positive(> 0)
        "regression_labels": []
    },
    "valid": {***}, # same as the "train" 
    "test": {***}, # same as the "train"
}

For MOSI and MOSEI, the pre-extracted text features are from BERT, different from the original glove features in the CMU-Multimodal-SDK.

For SIMS, if you want to extract features from raw videos, you need to install Openface Toolkits first, and then refer our codes in the data/DataPre.py.

python data/DataPre.py --data_dir [path_to_Dataset] --language ** --openface2Path  [path_to_FeatureExtraction]

For bert models, you also can download Bert-Base, Chinese from Google-Bert. And then, convert tensorflow into pytorch using transformers-cli

  1. Clone this repo and install requirements.
git clone https://github.com/thuiar/Self-MM
cd Self-MM
conda create --name self_mm python=3.7
source activate self_mm
pip install -r requirements.txt
  1. Make some changes Modify the config/config_tune.py and config/config_regression.py to update dataset pathes.

  2. Run codes

python run.py --modelName self_mm --datasetName mosi

Results

Detailed results are shown in MMSA > results/result-stat.md.

Paper


Please cite our paper if you find our work useful for your research:

@inproceedings{yu2021le,
  title={Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis},
  author={Yu, Wenmeng and Xu, Hua and Ziqi, Yuan and Jiele, Wu},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2021}
}

More Repositories

1

MMSA

MMSA is a unified framework for Multimodal Sentiment Analysis.
Python
654
star
2

GNN-GBDT-Guided-Fast-Optimizing-Framework

GNN&GBDT-Guided Fast Optimizing Framework for Large-scale Integer Programming(Ye et al., ICML 2023): https://openreview.net/pdf?id=tX7ajV69wt
Python
312
star
3

TEXTOIR

TEXTOIR is the first opensource toolkit for text open intent recognition. (ACL 2021)
Python
195
star
4

MMSA-FET

A Tool for extracting multimodal features from videos.
Python
133
star
5

DeepAligned-Clustering

Discovering New Intents with Deep Aligned Clustering (AAAI 2021)
Python
120
star
6

OKD-Reading-List

Papers for Open Knowledge Discovery
TeX
117
star
7

Cross-Modal-BERT

CM-BERT: Cross-Modal BERT for Text-Audio Sentiment Analysis(MM2020οΌ‰
Python
103
star
8

AWESOME-MSA

Paper List for Multimodal Sentiment Analysis
94
star
9

M-SENA

M-SENA: All-in-One Platform for Multimodal Sentiment Analysis
79
star
10

MIntRec

MIntRec: A New Dataset for Multimodal Intent Recognition (ACM MM 2022)
Python
74
star
11

Adaptive-Decision-Boundary

Deep Open Intent Classification with Adaptive Decision Boundary (AAAI 2021)
Python
70
star
12

ch-sims-v2

Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module
Python
54
star
13

TEXTOIR-DEMO

TEXTOIR: An Integrated and Visualized Platform for Text Open Intent Recognition (ACL 2021)
JavaScript
46
star
14

CDAC-plus

Discovering New Intents via Constrained Deep Adaptive Clustering with Cluster Refinement (AAAI2020)
Jupyter Notebook
44
star
15

DeepUnkID

Deep Unknown Intent Detection with Margin Loss (ACL2019)
Jupyter Notebook
34
star
16

TCL-MAP

TCL-MAP is a powerful method for multimodal intent recognition (AAAI 2024)
Python
31
star
17

CRL

Implementation of the research paper Consistent Representation Learning for Continual Relation Extraction (Findings of ACL 2022)
Python
25
star
18

TFR-Net

This repository contains the official implementation code of the paper Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment Analysis, accepted at ACMMM 2021.
Python
25
star
19

MIntRec2.0

MIntRec 2.0 is the first large-scale dataset for multimodal intent recognition and out-of-scope detection in multi-party conversations (ICLR 2024)
Python
20
star
20

OpenVNA

[ACL 2024 SDT] OpenVNA is an open-source framework designed for analyzing the behavior of multimodal language understanding systems under noisy conditions.
Python
15
star
21

AWESOME-Dialogue

Paper List for Dialogue and Interactive Systems
15
star
22

UMC

Unsupervised Multimodal Clustering for Semantics Discovery in Multimodal Utterances (ACL 2024)
Python
14
star
23

thuiar.github.io

The official website of THUIAR.
HTML
10
star
24

Books

JavaScript
8
star
25

Robust-MSA

JavaScript
7
star
26

CTMWA

Crossmodal Translation based Meta Weight Adaption for Robust Image-Text Sentiment Analysis
Python
7
star
27

Meta-NA

Pytorch implementation for codes in Meta Noise Adaption Framework for Multimodal Sentiment Analysis with Feature Noise (Accepted by IEEE Transactions on Multimedia).
Python
6
star
28

TCM-CAS

Traditional Chinese Medicine Constitution Assessment System
4
star
29

Adaptive-Batch-ParEGO

This repository contains Matlab implementation of the algorithm framework for adaptive batch-ParEGO
MATLAB
3
star
30

MILPGen

Python
2
star
31

AWESOME-MTL

Paper List for Multi-task Learning
2
star
32

cmcnn

code for paper "Co-attentive multi-task convolutional neural network for facial expression recognition"
Python
2
star
33

Expensive-Multi-objective-Optimization

2
star
34

Light-MILPopt

2
star
35

AudioProcess

Related methods and tools for processing audio data
C++
1
star
36

Block-MOBO

This repository contains Matlab implementation of the algorithm framework for Block-MOBO.
MATLAB
1
star
37

ML4MILP

ML4MILP: the first benchmark dataset specifically designed to test ML-based algorithms for solving MILP problems
Python
1
star
38

GAR-Net

GAR-Net: A Graph Attention Reasoning Network for Conversation Understanding
Python
1
star