ml-jku/cloob ml-jku/cloob

  • Stars
    star
    152
  • Rank 244,685 (Top 5 %)
  • Language
    Python
  • License
    Other
  • Created about 3 years ago
  • Updated over 2 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
ml-jku/cloob
github

ml-jku

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP

Andreas FΓΌrst* 1, Elisabeth Rumetshofer* 1, Johannes Lehner1, Viet Tran1, Fei Tang3, Hubert Ramsauer1, David Kreil2, Michael Kopp2, GΓΌnter Klambauer1, Angela Bitto-Nemling1, Sepp Hochreiter1 2

1 ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria
2 Institute of Advanced Research in Artificial Intelligence (IARAI)
3 HERE Technologies
* Equal contribution

Detailed blog post on this paper at this link.

The full paper is available here.

Implementation of CLOOB

This repository contains the implemenation of CLOOB used to obtain the results reported in the paper. The implementation is based on OpenCLIP, an open source implementation of OpenAI's CLIP.

Setup

We provide an 'environment.yml' file to set up a conda environment with all required packages. Run the following command to clone the repository and create the environment.

# Clone repository and swtich into the directory
git clone https://github.com/ml-jku/cloob
cd cloob

# Create the environment and activate it
conda env create --file environment.yml
conda activate cloob

# Add the directory to the PYTHONPATH environment variable
export PYTHONPATH="$PYTHONPATH:$PWD/src"

Data

For pre-training we use the two datasets supported by OpenCLIP, namely Conceptual Captions and YFCC.

Conceptual Captions

OpenCLIP already provides a script to download and prepare the Conceptual Captions dataset, which contains 2.89M training images and 13k validation images. First, download the Conceptual Captions URLs and then run the script gather_cc.py.

python3 src/data/gather_cc.py path/to/Train_GCC-training.tsv path/to/Validation_GCC-1.1.0-Validation.tsv

YFCC

We use the same subset of ~15M images from the YFCC100M dataset as CLIP. They provide a list of (line number, photo identifier, photo hash) of each image contained in this subset here.

For more information see YFCC100m Subset on OpenAI's github.

Downstream Tasks

In the paper we report results on several downstream tasks. Except for ImageNet we provide links to already pre-processed versions (where necessary) of the respective test set.

Dataset Description Official Processed
Birdsnap This dataset contains images of North American bird species, however
our dataset is smaller than reported in CLIP as some samples are no longer available.		 Link Link
Country211 This dataset was published in CLIP and is a small subset of the YFCC100m dataset.
It consists of photos that can be assigned to 211 countries via GPS coordinates.
For each country 200 photos are sampled for the training set and 100 for testing.		 Link Link
Flowers102 Images of 102 flower categories commonly occuring in the United Kingdom were collected.
Several classes are very similar and there is a large variation in scale, pose and lighting.		 Link Link
GTSRB This dataset was released for a challenge held at the IJCNN 2011.
The dataset contains images of german traffic signs from more than 40 classes.		 Link Link
Stanford Cars This dataset contains images of 196 car models at the level of make,
model and year (e.g. Tesla Model S Sedan 2012).		 Link Link
UCF101 The dataset has been created by extracting the middle frame from each video. Link Link
ImageNet This dataset spans 1000 object classes and contains 1,281,167 training images,
50,000 validation images and 100,000 test images.		 Link -
ImageNet v2 The ImageNetV2 dataset contains new test data for the ImageNet benchmark. Link -

Usage

In the following there is an example command for pretraining on CC with an effective batch size of 512 when used on 4 GPUs.

python -u src/training/main.py \
--train-data="<dataset-dir>/conceptual_captions/Train-GCC-training_output.csv" \
--val-data="<dataset-dir>/conceptual_captions/Validation_GCC-1.1.0-Validation_output.csv" \
--path-data="<dataset-dir>/conceptual_captions" \
--imagenet-val="<dataset-dir>/imagenet/val" \
--warmup 20000 \
--batch-size=128 \
--lr=1e-3 \
--wd=0.1 \
--lr-scheduler="cosine-restarts" \
--restart-cycles=10 \
--epochs=70 \
--method="cloob" \
--init-inv-tau=30 \
--scale-hopfield=8 \
--workers=8 \
--model="RN50" \
--dist-url="tcp://127.0.0.1:6100" \
--batch-size-eval=512

Zeroshot evaluation of downstream tasks

We provide a Jupyter notebook to perform zeroshot evaluation with a trained model.

LICENSE

MIT LICENSE

More Repositories

1

hopfield-layers

Hopfield Networks is All You Need
Python
1,660
star
2

hopular

Hopular: Modern Hopfield Networks for Tabular Data
Python
299
star
3

DeepRC

DeepRC: Immune repertoire classification with attention-based deep massive multiple instance learning
Python
112
star
4

clamp

Code for the paper Enhancing Activity Prediction Models in Drug Discovery with the Ability to Understand Human Language
Python
85
star
5

HopCPT

Conformal Prediction for Time Series with Modern Hopfield Networks
Python
67
star
6

GINNs-Geometry-informed-Neural-Networks

Jupyter Notebook
54
star
7

helm

Python
52
star
8

sars-cov-inhibitors-chemai

Large-scale ligand-based virtual screening for potential SARS-Cov-2 inhibitors using a deep neural network
52
star
9

L2M

Learning to Modulate pre-trained Models in RL (Decision Transformer, LoRA, Fine-tuning)
Python
49
star
10

rudder

RUDDER: Return Decomposition for Delayed Rewards
45
star
11

mhn-react

Modern Hopfield Network (MHN) for template relevance prediction
Jupyter Notebook
40
star
12

mc-lstm

Experiments with Mass Conserving LSTMs
Python
38
star
13

UPT

Code for the paper Universal Physics Transformers
Python
37
star
14

hyper-dti

HyperPCM: Robust task-conditioned modeling of drug-target interactions
Python
34
star
15

MIM-Refiner

A Contrastive Learning Boost from Intermediate Pre-Trained Representations
Python
34
star
16

lsc

Large-scale comparison of machine learning methods for drug target prediction on ChEMBL
C++
33
star
17

cloome

Jupyter Notebook
32
star
18

MAE-CT

Python
31
star
19

quam

Quantification of Uncertainty with Adversarial Models
Jupyter Notebook
27
star
20

DIffUCO

Python
26
star
21

hopfield-boosting

Jupyter Notebook
25
star
22

semantic-image-text-alignment

Python
24
star
23

SDLG

SDLG is an efficient method to accurately estimate aleatoric semantic uncertainty in LLMs
Jupyter Notebook
23
star
24

vnegnn

Python
22
star
25

rudder-demonstration-code

Code for demonstration example-task in RUDDER blog
Python
21
star
26

align-rudder

Code to reproduce results on toy tasks and companion blog for the paper.
Python
20
star
27

oeaw_ai_summer_school

Introduction to Machine Learning and Neural Networks, including NNs, CNNs, RNNs, AE, VAE, and GAN
19
star
28

chef

Python
15
star
29

subgd

Code for "Few-Shot Learning by Dimensionality Reduction in Gradient Space"
Jupyter Notebook
14
star
30

mgenerators-failure-modes

Shows some of the ways molecule generation and optimization can go wrong
Jupyter Notebook
14
star
31

reactive-exploration

Code for the paper "Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learning"
Python
14
star
32

MHNfs

Jupyter Notebook
13
star
33

hti-cnn

Python
11
star
34

VAG-CO

Python
11
star
35

tensorflow-layer-library

Tensorflow Layer Library (TeLL)
Python
10
star
36

bgnn

Python
9
star
37

gapnet-pl

Python
8
star
38

GNN-VPA

A Variance-Preserving Aggregation Strategy for Graph Neural Networks
Python
5
star
39

melvin

Python
4
star
40

convex-init

Code for Principled Weight Initialisation for Input-Convex Neural Networks
Python
4
star
41

PlattScaling

Python Package for Platt scaling based on the algorithm according to Lin, Lin and Weng
Python
3
star
42

TPDistance

Computation of 3D Triangle Point Distances (on a GPU)
Python
3
star
43

diverse-hits

Code accompanying our paper on diversity-based comparison of goal-directed generators
Jupyter Notebook
3
star
44

covid

Python
2
star
45

haprfn-R

R
2
star
46

LRAM

A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Tasks
Python
1
star
47

ExCAPE

Code for DNNs in the ExCAPE-Project
Python
1
star
48

blog-post-performer

Blog post to the Performer paper including exemplary notebooks
1
star
49

ml-jku.github.io

Repository for the website of Institute for Machine Learning, Linz.
JavaScript
1
star
50

autoregressive_activity_prediction

This repo includes code for the autoregressive activity prediction for low-data drug discovery manuscript
Jupyter Notebook
1
star