• Stars
    star
    138
  • Rank 255,313 (Top 6 %)
  • Language
    Python
  • License
    MIT License
  • Created over 3 years ago
  • Updated over 3 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Implementing Lambda Networks using Pytorch

LambdaNetworks: Modeling long-range Interactions without Attention

Experimnets (CIFAR10)

Model k h u m Params (M) Acc (%)
ResNet18 baseline (ref) 14 93.02
LambdaResNet18 16 4 4 9 8.6 92.21 (70 Epochs)
LambdaResNet18 16 4 4 7 8.6 94.20 (67 Epochs)
LambdaResNet18 16 4 4 5 8.6 91.58 (70 Epochs)
LambdaResNet18 16 4 1 23 8 91.36 (69 Epochs)
ResNet50 baseline (ref) 23.5 93.62
LambdaResNet50 16 4 4 7 13 93.74 (70 epochs)

Usage

import torch

from model import LambdaConv, LambdaResNet50, LambdaResNet152

x = torch.randn([2, 3, 32, 32])
conv = LambdaConv(3, 128)
print(conv(x).size()) # [2, 128, 32, 32]

# reference
# https://discuss.pytorch.org/t/how-do-i-check-the-number-of-parameters-of-a-model/4325
def get_n_params(model):
    pp=0
    for p in list(model.parameters()):
        nn=1
        for s in list(p.size()):
            nn = nn*s
        pp += nn
    return pp

model = LambdaResNet50()
print(get_n_params(model)) # 14.9M (Ours) / 15M(Paper)

model = LambdaResNet152()
print(get_n_params(model)) # 32.8M (Ours) / 35M (Paper)

Parameters

Model k h u m Params (M), Paper Params (M), Ours
LambdaResNet50 16 4 1 23 15.0 14.9
LambdaResNet50 16 4 4 7 16.0 16.0
LambdaResNet152 16 4 1 23 35 32.8
LambdaResNet200 16 4 1 23 42 35.29

Ablation Parameters

k h u Params (M), Paper Params (M), Ours
ResNet baseline 25.6 25.5
8 2 1 14.8 15.0
8 16 1 15.6 14.9
2 4 1 14.7 14.6
4 4 1 14.7 14.66
8 4 1 14.8 14.66
16 4 1 15.0 14.99
32 4 1 15.4 15.4
2 8 1 14.7 14.5
4 8 1 14.7 14.57
8 8 1 14.7 14.74
16 8 1 15.1 14.1
32 8 1 15.7 15.76
8 8 4 15.3 15.26
8 8 8 16.0 16.0
16 4 4 16.0 16.0

More Repositories

1

Attention-Augmented-Conv2d

Implementing Attention Augmented Convolutional Networks using Pytorch
Python
639
star
2

Stand-Alone-Self-Attention

Implementing Stand-Alone Self-Attention in Vision Models using Pytorch
Python
454
star
3

MobileNetV3-Pytorch

Implementing Searching for MobileNetV3 paper using Pytorch
Python
291
star
4

BottleneckTransformers

Bottleneck Transformers for Visual Recognition
Python
265
star
5

Billion-scale-semi-supervised-learning

Implementing Billion-scale semi-supervised learning for image classification using Pytorch
Python
89
star
6

RandWireNN

Implementing Randomly Wired Neural Networks for Image Recognition, Using CIFAR-10 dataset, CIFAR-100 dataset
Jupyter Notebook
88
star
7

Synthesizer-Rethinking-Self-Attention-Transformer-Models

Implementing SYNTHESIZER: Rethinking Self-Attention in Transformer Models using Pytorch
Python
70
star
8

CLIP

CLIP: Connecting Text and Image (Learning Transferable Visual Models From Natural Language Supervision)
Python
69
star
9

Mixed-Depthwise-Convolutional-Kernels

Implementing MixNet: Mixed Depthwise Convolutional Kernels using Pytorch
Python
60
star
10

SimSiam

Exploring Simple Siamese Representation Learning
Python
57
star
11

Action-Localization

Action-Localization, Atomic Visual Actions (AVA) Dataset
Python
23
star
12

Bag-of-MLP

Bag of MLP
Python
19
star
13

PSPNet

Implementing Pyramid Scene Parsing Network (PSPNet) paper using Pytorch
Python
14
star
14

DiffusionModel

Re-implementating Diffusion model using Pytorch
Python
7
star
15

AssembleNet

Implementing AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures Explain using Pytorch
Python
7
star
16

OmniNet

OmniNet: Omnidirectional Representations from Transformers
Python
6
star
17

Backpropagation-CNN-basic

Python
6
star
18

Graph-Convolutional-Network

Python
5
star
19

Phasic-Policy-Gradient

Phasic-Policy-Gradient
Python
5
star
20

bag-of-rl

Bag of Reinforcement Learning Algorithm
Python
5
star
21

minimal-BERT

Bidirectional Encoder Representations from Transformers
Python
4
star
22

Vision-Language

Vision-Language, Solve GQA(Visual Reasoning in the Real World) dataset.
Python
3
star
23

minimal-cyclegan

Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
Python
3
star
24

Transformer

Implementing Attention Is All You Need paper. Transformer Model
Python
2
star
25

minimal-stylegan

Python
2
star
26

SlowFast

SlowFast Network
Python
1
star
27

minimal-segmentation

minimal-segmentation
Python
1
star