leaderj1001/BottleneckTransformers leaderj1001/BottleneckTransformers

  • Stars
    star
    272
  • Rank 151,235 (Top 3 %)
  • Language
    Python
  • License
    MIT License
  • Created almost 4 years ago
  • Updated over 3 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
leaderj1001/BottleneckTransformers
github

leaderj1001

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Bottleneck Transformers for Visual Recognition

Bottleneck Transformers for Visual Recognition

Update 2021/03/14

  • support Multi-head Attention

Experiments

Model heads Params (M) Acc (%)
ResNet50 baseline (ref) 23.5M 93.62
BoTNet-50 1 18.8M 95.11%
BoTNet-50 4 18.8M 95.78%
BoTNet-S1-50 1 18.8M 95.67%
BoTNet-S1-59 1 27.5M 95.98%
BoTNet-S1-77 1 44.9M wip

Summary

μŠ€ν¬λ¦°μƒ· 2021-01-28 μ˜€ν›„ 4 50 19

Usage (example)

  • Model
from model import Model

model = ResNet50(num_classes=1000, resolution=(224, 224))
x = torch.randn([2, 3, 224, 224])
print(model(x).size())
  • Module
from model import MHSA

resolution = 14
mhsa = MHSA(planes, width=resolution, height=resolution)

Reference

  • Paper link
  • Author: Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, Ashish Vaswani
  • Organization: UC Berkeley, Google Research

More Repositories

1

Attention-Augmented-Conv2d

Implementing Attention Augmented Convolutional Networks using Pytorch
Python
643
star
2

Stand-Alone-Self-Attention

Implementing Stand-Alone Self-Attention in Vision Models using Pytorch
Python
456
star
3

MobileNetV3-Pytorch

Implementing Searching for MobileNetV3 paper using Pytorch
Python
290
star
4

LambdaNetworks

Implementing Lambda Networks using Pytorch
Python
138
star
5

Billion-scale-semi-supervised-learning

Implementing Billion-scale semi-supervised learning for image classification using Pytorch
Python
89
star
6

RandWireNN

Implementing Randomly Wired Neural Networks for Image Recognition, Using CIFAR-10 dataset, CIFAR-100 dataset
Jupyter Notebook
89
star
7

CLIP

CLIP: Connecting Text and Image (Learning Transferable Visual Models From Natural Language Supervision)
Python
74
star
8

Synthesizer-Rethinking-Self-Attention-Transformer-Models

Implementing SYNTHESIZER: Rethinking Self-Attention in Transformer Models using Pytorch
Python
70
star
9

Mixed-Depthwise-Convolutional-Kernels

Implementing MixNet: Mixed Depthwise Convolutional Kernels using Pytorch
Python
60
star
10

SimSiam

Exploring Simple Siamese Representation Learning
Python
58
star
11

Action-Localization

Action-Localization, Atomic Visual Actions (AVA) Dataset
Python
25
star
12

Bag-of-MLP

Bag of MLP
Python
20
star
13

PSPNet

Implementing Pyramid Scene Parsing Network (PSPNet) paper using Pytorch
Python
14
star
14

DiffusionModel

Re-implementating Diffusion model using Pytorch
Python
7
star
15

AssembleNet

Implementing AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures Explain using Pytorch
Python
7
star
16

Backpropagation-CNN-basic

Python
6
star
17

OmniNet

OmniNet: Omnidirectional Representations from Transformers
Python
6
star
18

Graph-Convolutional-Network

Python
5
star
19

Phasic-Policy-Gradient

Phasic-Policy-Gradient
Python
5
star
20

bag-of-rl

Bag of Reinforcement Learning Algorithm
Python
5
star
21

minimal-BERT

Bidirectional Encoder Representations from Transformers
Python
4
star
22

Vision-Language

Vision-Language, Solve GQA(Visual Reasoning in the Real World) dataset.
Python
3
star
23

minimal-cyclegan

Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
Python
3
star
24

Transformer

Implementing Attention Is All You Need paper. Transformer Model
Python
2
star
25

minimal-stylegan

Python
2
star
26

SlowFast

SlowFast Network
Python
1
star
27

minimal-segmentation

minimal-segmentation
Python
1
star