๐
Glasses
Compact, concise and customizable deep learning computer vision library
Models have been stored into the hugging face hub!
Doc is here
TL;DR
This library has
- human readable code, no research code
- common component are shared across models
- same APIs for all models (you learn them once and they are always the same)
- clear and easy to use model constomization (see here)
- classification and segmentation
- emoji in the name ;)
Stuff implemented so far:
- Training data-efficient image transformers & distillation through attention
- Vision Transformer - An Image Is Worth 16x16 Words: Transformers For Image Recognition At Scale
- ResNeSt: Split-Attention Networks
- AlexNet- ImageNet Classification with Deep Convolutional Neural Networks
- DenseNet - Densely Connected Convolutional Networks
- EfficientNet - EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
- EfficientNetLite - Higher accuracy on vision models with EfficientNet-Lite
- FishNet - FishNet: A Versatile Backbone for Image, Region, and Pixel Level Prediction
- MobileNet - MobileNetV2: Inverted Residuals and Linear Bottlenecks
- RegNet - Designing Network Design Spaces
- ResNet - Deep Residual Learning for Image Recognition
- ResNetD - Bag of Tricks for Image Classification with Convolutional Neural Networks
- ResNetXt - Aggregated Residual Transformations for Deep Neural Networks
- SEResNet - Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks
- VGG - Very Deep Convolutional Networks For Large-scale Image Recognition
- WideResNet - Wide Residual Networks
- FPN - Feature Pyramid Networks for Object Detection
- PFPN - Panoptic Feature Pyramid Networks
- UNet - U-Net: Convolutional Networks for Biomedical Image Segmentation
- Squeeze and Excitation - Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks
- ECA - ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks
- DropBlock: A regularization method for convolutional networks
- Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
- Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks
- Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization
Installation
You can install glasses
using pip by running
pip install git+https://github.com/FrancescoSaverioZuppichini/glasses
Motivations
Almost all existing implementations of the most famous model are written with very bad coding practices, what today is called research code. I struggled to understand some of the implementations even if in the end were just a few lines of code.
Most of them are missing a global structure, they used tons of code repetition, they are not easily customizable and not tested. Since I do computer vision for living, I needed a way to make my life easier.
Getting started
The API are shared across all models!
import torch
from glasses.models import AutoModel, AutoTransform
# load one model
model = AutoModel.from_pretrained('resnet18').eval()
# and its correct input transformation
tr = AutoTransform.from_name('resnet18')
model.summary(device='cpu' ) # thanks to torchinfo
# at any time, see all the models
AutoModel.models_table()
Models
โโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Name โ Pretrained โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ resnet18 โ true โ
โ resnet26 โ true โ
โ resnet26d โ true โ
โ resnet34 โ true โ
โ resnet34d โ true โ
โ resnet50 โ true โ
...
Interpretability
import requests
from PIL import Image
from io import BytesIO
from glasses.interpretability import GradCam, SaliencyMap
from torchvision.transforms import Normalize
# get a cute dog ๐ถ
r = requests.get('https://i.insider.com/5df126b679d7570ad2044f3e?width=700&format=jpeg&auto=webp')
im = Image.open(BytesIO(r.content))
# un-normalize when done
mean, std = tr.transforms[-1].mean, tr.transforms[-1].std
postprocessing = Normalize(-mean / std, (1.0 / std))
# apply preprocessing
x = tr(im).unsqueeze(0)
_ = model.interpret(x, using=GradCam(), postprocessing=postprocessing).show()
Classification
from glasses.models import ResNet
from torch import nn
# change activation
model = AutoModel.from_pretrained('resnet18', activation = nn.SELU).eval()
# or directly from the model class
ResNet.resnet18(activation = nn.SELU)
# change number of classes
ResNet.resnet18(n_classes=100)
# freeze only the convolution weights
model = AutoModel.from_pretrained('resnet18')
model.freeze(who=model.encoder)
Get the inner features
# model.encoder has special hooks ready to be activated
# call the .features to trigger them
model.encoder.features
x = torch.randn((1, 3, 224, 224))
model(x)
[f.shape for f in model.encoder.features]
Change inner block
# what about resnet with inverted residuals?
from glasses.models.classification.efficientnet import InvertedResidualBlock
ResNet.resnet18(block = InvertedResidualBlock)
Segmentation
from functools import partial
from glasses.models.segmentation.unet import UNet, UNetDecoder
# vanilla Unet
unet = UNet()
# let's change the encoder
unet = UNet.from_encoder(partial(AutoModel.from_name, 'efficientnet_b1'))
# mmm I want more layers in the decoder!
unet = UNet(decoder=partial(UNetDecoder, widths=[256, 128, 64, 32, 16]))
# maybe resnet was better
unet = UNet(encoder=lambda **kwargs: ResNet.resnet26(**kwargs).encoder)
# same API
# unet.summary(input_shape=(1,224,224))
unet
More examples
# change the decoder part
model = AutoModel.from_pretrained('resnet18')
my_head = nn.Sequential(
nn.AdaptiveAvgPool2d((1,1)),
nn.Flatten(),
nn.Linear(model.encoder.widths[-1], 512),
nn.Dropout(0.2),
nn.ReLU(),
nn.Linear(512, 1000))
model.head = my_head
x = torch.rand((1,3,224,224))
model(x).shape #torch.Size([1, 1000])
Pretrained Models
I am currently working on the pretrained models and the best way to make them available
This is a list of all the pretrained models available so far!. They are all trained on ImageNet.
I used a batch_size=64
and a GTX 1080ti to evaluale the models.
top1 | top5 | time | batch_size | |
---|---|---|---|---|
vit_base_patch16_384 | 0.842 | 0.9722 | 1130.81 | 64 |
vit_large_patch16_224 | 0.82836 | 0.96406 | 893.486 | 64 |
eca_resnet50t | 0.82234 | 0.96172 | 241.754 | 64 |
eca_resnet101d | 0.82166 | 0.96052 | 213.632 | 64 |
efficientnet_b3 | 0.82034 | 0.9603 | 199.599 | 64 |
regnety_032 | 0.81958 | 0.95964 | 136.518 | 64 |
vit_base_patch32_384 | 0.8166 | 0.9613 | 243.234 | 64 |
vit_base_patch16_224 | 0.815 | 0.96018 | 306.686 | 64 |
deit_small_patch16_224 | 0.81082 | 0.95316 | 132.868 | 64 |
eca_resnet50d | 0.80604 | 0.95322 | 135.567 | 64 |
resnet50d | 0.80492 | 0.95128 | 97.5827 | 64 |
cse_resnet50 | 0.80292 | 0.95048 | 108.765 | 64 |
efficientnet_b2 | 0.80126 | 0.95124 | 127.177 | 64 |
eca_resnet26t | 0.79862 | 0.95084 | 155.396 | 64 |
regnety_064 | 0.79712 | 0.94774 | 183.065 | 64 |
regnety_040 | 0.79222 | 0.94656 | 124.881 | 64 |
resnext101_32x8d | 0.7921 | 0.94556 | 290.38 | 64 |
regnetx_064 | 0.79066 | 0.94456 | 176.3 | 64 |
wide_resnet101_2 | 0.7891 | 0.94344 | 277.755 | 64 |
regnetx_040 | 0.78486 | 0.94242 | 122.619 | 64 |
wide_resnet50_2 | 0.78464 | 0.94064 | 201.634 | 64 |
efficientnet_b1 | 0.7831 | 0.94096 | 98.7143 | 64 |
resnet152 | 0.7825 | 0.93982 | 186.191 | 64 |
regnetx_032 | 0.7792 | 0.93996 | 319.558 | 64 |
resnext50_32x4d | 0.77628 | 0.9368 | 114.325 | 64 |
regnety_016 | 0.77604 | 0.93702 | 96.547 | 64 |
efficientnet_b0 | 0.77332 | 0.93566 | 67.2147 | 64 |
resnet101 | 0.77314 | 0.93556 | 134.148 | 64 |
densenet161 | 0.77146 | 0.93602 | 239.388 | 64 |
resnet34d | 0.77118 | 0.93418 | 59.9938 | 64 |
densenet201 | 0.76932 | 0.9339 | 158.514 | 64 |
regnetx_016 | 0.76684 | 0.9328 | 91.7536 | 64 |
resnet26d | 0.766 | 0.93188 | 70.6453 | 64 |
regnety_008 | 0.76238 | 0.93026 | 54.1286 | 64 |
resnet50 | 0.76012 | 0.92934 | 89.7976 | 64 |
densenet169 | 0.75628 | 0.9281 | 127.077 | 64 |
resnet26 | 0.75394 | 0.92584 | 65.5801 | 64 |
resnet34 | 0.75096 | 0.92246 | 56.8985 | 64 |
regnety_006 | 0.75068 | 0.92474 | 55.5611 | 64 |
regnetx_008 | 0.74788 | 0.92194 | 57.9559 | 64 |
densenet121 | 0.74472 | 0.91974 | 104.13 | 64 |
deit_tiny_patch16_224 | 0.7437 | 0.91898 | 66.662 | 64 |
vgg19_bn | 0.74216 | 0.91848 | 169.357 | 64 |
regnety_004 | 0.73766 | 0.91638 | 68.4893 | 64 |
regnetx_006 | 0.73682 | 0.91568 | 81.4703 | 64 |
vgg16_bn | 0.73476 | 0.91536 | 150.317 | 64 |
vgg19 | 0.7236 | 0.9085 | 155.851 | 64 |
regnetx_004 | 0.72298 | 0.90644 | 58.0049 | 64 |
vgg16 | 0.71628 | 0.90368 | 135.398 | 64 |
vgg13_bn | 0.71618 | 0.9036 | 129.077 | 64 |
efficientnet_lite0 | 0.7041 | 0.89894 | 62.4211 | 64 |
vgg11_bn | 0.70408 | 0.89724 | 86.9459 | 64 |
vgg13 | 0.69984 | 0.89306 | 116.052 | 64 |
regnety_002 | 0.6998 | 0.89422 | 46.804 | 64 |
resnet18 | 0.69644 | 0.88982 | 46.2029 | 64 |
vgg11 | 0.68872 | 0.88658 | 79.4136 | 64 |
regnetx_002 | 0.68658 | 0.88244 | 45.9211 | 64 |
Assuming you want to load efficientnet_b1
:
from glasses.models import EfficientNet, AutoModel, AutoTransform
# load it using AutoModel
model = AutoModel.from_pretrained('efficientnet_b1').eval()
# or from its own class
model = EfficientNet.efficientnet_b1(pretrained=True)
# you may also need to get the correct transformation that must be applied on the input
tr = AutoTransform.from_name('efficientnet_b1')
In this case, tr
is
Compose(
Resize(size=240, interpolation=PIL.Image.BICUBIC)
CenterCrop(size=(240, 240))
ToTensor()
Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225))
)
Deep Customization
All models are composed by sharable parts:
Block
Layer
Encoder
Head
Decoder
Block
Each model has its building block, they are noted by *Block
. In each block, all the weights are in the .block
field. This makes it very easy to customize one specific model.
from glasses.models.classification.vgg import VGGBasicBlock
from glasses.models.classification.resnet import ResNetBasicBlock, ResNetBottleneckBlock, ResNetBasicPreActBlock, ResNetBottleneckPreActBlock
from glasses.models.classification.senet import SENetBasicBlock, SENetBottleneckBlock
from glasses.models.classification.resnetxt import ResNetXtBottleNeckBlock
from glasses.models.classification.densenet import DenseBottleNeckBlock
from glasses.models.classification.wide_resnet import WideResNetBottleNeckBlock
from glasses.models.classification.efficientnet import EfficientNetBasicBlock
For example, if we want to add Squeeze and Excitation to the resnet bottleneck block, we can just
from glasses.nn.att import SpatialSE
from glasses.models.classification.resnet import ResNetBottleneckBlock
class SEResNetBottleneckBlock(ResNetBottleneckBlock):
def __init__(self, in_features: int, out_features: int, squeeze: int = 16, *args, **kwargs):
super().__init__(in_features, out_features, *args, **kwargs)
# all the weights are in block, we want to apply se after the weights
self.block.add_module('se', SpatialSE(out_features, reduction=squeeze))
SEResNetBottleneckBlock(32, 64)
Then, we can use the class methods to create the new models following the existing architecture blueprint, for example, to create se_resnet50
ResNet.resnet50(block=ResNetBottleneckBlock)
The cool thing is each model has the same api, if I want to create a vgg13 with the ResNetBottleneckBlock
I can just
from glasses.models import VGG
model = VGG.vgg13(block=SEResNetBottleneckBlock)
model.summary()
Some specific model can require additional parameter to the block, for example MobileNetV2
also required a expansion
parameter so our SEResNetBottleneckBlock
won't work.
Layer
A Layer
is a collection of blocks, it is used to stack multiple blocks together following some logic. For example, ResNetLayer
from glasses.models.classification.resnet import ResNetLayer
ResNetLayer(64, 128, depth=2)
Encoder
The encoder is what encoders a vector, so the convolution layers. It has always two very important parameters.
- widths
- depths
widths is the wide at each layer, so how much features there are depths is the depth at each layer, so how many blocks there are
For example, ResNetEncoder
will creates multiple ResNetLayer
based on the len of widths
and depths
. Let's see some example.
from glasses.models.classification.resnet import ResNetEncoder
# 3 layers, with 32,64,128 features and 1,2,3 block each
ResNetEncoder(
widths=[32,64,128],
depths=[1,2,3])
All encoders are subclass of Encoder
that allows us to hook on specific stages to get the featuers. All you have to do is first call .features
to notify the model you want to receive the features, and then pass an input.
enc = ResNetEncoder()
enc.features
enc(torch.randn((1,3,224,224)))
print([f.shape for f in enc.features])
Remember each model has always a .encoder
field
from glasses.models import ResNet
model = ResNet.resnet18()
model.encoder.widths[-1]
The encoder knows the number of output features, you can access them by
Features
Each encoder can return a list of features accessable by the .features
field. You need to call it once before in order to notify the encoder we wish to also store the features
from glasses.models.classification.resnet import ResNetEncoder
x = torch.randn(1,3,224,224)
enc = ResNetEncoder()
enc.features # call it once
enc(x)
features = enc.features # now we have all the features from each layer (stage)
[print(f.shape) for f in features]
# torch.Size([1, 64, 112, 112])
# torch.Size([1, 64, 56, 56])
# torch.Size([1, 128, 28, 28])
# torch.Size([1, 256, 14, 14])
Head
Head is the last part of the model, it usually perform the classification
from glasses.models.classification.resnet import ResNetHead
ResNetHead(512, n_classes=1000)
Decoder
The decoder takes the last feature from the .encoder
and decode it. This is usually done in segmentation
models, such as Unet.
from glasses.models.segmentation.unet import UNetDecoder
x = torch.randn(1,3,224,224)
enc = ResNetEncoder()
enc.features # call it once
x = enc(x)
features = enc.features
# we need to tell the decoder the first feature size and the size of the lateral features
dec = UNetDecoder(start_features=enc.widths[-1],
lateral_widths=enc.features_widths[::-1])
out = dec(x, features[::-1])
out.shape
This object oriented structure allows to reuse most of the code across the models
name | Parameters | Size (MB) |
---|---|---|
cse_resnet101 | 49,326,872 | 188.17 |
cse_resnet152 | 66,821,848 | 254.91 |
cse_resnet18 | 11,778,592 | 44.93 |
cse_resnet34 | 21,958,868 | 83.77 |
cse_resnet50 | 28,088,024 | 107.15 |
deit_base_patch16_224 | 87,184,592 | 332.58 |
deit_base_patch16_384 | 87,186,128 | 357.63 |
deit_small_patch16_224 | 22,359,632 | 85.3 |
deit_tiny_patch16_224 | 5,872,400 | 22.4 |
densenet121 | 7,978,856 | 30.44 |
densenet161 | 28,681,000 | 109.41 |
densenet169 | 14,149,480 | 53.98 |
densenet201 | 20,013,928 | 76.35 |
eca_resnet101d | 44,568,563 | 212.62 |
eca_resnet101t | 44,566,027 | 228.65 |
eca_resnet18d | 16,014,452 | 98.41 |
eca_resnet18t | 1,415,684 | 37.91 |
eca_resnet26d | 16,014,452 | 98.41 |
eca_resnet26t | 16,011,916 | 114.44 |
eca_resnet50d | 25,576,350 | 136.65 |
eca_resnet50t | 25,573,814 | 152.68 |
efficientnet_b0 | 5,288,548 | 20.17 |
efficientnet_b1 | 7,794,184 | 29.73 |
efficientnet_b2 | 9,109,994 | 34.75 |
efficientnet_b3 | 12,233,232 | 46.67 |
efficientnet_b4 | 19,341,616 | 73.78 |
efficientnet_b5 | 30,389,784 | 115.93 |
efficientnet_b6 | 43,040,704 | 164.19 |
efficientnet_b7 | 66,347,960 | 253.1 |
efficientnet_b8 | 87,413,142 | 505.01 |
efficientnet_l2 | 480,309,308 | 2332.13 |
efficientnet_lite0 | 4,652,008 | 17.75 |
efficientnet_lite1 | 5,416,680 | 20.66 |
efficientnet_lite2 | 6,092,072 | 23.24 |
efficientnet_lite3 | 8,197,096 | 31.27 |
efficientnet_lite4 | 13,006,568 | 49.62 |
fishnet150 | 24,960,808 | 95.22 |
fishnet99 | 16,630,312 | 63.44 |
mobilenet_v2 | 3,504,872 | 24.51 |
mobilenetv2 | 3,504,872 | 13.37 |
regnetx_002 | 2,684,792 | 10.24 |
regnetx_004 | 5,157,512 | 19.67 |
regnetx_006 | 6,196,040 | 23.64 |
regnetx_008 | 7,259,656 | 27.69 |
regnetx_016 | 9,190,136 | 35.06 |
regnetx_032 | 15,296,552 | 58.35 |
regnetx_040 | 22,118,248 | 97.66 |
regnetx_064 | 26,209,256 | 114.02 |
regnetx_080 | 34,561,448 | 147.43 |
regnety_002 | 3,162,996 | 12.07 |
regnety_004 | 4,344,144 | 16.57 |
regnety_006 | 6,055,160 | 23.1 |
regnety_008 | 6,263,168 | 23.89 |
regnety_016 | 11,202,430 | 42.73 |
regnety_032 | 19,436,338 | 74.14 |
regnety_040 | 20,646,656 | 91.77 |
regnety_064 | 30,583,252 | 131.52 |
regnety_080 | 39,180,068 | 165.9 |
resnest101e | 48,275,016 | 184.15 |
resnest14d | 10,611,688 | 40.48 |
resnest200e | 70,201,544 | 267.8 |
resnest269e | 7,551,112 | 28.81 |
resnest26d | 17,069,448 | 65.11 |
resnest50d | 27,483,240 | 104.84 |
resnest50d_1s4x24d | 25,677,000 | 97.95 |
resnest50d_4s2x40d | 30,417,592 | 116.03 |
resnet101 | 44,549,160 | 169.94 |
resnet152 | 60,192,808 | 229.62 |
resnet18 | 11,689,512 | 44.59 |
resnet200 | 64,673,832 | 246.71 |
resnet26 | 15,995,176 | 61.02 |
resnet26d | 16,014,408 | 61.09 |
resnet34 | 21,797,672 | 83.15 |
resnet34d | 21,816,904 | 83.22 |
resnet50 | 25,557,032 | 97.49 |
resnet50d | 25,576,264 | 97.57 |
resnext101_32x16d | 194,026,792 | 740.15 |
resnext101_32x32d | 468,530,472 | 1787.3 |
resnext101_32x48d | 828,411,176 | 3160.14 |
resnext101_32x8d | 88,791,336 | 338.71 |
resnext50_32x4d | 25,028,904 | 95.48 |
se_resnet101 | 49,292,328 | 188.04 |
se_resnet152 | 66,770,984 | 254.71 |
se_resnet18 | 11,776,552 | 44.92 |
se_resnet34 | 21,954,856 | 83.75 |
se_resnet50 | 28,071,976 | 107.09 |
unet | 23,202,530 | 88.51 |
vgg11 | 132,863,336 | 506.83 |
vgg11_bn | 132,868,840 | 506.85 |
vgg13 | 133,047,848 | 507.54 |
vgg13_bn | 133,053,736 | 507.56 |
vgg16 | 138,357,544 | 527.79 |
vgg16_bn | 138,365,992 | 527.82 |
vgg19 | 143,667,240 | 548.05 |
vgg19_bn | 143,678,248 | 548.09 |
vit_base_patch16_224 | 86,415,592 | 329.65 |
vit_base_patch16_384 | 86,415,592 | 329.65 |
vit_base_patch32_384 | 88,185,064 | 336.4 |
vit_huge_patch16_224 | 631,823,080 | 2410.21 |
vit_huge_patch32_384 | 634,772,200 | 2421.46 |
vit_large_patch16_224 | 304,123,880 | 1160.14 |
vit_large_patch16_384 | 304,123,880 | 1160.14 |
vit_large_patch32_384 | 306,483,176 | 1169.14 |
vit_small_patch16_224 | 48,602,344 | 185.4 |
wide_resnet101_2 | 126,886,696 | 484.03 |
wide_resnet50_2 | 68,883,240 | 262.77 |
Credits
Most of the weights were trained by other people and adapted to glasses. It is worth cite
- pytorch-image-models
- torchvision