• Stars
    star
    537
  • Rank 82,649 (Top 2 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created over 1 year ago
  • Updated 7 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

[ICLR 2023 Oral] Image as Set of Points

Image as Set of Points - ICLR'23 [Oral, Top5%]

by Xu Ma*, Yuqian Zhou*, Huan Wang, Can Qin, Bin Sun, Chang Liu, Yun Fu.

arXiv webpage


Clustering Process (from scratch to last epoch):

An example of clustering training updates(from scratch to last epoch) for the four stages.
Elephant at sunset Elephant at sunset Elephant at sunset Elephant at sunset
More examples for clustering training updates(from scratch to last epoch).
Elephant at sunset Elephant at sunset Elephant at sunset Elephant at sunset Elephant at sunset

TO DO (Mar 9):

  • Fix a small bug in Line 170 and re-train all checkpoints.
  • update the checkpoints (conv1x1 -> nn.linear, shape doesn't match)
  • add large model and checkpoints
  • release codes/ checkpoints for CoC without region partition (re-trained with updated codes, get better results)
  • release the visualization script.

Image Classification

1. Requirements

torch>=1.7.0; torchvision>=0.8.0; pyyaml; timm; einops; apex-amp (if you want to use fp16);

data prepare: ImageNet with the following folder structure, you can extract ImageNet by this script.

โ”‚imagenet/
โ”œโ”€โ”€train/
โ”‚  โ”œโ”€โ”€ n01440764
โ”‚  โ”‚   โ”œโ”€โ”€ n01440764_10026.JPEG
โ”‚  โ”‚   โ”œโ”€โ”€ n01440764_10027.JPEG
โ”‚  โ”‚   โ”œโ”€โ”€ ......
โ”‚  โ”œโ”€โ”€ ......
โ”œโ”€โ”€val/
โ”‚  โ”œโ”€โ”€ n01440764
โ”‚  โ”‚   โ”œโ”€โ”€ ILSVRC2012_val_00000293.JPEG
โ”‚  โ”‚   โ”œโ”€โ”€ ILSVRC2012_val_00002138.JPEG
โ”‚  โ”‚   โ”œโ”€โ”€ ......
โ”‚  โ”œโ”€โ”€ ......

2. Pre-trained Context Cluster Models

We update the new checkpoints and logs (fix bugs, FC implementation). Feel free to download.

Model #params Image resolution Top1 Acc Throughtput Download
ContextCluster-tiny 5.3M 224 71.9 518.4 [checkpoint & logs]
ContextCluster-tiny2 5.3M 224 72.0 510.8 [checkpoint & logs]
ContextCluster-tiny_plain (w/o region partition) 5.3M 224 72.8 - [checkpoint]
ContextCluster-small 14.0M 224 77.8 513.0 [checkpoint & logs]
ContextCluster-medium 27.9M 224 81.2 325.2 [checkpoint & logs]

3. Validation

To evaluate our Context Cluster models, run:

MODEL=coc_tiny #{tiny, tiny2 small, medium}
python3 validate.py /path/to/imagenet  --model $MODEL -b 128 --checkpoint {/path/to/checkpoint} 

4. Train

We show how to train Context Cluster on 8 GPUs. The relation between learning rate and batch size is lr=bs/1024*1e-3. For convenience, assuming the batch size is 1024, then the learning rate is set as 1e-3 (for batch size of 1024, setting the learning rate as 2e-3 sometimes sees better performance).

MODEL=coc_tiny # coc variants
DROP_PATH=0.1 # drop path rates
python3 -m torch.distributed.launch --nproc_per_node=8 train.py --data_dir /dev/shm/imagenet --model $MODEL -b 128 --lr 1e-3 --drop-path $DROP_PATH --amp

5. Clustering Visualization

We provide a script to visualize the clustering results of CoC for a given stage, block, head.

Different layers/heads will present different clustering patterns.

# Use example (generated image will saved to images/cluster_vis/{model}):
python cluster_visualize.py --image {path_to_image} --model {model} --checkpoint {path_to_checkpoint} --stage {stage} --block {block} --head {head}
 

See folder pointcloud for point cloud classification taks on ScanObjectNN.

See folder detection for Detection and instance segmentation tasks on COCO.

See folder segmentation for Semantic Segmentation task on ADE20K.

BibTeX

@inproceedings{ma2023image,
    title={Image as Set of Points},
    author={Xu Ma and Yuqian Zhou and Huan Wang and Can Qin and Bin Sun and Chang Liu and Yun Fu},
    booktitle={The Eleventh International Conference on Learning Representations},
    year={2023},
    url={https://openreview.net/forum?id=awnvqZja69}
}

Acknowledgment

Our implementation is mainly based on the following codebases. We gratefully thank the authors for their wonderful works.

pointMLP, poolformer, pytorch-image-models, mmdetection, mmsegmentation.

License

The majority of Context Cluster is licensed under an Apache License 2.0

More Repositories

1

pointMLP-pytorch

[ICLR 2022 poster] Official PyTorch implementation of "Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework"
Python
481
star
2

Rewrite-the-Stars

[CVPR 2024] Rewrite the Stars
Python
242
star
3

Open-Set-Recognition

Open Set Recognition
Python
135
star
4

DCANet

[arXiv 2020] Deep Connected Attention Networks
Python
121
star
5

CollaborativeFiltering

matlab, collaborative filtering, MovieLens dataset๏ผŒThe movie recommendation system
MATLAB
107
star
6

FCViT

A Close Look at Spatial Modeling: From Attention to Convolution
Python
89
star
7

LIVE

[CVPR 2022 Oral] Towards Layer-wise Image Vectorization
Python
57
star
8

EfficientMod

[ICLR 2024 poster] Efficient Modulation for Vision Networks
Python
44
star
9

DataMining

Java implementation of the classic Data mining (big data) algorithm. Create a new Java project, and copy this project to the SRC directory is ok.
Java
27
star
10

SPANet

Codes of "SPANet: Spatial Pyramid Attention Network for Enhanced Image Recognition"
Python
22
star
11

ORL3

Face recognition implement based on LDA, PCA and SVM.
MATLAB
21
star
12

CV_papers

A paper List for computer vision.
18
star
13

Efficient_ImageNet_Classification

An efficient implementation for ImageNet classification
Python
15
star
14

CenterLoss

A Discriminative Feature Learning Approach for Deep Face Recognition
Python
8
star
15

DNN

A DNN learning project
Python
7
star
16

TSNE

Python
7
star
17

Non-Local

Compare Non-Local, GC, SE and Global_Average_Pooling
Python
7
star
18

SPANet_TMM

Python
6
star
19

Attention

Python
5
star
20

L21FS

L21FS
MATLAB
5
star
21

SparseSENet

For REU project
Python
4
star
22

ResidualAttention

Python
4
star
23

Dynamic-Conv

Python
4
star
24

NANet

Code for "Attention Meets Normalization and Beyond"
Python
4
star
25

awesome-vision-transformers-comparison

A detailed comparsion of Recent Vision Transformers on ImageNet1k
4
star
26

hpc_yolo3

an object detection framework for UNT REU 2019 project.
Python
3
star
27

2PTWSVM

2PTWSVM
MATLAB
3
star
28

ParameterFree

Official code for โ€œCascaded Context Dependency: An Extremely Lightweight Module for Deep Convolutional Neural Networksโ€
Python
3
star
29

pointsMLP

Python
3
star
30

DistKernel

Python
3
star
31

RDANet

residual decoupled attention on imagenet
Python
2
star
32

ShiftFormer

An Efficient Transformer Beyond Convolution, Self-Attention, and MLP
Python
2
star
33

Learning_keras

A learning demo for DNN based on keras
2
star
34

imagenet_nv

Python
2
star
35

ImageNet.fastai

Python
2
star
36

cifar

Python
2
star
37

imagenet.pytorch

Python
2
star
38

detection

Python
2
star
39

openmax

Python
2
star
40

RDA_cifar_GCP

RDA on CIFAR100
Python
2
star
41

incrementalAD

This is a project for Incremental Anomaly Detection, which is written in MATLAB. The used techniques including: incremental LDA, incremental SVM, SMOTE, hard negative mining.
MATLAB
2
star
42

MLproject

The project for Machine Learning
MATLAB
2
star
43

mmdet

Forked from mmdetection
Python
1
star
44

PRM

IJCAI 2020
Python
1
star
45

SuperPixelNet

1
star
46

metric_learning

Deep Metric Learning
Python
1
star
47

RENYI

1
star
48

mmdet0

Python
1
star
49

ma-xu.github.io

personal blog website
HTML
1
star
50

mmdetection3d-0.10.0

Python
1
star
51

springMVC2222

JavaScript
1
star
52

seg_every_thing2

Python
1
star
53

DeepMetric

Python
1
star
54

khdj

ๅบทๆŠคๅˆฐๅฎถ
HTML
1
star
55

Odevity

Python
1
star
56

EC

SpringMVCๆ•ดๅˆhibernate๏ผŒmysql็š„ๅŸบ็ก€ๅญฆไน ๆจก็‰ˆ
Java
1
star
57

mmdet1

Python
1
star
58

OLTR

Python
1
star
59

Library

all
JavaScript
1
star
60

ECommerce

test
JavaScript
1
star