val-iisc/3d-lmnet val-iisc/3d-lmnet

  • Stars
    star
    113
  • Rank 310,115 (Top 7 %)
  • Language
    Python
  • License
    MIT License
  • Created over 6 years ago
  • Updated over 5 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
val-iisc/3d-lmnet
github

val-iisc

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Repository for 3D-LMNet: Latent Embedding Matching for Accurate and Diverse 3D Point Cloud Reconstruction from a Single Image [BMVC 2018]

3D-LMNet

This repository contains the source codes for the paper 3D-LMNet: Latent Embedding Matching For Accurate and Diverse 3D Point Cloud Reconstruction From a Single Image.
Accepted at British Machine Vision Conference (BMVC 2018)

Citing this work

If you find this work useful in your research, please consider citing:

@inproceedings{mandikal20183dlmnet,
 author = {Mandikal, Priyanka and Navaneet, K L and Agarwal, Mayank and Babu, R Venkatesh},
 booktitle = {Proceedings of the British Machine Vision Conference ({BMVC})},
 title = {{3D-LMNet}: Latent Embedding Matching for Accurate and Diverse 3D Point Cloud Reconstruction from a Single Image},
 year = {2018}
}

Overview

3D-LMNet is a latent embedding matching approach for 3D point cloud reconstruction from a single image. To better incorporate the data prior and generate meaningful reconstructions, we first train a 3D point cloud auto-encoder and then learn a mapping from the 2D image to the corresponding learnt embedding. For a given image, there may exist multiple plausible 3D reconstructions depending on the object view. To tackle the issue of uncertainty in the reconstruction, we predict multiple reconstructions that are consistent with the input view, by learning a probablistic latent space using a view-specific ‘diversity loss’. We show that learning a good latent space of 3D objects is essential for the task of single-view 3D reconstruction.

Overview of 3D-LMNet

Dataset

ShapeNet

We train and validate our model on the ShapeNet dataset. We use the rendered images from the dataset provided by 3d-r2n2, which consists of 13 object categories. For generating the ground truth point clouds, we sample points on the corresponding object meshes from ShapeNet. We use the dataset split provided by r2n2 in all the experiments. Data download links are provided below:
Rendered Images (~12.3 GB): http://cvgl.stanford.edu/data2/ShapeNetRendering.tgz
ShapeNet pointclouds (~2.8 GB): https://drive.google.com/open?id=1cfoe521iTgcB_7-g_98GYAqO553W8Y0g
ShapeNet train/val split: https://drive.google.com/open?id=10FR-2Lbn55POB1y47MJ12euvobi6mgtc

Download each of the folders, extract them and move them into data/shapenet/.
The folder structure should now look like this:
--data/shapenet/
  --ShapeNetRendering/
  --ShapeNet_pointclouds/
  --splits/

Pix3D

We evaluate the generalization capability of our model by testing it on the real-world pix3d dataset. For the ground truth point clouds, we sample 1024 points on the provided meshes. Data download links are provided below:
Pix3D dataset (~20 GB): Follow the instructions in https://github.com/xingyuansun/pix3d
Pix3D pointclouds (~13 MB): https://drive.google.com/open?id=1RZakyBu9lPbG85SyconBn4sR8r2faInV

Download each of the folders, extract them and move them into data/pix3d/.
The folder structure should now look like this:
--data/pix3d/
  --img_cleaned_input/
  --img/
  --mask/
  --model/
  --pix3d_pointclouds/
  --pix3d.json

Usage

Install TensorFlow. We recommend version 1.3 so that the additional TensorFlow ops can be compiled. The code provided has been tested with Python 2.7, TensorFlow 1.3, and CUDA 8.0. The following steps need to be performed to run the codes given in this repository:

  1. Clone the repository:
git clone https://github.com/val-iisc/3d-lmnet.git
cd 3d-lmnet
  1. Tensorflow ops for losses (Chamfer and EMD) as well as for point cloud visualization need to be compiled. Run the makefile as given below. (Note that the the nvcc, cudalib, and tensorflow paths inside the makefile need to be updated to point to the locations on your machine):
make

Training

  • To train the point-cloud auto-encoder, run:
bash scripts/train_ae.sh

Note that the auto-encoder needs to be trained before training either of the latent matching setups.

  • To train the latent matching (lm) setup, run:
bash scripts/train_lm.sh
  • To train the probabilistic latent matching (plm) setup, run:
bash scripts/train_plm.sh

Trained Models

Create a folder called 'trained_models' inside the project folder:

mkdir trained_models

Evaluation

Follow the steps detailed above to download the dataset and pre-trained models.

ShapeNet

  • For computing the Chamfer and EMD metrics reported in the paper (all 13 categories), run:
bash scripts/metrics_shapenet_lm.sh

The computed metrics will be saved inside trained_models/lm/metrics_shapenet/

  • For the plm setup (chair category), run:
bash scripts/metrics_shapenet_plm.sh

The computed metrics will be saved inside trained_models/plm/metrics_shapenet/

Pix3D

  • For computing the Chamfer and EMD metrics reported in the paper (3 categories) for the real-world Pix3D dataset, run:
bash scripts/metrics_pix3d.sh

The computed metrics will be saved inside trained_models/lm/metrics_pix3d/

Demo

Follow the steps detailed above to download the dataset and pre-trained models.

ShapeNet

  • Run the following to visualize the results for latent matching (lm):
bash scripts/demo_shapenet_lm.sh

You can navigate to the next visualization by pressing 'q'. Close visualization using back arrow. You can visualize results for different categories by changing the value of the category flag.

  • Run the following to visualize the results for probabilistic latent matching (plm):
bash scripts/demo_shapenet_plm.sh

Pix3D

  • Run the following to visualize the results on the real-world Pix3D dataset:
bash scripts/demo_pix3d.sh

You can navigate to the next visualization by pressing 'q'. Close visualization using back arrow.

Sample Results

ShapeNet

Below are a few sample reconstructions from our trained model tested on ShapeNet. 3D-LMNet_ShapeNet_results

Pix3D

Below are a few sample reconstructions from our trained model tested on real-world Pix3D dataset. Note that we mask out the background using the provided masks before passing the images through the network. 3D-LMNet_Pix3D_results

More Repositories

1

lsc-cnn

Python
210
star
2

crowd-counting-scnn

This project is an implementation of the crowd counting model proposed in our CVPR 2017 paper - Switching Convolutional Neural Network(SCNN) for Crowd Counting. SCNN is an adaptation of the fully-convolutional neural network and uses an expert CNN that chooses the best crowd density CNN regressor for parts of the scene from a bag of regressors. This helps it tackle intra-scene crowd density variation and obtain SOTA results
Python
146
star
3

deligan

This project is an implementation of the Generative Adversarial Network proposed in our CVPR 2017 paper - DeLiGAN : Generative Adversarial Networks for Diverse and Limited Data. DeLiGAN is a simple but effective modification of the GAN framework and aims to improve performance on datasets which are diverse yet small in size.
Python
111
star
4

cnn-fixations

Visualising predictions of deep neural networks
98
star
5

sketch-parse

Code, demos and data for SketchParse (a neural network for sketch segmentation). Paper:
Jupyter Notebook
80
star
6

GD-UAP

Generalized Data-free Universal Adversarial Perturbations
Python
69
star
7

SDAT

[ICML 2022]Source code for "A Closer Look at Smoothness in Domain Adversarial Training",
Python
68
star
8

fast-feature-fool

Data independent universal adversarial perturbations
Python
61
star
9

densepcr

Repository for 'Dense 3D Point Cloud Reconstruction Using a Deep Pyramid Network' [WACV 2019]
Python
52
star
10

expresso

expresso
Python
44
star
11

NoisyTwins

[CVPR 2023] Source code for NoisyTwins: Class-consistent and Diverse Image Generation Through StyleGANs
Python
34
star
12

DeiT-LT

[CVPR 2024] Code for our Paper "DeiT-LT: Distillation Strikes Back for Vision Transformer training on Long-Tailed Datasets"
Python
33
star
13

pose_estimation

Code for our work on pose-estimation using template 3D models.
Jupyter Notebook
32
star
14

nag

[CVPR 2018] Tensorflow implementation of NAG : Network for Adversary Generation
Python
32
star
15

ss_human_mesh

Code repository for the paper: Appearance Consensus Driven Self-Supervised Human Mesh Recovery
Python
31
star
16

css-ccnn

Implementation for "Completely Self-Supervised Crowd Counting via Distribution Matching" (http://arxiv.org/abs/2009.06420)
Python
29
star
17

Hard-Label-Model-Stealing

Python
26
star
18

3d-psrnet

Repository for 3D-PSRNet: Part Segmented 3D Point Cloud Reconstruction [ECCVW 2018]
Python
24
star
19

GAMA-GAT

Guided Adversarial Attack for Evaluating and Enhancing Adversarial Defenses, NeurIPS Spotlight 2020
Python
24
star
20

Saddle-LongTail

[NeurIPS 2022] Source code for our paper "Escaping Saddle Points for Effective Generalization on Class-Imbalanced Data"
Python
22
star
21

BPFC

Towards Achieving Adversarial Robustness by Enforcing Feature Consistency Across Bit Planes
Python
22
star
22

capnet

Code release for "CAPNet: Continuous Approximation Projection For 3D Point Cloud Reconstruction Using 2D Supervision", (AAAI-19)
Python
21
star
23

StickerDA

[ECCV22] Concurrent Subsidiary Supervision for Unsupervised Source-Free Domain Adaptation
Python
18
star
24

VL2V-ADiP

[CVPR 2024] Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification
Python
18
star
25

USFDA

Universal Source-Free Domain Adaptation (CVPR 2020)
17
star
26

NuAT

Towards Efficient and Effective Adversarial Training, NeurIPS 2021
Python
16
star
27

gSRGAN

[ECCV2022] Source Code for "Improving GANs for Long-Tailed Data through Group Spectral Regularization"
Python
16
star
28

DAJAT

Official Code for Efficient and Effective Augmentation Strategy for Adversarial Training (NeurIPS-2022)
Python
15
star
29

s3vaada

Submodular Subset Selection for Active Domain Adaptation (ICCV 2021)
Python
15
star
30

sketch-object-recognition

C++
15
star
31

RMLVQA

Python
12
star
32

class-balancing-gan

Class Balancing GAN with a Classifier In The Loop (UAI 2021)
Python
12
star
33

InheriTune

Code Release for the CVPR 2020 (oral) paper, "Towards Inheritable Models for Open-set Domain Adaptation".
11
star
34

DART

[CVPR-2023] Official Code for DART: Diversify-Aggregate-Repeat Training Improves Generalization of Neural Networks
Python
10
star
35

SFDA-Seg

Generalize then Adapt: Source-free Domain Adaptation for Semantic Segmentation (ICCV 2021)
9
star
36

MixupDA

[ICML22] Balancing Discriminability and Transferability for Source-Free Domain Adaptation
Python
9
star
37

OAAT

Official Code for Scaling Adversarial Training to Large Perturbation Bounds (ECCV-2022)
Python
9
star
38

sketchguess

Repository for code, models and datasets for the paper Game of Sketches: Deep Recurrent Models of Pictionary-style Word Guessing accepted at 36th AAAI Conference on Artificial Intelligence (AAAI-18), New Orleans, USA
Python
9
star
39

FLSS

Official code for the paper - Boosting Adversarial Robustness using Feature Level Stochastic Smoothing
Python
7
star
40

CostSensitiveSelfTraining

[NeurIPS 2022] This repository contains the code for our work CSST: Cost Sensitive Self Training for Optimizing Non-Decomposable Objectives
Python
7
star
41

AAA

Python
6
star
42

SelMix

[ICLR 2024] [Spotlight]Code for our paper SelMix: Selective Mixup FineTuning
Python
4
star
43

EffSSL

Towards Efficient and Effective Self-Supervised Learning of Visual Representations
Python
4
star
44

sketch-object-part-analysis

Code and data related to analysis of object sketches at a semantic part-level
3
star
45

HSR

Code for Hierarchical Semantic Regularization of Latent Spaces in StyleGANs (ECCV 2022)
Python
3
star
46

DLCV

Webpage for DS265: Deep Learning in Computer Vision Course
CSS
2
star
47

gat

Repository for the Gray-box Adversarial Training project presented at ECCV 2018.
Python
2
star
48

swiden

Jupyter Notebook
2
star
49

DSiT-SFDA

[ICCV23] Domain-Specificity Inducing Transformers for Source-Free Domain Adaptation
1
star
50

SAT-Rx

Regularizers for Single-step Adversarial training
Python
1
star