• Stars
    star
    134
  • Rank 265,142 (Top 6 %)
  • Language
    C++
  • Created over 8 years ago
  • Updated almost 3 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Still a work in progress and adding code..

#SceneNet

All the necessary source code for SceneNet SceneNet: Understanding Real World Indoor Scenes with Synthetic Data will be available here soon.

#Updates

This code enables depth and annotation rendering given a 3D model and trajectory. We provide a sample 3D model in the data folder and a trajectory in data/room_89_simple_data folder. More details will be updated soon. Things to do

  • Simulated Annealing
  • RGB Rendering to be merged with the code
  • Releasing Trajectories
  • Converting Depth to DHA format
  • Emphasise on Negative Focal Length

#Dependencies

#Build

mkdir build
cd build
cmake .. -DCUDA_PROPAGATE_HOST_FLAGS=0
make -j8

#Demo in your build, run

./opengl_depth_rendering ../data/room_89_simple.obj

You should have annotated images in the folder data/room_89_simple_data that should look like these

Montage-0

#Adding noise to the depth maps

Video

#SceneNet Basis Models (Work In Progress)

#Sample Texture Library

Download the sample texture library to texture the models from here. Textured bedrooms are shown below.

Montage-0

#Website

More information is available here at our website robotvault.bitbucket.org

#Labels

Label name and number mapping is from Eigen et al. arXiv 2015, ICCV 2015

Label Number Label Name
1 Bed
2 Books
3 Ceiling
4 Chair
5 Floor
6 Furniture
7 Objects
8 Picture
9 Sofa
10 Table
11 TV
12 Wall
13 Window

#Conversion from SUN RGB-D/NYUv2 37/40 Labels

Get the 40 class mapping for NYUv2 from http://www.cs.berkeley.edu/~sgupta/cvpr13/. SUN RGB-D already provide the 37 class mapping in their meta-data files.

Label name and number are from SUN RGB-D/ NYUv2. Their corresponding SceneNet/Eigen et al. mapping is in the last column.

SUN RGB-D/NYUv2Label Number Label Name Eigen et al./SceneNet Mapping
1 Wall 12
2 Floor 5
3 Cabinet 6
4 Bed 1
5 Chair 4
6 Sofa 9
7 Table 10
8 Door 12
9 Window 13
10 BookShelf 6
11 Picture 8
12 Counter 6
13 Blinds 13
14 Desks 10
15 Shelves 6
16 Curtain 13
17 Dresser 6
18 Pillow 7
19 Mirror 7
20 Floor-mat 5
21 Clothes 7
22 Ceiling 3
23 Books 2
24 Refrigerator 6
25 Television 11
26 Paper 7
27 Towel 7
28 Shower-curtain 7
29 Box 7
30 Whiteboard 7
31 Person 7
32 NightStand 6
33 Toilet 7
34 Sink 7
35 Lamp 7
36 Bathtub 7
37 Bag 7
38 Other-structure 7
39 Other-furniture 6
40 Other-prop 7

#Accuracy Script

Compute the global/per-class accuracy with getAccuracyNYU.m script provided in the repository.

#Latex Code

\usepackage[table]{xcolor}
\definecolor{bedColor}{rgb}{0, 0, 1}
\definecolor{booksColor}{rgb}{0.9137,0.3490,0.1882}
\definecolor{ceilColor}{rgb}{0, 0.8549, 0}
\definecolor{chairColor}{rgb}{0.5843,0,0.9412}
\definecolor{floorColor}{rgb}{0.8706,0.9451,0.0941}
\definecolor{furnColor}{rgb}{1.0000,0.8078,0.8078}
\definecolor{objsColor}{rgb}{0,0.8784,0.8980}
\definecolor{paintColor}{rgb}{0.4157,0.5333,0.8000}
\definecolor{sofaColor}{rgb}{0.4588,0.1137,0.1608}
\definecolor{tableColor}{rgb}{0.9412,0.1373,0.9216}
\definecolor{tvColor}{rgb}{0,0.6549,0.6118}
\definecolor{wallColor}{rgb}{0.9765,0.5451,0}
\definecolor{windColor}{rgb}{0.8824,0.8980,0.7608}
\begin{table*}
\begin{tabular}{l}
\textbf{13 class semantic segmentation: NYUv2} \\
\end{tabular}
\centering
\begin{tabular}{|l|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|}
\hline
Training  & \cellcolor{bedColor}\rotatebox{90}{bed} & \cellcolor{booksColor}\rotatebox{90}{books}  & \cellcolor{ceilColor}\rotatebox{90}{ceil.} & \cellcolor{chairColor}\rotatebox{90}{chair}  & \cellcolor{floorColor}\rotatebox{90}{floor}  & \cellcolor{furnColor}\rotatebox{90}{furn}   & \cellcolor{objsColor}\rotatebox{90}{objs.} & \cellcolor{paintColor}\rotatebox{90}{paint.} & \cellcolor{sofaColor}\rotatebox{90}{sofa}   & \cellcolor{tableColor}\rotatebox{90}{table}  & \cellcolor{tvColor}\rotatebox{90}{tv}     & \cellcolor{wallColor}\rotatebox{90}{wall}   & \cellcolor{windColor}\rotatebox{90}{window} \\ \hline
NYU-DHA & 67.7 & 6.5 & 69.9 & 47.9 & \textbf{96.2} & 53.8 & 46.5 & 11.3 & 50.7 & 41.6 & 10.8 & 85.0 & 25.8 \\ \hline
SceneNet-DHA & 60.8 & 2.0 & 44.2 & 68.3 & 90.2 & 26.4 & 27.6 &  6.3 & 21.1 & 42.2 & 0 & \textbf{92.0} & 0.0 \\ \hline
SceneNet-FT-NYU-DHA & 70.8 & 5.3 & 75.0 & 58.9 & 95.9 & 63.3 & 48.4 & 15.2 & 58.0 & 43.6 & 22.3 &  85.1  & 29.9 \\ \hline
NYU-DO-DHA & 69.6 & 3.1 & 69.3 & 53.2 & 95.9 & 60.0 & 49.0 & 11.6 & 52.7 & 40.2 & 17.3 & 85.0 & 27.1 \\ \hline
SceneNet-DO-DHA & 67.9 & 4.7 & 41.2 & 67.7 & 87.9 & 38.4 & 25.6 &  6.3 & 16.3 & 43.8 & 0 & 88.6 & 1.0 \\ \hline
SceneNet-FT-NYU-DO-DHA & \textbf{70.8} & 5.5 & 76.2 & 59.6 & 95.9 & \textbf{62.3} & \textbf{50.0} & 18.0 & \textbf{61.3} & 42.2 & 22.2 & 86.1 & 32.1 \\ \hline
Eigen \textit{et al.} (rgbd+normals) \cite{Eigen:etal:ICCV2015} & 61.1 & \textbf{49.7} & 78.3 & \textbf{72.1} & 96.0 & 55.1 & 40.7 &\textbf{58.7} & 45.8 &\textbf{44.9}& \textbf{41.9} & 88.7 & \textbf{57.7}  \\ \hline
Hermans \textit{et al.}(rgbd+crf)\cite{Hermans:etal:ICRA2014} & 68.4 & N/A & \textbf{83.4} & 41.9 & 91.5 & 37.1 & 8.6 & N/A & 28.5 & 27.7 & 38.4 & 71.8 & 46.1 \\ \hline
\end{tabular}
\vspace{0.5mm} \vspace{0.5mm}
\caption{Results on NYUv2 test data for 13 semantic classes. We see a similar pattern here --- adding synthetic data helps immensely in improving the performance of nearly all functional categories of objects using DHA as input channels. As expected, accuracy on \textit{books}, \textit{painting}, \textit{tv}, and \textit{windows}, is compromised highlighting that the role of depth as a modality to segment these objects is limited. Note that we recomputed the accuracies of \cite{Eigen:etal:ICCV2015} using their publicly available annotations of 320$\times$240 and resizing them to 224$\times$224. Hermans \textit{et al.} \cite{Hermans:etal:ICRA2014} use ``\textit{Decoration}" and ``\textit{Bookshelf}" instead of \textit{painting} and \textit{books} as the other two classes. Therefore, they are not directly comparable. Also, their annotations are not publicly available but we have still added their results in the table. Note that they use 640$\times$480. Poor performance of SceneNet-DHA and SceneNet-DO-DHA on \textit{tv} and \textit{windows} is mainly due to limited training data for these classes in SceneNet.}
\label{table: CA breakdown for 13 classes}
\end{table*}

Relevant Documents

The whole process of data generation and experiments are in the following papers. If you find our work valuable and helpful, please consider citing the relevant document. Thank you.

SceneNet: an Annotated Model Generator for Indoor Scene Understanding, ICRA 2016
Ankur Handa, Viorica Patraucean, Simon Stent, Roberto Cipolla

@inproceedings{Handa:etal:ICRA2016,
  author    = {Ankur Handa and 
               Viorica P{\u a}tr{\u a}ucean and
               Simon Stent and
               Roberto Cipolla},
  title     = {SceneNet: an Annotated Model Generator for Indoor Scene Understanding},
  booktitle = {ICRA},
  year      = {2016}
}

Understanding Real World Indoor Scenes With Synthetic Data, CVPR 2016
Ankur Handa, Viorica Patraucean, Vijay Badrinarayanan, Simon Stent, Roberto Cipolla

@inproceedings{Handa:etal:CVPR2016,
  author    = {Ankur Handa and 
               Viorica P{\u a}tr{\u a}ucean and
               Vijay Badrinarayanan and 
               Simon Stent and
               Roberto Cipolla},
  title     = {Understanding Real World Indoor Scenes With Synthetic Data},
  booktitle = {CVPR},
  year      = {2016}
}

An up-to-date version is maintained on the arXiv

SceneNet: Understanding Real World Indoor Scenes with Synthetic Data arXiv link
Ankur Handa, Viorica Patraucean, Vijay Badrinarayanan, Simon Stent, Roberto Cipolla

@inproceedings{Handa:etal:arXiv2016,
  author    = {Ankur Handa and 
               Viorica P{\u a}tr{\u a}ucean and
               Vijay Badrinarayanan and 
               Simon Stent and
               Roberto Cipolla},
  title     = {SceneNet: Understanding Real World Indoor Scenes With Synthetic Data},
  booktitle = {arXiv},
  year      = {2015}
}

#License

All the code and data are released under a creative commons license which is purely for research purposes only. Please view the summary here http://creativecommons.org/licenses/by-nc/4.0/

More Repositories

1

gvnn

gvnn: Geometric Vision with Neural Networks
Lua
440
star
2

nerf2D

Adding positional encoding to the input preserves sharp edges in the image
Python
202
star
3

sunrgbd-meta-data

train test labels for sunrgbd
MATLAB
166
star
4

nyuv2-meta-data

all the meta data needed for nyuv2
105
star
5

simkinect

Simulating Kinect Noise: adding noise to clean depth-maps rendered with a graphics engine.
Python
69
star
6

DeformationLoopClosure

Deformation Loop Closure sample code to enable non-rigid alignment of point clouds
C++
55
star
7

robot-assets

A repository of various URDFs and assets needed for robot manipulation
Python
37
star
8

tf-unet

tensorflow version of unet
Python
29
star
9

sim2realAI

We are indexing the progress in simulations to real world transfer for perception and control
28
star
10

pytorch-SceneNetRGBD

Implementation of UNet as used in SceneNet RGB-D paper
Python
21
star
11

implicit-bc-2d

implicit behaviour cloning toy 2d example
Python
13
star
12

dexpilot

paper on dexpilot
13
star
13

blender_scripts

scripts to convert model formats with blender
Python
11
star
14

TVL1Denoising

Experiments on TVL1 denoising
C++
10
star
15

HTC-Vive-Setup-Ubuntu

Step by step guide to setting up HTC Vive on Ubuntu
9
star
16

SceneGraphRendering

rendering rgb-d frames and more
C++
6
star
17

tf-new-op-example

tensorflow op example
C++
4
star
18

VLP16DataReader

Data Grabber for VLP16 and displays the point cloud in OpenGL window
CMake
4
star
19

tinyobjloader

Minimal version of tinyobjloader I found
C++
3
star
20

OffScreenDepthRender

Offscreen rendering of depth maps using OpenGL with Pangolin
C++
3
star
21

living_room_iclnuim

ICL-NUIM dataset living room code
Shell
2
star
22

rand_conv

applying random convolutions to an image
Python
2
star
23

imageutilities

modified version of the imageutilities GPU library from TU Graz
C++
2
star
24

ankurhanda.github.io

website
HTML
2
star
25

alignYAxisWithGravity

aligning the floor normals with the gravity vector based on http://www.cs.berkeley.edu/~sgupta/pdf/GuptaArbelaezMalikCVPR13.pdf
C++
2
star
26

stanford-scene-database

Automatically exported from code.google.com/p/stanford-scene-database
C
1
star
27

ORB_SLAM_Pangolin

C++
1
star
28

matplotlib_utils

utility functions for quick matplotlib plotting
Python
1
star
29

scenenet-rgbd-notes

1
star
30

py_contact_models

Python
1
star
31

libcvd

popular image library back in ancient times
C++
1
star
32

python_plotting

python utils for plotting
Python
1
star
33

Pangolin-local

local copy of pangolin
C++
1
star
34

Superresolution

Super resolution paper implemented from Markus Unger and Pock's variational optimisation technique
C++
1
star
35

caffe-dense-img-seg

caffe per pixel segmentation
C++
1
star
36

MagmaCublasTestFunctions

A repository of codes to get comfortable with Magma and Cublas
C
1
star