• Stars
    star
    385
  • Rank 111,464 (Top 3 %)
  • Language
    Python
  • License
    Other
  • Created over 7 years ago
  • Updated almost 6 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

LabelFusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes

LabelFusion

docs/fusion_short.gif

This repo holds the code for LabelFusion.

Quick Links

Getting Started

First step is to download and set up our Docker.

If alternatively you'd like to natively install, this document may be helpful: https://github.com/RobotLocomotion/LabelFusion/blob/master/docs/setup.rst

Inspecting Data from LabelFusion

If you've downloaded some of LabelFusion data and would like to inspect some of it, we recommend the following:

  1. Run our docker image (instructions here: https://hub.docker.com/r/robotlocomotion/labelfusion/)
  2. Inside the docker image navigate to a log directory and run the alignment tool. Even though the data has already been labeled, you can inspect the results:
cd ~/labelfusion/data/logs_test/2017-06-16-20
run_alignment_tool

You should see a GUI like the following:

docs/labelfusion_screenshot.png

  1. Inspect labeled images (cd path-to-labelfusion-data/logs_test/2017-06-16-20/images and browse the images)
  2. Run a script to print out the overall status of the dataset (note this may take ~10-20 seconds to run for the full dataset): dataset_update_status -o

Training on Object Detection, Segmentation, and/or Pose data

docs/color_masks.gif

docs/bbox_trim.gif

LabelFusion provides training data which is able to train a variety of perception systems. This includes:

  • semantic segmentation (pixelwise classification)
  • 2D object detection (bounding box + classification) -- note that we provide the segmentation masks, not the bounding boxes, but the bounding boxes could be computed from the masks
  • 6 DoF object poses
  • 3D object detection (bounding box + classidication) -- the 3D bounding box can be computed from the 6 DoF object poses together with their mesh.
  • 6 DoF camera pose - this is provided without any labeling, just through the use of the dense SLAM method we use, ElasticFusion

Please see this document to better understand how the data is structured: https://github.com/RobotLocomotion/LabelFusion/blob/master/docs/data_organization.rst

At the time of publication for LabelFusion, we used this repo to train segmentation networks: https://github.com/DrSleep/tensorflow-deeplab-resnet

Quick Pipeline Instructions for Making New Labeled Data with LabelFusion

This is the quick version. If you'd prefer to go step-by-step manually, see Pipeline_Instructions.

Collect raw data from Xtion

First, cdlf && cd data/logs, then make a new directory for your data. In one terminal, run:

openni2-camera-lcm

In another, run:

lcm-logger

Your data will be saved in current directory as lcmlog-*.

Process into labeled training data

First we will launch a log player with a slider, and a viewer. The terminal will prompt for a start and end time to trim the log, then save the outputs:

run_trim

Next, we prepare for object pose fitting, by running ElasticFusion and formatting the output:

run_prep

Next, launch the object alignment tool and follow the three steps:

run_alignment_tool
  1. Check available object types:
  • In your data directory, open object_data.yaml and review the available objects, and add the objects / meshes that you need.
    • If you need multiple instances of the same object, you will need to create separate copies of the object with unique names (e.g. drill-1, drill-2, ...). For networks that do object detection, ensure that you remove this distinction from your labels / classes.
  1. Align the reconstructed point cloud:

    • Open measurement panel (View -> Measurement Panel), then check Enabled in measurement panel

    • Use shift + click and click two points: first on the surface of the table, then on a point above the table

    • Open Director terminal with F8 and run:

      gr.rotateReconstructionToStandardOrientation()
      
    • Close the run_alignment_tool application (ctrl + c) and rerun.

  2. Segment the pointcloud above the table

    • Same as above, use shift + click and click two points: first on the

    surface of the table, then on a point above the table - Open Director terminal with F8 and run:

    gr.segmentTable()
    gr.saveAboveTablePolyData()
    
    • Close the run_alignment_tool application (ctrl + c) and rerun.
  3. Align each object and crop point clouds.

    • Assign the current object you're aligning, e.g.:

      objectName = "drill"
      
    • Launch point cloud alignment:

      gr.launchObjectAlignment(objectName)
      

      This launches a new window. Click the same three points in model and on pointcloud. Using shift + click to do this. After you do this the affordance should appear in main window using the transform that was just computed.

      • If the results are inaccurate, you can rerun the above command, or you can double-click on each affordance and move it with an interactive marker: left-click to translate along an axis, right-click to rotate along an axis.
    • When you are done with an object's registration (or just wish to save intermediate poses), run:

      gr.saveRegistrationResults()
      

After the alignment outputs have been saved, we can create the labeled data:

run_create_data

By default, only RGB images and labels will be saved. If you'd also like to save depth images, use the -d flag:

run_create_data -d

Citing LabelFusion

If you find LabelFusion useful in your work, please consider citing:

@inproceedings{marion2018label,
  title={Label Fusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes},
  author={Marion, Pat and Florence, Peter R and Manuelli, Lucas and Tedrake, Russ},
  booktitle={2018 IEEE International Conference on Robotics and Automation (ICRA)},
  pages={3325--3242},
  year={2018},
  organization={IEEE}
}

More Repositories

1

drake

Model-based design and verification for robotics.
C++
3,265
star
2

pytorch-dense-correspondence

Code for "Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation"
Python
557
star
3

director

A robotics interface and visualization framework, with extensive applications for working with http://drake.mit.edu
Python
178
star
4

gcs-science-robotics

Motion Planning around Obstacles with Convex Optimization by Marcucci et al, 2023
Python
155
star
5

drake-external-examples

Examples of how to use Drake in your own project.
C++
101
star
6

drake-ros

Experimental prototyping (for now)
Python
89
star
7

LittleDog

Example of quadruped planning and control in drake using LittleDog -- a small quadruped robot from Boston Dynamics.
MATLAB
67
star
8

spartan

A project repo for robotics research and applications using drake and director.
Python
31
star
9

models

Shareable model files (urdf/sdf + meshes, etc) for our robotics projects
Starlark
25
star
10

drake-iiwa-driver

C++
24
star
11

xfoil

Fork of Mark Drela's XFOIL software (http://raphael.mit.edu/xfoil) which builds using cmake.
Fortran
24
star
12

libbot

Deprecated git mirror of the svn repository formerly located at https://code.google.com/p/libbot. Please use the upstream https://github.com/libbot2/libbot2 or new fork https://github.com/RobotLocomotion/libbot2 instead.
C
23
star
13

drake-blender

Drake glTF Render Client-Server API using Blender
Python
14
star
14

meshConverters

simple wrapper on vcg to give small platform independent mesh converters
C++
12
star
15

RobotLocomotion.github.io

Automatically generated documentation for Drake.
HTML
11
star
16

Hubo

Humanoid robot from KAIST
MATLAB
11
star
17

avl

Fork of Mark Drela's AVL software (http://raphael.mit.edu/avl) which builds using cmake.
Fortran
10
star
18

robot-plan-runner

C++
10
star
19

6-881-examples

Python
7
star
20

realsense2-lcm-driver

Publishes RealSense data to LCM
C++
6
star
21

drake-ci

Continuous integration scripts for Drake.
CMake
6
star
22

cmake

some useful shared cmake utilities
CMake
5
star
23

manip_dataset

Repo with instructions on how to download the dataset
Python
5
star
24

drake-franka-driver

Driver software for the Franka robots.
C++
4
star
25

homebrew-director

Homebrew formulae for Drake.
Ruby
3
star
26

ros_drake

CMake
2
star
27

drake-schunk-driver

Driver software for the Schunk gripper.
C++
2
star
28

bazel-external-data

Handle external large files for use with Bazel.
Python
2
star
29

drake-python3.7

Work in progress. A fork of the Drake toolbox for model-based design and verification for robotics that uses Python 3.7 on Ubuntu 18.04 (Bionic Beaver) and pip to manage Python dependencies.
C++
2
star
30

ros-drake-vendor

Maintainer scripts that package Drake in the ROS build farm
CMake
1
star
31

ipopt-mirror

Fortran
1
star
32

eigen-mirror

C++
1
star
33

ros_drake-release

1
star
34

Pigeon

Full pigeon model w/ Harvard
MATLAB
1
star
35

pdc-ros

Simple wrapper for Kuka pytorch tools + ROS
Python
1
star