Floor-SP: Inverse CAD for Floorplans by Sequential Room-wise Shortest Path
[Project Page] [Video]
Update (June 2022): our recent work, HEAT: Holistic Edge Attention Transformer for Structured Reconstruction, has better performance and efficiency than Floor-SP on floorplan reconstruction and works well for other planar graph reconstruction tasks. Please check it out if interested.
Contents
Overview
This is the official implementation of the paper Floor-SP: Inverse CAD for Floorplans by Sequential Room-wise Shortest Path, published on ICCV 2019.
Floor-SP takes aligned RGBD scans of an indoor space as the input, converts the input into 2D top-view images, and produces floorplan estimation. Detailed steps will be explained in following sections.
If you find the paper and the code helpful, please consider citing our paper:
@InProceedings{cjc2019floorsp,
title={Floor-SP: Inverse CAD for Floorplans by Sequential Room-wise Shortest Path},
author={Jiacheng Chen, Chen Liu, Jiaye Wu, Yasutaka Furukawa},
booktitle={The IEEE International Conference on Computer Vision (ICCV)},
year={2019}
}
Data download
The data will be release by Beike(www.ke.com) around Nov 2019, the link will be provided here. We will then provide a more detailed instruction on how to process the data and get output from Floor-SP.
Oct 28: 100 complete house scans have been released by Beike (www.ke.com) at link or link. Beike will send the data links via email after you fill out the form. A more detailed guideline on the data format will be provided soon by Beike. The 100 scans include the test set used in our paper, the IDs of the test scenes are provided in data/test_ids.txt.
To prepare the data, first fill the form on Beike's website to get the email containing the download links. Download the email from Beike and save the html file into ./data/email.html. Then run ./data/download.sh to download all the 100 data and unzip them automatically:
cd data
sh download.sh
cd ..
Then follow preprocess.sh and run.sh to run the whole floor-sp pipeline on the released dataset. Please refer to the rest of this README doc for more details.
Data preprocessing
We need to pre-process aligned RGBD scans to get the global point cloud for an indoor space. The global point cloud will then be converted into top-view 2D density map and mean surface normal map, which are the inputs to our Floor-SP. In this part, we also parse the annotation files and roughly clean the annotations by filtering invalid geometric structures.
-
Run
./utils/data_process.pyto pre-process data. This merges local point clouds into a global one and parses annotations. This part also makes some simple cleaning on the annotations. Sometimes the given data does not contain the json annotation file, those samples are skipped. -
Run the data writer
./utils/data_writer.pyto generate training/testing data for Floor-SP. Floor-SP uses the files underseparate_room_datadirectory. This writer is adapted from the data writer of FloorNet[2]. -
Note that the above instructions are for processing data with annotation files. If the aim is to run pre-trained Floor-SP on new point clouds, we don't need annotations. A simpler data pre-processing writer could be implemented here.
Floor-SP
1. Mask-RCNN
(All files are in ./mask-rcnn/)
Mask-RCNN [3] is the first part in Floor-SP's pipeline. It generates room instance segmentations, these segmentations form the region term in our definition of the room-aware floorplan reconstruction. Also, the room masks also serve as inputs to modules predicting corner/edge likelihood maps.
The implementation of this part is based on a public pytorch implementation of Mask-RCNN. See ./mask-rcnn/README.md for information related to installation and environment setup. This mask-rcnn focuses on detecting room instances with segmentations. The data is by default stored in ./FloorNet/data/separate_room_data after running the pre-processing.
./mask-rcnn/main.py is the script for training and evaluation of the trained mask-rcnn. We provide the pre-trained weights so ideally you don't need to re-train the model. We will only need to run the inference part for getting room segmentation results on new data.
The link to the pre-trained weights is link to the pre-trained weights. You can also find the weights for other modules of Floor-SP in the tar file.
-
./mask-rcnn/inference_corner.pyruns pre-trained mask-rcnn for room instance segmentation and prepares the data for corner/edge estimation for indoor scenes. We run this on both the training and test set to prepare data for later steps. -
./mask-rcnn/inference_room.pyruns pre-trained mask-rcnn for room instance segmentation and prepares room-related data. The data can be used for training the room-corner association module, which predicts the affiliation between corner instance and room instance. This script also saves the labels/visualization colors of all room instances properly, which are important in generating visualization results. This script is also run for both training and test set
2. Corner/edgeness modules
(see ./floor-sp/)
Floor-SP uses Dialated ResNet (DRN) based neural networks [4] to estimate corner likelihood map and edgeness likelihood map. These maps are then converted into various energy terms in the room-aware floorplan reconstruction formulation.
The model is in ./floor-sp/models/corner_net, CornerEdgeNet predicts corner/edgeness maps simultaneously.
./floor-sp/mains/corner_main.py is the script for the training / inference of Floor-SP's corner+edgeness module. The module takes density map + mean surface normal map as input and generates corner / edgeness likelihood maps. These maps are important components of Floor-SP's overall formulation.
Similarly, ./floor-sp/mains/associate_main.py is the script for the training/ inference of a room-corner affiliation module. This module predicts which corner belongs to which room and is useful for reducing the search space for room-wise coordinate descent. However, this is an optional module for Floor-SP since we found that simply using heuristics to pick up candidate corners for every room suffice. We provide the heuristics-based version in ./floor-sp/mains/associate_heuristics.py. Running either script generates intermediate data for later steps. We use the DNN-based association module to generate results presented in the paper, but the heuristics-based version produces almost the same results.
Detailed instructions can be checked in ./floor-sp/README.md.
3. Sequential room-wise shortest path
We devise the room-wise coordinate descent strategy (sequential room-wise shortest path) to optimize room structures for the floorplan. (details are in ./floor-sp/utils/floorplan_utils/)
Room-wise coordinate descent solves the room-aware floorplan reconstruction, an energy minimization problem, by using a shortest path algorithm as the solver. The paper Piecewise Planar and Compact Floorplan Reconstruction from Images [1] uses shortest path algorithm to find an optimal global floorplan (actually an outer-most boundary) for an indoor space. Following this idea, our room-wise coordinate descent runs shortest path algorithm iteratively to solve the optimal per-room structure. The rooms are processed sequentially and there could be multiple rounds of optimization just as in traditional coordinate descent.
The energy minimization problem is established using room instance segmentations, room corner likelihood maps and room edge likelihood maps that we generated in previous parts of Floor-SP. ./floor-sp/mains/extract_floorplan.py is the script for running room-wise coordinate descent, together with room merging and the final visualization. The algorithm related code can be found in ./floor-sp/utils/floorplan_utils.
4. Room merging
Room merging to get final floorplan. + Visualization / testing, etc. (./floor-sp/utils/floorplan_utils/merge.py)
The final stage of Floor-SP is to merge the optimal per-room structure to get the final floorplan. This is not the technical core. Since the energy formulation in room-aware floorplan reconstruction takes the consistency between neighbouring rooms into consideration, the room structures (i.e. room loops) produced by room-wise coordinate descent are usually in good status already. The merging stage is just for making the final visualization looks beautiful, there will not be new rooms or new corners created. In the merging stage, edges that are almost co-linear and close to each other are merged to be on the same straight line, then corners closed to each other within a threshold are also merged into one.
Note: The implementation of the merging of co-linear edges is not optimal, we simply shift all edges in the set of co-linear edges to make them on the same line with the first one in the set, while the ideal implementation should compute an average location and move all edges to that place. This simpler implementation works well for non-product-level use cases.
Environment Setup
The implementation is based on Python3.5 and Pytorch0.4.0. The file requirements.txt contains related packages and their corresponding versions in the environment for running the whole Floor-SP. You can do pip install -r requirements.txt to install all of them.
Algorithm Dependencies
Floor-SP makes use of several algorithms/models to build up the whole floorplan reconstruction system, they are listed as follows:
-
Running shortest path algorithm in a 2-D grid for solving an optimal structure [1], based on which we designd the room-wise coordinate descent algorithm.
-
Mask-RCNN for getting room instance segmentations [3].
-
Dialated Residual Networks for implementing the room/edgeness module [4].
References
[1] R. Cabral and Y. Furukawa. Piecewise planar and compact floorplan reconstruction from images. In IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pages 628β635. IEEE, 2014
[2] C. Liu, J. Wu, and Y. Furukawa. Floornet: A unified framework for floorplan reconstruction from 3d scans. In Proceedings of the European Conference on Computer Vision(ECCV), pages 201β217, 2018
[3] K. He, G. Gkioxari, P. Dollar, and R. B. Girshick. Mask r-cnn. 2017 IEEE International Conference on Computer Vision (ICCV), pages 2980β2988, 2017
[4] F. Yu, V. Koltun, and T. A. Funkhouser. Dilated residual networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 636β644, 2017