monocular-depth-unity
Real-time Depth from Monocular Image using the MiDaS v2 neural network with Unity's Barracuda inference framework.
This project includes the correct .onnx model that works well with Barracuda. You can always re-download the model from its official source if you encounter the infamous InvalidProtocolBufferException error.
See discussions that led to the choice of this specific model here (Unity) and here (Intel ISL). (June 2021)
Requirements
| Platform | Version |
|---|---|
| Unity | 2021.2 or higher |
| com.unity.barracuda | 3.0.0 |
| com.unity.collections | 2.1.0-pre.11 |
| com.unity.mathematics | 1.2.6 |
| com.unity.burst | 1.8.3 |
Installation
Unity Package Manager
Latest Version (Recommended)
Add this line to your manifest.json:
"ulc-nn-depth":"https://github.com/GeorgeAdamon/monocular-depth-unity.git?path=/MonocularDepthBarracuda/Packages/DepthFromImage#main",Legacy 1.0.0 Release
Add this line to your manifest.json:
"ulc-nn-depth":"https://github.com/GeorgeAdamon/monocular-depth-unity.git?path=/MonocularDepthBarracuda/Packages/DepthFromImage#v1.0.0",Usage
Step 0
Find the DEPTH_FROM_IMAGE prefab
Step 1
Use the Texture you like in the Input Texture slot. Works with RenderTextures and Texture2D objects. Video is supported through RenderTextures.
Step 2
Parameterize the visual output in the Depth Mesher object. Use Shader method for best performance, or Mesh to get an actual tangible mesh.
If Color Texture is left blank, the mesh will be colorized with the depth data by default.
Performance
Sustained 60 fps on GTX 970 & i7 5930K (2015 rig) when using the shader-based displacement. Sustained 160 fps on RTX 3080Ti & i7 5930K when using the shader-based displacement.
Issues
The mesh-based displacement doesn't fully utilize the AsyncGPUReadback command. A command queue needs to be implemented, to process pending commands.