RunwayML for Unity
Generate images, process textures, and create new rendering pipelines using machine learning models inside Unity.
Prerequisites
-
RunwayML: Download the latest release of RunwayML and sign up for an account. Visit our installation guide for more details. If you encounter any issues installing RunwayML, feel free to contact us through the support page.
-
Unity: Unity version 2019.2.3f1 or greater is required.
Installation
There are two ways to use the RunwayML for Unity plugin.
-
Download the starter project, containing a simple scene to get you started as quickly as possible with RunwayML for Unity.
or
-
Import the RunwayML for Unity scripts to your project. The latest
RunwayForUnity_{version}.unitypackage
is available in the releases page. The.unitypackage
contains the necessary scripts in order to use RunwayML models in Unity.
Getting Started
First, you need to first launch the RunwayML application and sign in with your account.
Next, download and unzip the RunwayML for Unity project. Import the project to Unity by opening Unity Hub and clicking on the ADD
button next to Projects
and selecting your project folder.
Once the project has been imported, click on the newly added project to launch the Unity Editor.
Using the RunwayML Panel
To interact with RunwayML inside Unity, you first need to open the RunwayML panel inside Unity. Click Window
on the top menu and then Runway
to open the RunwayML panel.
The RunwayML panel is split into four sections: the (1) Model Selection, (2) Input, (3) Output, and (4) Run Options.
Note: If you are seeing a
RUNWAY NOT FOUND
message instead of the view, ensure that the RunwayML application is running and that you are signed in.
(1) Model Selection
Here, you can select the RunwayML model that you'd like to use, and select settings for initializing your model. To learn more about the capabilities and use-cases of different models in RunwayML, watch our tutorial on discovering RunwayML models.
(2) Input
Choose the inputs that you want to process with your model. Currently, the following input data types are supported:
Image Inputs
You can use any texture or camera as image input for your model.
Use Select
to choose any texture or camera to use as input for your model.
Use Preview
to create a new window that displays the current content of your input.
Use Save
to save the current input content to disk.
Segmentation Inputs
For models that take Segmentation
data type as input, RunwayML for Unity allows you to create segmentation maps from objects in your scene, where each object is encoded by a color based on one of the categories supported by the selected model.
To label objects with a specific category supported by the model, click on one or more objects in your scene. Then select the label you want to tag the objects with (e.g. tree), and click on the Label selected objects
button.
Use Select
to choose any camera to use as input for your model. A segmentation map based on the camera display will be used as input.
Use Preview
to create a new window that displays the current content of your input.
Use Save
to save the current input content to disk.
(3) Output
The model's output will be displayed here. Currently, only models that produce image-based outputs are supported.
Use Preview
to create a new window that displays the current content of your output.
Use Save
to save the current output content to disk.
(4) Run Options
There are two options to consider with regards to how your model is run:
Run Location: Choose whether to run the model locally (Local
) or in RunwayML's cloud infrastructure (Remote
). When running models in RunwayML's cloud infrastructure, you'll be running them on fast GPU enabled computers. When running models locally, you'll need to download and install them individually. You can currently download models locally only through the RunwayML application. Go to your Settings
in RunwayML to view the currently installed models. Once you have installed a model locally, you can start the model from the RunwayML for Unity plugin.
Run Continuously: Check to keep processing the input with the selected model. Useful for performing continuous rendering when camera is selected as input.
Once you have selected the options for your model, click Start Model
to run your model. Once the model is running, you can click Process
to run inference on the current inputs using the model.
Acknowledgements
The ImageSynthesis
and ColorEncoding
script are modified versions of scripts from Unity's Image Synthesis for Machine Learning.
Contributing
This is still a work in progress. Contributions are welcomed!