• Stars
    star
    181
  • Rank 212,110 (Top 5 %)
  • Language
    C#
  • License
    MIT License
  • Created about 5 years ago
  • Updated about 5 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

RunwayML for Unity ๐ŸŽฏ

RunwayML for Unity

Generate images, process textures, and create new rendering pipelines using machine learning models inside Unity.

RunwayML for Unity

Prerequisites

  • RunwayML: Download the latest release of RunwayML and sign up for an account. Visit our installation guide for more details. If you encounter any issues installing RunwayML, feel free to contact us through the support page.

  • Unity: Unity version 2019.2.3f1 or greater is required.

Installation

There are two ways to use the RunwayML for Unity plugin.

  1. Download the starter project, containing a simple scene to get you started as quickly as possible with RunwayML for Unity.

    or

  2. Import the RunwayML for Unity scripts to your project. The latest RunwayForUnity_{version}.unitypackage is available in the releases page. The .unitypackage contains the necessary scripts in order to use RunwayML models in Unity.

Getting Started

First, you need to first launch the RunwayML application and sign in with your account.

Next, download and unzip the RunwayML for Unity project. Import the project to Unity by opening Unity Hub and clicking on the ADD button next to Projects and selecting your project folder.

Once the project has been imported, click on the newly added project to launch the Unity Editor.

Using the RunwayML Panel

To interact with RunwayML inside Unity, you first need to open the RunwayML panel inside Unity. Click Window on the top menu and then Runway to open the RunwayML panel.

The RunwayML panel is split into four sections: the (1) Model Selection, (2) Input, (3) Output, and (4) Run Options.

RunwayML for Unity

Note: If you are seeing a RUNWAY NOT FOUND message instead of the view, ensure that the RunwayML application is running and that you are signed in.

(1) Model Selection

Here, you can select the RunwayML model that you'd like to use, and select settings for initializing your model. To learn more about the capabilities and use-cases of different models in RunwayML, watch our tutorial on discovering RunwayML models.

(2) Input

Choose the inputs that you want to process with your model. Currently, the following input data types are supported:

Image Inputs

You can use any texture or camera as image input for your model.

Use Select to choose any texture or camera to use as input for your model. Use Preview to create a new window that displays the current content of your input. Use Save to save the current input content to disk.

Segmentation Inputs

For models that take Segmentation data type as input, RunwayML for Unity allows you to create segmentation maps from objects in your scene, where each object is encoded by a color based on one of the categories supported by the selected model.

To label objects with a specific category supported by the model, click on one or more objects in your scene. Then select the label you want to tag the objects with (e.g. tree), and click on the Label selected objects button.

Use Select to choose any camera to use as input for your model. A segmentation map based on the camera display will be used as input. Use Preview to create a new window that displays the current content of your input. Use Save to save the current input content to disk.

(3) Output

The model's output will be displayed here. Currently, only models that produce image-based outputs are supported.

Use Preview to create a new window that displays the current content of your output. Use Save to save the current output content to disk.

(4) Run Options

There are two options to consider with regards to how your model is run:

Run Location: Choose whether to run the model locally (Local) or in RunwayML's cloud infrastructure (Remote). When running models in RunwayML's cloud infrastructure, you'll be running them on fast GPU enabled computers. When running models locally, you'll need to download and install them individually. You can currently download models locally only through the RunwayML application. Go to your Settings in RunwayML to view the currently installed models. Once you have installed a model locally, you can start the model from the RunwayML for Unity plugin.

Run Continuously: Check to keep processing the input with the selected model. Useful for performing continuous rendering when camera is selected as input.

Once you have selected the options for your model, click Start Model to run your model. Once the model is running, you can click Process to run inference on the current inputs using the model.

Acknowledgements

The ImageSynthesis and ColorEncoding script are modified versions of scripts from Unity's Image Synthesis for Machine Learning.

Contributing

This is still a work in progress. Contributions are welcomed!

More Repositories

1

stable-diffusion

Latent Text-to-Image Diffusion
Jupyter Notebook
3,267
star
2

guided-inpainting

Towards Unified Keyframe Propagation Models
Python
220
star
3

RunwayML-for-Photoshop

RunwayML for Photoshop
TypeScript
143
star
4

processing-library

RunwayML-for-Processing
Java
132
star
5

runway

[DEPRECATED ALPHA VERSION] Run, build and train machine learning models using a visual interface and then connect your models to your favorite software.
128
star
6

learn

๐Ÿ‘ฉโ€๐Ÿซ Learn how to use RunwayML with tutorials, guides, and examples!
127
star
7

model-sdk

๐Ÿ’ฅ๐Ÿ Runway Model SDK: port your own machine learning models to Runway
Python
102
star
8

ofxRunway

RunwayML + openFrameworks โ˜„๏ธ
Makefile
101
star
9

touchDesigner

RunwayML + TouchDesigner ๐ŸŒ
101
star
10

javascript

RunwayML + JavaScript ๐Ÿ‘พ
JavaScript
100
star
11

processing

[ARCHIVED] Runway + Processing ๐Ÿ”ฅ
Processing
95
star
12

p5js

RunwayML + p5.js ๐Ÿ 
JavaScript
94
star
13

model-template

๐ŸŒ๏ธโ€โ™‚๏ธModel Template: Basic template structure of a Runway model.
Python
56
star
14

hosted-models

Interact with Runway Hosted Models with only a few lines of code!
TypeScript
55
star
15

unity

[ARCHIVED] Runway + Unity ๐Ÿฆš
C#
49
star
16

awssecret2env

Convert secrets stored in AWS Secrets Manager to environment variables
Go
48
star
17

maxmsp

RunwayML + Max/MSP ๐Ÿฒ
Max
45
star
18

Intro-Synthetic-Media

Introduction to Synthetic Media Class at ITP/NYU.
JavaScript
38
star
19

alpha_models

[ALPHA] Models for Runway
PureBasic
31
star
20

OpenRNDR

RunwayML + OpenRNDR ๐Ÿฆ€
Kotlin
24
star
21

arduino

RunwayML + Arduino ๐ŸฆŠ
C++
24
star
22

RunwayML-for-Grasshopper

RunwayML for Grasshopper
C#
19
star
23

puredata

RunwayML + Pure Data ๐Ÿฆœ
14
star
24

Arbitrary-Image-Stylization

Python
7
star
25

sdk-node

TypeScript
5
star
26

sdk-python

Python
4
star
27

design

Runway design resources
1
star