ofxRunway
RunwayML + openFrameworks
openFrameworks addon that allows you to easily integrate RunwayML into your openFrameworks App. It contains examples on most of the currently available models.
Install
Download and copy or git clone
this addon into your openFrameworks addons folder.
Generate or update the example projects by dragging their folders onto the projectGenerator and hitting "update" or "generate".
Dependencies
You'll need to install the following addons for this addon to work:
Instructions
Open an example and compile. When running it you must have your RunwayML app running, with the same model as the example running in a Workspace. The examples are named after each model.
For models that accept images as input, there are usually two examples: one which uses your computer's webcam and continously streams frames to RunwayML, and the other one opens an image file and processes it once.
Warning
Visual Studio
Some users have reported a compilation error on Windows, particularly but probably not limited to Visual Studio 2019, producing the following error:
1>------ Build started: Project: example-GPT-2, Configuration: Debug x64 ------
1>ofxNetworkUtils.cpp
1>C:\of\addons\ofxNetworkUtils\src\ofxNetworkUtils.cpp(16,1): fatal error C1021: invalid preprocessor command 'warning'
1>Done building project "example-GPT-2.vcxproj" -- FAILED.
========== Build: 0 succeeded, 1 failed, 1 up-to-date, 0 skipped ==========
The fix is to simply comment out the lines that start with #warning
in file openframeworks_folder\addons\ofxNetworkUtils\src\ofxNetworkUtils.cpp
Explanation
RunwayML's method of communication with other applications is using some network protocol. So far, HTTP is the one implemented.
ofxRunway takes charge of making the needed requests via HTTP on its own thread, as well as receiving the response and processing it when needed; if the response is an image it will decode it, and put into an ofImage.
RunwayML expects to receive and return data in the JSON format.
The particular format for what is sent and received can be queried by calling
http://localhost:8000/info
which will return a JSON file with the structure for the input and output packages.
You can find the host name as well as the input and output specification of the model you want to run directly inside the RunwayML app. Select your model in the Workspace, and then select Network at the top of the right side bar to find the specifications.
Classes
ofxRunway has the following classes:
ofxRunwayListener
- This is an abstract class.
- The idea is that you extend your
ofApp
class by delcaring it like thisclass ofApp : public ofBaseApp, public ofxRunwayListener`
as well as adding and implementing the following callback functions:
cpp void runwayInfoEvent(ofJson& info){} void runwayErrorEvent(string& message){}
which will get called whenever needed by ofxRunway
. You should use this as a means for communication with ofxRunway
.
- In order for these callbacks to work you need to setup your
ofxRunway
instance by calling
runway.setup(this, "http://localhost:8000");
assuming that you are calling it from within ofApp
and the second parameter is the correct one.
ofxRunway
- Is the main class, which is in charge of managing the sending and receiving of data.
- All the HTTP calls are executed on a different thread, so it won't block your app while waiting for a response.
- It includes several handy functions for sending and receiving different data types.
- It is designed to be thread safe. You don't need to take care of mutexes.
Public Methods
setup
-
Run any of these during setup or if for some reason you want to reset the connection during runtime.
-
Only use the version with two arguments when you have implemented your class extending
ofxRunwayListener
as described previously. -
You can get the host string from RunwayML app, which is usually defaulted to
"http://localhost:8000"
;bool setup(const string& host); bool setup(ofxRunwayListener& listenerClass, const string& host); bool setup(ofxRunwayListener* listenerClass, const string& host);
send
- Send an
ofxRunwayData
instance with the necessary data by callingvoid send(ofxRunwayData & data);
- This will create the call on its own thread, so it won't block your main app.
- There are also the following handy functions to be used when you only need to send a single item:
bool send(const string& name, const ofBaseHasPixels& img, ofxRunwayImageType type =OFX_RUNWAY_JPG, int resize_width = 0, int resize_height = 0); bool send(const string& name, const ofPixels& pix, ofxRunwayImageType type = OFX_RUNWAY_JPG, int resize_width = 0, int resize_height = 0); bool send(const string& name, string& data);
tryReceive
bool tryReceive(ofxRunwayData & data);
This function will check if there is any new data, and if there is, it will move or copy this new data to the ofxRunwayData
instance you pass to it and will return true. If there is no new data it will return false. It is intended to be called from within your ofApp::update
function. There is no need to make any other checks or conditions; if there is nothing it will return immediately with very little overhead.
Use it when you want to access directly the data received from the RunwayML app, otherwise use the handy get functions further down.
ofxRunwayData dataToReceive;
while (runway.tryReceive(dataToReceive)) {
// dataToReceive.data contains the raw JSON data sent by RunwayML. And it is here where you should perform what ever you need with it.
break;
}
get
Just like the send methods there are a few handy functions that you can use when expecting a single element to be returned:
bool get(const string& name, ofImage& img);
bool get(const string& name, ofPixels& pix);
bool get(const string& name, string& data);
bool get(vector<ofxRunwayCaption>& captions, float imgWidth, float imgHeight);
bool get(vector<ofxRunwayPose>& poses, float imgWidth, float imgHeight, ofxRunwayPoseType poseType);
bool get(vector<ofxRunwayFaceLandmarks>& landmarks, float imgWidth, float imgHeight);
Other methods
-
bool isBusy();
Will return true if
ofxRunway
has sent a request and is waiting for a response, otherwise will return false; -
ofxRunwayState getState();
Will return the current state of
ofxRunway
. This is a thread-safe function.ofxRunwayState
is an enum which has the following possible valuesOFX_RUNWAY_DISCONNECTED
initial default stateOFX_RUNWAY_SETUP
it has been set up but no attempt of getting any data from the RunwayML server has been doneOFX_RUNWAY_CONNECTED
set up and connected to RunwayML server but no data has been received backOFX_RUNWAY_RUNNING
connected and getting data from serverOFX_RUNWAY_CONNECTION_REFUSED
There was an error while trying to connect to server. Probably RunwayML is not running or the model in the RunwayML Workspace is not running.
-
string getStateAsString(bool bVerbose = false);
- Will return the status as as string. If you pass
true
as an argument it will return extra info. - Used by
drawStatus()
- Will return the status as as string. If you pass
-
const string& getHost();
- Returns the host address.
-
ofRectangle drawStatus(int x = 20, int y = 20, bool bVerbose = false);
- Draws the current status using ofBitmapFont
x
X-axis coordinate where to draw the texty
y-axis coordinate where to draw the textbVerbose
if true draw the verbose (full text) for the current state.- returns the bounding box of the text drawn.
-
const ofJson& getInputType(const string& name);
- Returns the
ofJson
object with info for the input types with the name passed as an argument.
- Returns the
-
const ofJson& getOutputType(const string& name);
- Returns the
ofJson
object with info for the output types with the name passed as an argument.
- Returns the
-
const ofJson& getInputTypes();
- Returns the full
ofJson
object with info about the sent data.
- Returns the full
-
const ofJson& getOutputTypes();
- Returns the full
ofJson
object with info about the received data.
- Returns the full
-
bool isServerAvailable();
- Returns true if connected to the server.
Public variables
-
ofEvent<ofJson> infoEvent;
- Event triggered when a new info event has arrived. This will happen as soon as
ofxRunway
successfully connects to the RunwayML app. It will usually only happen once.
- Event triggered when a new info event has arrived. This will happen as soon as
-
ofEvent<string> errorEvent;
- Event triggered when a there has been an error, either in sending or receiving. The event will contain a string with information about the error.
ofxRunwayData
- This class provides useful functions for setting and getting data into the JSON data required by RunwayML.
- Essentially it stores the data in an
ofJson
object, although it has a lot of very useful functions for setting and getting the data.
Public methods
Getters
-
The following
get
functions require a string as the first argument which is the name of the element you want to get or set. These names are the ones given by RunwayML in its infoEvent (you'll see these printed in the console). -
The second argument of these is a reference to an object of the type you want to get.
-
It will return true if the element was found and its info was copied into the reference passed in the second argument.
-
All these behave in the same way and only differ on the data type asked for.
-
These are named according to the data type that these will get.
bool getImage(const string& name,ofBaseHasPixels& pixels);
bool getImage(const string& name, ofPixels& pixels);
bool getBoolean(const string& name, bool &b);
bool getInt(const string& name, int& i);
bool getFloat(const string& name, float& f);
bool getString(const string& name, string& s);
bool getFloats(const string& name, vector<float>& f);
bool getStrings(const string& name, vector<string>& s);
bool getFloatVectors(const string& name, vector<vector<float> >& v);
-
example
ofxRunwayData dataToReceive;// this might be declared elsewhere
ofImage img;// this might be declared elsewhere
if(dataToReceive.getImage("image", img)){
//if it returned true, then it means that an image was extracted form the `dataToReceive` object, decoded and copied into the `img` object.
}
Get captions
There is theofxRunwayCaption
struct, whose declaration is the following.
struct ofxRunwayCaption{
ofRectangle rect;
string label;
void draw();
};
-
The following two functions allow you to get a collection of
ofxRunwayCaption
objects. The difference between the two is that one isstatic
while the other is not. -
You pass a reference to a
vector<ofxRunwayCaption>
along with the width and height of the image over which the captions go, so these can get scaled accordingly.bool getCaptions(vector<ofxRunwayCaption>& captions, float imgWidth, float imgHeight);
static bool getCaptions(vector<ofxRunwayCaption>& captions, const ofJson& data, float imgWidth, float imgHeight);
Get segmentation map
-
The segmentation map is simply a bunch of unique colors which have been associated with a label (string).
typedef std::map<ofColor, string, colorComp> SegmentationMap;
- static bool getSegmentationMap(SegmentationMap & segMap, const ofJson& info);
- static string findSegmentationLabel(const SegmentationMap & segMap, const ofBaseHasPixels& pixels, size_t x, size_t y);
- static string findSegmentationLabel(const SegmentationMap & segMap, const ofPixels& pixels, size_t x, size_t y);
Public variables
ofJson data;
This is where the actual data is stored.
Contributing
🎉Community Contribution
This is still a work in progress. Contributions are welcomed!
History
- v1. Original addon by Gene Kogan 🎉
- v2. Updated, with many examples by Roy Macdonald 🎉
Note: v1 and v2 are not compatible even though they share a lot of code.
Note: This project is heavily inspired by the RunwayML Processing Library made by George Profenza