• Stars
    star
    178
  • Rank 214,989 (Top 5 %)
  • Language
    HTML
  • License
    MIT License
  • Created almost 7 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Remotely-controllable 3D viewer, built on top of three.js

MeshCat

MeshCat is a remotely-controllable 3D viewer, built on top of three.js. The MeshCat viewer runs in a browser and listens for geometry commands over WebSockets. This makes it easy to create a tree of objects and transformations by sending the appropriate commands over the websocket.

The MeshCat viewer is meant to be combined with an interface in the language of your choice. Current interfaces are:

API

MeshCat can be used programmatically from JS or over a WebSocket connection.

Programmatic API

To create a new MeshCat viewer, use the Viewer constructor:

let viewer = new MeshCat.Viewer(dom_element);

where dom_element is the div in which the viewer should live. The primary interface to the viewer is the handle_command function, which maps directly to the behaviors available over the WebSocket.

Viewer.handle_command(cmd)
Handle a single command and update the viewer. cmd should be a JS object with at least the field type. Available command types are:
set_object
Set the 3D object at a given path in the scene tree from its JSON description. Any transforms previously applied to that path will be preserved and any children of that path will continue to exist. To remove transforms and delete all children from a given path, you should send a delete command first (see below).

Internally, we append a final path segment, <object>, to the provided path before creating the object. This is done because clients may disagree about what the "intrinsic" transform of a particular geometry is (for example, is a "Box" centered on the origin, or does it have one corner at the origin?). Clients can use the matrix field of the JSON object to store that intrinsic transform, and that matrix will be preserved by attaching it to the <object> path. Generally, you shouldn't need to worry about this: if you set an object at the path /meshcat/foo, then you can set the transform at /meshcat/foo and everything will just work.

Additional fields:

path
A "/"-separated string indicating the object's path in the scene tree. An object at path "/foo/bar" is a child of an object at path "/foo", so setting the transform of (or deleting) "/foo" will also affect its children.
object
The Three.js Object, with its geometry and material, in JSON form as a JS object. The format accepted is, essentially, anything that ObjectLoader can handle, or, similarly, anything you might get by calling the toJSON() method of a Three.js Object3D.
Example:
{
    type: "set_object",
    path: "/meshcat/boxes/box1",
    object: {
        metadata: {version: 4.5, type: "Object"},
        geometries: [
            {
                uuid: "cef79e52-526d-4263-b595-04fa2705974e",
                type: "BoxGeometry",
                width: 1,
                height: 1,
                depth:1
            }
        ],
        materials: [
            {
                uuid: "0767ae32-eb34-450c-b65f-3ae57a1102c3",
                type: "MeshLambertMaterial",
                color: 16777215,
                emissive: 0,
                side: 2,
                depthFunc: 3,
                depthTest: true,
                depthWrite: true
            }
        ],
        object: {
            uuid: "00c2baef-9600-4c6b-b88d-7e82c40e004f",
            type: "Mesh",
            geometry: "cef79e52-526d-4263-b595-04fa2705974e",
            material: "0767ae32-eb34-450c-b65f-3ae57a1102c3"
        }
    }
}
                
Note the somewhat indirect way in which geometries and materials are specified. Each Three.js serialized object has a list of geometries and a list of materials, each with a UUID. The actual geometry and material for a given object are simply references to those existing UUIDs. This enables easy re-use of geometries between objects in Three.js, although we don't really rely on that in MeshCat. Some information about the JSON object format can be found on the Three.js wiki.
set_transform
Set the homogeneous transform for a given path in the scene tree. An object's pose is the concatenation of all of the transforms along its path, so setting the transform of "/foo" will move the objects at "/foo/box1" and "/foo/robots/HAL9000".

Additional fields:

path
A "/"-separated string indicating the object's path in the scene tree. An object at path "/foo/bar" is a child of an object at path "/foo", so setting the transform of (or deleting) "/foo" will also affect its children.
matrix
The homogeneous transformation matrix, given as a 16-element Float32Array in column-major order.
Example:
{
    type: "set_transform",
    path: "/meshcat/boxes",
    matrix: new Float32Array([1, 0, 0, 0,
                            0, 1, 0, 0,
                            0, 0, 1, 0,
                            0.5, 0.0, 0.5, 1])
}
                
delete
Delete the object at the given path as well as all of its children.

Additional fields:

path
A "/"-separated string indicating the object's path in the scene tree. An object at path "/foo/bar" is a child of an object at path "/foo", so setting the transform of (or deleting) "/foo" will also affect its children.
Example:
{
    type: "delete",
    path: "/meshcat/boxes"
}
                
set_property
Set a single named property of the object at the given path. If no object exists at that path, an empty one is automatically created.

Note: as we append an extra path element with the name <object> to every item created with set_object, if you want to modify a property of the object itself, rather than the group containing it, you should ensure that your path is of the form /meshcat/foo/<object>

Additional fields:

property
The name of the property to set, as a string.
value
The new value.
Example 1:
{
    type: "set_property",
    path: "/Cameras/default/rotated/<object>",
    property: "zoom",
    value: 2.0
}
                
Example 2:
{
    type: "set_property",
    path: "/Lights/DirectionalLight/<object>",
    property: "intensity",
    value: 1.0
}
                
set_animation
Create an animation of any number of properties on any number of objects in the scene tree, and optionally start playing that animation.

Additional fields:

animations
A list of objects, each with two fields:
path
The path to the object whose property is begin animated. As with set_property above, you will need to append <object> to the path to set an object's intrinsic property.
clip
A Three.js AnimationClip in JSON form. The clip in turn has the following fields:
fps
The frame rate of the clip
name
A name for this clip. Not currently used.
tracks
The tracks (i.e. the properties to animate for this particular object. In Three.js, it is possible for a track to specify the name of the object it is attached to, and Three.js will automatically perform a depth-first search for a child object with that name. We choose to ignore that feature, since MeshCat already has unambiguous paths. So each track should just specify a property in its name field, with a single "." before that property name to signify that it applies to exactly the object given by the path above.

Each track has the following fields:

name
The property to be animated, with a leading "." (e.g. ".position")
type
The Three.js data type of the property being animated (e.g. "vector3" for the position property)
keys
The keyframes of the animation. The format is a list of objects, each with a field time (in frames) and value indicating the value of the animated property at that time.
options
Additional options controlling the animation. Currently supported values are:
play
Boolean [true]. Controls whether the animation should play immediately.
repetitions
Integer [1]. Controls the number of repetitions of the animation each time you play it.
Example:
{
    type: "set_animation",
    animations: [{
            path: "/Cameras/default",
            clip: {
                fps: 30,
                name: "default",
                tracks: [{
                    name: ".position"
                    type: "vector3",
                    keys: [{
                        time: 0,
                        value: [0, 1, .3]
                    },{
                        time: 80,
                        value: [0, 1, 2]
                    }],
                }]
            }
        },{
            path: "/meshcat/boxes",
            clip: {
                fps: 30,
                name: "default",
                tracks: [{
                    name: ".position"
                    type: "vector3",
                    keys: [{
                        time: 0,
                        value: [0, 1, 0]
                    },{
                        time: 80,
                        value: [0, -1, 0]
                    }],
                }]
            }
        }],
    options: {
        play: true,
        repetitions: 1
    }
}
                
set_target
Set the target of the 3D camera, around which it rotates. This is expressed in a left-handed coordinate system where y is up.

Example:

{
    "type": "set_target",
    value: [0., 1., 0.]
}
                
This sets the camera target to `(0, 1, 0)`
capture_image
Capture an image from the viewport. At the moment it will return the image at a provided resolution (by default 1920x1080).
{
    "type": "capture_image",
    "xres": 1920,
    "yres": 1080
}
                
This sets the camera target to `(0, 1, 0)`

WebSocket API

Viewer.connect(url)
Set up a web socket connection to a server at the given URL. The viewer will listen for messages on the socket as binary MsgPack blobs. Each message will be decoded using msgpack.decode() from msgpack-javascript and the resulting object will be passed directly to Viewer.handle_command() as documented above.

Note that we do support the MsgPack extension types listed in msgpack-javascript#extension-types, with additional support for the Float32Array type which is particularly useful for efficiently sending point data and for Uint32Array.

Useful Paths

The default MeshCat scene comes with a few objects at pre-set paths. You can replace, delete, or transform these objects just like anything else in the scene.

/Lights/DirectionalLight
The single directional light in the scene.
/Lights/AmbientLight
The ambient light in the scene.
/Grid
The square grid in the x-y plane, with 0.5-unit spacing.
/Axes
The red, green, and blue XYZ triad at the origin of the scene (invisible by default, click on "open controls" in the upper right to toggle its visibility).
/Cameras
The camera from which the scene is rendered (see below for details)
/Background
The background texture, with properties for "top_color" and "bottom_color" as well as a boolean "visible".

Camera Control

The camera is just another object in the MeshCat scene, so you can move it around with set_transform commands like any other object. You can also use set_target to change the camera's target. Please note that replacing the camera with set_object is not currently supported (but we expect to implement this in the future).

Controlling the camera is slightly more complicated than moving a single object because the camera actually has two important poses: the origin about which the camera orbits when you click-and-drag with the mouse, and the position of the camera itself. In addition, cameras and controls in Three.js assume a coordinate system in which the Y axis is upward. In robotics, we typically have the Z axis pointing up, and that's what's done in MeshCat. To account for this, the actual camera lives inside a few path elements:

/Cameras/default/rotated/<object>

The /rotated path element exists to remind users that its transform has been rotated to a Y-up coordinate system for the camera inside.

There is one additional complication: the built-in orbit and pan controls (which allow the user to move the view with their mouse) use the translation of only the intrinsic transform of the camera object itself to determine the radius of the orbit. That means that, in practice, you can allow the user to orbit by setting the position property at the path /Cameras/default/rotated/<object> to a nonzero value like [2, 0, 0], or you can lock the orbit controls by setting the position property at that path to [0, 0, 0]. Remember that whatever translation you choose is in the rotated, Y-up coordinate system that the Three.js camera expects. We're sorry.

Examples

To move the camera's center of attention to the point [1, 2, 3], while still allowing the user to orbit and pan manually, we suggest setting the transform of the /Cameras/default path to whatever center point you want and setting the position property of /Cameras/default/rotated/<object> to [2, 0, 0]. That means sending two commands:

{
    type: "set_transform",
    path: "/Cameras/default",
    matrix: new Float32Array([1, 0, 0, 0,
                              0, 1, 0, 0,
                              0, 0, 1, 0,
                              1, 2, 3, 1])  // the translation here is the camera's
                                            // center of attention
},
{
    type: "set_property",
    path: "/Cameras/default/rotated/<object>",
    property: "position",
    value: [2, 0, 0] // the offset of the camera about its point of rotation
}

To move the camera itself to the point [1, 2, 3] and lock its controls, we suggest setting the transform of /Cameras/default to the exact camera pose and setting the position property of /Cameras/default/rotated/<object> to [0, 0, 0]:

{
    type: "set_transform",
    path: "/Cameras/default",
    matrix: new Float32Array([1, 0, 0, 0,
                              0, 1, 0, 0,
                              0, 0, 1, 0,
                              1, 2, 3, 1])  // the translation here is now the camera's
                                            // exact pose
},
{
    type: "set_property",
    path: "/Cameras/default/rotated/<object>",
    property: "position",
    value: [0, 0, 0] // set to zero to lock the camera controls
}

Developing MeshCat

The MeshCat javascript sources live in src/index.js. We use webpack to bundle up MeshCat with its Three.js dependencies into a single javascript bundle. If you want to edit the MeshCat source, you'll need to regenerate that bundle. Fortunately, it's pretty easy:

  1. Install yarn. This should also install node and npm. Try running yarn -v and npm -v to make sure those programs are installed.
  2. Run yarn
    • This will read the project.json file and install all the necessary javascript dependencies.
  3. Run npm run build
    • This will run webpack and create the bundled output in dist/main.min.js. The build script will also watch for changes to the MeshCat source files and regenerate the bundle whenever those source files change.
  4. Try it out! You can load the bundled main.min.js in your own application, or you can open up dist/index.html in your browser.

Note that due to caching, you may need to do a hard refresh (shift+F5 or ctrl+shift+R) in your browser to reload the updated javascript bundle.

More Repositories

1

MeshCat.jl

WebGL-based 3D visualizer in Julia
Julia
232
star
2

meshcat-python

WebGL-based 3D visualizer for Python
Python
212
star
3

iris-distro

Iterative Regional Inflation by SDP
MATLAB
119
star
4

swig-eigen-numpy

A demonstration of a SWIG wrapper for a C++ library containing Eigen matrix types for use with Python and NumPy
C
53
star
5

CommonSubexpressions.jl

Naïve combined subexpression elimination in Julia
Julia
35
star
6

cryptics

Cryptic crossword solver
HTML
35
star
7

DynamicWalking2018.jl

Julia Robotics tutorial presented at Dynamic Walking 2018
Jupyter Notebook
30
star
8

ConditionalJuMP.jl

Automatic transformation of implications and complementarity into mixed-integer models in Julia
Julia
29
star
9

AdaptiveDistanceFields.jl

Adaptively sampled distance fields in Julia
Jupyter Notebook
28
star
10

NNLS.jl

Non-Negative Least Squares and Quadratic Program solver in Julia
Julia
27
star
11

ScratchHolograms

A set of tools to make abrasion (scratch) holography easier to do by hand
Ruby
25
star
12

StrandbeestRobot.jl

Simulation of a 12-legged parallel walking mechanism in Julia, inspired by Theo Jansen's Strandbeest
Jupyter Notebook
19
star
13

DrakeVisualizer.jl

Interface to the RobotLocomotion Drake Visualizer tool from Julia
Julia
18
star
14

ConcreteInterfaces.jl

Julia
15
star
15

RigidBodyTreeInspector.jl

Simple visualization for the RigidBodyDynamics.jl library
Julia
13
star
16

DetroitTechWatch2020.jl

Jupyter Notebook
11
star
17

Flatten.jl

Flatten Julia types to tuples or vectors, and un-flatten them later
Julia
11
star
18

LoewnerJohnEllipsoids.jl

Loewner-John inner and outer ellipsoids
Julia
10
star
19

SublimeLinter-contrib-mlint

A SublimeLinter3 interface to MATLAB mlint
Python
9
star
20

Ev3dev.jl

Julia bindings for ev3dev
Jupyter Notebook
9
star
21

WPepub

Convert a wordpress blog into an epub book
Python
9
star
22

CrypticCrosswords.jl

Cryptic crossword clue solver written in Julia
Julia
8
star
23

phd-thesis-defense

Slides from my PhD thesis defense at MIT CSAIL, October 2018
Shell
8
star
24

Mayday.jl

Sums-of-Squares optimization in Julia, powered by JuMP
Julia
7
star
25

LocalScopeBenchmarks.jl

A slightly friendlier interface to BenchmarkTools.jl
Julia
6
star
26

MatlabOptimizationTools

Some helpful scripts for running optimizations in Matlab
MATLAB
6
star
27

LVIS-dev

Experimental implementation of LVIS in Julia
Shell
6
star
28

CouenneNL.jl

Julia
5
star
29

ChartParsers.jl

Basic top-down and bottom-up chart parsing for context-free grammars in Julia
Julia
5
star
30

PyLCM.jl

Julia bindings for LCM, the Lightweight Communications and Marshalling library, via PyCall
Python
5
star
31

PyAnalogInt

Analog integers in python
Python
4
star
32

quadruped-walking-exploration

MATLAB
4
star
33

LCPSim.jl

Julia
4
star
34

pydrake-julia

Use Drake from Julia by way of Python
Jupyter Notebook
4
star
35

EdgeCameras.jl

Julia implementation of Bouman et al.'s edge camera algorithm
Jupyter Notebook
4
star
36

drake-mpc

Jupyter Notebook
3
star
37

en

fork of https://www.nodebox.net/code/index.php/Linguistics
Python
3
star
38

BeatTrack

Predictive beat tracking, eventually to be used with Bemis100
Python
3
star
39

juliet

More powerful package installation for Julia
Julia
3
star
40

StateMachineIterators.jl

Iterable finite state machines in Julia
Julia
3
star
41

Collective.jl

Your enigmatological distinctiveness will be added to our own
Julia
3
star
42

CrypticServer.jl

Julia
2
star
43

StaticPolynomials.jl

Jupyter Notebook
2
star
44

Trivia.jl

OpenTDB trivia interface in Julia
Julia
2
star
45

oscilloscope-music

Decoding oscilloscope music in Julia
Jupyter Notebook
2
star
46

convex-segmentation

Find convex regions in a binary image
MATLAB
2
star
47

SpatialFields.jl

Scalar and vector fields, including Hermite Radial Basis Function fields
Jupyter Notebook
2
star
48

DygraphsOverviewMode

An overview mode for dygraphs that makes it easy to navigate through a dataset
JavaScript
2
star
49

interval-inverse-kinematics

Jupyter Notebook
2
star
50

rdeits-vim

An ongoing attempt to turn vim into a nice editing environment for myself.
Vim Script
2
star
51

julia-ev3-report

TeX
2
star
52

Identify-Polygons

Identify 2D polygons from a point cloud using a genetic algorithm
Python
2
star
53

adaptive-PIV

An FPGA implementation of Particle Image Velocimetry for 6.375 at MIT
MATLAB
2
star
54

FluxExtensions.jl

Julia
1
star
55

swigmake

Helpful swig interface files
C
1
star
56

threshold-qp

1
star
57

Puzzle-Anagrams

Anagram tools for an MIT puzzle
Python
1
star
58

julia-matlab-fortran-fem-benchmark

Julia, Matlab, and Fortan Finite Element FEM benchmark and direct comparison
Fortran
1
star
59

gaited-footstep-planner-exploration

MATLAB
1
star
60

sublime-config

Sublime Text 2 Configuration
1
star
61

6.438-formula-sheet

1
star
62

Noticer.jl

Julia
1
star
63

LearningMPC.jl

Julia
1
star
64

ImplicitSkinning.jl

Implementation of implicit surface reconstruction with HRBF by Vaillant et al.
Jupyter Notebook
1
star
65

dotfiles

My various *nix configuration files`
Vim Script
1
star
66

julia-metaprogramming

Jupyter Notebook
1
star
67

crazytags

Python
1
star
68

coasters

Cedar Point roller coaster analysis
Python
1
star
69

SCOWL-mirror

Mirror of SCOWL from https://sourceforge.net/projects/wordlist/files/SCOWL/2018.04.16/
Shell
1
star
70

MultiprecisionLP.jl

Julia
1
star
71

action-sequence

Convert a set of images into a multiple-exposure action sequence
Python
1
star
72

pypolyhedron

Mirror of PyPolyhedron by Pearu Peterson from http://cens.ioc.ee/projects/polyhedron/
C
1
star
73

juggling

MATLAB
1
star
74

rrt-explorations

Playing around with Rapidly Expanding Random Trees
1
star
75

rdeits.github.com

HTML
1
star
76

Behaviors.jl

Finite-state-machine behaviors
Julia
1
star
77

TiddlyWiki-Project-Journal

A modification of Jeremy Ruston's TiddlyWiki for my own project and research tracking.
1
star
78

iris

IRIS algorithm for finding large convex volumes of free space
MATLAB
1
star
79

icra-2019-lvis

Slides for my talk at ICRA 2019
Shell
1
star