• Stars
    star
    1,639
  • Rank 26,846 (Top 0.6 %)
  • Language
    JavaScript
  • License
    Apache License 2.0
  • Created about 5 years ago
  • Updated 12 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A model-agnostic visual debugging tool for machine learning

Gitpod Ready-to-Code Build Status CII Best Practices

Manifold

This project is stable and being incubated for long-term support.

Manifold

Manifold is a model-agnostic visual debugging tool for machine learning.

Understanding ML model performance and behavior is a non-trivial process, given the intrisic opacity of ML algorithms. Performance summary statistics such as AUC, RMSE, and others are not instructive enough for identifying what went wrong with a model or how to improve it.

As a visual analytics tool, Manifold allows ML practitioners to look beyond overall summary metrics to detect which subset of data a model is inaccurately predicting. Manifold also explains the potential cause of poor model performance by surfacing the feature distribution difference between better and worse-performing subsets of data.

Table of contents

Prepare Your Data

There are 2 ways to input data into Manifold:

In either case, data that's directly input into Manifold should follow this format:

const data = {
  x:     [...],         // feature data
  yPred: [[...], ...]   // prediction data
  yTrue: [...],         // ground truth data
};

Each element in these arrays represents one data point in your evaluation dataset, and the order of data instances in x, yPred and yTrue should all match. The recommended instance count for each of these datasets is 10000 - 15000. If you have a larger dataset that you want to analyze, a random subset of your data generally suffices to reveal the important patterns in it.

x: {Object[]}

A list of instances with features. Example (2 data instances):

[{feature_0: 21, feature_1: 'B'}, {feature_0: 36, feature_1: 'A'}];
yPred: {Object[][]}

A list of lists, where each child list is a prediction array from one model for each data instance. Example (3 models, 2 data instances, 2 classes ['false', 'true']):

[
  [{false: 0.1, true: 0.9}, {false: 0.8, true: 0.2}],
  [{false: 0.3, true: 0.7}, {false: 0.9, true: 0.1}],
  [{false: 0.6, true: 0.4}, {false: 0.4, true: 0.6}],
];
yTrue: {Number[] | String[]}

A list, ground truth for each data instance. Values must be numbers for regression models, must be strings that match object keys in yPred for classification models. Example (2 data instances, 2 classes ['false', 'true']):

['true', 'false'];

Interpret visualizations

This guide explains how to interpret Manifold visualizations.

Manifold consists of:

Performance Comparison View

This visualization is an overview of performance of your model(s) across different segments of your data. It helps you identify under-performing data subsets for further inspection.

Reading the chart

performance comparison view

  1. X axis: performance metric. Could be log-loss, squared-error, or raw prediction.
  2. Segments: your dataset is automatically divided into segments based on performance similarity between instances, across models.
  3. Colors: represent different models.

performance comparison view unit

  1. Curve: performance distribution (of one model, for one segment).
  2. Y axis: data count/density.
  3. Cross: the left end, center line, and right end are the 25th, 50th and 75th percentile of the distribution.

Explanation

Manifold uses a clustering algorithm (k-Means) to break prediction data into N segments based on performance similarity.

The input of the k-Means is per-instance performance scores. By default, that is the log-loss value for classification models and the squared-error value for regression models. Models with a lower log-loss/squared-error perform better than models with a higher log-loss/squared-error.

If you're analyzing multiple models, all model performance metrics will be included in the input.

Usage

  • Look for segments of data where the error is higher (plotted to the right). These are areas you should analyze and try to improve.

  • If you're comparing models, look for segments where the log-loss is different for each model. If two models perform differently on the same set of data, consider using the better-performing model for that part of the data to boost performance.

  • After you notice any performance patterns/issues in the segments, slice the data to compare feature distribution for the data subset(s) of interest. You can create two segment groups to compare (colored pink and blue), and each group can have 1 or more segments.

Example

performance comparison view example

Data in Segment 0 has a lower log-loss prediction error compared to Segments 1 and 2, since curves in Segment 0 are closer to the left side.

In Segments 1 and 2, the XGBoost model performs better than the DeepLearning model, but DeepLearning outperforms XGBoost in Segment 0.


Feature Attribution View

This visualization shows feature values of your data, aggregated by user-defined segments. It helps you identify any input feature distribution that might correlate with inaccurate prediction output.

Reading the chart

feature attribution view

  1. Histogram / heatmap: distribution of data from each data slice, shown in the corresponding color.
  2. Segment groups: indicates data slices you choose to compare against each other.
  3. Ranking: features are ranked by distribution difference between slices.

feature attribution view unit

  1. X axis: feature value.
  2. Y axis: data count/density.
  3. Divergence score: measure of difference in distributions between slices.

Explanation

After you slice the data to create segment groups, feature distribution histograms/heatmaps from the two segment groups are shown in this view.

Depending on the feature type, features can be shown as heatmaps on a map for geo features, distribution curve for numerical features, or distribution bar chart for categorical features. (In bar charts, categories on the x-axis are sorted by instance count difference. Look for differences between the two distributions in each feature.)

Features are ranked by their KL-Divergence - a measure of difference between the two contrasting distributions. The higher the divergence is, the more likely this feature is correlated with the factor that differentiates the two Segment Groups.

Usage

  • Look for the differences between the two distributions (pink and blue) in each feature. They represent the difference in data from the two segment groups you selected in the Performance Comparison View.

Example

feature attribution view example

Data in Groups 0 and 1 have obvious differences in Features 0, 1, 2 and 3; but they are not so different in features 4 and 5.

Suppose Data Groups 0 and 1 correspond to data instances with low and high prediction error respectively, this means that data with higher errors tend to have lower feature values in Features 0 and 1, since peak of pink curve is to the left side of the blue curve.


Geo Feature View

If there are geospatial features in your dataset, they will be displayed on a map. Lat-lng coordinates and h3 hexagon ids are currently supoorted geo feature types.

Reading the chart

geo feature view lat-lng

  1. Feature name: when multiple geo features exist, you can choose which one to display on the map.
  2. Color-by: if a lat-lng feature is chosen, datapoints are colored by group ids.
  3. Map: Manifold defaults to display the location and density of these datapoints using a heatmap.

geo feature view hex id

  1. Feature name: when choosing a hex-id feature to display, datapoints with the same hex-id are displayed in aggregate.
  2. Color-by: you can color the hexagons by: average model performance, percentage of segment group 0, or total count per hexagon.
  3. Map: all metrics that are used for coloring are also shown in tooltips, on the hexagon level.

Usage

  • Look for the differences in geo location between the two segment groups (pink and grey). They represent the spation distribution difference between the two subsets you previously selected.

Example

In the first map above, Group 0 has a more obvious tendency to be concentrated in downtown San Francisco area.

Using the Demo App

To do a one-off evaluation using static outputs of your ML models, use the demo app. Otherwise, if you have a system that programmatically generates ML model outputs, you might consider using the Manifold component directly.

Running Demo App Locally

Run the following commands to set up your environment and run the demo:

# install all dependencies in the root directory
yarn
# demo app is in examples/manifold directory
cd examples/manifold
# install dependencies for the demo app
yarn
# run the app
yarn start

Now you should see the demo app running at localhost:8080.

Upload CSV to Demo App

csv upload interface

Once the app starts running, you will see the interface above asking you to upload "feature", "prediction" and "ground truth" datasets to Manifold. They correspond to x, yPred, and yTrue in the "prepare your data" section, and you should prepare your CSV files accordingly, illustrated below:

Field x (feature) yPred (prediction) yTrue (ground truth)
Number of CSVs 1 multiple 1
Illustration of CSV format

Note, the index columns should be excluded from the input file(s). Once the datasets are uploaded, you will see visualizations generated by these datasets.

Using the Component

Embedding the Manifold component in your app allows you to programmatically generate ML model data and visualize. Otherwise, if you have some static output from some models and want to do a one-off evaluation, you might consider using the demo app directly.

Here are the basic steps to import Manifold into your app and load data for visualizing. You can also take a look at the examples folder.

Install Manifold

$ npm install @mlvis/manifold styled-components styletron-engine-atomic styletron-react

Load and Convert Data

In order to load your data files to Manifold, use the loadLocalData action. You could also reshape your data into the required Manifold format using dataTransformer.

import {loadLocalData} from '@mlvis/manifold/actions';

// create the following action and pass to dispatch
loadLocalData({
  fileList,
  dataTransformer,
});
fileList: {Object[]}

One or more datasets, in CSV format. Could be ones that your backend returns.

dataTransformer: {Function}

A function that transforms fileList into the Manifold input data format. Default:

const defaultDataTransformer = fileList => ({
  x: [],
  yPred: [],
  yTrue: [],
});

Mount reducer

Manifold uses Redux to manage its internal state. You need to register manifoldReducer to the main reducer of your app:

import manifoldReducer from '@mlvis/manifold/reducers';
import {combineReducers, createStore, compose} from 'redux';

const initialState = {};
const reducers = combineReducers({
  // mount manifold reducer in your app
  manifold: manifoldReducer,

  // Your other reducers here
  app: appReducer,
});

// using createStore
export default createStore(reducer, initialState);

Mount Component

If you mount manifoldReducer in another address instead of manifold in the step above, you need to specify the path to it when you mount the component with the getState prop. width and height are both needed explicitly. If you have geospatial features and need to see them on a map, you also need a mapbox token.

import Manifold from '@mlvis/manifold';
const manifoldGetState = state => state.pathTo.manifold;
const yourMapboxToken = ...;

const Main = props => (
  <Manifold
    getState={manifoldGetState}
    width={width}
    height={height}
    mapboxToken={yourMapboxToken}
  />
);

Styling

Manifold uses baseui, which uses Styletron as a styling engine. If you don't already use Styletron in other parts of your app, make sure to wrap Manifold with the styletron provider.

Manifold uses the baseui theming API. The default theme used by Manifold is exported as THEME. You can customize the styling by extending THEME and passing it as a theme prop of the Manifold component.

import Manifold, {THEME} from '@mlvis/manifold';
import {Client as Styletron} from 'styletron-engine-atomic';
import {Provider as StyletronProvider} from 'styletron-react';

const engine = new Styletron();
const myTheme = {
  ...THEME,
  colors: {
    ...THEME.colors,
    primary: '#ff0000',
  },
}

const Main = props => (
  <StyletronProvider value={engine}>
    <Manifold
      getState={manifoldGetState}
      theme={myTheme}
    />
  </StyletronProvider>
);

Built With

Contributing

Please read our code of conduct before you contribute! You can find details for submitting pull requests in the CONTRIBUTING.md file. Refer to the issue template.

Versioning

We document versions and changes in our changelog - see the CHANGELOG.md file for details.

License

Apache 2.0 License

More Repositories

1

react-vis

Data Visualization Components
JavaScript
8,632
star
2

baseweb

A React Component library implementing the Base design language
TypeScript
8,574
star
3

cadence

Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.
Go
7,654
star
4

RIBs

Uber's cross-platform mobile architecture framework.
Kotlin
7,624
star
5

kraken

P2P Docker registry capable of distributing TBs of data in seconds
Go
5,768
star
6

prototool

Your Swiss Army Knife for Protocol Buffers
Go
5,051
star
7

causalml

Uplift modeling and causal inference with machine learning algorithms
Python
4,592
star
8

h3

Hexagonal hierarchical geospatial indexing system
C
4,491
star
9

NullAway

A tool to help eliminate NullPointerExceptions (NPEs) in your Java code with low build-time overhead
Java
3,469
star
10

AutoDispose

Automatic binding+disposal of RxJava streams.
Java
3,343
star
11

aresdb

A GPU-powered real-time analytics storage and query engine.
Go
2,976
star
12

react-digraph

A library for creating directed graph editors
JavaScript
2,573
star
13

piranha

A tool for refactoring code related to feature flag APIs
Java
2,206
star
14

orbit

A Python package for Bayesian forecasting with object-oriented design and probabilistic models under the hood.
Python
1,782
star
15

ios-snapshot-test-case

Snapshot view unit tests for iOS
Objective-C
1,755
star
16

petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
Python
1,724
star
17

needle

Compile-time safe Swift dependency injection framework
Swift
1,708
star
18

okbuck

OkBuck is a gradle plugin that lets developers utilize the Buck build system on a gradle project.
Java
1,535
star
19

UberSignature

Provides an iOS view controller allowing a user to draw their signature with their finger in a realistic style.
Objective-C
1,281
star
20

nanoscope

An extremely accurate Android method tracing tool.
HTML
1,237
star
21

tchannel

network multiplexing and framing protocol for RPC
Thrift
1,148
star
22

queryparser

Parsing and analysis of Vertica, Hive, and Presto SQL.
Haskell
1,067
star
23

fiber

Distributed Computing for AI Made Simple
Python
1,041
star
24

neuropod

A uniform interface to run deep learning models from multiple frameworks
C++
927
star
25

uReplicator

Improvement of Apache Kafka Mirrormaker
Java
888
star
26

pam-ussh

uber's ssh certificate pam module
Go
826
star
27

ringpop-go

Scalable, fault-tolerant application-layer sharding for Go applications
Go
810
star
28

h3-js

h3-js provides a JavaScript version of H3, a hexagon-based geospatial indexing system.
JavaScript
779
star
29

xviz

A protocol for real-time transfer and visualization of autonomy data
JavaScript
760
star
30

mockolo

Efficient Mock Generator for Swift
Swift
759
star
31

h3-py

Python bindings for H3, a hierarchical hexagonal geospatial indexing system
Python
739
star
32

streetscape.gl

Visualization framework for autonomy and robotics data encoded in XVIZ
JavaScript
702
star
33

react-view

React View is an interactive playground, documentation and code generator for your components.
TypeScript
685
star
34

nebula.gl

A suite of 3D-enabled data editing overlays, suitable for deck.gl
TypeScript
648
star
35

RxDogTag

Automatic tagging of RxJava 2+ originating subscribe points for onError() investigation.
Java
642
star
36

peloton

Unified Resource Scheduler to co-schedule mixed types of workloads such as batch, stateless and stateful jobs in a single cluster for better resource utilization.
Go
632
star
37

motif

A simple DI API for Android / Java
Kotlin
532
star
38

signals-ios

Typeful eventing
Objective-C
526
star
39

tchannel-go

Go implementation of a multiplexing and framing protocol for RPC calls
Go
476
star
40

marmaray

Generic Data Ingestion & Dispersal Library for Hadoop
Java
474
star
41

grafana-dash-gen

grafana dash dash dash gen
JavaScript
467
star
42

zanzibar

A build system & configuration system to generate versioned API gateways.
Go
445
star
43

clay

Clay is a framework for building RESTful backend services using best practices. It’s a wrapper around Flask.
Python
441
star
44

astro

Astro is a tool for managing multiple Terraform executions as a single command
Go
431
star
45

NEAL

πŸ”ŽπŸž A language-agnostic linting platform
OCaml
423
star
46

react-vis-force

d3-force graphs as React Components.
JavaScript
402
star
47

arachne

An always-on framework that performs end-to-end functional network testing for reachability, latency, and packet loss
Go
387
star
48

cadence-web

Web UI for visualizing workflows on Cadence
JavaScript
376
star
49

Python-Sample-Application

Python
373
star
50

rides-ios-sdk

Uber Rides iOS SDK (beta)
Swift
364
star
51

stylist

A stylist creates cool styles. Stylist is a Gradle plugin that codegens a base set of Android XML themes.
Kotlin
355
star
52

storagetapper

StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service
Go
331
star
53

swift-concurrency

Concurrency utilities for Swift
Swift
321
star
54

cyborg

Display Android Vectordrawables on iOS.
Swift
301
star
55

RemoteShuffleService

Remote shuffle service for Apache Spark to store shuffle data on remote servers.
Java
288
star
56

rides-android-sdk

Uber Rides Android SDK (beta)
Java
286
star
57

h3-go

Go bindings for H3, a hierarchical hexagonal geospatial indexing system
Go
267
star
58

h3-java

Java bindings for H3, a hierarchical hexagonal geospatial indexing system
Java
259
star
59

h3-py-notebooks

Jupyter notebooks for h3-py, a hierarchical hexagonal geospatial indexing system
Jupyter Notebook
239
star
60

hermetic_cc_toolchain

Bazel C/C++ toolchain for cross-compiling C/C++ programs
Starlark
219
star
61

artist

An artist creates views. Artist is a Gradle plugin that codegens a base set of Android Views.
Kotlin
209
star
62

geojson2h3

Conversion utilities between H3 indexes and GeoJSON
JavaScript
206
star
63

tchannel-node

JavaScript
205
star
64

RxCentralBle

A reactive, interface-driven central role Bluetooth LE library for Android
Java
199
star
65

uberalls

Track code coverage metrics with Jenkins and Phabricator
Go
187
star
66

SwiftCodeSan

SwiftCodeSan is a tool that "sanitizes" code written in Swift.
Swift
172
star
67

rides-python-sdk

Uber Rides Python SDK (beta)
Python
169
star
68

doubles

Test doubles for Python.
Python
166
star
69

logtron

A logging MACHINE
JavaScript
158
star
70

cassette

Store and replay HTTP requests made in your Python app
Python
138
star
71

cadence-java-client

Java framework for Cadence Workflow Service
Java
137
star
72

UBTokenBar

Flexible and extensible UICollectionView based TokenBar written in Swift
Swift
136
star
73

athenadriver

A fully-featured AWS Athena database driver (+ athenareader https://github.com/uber/athenadriver/tree/master/athenareader)
Go
133
star
74

tchannel-java

A Java implementation of the TChannel protocol.
Java
132
star
75

android-template

This template provides a starting point for open source Android projects at Uber.
Java
127
star
76

bayesmark

Benchmark framework to easily compare Bayesian optimization methods on real machine learning tasks
Python
126
star
77

crumb

An annotation processor for breadcrumbing metadata across compilation boundaries.
Kotlin
124
star
78

py-find-injection

Look for SQL injection attacks in python source code
Python
119
star
79

rides-java-sdk

Uber Rides Java SDK (beta)
Java
101
star
80

startup-reason-reporter

Reports the reason why an iOS App started.
Objective-C
97
star
81

uber-poet

A mock swift project generator & build runner to help benchmark various module dependency graphs.
Python
94
star
82

charlatan

A Python library to efficiently manage and install database fixtures
Python
89
star
83

cadence-java-samples

Java
89
star
84

simple-store

Simple yet performant asynchronous file storage for Android
Java
82
star
85

swift-abstract-class

Compile-time abstract class validation for Swift
Swift
82
star
86

tchannel-python

Python implementation of the TChannel protocol.
Python
77
star
87

eight-track

Record and playback HTTP requests
JavaScript
70
star
88

client-platform-engineering

A collection of cookbooks, scripts and binaries used to manage our macOS, Ubuntu and Windows endpoints
Ruby
69
star
89

multidimensional_urlencode

Python library to urlencode a multidimensional dict
Python
67
star
90

lint-checks

A set of opinionated and useful lint checks
Kotlin
67
star
91

uncaught-exception

Handle uncaught exceptions.
JavaScript
66
star
92

swift-common

Common code used by various Uber open source projects
Swift
64
star
93

uberscriptquery

UberScriptQuery, a SQL-like DSL to make writing Spark jobs super easy
Java
57
star
94

sentry-logger

A Sentry transport for Winston
JavaScript
55
star
95

graph.gl

WebGL2-Powered Visualization Components for Graph Visualization
JavaScript
50
star
96

nanoscope-art

C++
47
star
97

assume-role-cli

CLI for AssumeRole is a tool for running programs with temporary credentials from AWS's AssumeRole API.
Go
47
star
98

airlock

A prober to probe HTTP based backends for health
JavaScript
47
star
99

mutornadomon

Easy-to-install monitor endpoint for Tornado applications
Python
46
star
100

kafka-logger

A kafka logger for winston
JavaScript
45
star