• Stars
    star
    451
  • Rank 93,700 (Top 2 %)
  • Language
    Go
  • License
    MIT License
  • Created about 7 years ago
  • Updated 17 days ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A build system & configuration system to generate versioned API gateways.

GoDoc Build Status Coverage Status Go Report Card

Zanzibar is an extensible framework to build configuration driven web applications. The goal of Zanzibar is to simplify application development into two steps:

  1. write configurations for the application and its components;
  2. write code to implement and test business logic.

Based on the configurations, Zanzibar generates boilerplates and glue code, wires them up with your business domain code and the runtime components Zanzibar provides to create a deployable binary.

The builtin components of Zanzibar makes it easy to develop microservices and gateway services that proxy or orchestrate microservices. It is also simple to extend Zanzibar with custom plugins to ease the development of applications that suit your specific needs.

Table of Contents

Concepts

Zanzibar is built on three pillars: module, config, code generation.

Module

Modules are the components that a Zanzibar application is made of. A module belongs to a ModuleClass, has a type and can have dependencies on other modules.

ModuleClass

ModuleClass abstracts the functionality of a specific class of components. Zanzibar predefines a few module classes, i.e., client, endpoint, middleware and service. Each represents a corresponding abstraction:

ModuleClass Abstraction
client clients to communicate with downstreams, e.g., database clients and RPC clients
endpoint application interfaces exposed to upstreams
middleware common functionality that has less to do with business logic, e.g., rate limiting middleware
service a collection of endpoints that represents high level application abstraction, e.g., a demo service that prints "Hello World!"

Type

The module type differentiates module instances of the same ModuleClass with further classification. Types are somewhat arbitrary as they are not necessarily abstractions but indications about how Zanzibar should treat the modules.

Client

A client module could be of type http, tchannel or custom, where http or tchannel means Zanzibar will generate a client with given configuration that speaks that protocol while custom means the client is fully provided and Zanzibar will use it as is without code generation. In other words, http and tchannel clients are configuration driven (no user code) whereas custom clients are user-defined and can be "smart clients".

Endpoint

An endpoint module could also be of type http or tchannel, which determines the protocol that the endpoint will be made available to invoke externally via the Zanzibar router. While endpoint modules do not have custom type, each method of an endpoint module has a workflowType that indicates the type of workflow the endpoint method fulfills. The builtin workflow type is httpClient, tchannelClient and custom, where httpClient and tchannelClient means the endpoint method workflow is to proxy to a client, and custom means the workflow is fulfilled by user code, see more in Custom Workflow.

Note that workflow type is likely to be deprecated in the future so that proxy to a client will be no longer a builtin option.

Middleware

The builtin type of middleware module is default.

Service

The builtin service type is gateway (it is likely to change in the future, because default is probably a better name).

Note Zanzibar has support for user-defined module classes and module types in case the builtin types are not sufficient. The preferred way of extending Zanzibar is through user defined module classes and module types.

Dependency Injection

Module dependencies describe the relationships among various modules. The dependency relationship is critical to correctly assemble the modules to a full application.

Dependency Injection

A module is expected to define its immediate or direct dependencies. Zanzibar generates a module constructor with dependent modules as parameters, and passes the dependencies to the constructor during initilizaiton.

Module Initialization

Zanzibar also constructs a full global dependency graph for a given set of modules. This graph is used to initialize modules in the correct order, e.g. leaf modules are initialized first and then passed to the constructors of parent modules for initialization.

Dependency Rules

To establish and enforce abstraction boundaries, dependency rules at ModuleClass level are necessary. Zanzibar predefines the following dependency rules for built module classes:

ModuleClass DependsOn DependedBy
client N/A middleware, endpoint
middleware client endpoint
endpoint client, middleware service
service endpoint N/A

This table exhausts the possible immediate or direct dependency relationships among builtin module classes. Take endpoint module class for example, an endpoint module can depend on client or middleware modules but not endpoint or service modules. The reasoning for such rules aligns with the abstractions the module classes represent.

The ModuleClass struct has DependsOn and DependedBy public fields, which makes it simple to extend the dependency rules with custom module class, e.g., we can define a custom module class task that abstracts common business workflow by setting its DependsOn field to client and DependedBy field to endpoint.

Config

Configurations are the interface that developers interact with when using the Zanzibar framework, they make up most of Zazibar's API. Various configurarions contain essential meta information of a Zanzibar application and its components. They are source of truth of the application.

Config Layout

Because configurations are the core of a Zanzibar application, we create a root directory to host configuration files when starting a Zanzibar application. There are a few typical directories and files under the root directory. Take example-gateway for example:

example-gateway                 # root directory
โ”œโ”€โ”€ bin                         # directory for generated application binaries
โ”‚ย ย  โ””โ”€โ”€ example-gateway         # generated example-gateway binary
โ”œโ”€โ”€ build                       # directory for all generated code
โ”‚ย ย  โ”œโ”€โ”€ clients                 # generated mocks and module initializers for clients
โ”‚ย ย  โ”œโ”€โ”€ endpoints               # generated mocks and module initializers for endpoints
โ”‚ย ย  โ”œโ”€โ”€ gen-code                # generated structs and (de)serializers by Thrift compiler
โ”‚ย ย  โ”œโ”€โ”€ middlewares             # generated module initializers for middlewares
โ”‚ย ย  โ”‚   โ””โ”€โ”€ default             # generated module initializers for default middlewares
โ”‚ย ย  โ””โ”€โ”€ services                # generated mocks and module intialziers for services
โ”œโ”€โ”€ build.yaml                  # config file for Zanzibar code generation, see below for details
โ”œโ”€โ”€ clients                     # config directory for modules of client module class
โ”‚ย ย  โ””โ”€โ”€ bar                     # config directory for a client named 'bar'
โ”œโ”€โ”€ config                      # config directory for application runtime properties
โ”‚ย ย  โ”œโ”€โ”€ production.yaml         # config file for production environment
โ”‚ย ย  โ””โ”€โ”€ test.yaml               # config file for test environment
โ”œโ”€โ”€ copyright_header.txt        # optional copyright header for open source application
โ”œโ”€โ”€ endpoints                   # config directory for modules of endpoint module class
โ”‚ย ย  โ””โ”€โ”€ bar                     # config directory for an endpoint named 'bar'
โ”œโ”€โ”€ idl                         # idl directory for all thrift files
โ”‚ย ย  โ”œโ”€โ”€ clients                 # idl directory for client thrift files
โ”‚ย ย  โ””โ”€โ”€ endpoints               # idl directory for endpoint thrift files
โ”œโ”€โ”€ middlewares                 # config directory for modules of middleware module class
โ”‚ย ย  โ”œโ”€โ”€ transform-response      # config directory for a middleware named 'transform-response'
โ”‚ย ย  โ”œโ”€โ”€ default                 # directory for all default middlewares
โ”‚ย ย  โ”‚   โ””โ”€โ”€ log-publisher       # config directory for a default middleware named 'log-publisher'
โ”‚ย ย  โ””โ”€โ”€ default.yaml            # config file describing default middlewares and their execution order   
โ””โ”€โ”€ services                    # config directory for modules of service module class
    โ””โ”€โ”€ example-gateway         # config directory for a service named 'example-gateway'

Module Config

Each module must have a config file so that it can be recognized by Zanzibar. This section explains how the module config files are organized and what goes into them.

General Layout

Under the application root directory, there should be a corresponding top level config directory for each module class. For Zanzibar builtin module classes, the name of the directory is the plural of the module class name, e.g., a clients directory for client module class. The directory name is used when registering generator for a module class (example). While it is not required, the same directory naming convention should be followed when defining custom module classes.

Under a module class directory, there should be a corresponding config directory for each module, e.g., the clients directory has a few subdirectories and each of them corresponds to a module.

Under a module directory, there should be a YAML file that contains the meta information of that module. It is required that the file is named of {$ModuleClass}-config.yaml, e.g. the path to the YAML config file of bar client module is clients/bar/client-config.yaml, similarly the path to the YAML config file of bar endpoint module is endpoints/bar/endpoint-config.yaml.

Non-Config Content

Besides the YAML config file, the module directory also contains other necessary directories/files. For example, the quux client is a custom (non-generated) client, its module config directory has following layout:

quxx                        # client module config directory
โ”œโ”€โ”€ client-config.yaml      # client module config file
โ”œโ”€โ”€ fixture                 # directory for fixtures used for testing
โ”‚ย ย  โ””โ”€โ”€ fixure.go           # fixtures that can be used by a generated mock client for testing
โ””โ”€โ”€ quux.go                 # custom client implementation, package is imported by generated code

For client and endpoint modules of builtin type custom, Zanzibar expects user code to be placed in the module directory. This is important because Zaznibar-generated code refers to user code by importing the package of the module directory path. Furthermore, user code of custom client and endpoint modules must also define and implement necessary public types and interfaces so that Zanzibar can wire up the modules.

Custom Client

For client module of custom type, user code must define a Client interface and a NewClient constructor that returns the Client interface. Below is the example code snippet for the quux custom client:

package quux

import "github.com/uber/zanzibar/examples/example-gateway/build/clients/quux/module"

type Client interface {
	Echo(string) string
}

func NewClient(deps *module.Dependencies) Client {
	return &quux{}
}

type quux struct{}

func (c *quux) Echo(s string) string { return s }

Note the type of deps parameter passed to NewClient constructor function is generated by Zanzibar, as indicated by the import path. Zanzibar takes care of initializing and passing in the acutal deps argument, as mentioned in Dependency Injection.

Circuit Breaker

For increasing overall system resiliency, zanzibar uses a Circuit Breaker which avoids calling client when there is an increase in failure rate beyond a set threshold. After a sleepWindowInMilliseconds, client calls are attempted recovery by going in half-open and then close state.

circuitBreakerDisabled: Default false. To disable the circuit-breaker:

    "clients.<clientID>.circuitBreakerDisabled" : true

maxConcurrentRequests: Default 50. To set how many requests can be run at the same time, beyond which requests are rejected:

   "clients.<clientID>.maxConcurrentRequests": 50

errorPercentThreshold: Default 20. To set error percent threshold beyond which to trip the circuit open:

    "clients.<clientID>.errorPercentThreshold": 20

requestVolumeThreshold: Default 20. To set minimum number of requests that will trip the circuit in a rolling window of 10 (For example, if the value is 20, then if only 19 requests are received in the rolling window of 10 seconds the circuit will not trip open even if all 19 failed):

    "clients.<clientID>.requestVolumeThreshold" : true

sleepWindowInMilliseconds: Default 5000. To set the amount of time, after tripping the circuit, to reject requests before allowing attempts again to determine if the circuit should again be closed:

    "clients.<clientID>.sleepWindowInMilliseconds" : true
Custom Workflow

For endpoint module of custom workflow type, user code must define a New{$endpoint}{$method}Workflow constructor that returns the Zanzibar-generated {$endpoint}{$method}Workflow interface which has a sole Handle method. Below is the example code snippet for the contacts custom endpoint:

package contacts

import (
	"context"

	"github.com/uber/zanzibar/examples/example-gateway/build/endpoints/contacts/module"
	"github.com/uber/zanzibar/examples/example-gateway/build/endpoints/contacts/workflow"
	contacts "github.com/uber/zanzibar/examples/example-gateway/build/gen-code/endpoints-idl/endpoints/contacts/contacts"

	zanzibar "github.com/uber/zanzibar/runtime"
	"go.uber.org/zap"
)

func NewContactsSaveContactsWorkflow(
    c *module.ClientDependencies,
    l *zap.Logger,
) workflow.ContactsSaveContactsWorkflow { return &saveContacts{ ... } }

type saveContacts struct { ... }

func (w *saveContacts) Handle(
	ctx context.Context,
	headers zanzibar.Header,
	r *contacts.SaveContactsRequest,
) (*contacts.SaveContactsResponse, zanzibar.Header, error) { ... }

The idea of the workflow constructor is similar to the client constructor, with a couple of differences:

  • the first parameter is specifically ClientDependencies and there is an additional logger parameter, this will be changed in the future so that the dependency parameter is generalized;
  • the return value is an interface generated by Zanzibar, the parameter and return value of the Handle method refers to structs generated by Thrift compiler based on the endpoint thrift file configured in the endpoint-config.yaml, see more in Config Schema.
Grouping

Zanzibar allows nesting module config directories in the sub-directories of a module class config directory. This is useful to group related modules under a sub-directory. For example, the tchannel directory groups all TChannel endpoint modules:

endpoints
โ”œโ”€โ”€ ...
โ””โ”€โ”€ tchannel                    # this directory does not correspond to a module, it represents a group
    โ””โ”€โ”€ baz                     # module config directory under the 'tchannel' group
        โ”œโ”€โ”€ baz_call.go
        โ”œโ”€โ”€ baz_call_test.go
        โ”œโ”€โ”€ call.yaml
        โ””โ”€โ”€ endpoint-config.yaml
Config Schema

Modules of different ModuleClass and type are likely to have different fields in their config files. Developers are expected to write module config files according to the schemas.

Note: fields are absent in config schemas but present in examples are experimental.

The endpoint module config is different from other module classes as it has multiple YAML files, where each endpoint method corresponds to a YAML file and the endpoint-config.yaml file refers to them.

endpoints/multi
โ”œโ”€โ”€ endpoint-config.yaml    # has a field 'endpoints' that is a list and contains helloA and helloB
โ”œโ”€โ”€ helloA.yaml             # config file for method helloA
โ””โ”€โ”€ helloB.yaml             # config file for method helloB

The reason for such layout is to avoid a large endpoint-config.yaml file when an endpoint has many methods.

Application Config

Besides the module configs, Zanzibar also expects a YAML file that configures necessary properties to boostrap the code generation process of a Zanzibar application. The schema for application config is defined here.

Unlike the module configs, there is no restriction on how this config file should be named. It can be named {$appName}.yaml or build.yaml as it is in example-gateway, as long as it is passed correctly as an argument to the code generation runner.

In this config file, you can specify the paths from which to discover modules. You can also specify default dependencies.

Default Dependencies allow module classes to include instances of other module classes as default dependencies. This means that no explicit configurations are required for certain module instances to be included as a dependency. e.g., we can include clients/logger as a default dependency for endpoint, and every endpoint will have clients/logger as a dependency in its module/dependencies.go file, even if the endpoint's endpoint-config.yaml file does not list clients/logger as a dependency.

Note that these paths support Glob patterns.

Code Generation

Zanzibar provides HTTP and TChannel runtime components for both clients and servers. Once all the configs are properly defined, Zanzibar is able to parse the config files and generate code and wire it up with the runime components to produce a full application. All generated code is placed in the build directory.

Go Structs and (de)serializers

Zanzibar expects non-custom clients and endpoints to define their interfaces using Thrift (Zanzibar Thrift file semantics). For example, the bar endpoint defines its interfaces using the bar.thrift as specified in hello.yaml. The data types in such thrift files must have their equivalents in Go.

  • For tchannel clients/endpoints, network communication is Thrift over TChannel. Zanzibar uses thriftrw to generate Go structs and thrift (de)serializers;
  • For http clients/endpoints, network communication is JSON over HTTP. Zanzibar uses thriftrw to generate Go structs and then uses easyjson to generate JSON (de)serializers.

The pre-steps.sh script takes care of this part of the code generation, and places the generated code under build/gen-code directory.

Zanzibar-generated Code

Everything except gen-code under build directory is generated by Zanzibar. Zanzibar parses config files for each module to gathers meta information and then executing various templates by applying them to the meta data. Here is what is generated for each builtin module class:

  • client: dependency type, client interface and constructor if non-custom, mock client constructor
  • middleware: dependency type, middleware type and constructor (unstable)
  • endpoint: dependency type, endpoint type and constructor, workflow interface, workflow if non-custom, mock workflow constructor if custom
  • service: dependency type and initializer, main.go, mock service constructor, service constructor

How to Use

Install

Assuming you are using a vendor package management tool like Glide, then the minimal glide.yaml file would look like:

- package: go.uber.org/thriftrw
  version: ^1.8.0
- package: github.com/mailru/easyjson
  version: master
- package: github.com/uber/zanzibar
  version: master

Code Gen

After installing the packages, create your module configs and application config in your application root directory. Then you are ready to run the following script to kick off code generation:

# put this script in application root directory

CONFIG_DIR="."
BUILD_DIR="$CONFIG_DIR/build"
THRIFTRW_SRCS=""

# find all thrift files specified in the config files
config_files=$(find "." -name "*-config.yaml" ! -path "*/build/*" ! -path "*/vendor/*" | sort)
for config_file in ${config_files}; do
	dir=$(dirname "$config_file")
	yaml_files=$(find "$dir" -name "*.yaml")
	for yaml_file in ${yaml_files}; do
		thrift_file=$(yq -r '.. | .idlFile? | select(strings | endswith(".thrift"))' "$yaml_file")
		[[ -z ${thrift_file} ]] && continue
		[[ ${THRIFTRW_SRCS} == *${thrift_file}* ]] && continue
        THRIFTRW_SRCS+=" $CONFIG_DIR/idl/$thrift_file"
	done
done

bash ./vendor/github.com/uber/zanzibar/codegen/runner/pre-steps.sh "$BUILD_DIR" "$CONFIG_DIR" "zanzibar" "$THRIFTRW_SRCS"
go run ./vendor/github.com/uber/zanzibar/codegen/runner/runner.go --config="$CONFIG_DIR/build.yaml"

Note the above script will be abstracted for easier usage in the future.

Testing

Zanzibar comes with builtin integration testing frameworks to help test business logic with ease. Setting genMock to true will trigger Zanzibar to generate mock client, workflow and service constructors. The mock clients, being the leaf nodes in the dependency graph, are wired with the rest modules to create a testing application, which you can test against by setting expectations of the mock clients. The generated test helpers make writing tests straightforward and concise.

Entry Points

Currently Zanzibar provides two entry points to write integration tests: service and endpoint.

Service

Service level integration testing treats your application as a black box. Zanzibar starts a local server for your application and you write tests by sending requests to the server and verify the response is expected.

func TestSaveContacts(t *testing.T) {
	ms := ms.MustCreateTestService(t)
	ms.Start()
	defer ms.Stop()

	ms.MockClients().Contacts.ExpectSaveContacts().Success()

	endpointReqeust := &endpointContacts.SaveContactsRequest{
		Contacts: []*endpointContacts.Contact{},
	}
	rawBody, _ := endpointReqeust.MarshalJSON()

	res, err := ms.MakeHTTPRequest(
		"POST", "/contacts/foo/contacts", nil, bytes.NewReader(rawBody),
	)
	if !assert.NoError(t, err, "got http error") {
		return
	}

	assert.Equal(t, "202 Accepted", res.Status)
}
Endpoint

Endpoint level integration testing allows focusing on testing the business logic without a full server setup. It is lightweighted and feels more like unit tests.

func TestSaveContactsCallWorkflow(t *testing.T) {
	mh, mc := mockcontactsworkflow.NewContactsSaveContactsWorkflowMock(t)

	mc.Contacts.ExpectSaveContacts().Success()

	endpointReqeust := &endpointContacts.SaveContactsRequest{
		UserUUID: "foo",
		Contacts: []*endpointContacts.Contact{},
	}

	res, resHeaders, err := mh.Handle(context.Background(), nil, endpointReqeust)

	if !assert.NoError(t, err, "got error") {
		return
	}
	assert.Nil(t, resHeaders)
	assert.Equal(t, &endpointContacts.SaveContactsResponse{}, res)
}

The above snippets can be found in save_contacts_test.go.

Fixture

Zanzibar uses gomock to generate client mocks. To avoid manually setting the same fixture expectations again and again, Zanzibar augments gomock-generated mocks with fixture support. For example, the client-config.yaml file of contacts client has a fixture field:

fixture:
  importPath: github.com/uber/zanzibar/examples/example-gateway/clients/contacts/fixture
  scenarios:
    SaveContacts:
    - success

This basically says the saveContacts method has a success scenario which is defined in the fixture package indicated by the importPath. The fixture package is provided by users and here is what it looks like:

package fixture

import (
	mc "github.com/uber/zanzibar/examples/example-gateway/build/clients/contacts/mock-client"
	gen "github.com/uber/zanzibar/examples/example-gateway/build/gen-code/clients-idl/clients/contacts/contacts"
)

var saveContactsFixtures = &mc.SaveContactsScenarios{
	Success: &mc.SaveContactsFixture{
		Arg0Any: true,
		Arg1Any: true,
		Arg2: &gen.SaveContactsRequest{
			UserUUID: "foo",
		},

		Ret0: &gen.SaveContactsResponse{},
	},
}

// Fixture ...
var Fixture = &mc.ClientFixture{
	SaveContacts: saveContactsFixtures,
}

With that, in your tests you will be able to write

mc.Contacts.ExpectSaveContacts().Success()

rather than

s.mockClient.EXPECT().SaveContacts(arg0, arg1, arg2).Return(ret0, ret1, ret2)

Check out fixture abstraction to see how it works.

Extend Zanzibar

Once the concepts of module, config and code generation are clear, extending Zanzibar becomes straightforward. There are two ways to extend Zanzibar.

New ModuleClass or Type

To extend Zanzibar with new module class or type is simply to extend each of its three pillars. For example, we want to add a new task module class to abstract common business workflow, here is what we need to do for each pillar:

  • module: understand what meta information is needed for each task module;
  • config: add a tasks directory under the application root directory, define proper schema for task module class;
  • code generation: add templates for task if necessary, create a code generator that implements the BuildGenerator interface and register it onto the module system for the task module class.

The same idea applies for adding new types of an existing module class.

PostGenHook

Zanzibar provides post-generation hooks which has access to the meta information of all modules. You can do whatever (mutating the input is probably not a good idea) suits your needs within a post-generation hook. Zanzibar invokes post-generation hooks as the very last step of code generation. In fact, mocks are all generated via post-generation hooks.

Development

Installation

mkdir -p $GOPATH/src/github.com/uber
git clone [email protected]:uber/zanzibar $GOPATH/src/github.com/uber/zanzibar
cd $GOPATH/src/github.com/uber/zanzibar
GO111MODULE=off make install

Running make generate

make generate

Running the tests

make test

Running the benchmarks

for i in `seq 5`; do make bench; done

Running the end-to-end benchmarks

First fetch wrk

git clone https://github.com/wg/wrk ~/wrk
cd ~/wrk
make
sudo ln -s $HOME/wrk/wrk /usr/local/bin/wrk

Then you can run the benchmark comparison script

# Assume you are on feature branch ABC
./benchmarks/compare_to.sh master

Running the server

First create log dir...

sudo mkdir -p /var/log/my-gateway
sudo chown $USER /var/log/my-gateway
chmod 755 /var/log/my-gateway

sudo mkdir -p /var/log/example-gateway
sudo chown $USER /var/log/example-gateway
chmod 755 /var/log/example-gateway
make run
# Logs are in /var/log/example-gateway/example-gateway.log

Adding new dependencies

We use glide @ 0.12.3 to add dependencies.

Download glide @ 0.12.3 and make sure it's available in your path

If we want to add a dependency:

  • Add a new section to the glide.yaml with your package and version
  • run glide up --quick
  • check in the glide.yaml and glide.lock

If you want to update a dependency:

  • Change the version field in the glide.yaml
  • run glide up --quick
  • check in the glide.yaml and glide.lock

More Repositories

1

react-vis

Data Visualization Components
JavaScript
8,666
star
2

baseweb

A React Component library implementing the Base design language
TypeScript
8,637
star
3

cadence

Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.
Go
7,872
star
4

RIBs

Uber's cross-platform mobile architecture framework.
Kotlin
7,681
star
5

kraken

P2P Docker registry capable of distributing TBs of data in seconds
Go
5,889
star
6

prototool

Your Swiss Army Knife for Protocol Buffers
Go
5,046
star
7

causalml

Uplift modeling and causal inference with machine learning algorithms
Python
4,788
star
8

h3

Hexagonal hierarchical geospatial indexing system
C
4,625
star
9

NullAway

A tool to help eliminate NullPointerExceptions (NPEs) in your Java code with low build-time overhead
Java
3,532
star
10

AutoDispose

Automatic binding+disposal of RxJava streams.
Java
3,360
star
11

aresdb

A GPU-powered real-time analytics storage and query engine.
Go
2,986
star
12

react-digraph

A library for creating directed graph editors
JavaScript
2,590
star
13

piranha

A tool for refactoring code related to feature flag APIs
Java
2,233
star
14

orbit

A Python package for Bayesian forecasting with object-oriented design and probabilistic models under the hood.
Python
1,808
star
15

ios-snapshot-test-case

Snapshot view unit tests for iOS
Objective-C
1,769
star
16

needle

Compile-time safe Swift dependency injection framework
Swift
1,757
star
17

petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
Python
1,755
star
18

manifold

A model-agnostic visual debugging tool for machine learning
JavaScript
1,636
star
19

okbuck

OkBuck is a gradle plugin that lets developers utilize the Buck build system on a gradle project.
Java
1,536
star
20

UberSignature

Provides an iOS view controller allowing a user to draw their signature with their finger in a realistic style.
Objective-C
1,285
star
21

nanoscope

An extremely accurate Android method tracing tool.
HTML
1,242
star
22

tchannel

network multiplexing and framing protocol for RPC
Thrift
1,150
star
23

queryparser

Parsing and analysis of Vertica, Hive, and Presto SQL.
Haskell
1,069
star
24

fiber

Distributed Computing for AI Made Simple
Python
1,039
star
25

neuropod

A uniform interface to run deep learning models from multiple frameworks
C++
929
star
26

uReplicator

Improvement of Apache Kafka Mirrormaker
Java
902
star
27

pam-ussh

uber's ssh certificate pam module
Go
833
star
28

ringpop-go

Scalable, fault-tolerant application-layer sharding for Go applications
Go
816
star
29

h3-js

h3-js provides a JavaScript version of H3, a hexagon-based geospatial indexing system.
JavaScript
804
star
30

mockolo

Efficient Mock Generator for Swift
Swift
788
star
31

h3-py

Python bindings for H3, a hierarchical hexagonal geospatial indexing system
Python
762
star
32

xviz

A protocol for real-time transfer and visualization of autonomy data
JavaScript
760
star
33

streetscape.gl

Visualization framework for autonomy and robotics data encoded in XVIZ
JavaScript
702
star
34

react-view

React View is an interactive playground, documentation and code generator for your components.
TypeScript
690
star
35

nebula.gl

A suite of 3D-enabled data editing overlays, suitable for deck.gl
TypeScript
669
star
36

RxDogTag

Automatic tagging of RxJava 2+ originating subscribe points for onError() investigation.
Java
645
star
37

peloton

Unified Resource Scheduler to co-schedule mixed types of workloads such as batch, stateless and stateful jobs in a single cluster for better resource utilization.
Go
636
star
38

motif

A simple DI API for Android / Java
Kotlin
530
star
39

signals-ios

Typeful eventing
Objective-C
526
star
40

tchannel-go

Go implementation of a multiplexing and framing protocol for RPC calls
Go
480
star
41

grafana-dash-gen

grafana dash dash dash gen
JavaScript
478
star
42

marmaray

Generic Data Ingestion & Dispersal Library for Hadoop
Java
472
star
43

clay

Clay is a framework for building RESTful backend services using best practices. Itโ€™s a wrapper around Flask.
Python
441
star
44

astro

Astro is a tool for managing multiple Terraform executions as a single command
Go
431
star
45

NEAL

๐Ÿ”Ž๐Ÿž A language-agnostic linting platform
OCaml
424
star
46

react-vis-force

d3-force graphs as React Components.
JavaScript
402
star
47

arachne

An always-on framework that performs end-to-end functional network testing for reachability, latency, and packet loss
Go
389
star
48

cadence-web

Web UI for visualizing workflows on Cadence
JavaScript
377
star
49

Python-Sample-Application

Python
374
star
50

rides-ios-sdk

Uber Rides iOS SDK (beta)
Swift
367
star
51

stylist

A stylist creates cool styles. Stylist is a Gradle plugin that codegens a base set of Android XML themes.
Kotlin
355
star
52

storagetapper

StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service
Go
334
star
53

swift-concurrency

Concurrency utilities for Swift
Swift
323
star
54

RemoteShuffleService

Remote shuffle service for Apache Spark to store shuffle data on remote servers.
Java
319
star
55

cyborg

Display Android Vectordrawables on iOS.
Swift
300
star
56

rides-android-sdk

Uber Rides Android SDK (beta)
Java
288
star
57

h3-go

Go bindings for H3, a hierarchical hexagonal geospatial indexing system
Go
283
star
58

h3-java

Java bindings for H3, a hierarchical hexagonal geospatial indexing system
Java
262
star
59

hermetic_cc_toolchain

Bazel C/C++ toolchain for cross-compiling C/C++ programs
Starlark
255
star
60

h3-py-notebooks

Jupyter notebooks for h3-py, a hierarchical hexagonal geospatial indexing system
Jupyter Notebook
246
star
61

geojson2h3

Conversion utilities between H3 indexes and GeoJSON
JavaScript
217
star
62

artist

An artist creates views. Artist is a Gradle plugin that codegens a base set of Android Views.
Kotlin
210
star
63

tchannel-node

JavaScript
205
star
64

RxCentralBle

A reactive, interface-driven central role Bluetooth LE library for Android
Java
198
star
65

uberalls

Track code coverage metrics with Jenkins and Phabricator
Go
187
star
66

SwiftCodeSan

SwiftCodeSan is a tool that "sanitizes" code written in Swift.
Swift
173
star
67

rides-python-sdk

Uber Rides Python SDK (beta)
Python
170
star
68

doubles

Test doubles for Python.
Python
165
star
69

logtron

A logging MACHINE
JavaScript
158
star
70

cadence-java-client

Java framework for Cadence Workflow Service
Java
140
star
71

athenadriver

A fully-featured AWS Athena database driver (+ athenareader https://github.com/uber/athenadriver/tree/master/athenareader)
Go
139
star
72

cassette

Store and replay HTTP requests made in your Python app
Python
138
star
73

UBTokenBar

Flexible and extensible UICollectionView based TokenBar written in Swift
Swift
136
star
74

tchannel-java

A Java implementation of the TChannel protocol.
Java
132
star
75

bayesmark

Benchmark framework to easily compare Bayesian optimization methods on real machine learning tasks
Python
131
star
76

android-template

This template provides a starting point for open source Android projects at Uber.
Java
128
star
77

crumb

An annotation processor for breadcrumbing metadata across compilation boundaries.
Kotlin
122
star
78

py-find-injection

Look for SQL injection attacks in python source code
Python
119
star
79

rides-java-sdk

Uber Rides Java SDK (beta)
Java
104
star
80

startup-reason-reporter

Reports the reason why an iOS App started.
Objective-C
97
star
81

uber-poet

A mock swift project generator & build runner to help benchmark various module dependency graphs.
Python
95
star
82

cadence-java-samples

Java
93
star
83

charlatan

A Python library to efficiently manage and install database fixtures
Python
89
star
84

swift-abstract-class

Compile-time abstract class validation for Swift
Swift
83
star
85

simple-store

Simple yet performant asynchronous file storage for Android
Java
82
star
86

tchannel-python

Python implementation of the TChannel protocol.
Python
77
star
87

client-platform-engineering

A collection of cookbooks, scripts and binaries used to manage our macOS, Ubuntu and Windows endpoints
Ruby
72
star
88

eight-track

Record and playback HTTP requests
JavaScript
70
star
89

lint-checks

A set of opinionated and useful lint checks
Kotlin
68
star
90

multidimensional_urlencode

Python library to urlencode a multidimensional dict
Python
67
star
91

uncaught-exception

Handle uncaught exceptions.
JavaScript
66
star
92

swift-common

Common code used by various Uber open source projects
Swift
65
star
93

uberscriptquery

UberScriptQuery, a SQL-like DSL to make writing Spark jobs super easy
Java
59
star
94

sentry-logger

A Sentry transport for Winston
JavaScript
55
star
95

graph.gl

WebGL2-Powered Visualization Components for Graph Visualization
JavaScript
52
star
96

nanoscope-art

C++
48
star
97

assume-role-cli

CLI for AssumeRole is a tool for running programs with temporary credentials from AWS's AssumeRole API.
Go
47
star
98

airlock

A prober to probe HTTP based backends for health
JavaScript
47
star
99

mutornadomon

Easy-to-install monitor endpoint for Tornado applications
Python
46
star
100

kafka-logger

A kafka logger for winston
JavaScript
45
star