• This repository has been archived on 04/Mar/2022
  • Stars
    star
    5,051
  • Rank 7,669 (Top 0.2 %)
  • Language
    Go
  • License
    MIT License
  • Created about 6 years ago
  • Updated almost 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Your Swiss Army Knife for Protocol Buffers

Prototool

MIT License GitHub Release Build Status Coverage Status Docker Image Homebrew Package AUR Package

Update: We recommend checking out Buf, which is under active development. There are a ton of docs for getting started, including for migration from Prototool.

Protobuf is one of the best interface description languages out there - it's widely adopted, and after over 15 years of use, it's practically bulletproof. However, working with Protobuf and maintaining consistency across your Protobuf files can be a pain - protoc, while being a tool that has stood the test of time, is non-trivial to use, and the Protobuf community has not developed common standards with regards to stub generation. Prototool aims to solve this by making working with Protobuf much simpler.

Prototool lets you:

  • Handle installation of protoc and the import of all of the Well-Known Types behind the scenes in a platform-independent manner.
  • Standardize building of your Protobuf files with a common configuration.
  • Lint your Protobuf files with common linting rules according to Google' Style Guide, Uber's V1 Style Guide, Uber's V2 Style Guide, or your own set of configured lint rules.
  • Format your Protobuf files in a consistent manner.
  • Create Protobuf files from a template that passes lint, taking care of package naming for you.
  • Generate stubs using any plugin based on a simple configuration file, including handling imports of all the Well-Known Types.
  • Call gRPC endpoints with ease, taking care of the JSON to binary conversion for you.
  • Check for breaking changes on a per-package basis, verifying that your API never breaks.
  • Output errors and lint failures in a common file:line:column:message format, making integration with editors possible, Vim integration is provided out of the box.

Prototool accomplishes this by downloading and calling protoc on the fly for you, handing error messages from protoc and your plugins, and using the generated FileDescriptorSets for internal functionality, as well as wrapping a few great external libraries already in the Protobuf ecosystem. Compiling, linting and formatting commands run in around 3/100ths of second for a single Protobuf file, or under a second for a larger number (500+) of Protobuf files.

Table Of Contents

Installation

Prototool can be installed on Mac OS X or Linux through a variety of methods.

See install.md for full instructions.

Quick Start

We'll start with a general overview of the commands. There are more commands, and we will get into] usage below, but this shows the basic functionality.

prototool help
prototool lint idl/uber # search for all .proto files recursively, obeying exclude_paths in prototool.yaml or prototool.json files
prototool lint # same as "prototool lint .", by default the current directory is used in directory mode
prototool create foo.proto # create the file foo.proto from a template that passes lint
prototool files idl/uber # list the files that will be used after applying exclude_paths from corresponding prototool.yaml or prototool.json files
prototool lint --list-linters # list all current lint rules being used
prototool lint --list-all-lint-groups # list all available lint groups, currently "google" and "uber"
prototool compile idl/uber # make sure all .proto files in idl/uber compile, but do not generate stubs
prototool generate idl/uber # generate stubs, see the generation directives in the config file example
prototool grpc idl/uber --address 0.0.0.0:8080 --method foo.ExcitedService/Exclamation --data '{"value":"hello"}' # call the foo.ExcitedService method Exclamation with the given data on 0.0.0.0:8080
prototool descriptor-set --include-imports idl/uber # generate a FileDescriptorSet for all files under idl/uber, outputting to stdout, a given file, or a temporary file
prototool break check idl/uber --git-branch master # check for breaking changes as compared to the Protobuf definitions in idl/uber on the master branch

Full Example

See the example directory.

The make command make example runs prototool while installing the necessary plugins.

Configuration

Prototool operates using a config file named either prototool.yaml or prototool.json. Only one of prototool.yaml or prototool.json can exist in a given directory. For non-trivial use, you should have a config file checked in to at least the root of your repository. It is important because the directory of an associated config file is passed to protoc as an include directory with -I, so this is the logical location your Protobuf file imports should start from.

Recommended base config file:

protoc:
  version: 3.11.0
lint:
  group: uber2

See protoc.md for how Prototool handles working with protoc.

The command prototool config init will generate a config file in the current directory with the currently recommended options set.

When specifying a directory or set of files for Prototool to operate on, Prototool will search for config files for each directory starting at the given path, and going up a directory until hitting root. If no config file is found, Prototool will use default values and operate as if there was a config file in the current directory, including the current directory with -I to protoc.

If multiple prototool.yaml or prototool.json files are found that match the input directory or files, an error will be returned.

See etc/config/example/prototool.yaml all available options.

File Discovery

In most Prototool commands, you will see help along the following lines:

$ prototool help lint
Lint proto files and compile with protoc to check for failures.

Usage:
  prototool lint [dirOrFile] [flags]

dirOrFile can take two forms:

  • You can specify exactly one directory. If this is done, Prototool goes up until it finds a prototool.yaml or prototool.json file (or uses the current directory if none is found), and then uses this config for all .proto files under the given directory recursively, except for files in the excludes lists in prototool.yaml or prototool.json files.
  • You can specify exactly one file. This has the effect as if you specified the directory of this file (using the logic above), but errors are only printed for that file. This is useful for e.g. Vim integration.
  • You can specify nothing. This has the effect as if you specified the current directory as the directory.

The idea with "directory builds" is that you often need more than just one file to do a protoc call, for example if you have types in other files in the same package that are not referenced by their fully-qualified name, and/or if you need to know what directories to specify with -I to protoc (by default, the directory of the prototool.yaml or prototool.json file is used).

Command Overview

Let's go over some of the basic commands.

prototool config init

Create a prototool.yaml file in the current directory with the currently recommended options set.

Pass the --document flag to generate a prototool.yaml file with all other options documented and commented out.

Pass the --uncomment flag to generate prototool.yaml file with all options documented but uncommented.

See etc/config/example/prototool.yaml for the config file that prototool config init --uncomment generates.

prototool compile

Compile your Protobuf files, but do not generate stubs. This has the effect of calling protoc with -o /dev/null.

Pass the --dry-run flag to see the protoc commands that Prototool runs behind the scenes.

prototool generate

Compile your Protobuf files and generate stubs according to the rules in your prototool.yaml or prototool.json file.

See etc/config/example/prototool.yaml for all available options. There are special options available for Golang plugins, and plugins that output a single file instead of a set of files. Specifically, you can output a single JAR for the built-in protoc java plugin, and you can output a file with the serialized FileDescriptorSet using the built-in protoc descriptor_set plugin, optionally also calling --include_imports and/or --include_source_info.

Pass the --dry-run flag to see the protoc commands that Prototool runs behind the scenes.

See example/proto/prototool.yaml for a full example.

prototool lint

Lint rules can be set using the configuration file. See the configuration at etc/config/example/prototool.yaml for all available options. There are three pre-configured groups of rules, the setting of which is integral to the prototool lint, prototool create, and prototool format commands:

  • uber2: This lint group follows the V2 Uber Style Guide, and makes some modifications to more closely follow the Google Cloud APIs file structure, as well as adding even more rules to enforce more consistent development patterns. This is the lint group we recommend using.
  • uber1: This lint group follows the V1 Uber Style Guide. For backwards compatibility reasons, this is the default lint group, however we recommend using the uber2 lint group.
  • google: This lint group follows the Google Style Guide. This is a small group of rules meant to enforce basic naming. The style guide is copied to etc/style/google/google.proto.

The flag --generate-ignores will help with migrating to a given lint group by generating the configuration to ignore existing lint failures on a per-file basis.

See lint.md for full instructions.

prototool format

Format a Protobuf file and print the formatted file to stdout. There are flags to perform different actions:

  • -d Write a diff instead.
  • -f Fix the file according to the Style Guide. This will have different behavior if the uber2 lint group is set.
  • -l Write a lint error in the form file:line:column:message if a file is unformatted.
  • -w Overwrite the existing file instead.
prototool create

Create Protobuf files from a template. With the provided Vim integration, this will automatically create new files that pass lint when a new file is opened.

See create.md for full instructions.

prototool files

Print the list of all files that will be used given the input dirOrFile. Useful for debugging.

prototool break check

Protobuf is a great way to represent your APIs and generate stubs in each language you develop with. As such, Protobuf APIs should be stable so as not to break consumers across repositories. Even in a monorepo context, making sure that your Protobuf APIs do not introduce breaking changes is important so that different deployed versions of your services do not have wire incompatibilities.

Prototool exposes a breaking change detector through the prototool break check command. This will check your current Protobuf definitions against a past version of your Protobuf definitions to see if there are any source or wire incompatible changes. Some notes on this command:

  • The breaking change detection operates on a per-package basis, not per-file - definitions can be moved between files within the same Protobuf package without being considered breaking.
  • The breaking change detector can either check against a given git branch or tag, or it can check against a previous state saved with the prototool break descriptor-set command.
  • The breaking change detector understands the concept of beta vs. stable packages, discussed in the V2 Style Guide. By default, the breaking change detector will not check beta packages for breaking changes, and will not allow stable packages to depend on beta packages, however both of these options are configurable in your prototool.yaml file.

See breaking.md for full instructions.

prototool descriptor-set

Produce a serialized FileDescriptorSet for all Protobuf definitions. By default, the serialized FileDescriptorSet is printed to stdout. There are a few options:

  • --include-imports, --include-source-info are analagous to protoc's --include_imports, --include_source_info flags.
  • --json outputs the FileDescriptorSet as JSON instead of binary.
  • -o writes the FileDescriptorSet to the given output file path.
  • --tmp writes the FileDescriptorset to a temporary file and prints the file path.

The outputted FileDescriptorSet is a merge of all produced FileDescriptorSets for each Protobuf package compiled.

This command is useful in a few situations.

One such situation is with external gRPC tools such as grpcurl or ghz. Both tools take a path to a serialized FileDescriptorSet for use to figure out the request/response structure of RPCs when the gRPC reflection service is not available. prototool descriptor-set can be used to generate these FileDescriptorSets on the fly.

grpcurl -protoset $(prototool descriptor-set --include-imports --tmp) ...
ghz -protoset $(prototool descriptor-set --include-imports --tmp) ...

You can also just save the file once and not re-compile each time.

prototool descriptor-set --include-imports -o descriptor_set.bin
grpcurl -protoset descriptor_set.bin ...
ghz -protoset descriptor_set.bin ...

Another situation is to use jq to make arbitrary queries on your Protobuf definitions.

For example, if your Protobuf definitions are in path/to/proto, the following will print all message names.

prototool descriptor-set path/to/proto --json | \
  jq '.file[] | select(.messageType != null) | .messageType[] | .name' | \
  sort | uniq
prototool grpc

Call a gRPC endpoint using a JSON input. What this does behind the scenes:

  • Compiles your Protobuf files with protoc, generating a FileDescriptorSet.
  • Uses the FileDescriptorSet to figure out the request and response type for the endpoint, and to convert the JSON input to binary.
  • Calls the gRPC endpoint.
  • Uses the FileDescriptorSet to convert the resulting binary back to JSON, and prints it out for you.

See grpc.md for full instructions.

Tips and Tricks

Prototool is meant to help enforce a consistent development style for Protobuf, and as such you should follow some basic rules:

  • Have all your imports start from the directory your prototool.yaml or prototool.json file is in. While there is a configuration option protoc.includes to denote extra include directories, this is not recommended.
  • Have all Protobuf files in the same directory use the same package.
  • Do not use long-form go_package values, ie use foopb, not github.com/bar/baz/foo;foopb. This helps prototool generate do the best job.

Vim Integration

This repository is a self-contained plugin for use with the ALE Lint Engine. The Vim integration will currently compile, provide lint errors, do generation of your stubs, and format your files on save. It will also optionally create new files from a template when opened.

See vim.md for full instructions.

Stability

Prototool is generally available, and conforms to SemVer, so Prototool will not have any breaking changes on a given major version, with some exceptions:

  • Commands under the x top-level command are experimental, and may change or be deleted between minor versions of Prototool. We expect such commands to be promoted to stable within a few minor releases, however development is still in-progress.
  • The output of the formatter may change between minor versions. This has not happened yet, but we may change the format in the future to reflect things such as max line lengths.
  • The breaking change detector's output format currently does not output filename, line, or column. This is an expected upgrade in the future, so the output will likely change. This is viewed as purely an upgrade, so until this is done, do not parse prototool break check output in scripts.
  • The breaking change detector may have additional checks added between minor versions, and therefore a change that might not have been breaking previously might become a breaking change. This may become stable in the near future, and at this time we'll denote that no more checks will be added.

Development

See development.md for concerns related to Prototool development.

See maintenance.md for maintenance-related tasks.

FAQ

See faq.md for answers to frequently asked questions.

Special Thanks

Prototool uses some external libraries that deserve special mention and thanks for their contribution to Prototool's functionality:

  • github.com/emicklei/proto - The Golang Protobuf parsing library that started it all, and is still used for the linting and formatting functionality. We can't thank Ernest Micklei enough for his help and putting up with all the filed issues.
  • github.com/jhump/protoreflect - Used for the JSON to binary and back conversion. Josh Humphries is an amazing developer, thank you so much.
  • github.com/fullstorydev/grpcurl - Still used for the gRPC functionality. Again a thank you to Josh Humphries and the team over at FullStory for their work.

More Repositories

1

react-vis

Data Visualization Components
JavaScript
8,632
star
2

baseweb

A React Component library implementing the Base design language
TypeScript
8,574
star
3

cadence

Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.
Go
7,654
star
4

RIBs

Uber's cross-platform mobile architecture framework.
Kotlin
7,624
star
5

kraken

P2P Docker registry capable of distributing TBs of data in seconds
Go
5,768
star
6

causalml

Uplift modeling and causal inference with machine learning algorithms
Python
4,592
star
7

h3

Hexagonal hierarchical geospatial indexing system
C
4,491
star
8

NullAway

A tool to help eliminate NullPointerExceptions (NPEs) in your Java code with low build-time overhead
Java
3,469
star
9

AutoDispose

Automatic binding+disposal of RxJava streams.
Java
3,343
star
10

aresdb

A GPU-powered real-time analytics storage and query engine.
Go
2,976
star
11

react-digraph

A library for creating directed graph editors
JavaScript
2,573
star
12

piranha

A tool for refactoring code related to feature flag APIs
Java
2,206
star
13

orbit

A Python package for Bayesian forecasting with object-oriented design and probabilistic models under the hood.
Python
1,782
star
14

ios-snapshot-test-case

Snapshot view unit tests for iOS
Objective-C
1,755
star
15

petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
Python
1,724
star
16

needle

Compile-time safe Swift dependency injection framework
Swift
1,708
star
17

manifold

A model-agnostic visual debugging tool for machine learning
JavaScript
1,639
star
18

okbuck

OkBuck is a gradle plugin that lets developers utilize the Buck build system on a gradle project.
Java
1,535
star
19

UberSignature

Provides an iOS view controller allowing a user to draw their signature with their finger in a realistic style.
Objective-C
1,281
star
20

nanoscope

An extremely accurate Android method tracing tool.
HTML
1,237
star
21

tchannel

network multiplexing and framing protocol for RPC
Thrift
1,148
star
22

queryparser

Parsing and analysis of Vertica, Hive, and Presto SQL.
Haskell
1,067
star
23

fiber

Distributed Computing for AI Made Simple
Python
1,041
star
24

neuropod

A uniform interface to run deep learning models from multiple frameworks
C++
927
star
25

uReplicator

Improvement of Apache Kafka Mirrormaker
Java
888
star
26

pam-ussh

uber's ssh certificate pam module
Go
826
star
27

ringpop-go

Scalable, fault-tolerant application-layer sharding for Go applications
Go
810
star
28

h3-js

h3-js provides a JavaScript version of H3, a hexagon-based geospatial indexing system.
JavaScript
779
star
29

xviz

A protocol for real-time transfer and visualization of autonomy data
JavaScript
760
star
30

mockolo

Efficient Mock Generator for Swift
Swift
759
star
31

h3-py

Python bindings for H3, a hierarchical hexagonal geospatial indexing system
Python
739
star
32

streetscape.gl

Visualization framework for autonomy and robotics data encoded in XVIZ
JavaScript
702
star
33

react-view

React View is an interactive playground, documentation and code generator for your components.
TypeScript
685
star
34

nebula.gl

A suite of 3D-enabled data editing overlays, suitable for deck.gl
TypeScript
648
star
35

RxDogTag

Automatic tagging of RxJava 2+ originating subscribe points for onError() investigation.
Java
642
star
36

peloton

Unified Resource Scheduler to co-schedule mixed types of workloads such as batch, stateless and stateful jobs in a single cluster for better resource utilization.
Go
632
star
37

motif

A simple DI API for Android / Java
Kotlin
532
star
38

signals-ios

Typeful eventing
Objective-C
526
star
39

tchannel-go

Go implementation of a multiplexing and framing protocol for RPC calls
Go
476
star
40

marmaray

Generic Data Ingestion & Dispersal Library for Hadoop
Java
474
star
41

grafana-dash-gen

grafana dash dash dash gen
JavaScript
467
star
42

zanzibar

A build system & configuration system to generate versioned API gateways.
Go
445
star
43

clay

Clay is a framework for building RESTful backend services using best practices. It’s a wrapper around Flask.
Python
441
star
44

astro

Astro is a tool for managing multiple Terraform executions as a single command
Go
431
star
45

NEAL

πŸ”ŽπŸž A language-agnostic linting platform
OCaml
423
star
46

react-vis-force

d3-force graphs as React Components.
JavaScript
402
star
47

arachne

An always-on framework that performs end-to-end functional network testing for reachability, latency, and packet loss
Go
387
star
48

cadence-web

Web UI for visualizing workflows on Cadence
JavaScript
376
star
49

Python-Sample-Application

Python
373
star
50

rides-ios-sdk

Uber Rides iOS SDK (beta)
Swift
364
star
51

stylist

A stylist creates cool styles. Stylist is a Gradle plugin that codegens a base set of Android XML themes.
Kotlin
355
star
52

storagetapper

StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service
Go
331
star
53

swift-concurrency

Concurrency utilities for Swift
Swift
321
star
54

cyborg

Display Android Vectordrawables on iOS.
Swift
301
star
55

RemoteShuffleService

Remote shuffle service for Apache Spark to store shuffle data on remote servers.
Java
288
star
56

rides-android-sdk

Uber Rides Android SDK (beta)
Java
286
star
57

h3-go

Go bindings for H3, a hierarchical hexagonal geospatial indexing system
Go
267
star
58

h3-java

Java bindings for H3, a hierarchical hexagonal geospatial indexing system
Java
259
star
59

h3-py-notebooks

Jupyter notebooks for h3-py, a hierarchical hexagonal geospatial indexing system
Jupyter Notebook
239
star
60

hermetic_cc_toolchain

Bazel C/C++ toolchain for cross-compiling C/C++ programs
Starlark
219
star
61

artist

An artist creates views. Artist is a Gradle plugin that codegens a base set of Android Views.
Kotlin
209
star
62

geojson2h3

Conversion utilities between H3 indexes and GeoJSON
JavaScript
206
star
63

tchannel-node

JavaScript
205
star
64

RxCentralBle

A reactive, interface-driven central role Bluetooth LE library for Android
Java
199
star
65

uberalls

Track code coverage metrics with Jenkins and Phabricator
Go
187
star
66

SwiftCodeSan

SwiftCodeSan is a tool that "sanitizes" code written in Swift.
Swift
172
star
67

rides-python-sdk

Uber Rides Python SDK (beta)
Python
169
star
68

doubles

Test doubles for Python.
Python
166
star
69

logtron

A logging MACHINE
JavaScript
158
star
70

cassette

Store and replay HTTP requests made in your Python app
Python
138
star
71

cadence-java-client

Java framework for Cadence Workflow Service
Java
137
star
72

UBTokenBar

Flexible and extensible UICollectionView based TokenBar written in Swift
Swift
136
star
73

athenadriver

A fully-featured AWS Athena database driver (+ athenareader https://github.com/uber/athenadriver/tree/master/athenareader)
Go
133
star
74

tchannel-java

A Java implementation of the TChannel protocol.
Java
132
star
75

android-template

This template provides a starting point for open source Android projects at Uber.
Java
127
star
76

bayesmark

Benchmark framework to easily compare Bayesian optimization methods on real machine learning tasks
Python
126
star
77

crumb

An annotation processor for breadcrumbing metadata across compilation boundaries.
Kotlin
124
star
78

py-find-injection

Look for SQL injection attacks in python source code
Python
119
star
79

rides-java-sdk

Uber Rides Java SDK (beta)
Java
101
star
80

startup-reason-reporter

Reports the reason why an iOS App started.
Objective-C
97
star
81

uber-poet

A mock swift project generator & build runner to help benchmark various module dependency graphs.
Python
94
star
82

charlatan

A Python library to efficiently manage and install database fixtures
Python
89
star
83

cadence-java-samples

Java
89
star
84

simple-store

Simple yet performant asynchronous file storage for Android
Java
82
star
85

swift-abstract-class

Compile-time abstract class validation for Swift
Swift
82
star
86

tchannel-python

Python implementation of the TChannel protocol.
Python
77
star
87

eight-track

Record and playback HTTP requests
JavaScript
70
star
88

client-platform-engineering

A collection of cookbooks, scripts and binaries used to manage our macOS, Ubuntu and Windows endpoints
Ruby
69
star
89

multidimensional_urlencode

Python library to urlencode a multidimensional dict
Python
67
star
90

lint-checks

A set of opinionated and useful lint checks
Kotlin
67
star
91

uncaught-exception

Handle uncaught exceptions.
JavaScript
66
star
92

swift-common

Common code used by various Uber open source projects
Swift
64
star
93

uberscriptquery

UberScriptQuery, a SQL-like DSL to make writing Spark jobs super easy
Java
57
star
94

sentry-logger

A Sentry transport for Winston
JavaScript
55
star
95

graph.gl

WebGL2-Powered Visualization Components for Graph Visualization
JavaScript
50
star
96

nanoscope-art

C++
47
star
97

assume-role-cli

CLI for AssumeRole is a tool for running programs with temporary credentials from AWS's AssumeRole API.
Go
47
star
98

airlock

A prober to probe HTTP based backends for health
JavaScript
47
star
99

mutornadomon

Easy-to-install monitor endpoint for Tornado applications
Python
46
star
100

kafka-logger

A kafka logger for winston
JavaScript
45
star