• Stars
    star
    122
  • Rank 287,103 (Top 6 %)
  • Language
    Kotlin
  • License
    Apache License 2.0
  • Created about 7 years ago
  • Updated over 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

An annotation processor for breadcrumbing metadata across compilation boundaries.

Crumb

Crumb is an annotation processor that exposes a simple and flexible API to breadcrumb metadata across compilation boundaries. Working with dependencies manually is usually fine, but there's often cases where developers will want to automatically gather and act on information from those dependencies (code generation, gathering metrics, etc). Tools like ServiceLoader can solve some cases like this, but lack flexibility and can be slow at runtime.

This is where Crumb comes in. Crumb's API is an annotation-based, consumer/producer system where extensions can opt in to consuming or producing metadata. Extensions run at compile-time to produce or consume this metadata, while Crumb's processor manages this metadata for them (serializing, storing, retrieving, orchestrating the data to appropriate consumers, etc). This allows developers to propagate arbitrary data across compilation boundaries.

Some example usages:

  • Implementing compile-time ServiceLoader-style automatic discovery of downstream implementations of an interface
  • Automatically gathering adapters for model serialization (such as TypeAdapters for json serialization with Gson)
  • Automatic registration or reporting of experiments in feature libraries
  • Automatic registration of buildable components in a DI system, such as Dagger modules

More in-depth examples can be found at the bottom of this README.

Download

Maven Central

compile 'com.uber.crumb:crumb-annotations:x.y.z'
compile 'com.uber.crumb:crumb-core:x.y.z'
compile 'com.uber.crumb:crumb-compiler:x.y.z'
compile 'com.uber.crumb:crumb-compiler-api:x.y.z'

Snapshots of the development version are available in Sonatype's snapshots repository.

API

Annotations

There are four annotations in the crumb-annotations artifact:

@CrumbProducer - This annotation can be used on custom annotations to signal to the processor that elements annotated with the custom annotation are used to produce metadata.

@CrumbConsumer - This annotation can be used on custom annotations to signal to the processor that elements annotated with the custom annotation are used to consume metadata.

@CrumbQualifier - This annotation can be used on custom annotations to indicate that elements annotated with the custom annotation are relevant for Crumb and used by extensions.

@CrumbConsumable - A convenience annotation that can be used to indicate that this type should be available to the Crumb processor and any of its extensions (since processors have to declare which annotations they support).

Extensions API

There are two extension interfaces that follow a Producer/Consumer symmetry. The API (and compiler implementation) is in Kotlin, but seamlessly interoperable with Java. The API is SPI-based, so implementations can be wired up with something like AutoService.

Both interfaces extend from a CrumbExtension base interface, that just has a method key(). This method has a default implementation in Kotlin that just returns the fully qualified class name of the extension. This is used to key the extension name when storing and retrieving metadata.

The API usually gives a CrumbContext instance when calling into extensions, which just contains useful information like references to the ProcessingEnvironment or RoundEnvironment.

CrumbProducerExtension - This interface is used to declare a producer extension. These extensions are called into when a type is trying to produce metadata to write to the classpath. The API is:

  • supportedProducerAnnotations() - Returns a set of supported annotations. Has a default implementation in Kotlin (empty), and is used to indicate to the compiler which annotations should be included in processing (since annotation processors have to declared which annotations they need).
  • isProducerApplicable(context: CrumbContext, type: TypeElement, annotations: Collection<AnnotationMirror> - Returns a boolean indicating whether or not this producer is applicable to a given type/annotations combination. The annotations are any @CrumbQualifier-annotated annotations found on type. Extensions may use whatever signaling they see fit though.
  • produce(context: CrumbContext, type: TypeElement, annotations: Collection<AnnotationMirror> - This is the call to produce metadata, and just returns a Map<String, String> (typealias'd in Kotlin to ProducerMetadata). Consumers can put whatever they want in this map (so be responsible!). The type and annotations parameters are the same as from isProducerApplicable().

CrumbConsumerExtension - This interface is used to declare a consumer extension. These extensions are called into when a type is trying to consume metadata to from the classpath. The API is:

  • supportedConsumerAnnotations() - Returns a set of supported annotations. Has a default implementation in Kotlin (empty), and is used to indicate to the compiler which annotations should be included in processing (since annotation processors have to declared which annotations they need).
  • isConsumerApplicable(context: CrumbContext, type: TypeElement, annotations: Collection<AnnotationMirror> - Returns a boolean indicating whether or not this consumer is applicable to a given type/annotations combination. The annotations are any @CrumbQualifier-annotated annotations found on type. Extensions may use whatever signaling they see fit though.
  • consume(context: CrumbContext, type: TypeElement, annotations: Collection<AnnotationMirror>, metadata: Set<ConsumerMetadata>) - This is the call to consume metadata, and is given a Set<Map<String, String>> (typealias'd in Kotlin to ConsumerMetadata). This is a set of all ProducerMetadata maps discovered on the classpath returned for this extension's declared key(). The type and annotations parameters are the same as from isConsumerApplicable().

CrumbManager

Crumb's core functionality can be leveraged independently from the compiler artifact via the crumb-core artifact. This can be useful for integration within existing tooling, and contains a CrumbManager and CrumbLog API. The crumb-compiler artifact is an advanced frontend over this utility.

CrumbManager has a simple load and store API, and CrumbLog is a logging mechanism to help with debugging issues.

Full docs can be found here: https://uber.github.io/crumb/0.x/

Packaging

Crumb works via generating synthetic types that hold @CrumbIndex annotations that hold information. These must be present in consumers compilation classpath to be used, but can be safely stripped (via tools such as R8, Proguard, etc) in production applications as they should appear to be unused.

Example: Plugin Loader

To demonstrate the functionality of Crumb we will have a hypothetical plugin system that automatically gathers and instantiates implementations of the Translations interface from downstream dependencies. Conceptually this is similar to a ServiceLoader, but at compile-time and with annotations.

To prevent a traditional approach of manually loading the implementations, Crumb makes it possible to automatically discover and utilize the Translations classes on the classpath.

Producing metadata

A given Translations implementation looks like this in a library:

public class EnglishTranslations implements Translations {
  // Implemented stuff!
}

The plugin implementation then needs to be registered into the plugin manager upstream. A Crumb extension can convey this information to consumers of the library by writing its location to Crumb and retrieving it on the other side. For this example, a custom @Plugin annotation is used to mark these translations implementations.

@CrumbProducer
public @interface Plugin {}

Note that it's annotated with @CrumbProducer so that the CrumbProcessor knows that this @Plugin annotation is used to produce metadata. Now this annotation can be applied to the implementation class:

@Plugin
public class EnglishTranslations implements Translations {
  // Implemented stuff!
}

Now that the implementation is denoted via the @Plugin annotation, the next step is implementing the ProducerExtension for this:

@AutoService(ProducerExtension.class)
public class PluginsCompiler implements ProducerExtension {

  @Override
  public String key() {
    return "PluginsCompiler";
  }

  @Override
  public boolean isProducerApplicable(CrumbContext context,
      TypeElement type,
      Collection<AnnotationMirror> annotations) {
    // Check for the @Plugin annotation here
  }

  @Override
  public Map<String, String> produce(CrumbContext context,
      TypeElement type,
      Collection<AnnotationMirror> annotations) {
    // <Error checking>
    return ImmutableMap.of(METADATA_KEY,
            type.getQualifiedName().toString());
  }
}

Crumb will take the returned metadata and make it available to any extension that also declared the key returned by key().

  • context is a holder class with access to the current ProcessingEnvironment and RoundEnvironment
  • type is the @CrumbProducer-annotated type (EnglishTranslations)
  • annotations are the @CrumbQualifier-annotated annotations found on that type. For simplicity, all holders are required to have a static obtain() method.

Consuming metadata

For the consumer side, our example will have a top-level TranslationsPluginManager class that just delegates to discovered downstream translations. With a ConsumerExtension, downstream services can be consumed and codegen'd directly with JavaPoet. For simplicity, this manager will follow an auto-value style pattern of having an abstract class with the generated implementation as a subclass.

The desired API looks like this:

public abstract class TranslationsPluginManager {

  public static Set<Translations> obtain() {
    return Plugins_TranslationsPluginManager.PLUGINS;
  }

}

Crumb can be wired in here. The symmetric counterpart to @CrumbProducer is @CrumbConsumer, so this example uses a similar @PluginPoint annotation here for consuming. This time it's annotated with @CrumbConsumer to indicate that it's for consumption.

@CrumbConsumer
public @interface PluginPoint {
  /* The target plugin interface. */
  Class<?> value();
}

This is then added to the manager class, specifying the Translations class as its target interface so that it only registers implementations of that interface.

@PluginPoint(Translations.class)
public abstract class TranslationsPluginManager {

  public static Set<Translations> obtain() {
    return Plugins_TranslationsPluginManager.PLUGINS;
  }

}

This is all the information needed for the ConsumerExtension. Implementation of it looks like this:

@AutoService(ConsumerExtension.class)
public class PluginsCompiler implements ConsumerExtension {

  @Override
  public String key() {
    return "PluginsCompiler";
  }

  @Override
  public boolean isConsumerApplicable(CrumbContext context,
      TypeElement type,
      Collection<AnnotationMirror> annotations) {
    // Check for the PluginPoint annotation here
  }

  @Override
  public void consume(CrumbContext context,
      TypeElement type,
      Collection<AnnotationMirror> annotations,
      Set<Map<String, String>> metadata) {
    // Each map is an instance of the Map we returned in the producer above

    PluginPoint targetPlugin = type.getAnnotation(PluginPoint.class).value(); // Not how it actually works, but here for readability

    // List of plugin TypeElements
    ImmutableSet<TypeElement> pluginClasses =
        metadata
            .stream()
            // Pull our metadata out by the key used to put it in
            .map(data -> data.get(METADATA_KEY))
            // Resolve the plugin implementation class
            .map(pluginClass ->
                context.getProcessingEnv().getElementUtils().getTypeElement(pluginClass))
            // Filter out anything that doesn't implement the targetPlugin interface
            .filter(pluginType ->
                context
                    .getProcessingEnv()
                    .getTypeUtils()
                    .isAssignable(pluginType.asType(), targetPlugin))
            .collect(toImmutableSet());

    // pluginClasses contains a set of all downstream plugin type implementations. This
  }
}

This closes the loop from the producers to the consumer. pluginClasses contains a set of all downstream plugin type implementations and could leverage JavaPoet to generate a backing implementation that looks like this:

public final class Plugins_TranslationsPluginManager extends TranslationsPluginManager {
  public static final Set<Translations> PLUGINS = new LinkedHashSet<>();

  static {
    PLUGINS.add(new EnglishTranslations());
  }
}

Note that both extension examples are called PluginsCompiler. Each interface is fully interoperable with the other, so it's possible to make one extension that implements both interfaces for code sharing.

@AutoService({ProducerExtension.class, ConsumerExtension.class})
public class PluginsCompiler implements ProducerExtension, ConsumerExtension {
  // ...
}

The complete implemented version of this example can be found under the :sample:plugins-compiler directory.

There's also an example experiments-compiler demonstrating how to trace enum-denoted experiments names to consumers.

pluginsamplediagram

License

Copyright (C) 2018 Uber Technologies

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

More Repositories

1

react-vis

Data Visualization Components
JavaScript
8,702
star
2

baseweb

A React Component library implementing the Base design language
TypeScript
8,661
star
3

cadence

Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.
Go
8,006
star
4

RIBs

Uber's cross-platform mobile architecture framework.
Kotlin
7,719
star
5

kraken

P2P Docker registry capable of distributing TBs of data in seconds
Go
6,002
star
6

prototool

Your Swiss Army Knife for Protocol Buffers
Go
5,044
star
7

causalml

Uplift modeling and causal inference with machine learning algorithms
Python
4,894
star
8

h3

Hexagonal hierarchical geospatial indexing system
C
4,731
star
9

NullAway

A tool to help eliminate NullPointerExceptions (NPEs) in your Java code with low build-time overhead
Java
3,596
star
10

AutoDispose

Automatic binding+disposal of RxJava streams.
Java
3,365
star
11

aresdb

A GPU-powered real-time analytics storage and query engine.
Go
2,996
star
12

react-digraph

A library for creating directed graph editors
JavaScript
2,605
star
13

piranha

A tool for refactoring code related to feature flag APIs
Java
2,250
star
14

orbit

A Python package for Bayesian forecasting with object-oriented design and probabilistic models under the hood.
Python
1,833
star
15

ios-snapshot-test-case

Snapshot view unit tests for iOS
Objective-C
1,788
star
16

needle

Compile-time safe Swift dependency injection framework
Swift
1,782
star
17

petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
Python
1,766
star
18

manifold

A model-agnostic visual debugging tool for machine learning
JavaScript
1,643
star
19

okbuck

OkBuck is a gradle plugin that lets developers utilize the Buck build system on a gradle project.
Java
1,536
star
20

UberSignature

Provides an iOS view controller allowing a user to draw their signature with their finger in a realistic style.
Objective-C
1,286
star
21

nanoscope

An extremely accurate Android method tracing tool.
HTML
1,243
star
22

tchannel

network multiplexing and framing protocol for RPC
Thrift
1,152
star
23

queryparser

Parsing and analysis of Vertica, Hive, and Presto SQL.
Haskell
1,072
star
24

fiber

Distributed Computing for AI Made Simple
Python
1,040
star
25

neuropod

A uniform interface to run deep learning models from multiple frameworks
C++
932
star
26

uReplicator

Improvement of Apache Kafka Mirrormaker
Java
907
star
27

pam-ussh

uber's ssh certificate pam module
Go
839
star
28

h3-js

h3-js provides a JavaScript version of H3, a hexagon-based geospatial indexing system.
JavaScript
826
star
29

ringpop-go

Scalable, fault-tolerant application-layer sharding for Go applications
Go
822
star
30

mockolo

Efficient Mock Generator for Swift
Swift
805
star
31

h3-py

Python bindings for H3, a hierarchical hexagonal geospatial indexing system
Python
790
star
32

xviz

A protocol for real-time transfer and visualization of autonomy data
JavaScript
760
star
33

streetscape.gl

Visualization framework for autonomy and robotics data encoded in XVIZ
JavaScript
702
star
34

react-view

React View is an interactive playground, documentation and code generator for your components.
TypeScript
690
star
35

nebula.gl

A suite of 3D-enabled data editing overlays, suitable for deck.gl
TypeScript
681
star
36

RxDogTag

Automatic tagging of RxJava 2+ originating subscribe points for onError() investigation.
Java
648
star
37

peloton

Unified Resource Scheduler to co-schedule mixed types of workloads such as batch, stateless and stateful jobs in a single cluster for better resource utilization.
Go
638
star
38

motif

A simple DI API for Android / Java
Kotlin
530
star
39

signals-ios

Typeful eventing
Objective-C
528
star
40

grafana-dash-gen

grafana dash dash dash gen
JavaScript
482
star
41

tchannel-go

Go implementation of a multiplexing and framing protocol for RPC calls
Go
481
star
42

marmaray

Generic Data Ingestion & Dispersal Library for Hadoop
Java
475
star
43

zanzibar

A build system & configuration system to generate versioned API gateways.
Go
455
star
44

clay

Clay is a framework for building RESTful backend services using best practices. It’s a wrapper around Flask.
Python
441
star
45

astro

Astro is a tool for managing multiple Terraform executions as a single command
Go
432
star
46

NEAL

🔎🐞 A language-agnostic linting platform
OCaml
426
star
47

react-vis-force

d3-force graphs as React Components.
JavaScript
402
star
48

arachne

An always-on framework that performs end-to-end functional network testing for reachability, latency, and packet loss
Go
392
star
49

cadence-web

Web UI for visualizing workflows on Cadence
JavaScript
380
star
50

Python-Sample-Application

Python
377
star
51

rides-ios-sdk

Uber Rides iOS SDK (beta)
Swift
369
star
52

stylist

A stylist creates cool styles. Stylist is a Gradle plugin that codegens a base set of Android XML themes.
Kotlin
357
star
53

storagetapper

StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service
Go
336
star
54

swift-concurrency

Concurrency utilities for Swift
Swift
326
star
55

RemoteShuffleService

Remote shuffle service for Apache Spark to store shuffle data on remote servers.
Java
319
star
56

cyborg

Display Android Vectordrawables on iOS.
Swift
300
star
57

h3-go

Go bindings for H3, a hierarchical hexagonal geospatial indexing system
Go
291
star
58

rides-android-sdk

Uber Rides Android SDK (beta)
Java
291
star
59

hermetic_cc_toolchain

Bazel C/C++ toolchain for cross-compiling C/C++ programs
Starlark
273
star
60

h3-java

Java bindings for H3, a hierarchical hexagonal geospatial indexing system
Java
268
star
61

h3-py-notebooks

Jupyter notebooks for h3-py, a hierarchical hexagonal geospatial indexing system
Jupyter Notebook
252
star
62

geojson2h3

Conversion utilities between H3 indexes and GeoJSON
JavaScript
221
star
63

artist

An artist creates views. Artist is a Gradle plugin that codegens a base set of Android Views.
Kotlin
211
star
64

tchannel-node

JavaScript
204
star
65

RxCentralBle

A reactive, interface-driven central role Bluetooth LE library for Android
Java
197
star
66

uberalls

Track code coverage metrics with Jenkins and Phabricator
Go
186
star
67

SwiftCodeSan

SwiftCodeSan is a tool that "sanitizes" code written in Swift.
Swift
176
star
68

rides-python-sdk

Uber Rides Python SDK (beta)
Python
171
star
69

doubles

Test doubles for Python.
Python
165
star
70

logtron

A logging MACHINE
JavaScript
158
star
71

athenadriver

A fully-featured AWS Athena database driver (+ athenareader https://github.com/uber/athenadriver/tree/master/athenareader)
Go
145
star
72

cadence-java-client

Java framework for Cadence Workflow Service
Java
140
star
73

cassette

Store and replay HTTP requests made in your Python app
Python
138
star
74

UBTokenBar

Flexible and extensible UICollectionView based TokenBar written in Swift
Swift
136
star
75

bayesmark

Benchmark framework to easily compare Bayesian optimization methods on real machine learning tasks
Python
133
star
76

tchannel-java

A Java implementation of the TChannel protocol.
Java
132
star
77

android-template

This template provides a starting point for open source Android projects at Uber.
Java
128
star
78

py-find-injection

Look for SQL injection attacks in python source code
Python
119
star
79

rides-java-sdk

Uber Rides Java SDK (beta)
Java
104
star
80

startup-reason-reporter

Reports the reason why an iOS App started.
Objective-C
97
star
81

uber-poet

A mock swift project generator & build runner to help benchmark various module dependency graphs.
Python
96
star
82

cadence-java-samples

Java
95
star
83

charlatan

A Python library to efficiently manage and install database fixtures
Python
89
star
84

simple-store

Simple yet performant asynchronous file storage for Android
Java
84
star
85

swift-abstract-class

Compile-time abstract class validation for Swift
Swift
84
star
86

tchannel-python

Python implementation of the TChannel protocol.
Python
77
star
87

client-platform-engineering

A collection of cookbooks, scripts and binaries used to manage our macOS, Ubuntu and Windows endpoints
Ruby
71
star
88

eight-track

Record and playback HTTP requests
JavaScript
70
star
89

lint-checks

A set of opinionated and useful lint checks
Kotlin
70
star
90

multidimensional_urlencode

Python library to urlencode a multidimensional dict
Python
67
star
91

uncaught-exception

Handle uncaught exceptions.
JavaScript
66
star
92

swift-common

Common code used by various Uber open source projects
Swift
66
star
93

uberscriptquery

UberScriptQuery, a SQL-like DSL to make writing Spark jobs super easy
Java
59
star
94

sentry-logger

A Sentry transport for Winston
JavaScript
56
star
95

graph.gl

WebGL2-Powered Visualization Components for Graph Visualization
JavaScript
53
star
96

nanoscope-art

C++
49
star
97

assume-role-cli

CLI for AssumeRole is a tool for running programs with temporary credentials from AWS's AssumeRole API.
Go
47
star
98

airlock

A prober to probe HTTP based backends for health
JavaScript
47
star
99

mutornadomon

Easy-to-install monitor endpoint for Tornado applications
Python
46
star
100

kafka-logger

A kafka logger for winston
JavaScript
45
star