• Stars
    star
    3,753
  • Rank 11,749 (Top 0.3 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created about 9 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A Microservice Toolkit from The New York Times

Gizmo Microservice Toolkit GoDoc Build Status Coverage Status

As of late April, 2021 Gizmo has been placed in maintenance mode.

This toolkit provides packages to put together server and pubsub daemons with the following features:

  • Standardized configuration and logging
  • Health check endpoints with configurable strategies
  • Configuration for managing pprof endpoints and log levels
  • Basic interfaces to define expectations and vocabulary
  • Structured logging containing basic request information
  • Useful metrics for endpoints
  • Graceful shutdowns

Install

Notice the capitalization!

go get github.com/NYTimes/gizmo/...

Import Path Change Notice

The New York Times recently changed the github organization from NYTimes to nytimes. This should not affect the installation as long as you use the proper casing NYTimes and not nytimes per installation instructions above.

However, the intention is to migrate the import paths to be consistent with how it's shown on GitHub. This will be a breaking change and we will introduce a major tag when we update the code. Therefore, the import path will go from github.com/NYTimes/gizmo/server to github.com/nytimes/gizmo/v2/server. This ensures that people will not have type-mismatches between import path changes.

Packages

The server package is the bulk of the toolkit and relies on server.Config to manage Server implementations.

It offers 1 server implementation:

SimpleServer, which is capable of handling basic HTTP and JSON requests via 5 of the available Service implementations: SimpleService, JSONService, ContextService, MixedService and a MixedContextService.

The server/kit package embodies Gizmo's goals to combine with go-kit.

  • In this package you'll find:
    • A more opinionated server with fewer choices.
    • go-kit used for serving HTTP/JSON & gRPC used for serving HTTP2/RPC
    • Monitoring, traces and metrics are automatically registered if running within App Engine, Cloud Run, Kubernetes Engine, Compute Engine or AWS EC2 Instances.
      • to change the name and version for Error reporting and Traces use SERVICE_NAME and SERVICE_VERSION environment variables.
    • Logs go to stdout locally or directly to Stackdriver when in GCP.
    • Using Go's 1.8 graceful HTTP shutdown.
    • Services using this package are expected to deploy to GCP.

The observe package provides observability helpers for metrics and tracing through OpenCensus

  • server/kit (and soon SimpleServer) utilizes this package to create a StackDriver exporter with sane defaults
  • GoogleProjectID, IsGAE, and IsCloudRun can help you make decisions about the underlying platform

The auth package provides primitives for verifying inbound authentication tokens:

  • The PublicKeySource interface is meant to provide *rsa.PublicKeys from JSON Web Key Sets.
  • The Verifier struct composes key source implementations with custom decoders and verifier functions to streamline server side token verification.

The auth/gcp package provides 2 Google Cloud Platform based auth.PublicKeySource and oauth2.TokenSource implementations:

  • The "Identity" key source and token source rely on GCP's identity JWT mechanism for asserting instance identities. This is the preferred method for asserting instance identity on GCP.
  • The "IAM" key source and token source rely on GCP's IAM services for signing and verifying JWTs. This method can be used outside of GCP, if needed and can provide a bridge for users transitioning from the 1st generation App Engine (where Identity tokens are not available) runtime to the 2nd.

The auth/gcp package also includes an Authenticator, which encapsulates a Google Identity verifier and oauth2 credentials to manage a basic web auth flow.

The config package contains a handful of useful functions to load to configuration structs from JSON files or environment variables.

There are also many structs for common configuration options and credentials of different Cloud Services and Databases.

The pubsub package contains two (publisher and subscriber) generic interfaces for publishing data to queues as well as subscribing and consuming data from those queues.

There are 4 implementations of pubsub interfaces:

  • For pubsub via Amazon's SNS/SQS, you can use the pubsub/aws package

  • For pubsub via Google's Pubsub, you can use the pubsub/gcp package

  • For pubsub via Kafka topics, you can use the pubsub/kafka package

  • For publishing via HTTP, you can use the pubsub/http package

The pubsub/pubsubtest package contains test implementations of the pubsub.Publisher, pubsub.MultiPublisher, and pubsub.Subscriber interfaces that will allow developers to easily mock out and test their pubsub implementations.

Examples

  • Several reference implementations utilizing server and pubsub are available in the examples subdirectory.
  • There are also examples within the GoDoc: here

If you experience any issues please create an issue and/or reach out on the #gizmo channel in the Gophers Slack Workspace with what you've found.

The Gizmo logo was based on the Go mascot designed by Renรฉe French and copyrighted under the Creative Commons Attribution 3.0 license.

More Repositories

1

covid-19-data

A repository of data on coronavirus cases and deaths in the U.S.
6,989
star
2

objective-c-style-guide

The Objective-C Style Guide used by The New York Times
5,848
star
3

Store

Android Library for Async Data Loading and Caching
Java
3,531
star
4

NYTPhotoViewer

A modern photo viewing experience for iOS.
Objective-C
2,847
star
5

pourover

A library for simple, fast filtering and sorting of large collections in the browser. There is a community-maintained fork that addresses a handful of post-NYT issues available via @hhsnopek's https://github.com/hhsnopek/pourover
JavaScript
2,393
star
6

kyt

Starting a new JS app? Build, test and run advanced apps with kyt ๐Ÿ”ฅ
JavaScript
1,922
star
7

react-tracking

๐ŸŽฏ Declarative tracking for React apps.
JavaScript
1,876
star
8

ice

track changes with javascript
JavaScript
1,708
star
9

backbone.stickit

Backbone data binding, model binding plugin. The real logic-less templates.
JavaScript
1,641
star
10

library

A collaborative documentation site, powered by Google Docs.
JavaScript
1,143
star
11

openapi2proto

A tool for generating Protobuf v3 schemas and gRPC service definitions from OpenAPI specifications
Go
940
star
12

gziphandler

Go middleware to gzip HTTP responses
Go
857
star
13

svg-crowbar

Extracts an SVG node and accompanying styles from an HTML document and allows you to download it all as an SVG file.
JavaScript
840
star
14

ingredient-phrase-tagger

Extract structured data from ingredient phrases using conditional random fields
Python
784
star
15

Emphasis

Dynamic Deep-Linking and Highlighting
JavaScript
576
star
16

tamper

Ruby
499
star
17

three-loader-3dtiles

This is a Three.js loader module for handling OGC 3D Tiles, created by Cesium. It currently supports the two main formats, Batched 3D Model (b3dm) - based on glTF Point cloud.
TypeScript
444
star
18

react-prosemirror

A library for safely integrating ProseMirror and React.
TypeScript
418
star
19

rd-blender-docker

A collection of Docker containers for running Blender headless or distributed โœจ
Python
415
star
20

Register

Android Library and App for testing Play Store billing
Kotlin
381
star
21

text-balancer

Eliminate typographic widows and other type crimes with this javascript module
JavaScript
373
star
22

document-viewer

The NYTimes Document Viewer
JavaScript
310
star
23

ios-360-videos

NYT360Video plays 360-degree video streamed from an AVPlayer on iOS.
Objective-C
273
star
24

three-story-controls

A three.js camera toolkit for creating interactive 3d stories
TypeScript
247
star
25

backbone.trackit

Manage unsaved changes in a Backbone Model.
JavaScript
202
star
26

aframe-loader-3dtiles-component

A-Frame component using 3D-Tiles
JavaScript
187
star
27

marvin

A go-kit HTTP server for the App Engine Standard Environment
Go
177
star
28

drone-gke

Drone plugin for deploying containers to Google Kubernetes Engine (GKE)
Go
165
star
29

Chronicler

A better way to write your release notes.
JavaScript
162
star
30

nginx-vod-module-docker

Docker image for nginx with Kaltura's VoD module used by The New York Times
Dockerfile
161
star
31

collectd-rabbitmq

A collected plugin, written in python, to collect statistics from RabbitMQ.
Python
143
star
32

public_api_specs

The API Specs (in OpenAPI/Swagger) for the APIs available from developer.nytimes.com
136
star
33

gunsales

Statistical analysis of monthly background checks of gun purchases
R
130
star
34

gcp-vault

A client for securely retrieving secrets from Vault in Google Cloud infrastructure
Go
119
star
35

rd-bundler-3d-plugins

Bundler plugins for optimizing glTF 3D models
JavaScript
119
star
36

Fech

Deprecated. Please see https://github.com/dwillis/Fech for a maintained fork.
Ruby
115
star
37

data-training

Files from the NYT data training program, available for public use.
114
star
38

drone-gae

Drone plugin for managing deployments and services on Google App Engine (GAE)
Go
97
star
39

mock-ec2-metadata

Go
95
star
40

encoding-wrapper

Collection of Go wrappers for Video encoding cloud providers (moved to @video-dev)
Go
85
star
41

redux-taxi

๐Ÿš• Component-driven asynchronous SSR in isomorphic Redux apps
JavaScript
70
star
42

video-captions-api

Agnostic API to generate captions for media assets across different transcription services.
Go
61
star
43

lifeline

A cron-based alternative to running daemons
Ruby
58
star
44

gcs-helper

Tool for proxying and mapping HTTP requests to Google Cloud Storage (GCS).
Go
54
star
45

logrotate

Go
54
star
46

httptest

A simple concurrent HTTP testing tool
Go
48
star
47

kyt-starter-universal

Deprecated, see: https://github.com/NYTimes/kyt/tree/master/packages/kyt-starter-universal
JavaScript
33
star
48

nytcampfin

A thin Python client for The New York Times Campaign Finance API
Python
27
star
49

safejson

safeJSON provides replacements for the 'load' and 'loads' methods in the standard Python 'json' module.
Python
27
star
50

thumbor-docker-image

Docker image for Thumbor smart imaging service
26
star
51

times_wire

A thin Ruby client for The New York Times Newswire API
Ruby
26
star
52

haiti-debt

Historical data on Haitiโ€™s debt payments to France collected by The New York Times.
21
star
53

hhs-child-migrant-data

Data from the U.S. Department of Human Health and Services on children who have migrated to the United States without an adult.
21
star
54

jsonlogic

Clojure
20
star
55

elemental-live-client

JS library to communicate with Elemental live API.
JavaScript
19
star
56

Open-Source-Science-Fair

The New York Times Open Source Science Fair
JavaScript
19
star
57

tweetftp

Ruby Implementation of the Tweet File Transfer Protocol (APRIL FOOLS JOKE)
Ruby
19
star
58

prosemirror-change-tracking-prototype

JavaScript
18
star
59

plumbook

Data from the Plum Book, published by the GPO every 4 years
17
star
60

libvmod-queryfilter

Simple querystring filter/sort module for Varnish Cache v3-v6
M4
16
star
61

sneeze

Python
16
star
62

querqy-clj

Search Query Rewriting for Elasticsearch and more! Built on Querqy.
Clojure
14
star
63

sqliface

handy interfaces and test implementations for Go's database/sql package
Go
14
star
64

grocery

The grocery package provides easy mechanisms for storing, loading, and updating Go structs in Redis.
Go
13
star
65

oak-byo-react-prosemirror-redux

JavaScript
13
star
66

vase.elasticsearch

Vase Bindings for Elasticsearch
Clojure
11
star
67

library-customization-example

An example repo that customizes Library behavior
SCSS
11
star
68

counter

count things, either as a one-off or aggregated over time
Ruby
11
star
69

kyt-starter

The default starter-kyt for kyt apps.
JavaScript
10
star
70

tulsa-1921-data

Data files associated with our story on the 1921 race massacre in Tulsa, Oklahoma.
10
star
71

open-blog-projects

A repository for code examples that are paired with our Open Blog posts
Swift
9
star
72

rd-mobile-pg-demos

HTML
9
star
73

pocket_change

Python
9
star
74

mentorship

7
star
75

sort_by_str

SQL-like sorts on your Enumerables
Ruby
7
star
76

drone-gdm

Drone.io plugin to facilitate the use of Google Deployment Manager in drone deploy phase.
Go
6
star
77

kyt-starter-static

Deprecated, see: https://github.com/NYTimes/kyt/tree/master/packages/kyt-starter-static
JavaScript
6
star
78

s3yum

Python
5
star
79

pocket

Python
5
star
80

drone-openapi

A Drone plugin for publishing Open API service specifications
Go
5
star
81

threeplay

Go client for the 3Play API.
Go
4
star
82

amara

Amara client for Go
Go
4
star
83

go-compare-expressions

Go
3
star
84

kaichu

Python
3
star
85

license

NYT Apache 2.0 license
3
star
86

prosemirror-tooltip

JavaScript
2
star
87

photon-dev_demo

A "Sustainable Systems, Powered By Python" Demo Repository (1 of 3)
Shell
2
star
88

std-cat

Content Aggregation Technology โ€” a standard for content aggregation on the Web
HTML
1
star