• Stars
    star
    290
  • Rank 139,617 (Top 3 %)
  • Language
    Go
  • License
    MIT License
  • Created about 4 years ago
  • Updated 6 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Example Go monolith with embedded microservices and The Clean Architecture

Example Go monolith with embedded microservices and The Clean Architecture

PkgGoDev Go Report Card CI/CD CircleCI Coverage Status Project Layout Release

This project shows an example of how to implement monolith with embedded microservices (a.k.a. modular monolith). This way you'll get many upsides of monorepo without it complexity and at same time most of upsides of microservice architecture without some of it complexity.

The embedded microservices use Uncle Bob's "Clean Architecture", check Example Go microservice for more details.

Table of Contents

Overview

Structure of Go packages

  • api/* - definitions of own and 3rd-party (in api/ext-*) APIs/protocols and related auto-generated code
  • cmd/* - main application(s)
  • internal/* - packages shared by embedded microservices, e.g.:
    • internal/config - configuration (default values, env) shared by embedded microservices' subcommands and tests
    • internal/dom - domain types shared by microservices (Entities)
  • ms/* - embedded microservices, with structure:
    • internal/config - configuration(s) (default values, env, flags) for microservice's subcommands and tests
    • internal/app - define interfaces ("ports") for The Clean Architecture (or "Ports and Adapters" architecture) and implements business-logic
    • internal/srv/* - adapters for served APIs/UI
    • internal/sub - adapter for incoming events
    • internal/dal - adapter for data storage
    • internal/migrations - DB migrations (in both SQL and Go)
    • internal/svc/* - adapters for accessing external services
  • pkg/* - helper packages, not related to architecture and business-logic (may be later moved to own modules and/or replaced by external dependencies), e.g.:
    • pkg/def/ - project-wide defaults
  • */old/* - contains legacy code which shouldn't be modified - this code is supposed to be extracted from old/ directories (and refactored to follow Clean Architecture) when it'll need any non-trivial modification which require testing

Features

  • Project structure (mostly) follows Standard Go Project Layout.
  • Strict but convenient golangci-lint configuration.
  • Embedded microservices:
    • Well isolated from each other.
    • Can be easily extracted from monolith into separate projects.
    • Share common configuration (both env vars and flags).
    • Each has own CLI subcommands, DB migrations, ports, metrics, …
  • Easily testable code (thanks to The Clean Architecture).
  • Avoids (and resists to) using global objects (to ensure embedded microservices won't conflict on these global objects).
  • CLI subcommands support using cobra.
  • Graceful shutdown support.
  • Configuration defaults can be overwritten by env vars and flags.
  • Example JSON-RPC 2.0 over HTTP API, with CORS support.
  • Example gRPC API:
  • Example OpenAPI 2.0 using grpc-gateway, with CORS suport:
    • Access to gRPC using HTTP/1 (except bi-directional streaming).
    • Generates swagger.json from gRPC .proto files.
    • Embedded Swagger UI.
  • Example DAL (data access layer):
    • MySQL 5.7 (strictest SQL mode).
    • PostgreSQL 11 (secure schema usage pattern).
  • Example tests, both unit and integration.
  • Production logging using structlog.
  • Production metrics using Prometheus.
  • Docker and docker-compose support.
  • Smart test coverage report, with optional support for coveralls.io.
  • Linters for Dockerfile and shell scripts.
  • CI/CD setup for GitHub Actions and CircleCI.

Development

Requirements

Setup

  1. After cloning the repo copy env.sh.dist to env.sh.
  2. Review env.sh and update for your system as needed.
  3. It's recommended to add shell alias alias dc="if test -f env.sh; then source env.sh; fi && docker-compose" and then run dc instead of docker-compose - this way you won't have to run source env.sh after changing it.

HTTPS

  1. This project requires https:// and will send HSTS and CSP HTTP headers, and also it uses gRPC with authentication which also require TLS certs, so you'll need to create certificate to run it on localhost - follow instructions in Create local CA to issue localhost HTTPS certificates.
  2. Or you can just use certificates in configs/insecure-dev-pki, which was created this way:
$ . ./env.sh   # Sets $EASYRSA_PKI=configs/insecure-dev-pki.
$ /path/to/easyrsa init-pki
$ echo Dev CA $(go list -m) | /path/to/easyrsa build-ca nopass
$ /path/to/easyrsa --days=3650 "--subject-alt-name=DNS:postgres" build-server-full postgres nopass
$ /path/to/easyrsa --days=3650 "--subject-alt-name=DNS:localhost" build-server-full ms-auth nopass
$ /path/to/easyrsa --days=3650 "--subject-alt-name=IP:127.0.0.1" build-server-full ms-auth-int nopass

Usage

To develop this project you'll need only standard tools: go generate, go test, go build, docker build. Provided scripts are for convenience only.

  • Always load env.sh in every terminal used to run any project-related commands (including go test): source env.sh.
    • When env.sh.dist change (e.g. by git pull) next run of source env.sh will fail and remind you to manually update env.sh to match current env.sh.dist.
  • go generate ./... - do not forget to run after making changes related to auto-generated code
  • go test ./... - test project (excluding integration tests), fast
  • ./scripts/test - thoroughly test project, slow
  • ./scripts/test-ci-circle - run tests locally like CircleCI will do
  • ./scripts/cover - analyse and show coverage
  • ./scripts/build - build docker image and binaries in bin/
    • Then use mentioned above dc (or docker-compose) to run and control the project.
      • Access project at host/port(s) defined in env.sh.

Cheatsheet

dc up -d --remove-orphans               # (re)start all project's services
dc logs -f -t                           # view logs of all services
dc logs -f SERVICENAME                  # view logs of some service
dc ps                                   # status of all services
dc restart SERVICENAME
dc exec SERVICENAME COMMAND             # run command in given container
dc stop && dc rm -f                     # stop the project
docker volume rm PROJECT_SERVICENAME    # remove some service's data

It's recommended to avoid docker-compose down - this command will also remove docker's network for the project, and next dc up -d will create a new network… repeat this many enough times and docker will exhaust available networks, then you'll have to restart docker service or reboot.

Run

Docker

$ docker run -i -t --rm ghcr.io/powerman/go-monolith-example:0.2.0 -v
mono version v0.2.0 7562a1e 2020-10-22_03:12:04 go1.15.3

Source

Use of the ./scripts/build script is optional (it's main feature is embedding git version into compiled binary), you can use usual go get|install|build to get the application instead.

$ ./scripts/build
$ ./bin/mono -h
Example monolith with embedded microservices

Usage:
  mono [flags]
  mono [command]

Available Commands:
  help        Help about any command
  ms          Run given embedded microservice's command
  serve       Starts embedded microservices

Flags:
  -h, --help                    help for mono
      --log.level OneOfString   log level [debug|info|warn|err] (default debug)
  -v, --version                 version for mono

Use "mono [command] --help" for more information about a command.

$ ./bin/mono serve -h
Starts embedded microservices

Usage:
  mono serve [flags]

Flags:
      --example.metrics.port Port             port to serve Prometheus metrics (default 17002)
      --example.mysql.dbname NotEmptyString   MySQL database name (default example)
      --example.mysql.pass String             MySQL password
      --example.mysql.user NotEmptyString     MySQL username (default root)
      --example.port Port                     port to serve (default 17001)
  -h, --help                                  help for serve
      --host NotEmptyString                   host to serve (default home)
      --host-int NotEmptyString               internal host to serve (default home)
      --mono.port Port                        port to serve monolith introspection (default 17000)
      --mysql.host NotEmptyString             host to connect to MySQL (default localhost)
      --mysql.port Port                       port to connect to MySQL (default 33306)
      --nats.urls NotEmptyString              URLs to connect to NATS (separated by comma) (default nats://localhost:34222)
      --stan.cluster_id NotEmptyString        STAN cluster ID (default local)
      --timeout.shutdown Duration             must be less than 10s used by 'docker stop' between SIGTERM and SIGKILL (default 9s)
      --timeout.startup Duration              must be less than swarm's deploy.update_config.monitor (default 3s)

Global Flags:
      --log.level OneOfString   log level [debug|info|warn|err] (default debug)

$ ./bin/mono -v
mono version v0.2.0 7562a1e 2020-10-22_03:19:37 go1.15.3

$ ./bin/mono serve
         mono: inf      main: `started` version v0.2.0 7562a1e 2020-10-22_03:19:37
         mono: inf     serve: `serve` home:17000 [monolith introspection]
      example: inf     natsx: `NATS connected` url=nats://localhost:34222
      example: inf     goose: OK    00001_down_not_supported.sql
      example: inf     goose: OK    00002_noop.go
      example: inf     goose: OK    00003_example.sql
      example: inf     goose: goose: no migrations to run. current version: 3
      example: inf     natsx: `STAN connected` clusterID=local clientID=example
      example: inf     serve: `serve` home:17001 [JSON-RPC 2.0]
      example: inf     serve: `serve` home:17002 [Prometheus metrics]
      example: inf  jsonrpc2: 192.168.2.1:46344     IncExample: `handled` 1
      example: inf  jsonrpc2: 192.168.2.1:46352     Example: `handled` 1
      example: inf  jsonrpc2: 192.168.2.1:46356     Example: `handled` 2
      example: ERR  jsonrpc2: 192.168.2.1:46364     Example: `failed to handle` err: unauthorized 0
^C
      example: inf     serve: `shutdown` [JSON-RPC 2.0]
      example: inf     serve: `shutdown` [Prometheus metrics]
         mono: inf     serve: `shutdown` [monolith introspection]
         mono: inf      main: `finished` version v0.2.0 7562a1e 2020-10-22_03:19:37

TODO

  • Add security-related headers for HTTPS endpoints (HSTS, CSP, etc.), also move default host from localhost to avoid poisoning it with HSTS.
  • Embed https://github.com/powerman/go-service-example as an example of embedding microservices from another repo.
  • Add example of internal/svc/* adapters calling some other services.
  • Add LPC (local procedure call API between embedded microservices), probably using https://github.com/fullstorydev/grpchan.
  • Add complete CRUD example as per Google API Design Guide (with PATCH/FieldMask), probably with generation of models conversion code using https://github.com/bold-commerce/protoc-gen-struct-transformer.
  • Add NATS/STAN publish/subscribe example in internal/sub (or maybe use JetStream instead of STAN?).
  • Switch from github.com/lib/pq to github.com/jackc/pgx.

More Repositories

1

dockerize

Utility to simplify running applications in docker containers
Go
177
star
2

go-service-example

Example Go service using go-swagger and Clean Architecture
Go
161
star
3

asciidoc-cheatsheet

Asciidoc cheatsheet for GitHub
Perl
144
star
4

rpc-codec

JSON-RPC 2.0 codec for Go net/rpc standard library
Go
96
star
5

vim-plugin-viewdoc

Vim plugin: flexible viewer for any documentation
Vim Script
89
star
6

vim-plugin-ruscmd

Vim plugin: support command mode in Russian keyboard layout
Vim Script
60
star
7

wcwidth-icons

Support fonts with double-width icons in xterm/rxvt-unicode/zsh/vim/…
C
41
star
8

vim-plugin-autosess

Vim plugin: auto save/load sessions
Vim Script
35
star
9

dotvim

~/.vim/
Vim Script
24
star
10

structlog

Structured logger for Go
Go
19
star
11

vcprompt

Version control information in your prompt
C
14
star
12

powerman-overlay

Powerman's Gentoo overlay
Shell
12
star
13

nerd-fonts-sh

Shell variables with names for Nerd fonts icons
Shell
9
star
14

tail

Go package tail implements behaviour of `tail -F` to follow rotated log files
Go
8
star
15

inferno-re2

OS Inferno driver: re2 library
Limbo
7
star
16

Narada

Framework for ease deploy and support microservice projects
Perl
7
star
17

jquery-tbodyscroll

jQuery plugin: add scrolling for table tbody element
JavaScript
6
star
18

asciidoc-habrahabr-backend

AsciiDoc backend for generating Habrahabr friendly HTML
6
star
19

check

Helpers to complement Go testing package
Go
6
star
20

sensitive

Package sensitive provides base types who's values should never be seen by the human eye, but still used for configuration.
Go
4
star
21

inferno-opt-setup

Setup projects in /opt for OS Inferno
Shell
4
star
22

testcert

TLS certs for use in Go tests
Go
4
star
23

must

Go
4
star
24

inferno-opt-skel

Example skeleton /opt project for OS Inferno
Brainfuck
4
star
25

alpine-runit-volume

Docker base image to run microservice with a data volume
Shell
3
star
26

gh-make-labels

Make labels for GitHub repo
Go
3
star
27

inferno-opt-mkfiles

mkfiles to use in OS Inferno projects compatible with /opt
Shell
3
star
28

inferno-contrib-tap

Limbo module: Test Anything Protocol
Brainfuck
3
star
29

vcprompt-fast

Improved and faster vcprompt: tool to get VCS info in shell PS1 prompt
Go
3
star
30

inferno-cjson

OS Inferno driver: fast JSON tokenizer
Brainfuck
3
star
31

perl-Log-Fast

Perl module: Log::Fast - Fast and flexible logger
Perl
3
star
32

perl-JSON-RPC2

Perl module: JSON::RPC2 - Transport-independent implementation of JSON-RPC 2.0
Perl
3
star
33

pqx

Helpers for use with Go Postgres driver github.com/lib/pq
Go
3
star
34

gotmpl

Command line tool for processing template file using Go text/template syntax
Go
3
star
35

flazsh

Fastest ZSH you've ever seen
Shell
2
star
36

userjs-github-asciidoc

UserJS for GitHub: fix Asciidoc rendering
JavaScript
2
star
37

vim-plugin-fixtermkeys

Fix terminal Ctrl Alt Shift modifiers for keys like Tab CR Space BS cursor and others
Vim Script
2
star
38

vaultgnupg

Scripts to link ansible-vault and GnuPG
Shell
2
star
39

getenv

Parse environment variables for Go
Go
2
star
40

deepequal

Go package with improved reflect.DeepEqual
Go
2
star
41

gotest

Helpers for Go tests
Go
2
star
42

structlog-usage-example

Usage example for Go package github.com/powerman/structlog
Go
2
star
43

narada4d

Manage data schema version
Go
2
star
44

appcfg

Get valid application configuration from flags/env/config files/consul/… for Go
Go
2
star
45

perl-Test-Mock-Time

Perl module: Test::Mock::Time - Deterministic time & timers for event loop tests
Perl
2
star
46

grub2-theme-powerman

Grub2 gfxmenu theme "Powerman"
2
star
47

inferno-contrib-retrymount

Automatically retry mount if mount point become unaccessible in OS Inferno
Brainfuck
2
star
48

inferno-contrib-hashtable

Limbo module: polymorphic hash table
Brainfuck
1
star
49

perl-Async-Defer

Perl module: Async::Defer - VM to write and run async code in usual sync-like way
Perl
1
star
50

perl-IO-Stream-HTTP-Persistent

Perl module: IO::Stream::HTTP::Persistent - HTTP persistent connections plugin
Perl
1
star
51

inferno-contrib-regmonitor

Limbo module: monitor registry(4) services
Brainfuck
1
star
52

perl-MojoX-JSONRPC2-HTTP

Perl module: MojoX::JSONRPC2::HTTP - Client for JSON RPC 2.0 over HTTP
Perl
1
star
53

narada-plugin-mojo

Mojolicious webapp for Narada projects
ApacheConf
1
star
54

inferno-contrib-growing

Limbo module: dynamically growing arrays
Brainfuck
1
star
55

narada-base

Base files used to initialize new Narada projects
1
star
56

inferno-contrib-logger

Limbo module: verbose logger
Brainfuck
1
star
57

asciidoc-9man-backend

AsciiDoc backend for generating man pages for OS Inferno and Plan9
1
star
58

sqlxx

General purpose extensions to golang's github.com/jmoiron/sqlx
Go
1
star
59

inferno-contrib-iobuf

Limbo module: buffered read/write
Brainfuck
1
star
60

inferno-contrib-watchdog

Watchdog for running services in OS Inferno
Brainfuck
1
star
61

chanq

Go package provides outgoing queue for channel to use in select case
Go
1
star
62

nginx-config

Nginx boilerplate config
DIGITAL Command Language
1
star
63

perl-IO-Stream-Proxy-SOCKSv5

Perl module: IO::Stream::Proxy::SOCKSv5 - SOCKSv5 proxy plugin for IO::Stream
Perl
1
star
64

constvar

Constant values in global Go variables
Go
1
star
65

urlvalues

Go package for unmarshaling url.Values to struct with strict validation
Go
1
star
66

conv

Convert types for Go
Go
1
star
67

perl-Mojolicious-Plugin-JSONRPC2

Perl module: Mojolicious::Plugin::JSONRPC2 - JSON RPC 2.0 over HTTP
Perl
1
star
68

perl-Inferno-RegMgr

Perl module: Inferno::RegMgr - Keep connection to OS Inferno's registry(4) and it tasks
Perl
1
star
69

mysqlx

Helpers for use with Go MySQL driver github.com/go-sql-driver/mysql
Go
1
star
70

inferno-contrib-register

Keep given service addr/attrs registered in OS Inferno registry
Brainfuck
1
star
71

perl-Crypt-MatrixSSL3

Perl module: Crypt::MatrixSSL3 - Perl extension for SSL and TLS using MatrixSSL.org
Perl
1
star
72

perl-Sub-Throttler

Perl module: Sub::Throttler - Rate limit sync and async function calls
Perl
1
star
73

perl-IO-Stream-Proxy-SOCKSv4

Perl module: IO::Stream::Proxy::SOCKSv4 - SOCKSv4 proxy plugin for IO::Stream
Perl
1
star
74

perl-IO-Stream-Proxy-HTTPS

Perl module: IO::Stream::Proxy::HTTPS - HTTPS proxy plugin for IO::Stream
Perl
1
star
75

go-service-stateless-example

Example how to build and test stateless Go microservice with Docker, Consul and Nginx
Go
1
star