• This repository has been archived on 17/Sep/2021
  • Stars
    star
    304
  • Rank 137,274 (Top 3 %)
  • Language
    Java
  • License
    Apache License 2.0
  • Created about 8 years ago
  • Updated over 5 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Experimental CQRS and Event Sourcing service

** Capital One built this project to help our engineers as well as users in the community. We are no longer able to fully support the project. We have archived the project as of Jul 9 2019 where it will be available in a read-only state. Feel free to fork the project and maintain your own version. **

CQRS Manager for Distributed Reactive Services

CQRS Manager for Distributed Reactive Services (herein abbreviated CMDR), is a reference implementation for the key component in a specific architecture for building distributed information services following a Log-centric REST+CQRS+ES design.

The role of the CMDR component is to handle incoming Commands by:

  1. Ensuring their conformity to Schemas (coming soon)
  2. Writing them down to the Log (Kafka in this implementation)

Additionally, CMDR indexes all Commands and Events from their respective Kafka topics in order to:

  1. Respond to read (GET) requests for information about Commands and Events
  2. Provide a Server Sent Events (SSE) interface to both Commands (/commands/updates) and Events (/events/updates)

Status

IMPORTANT! This is alpha-quality software, meant mostly to demonstrate the Log-centric REST+CQRS+ES architecture described in the linked talks, and to facilitate learning and discussion.

As this implementation reaches maturity, and becomes suited for production-use, this note will be removed from the README.

Documentation

CMDR is the component in your system that handles all incoming actions/writes, but it's totally ignorant of your business logic and domain. You'll need to build microservices that implement your business logic, and read-only REST endpoints to expose the resulting data. You'll end up with a system architecture that looks like this, and has many benefits.

This README will help you get CMDR running, but doesn't tell you how to integrate with it or why you might want to. See the rationale for why you'd use CMDR, and the contract documentation for details about how to integrate with CMDR.

Running in development

CMDR is a Leiningen project, and behaves as you would expect in terms of running tests, launching REPLs, packaging uberjars, etc.

CMDR runs as two microservices (com.capitalone.commander.indexer and com.capitalone.commander.rest) that coordinate via Apache Kafka and a JDBC-compliant database like PostgreSQL. In order to run the CMDR services, you'll first need to run a Kafka Cluster and a database.

Supporting Services

There is a handy Makefile for running and interacting with the supporting services and/or the example application. This Makefile uses docker and docker-compose under the hood to orchestrate the various services.

Running the supporting services (Kafka, ZooKeeper, PostgreSQL) via the Makefile is the easiest way to get started. However, if you want to run the supporting services manually, you can use a package manager to install the services.

Makefile

The easiest way to get CMDR's supporting services up-and-running quickly in development is via the Makefile.

$ make services
$ make service-bootstrap # bootstraps database, etc.

Then run the CMDR services locally from the REPL or via lein run as described below, and visit http://localhost:3000/ui/ to see the Swagger/OpenAPI user interface to the CMDR service.

Or you can run an entire example system (including both CMDR services and an example business logic service):

$ make example

Then you can visit the CMDR service at http://localhost:3000/ui/, and the sample application's REST API at http://localhost:8080/customers.

Running Manually

The following instructions provide an example for those running on Mac OS X using the Homebrew package manager to install the supporting services.

Install
Kafka and ZooKeeper

Kafka uses Apache ZooKeeper to maintain runtime state and configuration consistently across the cluster. You'll need to install both Kafka and ZooKeeper.

$ brew update
$ brew install kafka
PostgresQL

To install postgres:

$ brew install postgresql
Running
Kafka and ZooKeeper

In one shell:

$ zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties

In a second shell:

$ kafka-server-start /usr/local/etc/kafka/server.properties
PostgreSQL

In third shell:

$ postgres -D /usr/local/var/postgres

Then bootstrap the database in a fourth shell:

$ bin/run com.capitalone.commander.database 'jdbc:postgresql://localhost/postgres?user=postgres&password=postgres' commander commander commander

You'll want to use commander as the password for the commander user for the dev environment to work.

Clojure REPL

In the third shell:

# build Protobufs classes (only necessary on first run, or when making changes to .proto files)
$ lein do javac, compile

# launch repl
$ lein repl

Connect an nrepl client -- perhaps in your favorite editor, or else type directly into the REPL session in your shell:

(go)

Workflow

This application follows the reloaded workflow using the component pattern, providing a clean system architecture and reloadable REPL development environment.

Running in development

At the Clojure REPL, you first need to run database migrations.

(migrate-database)

You can then run the system:

(go)

And then reload the entire app, refreshing all the code:

(reset)

You can also run the tests right from the REPL (in addition to running via lein test):

(test)

Running tests

To run tests:

$ lein test

or

$ lein auto test

to run tests automatically every time a file is saved.

Swagger/OpenAPI

When running the API, you can view the Swagger/OpenAPI UI.

Swagger/OpenAPI functionality provided by pedestal-api.

Build

Uberjar

You can build a self-contained JAR that includes all dependencies via:

$ lein do clean, uberjar

You then run the application:

$ java -jar target/uberjar/cmdr-standalone.jar -m com.capitalone.commander.indexer
$ # OR
$ java -jar target/uberjar/cmdr-standalone.jar -m com.capitalone.commander.rest

Docker

The Docker image is based on the Alpine Linux version of the official Clojure repository on Docker Hub

$ docker build -t my-cmdr-build .

You can then run either of the CMDR services via the built image:

$ docker run my-cmdr-build com.capitalone.commander.indexer
$ # OR
$ docker run my-cmdr-build com.capitalone.commander.rest

Contributors

We welcome your interest in Capital One’s Open Source Projects (the “Project”). Any Contributor to the project must accept and sign a CLA indicating agreement to the license terms. Except for the license granted in this CLA to Capital One and to recipients of software distributed by Capital One, you reserve all right, title, and interest in and to your contributions; this CLA does not impact your rights to use your own contributions for any other purpose.

Link to CLA

This project adheres to the Open Source Code of Conduct. By participating, you are expected to honor this code.

License

Copyright 2016 Capital One Services, LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

More Repositories

1

DataProfiler

What's in your data? Extract schema, statistics and entities from datasets
Python
1,426
star
2

react-native-pathjs-charts

Android and iOS charts based on react-native-svg and paths-js
JavaScript
878
star
3

datacompy

Pandas, Polars, and Spark DataFrame comparison for humans and more!
Python
480
star
4

SWHttpTrafficRecorder

A simple library empowering you to record/capture HTTP(s) traffic of an iOS app for mocking/stubbing later.
Objective-C
205
star
5

fpe

A format-preserving encryption implementation in Go
Go
202
star
6

rubicon-ml

Capture all information throughout your model's development in a reproducible way and tie results directly to the model code!
Jupyter Notebook
127
star
7

giraffez

User-friendly Teradata client for Python
Python
108
star
8

locopy

locopy: Loading/Unloading to Redshift and Snowflake using Python.
Python
104
star
9

checks-out

Checks-Out pull request approval system
Go
76
star
10

dataCompareR

dataCompareR is an R package that allows users to compare two datasets and view a report on the similarities and differences.
R
75
star
11

stack-deployment-tool

Go
66
star
12

bash_shell_mock

A shell script mocking utility/framework for the BASH shell
Shell
66
star
13

architecture-viewer

Visualize your PlantUML sequence diagrams as interactive architecture diagrams!
JavaScript
60
star
14

go-future-context

A simple Future (Promise) library for Go.
Go
54
star
15

AI_Dictionary_English_Spanish

TeX
49
star
16

acronym-decoder

Acronym Decoder
TypeScript
43
star
17

synthetic-data

Generating complex, nonlinear datasets appropriate for use with deep learning/black box models which 'need' nonlinearity

Python
43
star
18

Particle-Cloud-Framework

Python
36
star
19

slackbot-destroyer

📣 ❌ Slack integration that can destroy all incoming messages from Slackbot.
Python
34
star
20

global-attribution-mapping

GAM (Global Attribution Mapping) explains the landscape of neural network predictions across subpopulations
Python
33
star
21

federated-model-aggregation

The Federated Model Aggregation (FMA) Service is a collection of installable python components that make up the generic workflow/infrastructure needed for federated learning.
Python
30
star
22

oas-nodegen

A library for generating completely customizable code from the Open API Specification (FKA Swagger) RESTful API documentation using the scripting power of Node.js.
JavaScript
28
star
23

easy-screenshots

Android Instrumentation Test Screenshots made Easy.
Java
21
star
24

edgetest

edgetest is a tox-inspired python library that will loop through your project's dependencies, and check if your project is compatible with the latest version of each dependency
Python
19
star
25

ablation

Evaluating XAI methods through ablation studies.
Python
15
star
26

serverless-shell

⚡️🐚 Serverless Shell with environment variables plugin
JavaScript
14
star
27

OAuthClient

Awesome OAuth Client for Java.
Java
13
star
28

otvPlots

ovtPlots: An R Package for Variable Level Monitoring
R
13
star
29

json-syntax

Generates functions to convert Python classes to and from JSON friendly objects.
Python
12
star
30

screen-object

screen-object (ruby gem for mobile app automation)
Ruby
12
star
31

jwt-security

JavaScript
11
star
32

BankAccountStarter-API-reference-app

CSS
10
star
33

CreditOffers-API-reference-app

JavaScript
10
star
34

Rewards-API-reference-app

JavaScript
10
star
35

local-crontab

🗺️⏰ Convert local crontabs to UTC crontabs
JavaScript
8
star
36

modtracker

JSON unmarshaling in Go that includes detection of modified fields
Go
7
star
37

grpc-cucumber-js

JavaScript
7
star
38

edgetest-hub

hub plugin for edgetest
Python
2
star
39

oas-nodegen-example

Example project that shows how to customize generated code to fit a specific design pattern using oas-nodegen
Java
2
star
40

edgetest-conda

Conda plugin for edgetest
Python
1
star
41

edgetest-pip-tools

pip-tools plugin for edgetest
Python
1
star