• This repository has been archived on 23/Sep/2020
  • Stars
    star
    125
  • Rank 286,335 (Top 6 %)
  • Language
    Java
  • License
    Apache License 2.0
  • Created over 7 years ago
  • Updated over 4 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

MirrorGate DevOps Dashboard

MirrorGate

MirrorGate is a WallBoard application meant to give teams fast feedback in all the different areas related to software development.

license GitHub issues GitHub stars GitHub forks Docker Stars Docker Pulls

Why that name?

MirrorGate is meant to display relevant information on how your software looks like from every perspective, from the planning to the user’s feedback. So it's the mirror where teams can see their work reflected, helping them to perform self-criticism and continuous improvement. It aims to improve software quality and time-to-market by making the team aware of how the software it is developing and its process looks like.

That said, to be honest, it all comes from this dialog from The Never Ending Story:

  • Engywook: Next is the Magic Mirror Gate. Atreyu has to face his true self.
  • Falcor: So what? That won't be too hard for him.
  • Engywook: Oh, that's what everyone thinks! But kind people find out that they are cruel. Brave men discover that they are really cowards! Confronted by their true selves, most men run away screaming!

Functionality

Right now MirrorGate offers different dashboard types and views:

Detail dashboard

Offers information on:

  • Sprint advance status and backlog refinement.
  • Program Increment (PI) advance status.
  • Incidences visualization by criticality.
  • Build status per repository.
  • Build statistics and failure tendency.
  • Marketplace feedback for mobile applications in iTunes, PlayStore or directly captured.
  • Active users from Google Analytics and Adobe Analytics.
  • AWS operation metrics.
  • Alerts.
  • Slack notifications.

ScreenCatpure

We expect to be adding much more information in the near future so stay tuned.

Aggregate dashboard

It also offers a view where you can display several product dashboards in a single view with a more summed up information.

ScreenCatpure

Smart components

Some components get all available space to display more detailed information. For instance:

  • Feedback

ScreenCatpure

  • Operations

ScreenCatpure

Backoffice

Additionally, MirrorGate offers a backoffice application where dashboards can be configured.

ScreenCatpure

Supported browsers

In MirrorGate we use some edge HTML and CSS features, thus only latest Chrome and Firefox versions are supported at the moment (i.e. IE and Safari are not currently supported).

Collecting feedback

MirrorGate ecosystem includes a markets-collector to be able to fetch feedback directly from the iOS and Android App Stores. Eventhough, it also offers an endpoint that might be invoked to directly send feedback on an specific product (directly from the front via form post or ajax request or from server side).

Running

MirrorGate server is provided as a docker image, so to run it simply execute the following commands in a terminal:

#Spinup mongo db
docker run --name mongo mongo > /dev/null &

#Run mirrorgate without security
docker run --env "SPRING_DATA_MONGODB_URI=mongodb://mongo:27017/dashboard" --env "SPRING_PROFILES_ACTIVE=embedded" --link="mongo" -p8080:8080 bbvaae/mirrorgate

Navigate to mirrorgate console in your local server, create a new dashboard and set the "Build Jobs" field to MirrorGateTest. Save it and navigate into the dashboard by cllicking the "eye" icon.

While keeping the dashboard opened run the following command:

curl -0 -v http://localhost:8080/api/builds \
-H "Expect:" \
-H 'Content-Type: application/json; charset=utf-8' \
-d @- << EOF
{
    "number" : "$(date +%s)",
    "buildStatus" : "Failure",
    "buildUrl": "#$(date +%s)",
    "timestamp": $(date +%s)000,
    "projectName" : "MirrorGateTest",
    "repoName" : "MyRepo",
    "branch" : "master"
}
EOF

You will now see the build status in the opened dashboard.

You now have an environment ready to receive information from the several collectors available.

To execute a simple local environment with Jira and Jenkins capabilities, please check the mirrorgate-sample-deployment project.

Architecture

In order to operate, MirrorGate requires several components:

  • Main MirrorGate application: typically executed by using the bbva-ae/mirrorgate docker container.
  • MongoDB database. Should be bound to the application by using the SPRING_DATA_MONGODB_URI environment variable (for example SPRING_DATA_MONGODB_URI=mongodb://localhost:27017/dashboarddb).
  • Collectors: collectors are components in charge of seeking and pushing information to the application. Currently, the following collectors exist:
    • Jenkins plugin: is a plugin that pushes information from a Jenkins CI server.
    • Jira collector: is a standalone application that polls Jira servers for changes every configurable amount of time.
    • Market collector: is a standalone process that polls smartphone applications marketplaces for user reviews.

Security

MirrorGate currently doesn't provide any authentication mechanism so if you want to secure it you will have to deploy it behind a reverse proxy.

It provides 3 different authorization depending on the value of the X-Forwarded-User header provided by the reverse proxy:

  • If the value of the header is ANONYMOUS, the access will be annonymous. The user will have read only access to the dashboards. This is meant to be use to place the dashboard in screens without the need of user authentication.
  • If the header is not set or equals COLLECTOR access to the /api endpoints will be granted, allowing the source system to push information. This is typically meant to be used for collectors and that's why it allows the header not to be provided in case you rely in perimetral security and the collectors access MirrorGate from behind the reverse proxy.
  • For any other case, the header is taken as the user id. It will have access to dashboards both write and read.

If you wan to completely disable the security in MirrorGate you can execute it with the env variable SPRING_PROFILES_ACTIVE=embedded.

For Developers

Build Dependencies

You need the following dependencies installed in order to build the project:

Project Structure

Contains folders for each of the modules:

  • mirrorgate-dashboard: contains the front-end sources.
  • mirrorgate-backoffice: contains the dashboard administration application.
  • mirrorgate-api: contains the API (back-end) sources.
  • docker: contains utilities to build a MirrorGate docker image.
  • tests: contains utilities to put all the pieces together and execute them as a whole while developing.

Check each of these folders for instructions on how to build, deploy and run each module.

How to execute

To execute MirrorGate locally:

  1. Ensure you have all the build dependencies installed.
  2. Clone this repository.
  3. Execute scripts/buildAndRun.sh.
  4. Wait some time until the message Tomcat started on port(s): 8080 appears.
  5. Open http://localhost:8080/mirrorgate/backoffice/index.html to access the WallBoards' backoffice.
  6. You should be able to navigate through the mock dashboards.
  7. Attempt to run some of the collectors (e.g. Jira) to be able to populate some information inside the database.

Contributing

Please read the contributing guide.

Credit

This project is inspired by CapitalOne's Hygieia.

More Repositories

1

kapow

Kapow! If you can script it, you can HTTP it.
Go
579
star
2

apicheck

The DevSecOps toolset for REST APIs
Python
263
star
3

kvm

Simple Docker KVM container
Shell
206
star
4

raft-badger

Raft backend implementation using BadgerDB
Go
101
star
5

patton

The clever vulnerability dependency finder
Gherkin
95
star
6

qed

The scalable, auditable and high-performance tamper-evident log project
Go
87
star
7

deeptracy

The Security Dependency Orchestrator Service
Python
86
star
8

timecop

Time series based anomaly detector
Python
83
star
9

waf-brain

Machine Learning WAF Based
Jupyter Notebook
77
star
10

chaos-monkey-engine

A Chaos Engineering swiss army knife
Python
45
star
11

patton-cli

The knife of the Admin & Security auditor
Python
42
star
12

susto

Systematic Universal Security Testing Orchestration
37
star
13

sqerzo

Tiny ORM for graph databases: Neo4j, RedisGraph, AWS Neptune or Gremlin
Python
34
star
14

open-cells

Open Cells routes & communicates web component applications.
JavaScript
31
star
15

Tarkin

A tool for anomaly detection over streaming data based on sentiment analysis
Jupyter Notebook
30
star
16

openstack-k8s

Openstack Deployment based on Kubernetes
Shell
29
star
17

data-refinery

Data transformation
Python
22
star
18

mist

Create complex tools execution Workflows for working together
Python
21
star
19

mercury-robust

mercury-robust is a framework to perform robust testing on ML models and datasets. It provides a collection of test that are easy to configure and helpful to guarantee robustness in your ML processes.
Jupyter Notebook
15
star
20

brainslug

Parasitic Computing Framework
Python
15
star
21

idsfree

Launch hacking tests in cloud providers securely, isolated and without raise security alerts in the provider
Python
14
star
22

gitsec

gitsec: GIT Secret Discovery
Python
14
star
23

UMAL

Modelling heterogeneous distributions with an Uncountable Mixture of Asymmetric Laplacians
Jupyter Notebook
13
star
24

spark-benchmarks

Benchmarking suite for Apache Spark
Scala
13
star
25

mercury-dataschema

Utility package that, given a Pandas DataFrame, it uses the DataSchema class which auto-infers feature types and automatically calculates different statistics depending on the types.
Python
13
star
26

ust2dsa

Improves Ubuntu security feed compatibility allowing it to be consumed by Debian vulnerability report tool, debsecan.
Haskell
12
star
27

masquerade

High-performance, real-time, multi-location data obfuscation tool
Go
12
star
28

openshift-utils

Openshift resources
Jupyter Notebook
12
star
29

mercury-explainability

mercury-explainability is a library with implementations of different state-of-the-art methods in the field of explainability. They are designed to work efficiently and to be easily integrated with the main Machine Learning frameworks.
Jupyter Notebook
12
star
30

mercury-reels

Reels is a library for analyzing sequences of events from transactional data to predict when related target events may occur in the future.
C++
11
star
31

mercury-monitoring

mercury-monitoring is a library to monitor data and model drift
Jupyter Notebook
10
star
32

mercury-settrie

A Python 3 library developed in C++ that enables efficient storage and querying of sets of sets. It can be used to perform fast document search. Uses the Settrie algorithm: https://osebje.famnit.upr.si/~savnik/papers/cdares13.pdf
C++
10
star
33

contract-testing

An experiment of contract testing using pact
Scala
9
star
34

waf-benchmark

Web Application Firewall Benchmark
Python
9
star
35

BeagleML

Machine Learning Model Trainer
9
star
36

docker-hdfs-alluxio-spark

Docker images and deployment configurations for a cluster of HDFS, Alluxio and Spark. Focusing on data locality. Support Openshift 3.4, and more comming.
Shell
8
star
37

k8s-configs

Kubernetes templates for openstack-k8s
Python
7
star
38

deeptracy-core

Deeptracy core for building deeptracy plugins
Python
7
star
39

economics-of-serverless

Economics of Serverless
Python
6
star
40

purescript-google-apps

Google Apps Script bindings for PureScript
PureScript
6
star
41

mirrorgate-jenkins-builds-collector

Jenkins plugin for collecting build results inside the MirrorGate dashboard
Java
6
star
42

hancock-wallet-hub

Docs:
TypeScript
5
star
43

hancock-dlt-adapter

Docs:
TypeScript
5
star
44

hancock-dlt-broker

Docs: https://bbva.github.io/hancock-dlt-broker/api.html
TypeScript
5
star
45

json-logex

Logger custom backend for console json output (elixir library)
Elixir
5
star
46

k8s-course

4
star
47

security-vault

Secrets bridge for containers
Go
4
star
48

deeptracy-api

REST API for Deeptracy project
Python
4
star
49

BeagleML-front

Machine Learning Model Trainer - Frontend
TypeScript
4
star
50

pacarana

A standalone ETL tool to generate advanced features for your Machine Learning projects
Scala
4
star
51

mirrorgate-jira-stories-collector

Jira collector for mirrorgate dashboard
Java
4
star
52

heat-templates

Heat Docker Templates
Shell
3
star
53

osc-docker

Docker container for Openstack clients
3
star
54

mirrorgate-aws-cloudwatch-metrics-collector

MirrorGate collector that gathers operation metrics from AWS CloudWatch
JavaScript
2
star
55

bump-go

Action for keeping the Go version of your project up-to-date
2
star
56

mirrorgate-bamboo-builds-collector

Bamboo builds collector for MirrorGate
JavaScript
2
star
57

deeptracy-dashboard

Deeptracy dashboard
TypeScript
2
star
58

gitbook

GitBook Documentation
2
star
59

Overlord

HTML
2
star
60

mirrorgate-sample-deployment

Sample docker-compose running the MirrorGate
2
star
61

mirrorgate-markets-collector

This Node application connects to a MirrorGate endpoint to get the id of the applications which comments and ratings must be recovered from Google Play and App Store.
JavaScript
2
star
62

mirrorgate-google-analytics-collector

Collector for analytics data from Google Analytics
JavaScript
2
star
63

cognito-mfa-singlestep-authorization-plugin

Java
1
star
64

ductype-manifesto

A functional shell inspired by algebraic data types and UNIX, designed to make command-line scripting easier and more powerful.
HTML
1
star
65

proctool

PoC: Artifact Tracing via Process IO Monitoring
Go
1
star
66

BeagleML-scheduler

Machine Learning Model Trainer - Scheduler
Python
1
star
67

crawlino

Crawling by rules definition
Python
1
star
68

buildbot-washer

Useful Patterns for Buildbot
Python
1
star
69

BeagleML-monitor

Machine Learning Model Trainer - Monitor
Python
1
star