• This repository has been archived on 16/Feb/2022
  • Stars
    star
    899
  • Rank 49,462 (Top 1.0 %)
  • Language
    Java
  • License
    Apache License 2.0
  • Created over 10 years ago
  • Updated about 7 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Fast Parallel Async HTTP client as a Service to monitor and manage 10,000 web servers. (Java+Akka)

REST Commander: Parallel Async HTTP Client as a Service Travis status

Formerly known as REST Superman. Fire thousands of HTTP requests and aggregate responses in a couple of clicks in seconds. Please check detail instructions, screenshots, documentations, sample code, REST APIs, and demos at www.restcommander.com and its demo video. What's new? Check related work review on efficient HTTP clients and concurrency and throttling model in Akka at eBay tech blog.

REST Commander is a fast parallel async HTTP/REST/SOAP client as a service to monitor and manage 10,000s of web servers. Sends requests to 1000 servers with response aggregation in 10 seconds. or 10,000 servers in 50 seconds.

Need standalone library alternative? Try Parallec.io (released 2015.11). With the feedbacks, lessons, and improvements from the past year of internal usage and open source of REST Commander, we now made its core as an easy to use standalone library. We added 15+ new features, rewritten 70%+ of the code, with 90%+ test coverage for confident usage and contribution. The key benefits of Parallec are flexible response aggregation and ease to send the results anywhere.

REST Commander serves as the agent master of cronus-agent (open sourced) for scalable software deployment, script execution, config push, and monitoring.

Version 2.0.0 : faster than faster, by restructuring via AKKA remoting and clustering, we are able to make REST Commander distributed and horizontally scalable. Distributed REST Commander can send request to 100K+ machines in eBay's cloud and gather result back in just 100s using 5 VM.

Version 2.0.0 | What and Why | Highlights | Performance | Run Instructions

REST Commander has been in top 10 trending out of 10 millions+ projects in Github in all languages on 01/21/2014 and 01/22/2014. It has been recommended and listed in top 20 trending out of 28K+ software in oschina, the largest open source community in China. It has also been featured and front-paged at InfoQ.

Structure Overview

Version 2.0.0 : distributed REST Commander

We restructure REST Commander via AKKA remoting and clustering. Detailed information and API document at here. Workflow and architecture design details at here. See git branch distributed_commander.

New Features

  • Speed : By restructuring REST Commander to make it distributed and horizontally scalable, it can send request to 100K+ machines in eBay's cloud and gather result back in just 100s using 5 VM.
  • Reliability : Automatically failover in case of failure on slave node, and adjust sending speed to avoid network congestion.
  • Scheduling : Provide task scheduling to handle multitasks.
  • Visibility : Show workload and available capacity of each slave node, track task progress.

Workflow & Architecture

The main workflow of distributed REST Commander is shown as below.

Work Flow

Distributed REST Commander is based on AKKA. Each functionality component in the pictrue above is implemented as an AKKA actor (except Job Manager). The whole system is based on message passing model.

What is REST Commander and Why I need it?

Commander is Postman at scale: a fast parallel async http client as a service with aggregated response and regular expression based string extraction. It is in Java (with Akka and Play Framework).

So what can Commander do? It speaks HTTP at scale, thus is powerful with many use cases. Here are some basic ones for automation on managing and monitoring tens of thousands web servers (See Sample Code). Commander itself is also "as a service": with its powerful REST API, you can define ad-hoc target servers, an HTTP request template, variable replacement, and a regular expression all in a single call.

Whenever comes to sending multiple HTTP requests in parallel, federated data aggregation or scalable task executions on HTTP, Think Commander First.

  • Monitor HTTP web servers: are you a company who have 50-5,000 web servers (e.g. tomcat, nginx, etc... ) running; and want to check every minute which servers are slow or misconfigured? Commander can get this done for you in an hour.
  • Config push to HTTP web servers: If your servers use REST/SOAP APIs to update its config and you want to enforce server-specific or uniform config on demand or with auto-remediation. Commander is your perfect choice.
  • HTTP web server management work flows combining the above 2: e.g., discover unhealthy webservers and then conduct operations (restart, config push) to them.

Highlights

  • Scalable and Fast: Utilizes Akka and Async HTTP Client to maximize concurrency; Sends and aggregates responses from 10,000+ Servers within 1 minute.
  • Powerful: Sends uniform or node-specific requests with near real-time response analysis or config pushes. Request level concurrency control.
  • Generic: Generic HTTP request. Generic response aggregation with user-defined regular expression matching. Generic variable replacement in request templates for node specific requests.
  • Ready to Use: Agility. Zero installation required. Changing requests and target servers with breeze. No database setup. Run locally in a single click.
  • User Friendly: Build in Java with Play Framework, Bootstrap and its Application Wizard, Commander enables sending requests in both easy-to-use web UI wizards and powerful REST APIs. Define ad hoc requests, target servers and the regex aggregation rule in a single REST call.
  • Agent-less Monitoring: Quickly check any HTTP results from an ad hoc list of servers with generic response aggregation by regular expression matching. FAST: No dependency or setup required.
  • Config Pushes: Push uniform config or node-specific configs to HTTP end points, as long as there are HTTP (REST/SOAP) APIs to perform.
  • N Requests to 1 Target: Concurrently fire a large number of different requests to a single target server. E.g. look up thousands of jobs status in a server. Concurrency control to accommodate server capacity.
  • IT Orchestration: Scalable multi-step HTTP work flows to thousands of HTTP endpoints.
  • Discover Outliers: Discover misconfigured servers from thousands of servers with http APIs in no time.
  • Remediation Automation: If your config change requests are idempotent, Commander can easily ensure correct config by scheduled config pushes.

Commander is powerful to send (1) the same request to different servers; (2) different requests to different servers; (3) different requests to the same server. Why we need them? Check out these live examples on (1) monitor websites; (2) poll job status (3) call the same weather WSDL web service with different zip codes.

Performance (SLA)

  • Measured from Commander running on a single off-the-shelf server.
  • 1000 servers requests and all responses aggregated in 7 seconds
  • 10,000 servers requests and all responses aggregated in 48 seconds
  • 20,000 servers requests and all responses aggregated in 70 seconds
  • 20,000 is far less than the maximum scale we tested and it is stable for months. We have not been able to find the scalability limit. :-)

Run Instructions

Directly Under Windows/Linux With Zero Installation:

  • Assuming have Java (JDK or most time just JRE) pre-installed.
WINDOWS
  • double click: start_application_win.bat . Just close the window to shutdown the server.
  • Then open browser: localhost:9000
  • After shutdown the application: double click: clean_application_pid_after_run_win.bat
LINUX or MAC
  • Note that for Linux/Mac user: need to chmod +x for play-1.2.4/play
  • sh /home/user/RestCommander/start_application_linux_or_mac.sh start

Run/Debug With Eclipse:

  • Clone project in Git from: https://github.com/eBay/restcommander
  • Extract to a folder, e.g., S:\GitSources\AgentMaster\AgentMaster. In command line run: S:\GitSources\AgentMaster\AgentMaster>play-1.2.4\play eclipsify AgentMaster
    • Note that for Linux/Mac user: need to chmod +x for play-1.2.4/play
  • Import existing project in Eclipse: import the AgentMaster folder.
  • Compile errors? Try rebuild and clean: (menu: Project->Clean->Clean all projects
  • Run application: under "eclipse" folder: AgentMaster.launch : run as AgentMaster
  • Then open browser: localhost:9000

Settings

Key files are under conf folder

  • agentcommand.conf : defines commands
  • nodegroup.conf : defines the node list: ad hoc; from ODB; from Statehub.
  • aggregation.conf : defines aggregation: using the default one to parse number out.
  • application.conf : play settings
  • actorconfig.conf : Akka settings
  • routes : MVC settings as dispatcher

Troubleshooting

  • Under windows: shutdown in the middle of sending requests or many requests: No buffer space available (maximum connections reached ?)

About REST Commander

Motivation:

REST and SOAP API (HTTP GET/POST/PUT/DELETE) has become the dominant approach in current platform and services. However, efficient REST / SOAP calls to multiple servers in parallel with server-specific requests and aggregated response analysis are still challenging.

Problem Statement:

Design and implement a user friendly and generic HTTP client able to conduct efficient HTTP calls to a large amount of servers in parallel with uniform or server-specific requests and aggregated response analysis.

Impact

  • Improve HTTP/SOAP/REST call efficiency by 100%-5000% with 3-15000 target nodes in parallel, compared to single server POSTMAN or sequential executed none-generic shell scripts.
  • Enable any uniform or server-specific REST (GET/POST/PUT/DELETE) calls to servers for periodical monitoring and configuration pushes in a simple UI with fast and reliable with responses aggregation. Automate tens of thousands of server's management and software pool management.
  • Innovation: After thoroughly reviewing related work of Postman, JMeter, Gatling, Apache Bench, Typhoeus and many other publication or tools, we are not aware of any existing ones are able to achieve the same speed, scale and functionality of generic response aggregation.

REST API Example

REST Commander supports both intuitive step-by-step wizards and REST APIs. Here is an simple example of uniform request to 3 target servers. In this example, the command and aggregation rule have been pre-defined. More complex API examples using none pre-defined command or aggregation rules can be found here.

Request: (assuming Commander runs on localhost:9000)

HTTP POST to: http://localhost:9000/commands/genUpdateSendCommandWithReplaceVarMapAdhocJson POST Body:

{
   "targetNodes":[
	  "www.restcommander.com",
	  "www.jeffpei.com",
	  "www.yangli907.com"
   ],
  
  "useNewAgentCommand":"false",
  "agentCommandType":"GET_VALIDATE_INTERNALS",
   "willAggregateResponse":true,
   "useNewAggregation":false,
   "aggregationType":"PATTERN_VI_SERVER_CPU",
   "replacementVarMap":{}
}	

Response:

{
	"aggregationMap": {
		"23.54": "1",
		"27.08": "1",
		"7.08": "1"
	},
	"aggregationValueToNodesList": [
		{
			"value": "23.54",
			"nodeList": [
				"www.yangli907.com"
			],
			"isError": false
		},
		{
			"value": "27.08",
			"nodeList": [
				"www.jeffpei.com"
			],
			"isError": false
		},
		{
			"value": "7.08",
			"nodeList": [
				"www.restcommander.com"
			],
			"isError": false
		}
	]
}

More Repositories

1

NMessenger

A fast, lightweight messenger component built on AsyncDisplaykit and written in Swift
Swift
2,424
star
2

nice-modal-react

A modal state manager for React.
TypeScript
1,947
star
3

akutan

A distributed knowledge graph store
Go
1,656
star
4

tsv-utils

eBay's TSV Utilities: Command line tools for large, tabular data files. Filtering, statistics, sampling, joins and more.
D
1,413
star
5

bayesian-belief-networks

Pythonic Bayesian Belief Network Package, supporting creation of and exact inference on Bayesian Belief Networks specified as pure python functions.
Python
1,122
star
6

NuRaft

C++ implementation of Raft core logic as a replication library
C++
962
star
7

parallec

Fast Parallel Async HTTP/SSH/TCP/UDP/Ping Client Java Library. Aggregate 100,000 APIs & send anywhere in 20 lines of code. Ping/HTTP Calls 8000 servers in 12 seconds. (Akka) www.parallec.io
Java
810
star
8

HeadGazeLib

A library to empower iOS app control through head gaze without a finger touch
Swift
754
star
9

Sequence-Semantic-Embedding

Tools and recipes to train deep learning models and build services for NLP tasks such as text classification, semantic search ranking and recall fetching, cross-lingual information retrieval, and question answering etc.
Python
459
star
10

modanet

ModaNet: A large-scale street fashion dataset with polygon annotations
327
star
11

flutter_glove_box

Various eBay tools for Flutter development
Dart
316
star
12

Neutrino

Neutrino is a software load balancer(SLB)
Scala
306
star
13

KPRN

Reasoning Over Knowledge Graph Paths for Recommendation
Lua
279
star
14

UAF

UAF - Universal Authentication Framework
Java
276
star
15

griffin

Model driven data quality service
JavaScript
240
star
16

cors-filter

CORS (Cross Origin Resource Sharing) is a mechanism supported by W3C to enable cross origin requests in web-browsers. CORS requires support from both browser and server to work. This is a Java servlet filter implementation of server-side CORS for web containers such as Apache Tomcat.
Java
231
star
17

Jungle

An embedded key-value store library specialized for building state machine and log store
C++
218
star
18

ebayui-core

Collection of Marko widgets; considered to be the core building blocks for all eBay components, pages & apps
TypeScript
209
star
19

sbom-scorecard

Generate a score for your sbom to understand if it will actually be useful.
Go
208
star
20

jsonpipe

A lightweight AJAX client for chunked JSON responses
JavaScript
204
star
21

ebay-font

A small utility to efficiently load custom web fonts
JavaScript
175
star
22

skin

Pure CSS framework designed & developed by eBay for a branded, e-commerce marketplace.
JavaScript
171
star
23

accelerator

The Accelerator is a tool for fast and reproducible processing of large amounts of data.
Python
150
star
24

firebase-remote-config-monitor

Monitors firebase remote config values, posting changes to slack
JavaScript
136
star
25

maxDNN

High Efficiency Convolution Kernel for Maxwell GPU Architecture
C++
132
star
26

go-ovn

A Go library for OVN Northbound/Southbound DB access using native OVSDB protocol
Go
107
star
27

Gringofts

Gringofts makes it easy to build a replicated, fault-tolerant, high throughput and distributed event-sourced system.
C++
102
star
28

parallec-samples

Single file examples and ready-to-use servers show how to use parallec.io library. Examples to aggregate APIs and publish to Elastic Search and Kafka, and many more. www.parallec.io
Java
92
star
29

userscript-proxy

HTTP proxy to inject scripts and stylesheets into existing sites.
JavaScript
84
star
30

xcelite

Java
81
star
31

mindpatterns

HTML Accessibility Pattern Examples
HTML
79
star
32

embedded-druid

Java
75
star
33

figma-include-accessibility-annotations

Include is a tool built to make annotating for accessibility (a11y) easier—easier for designers to spec and easier for developers to understand what is required.
JavaScript
73
star
34

RANSynCoders

Jupyter Notebook
72
star
35

ebay-oauth-python-client

Python OAuth SDK: Get OAuth tokens for eBay public APIs
Python
69
star
36

Design-Grid-Overlay

A Chrome extension to overlay a design grid on your web page; configurable to fit many design scenarios.
JavaScript
65
star
37

ebay-oauth-nodejs-client

🔑 Generate an OAuth token that can be used to call the eBay Developer REST APIs.
JavaScript
61
star
38

json-comparison

Powerful JSON comparison tool for identifying all the changes within JSON files
Java
60
star
39

xFraud

Jupyter Notebook
60
star
40

bascomtask

Lightweight parallel Java tasks
Java
59
star
41

DASTProxy

Java
57
star
42

jsonex

Java Object Serializer and Deserializer to JSON Format. Focuses on configuration friendliness, arbitrary object serialization and compact JSON format
Java
56
star
43

ebay-oauth-csharp-client

eBay OAuth C# Client Library
C#
53
star
44

nvidiagpubeat

nvidiagpubeat is an elastic beat that uses NVIDIA System Management Interface (nvidia-smi) to monitor NVIDIA GPU devices and can ingest metrics into Elastic search cluster, with support for both 6.x and 7.x versions of beats. nvidia-smi is a command line utility, based on top of the NVIDIA Management Library (NVML), intended to aid in the management and monitoring of NVIDIA GPU devices.
Go
53
star
45

nice-dag

nice-dag is a lightweight javascript library, which is used to present a DAG diagram.
TypeScript
47
star
46

SparkChamber

An event tracking framework for iOS
Swift
45
star
47

ebay-oauth-java-client

eBay OAuth APIs client for Java
Java
45
star
48

Winder

Winder is a simple state machine based on Quartz Scheduler. It helps to write multiple steps tasks on Quartz Scheduler. Winder derived from a state machine which is widly used in eBay Cloud. eBay Platform As A Service(PaaS) uses it to deploy software to hundreds of thousands virtual machines.
Java
45
star
49

AutoOpt

Automatic and Simultaneous Adjustment of Learning Rate and Momentum for Stochastic Gradient Descent
Python
44
star
50

GZinga

Java
43
star
51

YiDB

Java
43
star
52

collectbeat

Beats with discovery capabilities for environments like Kubernetes
Go
41
star
53

block-aggregator

C++
40
star
54

Jenkins-Pipeline-Utils

Global Jenkins Pipeline Library with common utilities.
Groovy
39
star
55

cassandra-river

Cassandra river for Elastic search.
Java
38
star
56

bsonpatch

A BSON implementation of RFC 6902 to compute the difference between two BSON documents
Java
38
star
57

arc

adaptive resources and components
JavaScript
35
star
58

regressr

A command line regression testing framework for testing HTTP services
Scala
34
star
59

ebashlib

A bash script battery which gathers several generic helper scripts for other repositories.
Shell
30
star
60

modshot

Takes screenshot of UI modules and compare with baselines using PhantomCSS
JavaScript
29
star
61

visual-html

Visual regression testing without the flakiness.
TypeScript
29
star
62

FeedSDK-Python

eBay Python Feed SDK - SDK for downloading large gzipped (tsv) item feed files and applying filters for curation
Python
29
star
63

accessibility-ruleset-runner

eBay Accessibility Ruleset Runner automates 20% of WCAG 2.0 AA recommendations, saving time on manual testing.
JavaScript
27
star
64

crossdomain-xhr

JavaScript
27
star
65

oink

REST based interface for PIG execution
Java
27
star
66

bonsai

open source version of the Bonsai library
Scala
26
star
67

ebayui-core-react

eBayUI React components
TypeScript
25
star
68

geosense

Self-contained jar to lookup timezone by lat+lon
Java
25
star
69

browser-telemetry

A Telemetry module for collecting errors, logs, metrics, uncaught exceptions etc on browser side.
JavaScript
25
star
70

oja

Lightweight Dependency Injection Framework for Node.JS Apps - Structure your application business logic
JavaScript
25
star
71

SketchSVG

Have icons in a Sketch file but don't want to manually extract and compress them as SVGs? Let our SketchSVG tool do it!
JavaScript
25
star
72

CustomRippleView

The Custom Ripple View library provides Android developers an easy way to customize and implement a Ripple Effect view.
Kotlin
24
star
73

FGrav

Dynamic Flame Graph Visualizations from raw data in your browser
JavaScript
24
star
74

nodash

Lightweight replacement for subset of Lodash
JavaScript
24
star
75

FeedSDK

Java SDK for downloading large gzipped (tsv) item feed files and applying filters for curation
Java
23
star
76

kube-credentials-plugin

A Jenkins plugin to store credentials in kubernetes
Java
21
star
77

releaser

A declarative API that syncs specs from git to kubernetes
Go
20
star
78

airflow-rest-api-plugin

A plugin of Apache Airflow that exposes REST endpoints for custom REST APIs.
Python
20
star
79

mtdtool

The Manual Test Demultiplexer is a desktop app (Mac and Windows) that provides an interface for driving manual testing on multiple physical devices.
Java
20
star
80

EBNObservable

A block-based Key-Value Observing (KVO) implementation with observable collections.
Objective-C
19
star
81

nice-form-react

A meta based form builder for React.
TypeScript
18
star
82

skin-react

Skin components built with React (Typescript)
TypeScript
18
star
83

accelerator-project_skeleton

Python
18
star
84

taxonomy-sdk

An SDK designed to bring transparency to the rapid evolution of our aspects metadata for our partners.
Java
18
star
85

wextracto

Python
17
star
86

HomeStore

Storage Engine for block and key/value stores.
C++
17
star
87

myriad

Java
17
star
88

event-notification-nodejs-sdk

NodeJS SDK designed to simplify processing of eBay notifications.
JavaScript
17
star
89

TDD-Albums

A Hands-On Tutorial for iPhone Developers Learning TDD
17
star
90

ebay-oauth-android-client

eBay OAuth Android Client library
Kotlin
16
star
91

fluid

Fluid Web Components
JavaScript
16
star
92

ostara

Java
16
star
93

lightning

Lightning is a Java based, super fast, multi-mode, asynchronous, and distributed URL execution engine from eBay
HTML
16
star
94

RTran

Road to Continous Upgrade
Scala
15
star
95

NautilusTelemetry

An iOS implementation of OpenTelemetry
Swift
15
star
96

hadoop-tsdb-connector

Java
15
star
97

Pine

Pine: Machine Learning Prediction As A Service
Scala
15
star
98

pynetforce

Network infrastructure automation service
Python
15
star
99

Vivid

A visual testing tool to compare two web pages visually and generate the pixel difference they have.
JavaScript
14
star
100

sisl

High Performance C++ data structures and utilities
C++
14
star