• This repository has been archived on 05/Jul/2024
  • Stars
    star
    154
  • Rank 242,095 (Top 5 %)
  • Language
  • Created over 7 years ago
  • Updated 5 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

FBNet-Command-Runner: A thrift service to run commands on heterogeneous Network devices with configurable parameters.

FBNet Command Runner (FCR)

A thrift service to run commands on heterogeneous Network devices with configurable parameters. It scales to a large number of devices.

It hides most of the devices specific details like:

  • Prompt processing
  • IP Address lookup
  • The base implementation only supports SSH. But other type of connections can be easily added.
  • Client can use any language of choice, to communicate with server using thrift call.

Requirements

  • python3.5+
  • asyncssh

Installing FCR

FCR can be quickly installed using pip. Just clone the git repo and install using the given requirement files

# Clone the git repo.
git clone --recursive https://github.com/facebookincubator/FCR.git

# Create a virtual environment
python3 -m venv venv
. venv/bin/activate

cd FCR
# use pip to install the required modules.
pip3 install -r requirements.txt
pip3 install .

FCR client

import asyncio

from fbnet.command_runner.thrift_client import AsyncioThriftClient

# Import FCR Thrift Types
from fbnet.command_runner_asyncio.CommandRunner import ttypes as fcr_ttypes

# Import FCR Service Client
from fbnet.command_runner_asyncio.CommandRunner.Command import Client as FcrClient

get the device and user information

import getpass

# Device Information
hostname = 'dev-001'
username = 'netbot'
password = getpass.getpass('%s Password: ' % username)
netbot Password: 路路路路路路路路

run

Run a commands on a single device. Multiple commands are separated by new lines.

# Destination device
device = fcr_ttypes.Device(hostname=hostname, username=username, password=password)

async def run(cmd, device):
    async with AsyncioThriftClient(FcrClient, 'localhost', 5000) as client:
        res = await client.run(cmd, device)
        # type of res is `struct CommandResult`
        print(res.output)

loop = asyncio.get_event_loop()
loop.run_until_complete(run('uname -a\nip -4 add list eth0', device))
netbot@dev-001:~$ uname -a
Linux dev-001 4.4.0-79-generic #100-Ubuntu SMP Wed May 17 19:58:14 UTC 2017 x86_64 GNU/Linux
netbot@dev-001:~$ ip -4 add list eth0
161: eth0@if162: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    inet 172.17.0.2/16 scope global eth0
       valid_lft forever preferred_lft forever

bulk_run

Multiple commands can be run on multiple devices using bulk_run api.

devices = [
    fcr_ttypes.Device(hostname='dev-%03d' % i, username=username, password=password)
    for i in range(1, 150)
]

dev_cmds = {dev: ['uname -r', 'whoami'] for dev in devices}

async def bulk_run(dev_cmds):
    async with AsyncioThriftClient(FcrClient, 'localhost', 5000) as client:
        results = await client.bulk_run(dev_cmds)
        # result is a map<devname, list<CommandResult>>
        return results

loop = asyncio.get_event_loop()
results = loop.run_until_complete(bulk_run(dev_cmds))

print("Some results from the devices")
for dev in devices[::10]:
    hostname = dev.hostname
    print(hostname, ':', results[hostname][0].output.splitlines()[1:])
Some results from the devices
dev-001 : ['4.4.0-79-generic']
dev-011 : ['4.4.0-79-generic']
dev-021 : ['4.4.0-79-generic']
dev-031 : ['4.4.0-79-generic']
dev-041 : ['4.4.0-79-generic']
dev-051 : ['4.4.0-79-generic']
dev-061 : ['4.4.0-79-generic']
dev-071 : ['4.4.0-79-generic']
dev-081 : ['4.4.0-79-generic']
dev-091 : ['4.4.0-79-generic']
dev-101 : ['4.4.0-79-generic']
dev-111 : ['4.4.0-79-generic']
dev-121 : ['4.4.0-79-generic']
dev-131 : ['4.4.0-79-generic']
dev-141 : ['4.4.0-79-generic']

Persisting connection to the device

Sometimes you want to keep the connection to the device open across multiple commands. For this FCR provides following APIS

# Destination device
device = fcr_ttypes.Device(hostname='dev-001', username=username, password=password)

async def fcr_session():

    async with AsyncioThriftClient(FcrClient, 'localhost', 5000) as client:
        # Open a session to the device
        session = await client.open_session(device)

        # Run commands on the open session
        res = await client.run_session(session, 'uname -a')
        print(res.output)

        res = await client.run_session(session, 'ip addr show | grep "inet\>"')
        print(res.output)

        # Finally Close the session
        await client.close_session(session)

loop = asyncio.get_event_loop()
results = loop.run_until_complete(fcr_session())
netbot@dev-001:~$ uname -a
Linux dev-001 4.4.0-79-generic #100-Ubuntu SMP Wed May 17 19:58:14 UTC 2017 x86_64 GNU/Linux
netbot@dev-001:~$ ip addr show | grep "inet\>"
    inet 127.0.0.1/8 scope host lo
    inet 172.17.0.2/16 scope global eth0

Customize FCR service

FCR is a thrift service that provides APIs to run remote commands on devices. To use FCR service you will need to extend the FcrServiceBase to adapt it to your specific environment. FCR provides interfaces to configure supported vendors and allows you to extend it get device information for your backend database.

from fbnet.command_runner.service import FcrServiceBase

class FCRService(FcrServiceBase):

    def __init__(self, args=None):
        super().__init__("FCR", args=args)

def main(args=None):
    service = FCRService()
    service.start(args)

You will need to customize the service to adapt to your specific environment. This will mostly work out of the box, but to use it effectively you will need to adapt it according to your specific environment

You will need to customize FCR service to work with your environment.

  • Device Vendor information: Information about vendors whose devices are in your Network. e.g. vendor name, device prompts. See below for more details.
  • Device Database: for loading metadata from your data source, e.g. names of devices, device address, device vendors.

Device Vendors

For FCR to work, it needs to know the prompts (regex) to expect from the devices. Each vendor can have a different set of prompts

  • setup commands: A set of initial commands to setup the connections. These commands are sent to the device everytime a new session is created. e.g. [term len 0,term width 511].
  • prompt regex: A list of expected prompts (regex) from devices of this vendor.
  • command timeouts: A default timeout specific to this vendor. This allows us to work with slow vendors
  • clear command: Command used to clear the last command. A sequence used to clear the command line. (Default: ^U)
  • session type: Session type to be used for the vendor. Most of the vendors will support the SSHCommandSession. Some vendor may provide a programmable interface, which may require a custom session type

This information can be provided to FCR service in json file. This file can be specified using '--device-vendors' command line options.

device_vendors.json

{
  "vendor_config": {
    "vendor1": {
        "vendor_name": "vendor1",
        "session_type": "ssh",
        "prompt_regex": ["[\\w.]+#\\s*"],
        "cli_setup": [
          "en",
          "term len 0"
        ],
        "shell_prompts": ["\\$"]
    },
    "vendor2": {
        "vendor_name": "vendor2",
        "session_type": "ssh",
        "prompt_regex": ["[\\w./:]+[$#]\\s*"],
        "cli_setup": [ "term len 0" ]
    },
  }

Device DB

FCR relies in a devices database to resolve device information. This database is populated with device_info objects. You will need to extend this database and implement _fetch_device_data() method

_fetch_device_data() needs to return an array of fbnet.command_runner.device_info objects. if needed you can extend fbnet.command_runner.device_info according to your specific environment.

from fbnet.command_runner.device_db import BaseDeviceDB
from fbnet.command_runner.options import Option

# You will likely get this data from your backend systems.
# But for illustration we will assume this is available in JSON
# format
import json
json_devdata = '''
[
  {"host": "rtr1", "chassis": "T1000", "ip": ["10.0.0.1","20.0.0.1"], "role": "router", "vendor": "vendor1"},
  {"host": "rtr2", "chassis": "T5000", "ip": ["10.0.0.2","20.0.0.2"], "role": "router", "vendor": "vendor2"},
  {"host": "rtr3", "chassis": "T2000", "ip": ["10.0.0.3","20.0.0.3"], "role": "switch", "vendor": "vendor2"}
]
'''

class DeviceDB(BaseDeviceDB):

    async def _fetch_device_data(self, name_filter=None):
        '''
        Fetch data from your backend database.

        This sample implementation assumes you have the data in a json format
        '''        
        devinfos = json.loads(json_devdata)

        return {self._make_dev(devinfo) for devinfo in devinfos }

    def _make_dev(devinfo):
        return DeviceInfo(
            self.service,
            hostname=devinfo['host'],
            username='default',     # typically a user with a bare minimum RO privileges
            password='passwd',
            pref_ips=devinfo['ip'], # a list of IP addresses in order of preferences
            ip=devinfo['ip'][0],    # the default fallback IP (after exhausting the pref_ips
            vendor_data=self.app.vendors.get(devinfo['vendor']),
            role=devinfo['role'],
            ch_model=devinfo['ch_model'])

CLI options for the server

bin/fcr_service.py --help
usage: fcr_service.py [-h] [--remote_call_overhead REMOTE_CALL_OVERHEAD]
                      [--lb_threshold LB_THRESHOLD] [-p PORT]
                      [--asyncio_debug]
                      [--log_level {debug,info,warning,error,critical}]
                      [--max_default_executor_threads MAX_DEFAULT_EXECUTOR_THREADS]
                      [--exit_max_wait EXIT_MAX_WAIT]
                      [--device_db_update_interval DEVICE_DB_UPDATE_INTERVAL]
                      [--device_name_filter DEVICE_NAME_FILTER]

A thrift service to run commands on heterogeneous Network devices with configurable parameters.

It hides most of the devices specific details:

* Prompt processing
* IP Address lookup
* Session Types

The clients can be implemented in any language supported by thrift

optional arguments:
  --asyncio_debug       turn on debug for asyncio (default: False)
  --device_db_update_interval DEVICE_DB_UPDATE_INTERVAL
                        device db update interval (in seconds). (default:
                        1800)
  --device_name_filter DEVICE_NAME_FILTER
                        A regex to restrict the database to matching device
                        names. This is passed as an argument to
                        self._fetch_devices_data() method. (default: None)
  --exit_max_wait EXIT_MAX_WAIT
                        Max time (seconds) to wait for session to terminate.
                        This allows existing session to complete gracefully.
                        (default: 300)
  --lb_threshold LB_THRESHOLD
                        Load Balance threashold for bulk_run calls. If number
                        of devices is greater than this threashold, the
                        requests are broken and send to other instances using
                        bulk_run_local() api (default: 100)
  --log_level {debug,info,warning,error,critical}
                        logging level (default: info)
  --max_default_executor_threads MAX_DEFAULT_EXECUTOR_THREADS
                        Max number of worker threads. These are used for
                        long running blocking non-async calls that are not
                        handled in async loop. The default should be good
                        enough for most use cases (default: 4)
  --remote_call_overhead REMOTE_CALL_OVERHEAD
                        Overhead for running commands remotely (for bulk
                        calls). This is subtracted from the requested timeout
                        when request are forwarded to remote service. This
                        allows the bulk_run() to completed within the
                        requested timeout (default: 20)
  -h, --help            show this help message and exit
  -p PORT, --port PORT  TCP port for FCR service (default: 5000)

License

FBNet Command Runner is MIT-licensed.

More Repositories

1

SocketRocket

A conforming Objective-C WebSocket client library.
Objective-C
9,534
star
2

katran

A high performance layer 4 load balancer
C
4,674
star
3

AITemplate

AITemplate is a Python framework which renders neural network into high performance CUDA/HIP C++ code. Specialized for FP16 TensorCore (NVIDIA GPU) and MatrixCore (AMD GPU) inference.
Python
4,515
star
4

velox

A C++ vectorized database acceleration library aimed to optimizing query engines and data processing systems.
C++
3,474
star
5

cinder

Cinder is Meta's internal performance-oriented production version of CPython.
Python
3,458
star
6

FBX2glTF

A command-line tool for the conversion of 3D model assets on the FBX file format to the glTF file format.
C++
2,060
star
7

spectrum

A client-side image transcoding library.
C++
1,987
star
8

oomd

A userspace out-of-memory killer
C++
1,795
star
9

fastmod

A fast partial replacement for the codemod tool
Rust
1,648
star
10

xar

executable archive format
Python
1,571
star
11

Bowler

Safe code refactoring for modern Python.
Python
1,532
star
12

submitit

Python 3.8+ toolbox for submitting jobs to Slurm
Python
1,245
star
13

gloo

Collective communications library with various primitives for multi-machine training.
C++
1,181
star
14

fizz

C++14 implementation of the TLS-1.3 standard
C++
1,128
star
15

dhcplb

dhcplb is Facebook's implementation of a load balancer for DHCP.
Go
1,046
star
16

below

A time traveling resource monitor for modern Linux systems
Rust
1,029
star
17

OnlineSchemaChange

A tool for performing online schema changes on MySQL.
Python
965
star
18

Glean

System for collecting, deriving and working with facts about source code.
Hack
923
star
19

Battery-Metrics

Library that helps in instrumenting battery related system metrics.
Java
736
star
20

retrie

Retrie is a powerful, easy-to-use codemodding tool for Haskell.
Haskell
500
star
21

superconsole

The superconsole crate provides a handler and building blocks for powerful, yet minimally intrusive TUIs. It is cross platform, supporting Windows 7+, Linux, and MacOS. Rustaceans who want to create non-interactive TUIs can use the component composition building block system to quickly deploy their code.
Rust
477
star
22

nvdtools

A set of tools to work with the feeds (vulnerabilities, CPE dictionary etc.) distributed by National Vulnerability Database (NVD)
Go
446
star
23

nimble

New file format for storage of large columnar datasets.
C++
419
star
24

infima

A UI framework that provides websites with the minimal CSS and JS needed to get started with building a modern responsive beautiful website
HTML
405
star
25

CG-SQL

CG/SQL is a compiler that converts a SQL Stored Procedure like language into C for SQLite. SQLite has no stored procedures of its own. CG/CQL can also generate other useful artifacts for testing and schema maintenance.
HTML
391
star
26

TTPForge

The TTPForge is a Cybersecurity Framework for developing, automating, and executing attacker Tactics, Techniques, and Procedures (TTPs).
Go
320
star
27

flowtorch

This library would form a permanent home for reusable components for deep probabilistic programming. The library would form and harness a community of users and contributors by focusing initially on complete infra and documentation for how to use and create components.
Jupyter Notebook
300
star
28

ptr

Python Test Runner.
Python
284
star
29

senpai

Senpai is an automated memory sizing tool for container applications.
Python
270
star
30

fbjni

A library designed to simplify the usage of the Java Native Interface
C++
260
star
31

dynolog

Dynolog is a telemetry daemon for performance monitoring and tracing. It exports metrics from different components in the system like the linux kernel, CPU, disks, Intel PT, GPUs etc. Dynolog also integrates with pytorch and can trigger traces for distributed training applications.
C++
251
star
32

gazebo

A Rust library containing a collection of small well-tested primitives.
Rust
235
star
33

reindeer

Reindeer is a tool to transform Rust Cargo dependencies into generated Buck build rules
Rust
177
star
34

dispenso

The project provides high-performance concurrency, enabling highly parallel computation.
C++
174
star
35

GeoLift

GeoLift is an end-to-end geo-experimental methodology based on Synthetic Control Methods used to measure the true incremental effect (Lift) of ad campaign.
R
171
star
36

oculus-linux-kernel

The Linux kernel code for Oculus devices
C
156
star
37

dataclassgenerate

DataClassGenerate (or simply DCG) is a Kotlin compiler plugin that addresses an Android APK size overhead from Kotlin data classes.
Kotlin
154
star
38

hsthrift

The Haskell Thrift Compiler. This is an implementation of the Thrift spec that generates code in Haskell. It depends on the fbthrift project for the implementation of the underlying transport.
Haskell
148
star
39

FioSynth

Tool which enables the creation of synthetic storage workloads, automates the execution and results collection of synthetic storage benchmarks.
Python
140
star
40

meta-code-verify

Code Verify is an open source web browser extension that confirms that your Facebook, Messenger, Instagram, and WhatsApp Web code hasn鈥檛 been tampered with or altered, and that the Web experience you鈥檙e getting is the same as everyone else鈥檚.
TypeScript
137
star
41

tacquito

Tacquito is an open source TACACs+ server written in Go that implements RFC8907
Go
93
star
42

go-qfext

a fast counting quotient filter implementation in golang
Go
91
star
43

momentum

A library for human kinematic motion and numerical optimization solvers to apply human motion
C++
89
star
44

ForgeArmory

ForgeArmory provides TTPs that can be used with the TTPForge (https://github.com/facebookincubator/ttpforge).
Swift
80
star
45

antlir

ANoTher Linux Image buildeR
Rust
76
star
46

sks

Secure Key Storage (SKS) is a library for Go that abstracts Security Hardware on laptops.
Go
72
star
47

dcrpm

A tool to detect and correct common issues around RPM database corruption.
Python
72
star
48

ConversionsAPI-Tag-for-GoogleTagManager

This repository will contain the artifacts needed for setting up Conversions API implementation on Google Tag Manager's serverside. Please follow the instructions https://www.facebook.com/business/help/702509907046774
Smarty
64
star
49

InjKit

Injection Kit. It is a java bytecode processing library for bytecode injection and transformation.
Java
55
star
50

obs-plugins

OBS Plugins
C++
54
star
51

glTFVariantMeld

An application that accepts files on the glTF format, interprets them as variants of an over-arching whole, and melds them together.
Rust
51
star
52

haberdashery

A collection of high-performance crypto implementations.
Rust
44
star
53

later

A framework for python asyncio with batteries included for people writing services in python asyncio
Python
39
star
54

go2chef

A Golang tool to bootstrap a system from zero so that it's able to run Chef to be managed
Go
39
star
55

CommutingZones

Commuting zones are geographic areas where people live and work and are useful for understanding local economies, as well as how they differ from traditional boundaries. These zones are a set of boundary shapes built using aggregated estimates of home and work locations. Data used to build commuting zones is aggregated and de-identified.
JavaScript
39
star
56

ConversionsAPI-Client-for-GoogleTagManager

This repository will contain the artifacts needed for setting up Conversions API implementation on Google Tag Manager's serverside. Primarily we will be hosting, - ConversionsAPI(Facebook) Client - listens on the events fired to GTM Server and maps them to common GTM schema. - ConversionsAPI(Facebook) Tag - server tag that fires events to CAPI.For more details on Design here https//fburl.com/uae68vlr
37
star
57

Facebook-Pixel-for-Wordpress

A plugin for advertisers who use Wordpress to enable them easily setup the facebook pixel.
JavaScript
37
star
58

strobelight

Meta's fleetwide profiler framework
C++
28
star
59

buck2-change-detector

Given a Buck2 built project and a set of changes (e.g. from source control) compute the targets that may have changed. Sometimes known as a target determinator, useful for optimizing a CI system.
Rust
28
star
60

wordpress-messenger-customer-chat-plugin

Messenger Customer Chat Plugin for WordPress
PHP
27
star
61

CP4M

CP4M is a conversational marketing platform which enables advertisers to integrate their customer-facing chatbots with FB Messenger/WhatsApp, in order to meet customers where they are and drive native conversations on the advertiser's owned infra.
Java
27
star
62

rush

RUSH (Reliable - unreliable - Streaming Protocol)
C++
26
star
63

MY_ENUM

Small c++ macro library to add compile-time introspection to c++ enum classes.
C++
17
star
64

SafeC

Library containing safer alternatives/wrappers for insecure C APIs.
C++
16
star
65

go-belt

It is an implementation-agnostic Go(lang) package to generalize observability tooling (logger, metrics, tracer and so on) and provide ability to use any of these tools with a standard context. Essentially it is an attempt to standardize observability API in Go.
Go
16
star
66

spark-ar-core-libs

Core libraries that can be used in Spark AR. You can import each library depends on your requirements.
TypeScript
15
star
67

scrut

Scrut is a testing toolkit for CLI applications. A tool to scrutinize terminal programs without fuss.
Rust
15
star
68

sado

A macOS signed-app shim for running daemons with reliable capabilities.
Swift
12
star
69

Portal-Kernel

Kernel Code for Portal.
C
11
star
70

npe-toolkit

Libraries, guides, blueprints, and sample code, to enable rapidly building 0-1 applications on iOS, Android and web.
TypeScript
10
star
71

Eigen-FBPlugins

This is collection of plugins extending Eigen arrays/matrices with main focus on using them for computer vision. In particular, this project should provide support for multichannel arrays (missing in vanilla Eigen) and seamless integration between Eigen types and OpenCV functions.
C++
9
star
72

isometric_pattern_matcher

A new isometric calibration pattern - which should/might lead to higher accuracy calibrations compared to existing solutions (checkerboards, patterns of circles).
C++
8
star
73

dnf-plugin-cow

Code to enable Copy on Write features being upstreamed in rpm and librepo
Shell
8
star
74

jupyterhub_fb_authenticator

JupyterHub Facebook Authenticator is a Facebook OAuth authenticator built on top of OAuthenticator.
Python
8
star
75

wireguard_py

Cython library for Wireguard
C
7
star
76

kernel-patches-daemon

Sync Patchwork series's with Github pull requests
Python
6
star
77

meta-fbvuln

OpenEmbedded meta-layer that allows producing a vulnerability manifest alongside a Yocto build. The produced manifest is suitable for ongoing vulnerability scanning of fielded software.
5
star
78

gazebo_lint

A Rust linter that provides various suggestions based on the new primitives offered in the `gazebo` library.
Rust
4
star
79

language-capirca

Adds syntax highlighting for Capirca filetypes in Atom. Capirca is an open source standard for writing vendor-neutral firewall policies as originally released by Google: https://github.com/google/capirca
3
star
80

cinderx

cinderx
C++
3
star
81

fbc_owrt_feed

Facebook Connectivity OpenWrt Feed. Package feed for OpenWrt router OS by Facebook Connectivity programme.
Lua
2
star
82

cutlass-fork

A Meta fork of NV CUTLASS repo.
C++
2
star
83

hwbits_lib

Abstraction of hardware register-level protocols in a python semantic names.
Python
1
star