• Stars
    star
    331
  • Rank 127,323 (Top 3 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created over 3 years ago
  • Updated 9 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A simplified library for decentralized, privacy preserving machine learning

SWARM LEARNING

Product version: 2.0.0

Swarm Learning is a decentralized, privacy-preserving Machine Learning framework. This framework utilizes the computing power at, or near, the distributed data sources to run the Machine Learning algorithms that train the models. It uses the security of a blockchain platform to share learnings with peers in a safe and secure manner. In Swarm Learning, training of the model occurs at the edge, where data is most recent, and where prompt, data-driven decisions are mostly necessary. In this completely decentralized architecture, only the insights learned are shared with the collaborating ML peers, not the raw data. This tremendously enhances data security and privacy.

Swarm Learning framework is made up of various components known as nodes, such as Swarm Learning (SL) nodes, Swarm Network (SN) nodes, Swarm Learning Command Interface (SWCI) nodes, and Swarm Operator (SWOP) nodes. Each node of Swarm Learning is modularized and runs in a separate container. The nodes represent different Swarm Learning functionality and not physical server nodes.

  • SL nodes run the core of Swarm Learning. An SL node works in collaboration with all the other SL nodes in the network. It regularly shares its learnings with the other nodes and incorporates their insights. SL nodes act as an interface between the user model application and other Swarm Learning components. SL nodes take care of distributing and merging model weights in a secured way.

  • SN nodes form the blockchain network. The current version of Swarm Learning uses an open-source version of Ethereum as the underlying blockchain platform. The SN nodes interact with each other using this blockchain platform to maintain and track progress. The SN nodes use this state and progress information to co-ordinate the working of the other swarm learning components.

    Sentinel Node is a special SN node. The Sentinel node is responsible for initializing the blockchain network. This is the first node to start.

NOTE: Only metadata is written to the blockchain. The model itself is not stored in the blockchain.
  • SWCI node is the command interface tool to the Swarm Learning framework. It is used to monitor the Swarm Learning framework. SWCI nodes can connect to any of the SN nodes in a given Swarm Learning framework to manage the framework. For more information on SWCI, see Swarm Learning Command Interface.

  • SWOP is an agent that can manage Swarm Learning operations. SWOP is responsible to execute tasks that are assigned to it. A SWOP node can execute only one task at a time. SWOP helps in executing tasks such as starting and stopping Swarm runs, building and upgrading ML containers, and sharing models for training. For more information about SWOP, see Swarm Operator node (SWOP).

  • Swarm Learning security and digital identity aspects are handled by X.509 certificates. Communication among Swarm Learning components are secured using X.509 certificates. User can either generate their own certificates or directly use certificates generated by any Standard Security software such as SPIRE. For more information on SPIRE, see https://thebottomturtle.io/Solving-the-bottom-turtle-SPIFFE-SPIRE-Book.pdf and https://spiffe.io/.

NOTE: Swarm Learning framework does not initialize if certificates are not provided.
  • Swarm Learning components communicate with each other using a set of TCP/IP ports.
NOTE: The participating nodes must be able to access each other's ports.

For more information on port details that must be opened, see Exposed Ports.

  • License Server installs and manages the license that is required to run the Swarm Learning framework. The licenses are managed by the AutoPass License Server (APLS) that runs on a separate node. For more information, see APLS User Guide.

Swarm Learning nodes works in collaboration with other Swarm Learning nodes in the network. It regularly shares its learnings with the other nodes and incorporates their insights. This process continues until the Swarm Learning nodes train the model to desired state.

User ML components

User can transform any Keras or PyTorch based ML program that is written using Python3 into a Swarm Learning ML program by making a few simple changes to the model training code by including the SwarmCallback API. For more information, see any of the examples included with the Swarm Learning package for a sample code.

The transformed user Machine Learning (user ML node) program can be run on the host or user can build it as a Docker container.

NOTE: HPE recommends users to build an ML Docker container.

The ML node is responsible to train and iteratively update the model. For each ML node, there is a corresponding SL node in the Swarm Learning framework, which performs the Swarm training. Each pair of ML and SL nodes must run on the same host. This process continues until the SL nodes train the model to the desired state.

NOTE: All the ML nodes must use the same ML platform either Keras (based on TensorFlow 2 backend) or PyTorch. Using Keras for some of the nodes and PyTorch for the other nodes is not supported.

Getting Started

  1. Prerequisites for Swarm Learning
  2. Upgrading from earlier evaluation versions
  3. Download and setup Swarm Learning using the SLM-UI installer
  4. Execute MNIST example
  5. Frequently Asked Questions
  6. Troubleshooting

NOTE: Accessing Hewlett Packard Enterprise Support clause and Concurrent swarm training feature mentioned in the documentation are applicable for enterprise customers ONLY.

Documentation

References

Acronyms and Abbreviations

Refer to Acronyms and Abbreviations for more information.

Getting in touch

Feedback and questions are appreciated. You can use the issue tracker to report bugs on GitHub.
or
Join the HPE Developer Slack Workspace and start a discussion in our #hpe-swarm-learning channel.

Contributing

Refer to Contributing for more information.

License

The distribution of Swarm Learning in this repository is for non-commercial and experimental use under this license.

See ATTRIBUTIONS and DATA LICENSE for terms and conditions for using the datasets included in this repository.

More Repositories

1

netperf

Netperf is a benchmark that can be used to measure the performance of many different types of networking. It provides tests for both unidirectional throughput, and end-to-end latency.
C
725
star
2

cacti

An integrated cache and memory access time, cycle time, area, leakage, and dynamic power model
C++
301
star
3

squest

Service request portal on top of Ansible Tower/AWX
Python
294
star
4

LinuxKI

LinuxKI Toolset (Trace-based performance analysis tool)
C
224
star
5

quartz

Quartz: A DRAM-based performance emulator for NVM
C
158
star
6

dlcookbook-dlbs

Deep Learning Benchmarking Suite
Python
130
star
7

POSH-HPEOneView

PowerShell language bindings library for HPE OneView.
PowerShell
125
star
8

mcpat

An integrated power, area, and timing modeling framework for multicore and manycore architectures
C++
122
star
9

PacketRusher

High performance 5G UE/gNB Simulator and CP/UP load tester.
Go
109
star
10

oneview-ansible

This project is no longer being developed and has limited support. Please use the newer Ansible Collection project: https://github.com/HewlettPackard/oneview-ansible-collection
Python
104
star
11

yoda

GitHub extension for agile project management using the issues subsystem.
JavaScript
90
star
12

python-hpOneView

DEPRECATED - no longer actively maintained. New repository: https://github.com/HewlettPackard/oneview-python
Python
87
star
13

python-ilorest-library-old

Python library for iLO RESTful API
Python
85
star
14

wireless-tools

Wireless Tools for Linux
C
85
star
15

PowerShell-ProLiant-SDK

PowerShell sample scripts for managing HPE servers
PowerShell
85
star
16

kraal

Enables the use of Kotlin coroutines and GraalVM native-image together
Kotlin
84
star
17

monkeyble

End-to-end testing framework for Ansible
Python
78
star
18

Atlas

Atlas: Programming for Persistent Memory
C++
74
star
19

foedus

FOEDUS: Fast Optimistic Engine for Data Unification Services
65
star
20

foedus_code

FOEDUS main source code repository
C++
64
star
21

structex

Go structure annotations that supports encoding and decoding; similar to C-style bitfields. Supports bitfield packing, self-describing layout parameters, and alignment.
Go
58
star
22

ansible-ilorest-role

Ansible role for installing the Python iLOrest library and showcasing a few examples.
Python
52
star
23

terraform-provider-oneview

Automates the provisioning of physical infrastructure from a private cloud using templates from HPE OneView with Terraform
Go
49
star
24

reconbf

Recon system hardening scanner
Python
47
star
25

sandpiper

Implementation of the Loopy Belief Propagation algorithm for Apache Spark
Scala
42
star
26

sparkle

C++
38
star
27

ilo-ansible-collection

Ansible Collection and Sample Playbooks for HPE iLO
Python
38
star
28

dockerfile-parser-rs

a Rust library for parsing, validating, and modifying Dockerfiles
Rust
37
star
29

jupyterhub-samlauthenticator

jupyterhub-samlauthenticator
Python
34
star
30

cloudformation-plus

A tool that adds features to AWS CloudFormation that reduce the amount of code you must write to deploy non-trivial applications.
Python
31
star
31

cmf

CMF library helps to collect and store information associated with ML pipelines. It tracks the lineages for artifacts and executions of distributed AI pipelines. It provides API's to record and query the metadata associated with ML pipelines. The framework adopts a data first approach and all artifacts recorded in the framework are versioned and identified by the content hash.
Python
30
star
32

lustre_exporter

Prometheus exporter for use with the Lustre parallel filesystem
Go
29
star
33

oneview-python

Python library for HPE OneView
Python
28
star
34

nvthreads

C
27
star
35

oneview-golang

Golang bindings for OneView api's
Go
27
star
36

nagios-plugins-hpilo

Nagios plug-in for iLO Agentless Management
Shell
26
star
37

mds

Managed Data Structures
Java
26
star
38

dpp

Device Provisioning Protocol is a Wi-Fi Alliance program to securely provision devices to obtain network access.
C
25
star
39

mdc-toolkit

24
star
40

Docker-SimpliVity

Ansible deployment playbooks with detailed deployment guides to rapidly provision a Docker dev and Docker ops environment in less than 30 minutes
Shell
24
star
41

woodchipper

An interactive command-line log processor
Rust
23
star
42

oneview-ansible-collection

Ansible Collection and Sample Playbooks for HPE OneView
Python
22
star
43

hpe-solutions-openshift

This GitHub site contains deployment guides and resources for deploying Red Hat OpenShift on HPE platforms.
Python
22
star
44

chef-provisioning-oneview

Chef Provisioning Driver for HPE OneView
Ruby
21
star
45

lustre-csi-driver

A Lustre container storage interface that allows Kubernetes to mount/unmount provisioned Lustre filesystems into containers.
Go
20
star
46

supersim

A flexible event-driven cycle-accurate network simulator
C++
19
star
47

hpe3par_ansible_module

HPE Alletra 9000 and HPE Primera and HPE 3PAR ansible module to configure, provision and manage storage systems and resources
Python
18
star
48

galadriel

SPIFFE Federation the easy way
Go
18
star
49

csa-ce

This repo contains scripts which start HPE Cloud Service Automation Community Edition 4.7 (HPE CSA CE) in Docker.
Shell
17
star
50

ezdemo

HPE Ezmeral Deployment tool for demos
Shell
17
star
51

ilo-chef

iLO Resource Provider for Chef
Ruby
17
star
52

oneview-chef

This project is no longer being developed and has limited support. In the near future this repository will be fully deprecated. Please consider using other OneView projects, such as Terraform and Ansible Collection
Ruby
17
star
53

oneview-osdeployment

Sample HPE OneView OS deployment integration
Python
16
star
54

oneview-sdk-java

Java SDK for HPE OneView
Java
16
star
55

ilo-sdk-ruby

iLO Software Development Kit for Ruby Programmers.
Ruby
15
star
56

criu-pmem

C
15
star
57

oneview-redfish-toolkit

HPE OneView Redfish Toolkit provides a REST service to answer DMTF's Redfish compliant requests by querying HPE OneView
Python
15
star
58

javascript-ilorest-library

JavaScript
15
star
59

mpgc

Multi-Process Garbage Collector
C++
15
star
60

logger

A log parsing engine written in Java for high performance. Additionally, logger is also highly configurable.
Java
14
star
61

pcp_exporter

Exporter for Performance CoPilot Metrics
Go
14
star
62

Aruba-FlaskwithNetworking

Python
14
star
63

zabbix-plugins-hpeilo

Shell
13
star
64

hpe-oneview-hubot

HPE OneView Chatbot Integration
JavaScript
13
star
65

osfci

Go
13
star
66

prometheus-parser-rs

a Rust library for parsing and validating Prometheus query expressions
Rust
12
star
67

nagios-hpeilo-restful-extension

Nagios Plug-in for iLO RESTful Extension
Shell
12
star
68

c-spiffe

C
12
star
69

LSGI

Large-Scale Graph Inference
HTML
12
star
70

oneview-powershell-samples

HPE OneView PowerShell sample scripts
PowerShell
12
star
71

oneview-sdk-ruby

This project is no longer being developed and has limited support. In the near future this repository will be fully deprecated. Please consider using other OneView projects, such as Golang and Python.
Ruby
12
star
72

zing-stats

Tool for generating summary stat reports and graphs from Gerrit (https://www.gerritcodereview.com/) and GitHub Enterprise review and pull requests data using https://plot.ly/.
Python
12
star
73

py-spiffe

Python library for SPIFFE support
Python
11
star
74

Jenkins-stats

Tool for gathering job data from the Jenkins CI system (https://jenkins-ci.org/) and generating summary stat reports and graphs. These reports cover metrics such as success/failure rates and job duration.
Python
11
star
75

image-streamer-tools

PowerShell
10
star
76

OpenShift-on-SimpliVity

10
star
77

lsrrb

Linux Software Raid Redundant Boot
Python
10
star
78

Docker-Synergy

Shell
10
star
79

shoveller

Rust
10
star
80

hpecli

Command-line interface for HPE products and services
Go
10
star
81

hpe3par_pstoolkit

The HPE Alletra 9000 and Primera and 3PAR PowerShell Toolkit supports cmdlets, which are wrappers around the native HPE Alletra 9000 or HPE Primera or HPE 3PAR storage CLI commands and Web Services API (WSAPI).
PowerShell
10
star
82

image-streamer-esxi

9
star
83

chef-ilorest-cookbook

Chef cookbook for installing the Python iLOrest library and showcasing a few examples.
Python
9
star
84

GlobalDashboardPS

A Powershell module for working with HPE OneView Global Dashboard
PowerShell
9
star
85

hpe-notebooks

Jupyter Notebook
9
star
86

hpe3par_python_sdk

HPE Alletra 9000 and HPE Primera and HPE 3PAR Software Development Kit for Python
Python
9
star
87

RiscVEdk2

C
9
star
88

HPEDSCC-PowerShell-Toolkit

A PowerShell Toolkit that enables management of a HPE GreenLake DSCC (Data Storage Cloud Console) environment via easy to use PowerShell commands which interface with the publically available RestAPI.
PowerShell
9
star
89

gull

a multi-node fabric-attached memory manager that provides simple abstractions for accessing and allocating NVM from fabric-attached memory
C++
9
star
90

iLOAmpPack-Redfish-API-Docs

iLO Amplifier Pack Redfish API Documentation
JavaScript
8
star
91

python-opsramp

Python binding for the OpsRamp API
Python
8
star
92

ironic-driver-oneview

HPE OneView driver for OpenStack Ironic bare-metal machine provisioning
8
star
93

simplivity-ansible

Ansible Modules for HPE SimpliVity
Python
8
star
94

hpe-nvm

Documentation and utilities for NVDIMM-N device management and support
8
star
95

oneview-puppet

This project is no longer being developed and has limited support. In the near future this repository will be fully deprecated. Please consider using other OneView projects, such as Terraform and Ansible Collection
Ruby
8
star
96

SHARP

Python
7
star
97

devid-provisioning-tool

Go
7
star
98

oneview-ansible-samples

A collection of sample code for oneview-ansible
7
star
99

dc-rl

HTML
7
star
100

simplivity-python

This library provides a Python interface to the HPE SimpliVity REST APIs
Python
7
star