• Stars
    star
    255
  • Rank 153,966 (Top 4 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created about 5 years ago
  • Updated 11 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

HTTP in TCP in Websockets in HTTP in TCP, Tunnel all the things!

Reverse Tunneling Dialer

remotedialer creates a two-way connection between a server and a client, so that a net.Dial can be performed on the server and actually connects to the client running services accessible on localhost or behind a NAT or firewall.

Architecture

Abstractions

remotedialer consists of structs that organize and abstract a TCP connection between a server and a client. Both client and server use Sessions, which provides a means to make a network connection to a remote endpoint and keep track of incoming connections. A server uses the Server object which contains a sessionManager which governs one or more Sessions, while a client creates a Session directly. A connection implements the io.Reader and io.WriteCloser interfaces, so it can be read from and written to directly. The connection's internal readBuffer monitors the size of the data it is carrying and uses backPressure to pause incoming data transfer until the amount of data is below a threshold.

Data flow

A client establishes a session with a server using the server's URL. The server upgrades the connection to a websocket connection which it uses to create a Session. The client then also creates a Session with the websocket connection.

The client sits in front of some kind of HTTP server, most often a kubernetes API server, and acts as a reverse proxy for that HTTP resource. When a user requests a resource from this remote resource, request first goes to the remotedialer server. The application containing the server is responsible for routing the request to the right client connection.

The request is sent through the websocket connection to the client and read into the client's connection buffer. A pipe is created between the client and the HTTP service which continually copies data between each socket. The request is forwarded through the pipe to the remote service, draining the buffer. The service's response is copied back to the client's buffer, and then forwarded back to the server and copied into the server connection's own buffer. As the user reads the response, the server's buffer is drained.

The pause/resume mechanism checks the size of the buffer for both the client and server. If it is greater than the threshold, a PAUSE message is sent back to the remote connection, as a suggestion not to send any more data. As the buffer is drained, either by the pipe to the remote HTTP service or the user's connection, the size is checked again. When it is lower than the threshold, a RESUME message is sent, and the data transfer may continue.

remotedialer in the Rancher ecosystem

remotedialer is used to connect Rancher to the downstream clusters it manages, enabling a user agent to access the cluster through an endpoint on the Rancher server. remotedialer is used in three main ways:

Agent config and tunnel server

When the agent starts, it initially makes a client connection to the endpoint /v3/connect/register, which runs an authorizer that sets some initial data about the node. The agent continues to connect to the endpoint /v3/connect on a loop. On each connection, it runs an OnConnect handler which pulls down node configuration data from /v3/connect/config.

Steve Aggregation

The steve aggregation server on the agent establishes a remotedialer Session with Rancher, making the steve API on the downstream cluster accessible from the Rancher server and facilitating resource watches.

Health Check

The clusterconnected controller in Rancher uses the established tunnel to check that clusters are still responsive and sets alert conditions on the cluster object if they are not.

Running Locally

remotedialer provides an example client and server which can be run in standalone mode FOR TESTING ONLY. These are found in the server/ and client/ directories.`

Compile

Compile the server and client:

go build -o server/server server/main.go
go build -o client/client client/main.go

Run

Start the server first.

./server/server

The server has debug mode off by default. Enable it with --debug.

The client proxies requests from the remotedialer server to a web server, so it needs to be run somewhere where it can access the web server you want to target. The remotedialer server also needs to be reachable by the client.

For testing purposes, a basic HTTP file server is provided. Build the server with:

go build -o dummy/dummy dummy/main.go

Create a directory with files to serve, then run the web server from that directory:

mkdir www
cd www
echo 'hello' > bigfile
/path/to/dummy

Run the client with

./client/client

Both server and client can be run with even more verbose logging:

CATTLE_TUNNEL_DATA_DEBUG=true ./server/server --debug
CATTLE_TUNNEL_DATA_DEBUG=true ./client/client

Usage

If the remotedialer server is running on 192.168.0.42, and the web service that the client can access is running at address 127.0.0.1:8125, make proxied requests like this:

curl http://192.168.0.42:8123/client/foo/http/127.0.0.1:8125/bigfile

where foo is the hardcoded client ID for this test server.

This test server only supports GET requests.

More Repositories

1

rancher

Complete container management platform
Go
22,538
star
2

os

Tiny Linux distro that runs the entire OS as Docker containers
Go
6,437
star
3

k3os

Purpose-built OS for Kubernetes, fully managed by Kubernetes.
Go
3,403
star
4

rke

Rancher Kubernetes Engine (RKE), an extremely simple, lightning fast Kubernetes distribution that runs entirely within containers.
Go
3,138
star
5

rio

Application Deployment Engine for Kubernetes
Go
2,282
star
6

local-path-provisioner

Dynamically provisioning persistent local storage with Kubernetes
Go
1,938
star
7

fleet

Deploy workloads from Git to large fleets of Kubernetes clusters
Go
1,450
star
8

convoy

A Docker volume plugin, managing persistent container volumes.
Go
1,308
star
9

rke2

Go
1,028
star
10

old-vm

(OBSOLETE) Package and Run Virtual Machines as Docker Containers
Go
646
star
11

ui

Rancher UI
JavaScript
587
star
12

cattle

Infrastructure orchestration engine for Rancher 1.x
Java
574
star
13

k3c

Lightweight local container engine for container development
Go
571
star
14

system-upgrade-controller

In your Kubernetes, upgrading your nodes
Go
502
star
15

dashboard

The Rancher UI
Vue
410
star
16

community-catalog

Catalog entries contributed by the community
Smarty
384
star
17

charts

Github based Helm Chart Index Repository providing charts crafted for Rancher Manager
Smarty
381
star
18

install-docker

Scripts for docker-machine to install a particular docker version
Shell
361
star
19

dapper

Docker build wrapper
Go
358
star
20

quickstart

HCL
357
star
21

cli

Rancher CLI
Go
331
star
22

terraform-provider-rke

Terraform provider plugin for deploy kubernetes cluster by RKE(Rancher Kubernetes Engine)
Go
328
star
23

opni

Multi Cluster Observability with AIOps
Go
323
star
24

kim

In ur kubernetes, buildin ur imagez
Go
323
star
25

trash

Minimalistic Go vendored code manager
Go
296
star
26

terraform-controller

Use K8s to Run Terraform
Go
290
star
27

elemental-toolkit

❄️ The toolkit to build, ship and maintain cloud-init driven Linux derivatives based on container images
Go
251
star
28

elemental

Elemental is an immutable Linux distribution built to run Rancher and its corresponding Kubernetes distributions RKE2 and k3s. It is built using the Elemental-toolkit
Go
228
star
29

terraform-provider-rancher2

Terraform Rancher2 provider
Go
222
star
30

rancher-compose

Docker compose compatible client to deploy to Rancher
Go
214
star
31

wrangler

Write controllers like a boss
Go
205
star
32

os-vagrant

Ruby
176
star
33

rancher-catalog

Smarty
155
star
34

docs

Documentation for Rancher products (for 2.0/new site)
SCSS
140
star
35

fleet-examples

Fleet usage examples
Shell
140
star
36

catalog-dockerfiles

Dockerfiles for Rancher Catalog containers
Shell
131
star
37

rancher-cleanup

Shell
125
star
38

api-spec

Specification for Rancher REST API implementation
121
star
39

k8s-intro-training

HTML
114
star
40

ansible-playbooks

Rancher 1.6 Installation. Doesn't support Rancher 2.0
Python
113
star
41

sherdock

Docker Image Manager
JavaScript
110
star
42

norman

APIs on APIs on APIs
Go
108
star
43

docker-from-scratch

Tiny Docker in Docker
Go
105
star
44

lb-controller

Load Balancer for Rancher services via ingress controllers backed up by a Load Balancer provider of choice
Go
97
star
45

pipeline

Go
96
star
46

k3k

Kubernetes in Kubernetes
Go
89
star
47

container-crontab

Simple cron runner for containers
Go
88
star
48

backup-restore-operator

Go
88
star
49

terraform-modules

Rancher Terraform Modules
HCL
85
star
50

os2

EXPERIMENTAL: A Rancher and Kubernetes optimized immutable Linux distribution based on openSUSE
Go
82
star
51

system-charts

Mustache
82
star
52

vagrant

Vagrant file to stand up a Local Rancher install with 3 nodes
Shell
79
star
53

rancher-dns

A simple DNS server that returns different answers depending on the IP address of the client making the request
Go
79
star
54

giddyup

Go
78
star
55

kontainer-engine

Provisioning kubernetes cluster at ease
Go
78
star
56

go-rancher

Go language bindings for Rancher API
Go
74
star
57

go-skel

Skeleton for Rancher Go Microservices
Shell
71
star
58

runc-cve

CVE patches for legacy runc packaged with Docker
Dockerfile
69
star
59

terraform-k3s-aws-cluster

HCL
67
star
60

agent

Shell
64
star
61

external-dns

Service updating external DNS with Rancher services records for Rancher 1.6
Go
63
star
62

terraform-provider-rancher2-archive

[Deprecated] Use https://github.com/terraform-providers/terraform-provider-rancher2
Go
62
star
63

kontainer-driver-metadata

This repository is to keep information of k8s versions and their dependencies like k8s components flags and system addons images.
Go
62
star
64

gitjob

Go
59
star
65

types

Rancher API types
Go
59
star
66

rancher.github.io

HTML
58
star
67

ui-driver-skel

Skeleton Rancher UI driver for custom docker-machine drivers
JavaScript
58
star
68

rancher-docs

Rancher Documentation
JavaScript
57
star
69

rke2-charts

Shell
56
star
70

os-services

RancherOS Service Compose Templates
Shell
54
star
71

client-python

A Python client for Rancher APIs
Python
49
star
72

hyperkube

Rancher hyperkube images
44
star
73

rancher-cloud-controller-manager

A kubernetes cloud-controller-manager for the rancher cloud
Go
44
star
74

steve

Kubernetes API Translator
Go
43
star
75

rodeo

Smarty
43
star
76

cis-operator

Go
43
star
77

rancherd

Bootstrap Rancher and k3s/rke2
Go
42
star
78

partner-charts

A catalog based on applications from independent software vendors (ISVs). Most of them are SUSE Partners.
Smarty
42
star
79

10acre-ranch

Build Rancher environment on GCE
Shell
41
star
80

secrets-bridge

Go
40
star
81

terraform-rancher-server

HCL
39
star
82

storage

Rancher specific storage plugins
Shell
39
star
83

k8s-sql

Storage backend for Kubernetes using Go database/sql
Go
37
star
84

lasso

Low level generic controller framework
Go
36
star
85

server-chart

[Deprecated] Helm chart for Rancher server
Shell
36
star
86

os-packer

Shell
36
star
87

pipeline-example-go

Go
36
star
88

cluster-template-examples

35
star
89

system-tools

This repo is for tools helping with various cleanup tasks for rancher projects. Example: rancher installation cleanup
Go
35
star
90

elemental-operator

The Elemental operator is responsible for managing the OS versions and maintaining a machine inventory to assist with edge or baremetal installations.
Go
33
star
91

image-mirror

Shell
31
star
92

rancher-metadata

A simple HTTP server that returns EC2-style metadata information that varies depending on the source IP address making the request.
Go
31
star
93

os-base

Base file system for RancherOS images
Shell
31
star
94

websocket-proxy

Go
29
star
95

rke-tools

Tools container for supporting functions in RKE
Go
29
star
96

gdapi-python

Python Binding to API spec
Python
28
star
97

wins

Windows containers connect to Windows host
Go
28
star
98

api-ui

Embedded UI for any service that implements the Rancher API spec
JavaScript
27
star
99

turtles

Rancher CAPI extension
Go
27
star
100

migration-tools

Go
27
star