Neco is a project to build and maintain highly automated on-premise data centers using Kubernetes at the center of its system. The project is led by a CNCF Silver member Cybozu which is also known as Kintone in the United States.
Neco is also the name of the architecture for systems built on the deliverables of the project.
Please read about Project Neco: Building Cloud Native On-Premise Data Centers.
Table of Contents:
Resources
Blogs
- English: https://blog.kintone.io/archive/category/neco
- Japanese: https://blog.cybozu.io/archive/category/Neco
Repositories
The project consists of a lot of repositories including:
- cybozu-go/neco: The main repository.
- cybozu-go/neco-apps: Kubernetes application manifests for Argo CD.
- cybozu/neco-containers: Dockerfiles to build container images.
- cybozu-go/sabakan: Versatile network boot server.
- cybozu-go/cke: Cybozu Kubernetes Engine.
- cybozu-go/coil: CNI plugin in favor of UNIX philosophy.
- cybozu-go/topolvm: Capacity-aware CSI plugin using LVM.
- cybozu-go/placemat: Virtual data center construction tool.
System overview
A Neco data center consists of a few (from 3 to 5) management servers called boot sever and thousands of servers for a Kubernetes cluster; pretty big, huh? In fact, Neco is designed to manage such large clusters.
Boot servers
Boot servers are symmetrical with each other except for host names and IP addresses.
All persistent data are stored in an etcd cluster. Thanks to etcd, adding or removing a boot server can be done easily.
Sabakan is a network boot server that supports DHCP and HTTP. It can also manage an inventory of Node servers and assign IP addresses automatically. Node servers should use UEFI HTTP boot to load Flatcar from sabakan.
CKE, or Cybozu Kubernetes Engines, is a CNCF certified Kubernetes installer. It queries Node server status to Sabakan and chooses Nodes to construct Kubernetes control plane and worker nodes. TLS certificates required for Kubernetes are issued by Vault.
Boot servers are set up using a custom Ubuntu installer. installer directory contains the tools to build the custom installer.
Neco Continuous Delivery System (Neco CD)
To bootstrap and maintain boot servers, a simple continuous delivery system called Neco CD is implemented in this repository.
Neco CD consists of these programs and artifacts:
neco-updater: a daemon to watch GitHub and notifyneco-workerof new artifacts.neco-worker: a daemon to install/update programs on a boot server.neco: CLI tool to bootstrap and configure boot servers.- Neco debian package: archive of program binaries to be installed on boot servers.
Debian packages are built by CircleCI and released on GitHub releases.
Read docs/cicd.md for details.
Kubernetes
A Kubernetes cluster created with CKE is a vanilla installation; it almost has nothing useful.
We have selected a set of applications such as MetalLB, Calico, or Teleport to make the vanilla Kubernetes fully featured. The manifests of the applications are maintained in neco-apps repository and continuously delivered by Argo CD.
Network
Design
Network of Neco is designed with these goals:
- Bare-metal performance, i.e., no overlays like VXLAN
- Scalability, especially for east-west traffic
- Vendor neutrality
The data center network for Neco should employ leaf-spine topology. More specifically, each data center rack works as an autonomous system (AS) and exchanges routes with BGP.
Since BGP is a layer-3 routing protocol, use of layer-2 can be limited within a single rack.
Node BGP and redundancy
Kubernetes nodes need to route packets for Pods. Without overlay networking, this means that each node needs to advertise routing information. In Neco, each node runs BIRD to speak BGP and advertise routes for Pods.
For high availability, network links must be redundant. Therefore, Neco requires that each rack has two Top-of-Rack (ToR) switches and each server has two ethernet links. To reduce convergence time of BGP, BFD is used together.
This means that ToR switches and node servers in a rack are all BGP speakers and need to connect each other as iBGP peers. To make configurations simple and flexible, ToR switches are configured as route reflectors.
The two ToR switches are independent; each has its own managing subnet. As a result, each server has two IP addresses for each ToR switch. To tolerate single link or ToR switch failure, the server has another IP address called the node address with /32 netmask and advertises it via BGP.
DHCP and location of boot servers
UEFI HTTP Boot requires a DHCP server which requires layer-2 broadcast domain. Without DHCP relay servers, a dedicated server must be run as a boot server for each rack because layer-2 domain does not span racks.
If ToR switches can provide DHCP relay, this requirement can be relaxed.
Tests
Virtual data center
To make continuous delivery reliable enough, thorough and comprehensive system tests need to be performed. Since Neco is the system to bootstrap and maintain an on-premise data center, this means that the tests should do the same thing, that is, to bootstrap and maintain a data center!
To simulate an on-premise data center, we have created placemat, a tool to construct a virtual data center using containers, virtual machines and Linux network stacks.
The virtual data center implements the aforementioned leaf-spine networks with BGP routers. To create the virtual data center, run the following commands on Ubuntu 22.04:
$ sudo curl -o /etc/apt/keyrings/docker-key.asc -fsSL https://download.docker.com/linux/ubuntu/gpg
$ echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker-key.asc] https://download.docker.com/linux/ubuntu jammy stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
$ sudo apt update
$ sudo apt install -y build-essential systemd-container lldpd qemu qemu-kvm socat picocom swtpm cloud-utils bird2 squid chrony dnsmasq jq freeipmi-tools unzip skopeo fakeroot docker-ce docker-ce-cli containerd.io
$ wget https://github.com/qemu/qemu/raw/master/pc-bios/bios.bin
$ wget https://github.com/qemu/qemu/raw/master/pc-bios/bios-256k.bin
$ sudo install -m 0644 -b bios.bin bios-256k.bin /usr/share/seabios/
$ wget https://github.com/cybozu-go/placemat/releases/download/v2.4.0/placemat2_2.4.0_amd64.deb
$ sudo dpkg -i ./placemat2_2.4.0_amd64.deb
$ git clone https://github.com/cybozu-go/neco
$ cd neco/dctest
$ make setup
$ make placemat
$ make test- To login a boot server in the virtual data center, run
./dcssh boot-0. - To stop and delete the virtual data center, run
make stop.
The setup commands above are examined on a GCP VM based on the Ubuntu 22.04 disk image. Some more setup steps are needed for the GCP environment.
- Enable nested virtualization by creating a custom image.
- Select a machine configuration with many vCPUs and much memory, e.g. n1-standard-32.
- Mount an additional local SSD scratch disk at
/var/scratch. - Stop and disable
systemd-resolved.serviceand prepareresolv.conf. - Install the latest Go and set
GOPATH.
Large tests
dctest directory contains test suites to run large tests using the virtual data center. Details are described in docs/dctest.md.
Middle/Small tests
Other than large tests, this repository contains smaller tests that can be run as follows:
- Setup the environment by
make setup. This is a one shot job. - Run etcd by
make start-etcd - Run tests by
go test -v -count 1 -race ./... - Stop etcd by
make stop-etcd
Directories
Other than Go packages, this repository has the following directories: