k8s-snowflake

Configs and scripts for bootstrapping an opinionated Kubernetes cluster anywhere.

Except it's my snowflake opinionated k8s distro :)

NOTE: current support is only for Azure and Google Cloud.

Table of Contents

• Provisioning
  • Base OS
  • Encrypted etcd secret data at rest
  • RBAC and Pod Security Policies
  • Container Runtime
  • Networking
• Azure
• Google Cloud
• Acknowledgements

Provisioning

These are opinionated scripts. If you don't like my opinions maybe consider using one of the hundred-thousand other tools for provisioning a cluster.

I literally made this because I didn't like the opinion of other things... so here we are. :P

I purposely tried to keep this as minimal and simple as possible from the OS base up.

Base OS

Every node uses Intel's Clear Linux as the base. This is for reasons of security and performance. If you would like to learn more on that you should click the link to their site.

Encrypted etcd secret data at rest

Data is encrypted with aescbc. You verify it's encrypted by following these instructions.

RBAC and Pod Security Policies

Kubernetes is installed with RBAC and is set up with a few roles and bindings that map to pod security policies.

There is a restricted pod security policy which does not allow running privileged pods and does not allow privilege escalation which is through the linux no_new_privs flag.

There is also a permissive pod security policy.

There are two cluster role bindings created (which grant permissions across namespaces):

• restricted: cannot create privileged pods, cannot escalate privileges, cannot run containers as root, cannot use the host network, IPC or PID namespace
• permissive: can create pods that are privileged and use the privileged pod security policy

Container Runtime

The cluster uses cri-containerd with runc as the container runtime.

Networking

The cluster uses cilium as a networking plugin. I like cilium because it uses BPF and XDP and their design is something I could wrap my head around. You should checkout their repo it's one of the cleanest implementations I have seen. You should checkout their really sweet BPF and XDP Reference Guide too!

Azure

Make sure you have the az tool installed. You can find instructions on downloading that here.

Make sure you are logged in.

To provision your cluster, clone this repo and run:

$ ./azure/setup.sh

The script automatically sets up an admin user with kubeconfig locally so you should be able to just run kubectl after!

NOTE: if you want to change the number of nodes, etc checkout the environment variables at the top of azure/setup.sh.

Google Cloud

Make sure you have the gcloud tool installed. You can find instructions on downloading that here.

Make sure you are logged in.

To provision your cluster, clone this repo and run:

$ VM_USER="your_ssh_user" ./gcloud/setup.sh

The script automatically sets up an admin user with kubeconfig locally so you should be able to just run kubectl after!

NOTE: if you want to change the number of nodes, etc checkout the environment variables at the top of gcloud/setup.sh.

Acknowledgements

Thanks to @kelseyhightower for kubernetes-the-hard-way which helped a lot of this.

If you are wondering why I didn't use something like cloud-init it's because Clear Linux has a pretty weirdly behaving version of cloud-init and I love bash, m'kay.