Cloud Foundry Container Runtime

A BOSH release for Kubernetes. Formerly named kubo.

• Slack: #cfcr on https://slack.cloudfoundry.org
• Pivotal Tracker: https://www.pivotaltracker.com/n/projects/2093412

Table of Contents

• Prerequisites
  • Hardware Requirements
• Deploying CFCR
  • Configuring CFCR
  • Using Proxy with CFCR
  • BOSH Lite
• Accessing the CFCR Cluster with kubectl
• Backup & Restore
• Monitoring
• DNS
• Deprecations
  • Deployment scripts and docs
  • Heapster

Prerequisites

• A BOSH Director configured with UAA, Credhub, and BOSH DNS runtime config. We recommend using BOSH Bootloader for this.

• Latest kubo-deployment tarball

• Accessing the master:

  • Single Master: Set up a DNS name pointing to your master's IP address
  • Multiple Masters: A TCP load balancer for your master nodes.
    • Use a TCP load balancer configured to connect to the master nodes on port 8443.
    • Add healthchecks using either a TCP dial or HTTPS by looking for a 200 OK response from /healthz.
    • if you have used BOSH Bootloader on GCP then you need to manually create a firewall rule. Allow access to port TCP 8443 to VMs in your BBL network tagged cfcr-master from your load balancer's IP.

• Cloud Config with

  • vm_types named minimal, small, and small-highmem (See cf-deployment for reference)
  • network named default
  • three availability zones azs named z1,z2,z3

  Note: the cloud-config properties can be customized by applying ops-files. See manifests/ops-files for some examples.

  If using loadbalancers then apply the vm_extension called cfcr-master-loadbalancer to the cloud-config to add the instances to your loadbalancers. See BOSH documentation for information on how to configure loadbalancers.

Hardware Requirements

Kubernetes uses etcd as its datastore. The official infrastructure requirements and example configurations for the etcd cluster can be found here.

Deploying CFCR

1. Upload the latest Xenial stemcell to the director.

2. Untar the kubo-deployment tarball and rename it kubo-deployment

3. Deploy

   Option 1. Single Master

   cd kubo-deployment
   
   bosh deploy -d cfcr manifests/cfcr.yml \
     -o manifests/ops-files/misc/single-master.yml \
     -o manifests/ops-files/add-hostname-to-master-certificate.yml \
     -v api-hostname=[DNS-NAME]

   Option 2. Three Masters

   cd kubo-deployment
   
   bosh deploy -d cfcr manifests/cfcr.yml \
     -o manifests/ops-files/add-vm-extensions-to-master.yml \
     -o manifests/ops-files/add-hostname-to-master-certificate.yml \
     -v api-hostname=[LOADBALANCER-ADDRESS]

   Note: Loadbalancer address should be the external address (hostname or IP) of the loadbalancer you have configured.

   Check additional configurations, such as setting Kubernetes cloud provider, in docs.

4. Add Kubernetes system components

   bosh -d cfcr run-errand apply-specs

5. Run the following to confirm the cluster is operational

   bosh -d cfcr run-errand smoke-tests

Configuring CFCR

Please check out our manifest and ops-files in kube-deployment for examples on how to configure kubo-release. Additionally, we have a doc page to describe how to configure Kubernetes components for the release.

CFCR can be deployed with Pod Security Policies. Check for more details in the doc

Configuring Proxy for CFCR

CFCR allows you to configure proxy for all components. Check recommendations for no proxy settings first.

BOSH Lite

CFCR clusters on BOSH Lite are intended for development. We run the deploy_cfcr_lite script to provision a cluster with the latest stemcell and master of kubo-release. This requires that the cloned kubo-release repository can be found from cd ../kubo-release from within the kubo-deployment directory.

cd kubo-deployment
./bin/deploy_cfcr_lite

Accessing the CFCR Cluster with kubectl

1. Login to the Credhub Server that stores the cluster's credentials:

   credhub login
   

2. Find the director name by running

   bosh env
   

3. Configure the kubeconfig for your kubectl client:

   cd kubo-deployment
   
   ./bin/set_kubeconfig <DIRECTOR_NAME>/cfcr https://[DNS-NAME-OR-LOADBALANCER-ADDRESS]:8443
   

Backup & Restore

We use BBR to perform backups and restores of the etcd node within a CFCR cluster, for both single and three master deployments. Our backup currently takes an etcd snapshot without interruptions to the cluster. However, for restore we take both the kube-apiserver and etcd offline to restore the cluster with the specified snapshot. Restore is a destructive operation that will completely overwrite any existing data on the cluster. For a closer look at the bbr scripts, check out:

• cfcr-etcd-release
• kubo-release

To run the bbr cli against a CFCR cluster, follow the steps under "BOSH Deployment" on the BBR documentation page.

Monitoring

Follow the recommendations in etcd's documentation for monitoring etcd metrics.

DNS

By default CFCR runs with CoreDNS in preference of Kube-DNS.

If you are migrating from an earlier version of CFCR, Kube-DNS can be removed by running:

kubectl delete deployment -n kube-system kube-dns

You may notice that a kube-dns service remains, this is also required by the CoreDNS spec.

Deprecations

Deployment scripts and docs

CFCR had a set of scripts, including deploy_bosh and deploy_k8s, that were the primary mechanism we supported to deploy BOSH and Kubernetes clusters. We no longer support these and have removed the corresponding documentation from https://docs-cfcr.cfapps.io

The BOSH oriented method documented in this README.md is the supported method to deploy Kubernetes clusters with CFCR.

Heapster

K8s 1.11 release kicked off the deprecation timeline for the Heapster component, see here for more info. As a result, we're in the process of replacing Heapster with Metrics Server in the upcoming releases of kubo-release.

Heapster can be removed by running:

kubectl delete deployment -n kube-system heapster