operator-framework/getting-started operator-framework/getting-started

  • This repository has been archived on 22/Sep/2020
  • Stars
    star
    535
  • Rank 82,940 (Top 2 %)
  • Language
  • Created over 6 years ago
  • Updated about 4 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
operator-framework/getting-started
github

operator-framework

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

An end-to-end scenario showing how to use the Operator Framework.

[DEPRECATED] Getting Started

This project is deprecated. Documentation for the Operator Framework has moved to https://operatorframework.io/

Overview

The Operator Framework (intro blog post) is an open source toolkit to manage Kubernetes native applications, called operators, in an effective, automated, and scalable way. Operators take advantage of Kubernetes's extensibility to deliver the automation advantages of cloud services like provisioning, scaling, and backup/restore while being able to run anywhere that Kubernetes can run.

This guide shows how to build a simple memcached operator and how to manage its lifecycle from install to update to a new version. For that, we will use two center pieces of the framework:

  • Operator SDK: Allows your developers to build an operator based on your expertise without requiring knowledge of Kubernetes API complexities.
  • Operator Lifecycle Manager: Helps you to install, update, and generally manage the lifecycle of all of the operators (and their associated services) running across your clusters.

Build an operator using the Operator SDK

BEFORE YOU BEGIN: links to the Operator SDK repo in this document are pinned to the master branch. Make sure you update the link such that it points to the correct Operator SDK repo version, which should match this repo's version or the operator-sdk version being used. For example, if you are using operator-sdk v0.12.0, update all links from this repo to the SDK repo with master -> v0.12.0. Otherwise you may see incorrect information.

The Operator SDK makes it easier to build Kubernetes native applications, a process that can require deep, application-specific operational knowledge. The SDK not only lowers that barrier, but it also helps reduce the amount of boilerplate code needed for many common management capabilities, such as metering or monitoring.

This section walks through an example of building a simple memcached operator using tools and libraries provided by the Operator SDK. This walkthrough is not exhaustive; for an in-depth explanation of these steps, see the SDK's user guide.

Requirements: Please make sure that the Operator SDK is installed on the development machine. Additionally, the Operator Lifecycle Manager must be installed in the cluster (1.8 or above to support the apps/v1beta2 API group) before running this guide.

Create a new project

  1. Use the CLI to create a new memcached-operator project:
$ mkdir -p $GOPATH/src/github.com/example-inc/
$ cd $GOPATH/src/github.com/example-inc/
$ export GO111MODULE=on
$ operator-sdk new memcached-operator
$ cd memcached-operator

This creates the memcached-operator project.

  1. Install dependencies by running go mod tidy

NOTE: Learn more about the project directory structure from the SDK project layout documentation.

Manager

The main program for the operator cmd/manager/main.go initializes and runs the Manager.

The Manager will automatically register the scheme for all custom resources defined under pkg/apis/... and run all controllers under pkg/controller/....

The Manager can restrict the namespace that all controllers will watch for resources:

mgr, err := manager.New(cfg, manager.Options{
	Namespace: namespace,
})

By default this will be the namespace that the operator is running in. To watch all namespaces leave the namespace option empty:

mgr, err := manager.New(cfg, manager.Options{
	Namespace: "",
})

Add a new Custom Resource Definition

Add a new Custom Resource Definition (CRD) API called Memcached, with APIVersion cache.example.com/v1alpha1 and Kind Memcached.

$ operator-sdk add api --api-version=cache.example.com/v1alpha1 --kind=Memcached

This will scaffold the Memcached resource API under pkg/apis/cache/v1alpha1/....

Define the Memcached spec and status

Modify the spec and status of the Memcached Custom Resource (CR) at pkg/apis/cache/v1alpha1/memcached_types.go:

type MemcachedSpec struct {
    // Size is the size of the memcached deployment
    Size int32 `json:"size"`
}
type MemcachedStatus struct {
    // Nodes are the names of the memcached pods 
    Nodes []string `json:"nodes"`
}

After modifying the *_types.go file always run the following command to update the generated code for that resource type:

$ operator-sdk generate k8s

Also run the following command in order to automatically generate the CRDs:

$ operator-sdk generate crds

You can see the changes applied in deploy/crds/cache.example.com_memcacheds_crd.yaml

Add a new Controller

Add a new Controller to the project that will watch and reconcile the Memcached resource:

$ operator-sdk add controller --api-version=cache.example.com/v1alpha1 --kind=Memcached

This will scaffold a new Controller implementation under pkg/controller/memcached/....

For this example replace the generated Controller file pkg/controller/memcached/memcached_controller.go with the example memcached_controller.go implementation.

The example Controller executes the following reconciliation logic for each Memcached CR:

  • Create a memcached Deployment if it doesn't exist
  • Ensure that the Deployment size is the same as specified by the Memcached CR spec
  • Update the Memcached CR status with the names of the memcached pods

The next two subsections explain how the Controller watches resources and how the reconcile loop is triggered. Skip to the Build section to see how to build and run the operator.

Resources watched by the Controller

Inspect the Controller implementation at pkg/controller/memcached/memcached_controller.go to see how the Controller watches resources.

The first watch is for the Memcached type as the primary resource. For each Add/Update/Delete event the reconcile loop will be sent a reconcile Request (a namespace/name key) for that Memcached object:

err := c.Watch(
    &source.Kind{Type: &cachev1alpha1.Memcached{}},
    &handler.EnqueueRequestForObject{},
  )

The next watch is for Deployments but the event handler will map each event to a reconcile Request for the owner of the Deployment. Which in this case is the Memcached object for which the Deployment was created. This allows the controller to watch Deployments as a secondary resource.

err := c.Watch(
    &source.Kind{Type: &appsv1.Deployment{}},
    &handler.EnqueueRequestForOwner{
        IsController: true,
        OwnerType:    &cachev1alpha1.Memcached{}},
    )

Reconcile loop

Every Controller has a Reconciler object with a Reconcile() method that implements the reconcile loop. The reconcile loop is passed the Request argument which is a Namespace/Name key used to lookup the primary resource object, Memcached, from the cache:

func (r *ReconcileMemcached) Reconcile(request reconcile.Request) (reconcile.Result, error) {
    // Lookup the Memcached instance for this reconcile request
    memcached := &cachev1alpha1.Memcached{}
    err := r.client.Get(context.TODO(), request.NamespacedName, memcached)
    ...
}

For a guide on Reconcilers, Clients, and interacting with resource Events, see the Client API doc.

Build and run the operator

Before running the operator, the CRD must be registered with the Kubernetes apiserver:

$ kubectl create -f deploy/crds/cache.example.com_memcacheds_crd.yaml

Once this is done, there are two ways to run the operator:

  • As a Deployment inside a Kubernetes cluster
  • As Go program outside a cluster

1. Run as a Deployment inside the cluster

Build the memcached-operator image and push it to your registry. The following example uses https://quay.io as the registry.

$ operator-sdk build quay.io/<user>/memcached-operator:v0.0.1
$ sed -i 's|REPLACE_IMAGE|quay.io/<user>/memcached-operator:v0.0.1|g' deploy/operator.yaml
$ docker push quay.io/<user>/memcached-operator:v0.0.1

Note If you are performing these steps on OSX, use the following sed command instead:

$ sed -i "" 's|REPLACE_IMAGE|quay.io/<user>/memcached-operator:v0.0.1|g' deploy/operator.yaml

The above command will replace the string REPLACE_IMAGE with the <image>:<tag> built above. Afterwards, verify that your operator.yaml file was updated successfully.

serviceAccountName: memcached-operator
containers:
- name: memcached-operator
  # Replace this with the built image name
  image: quay.io/<user>/memcached-operator:v0.0.1
  command:
  - memcached-operator
  imagePullPolicy: Always

IMPORTANT: Ensure that your cluster is able to pull the image pushed to your registry.

The Deployment manifest is generated at deploy/operator.yaml. Be sure to update the deployment image as shown above since the default is just a placeholder.

Setup RBAC and deploy the memcached-operator:

$ kubectl create -f deploy/service_account.yaml
$ kubectl create -f deploy/role.yaml
$ kubectl create -f deploy/role_binding.yaml
$ kubectl create -f deploy/operator.yaml

NOTE: To apply the RBAC you need to be logged in system:admin. (E.g. By using for OCP: oc login -u system:admin.)

Verify that the memcached-operator Deployment is up and running:

$ kubectl get deployment
NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
memcached-operator       1         1         1            1           1m

Verify that the memcached-operator pod is up and running:

$ kubectl get pod
NAME                                  READY     STATUS    RESTARTS   AGE
memcached-operator-7d76948766-nrcp7   1/1       Running   0          44s

IMPORTANT: Ensure that you built and pushed the image, and updated the operator.yaml file.

Verify that the operator is running successfully by checking its logs.

$ kubectl logs memcached-operator-7d76948766-nrcp7
{"level":"info","ts":1580855834.104447,"logger":"cmd","msg":"Operator Version: 0.0.1"}
{"level":"info","ts":1580855834.1044931,"logger":"cmd","msg":"Go Version: go1.13.6"}
{"level":"info","ts":1580855834.104505,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1580855834.1045163,"logger":"cmd","msg":"Version of operator-sdk: v0.15.1"}
{"level":"info","ts":1580855834.1049826,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1580855834.4423697,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1580855834.447401,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1580855834.7494223,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"}
{"level":"info","ts":1580855834.7497423,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1580855835.3955405,"logger":"metrics","msg":"Metrics Service object created","Service.Name":"memcached-operator-metrics","Service.Namespace":"default"}
{"level":"info","ts":1580855835.7000446,"logger":"cmd","msg":"Could not create ServiceMonitor object","error":"no ServiceMonitor registered with the API"}
{"level":"info","ts":1580855835.7005095,"logger":"cmd","msg":"Install prometheus-operator in your cluster to create ServiceMonitor objects","error":"no ServiceMonitor registered with the API"}
{"level":"info","ts":1580855835.7007008,"logger":"cmd","msg":"Starting the Cmd."}
{"level":"info","ts":1580855835.7014875,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1580855835.702304,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"memcached-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1580855835.803201,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"memcached-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1580855835.9041016,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"memcached-controller"}
{"level":"info","ts":1580855835.9044445,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"memcached-controller","worker count":1}

The following error will occur if your cluster was unable to pull the image:

$ kubectl get pod
NAME                                  READY     STATUS             RESTARTS   AGE
memcached-operator-6b5dc697fb-t62cv   0/1       ImagePullBackOff   0          2m

Following the logs in the error scenario described above.

$ kubectl logs memcached-operator-6b5dc697fb-t62cv
Error from server (BadRequest): container "memcached-operator" in pod "memcached-operator-6b5dc697fb-t62cv" is waiting to start: image can't be pulled

NOTE: Just for tests purposes make the image public and setting up the cluster to allow use insecure registry. ( E.g --insecure-registry 172.30.0.0/16 )

2. Run locally outside the cluster

This method is preferred during development cycle to deploy and test faster.

Run the operator locally with the default kubernetes config file present at $HOME/.kube/config:

$ operator-sdk run local
INFO[0000] Running the operator locally; watching namespace "default"
{"level":"info","ts":1593777657.892013,"logger":"cmd","msg":"Operator Version: 0.0.1"}
{"level":"info","ts":1593777657.892079,"logger":"cmd","msg":"Go Version: go1.14.4"}
{"level":"info","ts":1593777657.892084,"logger":"cmd","msg":"Go OS/Arch: darwin/amd64"}
{"level":"info","ts":1593777657.892087,"logger":"cmd","msg":"Version of operator-sdk: v0.18.2"}
...

You can use a specific kubeconfig via the flag --kubeconfig=<path/to/kubeconfig>.

Create a Memcached CR

Create the example Memcached CR that was generated at deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml:

$ cat deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml
apiVersion: "cache.example.com/v1alpha1"
kind: "Memcached"
metadata:
  name: "example-memcached"
spec:
  size: 3

$ kubectl apply -f deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml

Ensure that the memcached-operator creates the deployment for the CR:

$ kubectl get deployment
NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
memcached-operator       1         1         1            1           2m
example-memcached        3         3         3            3           1m

Check the pods and CR status to confirm the status is updated with the memcached pod names:

$ kubectl get pods
NAME                                  READY     STATUS    RESTARTS   AGE
example-memcached-6fd7c98d8-7dqdr     1/1       Running   0          1m
example-memcached-6fd7c98d8-g5k7v     1/1       Running   0          1m
example-memcached-6fd7c98d8-m7vn7     1/1       Running   0          1m
memcached-operator-7cc7cfdf86-vvjqk   1/1       Running   0          2m
$ kubectl get memcached/example-memcached -o yaml
apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  clusterName: ""
  creationTimestamp: 2018-03-31T22:51:08Z
  generation: 0
  name: example-memcached
  namespace: default
  resourceVersion: "245453"
  selfLink: /apis/cache.example.com/v1alpha1/namespaces/default/memcacheds/example-memcached
  uid: 0026cc97-3536-11e8-bd83-0800274106a1
spec:
  size: 3
status:
  nodes:
  - example-memcached-6fd7c98d8-7dqdr
  - example-memcached-6fd7c98d8-g5k7v
  - example-memcached-6fd7c98d8-m7vn7

Update the size

Change the spec.size field in the memcached CR from 3 to 4 and apply the change:

$ cat deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml
apiVersion: "cache.example.com/v1alpha1"
kind: "Memcached"
metadata:
  name: "example-memcached"
spec:
  size: 4

$ kubectl apply -f deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml

Confirm that the operator changes the deployment size:

$ kubectl get deployment
NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
example-memcached    4         4         4            4           5m

Cleanup

Delete the operator and its related resources:

$ kubectl delete -f deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml
$ kubectl delete -f deploy/operator.yaml
$ kubectl delete -f deploy/role_binding.yaml
$ kubectl delete -f deploy/role.yaml
$ kubectl delete -f deploy/service_account.yaml
$ kubectl delete -f deploy/crds/cache.example.com_memcacheds_crd.yaml

Reference implementation

The above walkthrough follows a similar implementation process to the one used to produce the memcached-operator in the SDK samples repo.

Manage the operator using the Operator Lifecycle Manager

NOTE: This section of the Getting Started Guide is out-of-date. We're working on some improvements to Operator SDK to streamline the experience of using OLM. For further information see, for example, this enhancement proposal. In the meantime, you might find the following documentation helpful:

The previous section has covered manually running an operator. In the next sections, we will explore using the Operator Lifecycle Manager (OLM) which is what enables a more robust deployment model for operators being run in production environments.

OLM helps you to install, update, and generally manage the lifecycle of all of the operators (and their associated services) on a Kubernetes cluster. It runs as an Kubernetes extension and lets you use kubectl for all the lifecycle management functions without any additional tools.

NOTE: Various public, OLM-ready operator projects are available at operatorhub.io.

Generate an operator manifest

The first step to leveraging OLM is to create a Cluster Service Version (CSV) manifest. An operator manifest describes how to display, create and manage the application, in this case memcached, as a whole. It is required for OLM to function.

The Operator SDK CLI can generate CSV manifests via the following command:

$ operator-sdk generate csv --csv-version 0.0.1 --update-crds

Several fields must be updated after generating the CSV. See the CSV generation doc for a list of required fields, and the memcached-operator CSV for an example of a complete CSV.

NOTE: You are able to preview and validate your CSV manifest syntax in the operatorhub.io CSV Preview tool.

Testing locally

The next step is to ensure your project deploys correctly with OLM and runs as expected. Follow this testing guide to deploy and test your operator.

NOTE: Also, check out some of the new OLM integrations in operator-sdk:

Promoting operator standards

We recommend running operator-sdk scorecard against your operator to see whether your operator's OLM integration follows best practices. For further information on running the scorecard and results, see the scorecard documentation.

NOTE: the scorecard is undergoing changes to give informative and helpful feedback. The original scorecard functionality will still be available while and after changes are made.

Conclusion

Hopefully, this guide was an effective demonstration of the value of the Operator Framework for building and managing operators. There is much more that we left out in the interest of brevity. The Operator Framework and its components are open source, so please feel encouraged to jump into each individually and learn what else you can do. If you want to discuss your experience, have questions, or want to get involved, join the Operator Framework mailing list.

More Repositories

1

operator-sdk

SDK for building Kubernetes applications. Provides high level APIs, useful abstractions, and project scaffolding.
Go
6,553
star
2

awesome-operators

A resource tracking a number of Operators out in the wild.
3,471
star
3

operator-lifecycle-manager

A management framework for extending Kubernetes with Operators
Go
1,710
star
4

java-operator-sdk

Java SDK for building Kubernetes Operators
Java
794
star
5

community-operators

The canonical source for Kubernetes Operators that appear on OperatorHub.io, OpenShift Container Platform and OKD.
Dockerfile
419
star
6

operator-sdk-samples

Contains samples of operators built using the operator-sdk
Shell
237
star
7

operator-registry

Operator Registry runs in a Kubernetes or OpenShift cluster to provide operator catalog data to Operator Lifecycle Manager.
Go
188
star
8

helm-app-operator-kit

Tool for creating a Kubernetes Operator from a Helm Chart
Shell
132
star
9

operator-marketplace

Marketplace Operator
Go
109
star
10

kubectl-operator

Manage Kubernetes Operators from the command line
Go
68
star
11

helm-operator-plugins

Experimental refactoring of the operator-framework's helm operator
Go
47
star
12

rukpak

RukPak runs in a Kubernetes cluster and defines APIs for installing cloud native content
Go
41
star
13

operator-lib

This is a library to help Operator developers
Go
40
star
14

operatorhub.io

TypeScript
39
star
15

josdk-spring-boot-starter

Java
36
star
16

api

Contains the API definitions used by OLM and Marketplace
Go
29
star
17

java-operator-plugins

Operator SDK Java Plugin
Go
26
star
18

community

Community organizational documentations and process for Operator Framework
25
star
19

olm-book

Documentation for OLM
25
star
20

operator-sdk-ansible-util

A collection of Ansible assets for use with Ansible-based operators built with the operator-sdk.
Python
18
star
21

catalogd

On-cluster FBC catalog content server
Go
14
star
22

operator-manifests

Artisanally packaged Kubernetes Operators
Go
14
star
23

kubernetes-webooks-framework

Framework and tooling to support writing admission controllers and conversion hooks for Kubernetes in Java
Java
12
star
24

enhancements

10
star
25

audit

audit operator bundles and catalogs, producing a report.
Go
10
star
26

deppy

Deppy: The dependency resolver for Kubernetes
Go
10
star
27

olm-docs

Hugo doc site for https://github.com/operator-framework/operator-lifecycle-manager
SCSS
10
star
28

java-controller-runtime

Repo for the Java controller-runtime.
Java
8
star
29

combo

Go
8
star
30

olm-broker

Service Catalog integration for OLM
Go
8
star
31

ansible-operator-plugins

Experimental extraction/refactoring of the Operator SDK's ansible operator plugin
Go
7
star
32

operator-manifest-tools

Tooling that enables software release pipelines for operator manifests
Go
7
star
33

operator-controller

Go
7
star
34

cool-catalog

Repository that showcases example workflows for building OLM Catalogs via Declarative Configurations
Dockerfile
6
star
35

oria-operator

Go
5
star
36

index-mermaid-graph

a tool that produces SVG diagrams using OLM graphs as the input
Go
5
star
37

go-appr

Go client bindings for App Registry
5
star
38

operator-framework.io

Website for the framework at https://operatorframework.io
HTML
4
star
39

mock-extension-apiserver

Go
3
star
40

bundle-example

Dockerfile
3
star
41

community-operator-catalog

Community Operator Catalog
Shell
3
star
42

logos

2
star
43

tekton-scorecard-image

a SDK Scorecard custom image for the tekton operator
Java
1
star
44

tilt-support

Library of shared Tilt helpers
Starlark
1
star
45

test-infra

1
star
46

flake-analyzer

Go
1
star