• Stars
    star
    418
  • Rank 103,620 (Top 3 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created almost 7 years ago
  • Updated about 1 month ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Data Import Service for kubernetes, designed with kubevirt in mind.

Containerized Data Importer

Go Report Card Coverage Status Licensed under Apache License version 2.0

Containerized-Data-Importer (CDI) is a persistent storage management add-on for Kubernetes. It's primary goal is to provide a declarative way to build Virtual Machine Disks on PVCs for Kubevirt VMs

CDI works with standard core Kubernetes resources and is storage device agnostic, while its primary focus is to build disk images for Kubevirt, it's also useful outside of a Kubevirt context to use for initializing your Kubernetes Volumes with data.

Introduction

Kubernetes extension to populate PVCs with VM disk images or other data

CDI provides the ability to populate PVCs with VM images or other data upon creation. The data can come from different sources: a URL, a container registry, another PVC (clone), or an upload from a client.

DataVolumes

CDI includes a CustomResourceDefinition (CRD) that provides an object of type DataVolume. The DataVolume is an abstraction on top of the standard Kubernetes PVC and can be used to automate creation and population of a PVC with data. Although you can use PVCs directly with CDI, DataVolumes are the preferred method since they offer full functionality, a stable API, and better integration with kubevirt. More details about DataVolumes can be found here.

Import from URL

This method is selected when you create a DataVolume with an http source. CDI will populate the volume using a pod that will download from the given URL and handle the content according to the contentType setting (see below). It is possible to configure basic authentication using a secret and specify custom TLS certificates in a ConfigMap.

Import from container registry

When a DataVolume has a registry source CDI will populate the volume with a Container Disk downloaded from the given image URL. The only valid contentType for this source is kubevirt and the image must be a Container Disk. More details can be found here.

Clone another PVC

To clone a PVC, create a DataVolume with a pvc source and specify namespace and name of the source PVC. CDI will attempt an efficient clone of the PVC using the storage backend if possible. Otherwise, the data will be transferred to the target PVC using a TLS secured connection between two pods on the cluster network. More details can be found here.

Upload from a client

To upload data to a PVC from a client machine first create a DataVolume with an upload source. CDI will prepare to receive data via an upload proxy which will transit data from an authenticated client to a pod which will populate the PVC according to the contentType setting. To send data to the upload proxy you must have a valid UploadToken. See the upload documentation for details.

Prepare an empty Kubevirt VM disk

The special source blank can be used to populate a volume with an empty Kubevirt VM disk. This source is valid only with the kubevirt contentType. CDI will create a VM disk on the PVC which uses all of the available space. See here for an example.

Import from oVirt

Virtual machine disks can be imported from a running oVirt installation using the imageio source. CDI will use the provided credentials to securely transfer the indicated oVirt disk image so that it can be used with kubevirt. See here for more information and examples.

Import from VMware

Disks can be imported from VMware with the vddk source. CDI will transfer the disks using vCenter/ESX API credentials and a user-provided image containing the non-redistributable VDDK library. See here for instructions.

Content Types

CDI features specialized handling for two types of content: Kubevirt VM disk images and tar archives. The kubevirt content type indicates that the data being imported should be treated as a Kubevirt VM disk. CDI will automatically decompress and convert the file from qcow2 to raw format if needed. It will also resize the disk to use all available space. The archive content type indicates that the data is a tar archive. Compression is not yet supported for archives. CDI will extract the contents of the archive into the volume. The content type can be selected by specifying the contentType field in the DataVolume. kubevirt is the default content type. CDI only supports certain combinations of source and contentType as indicated below:

  • http β†’ kubevirt, archive
  • registry β†’ kubevirt
  • pvc β†’ Not applicable - content is cloned
  • upload β†’ kubevirt
  • imageio β†’ kubevirt
  • vddk β†’ kubevirt

Deploy it

Deploying the CDI controller is straightforward. In this document the default namespace is used, but in a production setup a protected namespace that is inaccessible to regular users should be used instead.

$ export VERSION=$(curl -s https://api.github.com/repos/kubevirt/containerized-data-importer/releases/latest | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')
$ kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml
$ kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-cr.yaml

Use it

Create a DataVolume and populate it with data from an http source

$ kubectl create -f https://raw.githubusercontent.com/kubevirt/containerized-data-importer/$VERSION/manifests/example/import-kubevirt-datavolume.yaml

There are quite a few examples in the example manifests, check them out as a reference to create DataVolumes from additional sources like registries, S3, GCS and your local system.

Hack it

CDI includes a self contained development and test environment. We use Docker to build, and we provide a simple way to get a test cluster up and running on your laptop. The development tools include a version of kubectl that you can use to communicate with the cluster. A wrapper script to communicate with the cluster can be invoked using ./cluster-up/kubectl.sh.

$ mkdir $GOPATH/src/kubevirt.io && cd $GOPATH/src/kubevirt.io
$ git clone https://github.com/kubevirt/containerized-data-importer && cd containerized-data-importer
$ make cluster-up
$ make cluster-sync
$ ./cluster-up/kubectl.sh .....

For development on external cluster (not provisioned by our CI), check out the external provider.

Storage notes

CDI is designed to be storage agnostic. Since it works with the kubernetes storage APIs it should work well with any configuration that can produce a Bound PVC. The following are storage-specific notes that may be relevant when using CDI.

  • NFSv3 is not supported: CDI uses qemu-img to manipulate disk images and this program uses locking which is not compatible with the obsolete NFSv3 protocol. We recommend using NFSv4.

Connect with us

We'd love to hear from you, reach out on Github via Issues or Pull Requests!

Hit us up on Slack

Shoot us an email at: [email protected]

More details

  1. Hacking details
  2. Design docs
  3. Kubevirt documentation

More Repositories

1

kubevirt

Kubernetes Virtualization API and runtime in order to define and manage virtual machines.
Go
5,411
star
2

hyperconverged-cluster-operator

Operator pattern for managing multi-operator products
Go
148
star
3

demo

Easy to use KubeVirt demo based on minikube.
Shell
108
star
4

kubevirt-ansible

Set of Ansible roles & playbooks for KubeVirt deployment
Go
91
star
5

common-templates

Templates for running VMs on KubeVirt
Shell
80
star
6

kubevirtci

Contains cluster definitions and client tools to quickly spin up and destroy ephemeral and scalable k8s and ocp clusters for testing
Shell
79
star
7

hostpath-provisioner

Go
78
star
8

cloud-provider-kubevirt

Kubernetes cloud-provider for KubeVirt
Go
72
star
9

cluster-network-addons-operator

Deploy additional networking components on top of your Kubernetes cluster
Go
71
star
10

user-guide

This user guide will walk you through installation and various features.
Makefile
63
star
11

hostpath-provisioner-operator

Go
52
star
12

community

Community content
Python
49
star
13

node-maintenance-operator

A Kubernetes Operator to manage node maintenance through NodeMaintenance custom resources
Go
43
star
14

kubernetes-device-plugins

Incubating: A number of Kubernetes device-plugins developed and useful for KubeVirt
Go
43
star
15

macvtap-cni

A CNI + device plugin combo for virtualization workloads on Kubernetes.
Go
42
star
16

csi-driver

KubeVirt CSI driver
Go
39
star
17

libvirt

DEPRECATED Vanilla dockerized libvirt image, used as a base for kubevirt
Shell
39
star
18

client-go

Go client for kubevirt
Go
38
star
19

kubevirt-tutorial

Demo that guides users through an end to end KubeVirt experience.
HCL
30
star
20

kubevirt.github.io

KubeVirt website repo, documentation at https://kubevirt.io/user-guide/
HTML
30
star
21

kubevirt-tekton-tasks

VM specific tasks for Tekton Pipelines
Go
29
star
22

ssp-operator

Operator that deploys additional KubeVirt resources
Go
29
star
23

device-plugin-manager

Incubating: A framework for writing Kubernetes Device Plugins
Go
29
star
24

kubevirt-velero-plugin

Plugin to Velero which automates backing up and restoring KubeVirt/CDI objects
Go
28
star
25

client-python

An auto-generated python SDK to interact with KubeVirt resources
Python
28
star
26

terraform-provider-kubevirt

Terraform provider for Kubevirt platform
Go
27
star
27

containerdisks

Officially built and maintained containerdisks of the KubeVirt project
Go
25
star
28

project-infra

Project infrastructure administrative tools
Go
25
star
29

monitoring

KubeVirt monitoring dashboards
Go
24
star
30

common-instancetypes

Instancetypes and preferences for running VMs on KubeVirt
Go
22
star
31

vm-import-operator

Operator for importing VMs from RHV to KubeVirt
Go
16
star
32

vAdvisor

VM monitoring application based on WSGI, libvirt, flask and gevent, inspired by cAdvisor.
Python
14
star
33

vmctl

Use KubeVirt VMs with Deployments, Replica Sets, …
Go
13
star
34

must-gather

A client tool for gathering KubeVirt informations in a Kubernetes or OpenShift cluster
Shell
12
star
35

web-ui-operator

Operator for web-ui
Go
12
star
36

kubevirt.core

Lean Ansible bindings for KubeVirt
Python
11
star
37

api

KubeVirt API definition
Go
11
star
38

kubectl-virt-plugin

Holds all scripts to create packages and manifest file required for publishing the virtctl binary as a krew package for kubectl.
Shell
11
star
39

machine-remediation

Incubating: Deploy all components to recognize unhealthy machines and apply different remediation strategies on it
Go
10
star
40

api-reference

API documentation
HTML
8
star
41

web-ui-components

https://kubevirt.io/web-ui-components
JavaScript
8
star
42

bridge-marker

Shell
7
star
43

managed-tenant-quota

Go
7
star
44

kubevirt-ssp-operator

Shell
7
star
45

ansible-kubevirt-modules

Ansible modules for managing KubeVirt objects in Kubernetes clusters
Shell
7
star
46

kubesecondarydns

DNS for KubeVirt VirtualMachines secondary interfaces
Go
7
star
47

cluster-api-provider-external

The external provider implementation of the Cluster Management API.
Go
6
star
48

vm-console-proxy

Proxy for VM console connection.
Go
6
star
49

cloud-image-builder

Scripts and playbooks for creating and testing cloud images containing Kubernetes and KubeVirt
Shell
6
star
50

ci-health

Metrics about CI performance in repositories using Prow
Go
5
star
51

run

This repository is backing run.kubevirt.io.
Shell
5
star
52

controller-lifecycle-operator-sdk

Go
4
star
53

application-aware-quota

Empower users to customize and manage resource quotas per component in Kubernetes, utilizing the scheduling gates feature.
Go
4
star
54

katacoda-scenarios

Training scenarios for https://katacoda.com/kubevirt
Shell
4
star
55

v2v-job

A job to allow importing a VM from a support virt-v2v source into KubeVirt.
Shell
4
star
56

kubevirt-operator

Incubating: The KubeVirt Operator
Go
4
star
57

tekton-tasks-operator

Operator which deploys kubevirt-tekton-tasks in cluster
Go
3
star
58

kubevirt-template-validator

validating webhook to check templates sent to the cluster.
Go
3
star
59

containerized-data-importer-api

Containerized Data Importer API definitions
Go
3
star
60

web-ui-design

Images and Markdown documenting proposed user workflows for the UI
HTML
3
star
61

qe-tools

Different Ginkgo reporters for QE purposes
Go
3
star
62

test-infra-containers

All containers which are useful for testing (tgtd, cinder, ceph, gluster, ...)
Shell
2
star
63

cdi-operator

Makefile
2
star
64

k8s-seccomp-generator

Generates seccomp profile for kubernetes pods
Go
2
star
65

test-benchmarks

setup to create reproducible benchmarks tests with containers and KubeVirt VMs
Go
2
star
66

cockpit-demo

This is a demonstration of how KubeVirt could be displayed in Cockpit
JavaScript
2
star
67

libguestfs-appliance

Builds libguestfs appliances and container images to be consumed by other kubevirt projects
Dockerfile
2
star
68

cdi-api-reference

Containerized Data Importer API Reference
HTML
1
star
69

sig-release

A repo to hold and organize all the information required for KubeVirt releases
1
star
70

.github

Default Community health files.
1
star
71

kvm-info-nfd-plugin

Plugin for Kubernetes Node Feature Discovery for high(er) level KVM featues
C
1
star
72

node-labeller

Node-Labeller is deprecated and is moved to kubevirt repository
Go
1
star