Docker Machine Driver for Proxmox VE

This driver can be used to kickstart a VM in Proxmox VE to be used with Docker/Docker Machine.

Download and copy it into your PATH (don't forget to chmod +x) or build your own driver

Check if it works:

  $ docker-machine create --driver proxmoxve --help | grep -c proxmox
  35

Operation

Now you have two modes of operation:

use an iso to install a Docker distribution (e.g. RancherOS)
use a previously created cloud-init-based image VM template as a base

There are also other options to customize your VM which are not shown here, so please feel free to explore them with docker-machine create --driver proxmoxve --help

Preparing a special test user in PVE

If you want to test this docker-machine driver, i strongly recommend to secure it properly. Best way to do this to create a special user that has its own pool and storage for creating the test machines. This corresponds to the examples below.

Here is what I use (based on ZFS):

create a pool for use as --proxmoxve-proxmox-pool docker-machine
```
  pvesh create /pools -poolid docker-machine
```

create an user docker-machine with password D0ck3rS3cr3t

  pvesh create /access/users -userid docker-machine@pve -password D0ck3rS3cr3t

creating a special ZFS dataset and use it as PVE storage

  zfs create -o refquota=50G rpool/docker-machine-test
  zfs create rpool/docker-machine-test/iso
  pvesh create /storage -storage docker-machine -type zfspool -pool rpool/docker-machine-test
  pvesh create /storage -storage docker-machine-iso -type dir -path /rpool/docker-machine-test/iso -content iso
  pvesh set /pools/docker-machine -storage docker-machine
  pvesh set /pools/docker-machine -storage docker-machine-iso

set proper permissions for the user

  pvesh set /access/acl -path /pool/docker-machine -roles PVEVMAdmin,PVEDatastoreAdmin,PVEPoolAdmin -users docker-machine@pve

If you have additional test storages, you can also add them easily:

    pvesh set /pools/docker-machine -storage nfs
    pvesh set /pools/docker-machine -storage lvm
    pvesh set /pools/docker-machine -storage directory

Ceph is currently not directly tested by me, but there are fine people out there wo tried it.

Clone VM

This approach uses a predefined VM template with cloud-init support to be cloned and used. There a lot of ways to do that, here is an adopted one (courtesy of @travisghansen):

#!/bin/bash

set -x
set -e

export IMGID=9007
export BASE_IMG="debian-10-openstack-amd64.qcow2"
export IMG="debian-10-openstack-amd64-${IMGID}.qcow2"
export STORAGEID="docker-machine"

if [ ! -f "${BASE_IMG}" ];then
  wget https://cloud.debian.org/images/cloud/OpenStack/current-10/debian-10-openstack-amd64.qcow2
fi

if [ ! -f "${IMG}" ];then
  cp -f "${BASE_IMG}" "${IMG}"
fi

# prepare mounts
guestmount -a ${IMG} -m /dev/sda1 /mnt/tmp/
mount --bind /dev/ /mnt/tmp/dev/
mount --bind /proc/ /mnt/tmp/proc/

# get resolving working
mv /mnt/tmp/etc/resolv.conf /mnt/tmp/etc/resolv.conf.orig
cp -a --force /etc/resolv.conf /mnt/tmp/etc/resolv.conf

# install desired apps
chroot /mnt/tmp /bin/bash -c "apt-get update"
chroot /mnt/tmp /bin/bash -c "DEBIAN_FRONTEND=noninteractive apt-get install -y net-tools curl qemu-guest-agent nfs-common open-iscsi lsscsi sg3-utils multipath-tools scsitools"

# https://www.electrictoolbox.com/sshd-hostname-lookups/
sed -i 's:#UseDNS no:UseDNS no:' /mnt/tmp/etc/ssh/sshd_config

sed -i '/package-update-upgrade-install/d' /mnt/tmp/etc/cloud/cloud.cfg

cat > /mnt/tmp/etc/cloud/cloud.cfg.d/99_custom.cfg << '__EOF__'
#cloud-config

# Install additional packages on first boot
#
# Default: none
#
# if packages are specified, this apt_update will be set to true
#
# packages may be supplied as a single package name or as a list
# with the format [<package>, <version>] wherein the specifc
# package version will be installed.
#packages:
# - qemu-guest-agent
# - nfs-common

ntp:
  enabled: true

# datasource_list: [ NoCloud, ConfigDrive ]
__EOF__

cat > /mnt/tmp/etc/multipath.conf << '__EOF__'
defaults {
    user_friendly_names yes
    find_multipaths yes
}
__EOF__

# enable services
chroot /mnt/tmp systemctl enable open-iscsi.service || true
chroot /mnt/tmp systemctl enable multipath-tools.service || true

# restore systemd-resolved settings
mv /mnt/tmp/etc/resolv.conf.orig /mnt/tmp/etc/resolv.conf

# umount everything
umount /mnt/tmp/dev
umount /mnt/tmp/proc
umount /mnt/tmp

# create template
qm create ${IMGID} --memory 512 --net0 virtio,bridge=vmbr0
qm importdisk ${IMGID} ${IMG} ${STORAGEID} --format qcow2
qm set ${IMGID} --scsihw virtio-scsi-pci --scsi0 ${STORAGEID}:vm-${IMGID}-disk-0
qm set ${IMGID} --ide2 ${STORAGEID}:cloudinit
qm set ${IMGID} --boot c --bootdisk scsi0
qm set ${IMGID} --serial0 socket --vga serial0
qm template ${IMGID}

# set host cpu, ssh key, etc
qm set ${IMGID} --scsihw virtio-scsi-pci
qm set ${IMGID} --cpu host
qm set ${IMGID} --agent enabled=1
qm set ${IMGID} --autostart
qm set ${IMGID} --onboot 1
qm set ${IMGID} --ostype l26
qm set ${IMGID} --ipconfig0 "ip=dhcp"

Adapt to fit your needs and run it on your Proxmox VE until it works without any problems and creates a template in your Proxmox VE. You may need to install libguestfs-tools.

After the image is created, you can start to use the machine driver to create new VMs:

#!/bin/sh
set -ex

export PATH=$PWD:$PATH

PVE_NODE="proxmox"
PVE_HOST="192.168.1.5"

PVE_USER="docker-machine"
PVE_REALM="pve"
PVE_PASSWD="D0ck3rS3cr3t"

PVE_STORAGE_NAME="${1:-docker-machine}"
PVE_POOL="docker-machine"

VM_NAME="docker-clone"

docker-machine rm --force $VM_NAME >/dev/null 2>&1 || true

docker-machine --debug \
    create \
    --driver proxmoxve \
    --proxmoxve-proxmox-host $PVE_HOST \
    --proxmoxve-proxmox-node $PVE_NODE \
    --proxmoxve-proxmox-user-name $PVE_USER \
    --proxmoxve-proxmox-user-password $PVE_PASSWD \
    --proxmoxve-proxmox-realm $PVE_REALM \
    --proxmoxve-proxmox-pool $PVE_POOL \
    \
    --proxmoxve-provision-strategy clone \
    --proxmoxve-ssh-username 'debian' \
    --proxmoxve-ssh-password 'geheim' \
    --proxmoxve-vm-clone-vmid 9007 \
    \
    --proxmoxve-debug-resty \
    --proxmoxve-debug-driver \
    \
    $* \
    \
    $VM_NAME

eval $(docker-machine env $VM_NAME)

docker ps

Rancher OS

Use a recent, e.g. 1.5.6 version of RancherOS and copy the rancheros-proxmoxve-autoformat.iso to your iso image storage on your PVE
Create a script with the following contents and adapt to your needs:

PVE_NODE="proxmox-docker-machine"
PVE_HOST="192.168.1.10"

PVE_USER="docker-machine"
PVE_REALM="pve"
PVE_PASSWD="D0ck3rS3cr3t"

PVE_STORAGE_NAME="docker-machine"
PVE_STORAGE_SIZE="4"
PVE_POOL="docker-machine"

SSH_USERNAME="docker"
SSH_PASSWORD="tcuser"

PVE_MEMORY=2
PVE_CPU_CORES=4
PVE_IMAGE_FILE="docker-machine-iso:iso/rancheros-proxmoxve-autoformat-v1.5.5.iso"
VM_NAME="docker-rancher"

docker-machine rm --force $VM_NAME >/dev/null 2>&1 || true

docker-machine --debug \
    create \
    --driver proxmoxve \
    --proxmoxve-proxmox-host $PVE_HOST \
    --proxmoxve-proxmox-node $PVE_NODE \
    --proxmoxve-proxmox-user-name $PVE_USER \
    --proxmoxve-proxmox-user-password $PVE_PASSWD \
    --proxmoxve-proxmox-realm $PVE_REALM \
    --proxmoxve-proxmox-pool $PVE_POOL \
    \
    --proxmoxve-vm-storage-path $PVE_STORAGE_NAME \
    --proxmoxve-vm-storage-size $PVE_STORAGE_SIZE \
    --proxmoxve-vm-cpu-cores $PVE_CPU_CORES \
    --proxmoxve-vm-memory $PVE_MEMORY \
    --proxmoxve-vm-image-file "$PVE_IMAGE_FILE" \
    \
    --proxmoxve-ssh-username $SSH_USERNAME \
    --proxmoxve-ssh-password $SSH_PASSWORD \
    \
    --proxmoxve-debug-resty \
    --proxmoxve-debug-driver \
    \
    $VM_NAME


eval $(docker-machine env $VM_NAME)

docker ps

Run the script

At the first run, it is advisable to not comment out the debug flags. If everything works as expected, you can remove them.

Changes

Version 4

support for using clones+cloud-init (Thanks to @travisghansen)
enable custom network bridge without vlan tag (Thanks to @guyguy333)
including args to choice scsi model (Thanks to @bemanuel)
fix remove error, add further flags (Thanks to @Psayker)

Version 3

Renaming driver from proxmox-ve to proxmoxve due to identification problem with RancherOS's K8S implementation (Thanks to @Sellto for reporting #16)
fixing issue with created disk detection (Thanks to @Sellto for reporting #16)
Add IPAddress property needed by rancher to know the ip address of the created VM (Thanks to @Sellto)
Change the name of each flag for better display in the rancher Node Templates (Thanks to @Sellto)
Add number of CPU cores configuration paramater (Thanks to @Sellto)
LVM-thin fixes (Thanks to @vstconsulting)
Bridge and VLAN tag support (Thanks to @bemanuel)
Fixing filename detection including NFS support

Version 2

exclusive RancherOS support due to their special Proxmox VE iso files
adding wait cycles for asynchronous background tasks, e.g. create, stop etc.
use one logger engine
add guest username, password and ssh-port as new command line arguments
more and potentially better error handling

Version 1

Initial Version

lnxbil/docker-machine-driver-proxmox-ve

lnxbil

Reviews

Repository Details