hartwork/image-bootstrap hartwork/image-bootstrap

  • Stars
    star
    245
  • Rank 165,304 (Top 4 %)
  • Language
    Python
  • Created over 9 years ago
  • Updated almost 1 year ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
hartwork/image-bootstrap
github

hartwork

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

โ›… Creates Linux chroots and bootable virtual machine images; command line tool (Python 3)

Build Status

Table of Contents

About

Welcome to the home of image-bootstrap (and its little brother directory-bootstrap).

image-bootstrap is a command line tool to generate bootable virtual machine images and write them to a given block device.
Linux distributions supported by image-bootstrap currently include: Arch, Debian, Gentoo, Ubuntu.
When passing the --openstack parameter, images are prepared for use with OpenStack.

directory-bootstrap is a command line tool to install non-Debian Linux distributions into a given directory (similar to debootstrap for Debian/Ubuntu).
Distributions supported by directory-bootstrap currently include: Alpine Linux, Arch Linux, Gentoo, and Void Linux.

History

image-bootstrap started out as a re-write of grml-debootstrap. Primarily, it can be used to create Debian/Ubuntu or Arch images ready to be launched as a virtual machine.

In comparison to grml-debootstrap, by now image-bootstrap

  • installs to block devices only,

  • supports several approaches to installing GRUB 2.x, or extlinux, or no bootloader at all,

  • does not leak the host's hostname into the resulting image,

  • supports passing the root password off the command-line (and the eyes of other users and shell history),

  • supports using a custom /etc/resolv.conf file (e.g. to not leak your home router model name from /etc/resolv.conf into the image),

  • has support for installing Arch Linux, Gentoo and Ubuntu (besides Debian),

  • is able to create OpenStack images,

  • is written in Python rather than Bash/mksh, and

  • has more friendly terminal output.

directory-bootstrap came into life with the arrival of support for Arch Linux. Support for Gentoo followed, after. Support for Void Linux and CentOS chroots came into live during 34c3, December 2017. Support for Alpine Linux chroots came into in March 2018.

Example run

The following is a complete demo of installing Debian stretch to LVM volume /dev/vg/lv and launching the resulting image using KVM.

# ${EDITOR} root_password.txt

# sudo image-bootstrap --hostname stretch debian \
                                  --password-file root_password.txt /dev/vg/lv
     _                          __             __      __
    (_)_ _  ___ ____ ____  ___ / /  ___  ___  / /____ / /________ ____
   / /  ' \/ _ `/ _ `/ -_)/__// _ \/ _ \/ _ \/ __(_-</ __/ __/ _ `/ _ \
  /_/_/_/_/\_,_/\_, /\__/    /_.__/\___/\___/\__/___/\__/_/  \_,_/ .__/
               /___/                      v0.9.1 :: 2015-07-11  /_/

Software libre licensed under AGPL v3 or later.
Brought to you by Sebastian Pipping <[email protected]>.
Please report bugs at https://github.com/hartwork/image-bootstrap.  Thank you!

Selected approach "chroot-grub2-drive" for bootloader installation.
Checking for blkid... /sbin/blkid
Checking for chmod... /bin/chmod
Checking for chroot... /usr/sbin/chroot
Checking for cp... /bin/cp
Checking for debootstrap... /usr/sbin/debootstrap
Checking for find... /usr/bin/find
Checking for kpartx... /sbin/kpartx
Checking for mkdir... /bin/mkdir
Checking for mkfs.ext4... /sbin/mkfs.ext4
Checking for mount... /bin/mount
Checking for parted... /sbin/parted
Checking for partprobe... /sbin/partprobe
Checking for rm... /bin/rm
Checking for rmdir... /bin/rmdir
Checking for sed... /bin/sed
Checking for tune2fs... /sbin/tune2fs
Checking for umount... /bin/umount
Checking for uname... /bin/uname
Checking for unshare... /usr/bin/unshare

Checking for known unsupported architecture/machine combination...
Checking if "/dev/vg/lv" is a block device...
Reading root password from file "/home/user1/root_password.txt"...
Unsharing Linux namespaces (mount, UTS/hostname)...
Partitioning "/dev/vg/lv"...
Activating partition devices...
Creating file system on "/dev/mapper/vg-lvp1"...
Creating directory "/mnt/tmpFczeFl"...
Mounting partitions...
Creating directory "/mnt/tmpFczeFl/etc"...
Writing file "/mnt/tmpFczeFl/etc/hostname"...
Writing file "/mnt/tmpFczeFl/etc/resolv.conf" (based on file "/etc/resolv.conf")...
Bootstrapping Debian "stretch" into "/mnt/tmpFczeFl"...
Writing file "/mnt/tmpFczeFl/etc/hostname"...
Writing file "/mnt/tmpFczeFl/etc/resolv.conf" (based on file "/etc/resolv.conf")...
Writing file "/mnt/tmpFczeFl/etc/fstab"...
Writing file "/mnt/tmpFczeFl/etc/network/interfaces"...
Running pre-chroot scripts...
Mounting non-disk file systems...
Setting root password...
Writing device map to "/mnt/tmpFczeFl/boot/grub/device.map" (mapping "(hd9999)" to "/dev/dm-8")...
Installing bootloader to device "/dev/vg/lv" (actually "/dev/dm-8", approach "chroot-grub2-drive")...
Generating GRUB configuration...
Post-processing GRUB config...
Generating initramfs...
Unmounting non-disk file systems...
Cleaning chroot apt cache...
Running post-chroot scripts...
Unmounting partitions...
Removing directory "/mnt/tmpFczeFl"...
Deactivating partition devices...
Done.

# sudo kvm -hda /dev/vg/lv

Without --color never, the output above is actually in color.

Speeding things up

Using RAM instead of HDD/SSD

If you run image-bootstrap repeatedly and have enough RAM, you may want to create images on RAM storage rather than on disk. I use a setup with

  • a loop device (to have a block device)

  • over a sparse file (to save space)

  • in a tmpfs mount (to use RAM).

For example (assuming you have /tmp in RAM already):

# sudo mount -o remount,size=6g /tmp
# truncate --size 3g /tmp/disk3g
# LOOP_DEVICE="$(losetup --show -f /tmp/disk3g)"

# image-bootstrap .... arch ... "${LOOP_DEVICE}"
# qemu-img convert -p -f raw -O qcow2 "${LOOP_DEVICE}" /var/lib/arch-$(date -I).qcow2

# losetup -d "${LOOP_DEVICE}"
# rm /tmp/disk3g

Apt-Cacher NG -- a cache specific to Debian/Ubuntu

When creating multiple images, a local instance of Apt-Cacher NG and passing --mirror http://localhost:3142/debian to image-bootstrap may come in handy.

Polipo -- a generic HTTP cache

For a distribution-agnostic cache, using Polipo can greatly speed up consecutive runs. Invoke image-bootstrap with

# http_proxy=http://127.0.0.1:8123/ image-bootstrap ...

when using Polipo with default port configuration.

haveged -- an entropy generator

During image creation, cryptographic keys may need to be generated, e.g. for the OpenSSH server, at least temporarily. As key generation relies on availability of entropy, image creation may take longer in environments that are slow at adding to the entropy pool. To speed things up, running haveged at the host system could be an option, especially since all keys should be deleted from images, eventually. Otherwise, there is a risk of ending up with multiple systems having the same key allowing for attacks. I am unsure of the quality of entropy that haveged produces. Use is at your own risk.

Debian package

As long as image-bootstrap as not available in Debian, you can make an image-bootstrap Debian package yourself easily from Git as follows:

# git clone https://github.com/hartwork/image-bootstrap.git
Cloning into 'image-bootstrap'...
[..]

# make -C image-bootstrap/ deb
[..]

# ls *.deb
image-bootstrap_0.9.1_all.deb

# sudo dpkg -i image-bootstrap_0.9.1_all.deb
[..]

Usage (--help output)

In general, the usage is:

image-bootstrap [..] DISTRIBUTION [..] DEVICE

or

image-bootstrap --hostname NAME [DISTRO_AGNOSTIC] DISTRIBUTION [DISTRO_SPECIFIC] DEVICE

in a bit more detail.

A dump of the current --help output would be:

# image-bootstrap --help
usage: image-bootstrap [-h] [--version] [--color {never,always,auto}]
                       [--debug] [--quiet] [--verbose] [--arch ARCHITECTURE]
                       [--bootloader {auto,chroot-grub2-device,chroot-grub2-drive,host-extlinux,host-grub2-device,host-grub2-drive,none}]
                       [--bootloader-force] [--hostname NAME] [--openstack]
                       [--password PASSWORD | --password-file FILE]
                       [--resolv-conf FILE] [--disk-id ID]
                       [--first-partition-uuid UUID] [--machine-id ID]
                       [--scripts-pre DIRECTORY] [--scripts-chroot DIRECTORY]
                       [--scripts-post DIRECTORY] [--grub2-install COMMAND]
                       [--cache-dir DIRECTORY]
                       DISTRIBUTION ... DEVICE

Command line tool for creating bootable virtual machine images

positional arguments:
  DEVICE                block device to install to

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit

text output configuration:
  --color {never,always,auto}
                        toggle output color (default: auto)
  --debug               enable debugging
  --quiet               limit output to error messages
  --verbose             increase verbosity

machine configuration:
  --arch ARCHITECTURE   architecture (e.g. amd64)
  --bootloader {auto,chroot-grub2-device,chroot-grub2-drive,host-extlinux,host-grub2-device,host-grub2-drive,none}
                        approach to take during bootloader installation
                        (default: auto)
  --bootloader-force    apply more force when installing bootloader (default:
                        disabled)
  --hostname NAME       hostname to set (default: "machine")
  --openstack           prepare for use with OpenStack (default: disabled)
  --password PASSWORD   root password to set (default: password log-in
                        disabled)
  --password-file FILE  file to read root password from (default: password
                        log-in disabled)
  --resolv-conf FILE    file to copy nameserver entries from (default:
                        /etc/resolv.conf)
  --disk-id ID          specific disk identifier to apply, e.g. 0x12345678
  --first-partition-uuid UUID
                        specific UUID to apply to first partition, e.g.
                        c1b9d5a2-f162-11cf-9ece-0020afc76f16
  --machine-id ID       specific machine identifier to apply, e.g.
                        c1b9d5a2f16211cf9ece0020afc76f16

script integration:
  --scripts-pre DIRECTORY
                        scripts to run prior to chrooting phase, in
                        alphabetical order
  --scripts-chroot DIRECTORY
                        scripts to run during chrooting phase, in alphabetical
                        order
  --scripts-post DIRECTORY
                        scripts to run after chrooting phase, in alphabetical
                        order

command names:
  --grub2-install COMMAND
                        override grub2-install command

general configuration:
  --cache-dir DIRECTORY
                        directory to use for downloads (default:
                        /var/cache/directory-bootstrap/)

subcommands (choice of distribution):
  Run "image-bootstrap DISTRIBUTION --help" for details on options specific to that distribution.

  DISTRIBUTION          choice of distribution, pick from:
    arch                Arch Linux
    debian              Debian GNU/Linux
    gentoo              Gentoo
    ubuntu              Ubuntu

     _                          __             __      __
    (_)_ _  ___ ____ ____  ___ / /  ___  ___  / /____ / /________ ____
   / /  ' \/ _ `/ _ `/ -_)/__// _ \/ _ \/ _ \/ __(_-</ __/ __/ _ `/ _ \
  /_/_/_/_/\_,_/\_, /\__/    /_.__/\___/\___/\__/___/\__/_/  \_,_/ .__/
               /___/                      v2.0.0 :: 2020-02-28  /_/

Software libre licensed under AGPL v3 or later.
Brought to you by Sebastian Pipping <[email protected]>.
Please report bugs at https://github.com/hartwork/image-bootstrap.  Thank you!

To show options specific to Debian, run ..

# image-bootstrap debian --help
usage: image-bootstrap debian [-h] [--debootstrap COMMAND] [--release RELEASE]
                              [--mirror URL] [--debootstrap-opt OPTION]

optional arguments:
  -h, --help            show this help message and exit
  --release RELEASE     specify Debian release (default: stretch)
  --mirror URL          specify Debian mirror to use (e.g.
                        http://localhost:3142/debian for a local instance of
                        apt-cacher-ng; default:
                        http://httpredir.debian.org/debian)
  --debootstrap-opt OPTION
                        option to pass to debootstrap, in addition; can be
                        passed several times; use with --debootstrap-opt=...
                        syntax, i.e. with "="

command names:
  --debootstrap COMMAND
                        override debootstrap command

Hints on using image-bootstrap within a pipe

If you want to run image-bootstrap in a pipe to capture its output to both stdout/stderr to a single log file, be sure to run image-bootstrap in unbuffered mode, e.g.:

python3 -u image-bootstrap [OPTIONS] 2>&1 | tee my.log

The default shebang generated by python setuptools does not use -u. It's also not easily possible to have it pass -u since the env command does allow for for passing command parameters only in fairly recent versions. As a consequence, stdout/stderr won't be synchronized and error output you see in a log file will not exactly correspond to preceding/succeeding output on stdout. (See issue #71 for more details.)

Moreover, if you're using Bash and you need to keep track of image-bootstrap's exit code, be sure to run set -o pipefail prior to invoking image-bootstrap. (Please see the Pipelines section of the GNU Bash Reference Manual for more details.)

Known limitations

Installing to partition block devices

Linux does not like partitions in partitions much. It can be tricked using device mapper, though.

This is how to install to a partition using another partition as a temporary target. The temporary target must

  • be 2 GiB in space or more (to hold the whole distribution) and

  • smaller or equal than the actually target (for the later copy to work).

# dmsetup create dm-linear-vda4 --table "0 $(blockdev --getsz /dev/vda4) linear /dev/vda4 0"
# image-bootstrap --openstack arch /dev/mapper/dm-linear-vda4
# partprobe /dev/mapper/dm-linear-vda4
# pv /dev/mapper/dm-linear-vda4p1 > /dev/vda2
# dmsetup remove dm-linear-vda4p1
# dmsetup remove dm-linear-vda4

(/dev/vda2 is the real target, /dev/vda4 the temporary one.)

There are other ways to achieve the same.

More Repositories

1

git-delete-merged-branches

๐Ÿ”ฅ Command-line tool to delete merged Git branches
Python
858
star
2

grub2-theme-preview

๐ŸŒ‡ Preview a full GRUB 2.x theme (or just a background image) using KVM / QEMU
Python
286
star
3

jawanndenn

๐Ÿ“… Simple alternative to Doodle polls and scheduling (Python 3, Django 5, some JavaScript)
Python
182
star
4

django-berlin

๐Ÿ€ Ever wondered who's doing Django... in Berlin?
79
star
5

sandwine

๐Ÿท Command-line tool to run Windows apps with Wine and bwrap/bubblewrap isolation on Linux
Python
66
star
6

wnpp.debian.net

๐ŸŒ Code powering website "Debian Packages that Need Lovin'" created in 2009
Python
42
star
7

resolve-march-native

๐ŸŒ Tool to determine what GCC flags -march=native would resolve into
Python
32
star
8

docker-ssl-reverse-proxy

๐Ÿ”’ Easy-to-use auto-SSL reverse proxy as a Docker container based on Caddy and Letโ€™s Encrypt
Python
23
star
9

xiangqi-setup

๐ŸŽฒ Generate razor-sharp Xiangqi (Chinese chess) setup graphics; command line tool + themes
Python
22
star
10

binary-gentoo

๐Ÿฎ Collection of simple CLI tools to help build Gentoo packages on a non-Gentoo Linux host
Python
17
star
11

django-createsuperuserwithpassword

Django management command to create usable super users, programmatically
Python
16
star
12

svneverever

๐Ÿ”ฆ Collects path entries across SVN history (Python)
Python
15
star
13

surrogates

๐Ÿ˜Œ Encode and decode pairs of surrogate characters in Python 3
Python
11
star
14

django-migration-vis

to visualize django migration graphs with GraphVis
Python
11
star
15

golang-berlin

๐Ÿ€ Ever wondered who's doing Golang... in Berlin?
10
star
16

antijack

๐Ÿฅท seccomp-based anti-TTY-hijacking proof-of-concept (prevents TIOCSTI and TIOCLINUX)
C
10
star
17

go-wait-for-it

๐Ÿฆซ Wait for service(s) to be available before executing a command.
Go
9
star
18

visdriver

๐ŸŽฆ visdriver uses Winamp plug-ins to visualize audio; targets Wine on GNU/Linux primarily
C
8
star
19

fetchcommandwrapper

๐Ÿฎ Wrapper integrating aria2 (https://aria2.github.io/) into portage's FETCHCOMMAND for faster downloads (Python)
Python
8
star
20

no-cyclic-imports

โ™ป๏ธ Tool to detect and report on cyclic imports in a Python codebase
Python
8
star
21

mozilla-password-decrypt

๐Ÿ”“ Decrypt passwords stored by Firefox, Thunderbird, Iceweasel, Icedove using libnss3.so
Python
7
star
22

update-notifier-tray

๐Ÿ’ก Update indicator tray icon for Debian and Gentoo; meant to replace gone update-notifier
Python
7
star
23

shared-library-version-bump

๐ŸŒ A web-tool to help with bumping -version-info linker arguments with your upcoming release of a shared library
HTML
6
star
24

rnv

๐Ÿ  Relax NG Compact Syntax validator by David Tolpin; official upstream maintenance repository
C
5
star
25

vue-tristate-checkbox

Vue.js component implementing a cycling tristate checkbox that supports form submission
HTML
5
star
26

extract-hotline-miami-soundtrack

๐ŸŽต Extract soundtrack from Hotline Miami PC game .wad file (OGG Vorbis); command line tool
C
5
star
27

enum

๐Ÿ”ข Command line tool to enumerate values, replacing GNU seq and *BSD jot (moved from https://fedorahosted.org/enum/)
C
4
star
28

svgstrip

๐Ÿ›€ Tool that strips private information from SVG files (Python)
Python
4
star
29

import-watch

Trace module imports and detect/deny cyclic imports at runtime
Python
3
star
30

dsp_restless

The Restless Winamp Plugin (a.k.a. dsp_restless) of 2005
C++
3
star
31

backup-my-hub

๐Ÿ™ Simple command line tool to make local backups of GitHub repositories and gists (Python)
Python
3
star
32

rust-for-it

๐Ÿฆ€ Wait for one or more services (TCP port) to be available before executing a command; Rust version of wait-for-it
Rust
3
star
33

xiangqi-book-example

Example of a book featuring print-quality Xiangqi board setup graphics generated on-the-fly using xiangqi-setup; PDF output download at https://github.com/hartwork/xiangqi-book-example/releases/download/1.0.0/book_1_0_0.pdf
TeX
3
star
34

sdl_video_demo

Demos fast blitting of a video buffer to the screen with scaling while respecting aspect-ratio in C99 for both SDL 2 and SDL 1
C
2
star
35

klogripper

๐Ÿšฝ Kernel log ripper, i.e. a union of dmesg and tail -f (C)
C
2
star
36

newspipe

๐Ÿ“ฐ RSS/Atom aggregation to e-mail. Fork of Newspipe 1.1.9 with SSL and packaging (Python)
Python
2
star
37

slides_shell_we_continue

๐Ÿš Slides for a talk on Bash and the terminal (in German)
CSS
2
star
38

gentoo-workshop-troisdorf-2015-08-01

๐Ÿฎ Slides for a talk on how to write Ebuilds for Gentoo (in German); PDF: http://downloads.trolug.de/2015-09-01_gentoo-workshop-troisdorf_sebastian_pipping_paketierung.pdf
Makefile
2
star
39

slides_python_json_emoji_crash_story

๐Ÿ—ฟ Slides of talk "Python JSON Emoji crash story" at Django Meetup Berlin 2020-02-18; latest PDF: https://github.com/hartwork/slides_python_json_emoji_crash_story/releases/download/v4/python-json-emoji-crash-story-2020-02-18-v4.pdf
Makefile
2
star
40

OpenXiangqi

๐ŸŽฒ Chinese chess (Java, GNU AGPL v3 or later)
Java
1
star
41

out_bridge

The Outbridge Winamp Plugin (a.k.a. out_bridge) of 2005/2006
C++
1
star
42

gen_freeze

The Freeze Winamp Plugin (a.k.a. gen_freeze) of 2006
C++
1
star
43

dsp_freeverb_gpl

The GPL Freeverb Winamp Plugin (a.k.a. dsp_freeverb_gpl) of 2006
C++
1
star
44

flasktop

๐Ÿ” "top in the browser" using Flask and jQuery DataTables (Python, JavaScript)
Python
1
star
45

audio_pump_demo

๐ŸŽ™๏ธ Demos pumping audio from the microphone to headphones/speakers in C99 for PulseAudio, PortAudio and SDL 2
C
1
star
46

beamer-theme-matrix

๐ŸŒ LaTeX Beamer Theme Matrix of 2009 (Bash)
Shell
1
star
47

gen_allmusic

The Allmusic Hotkey Winamp Plugin (a.k.a. gen_allmusic) of 2006
C++
1
star