• Stars
    star
    117
  • Rank 301,828 (Top 6 %)
  • Language
    Shell
  • License
    MIT License
  • Created about 10 years ago
  • Updated 8 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

πŸ”© Scripts used to create Images on Scaleway - https://github.com/scaleway-community

Scaleway Image Toolbox

This repository contains the tools, documentation, examples and content for creating images on Scaleway.

Distribution images & Instant Apps

Official Scaleway images are found as git repositories on Github, in two organizations:

  • scaleway, which focuses on bases images and distributions, with names prefixed by "image-"
  • scaleway-community is dedicated to Instant Apps and ready-to-use images, where names are prefixed with "scaleway-".

Your contributions are welcome !

Your custom images can be anywhere, though, they don't even need to be in repositories.

Getting started

Simple images

On Scaleway, images are created by tagging an existing snapshot appropriately. This can be done with the scw CLI tool (or alternatively, using the console or directly through the API):

# Using the scw tool -- this assumes you've already set it up.
# Create a snapshot first:
$ scw commit -v 1 my-server my-snapshot
f3af311c-53f7-4f5d-9252-0bf69f017269

# Now make an image out of it:
$ scw tag --arch=x86_64 --bootscript="mainline 4.14"  my-snapshot my-image
a601dbac-08cb-4af3-9ff6-73b7e8dfd34f

$ scw images
REPOSITORY           TAG                 IMAGE ID            CREATED             REGION              ARCH
user/my-image        latest              a601dbac            5 seconds           [     par1]         [x86_64]
---- snipping all stock images 8< ----

Docker-based images

Using the tools contained in this repository, it is possible to create Scaleway images from Docker images. At its core, this process uses the tools we explored above and is pretty straightforward to use:

  1. Clone this repository and the repository of the image (if there is one) on a Scaleway instance
  2. In your terminal, navigate to the image-tools directory, and type make IMAGE_DIR=<the image's directory> scaleway_image
  3. ???
  4. Profit !

As a first example, let's simply rebuild our Ubuntu Xenial image without changing anything. Note that this is not needed to build your own Docker-based images, and is shown here only for demonstration purposes.

asciicast

Now that we've had a first taste, let's create a custom image based on Scaleway's Ubuntu image:

asciicast

To recap, here's what's needed for a custom Docker-based image:

  • A Dockerfile with an ARCH buildarg, based on any Docker image as long as the FROM chain can be traced back to one of Scaleway's official images on the Dockerhub: this is needed so that your custom image will ship with the needed tools, configuration and initialization scripts.
  • An env.mk file, containing metadata for the builder tool. It must be valid makefile syntax and will be included at the beginning of the build process. It must at least set the following variables: IMAGE_NAME, IMAGE_VERSION and IMAGE_TITLE but can also set other information, such as authorship, or extra buildargs for Docker with BUILD_ARGS.

All the rest is up to you and your creativity ! You can, for example, upload your images to the Dockerhub and start using them as bases for other images.

After building, the image and associated snapshots will be available for use with your account. Note that introducing name conflicts will force you to use the ids to interact with the images through the CLI.

Diving below the surface

This section mainly deals with explaining with what happens during the build process. Reading it is not required to be able to create images.

As explained above, images are created by tagging a snapshot. Snapshots capture the state of a server's volume at a certain point in time. The simplicity of this step is what makes it versatile: as long as you can somehow get a server to boot and populate one of its volume with some data, you can create a Scaleway Image.

From there, images can be built in many ways. An example would be to use a server with two volumes, using the first as a system volume and populating the second one as needed (parted, mkfs, debootstrap, pacstrap, etc...) before taking a snapshot of it. Snapshots can only be created when the server is off, though, so you would need to stop it. This can get a bit complicated when you need to build images repeatedly, and requires you to have a dedicated server for the pupose of image creation, so it might be a bit unpractical.

The build initrd

To make things easier, we created a lightweight tool for the purpose of image creation in the form of a special initrd. An initrd, short for initial ramdisk, is a minimal root filesystem that is loaded in memory and mounted from there. In its classic form, an initrd is used to setup the core components of the system and finding the real root filesystem, before surrendering execution to the OS' init.

This build initrd is a bit different, as it does not aim to hook into a complete boot and will never call init. It is a intended to be used via a remote boot through preset bootscripts, which are specifications of a kernel + initrd that we feed into IPXE, the network boot firmware. A server can be assigned a bootscript at creation. The ids of the build bootscripts for different architectures can be found in the bootscript_ids file at the root of this project. It can also be accessed with the name "Image creation bootscript".

Detailed information can be found in the initrd's repository, but let's recap its features broadly.

The build initrd offers different methods for building images, which are a collection of scripts. After setting up the network and retrieving metadata, it will use the method specified in the build_method parameter given through the metadata environment. Each method requires specific arguments.

The main method used by the image-tools suite is from-rootfs, which will retrieve a tarred rootfs from a given location and will apply it to the volume after setting up partitions (Linux, EFI) and their associated filesystem (ext4, fat32). GRUB EFI will then be installed to the partition. For example:

$ scw create --bootscript="image creation" --env="build_method=from-rootfs rootfs_url=<url of the rootfs.tar>" 50G

Other build methods also exist:

  • from-qcow2, which will copy a qcow2 file's content to the device,
  • unpartitioned-from-rootfs which will skip the partitions and EFI setup and use the raw device with an ext4 filesystem. Obviously, these images can only be booted remotely.

Finally, the build initrd will signal the completion of its job by opening a port given to it through the signal_build_done_port parameter, or 22 by default.

More Repositories

1

scaleway-cli

Command Line Interface for Scaleway
Go
843
star
2

docker-machine-driver-scaleway

🐳 Scaleway driver for Docker Machine
Go
248
star
3

terraform-provider-scaleway

Terraform Scaleway provider
Go
198
star
4

frontalization

Pytorch deep learning face frontalization model
Python
198
star
5

postal-address

πŸ“― Parse, normalize and render postal addresses.
Python
174
star
6

natasha

Natasha is a fast and scalable, DPDK powered, stateless NAT44 packet processor
C
125
star
7

python-scaleway

🐍 Python SDK to query Scaleway APIs.
Python
114
star
8

ultraviolet

A monorepo Design System with React components.
TypeScript
104
star
9

kernel-tools

🐧 Kernels on Scaleway
Makefile
104
star
10

scaleway-sdk-go

Integrate Scaleway with your Go application
Go
103
star
11

netbox-netprod-importer

Poll data from network devices in production and import it into netbox
Roff
94
star
12

c14-cli

[EOL] πŸ’» Manage your C14 Online Cold Storage from Command Line
Go
89
star
13

docs-content

Scaleway Documentation contents
MDX
78
star
14

serverless-scaleway-functions

Plugin for Serverless Framework to allow users to deploy their serverless applications on Scaleway Functions
JavaScript
69
star
15

image-builder

πŸ“ build server images on Scaleway
Dockerfile
66
star
16

scaleway-csi

Container Storage Interface (CSI) Driver for https://www.scaleway.com/block-storage/
Go
57
star
17

scaleway-cloud-controller-manager

Kubernetes Cloud Controller Manager for Scaleway
Go
53
star
18

initrd

πŸ”§ Initrd builder with qemu nbd metadata for Scaleway boot (armhf + x86_64)
Shell
51
star
19

action-scw

Github action to install and use scaleway-cli
TypeScript
49
star
20

serverless-examples

Example projects and patterns for use with Scaleway's serverless products
HCL
48
star
21

netbox2netshot

Inventory synchronization tool between Netbox and Netshot
Rust
46
star
22

python-netboxapi

Python client API for Netbox
Python
30
star
23

scaleway-sdk-js

Integrate Scaleway with your JS application
TypeScript
28
star
24

cert-manager-webhook-scaleway

A Scaleway DNS ACME webhook for cert-manager
Go
27
star
25

taskor

Async queued task library for Go
Go
27
star
26

juju-scaleway

☁️ Scaleway driver for Juju
Python
26
star
27

prometheus-scw-sd

Prometheus Service Discovery for Scaleway
Go
25
star
28

scaleway-operator

Scaleway Operator for Kubernetes
Go
25
star
29

scaleway-lib

scaleway-lib is a set of NPM packages used at Scaleway
TypeScript
25
star
30

scaleway-sdk-python

Integrate Scaleway with your Python applications
Python
24
star
31

serverless-gateway

Serverless API Gateway running on Scaleway
Python
18
star
32

crossplane-provider-scaleway

Crossplane Scaleway Provider
Go
16
star
33

homebrew-scaleway

🍺 `brew tap scaleway/scaleway`
Ruby
11
star
34

packer-plugin-scaleway

Packer plugin for Scaleway Builder
Go
10
star
35

serverless-functions-python

Utilities for testing your Python functions for Scaleway Serverless Functions.
Python
10
star
36

scaleway-form

Build amazing forms with Scaleway-ui and React-final-form πŸš€πŸš€
TypeScript
9
star
37

ansible

Ansible Collection for Scaleway
Python
8
star
38

scaleway-ui-website

Open Source Component Library
TypeScript
8
star
39

scaleway-functions-runtimes

Runtimes for Scaleway Function As A Service Platform
Go
8
star
40

serverless-api-framework-python

Framework for writing serverless APIs in Python, using Scaleway functions and containers.
Python
8
star
41

serverless-offline-scaleway

JavaScript
7
star
42

qa

πŸ’” Cloud testing suite
Makefile
6
star
43

action-scw-secret

TypeScript
6
star
44

serverless-functions-node

Offline testing for Scaleway Serverless Functions
TypeScript
6
star
45

sentry-wrapper

Log exceptions of a non sentry-capable setuptools entrypoint to Sentry
Python
6
star
46

cq-source-scaleway

CloudQuery Provider for Scaleway
Go
5
star
47

helm-charts

Scaleway Helm Charts
Mustache
5
star
48

octodns-scaleway

Scaleway provider for octoDNS
Python
5
star
49

terraform-scaleway-vpc-module

Terraform module for configuring a VPC
HCL
5
star
50

design-tokens

JSON design tokens used in Scaleway UI library
5
star
51

python2-secrets

Backport python3.6 secrets to python2
Python
5
star
52

serverless-functions-go

Scaleway Serverless Functions (FaaS) framework to increase developer experience, local testing and more
Go
5
star
53

port-range

🐍 A python module for port-range
Python
4
star
54

qiskit-scaleway

Scaleway provider implementation for Qiskit SDK
Python
4
star
55

waypoint-plugin-scaleway

Waypoint provider for Scaleway
Go
4
star
56

learn-nomad-cluster-setup

Infrastructure configuration to deploy a Nomad cluster on Scaleway
Shell
3
star
57

python-dkim

User-friendly interface as well as python bindings for libopendkim.
Python
3
star
58

kernels

Kernel repository used at Scaleway (branch-oriented)
3
star
59

tink-go-scwkms

Extension to Tink Go that provides Scaleway's Key Manager integration
Go
3
star
60

scaleway-packages

Repository of the RPM packages used in Scaleway Instance Images
2
star
61

demo-scaleway-ansible-baremetal

Scaleday 2019 demo baremetal
Python
2
star
62

tim

TIM is a plugin for testinfra parametrizing tests as described by a yaml file.
Python
1
star
63

snippets

Snippets to use Scaleway services
Python
1
star
64

tutorial-tem-serverless-terraform

HTML
1
star
65

terraform-scaleway-lb-module

HCL
1
star
66

dagster-scaleway

Python
1
star