• Stars
    star
    165
  • Rank 228,906 (Top 5 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created over 4 years ago
  • Updated 4 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Tool for synchronizing Helm Chart repositories.

CircleCI

charts-syncer

Sync chart packages and associated container images between chart repositories

Table of Contents

Usage

Sync all Helm Charts

$ charts-syncer sync

Sync Helm Charts from a specific date

$ charts-syncer sync --from-date 2020-05-15

Sync latest version of each Helm Chart

$ charts-syncer sync --latest-version-only

Advanced Usage

Sync Helm Charts and Container Images

By default, charts-syncer only sync Helm Charts packages, it does not copy the container images referenced by the chart. This feature can be enabled by setting the relocateContainerImages: true property in the config file i.e

# leverage .relok8s-images.yaml file inside the Charts to move the container images too
relocateContainerImages: true
source:
   ...
target:
   ...

In order for this option to work it is required that the source Helm Charts includes a .relok8s-images.yaml file with information about where to find the images inside chart. For more information about this file please refer to asset-relocation-tool-for-kubernetes readme.

Sync Helm Charts and associated container images between disconnected environments

There are scenarios where the source and target Helm Charts repositories are not reachable at the same time from the same location.

For those cases, charts-syncer supports a two steps relocation for offline Chart and container images transport, check the air gap docs.


Configuration

Below you can find an example configuration file. To know all the available configuration keys see the charts-syncer file as it includes explanatory comments for each configuration key.

#
# Example config file
#
source:
  repo:
    kind: HELM
    url: http://localhost:8080
    # Helm repository authentication, same for other repo types i.e OCI
    # auth:
    #   username: "USERNAME"
    #   password: "PASSWORD"
  # Container images registry authn
  # containers:
  #  auth:
  #     registry: 'REGISTRY'
  #     username: "USERNAME"
  #     password: "PASSWORD"
target:
  repoName: myrepo
  containerRegistry: k8s.container.registry.io
  containerRepository: repository/demo/k8s
  # Container images registry authn
  # containers:
  #   auth:
  #     username: "USERNAME"
  #     password: "PASSWORD"
  repo:
    kind: CHARTMUSEUM
    url: http://localhost:9090
    # Helm repository authentication
    # auth:
    #   username: "USERNAME"
    #   password: "PASSWORD"
charts:
  - redis
  - mariadb
# opt-out counterpart of "charts" property that explicit list the Helm charts to be skipped 
# either "charts" or "skipCharts" can be used at once
# skipCharts:
#  - mariadb

Note that the repo.url property you need to specify is the same one you would use to add the repo to helm with the helm repo add command.

Example: helm repo add bitnami https://charts.bitnami.com/bitnami.

Credentials for the Helm Chart repositories and container images registries can be provided using config file or the following environment variables:

Helm Chart repositories

  • SOURCE_REPO_AUTH_USERNAME

  • SOURCE_REPO_AUTH_PASSWORD

  • TARGET_REPO_AUTH_USERNAME

  • TARGET_REPO_AUTH_PASSWORD

Container images registries

  • SOURCE_CONTAINERS_AUTH_REGISTRY

  • SOURCE_CONTAINERS_AUTH_USERNAME

  • SOURCE_CONTAINERS_AUTH_PASSWORD

  • TARGET_CONTAINERS_AUTH_USERNAME

  • TARGET_CONTAINERS_AUTH_PASSWORD

Current available Kinds are HELM, CHARTMUSEUM, HARBOR and OCI. Below you can find the compatibility matrix between source and targets repositories.

Source Repo Target Repo Supported
HELM HELM
HELM CHARTMUSEUM
HELM HARBOR
HELM OCI
HELM LOCAL
CHARTMUSEUM HELM
CHARTMUSEUM CHARTMUSEUM
CHARTMUSEUM HARBOR
CHARTMUSEUM OCI
CHARTMUSEUM LOCAL
HARBOR HELM
HARBOR CHARTMUSEUM
HARBOR HARBOR
HARBOR OCI
HARBOR LOCAL
OCI HELM
OCI CHARTMUSEUM
OCI HARBOR
OCI OCI
OCI LOCAL
LOCAL HELM
LOCAL CHARTMUSEUM
LOCAL HARBOR
LOCAL OCI
LOCAL LOCAL

The list of charts in the config file is optional except for OCI repositories used as source. The rest of chart repositories kinds already support autodiscovery.

Harbor example

In the case of HARBOR kind repos, be aware that chart repository URLs are:

https://$HARBOR_DOMAIN/chartrepo/$HARBOR_PROJECT

So if HARBOR_DOMAIN=my.harbor.com and HARBOR_PROJECT=my-project, you would need to specify this repo in the config file like:

target:
 repo:
   kind: HARBOR
   url: https://my.harbor.com/chartrepo/my-project

OCI example

Since Harbor 2.0.0, there are two ways of storing charts. The legacy one uses chartmuseum under the hood and it corresponds to the HARBOR kind of this project. The new one however uses OCI to store helm charts as OCI artifacts. In case you are using Harbor with OCI backend you can use the following example:

target:
 repo:
   kind: OCI
   url: https://my.harbor.com/my-project/subpath

subpath in the previous url is optional in case your charts are not stored directly under your projects. It is worth mentioning that you can use Harbor robot accounts using OCI registries as source or target.

Also, take into account that if you use OCI as the source repository you must specify the list of charts to synchronize or a pointer to a charts index file in the repository.

Charts index for OCI-based repositories

By using a charts index file for OCI-Based repository you won't need to maintain a hardcoded list of chart names in the config file. charts-syncer will be able to auto-discover what charts need to be synchronized.

By default, the library will look up for a "charts-index:latest" charts index artifact within the source OCI repository.

However, this can be customized using the chartsIndex field using the format REGISTRY/PROJECT/[SUBPATH][:TAG|@sha256:DIGEST].

For example, if your URL is https://my-oci-registry.io/my-project/subpath and no chartsIndex is specified, charts-syncer will try to use my-oci-registry.io/my-project/subpath/charts-index:latest asset as index if it exists.

An example of the valid index format can be seen directly in its Protobuf definition. Worth to mention that the format of the charts index for OCI repositories is a custom one, not a traditional Helm index file.

source:
 repo:
   kind: OCI
   url: https://my-oci-registry.io/my-project/subpath
   # disableChartsIndex: true
   # Charts index location override, charts-index:latest by default
   chartsIndex: my-oci-registry.io/my-project/my-custom-index:prod

Finally, if no charts index is found, charts-syncer will require the list of charts in the config file:

source:
  repo:
    kind: OCI
    url: https://my-oci-registry.io/my-project/subpath
...
# Required if no index is provided or found
charts:
  - redis
  - mariadb
  - ...

Amazon Elastic Container Registry (ECR)

Amazon Elastic Container Registry (ECR) is an OCI registry, but it has two peculiarities that should be taken into account when interacting with charts-syncer.

The first peculiarity relates to authentication, usually you would have an IAM user with access_key_id and secret_access_key credentials. These are not the credentials that should be entered in the config file. To obtain the proper credentials, you need to obtain a temporary password to operate the ECR registry. By using the following command, you can get the password:

$ aws ecr get-login-password --region REGION

In the previous command remember to update the REGION placeholder by a proper value. The username will always be AWS.

The second peculiarity only affects when using ECR as a target registry since you need to create the repositories in advance. If the repositories don't exist when executing charts-syncer you will get an error like the following:

failed to do request: Post "https://AWS_ACCOUNT.dkr.ecr.AWS_REGION.amazonaws.com/v2/charts-syncer-test/charts/common/blobs/uploads/": EOF
failed to do request: Post "https://AWS_ACCOUNT.dkr.ecr.AWS_REGION.amazonaws.com/v2/charts-syncer-test/charts/apache/blobs/uploads/": EOF
failed to do request: Post "https://AWS_ACCOUNT.dkr.ecr.AWS_REGION.amazonaws.com/v2/charts-syncer-test/charts/mariadb/blobs/uploads/": EOF
failed to do request: Post "https://AWS_ACCOUNT.dkr.ecr.AWS_REGION.amazonaws.com/v2/charts-syncer-test/charts/mysql/blobs/uploads/": EOF
failed to do request: Post "https://AWS_ACCOUNT.dkr.ecr.AWS_REGION.amazonaws.com/v2/charts-syncer-test/charts/redis/blobs/uploads/": EOF

In that case make sure all the missing repositories are created before executing charts-syner again. Please refer to AWS documentation to see how to do it.

LOCAL example

In case charts-syncer is not able to directly push the modified charts to the desired target, it would be possible to sync the charts to a local folder using the LOCAL target kind and then use any other tool or process to upload these charts to the final charts repository.

target:
 repo:
   kind: LOCAL
   path: your_local_path

Requirements

In order for this tool to be able to successfully migrate a chart from a source repository to another it must fulfill the following requirements:

  • The images used by the chart must be specified in the following way in the values.yaml file:
image:
  registry: docker.io
  repository: bitnami/ghost
  tag: 3.22.2-debian-10-r0

The parent section name does not matter. In the previous example, instead of image it could be mainImage or whatever other name.

The important thing is that the image name is specified with registry, repository and tag.

The values of the parameters containerRegistry and containerRepositories from the configuration file will be used to update the registry and repository properties in the values.yaml. If these parameters are unset, the associated properties won't be modified. The tag property remains unchanged.

⚠️ Be aware that this tool expects the images to be already present in the target container registry.

Changes performed in a chart

In order to migrate a chart from one repository to another and retrieve the images from a new container registry, this tool performs the following changes in the chart code:

Update values.yaml and values-production.yaml (if exists)

These files are updated with the new container registry where the chart should pull the images from.

Update dependencies files

For Helm v2, these files are requirements.yaml and requirements.lock. For Helm v3, these files are Chart.yaml and Chart.lock

If the chart has any dependency, they should be registered in these files that will be updated to retrieve the dependencies from the target repository.

Update README.md

README files for bitnami charts include a TL;DR; section with instructions to add the helm repository to the helm CLI and a simple command to deploy the chart.

As the chart repository URL and chart repository name should have changed, the instructions in the README should be updated too.


Let's see the performed changes with an example. Imagine I sync the Ghost chart from the Bitnami chart repo to a local chartmuseum repo with no authentication.

I would use this config file:

source:
  repo:
    kind: HELM
    url: https://charts.bitnami.com/bitnami
target:
  repoName: myrepo
  containerRegistry: "my.registry.io"
  containerRepository: "test"
  repo:
    kind: CHARTMUSEUM
    url: http://localhost:8080

After executing the tool, these are the changes performed to the following files:

values.yaml

diff --git a/values.yaml b/values.yaml
index dff53b1..a9d5884 100755
--- a/values.yaml
+++ b/values.yaml
@@ -12,8 +12,8 @@
 ## ref: https://hub.docker.com/r/bitnami/ghost/tags/
 ##
 image:
-  registry: docker.io
-  repository: bitnami/ghost
+  registry: my.registry.io
+  repository: test/ghost
   tag: 3.22.2-debian-10-r0
   ## Specify a imagePullPolicy
   ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
@@ -40,8 +40,8 @@ image:
 ##
 volumePermissions:
   image:
-    registry: docker.io
-    repository: bitnami/minideb
+    registry: my.registry.io
+    repository: test/minideb
     tag: buster
     pullPolicy: Always
     ## Optionally specify an array of imagePullSecrets.

requirements.lock (only for Helm v2 charts)

diff --git a/requirements.lock b/requirements.lock
index ae8a2c5..ea23e53 100755
--- a/requirements.lock
+++ b/requirements.lock
@@ -1,9 +1,9 @@
 dependencies:
 - name: common
-  repository: https://charts.bitnami.com/bitnami
+  repository: http://localhost:8080
   version: 0.3.1
 - name: mariadb
-  repository: https://charts.bitnami.com/bitnami
+  repository: http://localhost:8080
   version: 7.6.1
-digest: sha256:9893236041ef5bdf2e972db020e72e8d68100eac4e280b9066d6e16c4061bcb3
-generated: "2020-07-06T18:13:45.662082005Z"
+digest: sha256:fbd22a3fc7b93ce6875a37902a3c8ccbb5dd3db2611ec9860b99e49d9f23196e
+generated: "2020-07-07T12:57:28.573258+02:00"

Chart.lock (only for Helm v3 charts)

diff --git a/Chart.lock b/Chart.lock
index ae1c198..eeed9a7 100644
--- a/Chart.lock
+++ b/Chart.lock
@@ -1,6 +1,6 @@
 dependencies:
 - name: zookeeper
-  repository: https://charts.bitnami.com/bitnami
+  repository: http://127.0.0.1:8080
   version: 5.21.9
-digest: sha256:3157eeec51b30e4011b34043df2dfac383a4bd11f76c85d07f54414a21dffc19
-generated: "2020-09-29T12:51:56.872354+02:00"
+digest: sha256:8aef6388d327cdf9b8f5714aadfe8112b2e2ff53494e86dbd42946d742d33ff0
+generated: "2020-09-30T16:15:20.548388+02:00"

README.md

diff --git a/README.md b/README.md
index 3fa7d7b..504894e 100755
--- a/README.md
+++ b/README.md
@@ -5,8 +5,8 @@
 ## TL;DR;

 ```console
-$ helm repo add bitnami https://charts.bitnami.com/bitnami
-$ helm install my-release bitnami/ghost
+$ helm repo add myrepo http://localhost:8080
+$ helm install my-release myrepo/ghost
 ```

 ## Introduction
@@ -29,7 +29,7 @@ Bitnami charts can be used with [Kubeapps](https://kubeapps.com/) for deployment
 To install the chart with the release name `my-release`:

 ```console
-$ helm install my-release bitnami/ghost
+$ helm install my-release myrepo/ghost
 ```

 The command deploys Ghost on the Kubernetes cluster in the default configuration. The [Parameters](#parameters) section lists the parameters that can be configured during installation.
@@ -168,7 +168,7 @@ Specify each parameter using the `--set key=value[,key=value]` argument to `helm
 ```console
 $ helm install my-release \
   --set ghostUsername=admin,ghostPassword=password,mariadb.mariadbRootPassword=secretpassword \
-    bitnami/ghost
+    myrepo/ghost
 ```

 The above command sets the Ghost administrator account username and password to `admin` and `password` respectively. Additionally, it sets the MariaDB `root` user password to `secretpassword`.
@@ -176,7 +176,7 @@ The above command sets the Ghost administrator account username and password to
 Alternatively, a YAML file that specifies the values for the above parameters can be provided while installing the chart. For example,

 ```console
-$ helm install my-release -f values.yaml bitnami/ghost
+$ helm install my-release -f values.yaml myrepo/ghost
 ```

In order to obtain these diffs check the developer docs.

Deploy to Kubernetes

Visit this guide to deploy a Kubernetes CronJob that will keep two Helm Chart repositories synced.

How to build

Check the developer docs.

License

Copyright © 2023 VMware, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.

You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

More Repositories

1

charts

Bitnami Helm Charts
Smarty
8,689
star
2

containers

Bitnami container images
Shell
3,113
star
3

minideb

A small image based on Debian designed for use in containers
Shell
2,010
star
4

vms

Bitnami VMs
206
star
5

readme-generator-for-helm

Auto generate READMEs for Helm Charts
JavaScript
206
star
6

kube-libsonnet

Bitnami's jsonnet library for building Kubernetes manifests
Jsonnet
173
star
7

bitnami-docker-redis-cluster

Bitnami Docker Image for Redis Cluster
Shell
131
star
8

bitnami-docker-openldap

Bitnami Docker Image for OpenLDAP
Shell
84
star
9

gonit

Drop-in replacement for monit written in Go, with Apache 2.0 license
Go
78
star
10

bitnami-docker-keycloak

Bitnami Docker Image for Keycloak
Shell
65
star
11

bitnami-docker-pgbouncer

Bitnami Docker Image for PgBouncer
Shell
44
star
12

wait-for-port

CLI tool for waiting until a TCP port reaches the desired state
Go
35
star
13

render-template

CLI tool for rendering templates with custom data
Go
27
star
14

azure-marketplace-charts

Helm Charts ready to launch on Azure Kubernetes Serice (AKS)
Mustache
27
star
15

ini-file

CLI tool for modifying .ini files
Go
23
star
16

charts-docs

Helm charts docs
HTML
20
star
17

healthcheck-tools

Set of Go tools to check different elements of your stack (SSL, SMTP, Permissions...)
Go
16
star
18

vulndb

The Bitnami Vulnerability Database
Shell
16
star
19

bitnami-docker-thanos

Bitnami Docker Image for Thanos
Dockerfile
14
star
20

bitnami-docker-golang

Bitnami Docker Image for Golang
Dockerfile
11
star
21

bitnami-docker-nginx-ldap-auth-daemon

Shell
11
star
22

bitnami-docker-ejbca

Bitnami Docker Image for EJBCA
Shell
10
star
23

bitnami-docker-openresty

Bitnami Docker Image for OpenResty
Shell
9
star
24

bitnami-docker-rclone

Bitnami Docker Image for rClone
Dockerfile
9
star
25

bitnami-docker-kubernetes-event-exporter

Bitnami Docker Image for Kubernetes Event Exporter
Dockerfile
8
star
26

bitnami-docker-aws-cli

Bitnami Docker Image for AWS CLI
Dockerfile
6
star
27

bitnami-docker-envoy

Bitnami Docker Image for Envoy
Dockerfile
5
star
28

bncert

Bitnami HTTPS Configuration Tool
Shell
5
star
29

bitnami-docker-keycloak-config-cli

Bitnami Docker Image for keycloak-config-cli
Dockerfile
5
star
30

bndiagnostic

Bitnami Diagnostic Tool
Shell
5
star
31

bitnami-docker-dotnet-sdk

Bitnami Docker Image for .NET SDK
Shell
4
star
32

bitnami-docker-spring-cloud-dataflow

Bitnami Docker Image for Spring Cloud Data Flow
Shell
4
star
33

bitnami-docker-grafana-image-renderer

Bitnami Docker Image for Grafana Image Renderer
Dockerfile
3
star
34

bitnami-docker-grafana-operator

Bitnami Docker Image for Grafana Operator
Dockerfile
3
star
35

bitnami-docker-jupyterhub

Bitnami Docker Image for JupyterHub
Dockerfile
3
star
36

bitnami-docker-argo-cd

Bitnami Docker Image for Argo CD
Shell
3
star
37

bitnami-docker-cert-manager-webhook

Bitnami Docker Image for Cert Manager Webhook
Dockerfile
3
star
38

bitnami-docker-azure-cli

Bitnami Docker Image for Azure CLI
Dockerfile
3
star
39

bitnami-docker-jupyter-base-notebook

Bitnami Docker Image for Jupyter Base Notebook
Dockerfile
3
star
40

bitnami-docker-wal-g

Dockerfile
3
star
41

bitnami-docker-haproxy-intel

Dockerfile
2
star
42

bitnami-docker-promtail

Dockerfile
2
star
43

bitnami-docker-trivy

Bitnami Docker Image for Trivy
Dockerfile
2
star
44

bitnami-docker-google-cloud-sdk

Bitnami Docker Image for Google Cloud SDK
Dockerfile
2
star
45

bitnami-docker-gitlab-runner

Bitnami Docker Image for Gitlab Runner
Shell
2
star
46

bitnami-docker-dotnet

Bitnami Docker Image for .NET
Shell
2
star
47

bitnami-docker-configurable-http-proxy

Bitnami Docker Image for Configurable HTTP Proxy
Dockerfile
2
star
48

bitnami-docker-haproxy

Bitnami Docker Image for HAProxy
Dockerfile
2
star
49

bitnami-docker-kiam

Bitnami Docker Image for Kiam
Dockerfile
2
star
50

bitnami-docker-pytorch-intel

Shell
2
star
51

bitnami-docker-grafana-loki

Dockerfile
2
star
52

bitnami-docker-tensorflow-intel

Bitnami Docker Image for TensorFlow for Intel
Shell
2
star
53

bitnami-docker-harbor-adapter-trivy

Bitnami Docker Image for Harbor Adapter Trivy
Shell
2
star
54

bitnami-docker-phppgadmin

Bitnami Docker Image for phpPgAdmin
Shell
2
star
55

bitnami-docker-rabbitmq-cluster-operator

Bitnami Docker Image for RabbitMQ Cluster Operator
Dockerfile
1
star
56

bitnami-docker-wordpress-intel

Bitnami Docker Image for WordPress for Intel
Shell
1
star
57

bitnami-docker-rmq-default-credential-updater

Bitnami Docker Image for RabbitMQ Default User Credential Updater
Dockerfile
1
star
58

bitnami-docker-python-snapshot

Bitnami Docker Image for Python using snapshots for the system packages repositories
Dockerfile
1
star
59

bitnami-docker-contour

Bitnami Docker Image for Contour
Dockerfile
1
star
60

bitnami-docker-dataplatform-exporter

Bitnami Docker Image for Data Platform BP 1
Dockerfile
1
star
61

bitnami-docker-kong-ingress-controller

Bitnami Docker Image for Kong Ingress Controller
Dockerfile
1
star
62

bitnami-docker-gitlab-runner-helper

Bitnami Docker Image for Gitlab Runner Helper
Shell
1
star
63

bitnami-docker-aspnet-core

Bitnami Docker Image for ASP.NET
Shell
1
star
64

bitnami-docker-sealed-secrets

Dockerfile
1
star
65

bitnami-docker-node-snapshot

Bitnami Docker Image for Node.js using snapshots for the system packages repositories
Dockerfile
1
star
66

bitnami-docker-gradle

Bitnami Docker Image for Gradle
Dockerfile
1
star
67

bitnami-docker-cosign

Bitnami Docker Image for Cosign
Dockerfile
1
star
68

bitnami-docker-argo-workflow-exec

Bitnami Docker Image for Argo Workflow Executor
Shell
1
star
69

bitnami-docker-cainjector

Bitnami Docker Image for CA Injector
Dockerfile
1
star
70

bitnami-docker-inc-intel

Shell
1
star
71

bitnami-docker-grafana-tempo

Bitnami Docker Image for Grafana Tempo
Dockerfile
1
star
72

bitnami-docker-kubeapps-kubeops

Bitnami Docker Image for Kubeapps Kubeops
Dockerfile
1
star
73

bitnami-docker-percona-xtrabackup

Bitnami Docker Image for Percona XtraBackup
Dockerfile
1
star
74

chart-categories-action

A GitHub action to check Helm Charts categories
Shell
1
star
75

bitnami-docker-rmq-messaging-topology-operator

Bitnami Docker Image for RabbitMQ Messaging Topology Operator
Dockerfile
1
star