• Stars
    star
    121
  • Rank 293,924 (Top 6 %)
  • Language
    Java
  • Created almost 14 years ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Jenkins Update Center backend

Jenkins Update Center Generator

This project is primarily used to generate the jenkins.io update center layout.

With a few modifications it could easily be used to generate your corporate update center as well.

Important
This tool is optimized for use by the Jenkins project in Jenkins project infrastructure. Expect incompatible changes, like options not used by the Jenkins project being dropped. Additionally, this documentation is focused on how the tool is used by the Jenkins project, rather than the customizations necessary to create private update sites.

Output

See site layout for more detailed documentation on site structure.

Information Sources

The generator pulls information from:

  • Artifactory artifact repository (see ArtifactoryRepositoryImpl.java)

    • Artifactory API for listing artifacts

    • downloading artifact files

    • downloading individual files in archives such as MANIFEST.MF

  • GitHub (see GitHubSource.java)

    • determine source code repository that actually exists (sometimes metadata is wrong)

    • plugin labels from repositories topics

    • default branch name

  • Jenkins usage statistics (see Popularities.java)

    • latest plugin installation numbers for popularity entries in update center JSON

  • Various metadata JSON files on reports.jenkins.io

    • Plugin issue tracker metadata JSON file, see IssueTrackerSource.java

    • Plugin maintainer mapping JSON file, see MaintainersSource.java

    • Maintainer info JSON file, see MaintainersSource.java

  • Local resource files in this repository

    • GitHub topic allowlist (resources/allowed-github-topics.properties)

    • Standalone deprecations (resources/deprecations.properties)

    • Artifact ignore list (with deprecation URLs) (resources/artifact-ignores.properties)

    • Label assignments (resources/label-definitions.properties)

    • Security warnings (resources/warnings.json)

    • Plugin URL overrides (resources/wiki-overrides.properties)

Features Controlled Through Resource Files

Categorizing plugins

Plugin labels are shown to users:

  • as categories in Jenkins before 2.224 and on the plugins site

  • as tags in Jenkins from 2.224 and also on the plugins site.

See this Jenkins resource file (look for UpdateCenter.PluginCategory) for the localization overrides applied to labels by Jenkins. Other labels are categorized into general Misc (custom-label-here) categories (Jenkins before 2.224) or displayed as is.

Two ways can be used to define these labels:

GitHub

Add topics to your GitHub repository. For a list of supported topics, see the resource file that contains all topics that can be set on GitHub repositories that will be reflected in update sites. Topics can be set with or without the prefix jenkins-. If a topics has that prefix, it is removed first: To add the label matrix for your plugin, you would add either matrix or jenkins-matrix on your repository.

Resource File

As an alternative to the above, plugin labels can be defined in the file resources/label-definitions.properties in this repository.

This is the preferable approach when a plugin isn’t in the jenkinsci GitHub organization, or a GitHub repository contains multiple plugins whose labels should be different.

Plugin URL Override

Plugins are generally expected to provide a <url> to their documentation in their POM. Historically, these URLs have been pages on the Jenkins wiki, but can point anywhere.

This requirement no longer exists, but it may still be useful to define a documentation URL for plugins that do not specify the correct URL.

The file resources/wiki-overrides.properties defines these wiki page overrides.

Any entries in this file are expected to be temporary, i.e. plugin maintainers are expected to fix their plugin’s metadata.

Deprecations

Plugins are considered deprecated by Jenkins 2.246 and newer when the update site metadata does one or both of the following:

  • Uses the label deprecated for the plugin. This can be done via GitHub repository topics, or the resources/label-definitions.properties described above. Jenkins will use the plugin URL as the reference URL for the deprecation notice.

  • Lists an entry with the plugin ID as key in the top-level deprecations map in update-center.json. This can be done through entries in the resources/deprecations.properties file (for plugins that continue being distributed) or in resources/artifact-ignores.properties for plugins that are suspended. The value from the properties file will be used as the URL for the deprecation notice in Jenkins. This entry and URL take precedence over a deprecated label, i.e. when both are set, the URL from the top-level element shall be used.

These two different approaches to plugin deprecation accomplish complementary goals:

  • The label approach is very simple and can easily be done by plugin maintainers themselves via GitHub labels. It is also backward compatible with any earlier version of Jenkins — it will just show the deprecation as a regular label. Additionally, it doesn’t bloat the JSON file size at all, since no special URL is needed.

  • The top-level deprecations element allows specifying a URL different from the plugin documentation URL as well as deprecating plugins no longer being distributed. Especially the latter is a common requirement when plugins integrate with services that no longer exist: It makes no sense to continue distributing them, but everyone having them already installed should be informed about it.

Removing plugins from distribution

The update center generator allows to specify that certain plugins, or plugin releases, should not be included in the output.

There are various reasons to need to do this, such as:

  • A plugin release causes major regressions and a fix is not immediately available.

  • A plugin integrates with a service that has been shut down.

Both use cases (entire plugins, or specific versions) are controlled via the file resources/artifact-ignores.properties. That file may also define URL of a deprecation notice that is shown to users who already installed the plugin. See that file for usage examples.

Security warnings

Since Jenkins 2.32.2 and 2.40, Jenkins can display security warnings about core and plugins. These warnings are part of the update center metadata downloaded by Jenkins. These warnings are defined in the file resources/warnings.json.

Usage

Invocation

Build (mvn clean verify) the generator and then invoke it as follows:

java -Dfile.encoding=UTF-8 -jar target/update-center2-*-SNAPSHOT-bin/update-center2-*-SNAPSHOT.jar --id default ...

The tool also supports batch mode execution, generating multiple update sites with a single invocation:

java -Dfile.encoding=UTF-8 -jar target/update-center2-*-SNAPSHOT-bin/update-center2-*-SNAPSHOT.jar --arguments-file <filename.txt>

filename.txt is a text file with a list of arguments on each line. Lines that start with # are comments and ignored. Example:

# one update site per line

# Minimal update sites for Jenkins <= 2.204 and 2.204.x LTS
--www-dir ./www2/2.204 --limit-plugin-core-dependency 2.204.999 --write-latest-core
--www-dir ./www2/stable-2.204 --limit-plugin-core-dependency 2.204.999 --write-latest-core --only-stable-core

# Minimal update sites for Jenkins <= 2.222 and 2.222.x LTS
--www-dir ./www2/2.222 --limit-plugin-core-dependency 2.222.999 --write-latest-core
--www-dir ./www2/stable-2.222 --limit-plugin-core-dependency 2.222.999 --write-latest-core --only-stable-core

# Experimental (alpha/beta) update site, no version caps, collect files for download (including experimental files)
--www-dir ./www2/experimental --with-experimental --downloads-directory ./download

# Latest update site for Jenkins > 2.222, with release-history.json, plugin-versions.json, plugin-documentation-urls.json, collect files for download, and generate plugin count
--generate-release-history --generate-plugin-versions --generate-plugin-documentation-urls --write-latest-core --write-plugin-count --www-dir ./www2/current --download-links-directory ./www2/download --downloads-directory ./download --latest-links-directory ./www2/current/latest

For a full list of arguments, invoke the tool as follows:

java -Dfile.encoding=UTF-8 -jar target/update-center2-*-SNAPSHOT-bin/update-center2-*-SNAPSHOT.jar --help
Note
--help isn’t a real argument, but usage instructions are printed when an invalid argument is provided.

Preparing local execution

Running ./site/generate.sh will first create the batch mode control file ./tmp/args.lst, before actually starting the tool. The following steps are therefore useful when trying to generate output corresponding to the real update sites during development:

  1. Implement changes in src/main/.

  2. Run ./site/generate.sh until the Java tool is actually launched, then abort. This requires some environment variables to be defined.

  3. Edit tmp/args.lst, changing or removing the --key, --certificate, and --root-certificate arguments as necessary.

  4. Run java -Dfile.encoding=UTF-8 -jar target/update-center2-*-SNAPSHOT-bin/update-center2-*-SNAPSHOT.jar --arguments-file tmp/args.lst

Alternatively, the closest you can get to real executions in local development:

  1. Implement changes in src/main/.

  2. Deploy a snapshot using mvn deploy. Requires an account in the Jenkins project, see Deploying changes below.

  3. Edit site/generate.sh to reference the specific snapshot you deployed (including timestamp) where it is downloaded using wget, see previous build output.

  4. Optionally, to speed things up, edit site/generate.sh and remove the arguments --downloads-directory "$DOWNLOAD_ROOT_DIR" from some of the invocations.

  5. Run ./site/generate.sh <www-dir> <downloads-dir>. The first argument is the output directory for metadata, the second argument is the output directory for downloads and unused unless the previous step 4 was skipped.

Running within an IDE

The project various artifacts to be used on a site hosting a jenkins update center The project produces a jar and a zip file containing all the required dependencies to run the generator.

If you want to run the generator from within your development environment, you can try to use the appassembler plugin as described below. The exec:java plugin won’t work.

mvn package appassembler:assemble
sh target/appassembler/bin/app --id default ...

Deploying changes

./site/generate.sh downloads and executes a specified version of update-center2. This is different from earlier iterations of this tool that always rebuilt from source. The current iteration requires a (possible snapshot deployment) first, that is then referenced in ./site.generate.sh.

Consequently, merging larger-scale changes to both the tool itself and the wrapper script need to be mindful of this dependency: A new release (or at minimum a snapshot deployment) is needed, which is then referenced in ./site/generate.sh.

Note
As of May 2020, everyone can deploy snapshots to Artifactory, so permissions issues shouldn’t hinder development.

Working with htaccess/mod_rewrite rules

The wrapper script site/generate.sh calls the script site/generate-htaccess.sh with chosen arguments. The latter script will generate the .htaccess file mostly containing mod_rewrite rules to redirect requests to appropriate tiered update sites. To learn more about tiers, see LAYOUT.md.

To test changes to site/generate-htaccess.sh, run site/test/test.sh. It executes site/generate-htaccess.sh and places it inside an Apache HTTPD Docker container and tests whether redirect rules are correctly applied.

Working with certificates

To sign JSON output files, create a development certificate:

openssl genrsa -out resources/certificates/demo.key 4096
openssl req -new -x509 -days 180 -key resources/certificates/demo.key -out resources/certificates/demo.crt -subj "/C=/ST=/L=/O=local-development/OU=local-development/CN=local-development/[email protected]"

Then add these arguments to your tool invocation (or arguments file):

--key resources/certificates/demo.key --certificate resources/certificates/demo.crt --root-certificate resources/certificates/demo.crt

To have your Jenkins instance accept update site JSON signed with this certificate, create a directory update-center-rootCAs/ in the Jenkins home directory, and copy the demo.crt file in there. Once update site JSON files are generated, configure Jenkins to download them in Manage Jenkins » Manage Plugin » Advanced: Either set up a local HTTP server so the URL would be something like http://localhost:8000/update-center.json, or specify a file:// URL like file:///Users/yourname/git/update-center2/www2/update-center.json

Note
For historical reason, the configured URL points to update-center.json, but the file actually downloaded by Jenkins (at least up to 2.235 as of this writing) is update-center.json.html.

More Repositories

1

jenkins.io

A static site for the Jenkins automation server
HTML
315
star
2

pipeline-library

Collection of custom steps and variables for our Jenkins instance(s)
Groovy
288
star
3

jenkins-infra

Jenkins main control repo for R10k and our Puppet Enterprise managed infrastructure
Ruby
108
star
4

evergreen

An automatically self-updating Jenkins distribution
JavaScript
95
star
5

repository-permissions-updater

Artifactory permissions synchronization tool and data set
Java
79
star
6

kubernetes-management

Jenkins Infrastructure Kubernetes Management
Shell
55
star
7

cn.jenkins.io

Chinese version of the website
HTML
34
star
8

azure

Documentation, tooling and other resources related to the Azure account used by the Jenkins project
HCL
27
star
9

crawler

tools crawler
Groovy
26
star
10

plugin-health-scoring

This project aims to introduce a metric system to calculate the health score of each plugin within the Jenkins ecosystem and reflect the final scores on the Plugin Site for the plugin maintainers and users.
Java
24
star
11

ircbot

Jenkins IRCbot
Java
23
star
12

documentation

Documentation for users of Jenkins project infrastructure
22
star
13

plugin-site

This is the frontend application driven by data from the Jenkins Plugin Site API.
JavaScript
20
star
14

wechat

WeChat for Jenkins (Jenkins 官方微信公众号)
Groovy
19
star
15

packer-images

This repository hosts the packer definitions for the Jenkins Infrastructure
Shell
17
star
16

helpdesk

Open your Infrastructure related issues here for the Jenkins project
Shell
16
star
17

pipeline-steps-doc-generator

Pipeline step documentation generator
Java
15
star
18

release

Contains every things needed to release jenkins core from the jenkins infra project
Shell
13
star
19

docker-openvpn

This project contents everything needed to build an openvpn docker image used by the Jenkins Infrastructure Project
Shell
10
star
20

jenkins-codeql

Custom CodeQL code scanning rules for Jenkins
Java
10
star
21

mock-ldap

Mock LDAP container useful for testing.
Dockerfile
9
star
22

plugins-wiki-docs

JavaScript
9
star
23

jira

Docker container to run JIRA
Shell
7
star
24

helm-charts

Helm chart repository containing the jenkins-infra public charts.
Mustache
7
star
25

account-app

Jenkins Account Management/Sign-up App
Java
7
star
26

iep

Infrastructure Enhancement Proposals
7
star
27

aws

Documentation, tooling and other resources related to the Jenkins Infrastructure Project parts hosted in Amazon Web Services (AWS).
HCL
7
star
28

community-functions

Azure Functions to implement community automation for the Jenkins project
JavaScript
6
star
29

datadog

Jenkins OSS infrastructure monitoring stuff
HCL
6
star
30

status

Jenkins Infrastructure Status page https://status.jenkins.io
HTML
6
star
31

docker-jenkins-lts

a docker image containing the latest jenkins lts release and plugins
Shell
6
star
32

confluence

Containerized Confluence that runs wiki.jenkins-ci.org
Shell
6
star
33

backend-extension-indexer

Generate the list of extension points and their known implementations
Java
6
star
34

uc

Go
5
star
35

backend-jenkins-plugin-info-plugin

Jenkins backend: jenkins-plugin-info-plugin
Java
5
star
36

jenkins-io-components

TypeScript
4
star
37

awestruct-ibeams

Archived as it has been inlined in jenkins.io (see https://github.com/jenkins-infra/jenkins.io/pull/6291). Collection of Awestruct extensions to make building sites more structurally sound.
Ruby
4
star
38

asciidoctor-jenkins-extensions

Extensions to asciidoctor syntax for Jenkins-related materials
Ruby
4
star
39

docker-builder

A Docker image providing tools for... building Docker Images (usually in rootless or daemon-less contexts)
Dockerfile
3
star
40

docker-jenkins-weeklyci

a docker image containing the latest jenkins weekly release and plugins
Shell
3
star
41

stories

A static site for the Jenkins is the way
JavaScript
3
star
42

.github

Global configurations for the Jenkins Infrastructure organization on GitHub
3
star
43

acceptance-tests

Infrastructure acceptance testing.
3
star
44

java.net-scm-issue-link

java.net issue update daemon
Java
3
star
45

docker-ldap

OpenLDAP server that runs accounts
Shell
3
star
46

stats.jenkins.io

Revamped Jenkins Infra Stats Website as a part of GSoC 2024
HTML
3
star
47

docker-mirrorbits

This repository hold everything need to mirrorbits based on https://github.com/etix/mirrorbits/
Dockerfile
2
star
48

docker-terraform

A Docker Image to provide a working environment around terraform
Dockerfile
2
star
49

infra-reports

Generate reports based on GitHub and Artifactory data (formerly github-reports)
JavaScript
2
star
50

bind

Docker container that runs DNS server
Dockerfile
2
star
51

javadoc

Groovy
2
star
52

docker-confluence-data

confluence-data is a repository containing confluence data exported to html
HTML
2
star
53

backend-commit-history-parser

Shell
2
star
54

azure-net

Network resources related to the Azure account used by the Jenkins project
HCL
2
star
55

captain-hook

Go
2
star
56

jenkins-wiki-exporter

Quick util to convert jenkins plugin wiki format to github markdown or asciidoc
JavaScript
2
star
57

docker-crond

Simple docker image running crond
Dockerfile
2
star
58

docker-packaging

This repository contains everything needed to build a docker image used to package jenkins core
Dockerfile
2
star
59

docs.jenkins.io

Versioned docs of jenkins.io
CSS
2
star
60

jenkins-usage-stats

Jenkins usage statistics reports generator
Go
1
star
61

digitalocean

Documentation, tooling and other resources related to the Jenkins Infrastructure Project parts hosted in Digital Ocean.
HCL
1
star
62

Enhancing-LLM-with-Jenkins-Knowledge

🚀 this project aims to develop an app using an existing open-source LLM with data collected for domain-specific Jenkins knowledge that can be fine-tuned locally and set up with a proper UI for the user to interact with.
Jupyter Notebook
1
star
63

github-reusable-workflows

Repository for reusable workflows
1
star
64

confluence-cache

Container that runs the pre-generated cache in front of Confluence
Ruby
1
star
65

usage-log-decrypter

Decrypts & produces anonymized summary of usage info
Java
1
star
66

pipeline-metadata-utils

Pipeline step metadata utility classes
Java
1
star
67

patron

Patron of Jenkins program
Groovy
1
star
68

interesting-category-action

Validate that the next draft release is interesting
Shell
1
star
69

docker-keycloak-theme

JavaScript
1
star
70

docker-hashicorp-tools

Dockerfile
1
star
71

infra-mirror

mirror script
Shell
1
star
72

rating

Community rating of releases
PHP
1
star
73

docker-helmfile

Docker Image used to execute Helmfile commands, with all the expected tooling
Dockerfile
1
star
74

jira-ldap-syncer

LDAP to JIRA bridge
Java
1
star
75

backend-confluence-spam-remover

Jenkins backend: confluence-spam-remover
Mathematica
1
star
76

jenkins-maven-cd-action

Shell
1
star
77

backend-release.rss

Generate release history RSS feed
Groovy
1
star
78

shared-tools

This repository hosts the shared tools (between local development and CI libraries) of the Jenkins infrastructure: Makefiles, CLIs, etc.
HCL
1
star
79

l10n-server

Jenkins backend: l10n-server
Java
1
star
80

merge-all-repo

This script merges all the Jenkins repositories into one repository
Groovy
1
star
81

jenkins-security-scan

GitHub Workflow and Action for the Jenkins Security Scan
1
star
82

homebrew-tap

Ruby
1
star
83

jenkins-contribution-stats

Shell
1
star
84

contributor-spotlight

Jenkins Contributor Spotlight feature
JavaScript
1
star