• Stars
    star
    183
  • Rank 210,154 (Top 5 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created over 7 years ago
  • Updated 5 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Repository that hosts Bazel toolchain configs for remote execution and related support tools.
Bazel CI
Build status

bazel-toolchains

https://github.com/bazelbuild/bazel-toolchains is a repository where Google hosts the source code for a CLI tool that can be used to generate Bazel toolchain configs. These configs are required to configure Bazel to issue commands that will execute inside a Docker container via a remote execution environment.

These toolchain configs include:

  • C/C++ CROSSTOOL file,
  • BUILD file with toolchain rules, and
  • wrapper scripts.

rbe_configs_gen - CLI Tool to Generate Configs

rbe_configs_gen is a CLI tool written in Go that can be used to generate toolchain configs for a given combination of Bazel release and docker image. The output of the tool are toolchain configs in one or more of the following formats:

  • Tarball
  • Config files copied directly to a local directory

rbe_configs_gen requires docker to be installed locally and internet access to work.

Config users are recommended to use the CLI tool to generate and self host their own configs. Pre-generated configs will be provided for new releases of Bazel & the RBE Ubuntu 16.04 without any SLOs. See Pre-generated Configs section below for details.

The rest of this section describes how to use the rbe_configs_gen tool.

Building

Building using Docker on Linux (Recommended)

Use the official Golang docker image to build the rbe_configs_gen binary using Go 1.16. This avoids having to install the Go toolchain locally but requires docker.

  1. Clone this repository and set it as the working directory:
$ git clone https://github.com/bazelbuild/bazel-toolchains.git
$ cd bazel-toolchains
  1. Run the following command:
$ docker run --rm -v $PWD:/srcdir -w /srcdir golang:1.16 go build -o rbe_configs_gen ./cmd/rbe_configs_gen/rbe_configs_gen.go
  1. Run rbe_configs_gen as follows to see the flags it accepts:
$ ./rbe_configs_gen --help

Building Locally

  1. Install Go for your platform if necessary. Tested to work with Go 1.16.

  2. Clone this repository

$ git clone https://github.com/bazelbuild/bazel-toolchains.git
$ cd bazel-toolchains
  1. Build the rbe_configs_gen executable
# Use -o rbe_configs_gen.exe on Windows
$ go build -o rbe_configs_gen ./cmd/rbe_configs_gen/rbe_configs_gen.go
  1. Run rbe_configs_gen as follows to see the flags it accepts:
# On Linux
$ ./rbe_configs_gen --help

# On Windows
$ rbe_configs_gen.exe

Generating Configs

Latest Bazel Version and Output Tarball

If you'd like to generate toolchain configs for the latest available Bazel release and the toolchain container l.gcr.io/google/rbe-ubuntu16-04:latest and produce a tarball with the generated configs run:

$ ./rbe_configs_gen \
    --toolchain_container=l.gcr.io/google/rbe-ubuntu16-04:latest \
    --output_tarball=rbe_default.tar \
    --exec_os=linux \
    --target_os=linux

The exec_os and target_os correspond to the Bazel execution & target platforms respectively.

You should see a tarball file rbe_default.tar locally containing the generated configs.

Specific Bazel Version and Output Directory

If you'd like to generate toolchain configs for a specific Bazel release, e.g., Bazel 4.0.0 (tested for versions >= 3.7.2) and the toolchain container l.gcr.io/google/rbe-ubuntu16-04:latest and copy the generated configs to path configs/path relative to a source repository at /path/to/source/repo run:

$ ./rbe_configs_gen \
    --bazel_version=4.0.0 \
    --toolchain_container=l.gcr.io/google/rbe-ubuntu16-04:latest \
    --output_src_root=/path/to/source/repo \
    --output_config_path=configs/path \
    --exec_os=linux \
    --target_os=linux

/path/to/source/repo should be the directory containing a Bazel WORKSPACE file. The toolchain configs will be extracted to /path/to/source/repo/configs/path.

The exec_os and target_os correspond to the Bazel execution & target platforms respectively.

Using Configs

.bazelrc

Copy/import a .bazelrc file from here. Pick the file that has the highest Bazel version in the filename that's less than or equal to the Bazel version you're using.

Option 1: Same Source Repository (Recommended)

If you copied the generated configs to the source repository where the rest of your code lives, and assuming the configs were copied to the path configs/path (i.e., the value specified to the flag --output_config_path when running rbe_configs_gen) relative to the directory containing the Bazel WORKSPACE file, all you need to do is replace all occurences of @rbe_default// in your .bazelrc file with //configs/path.

Option 2: Remote Github Repository

If you extract the contents of a generated toolchain configs tarball into the root of a Github repository e.g. github.com/example/configs-repo where this repository hosting the configs is different from the source repository where you'd like to use the configs, include the following in your WORKSPACE:

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")

git_repository(
    name = "rbe_default",
    # Replace this with the actual commit id of the Github repo you'd like to pin to.
    commit = "471da0273050b88d77529484ff89741ff586f9f5",
    remote = "https://github.com/example/configs-repo.git",
)

Option 3: Remote Tarball Archive

Then, assuming you've upload the toolchain configs tarball to a remote location available at the URL https://example.com/rbe-default.tar, include the following in your WORKSPACE file:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "rbe_default",
    sha256 = "<replace this with the 64 character sha256 digest of the configs tarball>",
    urls = ["https://example.com/rbe-default.tar"],
)

Custom Execution Properties

Certain remote execution backends support custom options such as selecting the VM machine type remote actions run on, configuring certain docker properties if the remote actions are executed in docker containers such as network access, privileged execution, allocated memory, etc. Bazel passes on any property specified to the exec_properties attribute to a platform definition to the underlying remote execution system.

If you're using RBE, continue reading to see how to specify custom execution properties.

First, in your WORKSPACE file, import the latest commit of this repository (replace the commit ID and sha256 digest with latest commit if necessary):

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
	name = "bazel_toolchains",
	urls = ["https://github.com/bazelbuild/bazel-toolchains/archive/dac71231098d891e5c4b74a2078fe9343feef510.tar.gz"],
	strip_prefix = "bazel-toolchains-dac71231098d891e5c4b74a2078fe9343feef510",
	sha256 = "56d5370eb99559b4c74f334f81bc8a298f728bd16d5a4333c865c2ad10fae3bc",
)

load("@bazel_toolchains//repositories:repositories.bzl", bazel_toolchains_repositories = "repositories")
bazel_toolchains_repositories()

Then declare a custom platform in a BUILD file. For now, let's assume this is the BUILD file at the root of your source repository (i.e., the BUILD file & WORKSPACE file are in the same directory):

load("@bazel_toolchains//rules/exec_properties:exec_properties.bzl", "create_rbe_exec_properties_dict")

platform(
	name = "custom_platform",
    # Inherit from the platform target generated by 'rbe_configs_gen' assuming the generated configs
    # were imported as a Bazel external repository named 'rbe_default'. If you extracted the
    # generated configs elsewhere in your source repository, replace the following with the label
    # to the 'platform' target in the generated configs.
	parents = ["@rbe_default//config:platform"],
    # Example custom execution property instructing RBE to use e2-standard-2 GCE VMs.
	exec_properties = create_rbe_exec_properties_dict(
		gce_machine_type = "e2-standard-2",
	),
)

See here for a list of parameters accepted by create_rbe_exec_properties_dict.

Finally, in your .bazelrc file, replace all options specifying a platform target with the above custom platform target instead. So for example, if your .bazelrc previously looked like

...
build:remote --extra_execution_platforms=@rbe_default//config:platform
build:remote --host_platform=@rbe_default//config:platform
build:remote --platforms=@rbe_default//config:platform
...

It should now look like

build:remote --extra_execution_platforms=//:custom_platform
build:remote --host_platform=//:custom_platform
build:remote --platforms=//:custom_platform

Pre-generated Configs

Pre-generated configs tarballs will be generated for every Bazel release starting with 4.0.0 & the latest available Ubuntu 16.04 Clang + JDK container and uploaded to GCS.

IMPORTANT: Ensure you read & agree with the terms of the LICENSE file included in the configs tarball before using pre-generated configs.

Basically, never depend directly on any of the URLs mentioned below to download toolchain configs in production because they may break without warning. Pre-generated configs are only provided as a convenience for experimenting with configuring Bazel for remote builds. Further, there are no guarantees on how long after a new release of Bazel or the Ubuntu 16.04 container mentioned above the corresponding pre-generated configs will be available. It's strongly recommended to generate and host your own toolchain configs by running the rbe_config_gen tool and test the functionality and correctness of the configs yourself before using them in production. Alternatively, you could also copy pre-generated configs and host it in a location you control after verifying correctness before using them in production.

See here for instructions on how to initialize your .bazelrc file.

Latest Bazel and Latest Ubuntu 16.04 Container

  1. Examine the contents of the JSON manifest of the latest configs.
$ curl https://storage.googleapis.com/rbe-toolchain/bazel-configs/rbe-ubuntu1604/latest/manifest.json
{
 "bazel_version": "4.0.0",
 "toolchain_container": "l.gcr.io/google/rbe-ubuntu16-04:latest",
 "image_digest": "f6568d8168b14aafd1b707019927a63c2d37113a03bcee188218f99bd0327ea1",
 "exec_os": "Linux",
 "configs_tarball_digest": "c0d428774cbe70d477e1d07581d863f8dbff4ba6a66d20502d7118354a814bea",
 "upload_time": "2021-02-18T06:02:32.997892223-08:00"
}
  1. The manifest indicates the configs are for Bazel 4.0.0, generated for the container l.gcr.io/google/rbe-ubuntu16-04@sha256:f6568d8168b14aafd1b707019927a63c2d37113a03bcee188218f99bd0327ea1 and the sha256 digest of the uploaded configs tarball is c0d428774cbe70d477e1d07581d863f8dbff4ba6a66d20502d7118354a814bea. To use these configs, add the following to your Bazel WORKSPACE file:
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "rbe_default",
    # Change the sha256 digest to the value of the `configs_tarball_digest` in the manifest you
    # got when you ran the curl command above.
    sha256 = "c0d428774cbe70d477e1d07581d863f8dbff4ba6a66d20502d7118354a814bea",
    urls = ["https://storage.googleapis.com/rbe-toolchain/bazel-configs/rbe-ubuntu1604/latest/rbe_default.tar"],
)

Specific Bazel and Latest Ubuntu 16.04 Container

  1. Say you'd like to use configs for Bazel 4.0.0 specifically.

  2. Check if a manifest exists for the Bazel version you're interested in (version should be >= 4.0.0).

# Replace "bazel_4.0.0" in the URL below with whatever "bazel_<version>" you'd like to you.
$ curl https://storage.googleapis.com/rbe-toolchain/bazel-configs/bazel_4.0.0/rbe-ubuntu1604/latest/manifest.json
{
 "bazel_version": "4.0.0",
 "toolchain_container": "l.gcr.io/google/rbe-ubuntu16-04:latest",
 "image_digest": "f6568d8168b14aafd1b707019927a63c2d37113a03bcee188218f99bd0327ea1",
 "exec_os": "Linux",
 "configs_tarball_digest": "c0d428774cbe70d477e1d07581d863f8dbff4ba6a66d20502d7118354a814bea",
 "upload_time": "2021-02-18T06:02:32.997892223-08:00"
}
  1. The manifest confirms the configs are for Bazel 4.0.0, generated for the container l.gcr.io/google/rbe-ubuntu16-04@sha256:f6568d8168b14aafd1b707019927a63c2d37113a03bcee188218f99bd0327ea1 and the sha256 digest of the uploaded configs tarball is c0d428774cbe70d477e1d07581d863f8dbff4ba6a66d20502d7118354a814bea. To use these configs, add the following to your Bazel WORKSPACE file:
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "rbe_default",
    # Change the sha256 digest to the value of the `configs_tarball_digest` in the manifest you
    # got when you ran the curl command above.
    sha256 = "c0d428774cbe70d477e1d07581d863f8dbff4ba6a66d20502d7118354a814bea",
    # Change "bazel_4.0.0" in the URL below with whatever "bazel_<version>" you downloaded the
    # manifest for in the previous step.
    urls = ["https://storage.googleapis.com/rbe-toolchain/bazel-configs/bazel_4.0.0/rbe-ubuntu1604/latest/rbe_default.tar"],
)

Where is rbe_autoconfig?

The rbe_autoconfig Bazel repository rule used to generate & use toolchain configs has been deprecated with release v4.0.0 of this repository being the last release that supports rbe_autoconfig. Release v4.0.0 supports Bazel versions up to 4.0.0.

More Repositories

1

bazel

a fast, scalable, multi-language and extensible build system
Java
22,794
star
2

starlark

Starlark Language
Starlark
2,435
star
3

bazelisk

A user-friendly launcher for Bazel.
Go
2,060
star
4

rules_go

Go rules for Bazel
Go
1,356
star
5

bazel-gazelle

Gazelle is a Bazel build file generator for Bazel projects. It natively supports Go and protobuf, and it may be extended to support new languages and custom rule sets.
Go
1,156
star
6

rules_docker

Rules for building and handling Docker images with Bazel
Starlark
1,074
star
7

buildtools

A bazel BUILD file formatter and editor
Go
1,007
star
8

examples

Examples for Bazel
Starlark
831
star
9

intellij

IntelliJ plugin for Bazel projects
Java
762
star
10

rules_rust

Rust rules for Bazel
Starlark
660
star
11

bazel-buildfarm

Bazel remote caching and execution service
Java
650
star
12

tulsi

An Xcode Project Generator For Bazel
Swift
547
star
13

rules_python

Bazel Python Rules
Starlark
521
star
14

rules_apple

Bazel rules to build apps for Apple platforms.
Starlark
509
star
15

bazel-watcher

Tools for building Bazel targets when source files change.
Go
440
star
16

bazel-skylib

Common useful functions and rules for Bazel
Starlark
393
star
17

sandboxfs

A virtual file system for sandboxing
Rust
371
star
18

rules_scala

Scala rules for Bazel
Starlark
360
star
19

rules_kotlin

Bazel rules for Kotlin
Kotlin
331
star
20

remote-apis

An API for caching and execution of actions on a remote system.
Go
328
star
21

rules_swift

Bazel rules to build Swift on Apple and Linux platforms
Starlark
311
star
22

rules_k8s

This repository contains rules for interacting with Kubernetes configurations / clusters.
Starlark
289
star
23

rules_typescript

MOVED to https://github.com/bazelbuild/rules_nodejs/tree/3.x/third_party/github.com/bazelbuild/rules_typescript
TypeScript
275
star
24

continuous-integration

Bazel's Continuous Integration Setup
Python
258
star
25

bazel-central-registry

The central registry of Bazel modules for the Bzlmod external dependency system.
Starlark
247
star
26

rules_pkg

Bazel rules for creating packages of many types (zip, tar, deb, rpm, ...)
Starlark
225
star
27

rules_cc

C++ Rules for Bazel
Starlark
184
star
28

rules_android

Android rules for Bazel
Starlark
175
star
29

rules_proto

Protocol buffer rules for Bazel
Starlark
163
star
30

rules_closure

Closure rules for Bazel
Java
153
star
31

vim-bazel

Vim support for Bazel
Vim Script
144
star
32

platforms

Constraint values for specifying platforms and toolchains
Starlark
108
star
33

stardoc

Stardoc: Starlark Documentation Generator
Java
107
star
34

proposals

Index of all Bazel proposals and design documents
106
star
35

rules_webtesting

Bazel rules to allow testing against a browser with WebDriver.
Go
96
star
36

apple_support

Apple support for Bazel rules
Starlark
83
star
37

emacs-bazel-mode

Emacs mode for Bazel
Emacs Lisp
80
star
38

rules_license

Starlark
77
star
39

rules_java

Java rules for Bazel
Starlark
72
star
40

reclient

Go
65
star
41

setup-bazelisk

Set up your GitHub Actions workflow with a specific version of Bazelisk
TypeScript
56
star
42

rules_sass

Sass rules for Bazel
Starlark
51
star
43

remote-apis-sdks

This repository contains client libraries for the Remote Execution API https://github.com/bazelbuild/remote-apis
Go
48
star
44

bazel-federation

Starlark
47
star
45

skydoc

Documentation generator for Skylark
Python
46
star
46

migration-tooling

Migration tools for Bazel
Java
44
star
47

codelabs

Shell
39
star
48

BUILD_file_generator

Generate BUILD files for your Java files
Java
39
star
49

eclipse

Eclipse For Bazel (deprecated, see https://github.com/salesforce/bazel-eclipse instead)
Java
32
star
50

bazel-integration-testing

Framework for integration tests that call Bazel
Java
32
star
51

rules_appengine

AppEngine rules for Bazel
Starlark
30
star
52

rules_android_ndk

Starlark
29
star
53

tools_remote

Java
28
star
54

bazel-bench

Benchmarking tool for bazel
Python
27
star
55

vim-ft-bzl

Vim Script
27
star
56

rules_perl

Perl rules for Bazel
Starlark
25
star
57

tools_android

Tools for use with building Android apps with Bazel
Starlark
24
star
58

homebrew-tap

This repository contains a collection of Homebrew (aka, Brew) "formulae" for Bazel
24
star
59

rules_d

D rules for Bazel
Starlark
24
star
60

bazel-blog

Content of the Bazel blog
HTML
20
star
61

bzlmod

Go
20
star
62

rules_testing

Starlark testing framework and utility libraries
Starlark
20
star
63

rules_gwt

Bazel rules for GWT
Starlark
19
star
64

bazel-website

Website for Bazel, a fast, scalable, multi-language and extensible build system
HTML
17
star
65

bazelcon

Artifacts from BazelCon
15
star
66

java_tools

Python
12
star
67

rules_groovy

Groovy rules for Bazel
Starlark
11
star
68

rules_postcss

PostCSS rules for Bazel
Starlark
10
star
69

rules_platform

Starlark
8
star
70

bazel_metrics

Python
7
star
71

community

Resources for community management efforts, such as SIGs
6
star
72

rules_angular

OBSOLETE see https://github.com/angular/angular/tree/master/packages/bazel
Python
5
star
73

gmaven_rules

This repository is deprecated. Please instead use https://github.com/bazelbuild/rules_jvm_external
Python
3
star
74

rules_utp

Starlark
2
star
75

.allstar

1
star
76

.github

1
star
77

bazel-worker-api

Java
1
star
78

build-event-stream

1
star
79

rules_sh

1
star