One weird trick for fast CI

Compilers hate this! (Just kidding, compilers are easy-going.)

For many build systems, you have to do a clean build to be sure you’re getting the correct result, so your CI has to always do a clean build. On the other hand, Bazel is designed so that you never have to do a bazel clean (if you ever run bazel clean and get a different answer, please file a bug!).

Thus, to cater to most build tools, CI systems generally run a loop like this:

However, for Bazel, doing a clean build every time wastes a lot of time and resources. For Bazel, the continuous integration system should look like this:

Of course, most build systems actually make this a bit difficult to set up, as you wouldn’t want this kind of configuration for a non-Bazel continuous build! However, Jenkins lets you set it up fairly simply. I’ll go through setting up a Jenkins instance that will build a local codebase every 5 minutes.

First, I’ll make my “source directory:”

$ mkdir jenkins-workspace
$ cd jenkins-workspace
$ touch WORKSPACE

I’m on OS X, so I installed the Jenkins pkg. When it is done installing, it pops up a web page prompts you to create your first job.

Click on the link and put in a name and select “Freestyle Build”, then hit “OK”.

On the next page, select “Advanced”. Select “Use custom workspace” and put in the directory from above.

Set it up to run every 5 minutes:

Screen Shot 2015-10-08 at 2.06.17 PM

Now add the following command as the build step:

This will build all targets in your workspace and then run all tests (so it will find any compile errors as well as test failures). //tools and //third_party contain a bunch of targets that won’t build out-of-the-box, so they’re filtered out using the -//target syntax (see the “Subtractive Patterns” section under Target Patterns).

Note that you don’t have to do any cleanup after the build, since Bazel never pollutes your source directory with build artifacts. You just want to pull, build, pull, build.

Now hit the “Save” button. It’ll take you back to your dashboard and begin the countdown to the first build. Click on the play button in the rightmost column to test it out (or just wait five minutes).

If you make changes, Jenkins will just build/test those changes:

If you haven’t made any changes, Bazel won’t have to do any work at all:

This can be a huge time saver for large projects.

  • ittai zeidman

    How does this fit with distributed caching? I’d like bazel to write to the cache only if the entire chain passes. Is that the behavior?
    Also, in a more realistic setting you would poll the SCM, right?

  • kristina1

    I’m not too familiar with Alpha’s distributed caching (so this may not be exactly right) but basically Bazel creates a hash that represents the action (the inputs, command, and outputs). That hash is sent to the distributed cache. If it’s a hit, then the output can be fetched without running anything. Otherwise, the action has to be run.

    The trick is figuring out how to compute the hash so you recognize when two actions are the same (e.g., `gcc -o thing thing.cc` is the same as `gcc thing.cc -o thing`) but never get a hit when the action doesn’t match (e.g., `find -L ./` is not the same as `find ./ -L`).

  • ittai zeidman

    Would you set up the job to periodic or being triggered by the scm (the trigger might come from polling of every second)?

  • kristina1

    Depends on how big your repo is/how active it is/how beefy your build machines are. There are several use cases I can think of:

    * In an editor: trigger a build every time you Ctrl-S. This might be noisy, though (I often save with obvious compile errors present). However, a lot of people request this internally.
    * Run on commit. This seems a bit saner, you could trigger it as a git hook.
    * For a repo that is too busy to run on every commit (e.g., if the repo has several commits per second), run periodically.

    So, until the repo got too busy, I’d probably go with “triggered by SCM.”

kristina chodorow's blog