Flag-Friday: debugging tests with –java_debug

To step through a Java test that you’re running with bazel test, use the --java_test flag:

$ bazel test --java_debug //src/test/java/com/example:hello-test
WARNING: Streamed test output requested so all tests will be run locally, without sharding, one at a time.
INFO: Found 1 test target...
Listening for transport dt_socket at address: 5005

At this point, switch over to your IDE and fire up a remote debugging configuration, with the host localhost:5005. The debugger will attach to the running process, your test will start, and the debugger will stop at your first breakpoint.

Saving the (prod) environment

You can create different environments (e.g., testing, prod, mobile, rainforest) with Bazel, then use them to make sure that targets only build with the right environment. This is a cool feature that’s undocumented (because it’s still in development, shhhhh, don’t tell anyone I told you about it).

Let’s say you have a prod SSH key that you don’t want used in development or test builds. You could restrict it to only be used in prod builds by defining the following:

environment(name = "dev")
environment(name = "prod")
environment(name = "testing")
 
environment_group(
    name = "buildenv",
    defaults = ["dev"],
    environments = [
        "dev",
        "prod",
        "testing",
    ],
)
 
filegroup(
    name = "ssh-key",
    restricted_to = [":prod"],
    srcs = ["key"],
)

Now whenever we use :ssh-key, it has to be in a prod-environment rule. For example, this works:

cc_binary(
    name = "prod-job",
    srcs = ["job.cc"],
    restricted_to = [":prod"],
    data = ["ssh-key"],
)

This doesn’t:

cc_test(
    name = "job-test",
    srcs = ["job_test.cc"],
    data = [":ssh-key"],
)

Building the second one gives:

$ bazel build :job-test
ERROR: /Users/kchodorow/test/a/BUILD:34:1: in cc_test rule //:job-test: dependency //:ssh-key doesn't support expected environment: //:dev.
ERROR: Analysis of target '//:job-test' failed; build aborted.
INFO: Elapsed time: 0.167s

Hopefully, if someone tried to add restricted_to = [":prod"] to a test, it’d “look wrong” and be easier to catch.

Note that you must set your defaults sanely: when I first tried this, I made the environment_group‘s defaults = ["prod"] and then was confused that I wasn’t getting any errors. Everything is built for the default environments unless specified otherwise!

This lets us say: “If a depends on b and b is restricted to a certain environment, then a must be restricted to the environment.” However, there is another direction to look at this from: if a is restricted to an environment, b must be compatible with that environment. To express this, you can use “compatible_with“:

filegroup(
    name = "dev-key",
    srcs = ["key.dev"],
    compatible_with = [
        ":dev",
        ":testing"
    ],
)

Now anything that’s restricted to “:dev” or “:testing” environments can depend on “:dev-key”. For example, these work:

cc_binary(
    name = "dev-job",
    srcs = ["job.cc"],
    data = [":dev-key"],
)
 
cc_test(
    name = "job-test",
    srcs = ["job_test.cc"],
    restricted_to = [":testing"],
    data = [":dev-key"],
)

This does not:

cc_binary(
    name = "prod-job",
    srcs = ["job.cc"],
    restricted_to = [":prod"],
    data = [":dev-key"],
)

The full matrix (assuming env is an environment) is:

b b restricted to env b compatible with env
a
a restricted to env
a compatible with env

Remember that environments are targets themselves, so avoid proliferating environments that aren’t global to the global scope (don’t make them publicly visible and keep them as private as possible).

Combining projects without converting to a monorepo

Bazel allows you to combine multiple directories from across your filesystem and pretend all of the sources are part of your project. This is a little hard to picture, so let’s use a concrete example.

Let’s say you have two projects you’re working on, checked out at ~/gitroot/spaghetti-stable and ~/gitroot/meatballs-master. You don’t want to combine them into one repository, but you have integration tests that run your meatballs service on top of your spaghetti platform and end-to-end tests that make requests (it forks them).

That is, the filesystem might look something like this:

gitroot/
  spaghetti-stable/
    WORKSPACE
    spaghetti/
      BUILD
      plate_of_spaghetti.cc
  meatballs-master/
    WORKSPACE
    meatballs/
      BUILD
      pile_of_meatballs.cc
      end_to_end_test.cc

I’ve made up these directories on GitHub (spaghetti and meatballs), if you want to take a look.

The spaghetti/BUILD file can be pretty simple:

cc_library(
    name = "spaghetti",
    srcs = ["plate_of_spaghetti.cc"],
    visibility = ["//visibility:public"],
)

The meatballs/BUILD file is similar, but you also have an end-to-end test that depends on both //spaghetti and //meatballs:

cc_library(
    name = "meatballs",
    srcs = ["pile_of_meatballs.cc"],
)
 
cc_test(
    name = "end_to_end_test",
    srcs = ["end_to_end_test.cc"],
    deps = [
        ":meatballs",
        "//spaghetti",
    ],
)

Note that we’re depending on //spaghetti, even though it’s not under meatballs-master/. We can combine the two directories during the build by running bazel with the --package_path argument:

$ bazel test --package_path %workspace%:/home/k/gitroot/spaghetti-stable:/usr/local/lib/bazel/base_workspace \
    //meatballs:end_to_end_test

This means: when you’re looking for package, first check ~/gitroot/meatballs-master (%workspace% is the current directory). Then check ~/gitroot/spaghetti-stable. Finally, check Bazel’s binary installer location (for internal tools Bazel needs during the build).

When your test finishes, take a look at ~/gitroot/meatballs-master/bazel-meatballs-master. This is called the execution root and it’s where Bazel actually runs build commands:

$ ls -l bazel-meatballs-master/
total 36
drwxr-x--- 2 k k 20480 Dec  8 14:08 _bin
drwxr-x--- 3 k k  4096 Dec  8 14:08 bazel-out
drwxr-x--- 2 k k  4096 Dec  8 14:08 external
lrwxrwxrwx 1 k k    64 Dec  8 14:08 meatballs -> home/k/test/meatballs-master/meatballs
lrwxrwxrwx 1 k k    64 Dec  8 14:08 spaghetti -> /home/k/test/spaghetti-stable/spaghetti
lrwxrwxrwx 1 k k    41 Dec  8 14:08 tools -> /usr/local/lib/bazel/base_workspace/tools

You can see that Bazel has combined the directories on the package path to create a single directory that contains both meatballs/ and spaghetti/ subdirectories. The source directory (~/gitroot/meatballs-master) is left unchanged.

If we were going to do this regularly, we could add the package path option to our .bazelrc file and then we don’t have to specify it every build.

To try this out, you can download the sources with:

$ git clone https://github.com/kchodorow/spaghetti-stable.git
$ git clone https://github.com/kchodorow/meatballs-master.git

Then cd into meatballs-master/ and run!

Non-technical advice for startups and open source projects

A former coworker recently asked me about what had worked well (and not) at MongoDB. I realized that I actually know a bunch of things about running an open source project/startup, some of which may not be common knowledge, so I figured I’d share some here.

Things changed dramatically as the company grew and the product matured, but this post is about the project’s infancy (when the company was less than, say, 20 people). At that point, our #1 job was pleasing users. This meant everything from getting back to mailing list questions within a few minutes to shifting long-term priorities based on what users wanted. We’d regularly have a user complain about an issue and a couple of hours later we’d tell them “Please try again from head, we’ve fixed the issue.”

Unfortunately, fast turnaround meant bugs. We weren’t using code review at this point (give us a break, it was 2008), so we could get things out as fast as we could code them, but they might not… exactly… work. Or might interact badly with other parts of the system. I’m not sure what the solution to this would be: I think fast iteration was part of what made us successful. People got stuff done.

(via Poorly Drawn Lines.)

Users loved how fast we fixed things, but the support load was insane (at times) and basically could not be handled by non-engineers (i.e., we could not hire a support staff). The only reason this worked was that the founders were twice as active as anyone else on the list, setting the example. We actually had a contest running for a while that if anyone had more messages than Eliot in a month, they would get an iPad. No one ever won that iPad.

If a bug couldn’t be fixed immediately, it was still incredibly important to get back to people fast. Users didn’t actually care if you couldn’t solve their problem, they wanted a response. (I mean, they obviously preferred it if we had a solution, but we got 100% goodwill if we responded with a solution, 95% goodwill if we responded with “crap, I’ve filed a bug” within 15 minutes, 50% goodwill within a day, and 0% after that.) I’m still not as good at this as I’d like to be, but I recommend being as responsive as possible, even if you can’t actually help.

Because our users were developers (and at this point, early adopters), they generally detested “business speak.” Occasionally we had a new non-technical person join who wan’t familiar with developer culture, which resulted in some really negative initial perceptions on Reddit (that was the most memorable one, but we also had some business-speak issues on Twitter and other channels).

In fact, when we first hired a non-technical person, I was really skeptical. I didn’t understand why an engineering company making a product for other engineers would need a non-engineer. Now I’d credit her with 50% of MongoDB’s success and a good deal of the developers’ happiness. Meghan would reach out to and contact users and contributors, get to know meetup and conference organizers, and was generally a “router” for all incoming requests that we didn’t know what to do with (“Can we get someone to come talk at our conference in Timbuktu?” “Could I have an internship?” “Can you send me 20 MongoDB mugs to give to my coworkers?”). In general, there was a surprising amount of non-technical stuff that was important for success. We had a piece of software that people definitely wanted and needed, but I don’t think it would have been nearly as successful without the non-technical parts.

One weird trick for fast CI

Compilers hate this! (Just kidding, compilers are easy-going.)

For many build systems, you have to do a clean build to be sure you’re getting the correct result, so your CI has to always do a clean build. On the other hand, Bazel is designed so that you never have to do a bazel clean (if you ever run bazel clean and get a different answer, please file a bug!).

Thus, to cater to most build tools, CI systems generally run a loop like this:

However, for Bazel, doing a clean build every time wastes a lot of time and resources. For Bazel, the continuous integration system should look like this:

Of course, most build systems actually make this a bit difficult to set up, as you wouldn’t want this kind of configuration for a non-Bazel continuous build! However, Jenkins lets you set it up fairly simply. I’ll go through setting up a Jenkins instance that will build a local codebase every 5 minutes.

First, I’ll make my “source directory:”

$ mkdir jenkins-workspace
$ cd jenkins-workspace
$ touch WORKSPACE

I’m on OS X, so I installed the Jenkins pkg. When it is done installing, it pops up a web page prompts you to create your first job.

Click on the link and put in a name and select “Freestyle Build”, then hit “OK”.

On the next page, select “Advanced”. Select “Use custom workspace” and put in the directory from above.

Set it up to run every 5 minutes:

Screen Shot 2015-10-08 at 2.06.17 PM

Now add the following command as the build step:

This will build all targets in your workspace and then run all tests (so it will find any compile errors as well as test failures). //tools and //third_party contain a bunch of targets that won’t build out-of-the-box, so they’re filtered out using the -//target syntax (see the “Subtractive Patterns” section under Target Patterns).

Note that you don’t have to do any cleanup after the build, since Bazel never pollutes your source directory with build artifacts. You just want to pull, build, pull, build.

Now hit the “Save” button. It’ll take you back to your dashboard and begin the countdown to the first build. Click on the play button in the rightmost column to test it out (or just wait five minutes).

If you make changes, Jenkins will just build/test those changes:

If you haven’t made any changes, Bazel won’t have to do any work at all:

This can be a huge time saver for large projects.

Creating a javadoc rule for Bazel

A couple of users have asked about how to generate javadoc with Bazel. There’s no built-in way, but I figured it might be useful to whip together a new rule to do so.

Here it is.

If you’d like to use this rule, download it to your workspace, load it in your build file, and give it a list of sources:

load("/javadoc", "javadoc")
 
javadoc(
    name = "app-doc",
    srcs = glob(["**/*.java"]),
)

Then build:

$ bazel build :app-doc
INFO: Found 1 target...
Target //:app-doc up-to-date:
  bazel-bin/app-doc.zip
INFO: Elapsed time: 0.141s, Critical Path: 0.00s

Now, if you look at bazel-bin/app-doc.zip, you can see that it contains the HTML tree generated by javadoc:

$ unzip -l bazel-bin/app-doc.zip
Archive:  bazel-bin/app-doc.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
     6990  2015-09-28 15:06   app-doc/SomeClass.html
      568  2015-09-28 15:06   app-doc/allclasses-frame.html
      548  2015-09-28 15:06   app-doc/allclasses-noframe.html
...

You can then unzip it wherever you want to view your docs.

Note that this is an extremely limited implementation: I just dashed this off in 20 minutes. It doesn’t support any javadoc options and probably doesn’t handle dependencies correctly. Let me know if you have any issues with it and I can implement improvements as needed.

Build, y u go slow?

yuno

When a build is taking too long, it can be very helpful to know what it’s doing. Bazel has built-in tooling that lets you visualize what each thread is doing at any given moment of a build and which build steps are slowing down your overall build.

To try out Bazel’s profiling tools, build your favorite (or, rather, least-favorite) target with the --profile option:

$ bazel build --profile=myprofile.out //snail:slow-lib

This will write the profile to a file called myprofile.out in the current directory. Once your build finishes, you can take a look at this file, but it’s not really designed to be read by humans. Instead, plug it into Bazel’s analyze-profile command:

$ bazel analyze-profile --html myprofile.out

Now you wait for Bazel to analyze the profile info and enjoy this picture of a snail I saw walking Domino the other day (penny included for scale):

Make your project build faster than this.

Make your project build faster than this.

(After taking this photo, I moved the snail onto the grass, since I’m pretty sure it was not delighted to be in the middle of a NYC sidewalk.)

Ding, analysis is probably done. Now you can open up myprofile.out.html and see your build, broken down into hundreds or thousands of individual steps. A screenshot of the output:

Screen Shot 2015-09-18 at 2.51.55 PM

I uploaded the HTML page so you can see the whole thing here and play with it (it will open in a new tab).

I used the //android target from the example app for the profile above, since it’s a little more meaty than a toy example.

The chart shows what all 200 build threads were doing during the build at any given time (one thread per row). The build is divided into several “phases” which are shown as different colored columns on the chart:

  1. The first 1.5 seconds (darkish grey) were spent initializing the build command, which means it was just parsing options and setting up the cache.
  2. The next ~1 second (green) was the loading phase, where Bazel figures out which packages it will need, downloads external dependencies, and finds and parses BUILD files.
  3. The next ~100ms (sliver of light grey) was the analyze dependencies phase, where Bazel figured out which dependencies were cached and clean and so did not need to be rebuilt.
  4. Finally, Bazel moved into the build phase (pink background), actually building all of the things that needed to be built.

There are several other phases that you can see on the legend, but they are, for the most part, too short to even be visible on the chart.

Below the chart, there’s a section for “Execution phase.” “Execution” can be a little confusing in this context: it’s referring to executing the build, not running your program. The execution phase maps to the pink phase in the chart above. In this section is a sub-section called “Critical path, ” which breaks down what your build was waiting on:

Critical path (13.339 s):
    Id        Time Percentage   Description
  6722     48.1 ms    0.36%   Zipaligning apk
  6721      344 ms    2.58%   Generating signed apk
  6720      540 ms    4.05%   Converting bazel-out/local_darwin-fastbuild/bin/android/android_deploy.jar to dex format
  6719      230 ms    1.73%   Building deploy jar android/android_deploy.jar
  6718     1.051 s    7.88%   Building android/libandroid.jar (0 files)
  6717     15.7 ms    0.12%   Extracting interface //android activities
  6716     1.744 s   13.07%   Building android/libactivities.jar (1 files)
  6715      785 ms    5.89%   Processing resources
  6712     3.737 s   28.01%   Building external/default_android_tools/src/tools/android/java/com/google/devtools/build/android/libandroid_builder_lib.jar (17 files) [for host]
  6711     4.843 s   36.30%   Writing file external/default_android_tools/src/tools/android/java/com/google/devtools/build/android/libandroid_builder_lib.jar-2.params [for host]
           1.73 ms    0.01%   [2 middleman actions]

As you can see, the build was “blocked” for more than 8 seconds building libandroid_builder_lib.jar and its params file. Luckily, these files shouldn’t change between builds (unless you update your Android SDK, which shouldn’t happen between every build). If I make a change to libactivities.jar (the actual meat of the program) and rebuild, I get:

$ bazel build --profile profile2 //android
INFO: Writing profile data to '/Users/kchodorow/gitroot/examples/tutorial/profile2'
INFO: Found 1 target...
INFO: From Generating unsigned apk:
 
THIS TOOL IS DEPRECATED. See --help for more information.
 
INFO: From Generating signed apk:
 
THIS TOOL IS DEPRECATED. See --help for more information.
 
Target //android:android up-to-date:
  bazel-bin/android/android_deploy.jar
  bazel-bin/android/android_unsigned.apk
  bazel-bin/android/android.apk
INFO: Elapsed time: 4.545s, Critical Path: 2.72s

Here is the new profile page for this incremental build.

Note that the critical path is only 2.718 seconds now, not 13.339! If we look at the profile, we can see that the new critical path is much more svelte:

Critical path (2.718 s):
    Id        Time Percentage   Description
   543     52.3 ms    1.92%   Zipaligning apk
   542      342 ms   12.59%   Generating signed apk
   541      577 ms   21.22%   Converting bazel-out/local_darwin-fastbuild/bin/android/android_deploy.jar to dex format
   540      245 ms    9.00%   Building deploy jar android/android_deploy.jar
   539     1.502 s   55.26%   Building android/libactivities.jar (1 files)

Now building libactivities.jar is the most heavyweight operation on the critical path, so we could tackle that by perhaps breaking it into separate libraries that don’t all have to be recompiled every time something changes.

The profiles Bazel generates can be… dense… so feel free to ask on the mailing list if you need any help interpreting them. Also, if you’re interested in more on the subject, check out the documentation on profiling.

Debugging flaky tests with Bazel

Suppose you have a test that is passing… most of the time. When you start debugging it, you might try running the test and, unhelpfully, it passes:

$ bazel test :flaker
INFO: Found 1 test target...
Target //:flaker up-to-date:
  bazel-bin/flaker
INFO: Elapsed time: 0.223s, Critical Path: 0.04s
//:flaker                                                                PASSED
 
Executed 1 out of 1 tests: 1 test passes.

At this point, if you simply run bazel test :flaker again, Bazel knows that no files that affect the test have changed, so it won’t bother re-running it:

$ bazel test :flaker
INFO: Found 1 test target...
Target //:flaker up-to-date:
  bazel-bin/flaker
INFO: Elapsed time: 0.207s, Critical Path: 0.00s
//:flaker                                                   (1/0 cached) PASSED 
Executed 0 out of 1 tests: 1 test passes.

Note the “cached” message: your test wasn’t even run! This is usually a good thing: you don’t want to waste processing power on retesting something that you already know passes. However, it isn’t very convenient if you know that your test is flaky. So what do you do now?

You can always run the test manually (./bazel-bin/flaker) over and over, or write a script that will run it, but neither is very convenient.

Enter Bazel’s --runs_per_test option.

Specifying this option runs your test multiple times, prints a summary of what happened, and (by default) only keeps the logs for the failing tests:

$ bazel test --runs_per_test=10 :flaker
INFO: Found 1 test target...
FAIL: //:flaker (run 10 of 10) (see /private/var/tmp/_bazel_kchodorow/16a1114002542b106523c47d490a1041/test/bazel-out/local_darwin-fastbuild/testlogs/flaker/test_run_10_of_10.log).
FAIL: //:flaker (run 4 of 10) (see /private/var/tmp/_bazel_kchodorow/16a1114002542b106523c47d490a1041/test/bazel-out/local_darwin-fastbuild/testlogs/flaker/test_run_4_of_10.log).
FAIL: //:flaker (run 5 of 10) (see /private/var/tmp/_bazel_kchodorow/16a1114002542b106523c47d490a1041/test/bazel-out/local_darwin-fastbuild/testlogs/flaker/test_run_5_of_10.log).
FAIL: //:flaker (run 9 of 10) (see /private/var/tmp/_bazel_kchodorow/16a1114002542b106523c47d490a1041/test/bazel-out/local_darwin-fastbuild/testlogs/flaker/test_run_9_of_10.log).
FAIL: //:flaker (run 3 of 10) (see /private/var/tmp/_bazel_kchodorow/16a1114002542b106523c47d490a1041/test/bazel-out/local_darwin-fastbuild/testlogs/flaker/test_run_3_of_10.log).
Target //:flaker up-to-date:
  bazel-bin/flaker
INFO: Elapsed time: 0.828s, Critical Path: 0.42s
//:flaker                                                                FAILED
 
Executed 1 out of 1 tests: 1 fails locally.

This ran the test 10 times, 5 runs of which failed. --runs_per_test lets you easily run a flaky test hundreds of times in a row (if necessary) to track down what’s going on. Bazel creates unique log files for each failing run automatically and discards the passing tests’ logs.

Getting more logs

By default, Bazel doesn’t keep logs around for passing tests (since you don’t usually care what happened when a test passed). However, during development, sometimes even passing logs can be handy. You can get logs for your passing tests using the --test_output=all flag:

$ bazel test --test_output=all :flaker
INFO: Found 1 test target...
INFO: From Testing //:flaker:
==================== Test output for //:flaker:
Should I pass or should I fail?
Thinking about it...



Okay, I'll pass.
================================================================================
Target //:flaker up-to-date:
  bazel-bin/flaker
INFO: Elapsed time: 0.873s, Critical Path: 0.71s
//:flaker                                                                PASSED

Executed 1 out of 1 tests: 1 test passes.

This way you can see stdout/stderr from tests that you’re still working on (or ones that are unexpectedly passing).

Turning off caching

Finally, as mentioned above, Bazel tries to cache test runs whenever possible. If you don’t want to use --runs_per_test but still want to rerun a test, you can specify --cache_test_results=no:

$ bazel test :flaker
INFO: Found 1 test target...
Target //:flaker up-to-date:
  bazel-bin/flaker
INFO: Elapsed time: 0.282s, Critical Path: 0.04s
//:flaker                                                                PASSED
 
Executed 1 out of 1 tests: 1 test passes.
$ bazel test --cache_test_results=no :flaker
INFO: Found 1 test target...
FAIL: //:flaker (see /private/var/tmp/_bazel_kchodorow/16a1114002542b106523c47d490a1041/test/bazel-out/local_darwin-fastbuild/testlogs/flaker/test.log).
Target //:flaker up-to-date:
  bazel-bin/flaker
INFO: Elapsed time: 0.178s, Critical Path: 0.03s
//:flaker                                                                FAILED
 
Executed 1 out of 1 tests: 1 fails locally.

It’s a bit redundant (you can get the same behavior from --runs_per_test), but sometimes I’m just in more of a --cache_test_results mood.

There are several other features Bazel has to simplify testing and debugging, but these are a few of the flags that I find most helpful.

Check out the Bazel user manual for more detailed documentation on:

Happy testing!

The Return of the Scala Rule Tutorial: The Execution

This builds on the first part of the tutorial. In this post, we will make the the rule actually produce an executable.

Capturing the output from scalac

At the end of the tutorial last time, we were calling scalac, but ignoring the result:

(cd /private/var/tmp/_bazel_kchodorow/92df5f72e3c78c053575a1a42537d8c3/blerg && \
  exec env - \
  /bin/bash -c 'external/scala/bin/scalac HelloWorld.scala; echo '\''blah'\'' > bazel-out/local_darwin-fastbuild/bin/hello-world.sh')

If you look at the directory where the action is running (/private/var/tmp/_bazel_kchodorow/92df5f72e3c78c053575a1a42537d8c3/blerg in my case) you can see that HelloWorld.class and HelloWorld$.class is created. This directory is called the execution root, it is where bazel executes build actions. Bazel uses separate directory trees for source code, executing build actions, and output files (bazel-out/). Files won’t get moved from the execution root to the output tree unless we tell Bazel we want them.

We want our compiled scala program to end up in bazel-out/, but there’s a small complication. With languages like Java (and Scala), a single source file might contain inner classes that cause multiple .class files to be generated by a single compile action. Bazel cannot know until it runs the action how many class files are going to be generated. However, Bazel requires that each action declare, in advance, what its outputs will be. The way to get around this is to package up the .class files and make the resulting archive the build output.

In this example, we’ll add the .class files into a .jar. Let’s add that to the outputs, which should now look like this:

  outputs = {
    'jar': "%{name}.jar",
    'sh': "%{name}.sh",
  },

In the impl function, our command is getting a bit complicated so I’m going to change it to an array of commands and then join them on “\n” in the action:

def impl(ctx):
    cmd = [
        "%s %s" % (ctx.file._scalac.path, ctx.file.src.path),
        "find . -name '*.class' -print > classes.list",
        "jar cf %s @classes.list" % (ctx.outputs.jar.path),
    ]
 
    ctx.action(
        inputs = [ctx.file.src],
	command = "\n".join(cmd),
        outputs = [ctx.outputs.jar]
    )

This will compile the src, find all of the .class files, and add them to the output jar. If we run this, we get:

$ bazel build -s :hello-world
INFO: Found 1 target...
>>>>> # //:hello-world [action 'Unknown hello-world.jar']
(cd /private/var/tmp/_bazel_kchodorow/92df5f72e3c78c053575a1a42537d8c3/blerg && \
  exec env - \
  /bin/bash -c 'external/scala/bin/scalac HelloWorld.scala
find . -name '\''*.class'\'' -print > classes.list
jar cf bazel-out/local_darwin-fastbuild/bin/hello-world.jar @classes.list')
Target //:hello-world up-to-date:
  bazel-bin/hello-world.jar
INFO: Elapsed time: 4.774s, Critical Path: 4.06s

Let’s take a look at what hello-world.jar contains:

$ jar tf bazel-bin/hello-world.jar
META-INF/
META-INF/MANIFEST.MF
HelloWorld$.class
HelloWorld.class

Looks good! However, we cannot actually run this jar, because java doesn’t know what the main class should be:

$ java -jar bazel-bin/hello-world.jar 
no main manifest attribute, in bazel-bin/hello-world.jar

Similar to the java_binary rule, let’s add a main_class attribute to scala_binary and put it in the jar’s manifest. Add 'main_class' : attr.string(), to scala_binary‘s attrs and change cmd to the following:

    cmd = [
        "%s %s" % (ctx.file._scalac.path, ctx.file.src.path),
        "echo Manifest-Version: 1.0 > MANIFEST.MF",
        "echo Main-Class: %s >> MANIFEST.MF" % ctx.attr.main_class,
        "find . -name '*.class' -print > classes.list",
	"jar cfm %s MANIFEST.MF @classes.list" % (ctx.outputs.jar.path),
    ]

Remember to update your actual BUILD file to add a main_class attribute:

# BUILD
load("/scala", "scala_binary")
 
scala_binary(
    name = "hello-world",
    src = "HelloWorld.scala",
    main_class = "HelloWorld",
)

Now building and running gives you:

$ bazel build :hello-world
INFO: Found 1 target...
Target //:hello-world up-to-date:
  bazel-bin/hello-world.jar
INFO: Elapsed time: 4.663s, Critical Path: 4.05s
$ java -jar bazel-bin/hello-world.jar 
Exception in thread "main" java.lang.NoClassDefFoundError: scala/Predef$
	at HelloWorld$.main(HelloWorld.scala:4)
	at HelloWorld.main(HelloWorld.scala)
Caused by: java.lang.ClassNotFoundException: scala.Predef$
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 2 more

Closer! Now it cannot find some scala libraries it needs. You can add it manually on the command line to see that our jar does actually does work if we specify the scala library jar, too:

$ java -cp $(bazel info output_base)/external/scala/lib/scala-library.jar:bazel-bin/hello-world.jar HelloWorld
Hello, world!

So we need our rule to generate an executable that basically runs this command, which can be accomplished by adding another action to our build. First we’ll add a dependency on scala-library.jar by adding it as a hidden attribute:

        '_scala_lib': attr.label(
            default=Label("@scala//:lib/scala-library.jar"),
            allow_files=True,
            single_file=True),

Making scala_binarys executable

Let’s pause here for a moment and switch gears: we’re going to tell bazel that scala_binarys are binaries. To do this, we add executable = True to the attrs and get rid of the reference to hello-world.sh in the outputs:

...
    outputs = {
        'jar': "%{name}.jar",
    },
    implementation = impl,
    executable = True,
)

This says that scala_binary(name = "foo", ...) should have an action that creates a binary called foo, which can be referenced via ctx.outputs.executable in the implementation function. We can now use bazel run :hello-world (instead of bazel build :hello-world; ./bazel-bin/hello-world.sh).

The executable we want to create is the java command from above, so we add the second action to impl, this one a file action (since we’re just generating a file with certain content, not executing a series of commands to generate a .jar):

    cp = "%s:%s" % (ctx.outputs.jar.basename, ctx.file._scala_lib.path)
    content = [
	"#!/bin/bash",
        "echo Running from $PWD",
	"java -cp %s %s" % (cp, ctx.attr.main_class),
    ]
    ctx.file_action(
	content = "\n".join(content),
	output = ctx.outputs.executable,
    )

Note that I also added a line to the file to echo where it is being run from. If we now use bazel run, you’ll see:

$ bazel run :hello-world
INFO: Found 1 target...
Target //:hello-world up-to-date:
  bazel-bin/hello-world.jar
  bazel-bin/hello-world
INFO: Elapsed time: 2.694s, Critical Path: 0.08s
 
INFO: Running command line: bazel-bin/hello-world
Running from /private/var/tmp/_bazel_kchodorow/92df5f72e3c78c053575a1a42537d8c3/blerg/bazel-out/local_darwin-fastbuild/bin/hello-world.runfiles
Error: Could not find or load main class HelloWorld
ERROR: Non-zero return code '1' from command: Process exited with status 1.

Whoops, it’s not able to find the jars! And what is that path, hello-world.runfiles, it’s running the binary from?

The runfiles directory

bazel run runs the binary from the runfiles directory, a directory that is different than the source root, execution root, and output tree mentioned above. The runfiles directory should contain all of the resources needed by the executable during execution. Note that this is not the execution root, which is used during the bazel build step. When you actually execute something created by bazel, its resources need to be in the runfiles directory.

In this case, our executable needs to access hello-world.jar and scala-library.jar. To add these files, the API is somewhat strange. You must return a struct containing a runfiles object from the rule implementation. Thus, add the following as the last line of your impl function:

return struct(runfiles = ctx.runfiles(files = [ctx.outputs.jar, ctx.file._scala_lib]))

Now if you run it again, it’ll print:

$ bazel run :hello-world
INFO: Found 1 target...
Target //:hello-world up-to-date:
  bazel-bin/hello-world.jar
  bazel-bin/hello-world
INFO: Elapsed time: 0.416s, Critical Path: 0.00s
 
INFO: Running command line: bazel-bin/hello-world
Running from /private/var/tmp/_bazel_kchodorow/92df5f72e3c78c053575a1a42537d8c3/blerg/bazel-out/local_darwin-fastbuild/bin/hello-world.runfiles
Hello, world!

Hooray!

However! If we run it as bazel-bin/hello-world, it won’t be able to find the jars (because we’re not in the runfiles directory). To find the runfiles directory regardless of where the binary is run from, change your content variable to the following:

    content = [
        "#!/bin/bash",
        "case \"$0\" in",
        "/*) self=\"$0\" ;;",
        "*)  self=\"$PWD/$0\";;",
        "esac",
        "(cd $self.runfiles; java -cp %s %s)" % (cp, ctx.attr.main_class),
    ]

This way, if it’s run from bazel run, $0 will be the absolute path to the binary (in my case, /private/var/tmp/_bazel_kchodorow/92df5f72e3c78c053575a1a42537d8c3/blerg/bazel-out/local_darwin-fastbuild/bin/hello-world). If it’s run via bazel-bin/hello-world, $0 will be just that: bazel-bin/hello-world. Either way, we’ll end up in the runfiles directory before executing the command.

Now our rule is successfully generating a binary. You can see the full code for this example on GitHub.

In the final part of this tutorial, we’ll fix the remaining issues:

  • No support for multiple source files, never mind dependencies.
  • [action 'Unknown hello-world.jar'] is pretty ugly.

Until next time!

Tutorial: how to write Scala rules for Bazel

Bazel comes with built-in support for several languages and allows you to write your own support for any other languages in Python.

Although you could probably get more abstract, let’s define a rule as something that takes some files, does something to them, and then gives you some output files. Specifically, for this example, we want a scala_binary rule where we can give it a Scala source file and it turns it into an executable binary.

Part 1: Creating a Scala source file

Let’s create a simple example of a Scala source file (mercilessly ripped from the Scala hello world example):

// HelloWorld.scala
object HelloWorld {
  def main(args: Array[String]) {
    println("Hello, world!")
  }
}

Note that I’ve never used Scala before today, so please let me know in the comments if I’ve made an mistakes.

Before proceeding, I think it’s a good idea to try building this without bazel (especially if you’re not too familiar with the language’s build tool… ahem) as a sanity check:

$ scalac HelloWorld.scala
$ scala HelloWorld
Hello, world!

Looking good! Now, let’s try to get bazel building that.

Adding a BUILD file and dummy scala_binary rule

We’ll create a BUILD file that references our (currently non-existent) scala_binary rule. This lets us plan out what we’ll need our rule to look like:

# BUILD
load('/scala', 'scala_binary')
 
scala_binary(
    name = "hello-world",
    src = "HelloWorld.scala",
)

The load() statement means that we’ll declare the scala_binary rule in a file called scala.bzl in the root of the workspace (due to the ‘/’ prefix on ‘/scala’). Let’s create that file now:

# scala.bzl
def impl(ctx):
    pass
 
scala_binary = rule(
    attrs = {
        'src': attr.label(
            allow_files=True,
            single_file=True),
    },
    outputs = {'sh': "%{name}.sh"},
    implementation = impl,
)

scala_binary‘s definition says that rules can have one attribute (other than name), src, which is a single file. The rule is supposed to output a file called name.sh, so for our example we should end up with hello-world.sh. The implementation of our rule should be happening in the function impl. The rule implementation doesn’t do anything yet, but we can at least try building now:

$ touch WORKSPACE # if you haven't already...
$ bazel build :hello-world
ERROR: /Users/kchodorow/blerg/BUILD:4:1: in scala_binary rule //:hello-world: 
: The following files have no generating action:
hello-world.sh
.
ERROR: Analysis of target '//:hello-world' failed; build aborted.
INFO: Elapsed time: 0.286s

The error is expected: our rule definition says that hello-world.sh should be an output, but there’s no code creating it yet. Let’s add some functionality to the implementation function by replacing existing function with the following:

def impl(ctx):
    ctx.action(
        inputs = [ctx.file.src],
        command = "echo %s > %s" % (ctx.file.src.path, ctx.outputs.sh.path),
        outputs = [ctx.outputs.sh]
    )

This adds an action to the build. It says that, if the inputs have changed (the src file), run the command (which right now is just echoing src‘s path) to the output file. Note that ctx.action(...) doesn’t actually run the action, it just adds that action to “things that need to be run in the future” for the rule.

Now if we build :hello-world again, we get:

$ bazel build -s :hello-world
INFO: Found 1 target...
>>>>> # //:hello-world [action 'Unknown hello-world.sh']
(cd /private/var/tmp/_bazel_kchodorow/92df5f72e3c78c053575a1a42537d8c3/blerg && \
  exec env - \
  /bin/bash -c 'echo HelloWorld.scala > bazel-out/local_darwin-fastbuild/bin/hello-world.sh')
Target //:hello-world up-to-date:
  bazel-bin/hello-world.sh
INFO: Elapsed time: 0.605s, Critical Path: 0.02s

I used bazel’s -s option here, which is very helpful for debugging what your rule is doing. It prints all of the subcommands a build is running. As you can see, now our rule has an action (>>>>> # //:hello-world [action 'Unknown hello-world.sh']) that creates bazel-bin/hello-world.sh by echoing the source file name. You can verify this by cating bazel-bin/hello-world.sh.

Adding a dependency on the scala compiler

We want our rule to actually call scalac. Even if you have the scala compiler installed on your system, you cannot simply create an action with a command = 'scalac MySourceFile.scala' line, as actions are run in a “clean room” environment: nothing* is there that you don’t specify.

As you probably don’t want to add the scala compiler to your workspace, open up your WORKSPACE file and add it as an external dependency:

# WORKSPACE
new_http_archive(
    name = "scala",
    url = "http://downloads.typesafe.com/scala/2.11.7/scala-2.11.7.tgz",
    sha256 = "ffe4196f13ee98a66cf54baffb0940d29432b2bd820bd0781a8316eec22926d0",
    build_file = "scala.BUILD",
)

Also create the scala.BUILD file in the root of your workspace:

# scala.BUILD
exports_files([
    "bin/scala",
    "bin/scalac",
    "lib/scala-library.jar"
])

Now add a dependency on scalac to your scala_binary rule by adding a “hidden attribute.” Add calling scalac in your impl, so your scala.bzl file looks something like this:

def impl(ctx):
    ctx.action(
        inputs = [ctx.file.src],
        command = "%s %s; echo 'blah' > %s" % (
            ctx.file._scalac.path, ctx.file.src.path, ctx.outputs.sh.path),
        outputs = [ctx.outputs.sh]
    )
 
scala_binary = rule(
    attrs = {
        'src': attr.label(
            allow_files=True,
            single_file=True),
        '_scalac': attr.label(
            default=Label("@scala//:bin/scalac"),
            executable=True,
            allow_files=True,
            single_file=True),
    },
    outputs = {'sh': "%{name}.sh"},
    implementation = impl,
)

Building now shows that scalac is successfully being run on our source file!

$ bazel build -s :hello-world
INFO: Found 1 target...
>>>>> # //:hello-world [action 'Unknown hello-world.sh']
(cd /private/var/tmp/_bazel_kchodorow/92df5f72e3c78c053575a1a42537d8c3/blerg && \
  exec env - \
  /bin/bash -c 'external/scala/bin/scalac HelloWorld.scala; echo '\''blah'\'' > bazel-out/local_darwin-fastbuild/bin/hello-world.sh')
Target //:hello-world up-to-date:
  bazel-bin/hello-world.sh
INFO: Elapsed time: 4.634s, Critical Path: 4.11s

There are still many issues with this implementation:

  • The output from calling scalac doesn’t actually go anywhere, hello-world.sh is still a dummy file.
  • No support for multiple source files, never mind dependencies.
  • [action 'Unknown hello-world.sh'] is pretty ugly.
  • You can’t call bazel run //hello-world, even though the output should be executable.

However, this post is already running long, so let’s wrap it up here and get to some of these issue in the next post. Until next time!

References

* Obviously there are some commands there (our original rule uses echo, for instance) and you can see what’s in the empty environment by writing env to an output file in an action. This can actually cause issues: sometimes commands in the default PATH have different behavior on different systems. To get a completely hermetic build, you should really provide every command your rule uses. However, we’re just using echo for debugging here anyway, so we’ll let it slide.

kristina chodorow's blog