Speeding up the build

[kevinburke]

Say I wanted to improve the performance of Bazel in the first run, what should I try? I'm curious about this in general. Here, 14 seconds are spent in the "analyzing" step. Is there a way to cache that for the next run on Travis CI? I'm caching $HOME/.bzrepos but not sure that does the trick.

$ make ci
bazel --batch --host_jvm_args=-Dbazel.DigestFunction=SHA1 test \
		--experimental_repository_cache="$HOME/.bzrepos" \
		--spawn_strategy=remote \
		--remote_rest_cache=https://remote.rest.stackmachine.com/cache \
		--test_output=errors \
		--strategy=Javac=remote \
		--features=race //... 2>&1 | ts '[%Y-%m-%d %H:%M:%.S]'
[2017-07-31 18:49:54.686348] Extracting Bazel installation...
[2017-07-31 18:50:01.534506] Loading: 
[2017-07-31 18:50:01.537702] Loading: 0 packages loaded
[2017-07-31 18:50:04.024261] Loading: 0 packages loaded
[2017-07-31 18:50:05.289372] Loading: 0 packages loaded
[2017-07-31 18:50:05.289656]     currently loading: 
[2017-07-31 18:50:06.541510] Analyzing: 4 targets (5 packages loaded)
[2017-07-31 18:50:07.589030] Analyzing: 4 targets (6 packages loaded)
[2017-07-31 18:50:09.541969] Analyzing: 4 targets (9 packages loaded)
[2017-07-31 18:50:16.083263] Analyzing: 4 targets (9 packages loaded)
[2017-07-31 18:50:19.224706] Analyzing: 4 targets (9 packages loaded)
[2017-07-31 18:50:22.008101] Analyzing: 4 targets (18 packages loaded)
[2017-07-31 18:50:22.747967] INFO: Analysed 4 targets (24 packages loaded).
[2017-07-31 18:50:22.750379] INFO: Found 2 targets and 2 test targets...
[2017-07-31 18:50:23.496402] [0 / 6] BazelWorkspaceStatusAction stable-status.txt
[2017-07-31 18:50:27.127856] [6 / 26] GoBuildTool external/io_bazel_rules_go/go/tools/builders/compile [for host]; 1s remote ... (2 actions running)
[2017-07-31 18:50:28.341258] INFO: Elapsed time: 29.771s, Critical Path: 1.76s
[2017-07-31 18:50:28.341599] INFO: Build completed successfully, 40 total actions
[2017-07-31 18:50:28.344651] //:go_default_test                                                       PASSED in 0.0s
[2017-07-31 18:50:28.345174] //:go_default_xtest                                                      PASSED in 0.0s
[2017-07-31 18:50:28.346077] 
[2017-07-31 18:50:28.346267] Executed 2 out of 2 tests: 2 tests pass.

[kevinburke]

Here's the "analysis" phase info from Travis CI:

=== ANALYSIS PHASE INFORMATION ===
Total analysis phase time               16.991 s
Total time (across all threads) spent on:
              Type    Total    Count     Average
    CREATE_PACKAGE    0.05%       20     4.06 ms
          VFS_STAT    0.18%    20552     0.01 ms
           VFS_DIR    0.47%      594     1.35 ms
        VFS_DELETE    0.02%        9     3.76 ms
          VFS_OPEN    0.00%       34     0.22 ms
          VFS_READ    0.07%    27499     0.00 ms
         VFS_WRITE    0.12%    29209     0.01 ms
          VFS_GLOB    1.12%     1223     1.54 ms
     SKYFRAME_EVAL    9.99%        1    16.904 s
       SKYFUNCTION   77.72%    17801     7.39 ms
     SKYLARK_LEXER    0.00%      120     0.02 ms
    SKYLARK_PARSER    0.01%      120     0.11 ms
   SKYLARK_USER_FN    9.72%      220     74.7 ms
SKYLARK_BUILTIN_FN    0.48%      886     0.93 ms

What is SKYFRAME_EVAL? is there a way to optimize it or cache its result?

[kevinburke]

I just tried switching all dependencies to download the zip (instead of git clone) but didn't get a speedup, see the profile information at the bottom here: https://travis-ci.org/kevinburke/rest/builds/259520850

(unless enabling profiling disables some caching steps)

[jayconrod]

We haven't spent much time optimizing the build yet, so there could be something slow in our Skylark code. Or it could be something in one of Bazel's built-in repositories.

The results are analysis are cached in memory when Bazel is run in the normal client/server mode. I don't think they are cached at all in batch mode. There probably won't be much benefit in Travis CI either way.

Travis CI has some caching functionality, but the recommendation is only to use it for artifacts that are computationally expensive to produce. The cache is stored on an external server, so if it just contains files downloaded from other sources (e.g., the Go SDK), downloading the cache will take just as long.

[jayconrod]

I'm going through our issues and closing old ones.

Performance is still not where I want it to be. I have a few pending PRs that should help matters a bit (#1608 is the last in this series). Fewer SDK and standard library files will be available in the sandbox. We'll also use the pre-compiled standard library when building in a compatible mode (not cross compiling, not msan, etc). That should speed things up a bit.

I've filed #1609 to track performance in more detail. I've already written a minimal benchmark, which is running nightly on my desktop, but that needs to be expanded a lot.