remote_download_toplevel doesn't download binary after testing

[lizan]

Description of the problem / feature request:

When --remote_download_toplevel is specified,

bazel test --remote_download_toplevel //test/...
bazel build --remote_download_toplevel <some_binary>

doesn't download <some_binary> while downloading binary is expected.

Running build first or clean some artifacts before build will be workaround.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

From envoy master, do test //test/exe/... and build //source/exe:envoy-static

What operating system are you running Bazel on?

Ubuntu 18.04

What's the output of bazel info release?

3.4.1

Any other information, logs, or outputs that you want to share?

It is possible because the test targets has the binary in its dependency so somehow bazel treated it is already downloaded, while test with --remote_download_toplevel does't download the binary itself.