[ci][byod] validate that correct ray is installed

[can-anyscale]

Why are these changes needed?

• Validate that ray is at the correct commit under test after building BYOD image. This is a feature that exists in the old way of running release tests, I just port the validation check here.

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• Testing Strategy
  • Unit tests
  • Release tests - https://buildkite.com/ray-project/release-tests-pr/builds/43467#0188ee54-d693-41ce-9b74-d1440781609e (just check that the main job that builds all BYOD succeed. I canceled all the tests to save capacity, since they are not what is tested here).

[aslonnie]

I am actually thinking about removing this ray.__commit__ thing in non-final releases. this is invalidating all the bazel build and test caches on every build that requires the wheel.

[can-anyscale]

Lol interesting, is this because the file that contains the commit hash has so many dependencies that a change to it cause the whole graph to be rebuilt? Can it be move outside of the dependency graph?

[aslonnie]

is this because the file that contains the commit hash has so many dependencies that a change to it cause the whole graph to be rebuilt? Can it be move outside of the dependency graph?

there are changes that are not affecting the wheel and does not need a wheel rebuild, but the __commit__ field is guaranteeing that a wheel will always get rebuilt for every commit.

[can-anyscale]

@aslonnie : have we decided to support this going forward, but do it in a way that doesn't break the cache, or we haven't decided what to do here yet?

[aslonnie]

have we decided to support this going forward, but do it in a way that doesn't break the cache, or we haven't decided what to do here yet?

I think having the check here for now is fine, but can we run the test not in the docker file? one can either use docker run and print out the commit and check, or docker create+docker cp to copy out the __commit__.py file, or even use a library to extract the file from the container image without creating a container.

Dockerfile is supposedly for build time, where the commit check is more like a test I think.

[aslonnie]

as in the other comment, can we have a test outside the dockerfile but after the build?

[can-anyscale]

We can run them as part of the test execution yeah. I like that, in that way, the issue will also be bisectable.