Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: trim trailing '/' on URL if provided in package YAML #45

Merged
merged 2 commits into from
May 1, 2024
Merged

fix: trim trailing '/' on URL if provided in package YAML #45

merged 2 commits into from
May 1, 2024

Conversation

alexconrey
Copy link
Contributor

@alexconrey alexconrey commented Apr 30, 2024 •
edited
Loading

Problem

Willing to concede that this may just be my own PEBKAC, but after struggling to understand why one repository would work but another one would not - I discovered that there are assumptions made around the URL passed in the sources section of the YAML. Specifically, the assumption covered here is that the URL does not end in a "/".

For most HTTP servers, they'll see a '//' and (more or less) understand what to do and redirect to the expected path. For some repositories, however, they parse these '//' as literal paths which return a 404 or otherwise non-2xx code with the contents of the file as expected.

Reproduction Steps

$ git diff examples/apt/bullseye.yaml
diff --git a/examples/apt/bullseye.yaml b/examples/apt/bullseye.yaml
index 06afe45..4d0c786 100644
--- a/examples/apt/bullseye.yaml
+++ b/examples/apt/bullseye.yaml
@@ -16,6 +16,8 @@ sources:
     url: https://snapshot-cloudflare.debian.org/archive/debian-security/20240210T223313Z
   - channel: bullseye-updates main
     url: https://snapshot-cloudflare.debian.org/archive/debian/20240210T223313Z/
+  - channel: bullseye stable
+    url: https://download.docker.com/linux/debian/dists/bullseye/
 
 archs:
   - "amd64"
@@ -31,3 +33,4 @@ packages:
   - "dpkg"
   - "apt"
   - "perl"
+  - "docker-ce"

$ bazel run @bullseye//:lock
DEBUG: /git/src/github.com/alexconrey/rules_distroless/apt/index.bzl:73:14: 
No lockfile was given, please run `bazel run @apt_security//:lock` to create the lockfile.
INFO: Repository bullseye_resolution instantiated at:
  /git/src/github.com/alexconrey/rules_distroless/WORKSPACE.bazel:29:10: in <toplevel>
  /git/src/github.com/alexconrey/rules_distroless/apt/index.bzl:65:17: in deb_index
Repository rule deb_resolve defined at:
  /git/src/github.com/alexconrey/rules_distroless/apt/private/resolve.bzl:135:30: in <toplevel>
WARNING: Download from https://download.docker.com/linux/debian/dists/bullseye//dists/bullseye/stable/binary-amd64/Packages.xz failed: class java.io.FileNotFoundException GET returned 404 Not Found
ERROR: An error occurred during the fetch of repository 'bullseye_resolution':
   Traceback (most recent call last):
        File "/git/src/github.com/alexconrey/rules_distroless/apt/private/resolve.bzl", line 74, column 33, in _deb_resolve_impl
                pkgindex = package_index.new(rctx, sources = sources, archs = manifest["archs"])
        File "/git/src/github.com/alexconrey/rules_distroless/apt/private/package_index.bzl", line 81, column 47, in _create
                (output, _) = _fetch_package_index(rctx, url, dist, comp, arch, "")
        File "/git/src/github.com/alexconrey/rules_distroless/apt/private/package_index.bzl", line 8, column 22, in _fetch_package_index
                r = rctx.download(
Error in download: java.io.IOException: Error downloading [https://download.docker.com/linux/debian/dists/bullseye//dists/bullseye/stable/binary-amd64/Packages.xz] to /home/ubuntu/.cache/bazel/_bazel_ubuntu/67a753ee4a5392426d2d342afb29bfb7/external/bullseye_resolution/bullseye/stable/amd64/Packages.xz: GET returned 404 Not Found
ERROR: /git/src/github.com/alexconrey/rules_distroless/WORKSPACE.bazel:29:10: fetching deb_resolve rule //external:bullseye_resolution: Traceback (most recent call last):
        File "/git/src/github.com/alexconrey/rules_distroless/apt/private/resolve.bzl", line 74, column 33, in _deb_resolve_impl
                pkgindex = package_index.new(rctx, sources = sources, archs = manifest["archs"])
        File "/git/src/github.com/alexconrey/rules_distroless/apt/private/package_index.bzl", line 81, column 47, in _create
                (output, _) = _fetch_package_index(rctx, url, dist, comp, arch, "")
        File "/git/src/github.com/alexconrey/rules_distroless/apt/private/package_index.bzl", line 8, column 22, in _fetch_package_index
                r = rctx.download(
Error in download: java.io.IOException: Error downloading [https://download.docker.com/linux/debian/dists/bullseye//dists/bullseye/stable/binary-amd64/Packages.xz] to /home/ubuntu/.cache/bazel/_bazel_ubuntu/67a753ee4a5392426d2d342afb29bfb7/external/bullseye_resolution/bullseye/stable/amd64/Packages.xz: GET returned 404 Not Found
ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/67a753ee4a5392426d2d342afb29bfb7/external/bullseye/BUILD.bazel:2:6: @bullseye//:lock depends on @bullseye_resolution//:lock in repository @bullseye_resolution which failed to fetch. no such package '@bullseye_resolution//': java.io.IOException: Error downloading [https://download.docker.com/linux/debian/dists/bullseye//dists/bullseye/stable/binary-amd64/Packages.xz] to /home/ubuntu/.cache/bazel/_bazel_ubuntu/67a753ee4a5392426d2d342afb29bfb7/external/bullseye_resolution/bullseye/stable/amd64/Packages.xz: GET returned 404 Not Found
ERROR: Analysis of target '@bullseye//:lock' failed; build aborted: Analysis failed
INFO: Elapsed time: 34.290s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (1 packages loaded, 0 targets configured)
ERROR: Build failed. Not running target

Solution

Note: Because the Docker repository is providing Packages.gz instead, I have locally cherry-picked #40
This local cherry-pick only proves the functionality of this change on a known atypical Debian/Ubuntu repository and should not require the contents of the cherry-picked commit.

  • Add convenience checks/sanitization if URLs are specified with trailing slashes
    • I'd also accept "hey, don't do that" as a solution - to which I would request an update to docs reflecting this expectation around url formatting
$ bazel run @bullseye//:lock
DEBUG: /git/src/github.com/alexconrey/rules_distroless/apt/index.bzl:73:14: 
No lockfile was given, please run `bazel run @apt_security//:lock` to create the lockfile.
DEBUG: /git/src/github.com/alexconrey/rules_distroless/apt/private/package_resolution.bzl:141:22: Warning: optional dependencies are not supported yet. https://github.com/GoogleContainerTools/rules_distroless/issues/27
DEBUG: /git/src/github.com/alexconrey/rules_distroless/apt/private/resolve.bzl:96:22: the following packages have unmet dependencies: awk
DEBUG: /git/src/github.com/alexconrey/rules_distroless/apt/private/package_resolution.bzl:141:22: Warning: optional dependencies are not supported yet. https://github.com/GoogleContainerTools/rules_distroless/issues/27
DEBUG: /git/src/github.com/alexconrey/rules_distroless/apt/private/package_resolution.bzl:141:22: Warning: optional dependencies are not supported yet. https://github.com/GoogleContainerTools/rules_distroless/issues/27
DEBUG: /git/src/github.com/alexconrey/rules_distroless/apt/private/package_resolution.bzl:141:22: Warning: optional dependencies are not supported yet. https://github.com/GoogleContainerTools/rules_distroless/issues/27
DEBUG: /git/src/github.com/alexconrey/rules_distroless/apt/private/package_resolution.bzl:141:22: Warning: optional dependencies are not supported yet. https://github.com/GoogleContainerTools/rules_distroless/issues/27
DEBUG: /git/src/github.com/alexconrey/rules_distroless/apt/private/package_resolution.bzl:141:22: Warning: optional dependencies are not supported yet. https://github.com/GoogleContainerTools/rules_distroless/issues/27
DEBUG: /git/src/github.com/alexconrey/rules_distroless/apt/private/package_resolution.bzl:141:22: Warning: optional dependencies are not supported yet. https://github.com/GoogleContainerTools/rules_distroless/issues/27
DEBUG: /git/src/github.com/alexconrey/rules_distroless/apt/private/resolve.bzl:96:22: the following packages have unmet dependencies: awk
DEBUG: /git/src/github.com/alexconrey/rules_distroless/apt/private/package_resolution.bzl:141:22: Warning: optional dependencies are not supported yet. https://github.com/GoogleContainerTools/rules_distroless/issues/27
DEBUG: /git/src/github.com/alexconrey/rules_distroless/apt/private/package_resolution.bzl:141:22: Warning: optional dependencies are not supported yet. https://github.com/GoogleContainerTools/rules_distroless/issues/27
DEBUG: /git/src/github.com/alexconrey/rules_distroless/apt/private/package_resolution.bzl:141:22: Warning: optional dependencies are not supported yet. https://github.com/GoogleContainerTools/rules_distroless/issues/27
DEBUG: /git/src/github.com/alexconrey/rules_distroless/apt/private/package_resolution.bzl:141:22: Warning: optional dependencies are not supported yet. https://github.com/GoogleContainerTools/rules_distroless/issues/27
DEBUG: /git/src/github.com/alexconrey/rules_distroless/apt/private/package_resolution.bzl:141:22: Warning: optional dependencies are not supported yet. https://github.com/GoogleContainerTools/rules_distroless/issues/27
WARNING: Download from https://download.docker.com/linux/debian/dists/bullseye/stable/binary-amd64/Packages.xz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://download.docker.com/linux/debian/dists/bullseye/stable/binary-arm64/Packages.xz failed: class java.io.FileNotFoundException GET returned 404 Not Found
INFO: Analyzed target @bullseye//:lock (78 packages loaded, 415 targets configured).
INFO: Found 1 target...
Target @bullseye_resolution//:lock up-to-date:
  bazel-bin/external/bullseye_resolution/lock
INFO: Elapsed time: 52.551s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/external/bullseye_resolution/lock external/bullseye_resolution/lock.json

Writing lockfile to examples/apt/bullseye.lock.json

Run the following command to add the lockfile or pass --autofix flag to do it automatically.

   buildozer set lock @@//examples/apt:bullseye.lock.json WORKSPACE.bazel:bullseye

@alexconrey alexconrey changed the title fix: trim '/' on URL if provided in package YAML fix: trim trailing '/' on URL if provided in package YAML Apr 30, 2024
thesayyn
thesayyn approved these changes May 1, 2024
View reviewed changes
Copy link
Collaborator

@thesayyn thesayyn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thank you for the comments!

@thesayyn thesayyn merged commit 06c648b into GoogleContainerTools:main May 1, 2024
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@thesayyn thesayyn thesayyn approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@alexconrey @thesayyn