Should the license cache be committed to version control?

[andreas-hakansson-ingka]

Is the intended workflow that the license cache should be committed to version control, with the rest of the project? I'm guessing yes? But I can't see the folder in this repository.

[jonabc]

@andreas-hakansson-ingka yep the files should be cached in the repo. there was a decision quite awhile back to not include the files in the repo but there have been usage changes since then and the decision was never re-evaluated. thank you for the call out about caching this tools license metadata, it's probably the right choice these days 🙇

[andreas-hakansson-ingka]

@jonabc thanks for the update. We're running into a bit of a problem, in some of our repositorsies though. We're using Licensed to scan go dependencies. Locally we use macOS for development, but we also run Licensed as part of our GitHub actions.

It seems that there are some slight differences in the dependencies that could be generated for Go depending on the host operating system. For example, if we run licensed cache locally then part of the cached dependencies are

    Caching golang.org/x/net/context/ctxhttp (v0.0.0-20210510120150-4163338589ed)
    Caching golang.org/x/net/http2 (v0.0.0-20210510120150-4163338589ed)
    Caching golang.org/x/net/internal/timeseries (v0.0.0-20210510120150-4163338589ed)
    Caching golang.org/x/net/trace (v0.0.0-20210510120150-4163338589ed)
    Caching golang.org/x/oauth2 (v0.0.0-20210514164344-f6687ab2804c)
    Caching golang.org/x/oauth2/authhandler (v0.0.0-20210514164344-f6687ab2804c)
    Caching golang.org/x/oauth2/google (v0.0.0-20210514164344-f6687ab2804c)
    Caching golang.org/x/oauth2/google/internal/externalaccount (v0.0.0-20210514164344-f6687ab2804c)
    Caching golang.org/x/oauth2/internal (v0.0.0-20210514164344-f6687ab2804c)
    Caching golang.org/x/oauth2/jws (v0.0.0-20210514164344-f6687ab2804c)
    Caching golang.org/x/oauth2/jwt (v0.0.0-20210514164344-f6687ab2804c)
    Caching golang.org/x/sync/semaphore (v0.0.0-20210220032951-036812b2e83c)

However, if we were to use this cache in GitHub Actions (running on an Ubuntu operating system) then out licensed status would fail with

* REDACTED.go.golang.org/x/sys/internal/unsafeheader
  filename: /home/runner/work/REDACTED/.licenses/go/golang.org/x/sys/internal/unsafeheader.dep.yml, version: v0.0.0-20210514084401-e8d321eab015, license: , allowed: false
    - cached dependency record not found

* REDACTED.go.golang.org/x/sys/unix
  filename: /home/runner/work/REDACTED/.licenses/go/golang.org/x/sys/unix.dep.yml, version: v0.0.0-20210514084401-e8d321eab015, license: , allowed: false
    - cached dependency record not found

if you look carefully then you will notice that .licenses/go/golang.org/x/sys/internal/unsafeheader.dep.yml and .licenses/go/golang.org/x/sys/unix.dep.yml were never generated by licensed cache when running on macOS, probably since these will only be derived if resolving dependencies on an unix operating system

This can be confirmed if we also run licensed cache during our GitHub actions execution

    Using golang.org/x/net/context/ctxhttp (v0.0.0-20210510120150-4163338589ed)
    Using golang.org/x/net/http2 (v0.0.0-20210510120150-4163338589ed)
    Using golang.org/x/net/internal/timeseries (v0.0.0-20210510120150-4163338589ed)
    Using golang.org/x/net/trace (v0.0.0-20210510120150-4163338589ed)
    Using golang.org/x/oauth2 (v0.0.0-20210514164344-f6687ab2804c)
    Using golang.org/x/oauth2/authhandler (v0.0.0-20210514164344-f6687ab2804c)
    Using golang.org/x/oauth2/google (v0.0.0-20210514164344-f6687ab2804c)
    Using golang.org/x/oauth2/google/internal/externalaccount (v0.0.0-20210514164344-f6687ab2804c)
    Using golang.org/x/oauth2/internal (v0.0.0-20210514164344-f6687ab2804c)
    Using golang.org/x/oauth2/jws (v0.0.0-20210514164344-f6687ab2804c)
    Using golang.org/x/oauth2/jwt (v0.0.0-20210514164344-f6687ab2804c)
    Using golang.org/x/sync/semaphore (v0.0.0-20210220032951-036812b2e83c)
    Caching golang.org/x/sys/internal/unsafeheader (v0.0.0-20210514084401-e8d321eab015)
    Caching golang.org/x/sys/unix (v0.0.0-20210514084401-e8d321eab015)

Notice the two last lines where is cached 2 additional dependencies and used the checked-in dependencies for the rest.

Do you have any suggestions on the correct workflow here? The way I see it is

1. Don't check-in .licenses and always make licensed cache part of our GitHub action workflow. This comes with the drawback that we will have slightly longer pipeline execution times
2. Check-in .licenses to help speed up execution time, but keep the additional licensed cache call to make sure we cache operating specific dependencies as well. If they fail in our CI environment, then we can add them to our reviewed section and push them again

Thoughts?

[jonabc]

@andreas-hakansson-ingka 👋 what environment is your executable being built in? Is it being built from linux (actions) or from macOS (dev)?

My recommendation would be to keep the license cache and status checks limited to the build environment so that you have matching licenses. If you build and release from multiple environments and they have different dependencies the only solution I'm aware of is to keep separate licenses available for each environment but that adds a bunch of hassle.

Since you mentioned GitHub actions, I'd recommend checking out https://github.com/jonabc/licensed-ci as a solution for an all-in-one verification and caching option. If that workflow fits what you need it could simplify some of your process.

[andreas-hakansson-ingka]

@jonabc so we sort of have this

• Dev (primarily macOS)
• CI/CD on actions (linux)
• Final artifact is a scratch docker image

Since you mentioned GitHub actions, I'd recommend checking out https://github.com/jonabc/licensed-ci as a solution for an all-in-one verification and caching option. If that workflow fits what you need it could simplify some of your processes.

Seen it, but it's a little bit too opinionated for us.. or rather our opinionated approach is not exactly like that action, so we simply rely on invoking a Makefile (from actions) and use the actions/cache to get a similar behavior

So if I understand you correctly you are suggesting

1. Don't commit the license cache to the repository
2. Generate and cache the license cache as part of our GitHub actions

[jonabc]

Don't commit the license cache to the repository

If you mean this as "don't commit the license cache to the repository during local dev" then yes. I'd still recommend committing the license files to the repo generally. Having the files in the repo allows you to easily manage situations that require manual intervention like license misdetection