cmd/go: go mod download -json should add 'Latest bool'

[hyangah]

go get <modulepath>@<version> triggers GET $GOPROXY/<modulepath>/@v/<version>.{info, mod, zip} queries when it talks with GOPROXY.

When it's the very first time the module proxy sees the version, the proxy may want to update its database and, if necessary, serve the newest version as the latest version of the module. Module owners may utilize this to tell the module proxy or Go Module Mirror as soon as the version becomes available in the origin.

But the proxy need to be careful because the the may be a commit or branch, and there is no guarantee that the newly fetched version is suitable as the latest version. The proxy may perform extra go list queries itself for every new version it observes, but it would be nice if the 'go mod download' command designed for the proxy provides this info in its output.

@rsc @bcmills @jayconrod @katiehockman @heschik

[bcmills]

But the proxy need to be careful because the [version] may be a commit or branch, and there is no guarantee that the newly fetched version is suitable as the latest version.

It seems to me that that information can already be resolved from the go mod download output.

Specifically, we can examine the Version field:

• If it is a pseudo-version (or other prerelease) derived from the latest version (or something older), then the proxy does not need to update latest.
• If it is a pseudo-version derived from a release version after latest, then the proxy is probably missing that release (and can find it using go list -versions).
• If it is a release version, then the proxy should update latest if (and only if) the new version is semantically higher than the existing latest version.

That leaves open the question of what to do if the proxy sees a pseudo-version or prerelease tag derived from something newer than latest, but those are likely rare anyway.

[hyangah]

Specifically, we can examine the Version field:

If it is a pseudo-version (or other prerelease) derived from the latest version (or something older), then the proxy does not need to update latest.
If it is a pseudo-version derived from a release version after latest, then the proxy is probably missing that release (and can find it using go list -versions).
If it is a release version, then the proxy should update latest if (and only if) the new version is semantically higher than the existing latest version.

This is already too much of complexity to implement correctly outside go commands. Would be nice if it's done through a library (maintained to be consistent with how go command sees) or throug go mod download. And we'd like to avoid the run of go list -versions followed by every go mod download if possible (due to our internal architectural limitation :-().

[bcmills]

Would be nice if it's done through a library (maintained to be consistent with how go command sees)

You mean like golang.org/x/mod/semver? 🙂

You could easily skip the “pseudo-version derived from a release version after latest” part, in which case you get:

if semver.Prerelease(m.Version) == "" && semver.Max(latest, m.Version) == m.Version {
	// Update latest.
	[…]
}

For the “derived from a release version after” case, perhaps we could add a semver.ReleaseBefore(v string) string, which would return the highest non-prerelease version (if any) immediately preceding v.

[hyangah]

@bcmills, do you mean the cached latest version in the proxy's database by 'latest' in your code snippet? (again, we are looking for a solution that avoids extra run of 'go list -versions' or 'go mod download module@latest' - because we are running each command in an isolated sandbox currently).

I think the pseudo-version can be newer than the actual latest (in the origin) if the pseudo-version was derived from a development branch or non-default branch.

[bcmills]

do you mean the cached latest version in the proxy's database by 'latest' in your code snippet?

Yes.

[bcmills]

I think the pseudo-version can be newer than the actual latest (in the origin) if the pseudo-version was derived from a development branch or non-default branch.

Yes: the pseudo-version is calculated based on the highest tag from a parent commit, which may be the latest tag or an arbitrarily higher pre-release tag. (For example, the commit may be on the default branch after the latest release, or may be after a higher prerelease tag on a development branch, or so on.)

However, note that every pseudo-version is also semantically a prerelease version: hence the semver.Prerelease(m.Version) == "" check in the code snippet.

[hyangah]

A bit more context behind this issue: we are investigating if it's possible to determine (without extra list query) whether the newly downloaded module version from go mod download is suitable for the 'latest' query when the origin doesn't have any released version tags. Only pseudo-versions are available so m.Version and latest in the code example are both pseudo-versions.

[bcmills]

we are investigating if it's possible to determine (without extra list query) whether the newly downloaded module version from go mod download is suitable for the 'latest' query when the origin doesn't have any released version tags.

Ah, ok!

In that case, I think you need only one more function, IsPseudoVersion(v string) bool, plus the ability to iterate over the versions known to the proxy:

if semver.Max(latest, m.Version) == m.Version {
	allowed := true
	if module.IsPseudoVersion(m.Version) {
		for _, v := range cachedVersions {
			if !module.IsPseudoVersion(v) {
				allowed = false
				break
			}
		}
	} else if semver.Prerelease(m.Version) != "" {
		onlyPseudoVersions := true
		for _, v := range cachedVersions {
			if semver.Prerelease(m.Version) == "" {
				allowed = false
				break
			}
		}
	}
	if allowed {
		// Update latest.
		[…]
	}
}

That's admittedly a fair amount of code, but it's comparable to what we would have to add to cmd/go anyway. Moreover, within cmd/go we would need to do a significant amount of extra work to determine whether the requested version is latest in general, and I don't think we want to do that extra work for every invocation of go mod download -json.

[rsc]

I don't believe you can derive latest-ness from the pseudo-version strings themselves. Whether something is latest depends, for Git, on what master is set to. Master can be changed arbitrarily in ways that don't respect semantic version ordering.

We should provide Latest bool in the go mod download output. Let the go command say for sure instead of making clients guess (or do a second invocation).

[rsc]

I think this will require a little bit of thought and then very little actual coding. If that's the case, it would be good to get this into Go 1.13 so that proxies can use it before 1.14 comes out.

[gopherbot]

Change https://golang.org/cl/183841 mentions this issue: cmd/go: add a Latest field to the output of 'go mod download -json'