Latest docker image only works for Concourse 6.1

[matt-royal]

Today we started to see an error with this output when puting to our concourse-pipeline resource:

Logging to /tmp/concourse-pipeline-resource-out.log671813192
2020/05/14 23:40:05 exit status 1 - WARNING:

fly version (6.1.0) is out of sync with the target (5.8.1). to sync up, run the following:

    /opt/resource/fly -t https://ci.prs.releng.rocks sync

error: oauth2: cannot fetch token: 401 Unauthorized
Response: {"error":"invalid_client","error_description":"Invalid client credentials."}

When we dug into the problem we noticed that the fly version (6.1.0) that now ships in the latest Docker image exits with a 1 when fly login is called and the CLI version is out of sync. Previous versions of fly login warned in this situation, but exited 0. This change breaks the logic in fly.Command.Login(), which assumes that login will succeed even if the CLI is out of sync, and it can THEN sync it.

We're still on Concourse 5.8.1, but I imagine this same problem will happen for people on Concourse 6.0.* or (in the future) 6.2.*. For now we've switched from the latest tag of the Docker image to 2.2.0 to avoid the problem.

If you feel this is an unintended breaking change in the fly CLI, we can instead file an issue there.

[matt-royal]

It looks like the fix here is as simple as calling fly sync before fly login. fly sync now takes a -c flag for the Concourse URL.

$ fly -v
6.1.0

$ fly sync -c https://ci.prs.releng.rocks
downloading fly from https://ci.prs.releng.rocks...

fly v5.8.1 [================================================] 19.5MiB/19.5MiB
done

$ fly -v
5.8.1

[vito]

Ugh. This broke when I fixed #68. Sorry, I think we just can't really expect this resource type to always be stable when tracking latest tag. I would just give it a proper tag - perhaps only specifying the major version. That's probably a good practice for most resource types anyhow. I shipped this as a major bump because I figured some things would break with the login changes in v6.1.

Supporting this resource type is like playing whack-a-mole. Keeping the version in sync with the outer Concourse and having to configure local auth for it is painful. I would recommend trying out the set_pipeline step if you can, and using tags to pin this specific resource type. We will be deprecating this resource type as soon as the set_pipeline step is finished, so it's probably a good idea to start trying it out early. We don't anticipate any major changes to it at this point, even though it's marked "experimental."

[matt-royal]

@vito Thank you for the quick response. The one limitation I see with set_pipeline that would prevent us from using it is that the pipeline name appears to be static. We generate dynamically-named pipelines based on the inputs. Are there plans to support that?

[vito]

@matt-royal The plan for that use case is concourse/rfcs#34 (pipeline instances). Rather than dynamically generating names, you would configure the pipelines under a common name and use "instance vars" to identify them further, e.g.:

set_pipeline: feature
instance_vars: {branch: foo}

This will be coming up soon in the backlog as it's part of the v10 roadmap. We should get to it as soon as archiving pipelines is done. It'll be there by the time we deprecate this resource type, in any case.

[matt-royal]

@vito Thank you. That might meet our needs. Will that proposal require static templates with only variables differing? Some of our upcoming features will require enough variance in pipeline setup that we're planning on using dynamic templating (possibly ytt) to vary the tasks and resources in each job.

[vito]

@matt-royal Cool! The set_pipeline step just takes a file: parameter, so you could use ytt in a prior step to generate the pipeline .yml however you like. The instance_vars: are passed along so you can use them for Concourse ((var)) syntax, but you don't have to actually use them in the template - they can just be used as identifiers.