Ability to re-trigger failed build with the same input versions

[davewalter]

When using the new version: every configuration for a get task, it is possible to arrive at a state where you have multiple builds of the same job running at the same time. If an earlier build fails, there is no way to re-trigger it with the same set of inputs. We haven't been able to determine a useful workaround; setting serial: true doesn't really help in this scenario, because the next build will start as soon as the first one fails.

It would be helpful if there were a way to re-trigger the job with the same inputs as a particular build (failed or otherwise).

Let us know if you need more details on this scenario or our desired fix. Thanks!

@davewalter and @rmasand

[vito]

We're thinking about splitting today's trigger build button (+). It is primarily used for three things today:

1. Impatience: I just pushed something or know that someone just published something, and I want the build to run now.
2. Retrying a build (this issue): I want to re-run the current build, either to see if it's flaky or to retry because something outside the build failed (e.g. github, a deployment, etc.).
3. Triggering a job that only ever manually runs, e.g. shipping a product after you've written release notes.

The flaw with case 1 is that there's a race condition. In the time between you loading the page and clicking the +, Concourse may have already found your stuff and queued a build. Now you have two, which is annoying.

The flaw with case 2 is you can only do it with the latest build, and also if you triggered a bunch, new versions may come in, potentially invalidating your flakiness trial. You could set version to a particular version in your pipeline, but that's annoying.

Case 3 pretty much works, but you don't know what versions it'll use until you run it. See #269

So, I think we should split + into two buttons. One that lives on the job, "sync", which will make sure everything's up-to-date and then queue up a build if it should (i.e. one's not queued already; same semantics as auto-triggering). The other button would be associated with a particular build of the job, and would re-trigger it with the same inputs. This covers cases 1 and 2.

The third case needs some more thinking since a "sync" button alone doesn't intuitively seem like enough given that the build only manually triggers.

[endzyme]

+1 this would be a great, and much needed, feature for Concourse CI

[charlieoleary]

+1 As well, this is a pretty critical feature. We've sort of circumvented it with empty commits (since we're using the PR resource), but it would be ideal to simply retry a failed job with the same inputs.

[ahelal]

Would love to see that. We are doing crazy stuff to try to trigger old commits. Is this open for external people to help ? and if so how ?

[primalmotion]

same thing here, I feel that concourse has everything to be able to do this fairly easily. It would work perfectly with the pull request resource.

[tracker-common]

+1000000 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍👍 👍👍👍 👍👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍👍 👍👍👍 👍👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍👍 👍👍👍 👍👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍👍 👍👍👍 👍👍

[gabro]

Same here, if you guys are busy can we help somehow?

[miromode]

+1

[kei-yamazaki]

👍

[timrchavez]

👍 This would go a long way to making Concourse a more viable choice for us.

[ls-yann-david]

Pleaseeeeee 👍

[VanAxe]

Retrigerring jobs would be a lot more elegant than empty commits! 😝 👍

[olhtbr]

👍

[tsantero]

is this issue currently being actively worked on internally? I need this asap, but also don't want to duplicate work. similarly, are there any contributor guidelines I should read?

[vito]

Notes from IPM:

• Preserve inputs on re-triggered build (duh) - make sure the scheduler/build starter doesn't re-compute them or use next_build_inputs
• We'll need a button in the UI for re-triggering (@Lindsayauchin), in addition to the existing trigger-build button
• We'll need to clear out the build events for the old build and "re-set" the build back to pending state (and reset whatever other build data is appropriate, i.e. start/end time)
• Also: clear out any outputs for the build, otherwise a re-trigger from green to red could result in "phantom outputs" satisfying passed constraints
• Re-compute build plan based on current pipeline config
  • If anyone is worried about this let us know - it's way easier to implement this way, and we figure there may be cases where a pipeline config change was made to fix the errant build anyway, in which case we'd want to pick up the new config.

We'll also spike on creating a new build instead of replacing it. They may actually be roughly the same difficulty. If we do this instead, we don't have to reset anything or clear out any outputs/etc.

[StevenArmstrong]

Could the trigger buttons be moved out from under the job and put on the main page somehow? It could have a trigger icon with a prompt or something? I say this as one of the main feedbacks our product lead gets of our concourse pipelines is around users calling the interface awkward and complaining they have to click through the green/red square of a job to trigger it. To mitigate this we have had to have a job called trigger for production deployments so users know how to trigger a production deployment. As a result of the clicking around we have even had requests to build a UI on top of concourse to make it easier and more dev friendly for roll forward and rollback triggers :(

[vito]

@StevenArmstrong So having a specific 'retrigger' button on failed builds in the pipeline? Sounds reasonable, though I would argue people should probably be clicking into the build first and understanding the failure rather than blindly re-triggering. 🤔 In any case we may want to discuss that as a separate issue that we can address after we take our first crack at this. :)

[hstenzel]

One question I have is how we can retrigger a build if we no longer have the log from the original?

[vito]

@hstenzel Build logs are purely cosmetic, the actual information regarding which versions/etc. are used is kept in the database and so re-triggering will still work.

[hstenzel]

Perhaps I'm missing something then. How would I retrigger a build for which I no longer have logs if the UI element is on the log screen?

…

On Mon, May 13, 2019, 2:58 PM Alex Suraci ***@***.***> wrote: @hstenzel <https://github.com/hstenzel> Build logs are purely cosmetic, the actual information regarding which versions/etc. are used is kept in the database and so re-triggering will still work. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#413?email_source=notifications&email_token=ABOOX5Z275ZG2V2MKEHPXW3PVG23TA5CNFSM4CCWGJC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVJHUGA#issuecomment-491944472>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABOOX52QDUJWEQEVIGCHT3DPVG23TANCNFSM4CCWGJCQ> .

[vito]

The only thing removed when logs are reaper are the build logs below the build header. The rest of the ui is still there and shows the build status and such. On Mon, May 13, 2019, 5:54 PM Harley Stenzel <notifications@github.com> wrote:

…

Perhaps I'm missing something then. How would I retrigger a build for which I no longer have logs if the UI element is on the log screen? On Mon, May 13, 2019, 2:58 PM Alex Suraci ***@***.***> wrote: > @hstenzel <https://github.com/hstenzel> Build logs are purely cosmetic, > the actual information regarding which versions/etc. are used is kept in > the database and so re-triggering will still work. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > < #413?email_source=notifications&email_token=ABOOX5Z275ZG2V2MKEHPXW3PVG23TA5CNFSM4CCWGJC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVJHUGA#issuecomment-491944472 >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/ABOOX52QDUJWEQEVIGCHT3DPVG23TANCNFSM4CCWGJCQ > > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#413?email_source=notifications&email_token=AAAAOWBU5MKVXKG2O76VNB3PVHPPRA5CNFSM4CCWGJC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVJVPLA#issuecomment-492001196>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAAOWCKKMZ2KKD2YDHMA7TPVHPPRANCNFSM4CCWGJCQ> .

[StevenArmstrong]

@vito it wasn't really just for retrigger. It was more if you are doing a redesign on what the + buttons or other buttons do it would be really good if they weren't nested under the job as users have said they find it confusing having to click through to manually initiate a new build when deploying to production. Instead I was suggesting having them a layer up as an icon on the pipeline page beside each job. You could then have a confirmation pop up if someone clicks it by mistake to confirm to build with latest or retrigger or cancel. This way you wouldn't need multiple buttons simply 1 button to build with multiple sub options without having to click through to the log to trigger anything. It's something that is frequently fed back about the UI from our users.

[Lindsayauchin]

@StevenArmstrong interesting idea. I think that from pipeline we have observed, like the (rabbit MQ team at Pivotal below) an action button to trigger a job on the pipeline page is just not scalable.

We are thinking about the user pains around triggering a build with the work being done on the resource version. You can follow a related issue #3403 to see our progress on the UX changes.

[hstenzel]

If I understand correctly, to retrigger a specific job I'd scroll left/right on the build page to find the correct job?

Also, I'd potentially want to retrigger a successful job too, thinking about the case of rebuilding an artifact that was accidentally lost.

Perhaps these questions are really more about the UI related to the feature.

[vito]

@hstenzel Yep - to re-trigger a build you would do so from the build's page. There's no such thing as re-triggering a job - you can trigger a new build of a job, but the re- part of re-trigger means you're running an already-existing build for a second time with the same inputs.

You will be able to re-trigger a build regardless of whether it succeeded or failed; they both have their use case: re-trigger a succeeded build to detect flakes, re-trigger failed build to allow artifacts to continue along the pipeline.

[StevenArmstrong]

@Lindsayauchin most people use pipeline groups on concourse to visualise big pipelines with many jobs. So I still think in combination with pipeline groups it could still be viable to have the trigger icons on the pipeline page.

[Lindsayauchin]

This has evolved into the Build re-triggering track of work. Iterative designs have been moved to smaller sliced stories and can be found in the Build re-triggering project here: https://github.com/concourse/concourse/projects/24

[vito]

For those following along: this has been implemented and will be in v6.0! I don't have an ETA yet since v6.0 includes very substantial internal changes that we're doing due diligence to test out. We're considering shipping a beta release first.

[vito]

Closing this out!

As implemented in v6.0, re-running a build will create a new build named e.g. "123.1", which will apprear adjacent to the original build in the build history. This placement in the history reflects the scheduler's iteration order for passed constraints - i.e. if you re-run a very old build it won't suddenly propagate the older versions downstream if there are "newer" successful builds after the re-run.

The new build will run with a newly constructed build plan based on the pipeline's current configuration, using the versions of each input from the original build. This should work in the common case of re-triggering recent flakes, but it can fail if the configuration has changed such that new get steps have been added, or the old version is no longer available because the resource changed. Hope that's good enough for MVP!

In the future we plan to fix this by having re-runs run with the the exact build plan that the original ran with, rather than constructing a new plan based on the current configuration. This is going to be tracked in a separate "epic", Build Lifecycle.