Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problematic rebuilding of a merged branch's pipeline #72

Closed
sino92 opened this issue Jan 23, 2019 · 4 comments
Closed

Problematic rebuilding of a merged branch's pipeline #72

sino92 opened this issue Jan 23, 2019 · 4 comments
Assignees
@sino92
Labels
bug Something isn't working

Comments

@sino92
Copy link
Contributor

sino92 commented Jan 23, 2019

Steps To Replicate:

  • Merge a branch
  • Realize you want to reopen the PR
  • Make changes on the same branch again
  • The pipeline build can't be triggered from Jenkins (empty)

Going to OC and starting the pipeline there works
@michaelsauter
Copy link
Member

Could you add the following info:

  • What is the error output you see in the triggered pipeline?
  • Does this happen both for pushing new commits to the branch (pipeline triggered via webhook) and when the pipeline is triggered via "Build Now" in Jenkins?

When I had a brief look together with you, this looked credentials related - somehow the pipeline seemed to forget the credentials ...

@sino92
Copy link
Contributor Author

sino92 commented Jan 28, 2019

I believe the credential issue we looked together was a combination of this case and an additional case: merge was made through command line and without approval.
As for this specific issue, there's no error when the pipeline is triggered from a push. But when triggered via "Build Now" we get:

SEVERE: Timer task io.fabric8.jenkins.openshiftsync.BuildSyncRunListener$1@151dd47d failed
  | java.lang.NullPointerException
  | at io.fabric8.jenkins.openshiftsync.BuildSyncRunListener.upsertBuild(BuildSyncRunListener.java:336)
  | at io.fabric8.jenkins.openshiftsync.BuildSyncRunListener.pollRun(BuildSyncRunListener.java:226)
  | at io.fabric8.jenkins.openshiftsync.BuildSyncRunListener.pollLoop(BuildSyncRunListener.java:202)
  | at io.fabric8.jenkins.openshiftsync.BuildSyncRunListener$1.doRun(BuildSyncRunListener.java:159)
  | at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
  | at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
  | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  | at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  | at java.lang.Thread.run(Thread.java:748)

Jan 28, 2019 8:53:23 PM io.fabric8.jenkins.openshiftsync.CredentialsUtils upsertCredential
  | INFO: Updated credential b2p2-cd-cd-user-with-password from Secret NamespaceName{b2p2-cd:cd-user-with-password} with revision: 127468902
  | Jan 28, 2019 8:53:23 PM org.eclipse.jetty.server.handler.ContextHandler$Context log
  | WARNING: Error while serving https://jenkins-b2p2-cd.22ad.bi-x.openshiftapps.com/job/b2p2-cd/job/b2p2-cd-app-570/build
  | java.lang.reflect.InvocationTargetException
  | at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:347)
  | at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)
  | at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)
  | at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129)
  | at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
  | at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:734)
  | at org.kohsuke.stapler.Stapler.invoke(Stapler.java:864)
  | at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:248)
  | at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
  | at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:734)
  | at org.kohsuke.stapler.Stapler.invoke(Stapler.java:864)
  | at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:248)
  | at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
  | at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:734)
  | at org.kohsuke.stapler.Stapler.invoke(Stapler.java:864)
  | at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
  | at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
  | at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
  | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
  | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
  | at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
  | at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftPermissionFilter.doFilter(OpenShiftPermissionFilter.java:242)
  | at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  | at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243)
  | at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  | at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
  | at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  | at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
  | at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  | at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
  | at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  | at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
  | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
  | at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64)
  | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
  | at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
  | at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
  | at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  | at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
  | at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  | at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
  | at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  | at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
  | at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  | at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
  | at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  | at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
  | at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  | at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
  | at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
  | at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  | at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
  | at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
  | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
  | at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
  | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
  | at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
  | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
  | at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
  | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
  | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
  | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
  | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
  | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
  | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
  | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
  | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
  | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
  | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
  | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
  | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
  | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
  | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
  | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
  | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
  | at org.eclipse.jetty.server.Server.handle(Server.java:531)
  | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
  | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
  | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
  | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
  | at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
  | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
  | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
  | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
  | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
  | at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
  | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
  | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
  | at java.lang.Thread.run(Thread.java:748)
  | Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kubernetes.default/apis/build.openshift.io/v1/namespaces/b2p2-cd/buildconfigs/app-570/instantiate . Message: builds.build.openshift.io "app-570-2" already exists. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=build.openshift.io, kind=builds, name=app-570-2, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=builds.build.openshift.io "app-570-2" already exists, metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=AlreadyExists, status=Failure, additionalProperties={}).
  | at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:472)
  | at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:411)
  | at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:381)
  | at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)
  | at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:328)
  | at io.fabric8.openshift.client.dsl.internal.BuildConfigOperationsImpl.instantiate(BuildConfigOperationsImpl.java:148)
  | at io.fabric8.openshift.client.dsl.internal.BuildConfigOperationsImpl.instantiate(BuildConfigOperationsImpl.java:68)
  | at io.fabric8.jenkins.openshiftsync.BuildDecisionHandler.shouldSchedule(BuildDecisionHandler.java:66)
  | at hudson.model.Queue.schedule2(Queue.java:590)
  | at hudson.model.Queue.schedule2(Queue.java:713)
  | at jenkins.model.ParameterizedJobMixIn.doBuild(ParameterizedJobMixIn.java:217)
  | at jenkins.model.ParameterizedJobMixIn$ParameterizedJob.doBuild(ParameterizedJobMixIn.java:408)
  | at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
  | at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343)
  | ... 88 more
 

@michaelsauter
Copy link
Member

@sino92 Thanks for the log output, that might help to find the issue. Did you already search for this error message? Could also be a known issue elsewhere ...

I am still pretty certain that merging via the CLI has no implications here.

@michaelsauter
Copy link
Member

I figured out what causes this.

When the webhook proxy deletes the pipeline, it does not delete the builds of that pipeline. Say you had 3 builds. When the pipeline is deleted and re-created later, it is done so without information about previous builds (lastVersion field in status is not set), however OpenShift somehow remembers the previous builds. New commits are blocked by the old builds. When lastVersion is modified to match the previous number of builds, then a new commit works - also when pushed via Git.

I guess we have to either delete the previous builds and/or remember lastVersion when deleting a pipeline ...

@michaelsauter michaelsauter transferred this issue from opendevstack/ods-jenkins-shared-library Feb 4, 2019
@michaelsauter michaelsauter added the bug Something isn't working label Feb 4, 2019
michaelsauter added a commit to BIX-Digital/ods-core that referenced this issue Feb 5, 2019
@michaelsauter
Delete associated builds in the background
aa51e78 
This prevents builds from being orphaned, see:
- https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/
-
https://docs.openshift.com/container-platform/3.11/rest_api/apis-build.openshift.io/v1.BuildConfig.html#Delete-apis-build.openshift.io-v1-namespaces-namespace-buildconfigs-name

Fixes opendevstack#72.
@michaelsauter michaelsauter mentioned this issue Feb 5, 2019
Merged
gerardcl pushed a commit that referenced this issue Oct 7, 2019
@michaelsauter
Delete associated builds in the background
9e8b4a2 
This prevents builds from being orphaned, see:
- https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/
-
https://docs.openshift.com/container-platform/3.11/rest_api/apis-build.openshift.io/v1.BuildConfig.html#Delete-apis-build.openshift.io-v1-namespaces-namespace-buildconfigs-name

Fixes #72.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sino92 sino92

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@michaelsauter @sino92