Extend Git retry mechanism to git push race conditions on secrets/templates repos

[gberche-orange]

Expected behavior

As a coab operator

• in order to avoid coab provisionning failure due to race conditions on secret/template repos https://github.com/orange-cloudfoundry/paas-templates/issues/1257
• I expect the configured retry policy from

  cf-ops-automation-broker/cf-ops-automation-broker-core/src/main/java/com/orange/oss/cloudfoundry/broker/opsautomation/ondemandbroker/git/RetryProperties.java

  Lines 8 to 26 in ef9776f

  	public class RetryProperties {
  	private int maxAttempts=4;
  	private boolean exponentialBackOff=true;
  	/**
  	* the delay to occur between retries
  	* See {@link RetryPolicy#withDelay(Duration)}
  	*/
  	private int delayMilliSeconds=5000;
  	/**
  	* when backoff is turned on, the max delay for each retries
  	* See {@link RetryPolicy#withBackoff(long, long, ChronoUnit)}
  	*/
  	private int maxDelayMilliSeconds=60000;
  	/**
  	* the max duration to perform retries for, else the execution will be failed.
  	* See {@link RetryPolicy#withMaxDuration(Duration)}
  	*/
  	private int maxDurationMilliSeconds=50000;

  to also be applied to rebases while avoiding too long delays

Observed behavior

The retry mechanism is apparently not working and failing at 1st push error

   2021-08-17T13:07:47.01+0000 [APP/PROC/WEB/0] OUT java.lang.IllegalArgumentException: java.lang.RuntimeException: failed to push: remote conflict. re-push following rebase failed with: secondPushFailure
s=[REJECTED_NONFASTFORWARD]
   2021-08-17T13:07:47.01+0000 [APP/PROC/WEB/0] OUT 	at com.orange.oss.cloudfoundry.broker.opsautomation.ondemandbroker.git.SimpleGitManager.commitPushRepo(SimpleGitManager.java:363) ~[cf-ops-automatio
n-broker-core-1.8.0.jar:1.8.0]
   2021-08-17T13:07:47.01+0000 [APP/PROC/WEB/0] OUT 	at com.orange.oss.cloudfoundry.broker.opsautomation.ondemandbroker.git.RetrierGitManager.lambda$commitPushRepo$4(RetrierGitManager.java:51) ~[cf-ops
-automation-broker-core-1.8.0.jar:1.8.0]
   2021-08-17T13:07:47.01+0000 [APP/PROC/WEB/0] OUT 	at net.jodah.failsafe.Functions.lambda$toSupplier$11(Functions.java:248) ~[failsafe-2.4.1.jar:2.4.1]
   2021-08-17T13:07:47.01+0000 [APP/PROC/WEB/0] OUT 	at net.jodah.failsafe.Functions.lambda$get$0(Functions.java:48) ~[failsafe-2.4.1.jar:2.4.1]
   2021-08-17T13:07:47.01+0000 [APP/PROC/WEB/0] OUT 	at net.jodah.failsafe.RetryPolicyExecutor.lambda$supply$0(RetryPolicyExecutor.java:62) ~[failsafe-2.4.1.jar:2.4.1]
   2021-08-17T13:07:47.01+0000 [APP/PROC/WEB/0] OUT 	at net.jodah.failsafe.Execution.executeSync(Execution.java:128) ~[failsafe-2.4.1.jar:2.4.1]
   2021-08-17T13:07:47.01+0000 [APP/PROC/WEB/0] OUT 	at net.jodah.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376) ~[failsafe-2.4.1.jar:2.4.1]
   2021-08-17T13:07:47.01+0000 [APP/PROC/WEB/0] OUT 	at net.jodah.failsafe.FailsafeExecutor.run(FailsafeExecutor.java:223) ~[failsafe-2.4.1.jar:2.4.1]
   2021-08-17T13:07:47.01+0000 [APP/PROC/WEB/0] OUT 	at com.orange.oss.cloudfoundry.broker.opsautomation.ondemandbroker.git.RetrierGitManager.commitPushRepo(RetrierGitManager.java:49) ~[cf-ops-automati
on-broker-core-1.8.0.jar:1.8.0]
   2021-08-17T13:07:47.01+0000 [APP/PROC/WEB/0] OUT 	at com.orange.oss.cloudfoundry.broker.opsautomation.ondemandbroker.git.PooledGitManager.commitPushRepo(PooledGitManager.java:69) ~[cf-ops-automation
-broker-core-1.8.0.jar:1.8.0]
   2021-08-17T13:07:47.01+0000 [APP/PROC/WEB/0] OUT 	at com.orange.oss.cloudfoundry.broker.opsautomation.ondemandbroker.git.GitProcessor.commitPushRepo(GitProcessor.java:72) ~[cf-ops-automation-broker-
core-1.8.0.jar:1.8.0]
   2021-08-17T13:07:47.01+0000 [APP/PROC/WEB/0] OUT 	at com.orange.oss.cloudfoundry.broker.opsautomation.ondemandbroker.git.GitProcessor.postCreate(GitProcessor.java:24) ~[cf-ops-automation-broker-core
-1.8.0.jar:1.8.0]
   2021-08-17T13:07:47.01+0000 [APP/PROC/WEB/0] OUT 	at com.orange.oss.cloudfoundry.broker.opsautomation.ondemandbroker.processors.ProcessorChain.create(ProcessorChain.java:46) ~[cf-ops-automation-brok

[gberche-orange]

The git retrier mechanism only focussed 502 transport exceptions, handleIf(e -> isCauseSubclassOf(e, org.eclipse.jgit.api.errors.TransportException.class)) see

cf-ops-automation-broker/cf-ops-automation-broker-core/src/main/java/com/orange/oss/cloudfoundry/broker/opsautomation/ondemandbroker/git/RetrierGitManager.java

Lines 16 to 24 in 72c09b2

	public RetrierGitManager(String repositoryAliasName, GitManager gitManager, RetryPolicy<Object> retryPolicy) {
	this.gitManager = gitManager;
	this.retryPolicy = retryPolicy;
	this.retryPolicy
	.onRetry(e -> logger.warn("Transient (?) failure, retrying. Cause: {}",
	e.getLastFailure().toString(), e.getLastFailure()))
	.onRetriesExceeded(e -> logger.warn("Aborting. Max attempts reached: #" + this.retryPolicy.getMaxAttempts() +
	" or max duration reached (" + this.retryPolicy.getMaxDuration().toString() + "). Rethrowing failure:" + e.getFailure()))
	.handleIf(e -> isCauseSubclassOf(e, org.eclipse.jgit.api.errors.TransportException.class))

Updated the issue title to consider this as an improvement