Increase the timeout when waiting for the workflows to 20 minutes. (k…

…ubeflow#172)

* We are seeing lots of test flakes doing to temporary problems connecting
  to the K8s master.

Related to kubeflow#169

py/kubeflow/testing/argo_client.py

@@ -30,9 +30,14 @@ def log_status(workflow):
 
 # Wait 2^x * 1 second between retries up to a max of 10 seconds between
 # retries.
-# Retry for a maximum of 3 minutes.
+# Retry for a maximum of 20 minutes.
+# We use a large timeout because we are seeing lots of unavailability with
+# our K8s master in our test cluster
+# See:
+# https://github.com/kubeflow/testing/issues/169
+# https://github.com/kubeflow/testing/issues/171
 @retry(wait_exponential_multiplier=1000, wait_exponential_max=10000,
-       stop_max_delay=3*60*1000,
+       stop_max_delay=20*60*1000,
        retry_on_exception=lambda e: not isinstance(e, util.TimeoutError))
 def wait_for_workflows(client, namespace, names,
                       timeout=datetime.timedelta(minutes=30),