dataflow/run_template: slightly increase max retry time in test (#4692)

* Slightly increase max retry time in test

Fixes #4686

It should be an uncommon occurrence that a job hangs in queue for more than 4 minutes, probably because many tests were triggered at a similar time.
I'm slightly increasing the max retry time to 5 minutes to leave more room for jobs to be cancelled, but most of the time it will be cancelled much quicker.

* Update dataflow/run_template/main_test.py

Co-authored-by: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>

Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com>
Co-authored-by: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>

dataflow/run_template/main_test.py

@@ -34,6 +34,8 @@
 import main
 
 
+RETRY_MAX_TIME = 5 * 60  # 5 minutes in seconds
+
 PROJECT = os.environ['GOOGLE_CLOUD_PROJECT']
 BUCKET = os.environ['CLOUD_STORAGE_BUCKET']
 
@@ -83,7 +85,7 @@ def get_job_id_from_name(job_name):
 
 
 # We retry the cancel operation a few times until the job is in a state where it can be cancelled
-@backoff.on_exception(backoff.expo, HttpError, max_time=240)
+@backoff.on_exception(backoff.expo, HttpError, max_time=RETRY_MAX_TIME)
 def dataflow_jobs_cancel(job_name):
     # to cancel a dataflow job, we need its ID, not its name
     job_id = get_job_id_from_name(job_name)