KFP SDK: suggest retry if pipeline launch returns 500 error ('warmup' apparently required)

[amygdala]

I've noticed that if I use the SDK to launch a pipeline run from a python script, I very often get a http 500 error if I haven't hit that endpoint recently, with: HTTP response body: upstream connect error or disconnect/reset before headers.
I don't know if the root cause is IAP-related or something else. If I retry again, things are fine. Apparently some kind of 'warmup' is required.

(Btw, this error is not related to the one described here, AFAICT: #788)

I'm working around this by catching the error and retrying in the client code, but I'm thinking the SDK itself should probably support some small number of retries.

[Ark-kun]

I see the same in my browser after not using it for a while.

[rileyjbauer]

This seems like a duplicate of #908. I also filed an issue on the main repo, kubeflow/kubeflow#2852 as I'm not yet sure where this problem is coming from, but it seems like it's deeper in the system