build(ci): Retry failed pulls from devservices

[billyvg]

This will retry failed pulls from devservices. This happens intermittently in CI (GHA), see https://github.com/getsentry/getsentry/runs/4069112530\?check_suite_focus\=true\#step:4:563

I am not sure of the root cause as docker-py obscures the actual error (docker/docker-py#2101), as a workaround, let us retry the pull a few times and see if this helps.

[armenzg]

Would it be good if we raise something if we run out of attempts? Or log an error?

[billyvg]

That's what https://github.com/getsentry/sentry/pull/29691/files#diff-1125ff5a13bc41637ad61c8bd8edbc84dc7a8436b8a6ad3a8b95f16d65156913R63 does

[armenzg]

Than you, sir! I just had a moment :)

[armenzg]

As a side note, should we switch to using docker pull on CI? We could have some script that extracts the versions from SENTRY_DEV_SERVICES in server.py

[billyvg]

I think we should stick to using sentry devservices and maybe in the future evaluate if we want to keep using docker-py over something like docker-compose?

[joshuarli]

I think we should stick to using sentry devservices and maybe in the future evaluate if we want to keep using docker-py over something like docker-compose?

As a side note, should we switch to using docker pull on CI? We could have some script that extracts the versions from SENTRY_DEV_SERVICES in server.py

I still think docker-py in devservices is the better choice right now. One thing that could be worth exploring is putting these pulls on multiple threads. That could help saturate network read and it would be especially a good speedup in case failed pulls happen. Maybe docker pull does some smarter things and is faster than docker-py though. In either case we don't need a script like that, just subprocess out to docker pull in each thread.