Providers requirements for every python version

[pierrejeambrun]

Allow to generate providers requirements for all python versions. Use all python versions compatible with the associated_airflow_version of the provider.

Parallelize image building for different python versions (images with all the airflow versions pre-installed)

Update the output file structure to look like this:

[potiuk]

Comment 1: It takes a long time to build the images with all the airflow versions in but we can optimize it later by pushing them to ghcr.io and using them as cache (should be separate step).

[potiuk]

All looks good I have just a few general comments (and some improvement proposals / test results will come as inline comments).

1. We will need this "all-airflow" images only once to generate all historical versions of providers. But we might find some errors/problems/need to regenerate those so we should keep it. I already have the cache built on my local machine too, so I might push them to our ghcr.io repo and pull rather than rebuild them. I can make some comments on how this can be done easily @pierrejeambrun (might be useful to learn how caching works) - or I can do it as a follow-up PR as you wish, let me know:).

2. Later when we turn it into "generate requirements for the last wave of providers only" the default should be to use the "latest release image" for those tests only - but we should add such capability as separate PR.

[pierrejeambrun]

Thank you for the reviews.

I agree, both updating the release process and image caching are important and should be done in follow up steps.

@potiuk I will address the other comments and do the image caching as a follow up right after. Seems cool to do and I would like to give it a try, I'll let you know if I need some help :)

[potiuk]

Looks fantastic :)

[pierrejeambrun]

Made a few adjustments to allow for more flexibility in the provider_ids/provider_versions combo.

This way we can:

• Regenerate requirements for all historical versions of a specific provider, (or all providers)
• Regenerate requirements for latest versions of a specific provider (or all providers -> release wave)
• Regenerate requirements for a specific provider and version

[potiuk]

Made a few adjustments to allow for more flexibility in the provider_ids/provider_versions combo.

This way we can:

• Regenerate requirements for all historical versions of a specific provider, (or all providers)
• Regenerate requirements for latest versions of a specific provider (or all providers -> release wave)
• Regenerate requirements for a specific provider and version

Whoa. Let me take a look.

[potiuk]

Yeah. That looks really good and flexible to handle both initial generation and regular. Only a small NIT.

[potiuk]

Images need to be regenerated after my changes :(