Multiple CI Enhancements

[microdev1]

This contains the following CI enhancements:

• Cache common submodules: extmod/ulab, lib/ and tools/.
• Share CP version across jobs to avoid fetching tags in each and every job.
• Change submodule fetching strategy:
  • tags[Y]: use git submodule update --init for frozen.
  • tags[N]: use git submodule update --init --depth=1 for everything else.
• Only build mpy-cross when required by the board.
• Refactor and alphabetically arrange matrix jobs.
• Use python 3.x for everything except espressif which is on 3.10.

Through multiple CI runs in my fork, I have found the following way of fetching submodules to be ideal:

• For large submodules (e.g. ports/): A shallow-clone with --depth=1.
• For smaller submodules (e.g. extmod/ulab, lib/, tools/): Caching has proven effective.
• For smaller submodules with tags (e.g. frozen): A normal fetch. I tried partial-clone with --filter=tree:0 but it doesn't produce a significant difference for small repos.

[dhalbert]

Thanks for working on this! I saw the build time of the main jobs as 1 hr 10min, which seems like a big improvement, but it's hard to compare with other runs, whose times include delays waiting for runners.

One query inline.

Did you test partial clone with --filter=tree:0 vs depth 1 for the large repos like esp-idf and the broadcom firmware repo?

If we used a partial clone for the main repo, would that make it easier to do the "what is changed checks" without invoking deeper and deeper shallow clones?

[microdev1]

it's hard to compare with other runs, whose times include delays waiting for runners...

Yes, comparing two check suites won't paint the correct picture, I have even noticed a difference of seconds-to-minutes for the same task between runners. I suggest you to take look at individual steps of a job.

Did you test partial clone with --filter=tree:0 vs depth 1 for the large repos like esp-idf and the broadcom firmware repo?

Yes, without tags --depth=1 is better and with tags --filter=tree:0 takes the lead when compared to a normal fetch for these.

If we used a partial clone for the main repo, would that make it easier to do the "what is changed checks" without invoking deeper and deeper shallow clones?

Using partial-clone does take less time when compared to the current --shallow-since and it also removes the need for a fetch in the "what is changed checks" but the difference is in the order of seconds and for a single job it doesn't justify moving away from actions/checkout and dealing with all that complexity.

[dhalbert]

Thanks! Let's try these as well.