Support recursive globbing (**/) for download paths

[jameshadfield]

When downloading assets from large builds (I'm looking at you nCoV) the --download arg is immensely helpful. Currently to download the intermediate files the nCoV workflow one needs to supply an argument for each directory, e.g.

--download results/* --download results/translations/* --download results/global/* --download results/asia/* ...

How feasible would it be to support recursive globbing, such as --download results/**?

[tsibley]

Yeah, this would be nice!

I wanted ** from the beginning, but support for it isn't built into Python's standard library, so I postponed implementing it. I don't know if there's a decent library focused just on path-matching/glob-parsing out there, but even if not, it should be pretty feasible to implement.

[tsibley]

Just a small note that I stumbled across Sphinx's path matching routines recently, which include an implementation of ** support. I haven't surveyed what else is available, and that due diligence should probably still be done, but this is one source to look at again in the future. Sphinx is BSD licensed, and so we could easily vendor their impl. as the basis for ours and adapt as needed.

[tsibley]

Following up to say that I think wcmatch is the library we want here: either wcmatch.glob's globmatch() or wcmatch.pathlib's Path.globmatch().

It turns out this sort of matching was going to be really handy for some other work I'm doing, and my searches this time turned up wcmatch. Trialing it out, it works great and has minimal deps. Aim is to match Bash, which is exactly what we want.

Once wcmatch is in the codebase, we can loop back and update the build --download flag to use it.