Filter external repos

[phackstock]

Closes #326.

This PR adds the feature to filter external repositories using include and exclude filters.
Any number of filters combining any attributes can be defined, example:

repositories:
  common-definitions:
    url: https://github.com/IAMconsortium/common-definitions.git/
definitions:
  variable:
    repository:
      name: common-definitions
      include:
        - name: [Primary Energy*, Final Energy*]
        - name: "Population*"
          tier: 1
      exclude:
        - name: "Final Energy|*|*"
  region:
    repository:
      name: common-definitions
      include:
        - hierarchy: [R5, R10]

For the variable section

in the example above we are including:

1. All variables starting with Primary Energy or Final Energy
2. All variables starting with Population and with the tier attribute equal to 1

From this list we are then excluding all variables that match "Final Energy||".
This means that the final resulting list will contain no Final Energy variables with
three or more levels.

For the region section

we are taking only R5 and R10 regions.

Changes

One of the changes that I have made is that all repositories in the definition section need to have the name key-word.
This would be a breaking change so all workflow repos that use external repositories would need to be updated.
I'd be happy to streamline it again so that just the name is allowed but for code simplicity I opted against that for now.

[danielhuppmann]

A few suggestions inline.

As with my other review, I’d strongly advise against using fnmatch.

[danielhuppmann]

Quick clarifying question, one can only have one include and one exclude statement per repo-import, right?

[phackstock]

Quick clarifying question, one can only have one include and one exclude statement per repo-import, right?

Correct. You can put multiple filters in both the include and exclude though.

[phackstock]

@danielhuppmann, I added a depth option now so you can match for the given level of variable depth.
I didn't implement the "1-" option, since it would make the code a lot less read- and maintainable and using fnmatch we already have that functionality.
Say you want to select "Final Energy" with the depth "1-" option, in pyam logic. You'd use

...
      include:
        - name: Final Energy*
          depth: [0, 1]

which I think is plenty readable.
Since this option uses fnmatch, I chose to keep it. If you want me to remove fnmatch, I can do that, however for now using it made the most sense to me.

[phackstock]

@danielhuppmann, as discussed bilaterally, I changed the string pattern matching to use the pyam character escape and regex.
Should be good now.

[danielhuppmann]

Thanks, looks good, a few minor suggestions inline.

One larger question: if an imported model-mapping includes a common region that is filtered out in the region-codelist, the initialization of the region-processor will fail, right? (That's ok, just make sure to create a follow-up issue so that this can be tackled later).

[phackstock]

One larger question: if an imported model-mapping includes a common region that is filtered out in the region-codelist, the initialization of the region-processor will fail, right? (That's ok, just make sure to create a follow-up issue so that this can be tackled later).

Looking at the code it looked fine to me but just to be sure I added a test in 60d77ce. I import all the mappings from common-definitions and only load the world region. It fails as expected.

[danielhuppmann]

Nice, thank you!