Stacked @card v1.2 : Customizing @card with current.card

[valayDave]

Stacked on : #893

Summary of Changes

This PR is an attempt to make MetaflowCards customizable. Metaflow already ships with MetaflowCardComponents. These are isolated components for a MetaflowCard. With this PR users will be able to dynamically add components in @step code. This PR introduces the current.card interface to collect MetaflowCardComponents from userland (@step code). This PR also introduces the id argument in the @card decorator and the --id option in the card create command.

Core Changes

1. Introduce a contruct called current.card to collect components from @step code
2. Addition of id property to @card and cli. id helps resolve custom components to the right card when using multiple decorators.

Since there can be many cards, this PR introduces the concept of user-editable cards. This seemed a necessary bifurcation because decorators can be dynamically added at runtime. So to distinguish what cards users can editing we provide MetaflowCards with a ALLOW_USER_COMPONENTS attribute.

If ALLOW_USER_COMPONENTS is set to True then the MetaflowCard is eligible to have components added from the @step code. We will call this an editable card.

Once the card the editable, we can collect custom components from the step code in the following way :

@card(type='blank',id='a')
@card(type='default')
@step
def train(self):
    from metaflow.cards import Markdown
    from metaflow import current
    current.card['a'].append(Markdown('# This is present in the blank card with id "a"'))
    current.card.append(Markdown('# This is present in the default card'))
    self.t = dict(
        hi = 1,
        hello = 2
    )
    self.next(self.end)

Consider the above snippet. In the above scenario there are two @card decorators. One card is of type blank and another of type default. The current.card object helps collect "components" from @step code to customize some card. In this case, it is a Markdown component. current.card only accepts objects which are subclasses of MetaflowCardComponent. Since it is the responsibility of current.card to resolve a component to the card mentioned in the decorator, current.card first tries to resolve a default editable card. A default editable card is a card that will have access to the current.card.append/current.card.extend methods. Since there can be many decorators and hence many cards, current.card.append resolves this once the last @card decorators call task_pre_step. In the above case when the user calls append, metaflow will only add Markdown to the default card. current.card['a'].append will add the Markdown to the blank card. Here a is an id that can help resolve the component to the card. Since cards can be of many types, some cards can also not be user-editable (Cards with ALLOW_USER_COMPONENTS=False). Those cards won't be eligible to access the current.card.append

current.card (CardComponentCollector)

The CardComponentCollector is the object responsible for resolving a MetaflowCardComponent to the card provided in the @card decorator.

Since there can be many cards, CardComponentCollector has a _finalize function. The _finalize function is called once the last @card decorator calls task_pre_step. The _finalize function will try to find the default editable card from all the @card decorators on the @step. The default editable card is the card that can access the current.card.append/current.card.extend methods. If there are multiple editable cards with no id then current.card will throw warnings when users call current.card.append. This is done because we cannot resolve which card the component belongs.

The @card decorator also exposes another argument called customize=True. Only one @card decorator over a @step can have customize=True. Since cards can also be added from CLI when running a flow, adding @card(customize=True) will set that particular card from the decorator as default editable. This means that current.card.append will append to the card belonging to @card with customize=True. If there is more than one decorator with customize=True we throw warnings that current.card.append won't append to any card.

One important feature of the current.card object is that it will not fail. Even when users try to access current.card.append with multiple editable cards, we throw warnings but don't fail. current.card will also not fail when a user tries to access a card of a non-existing id via current.card['mycard']. Since current.card['mycard'] gives reference to a list of MetaflowCardComponents, we return a non referenced list when users try to access current.card['my_non_existant_card']

TLDR

• current.card.append customizes the default editable card.
• Only one card can be default editable in a step.
• The card class must have ALLOW_USER_COMPONENTS=True to be considered default editable.
• Classes with ALLOW_USER_COMPONENTS=False are never default editable.
• The user can specify an id argument to a card, in which case the card is editable through current.card[id].append.
• A card with an id can be also defaulted editable if there are no other cards that are eligible to be default editable.
• If multiple default-editable cards exist but only one card doesn’t have an id, the card without an id is considered to be default editable.
• If we can’t resolve a single default editable card through the above rules, current.card.append calls show a warning but the call doesn’t fail.
• A card which is not default editable can be still edited through:
  • current.card['myid']
  • by looking it up by its type, e.g. current.card.get(type=’pytorch’).

Technical Details of Changes

Since the resolution of the default editable card is important for current.card, we need to ensure that we call the _finalize function when the last @card decorator calls task_pre_step. To keep track of this the changes introduce class methods to set the total number of card decorators per @step and increment a global step counter keeping a track of the number of decorators that have called task_pre_step. Once the last @card per @step calls task_pre_step we call the _finalize function.

The total number of card decorators per step is set in the step_init callback and the instantiation/addition of cards to the CardComponentCollector happens in the task_pre_step. Once the user code finishes running we serialize the MetaflowCardComponents added to current.card. CardComponentCollector has a _serialize function which helps
serialize the components of the card to a JSON list or a list of strings. In the serialization process it calls the MetaflowCardComponent.render() function. The MetaflowCardComponent.render() is expected to return a JSON serializable dict or string. This list is json dumped to a tempfile and passed to the card create subprocess.

The create command in the card_cli.py reads the component file as-is and passes the serialized component list when instantiating the card class.

[romain-intel]

Didn't look at everything in detail but hopefully some useful comments.

[romain-intel]

multiple typos in these two lines :)

[romain-intel]

simplify: return evt_name in self._called_once