Timm support

[A-Jacobson]

• Timm model wrapper with MosaicClassifier
• Timm Hparams interface
• Timm resnet50 yaml
• working W&B run

@hanlint , @Landanjs

[hanlint]

Overall, this design looks good. We might want a common schema across huggingface and TIMM though. So for hugging-face, our models would then be:

model
  hf:
     model_name: '<name_here>'

Would that work for you @moinnadeem ?

[A-Jacobson]

Timm is all image classification on imagenet so was super straightforward.

looking at HF, they have model factories such as AutoModelForMaskedLM for each task. We can use the same pattern, but we'd probably have to have a wrapper for each factory. @moinnadeem

So hf may look like:

masked language modeling

model
  hf-maskedlm:
     model_name: '<name_here>'

question answering

model
  hf-qa:
    model_name

and so on.

[hanlint]

Does model_name always map to a unique model factory type in HF? If so, we could just require hf, model_name and auto-infer the factory type.

[A-Jacobson]

I don't think so. I'm seeing "Bert-base" show up in multiple examples. Though, I'm not sure of the differences between tasks. It may just be different weights or a different classifier head.

[Landanjs]

LGTM! 2 comments on docstrings and a question on if we should include a **kwargs argument.

@mosaicml/research-engineering does anyone know if we could have a kwargs-like argument in yahp? Maybe defining a dict hparam kwargs: dict = hp.optional(default={}), then using it in a yaml like: kwargs.image_size: 512?

[jbloxham]

LGTM! 2 comments on docstrings and a question on if we should include a **kwargs argument.

@mosaicml/research-engineering does anyone know if we could have a kwargs-like argument in yahp? Maybe defining a dict hparam kwargs: dict = hp.optional(default={}), then using it in a yaml like: kwargs.image_size: 512?

YAHP supports JSON types, at least to some extent. I don't have any experience using it.

[moinnadeem]

I don't think so. I'm seeing "Bert-base" show up in multiple examples. Though, I'm not sure of the differences between tasks. It may just be different weights or a different classifier head.

Yeah, the bert-base-uncased is a single set of weights, but doing AutoModelForClassification.from_pretrained("bert-base-uncased") will create some randomly initialized weights for the classifier head.

In other words, there is a bijection between model names and weights.

[moinnadeem]

Timm is all image classification on imagenet so was super straightforward.

looking at HF, they have model factories such as AutoModelForMaskedLM for each task. We can use the same pattern, but we'd probably have to have a wrapper for each factory. @moinnadeem

So hf may look like:

masked language modeling

model
  hf-maskedlm:
     model_name: '<name_here>'

question answering

model
  hf-qa:
    model_name

and so on.

Hm, we have something very similar to this at the moment. I would like to make our current version more into a factory soon (see how we have very thin wrappers around the current HF configs). However, we do need to insert some additional variables, such as config.num_labels in the wrapper.

[hanlint]

Looks good, a few suggested changes. Do you have a w&b run with the TIMM resnet50 to verify convergence?

[A-Jacobson]

Looks good, a few suggested changes. Do you have a w&b run with the TIMM resnet50 to verify convergence?

https://wandb.ai/mosaic-ml/timm-imagenet/reports/Shared-panel-22-01-30-20-01-86--VmlldzoxNTAzMTky

resnet50 convergence run to 76.89 accuracy. note on the yaml said _quality = '76.51' so it seems we're within the margin for error.

[hanlint]

LGTM