Support running validation every fixed number of examples

[ehhuang]

Is your feature request related to a problem? Please describe.
For IterableDataset, we may not know the length of the dataset in advance. Running validation every X examples would be helpful.

[fellnerse]

I really would love this! Any updates on that?

[Ir1d]

👍 on this, I guess it should be easy to implement with a Hook

[fellnerse]

I'm able to do something similar that by setting up the trainer like this:

trainer = Trainer(
    ...,
    val_percent_check=kwargs["val_check_interval"],
    val_check_interval=kwargs["val_check_interval"],
)

If you set val_check_interval to (batch_size / len(train_dataset)) * nb_batches validating every nb_batches works. Just be sure to make the DataLoaders you use shuffle the data, otherwise the same data will be used over and over again for validation.

Also the ModelCheckpoint-call back has to be adjusted:

checkpoint_callback = ModelCheckpoint(
    ...,
    period=1 / kwargs["val_check_interval"],
)

Early-stopping seems not be influenced by the validation frequency.

Only caveat is that the validation batches are sampled now randomly from the validation dataset, and it's not guaranteed that all data is used after one epoch. Not too sure if that's an issue tho.