Streaming vision datasets

[knighton]

No description provided.

[abhi-mosaic]

Looks good so far! Left a few comments, looking forward to some end-to-end testing with streaming the newly-created datasets from S3 before approving this PR. Quick reminder that for the big datasets, we probably don't want to test more than 1 or 2 epochs because it will cost $$$ in egress (until we have cacheing).

We can also postpone support for cacheing to a future PR, since it is currently still broken in webdataset. Context: webdataset/webdataset#131

[ravi-mosaicml]

@knighton curious about the status of this PR?

[knighton]

@ravi-mosaicml webdataset is giving weird issues

[knighton]

item 1 is fixed in commit b6ead1a

for item 3, some minor differences are expected behavior. these sharded streaming datasets are new datasets

[abhi-mosaic]

I started to test this and talked to James about these comments, but I thought I should add them here:

1. pip install --user -e .[all] failed due to conflicting dependencies: ERROR: Cannot install awscli==1.22.67, mosaicml and mosaicml[all]==0.4.0 because these package versions have conflicting dependencies.
2. If a download fails, then is retried, it will use the folder from the failed attempt and fail again. James might have adjusted this one?
3. It appears the calculation for number of iterations is off. The training ends in ~900 fewer iterations (~1%) and the LR schedule is also adjusted:

The pink line is webdatasets and the blue line is my original runs. The performance seems to be a bit worse, but it is possible that it is within noise. I would need to do more runs, but I can try ResNet50 ImageNet first.

@Landanjs Just want to confirm that the # samples is expected to be lower, this is due to the constraints of sharded datasets. Basically we need an equal number of samples per shard, so n_samples % n_shards == 0 and to accomplish this we truncate the dataset. So we have lost a few examples for each vision dataset but I think @knighton was careful to keep the loss small to around ~1%.

This probably means that baselines need to be re-run, for all vision tasks. I would love to know if the results are meaningfully different or within the noise! I'm hoping the latter :)