Code

Streaming vision datasets#284

Code

Merged

hanlint merged 63 commits intodevmosaicml/composer:devfrom james/wdsmosaicml/composer:james/wdsCopy head branch name to clipboardMar 8, 2022

+1,185-93

Commits

Commits on Jan 27, 2022

• build_(vision datasets).

  knighton

  committed
  b6dab36

  Copy full SHA for b6dab36

Commits on Jan 28, 2022

• Streaming dataset: Web(MNIST, CIFAR-10/20/100, TinyImagenet-200, Imagenet-1K).

  knighton

  committed
  4b4e8d4

  Copy full SHA for 4b4e8d4

Commits on Feb 1, 2022

• WebDataset index files that indirect num_shards, (load, create)_webdataset.

  knighton

  committed
  302f666

  Copy full SHA for 302f666

• Make uniform.

  knighton

  committed
  4a61894

  Copy full SHA for 4a61894

Commits on Feb 2, 2022

• Link up webdataset creation -> load_dataset -> dataset hparam init.

  knighton

  committed
  aa89dda

  Copy full SHA for aa89dda

• Fix.

  knighton

  committed
  84e8254

  Copy full SHA for 84e8254

• isort.

  knighton

  committed
  fa22cf0

  Copy full SHA for fa22cf0

• yapf.

  knighton

  committed
  75aa4a8

  Copy full SHA for 75aa4a8

• build/ -> create/ ("build" is overloaded term).

  knighton

  committed
  842029c

  Copy full SHA for 842029c

• pyright (typing annotations).

  knighton

  committed
  ec582ff

  Copy full SHA for ec582ff

• Rm imagenet1k-multiproc example.

  knighton

  committed
  2fd7f53

  Copy full SHA for 2fd7f53

Commits on Feb 3, 2022

• Typing.

  knighton

  committed
  43023d6

  Copy full SHA for 43023d6

Commits on Feb 8, 2022

• isort, etc.

  knighton

  committed
  679e0f2

  Copy full SHA for 679e0f2

Commits on Feb 11, 2022

• ADE20k.

  knighton

  committed
  adb3289

  Copy full SHA for adb3289

Commits on Feb 14, 2022

• Refactor: dataset_s3_bucket.

  knighton

  committed
  9f5f561

  Copy full SHA for 9f5f561

Commits on Feb 22, 2022

• Fix (sharding).

  knighton

  committed
  3172c4c

  Copy full SHA for 3172c4c

Commits on Feb 23, 2022

• fix wds length

  abhi-mosaic

  committed
  48c4c52

  Copy full SHA for 48c4c52

• add wds cifar10 yaml

  abhi-mosaic

  committed
  cd34c82

  Copy full SHA for cd34c82

• add deps

  abhi-mosaic

  committed
  72ac7d4

  Copy full SHA for 72ac7d4

• minor fixes

  abhi-mosaic

  committed
  42f2bb4

  Copy full SHA for 42f2bb4

• typo

  abhi-mosaic

  committed
  142245b

  Copy full SHA for 142245b

Commits on Feb 24, 2022

• size_webdataset().

  knighton

  committed
  c41a94c

  Copy full SHA for c41a94c

• shuffle (following webdataset defaults).

  knighton

  committed
  9b56693

  Copy full SHA for 9b56693

Commits on Feb 25, 2022

• Add YAMLs.

  knighton

  committed
  47fafb9

  Copy full SHA for 47fafb9

Commits on Feb 26, 2022

• Fix.

  knighton

  committed
  e1a4d90

  Copy full SHA for e1a4d90

Commits on Feb 28, 2022

• Split JpgCls webdataset hparams.

  knighton

  committed
  ccafa8c

  Copy full SHA for ccafa8c

Commits on Mar 1, 2022

• Extract hyperparam for shuffle_buffer_per_worker.

  knighton

  committed
  55822f6

  Copy full SHA for 55822f6

Commits on Mar 3, 2022

• Merge branch 'dev' into james/wds

  knighton

  committed
  3c818ee

  Copy full SHA for 3c818ee

• Remove directory of webdataset versions of yamls save one, minor renames web_ to wds_.

  knighton

  committed
  6263224

  Copy full SHA for 6263224

• Make dataset_s3_bucket configurable, add typing annotations.

  knighton

  committed
  eb41cc4

  Copy full SHA for eb41cc4

• Drop use_synthetic from WebDataset datasets.

  knighton

  committed
  78bae50

  Copy full SHA for 78bae50

• Merge branch 'dev' into james/wds

  knighton

  authored
  9e4780b

  Copy full SHA for 9e4780b

• Make s3 bucket configurable for all webdatasets.

  knighton

  committed
  c997cbf

  Copy full SHA for c997cbf

• Make naming more uniform, make all wds s3/local configurable, etc.

  knighton

  committed
  a8947d5

  Copy full SHA for a8947d5

• Merge remote-tracking branch 'refs/remotes/origin/james/wds' into james/wds

  knighton

  committed
  b4c33b7

  Copy full SHA for b4c33b7

Commits on Mar 4, 2022

• Update ResNet50 yaml

  Landanjs

  committed
  2c91dd6

  Copy full SHA for 2c91dd6

• Fix: use_synthetic.

  knighton

  committed
  d0a96f7

  Copy full SHA for d0a96f7

• Fix (remove mixin).

  knighton

  committed
  c480aad

  Copy full SHA for c480aad

• Raise on error instead of silently failing.

  knighton

  committed
  25a3dd5

  Copy full SHA for 25a3dd5

• load_webdataset: s3_bucket -> remote (either s3 or local fs).

  knighton

  committed
  d831033

  Copy full SHA for d831033

• Set up webdataset with one call.

  Show description for aa45054
  knighton

  committed
  aa45054

  Copy full SHA for aa45054

• Class variables <-> yahp.

  knighton

  committed
  6e03882

  Copy full SHA for 6e03882

• Weaken docutils requirement to make .[all] happy.

  knighton

  committed
  b6ead1a

  Copy full SHA for b6ead1a

• Optional webdataset.

  knighton

  committed
  bfd2e4c

  Copy full SHA for bfd2e4c

Commits on Mar 5, 2022

• Also Wurlitzer.

  knighton

  committed
  1a75f8e

  Copy full SHA for 1a75f8e

• Lint.

  knighton

  committed
  e25f125

  Copy full SHA for e25f125

• Docstrings.

  knighton

  committed
  e0223d5

  Copy full SHA for e0223d5

• Merge branch 'dev' into james/wds

  knighton

  authored
  eecc937

  Copy full SHA for eecc937

• Python usage too modern.

  knighton

  committed
  13035c0

  Copy full SHA for 13035c0

• Skip over webdatasets (wds_).

  knighton

  committed
  cb84613

  Copy full SHA for cb84613

Commits on Mar 7, 2022

• Fix (split).

  knighton

  committed
  8df5392

  Copy full SHA for 8df5392

• Test hparams abstract base class solution.

  knighton

  committed
  a7b4334

  Copy full SHA for a7b4334

• Required hparams before optional hparams.

  knighton

  committed
  1d8fb9d

  Copy full SHA for 1d8fb9d

• All optional.

  knighton

  committed
  40eae59

  Copy full SHA for 40eae59

• Fix (default).

  knighton

  committed
  c525d3b

  Copy full SHA for c525d3b

• Merge branch 'dev' into james/wds

  hanlint

  authored
  523dc41

  Copy full SHA for 523dc41

• fix typing

  hanlint

  committed
  74e8dda

  Copy full SHA for 74e8dda

• fix lint

  hanlint

  committed
  aee6084

  Copy full SHA for aee6084

• lazy import load_webdataset

  hanlint

  committed
  393f2d5

  Copy full SHA for 393f2d5

• Skip over WebDatasets properly.

  knighton

  committed
  a14bf6c

  Copy full SHA for a14bf6c

• Ported Dockerfile README to rst, style fixes

  

  Bandish Shah

  committed
  c3cea69

  Copy full SHA for c3cea69

• Merge branch 'james/wds' of github.com:mosaicml/composer into james/wds

  

  Bandish Shah

  committed
  e8681e2

  Copy full SHA for e8681e2

• Minor doc tweak

  

  Bandish Shah

  committed
  a6ef182

  Copy full SHA for a6ef182