task: Pretraining dataset for Multilingual Quantizer training (Phase 1)

[hahuyhoang411]

Goal

Gathering all the data opensource available for the first multilingual training run. This test will focus mostly in Vietnamese dataset.

Tasklist

Multi-lingual Base

• LibriSpeech (CC-BY-4.0) or check parler-tss for the cleaned version
• Mozilla Common Voice (CC-0)
• GigaSpeech2 Refined (Vie, Thai & Indo, Apache-2.0)
• FLEURS (CC-BY-4.0)
• Emilia (CC-BY-NC-4.0)
• ...

Vietnamese

• ViVoice (1000+ hours, CC-BY-NC-SA-4.0)
• BUD500 (500+ hours, CC-BY-NC-SA-4.0)
• VLSP (100 hours)

Singlish

• National Speech Corpus (Singapore Open Data License)

[tuanlda78202]

These datasets will be saved on S3 (which can be useful for connecting RunPod later) and will be saved on disk with the A6000s cluster for training Quantizer Whisper Encoder and Ichigo.

[tuanlda78202]

Downloaded full Vietnamese datasets from HF and upload to S3, check it!

[tikikun]

is this duplication?

[dan-menlo]

Can we add the license information next to each dataset - I am concerned that we are using some datasets that are non-commercial licensed

[tuanlda78202]

Just updated license information for each dataset. I think this issue is done, and we can close.

[hahuyhoang411]

All good thanks @tuanlda78202