Skip to content
This repository has been archived by the owner on Oct 13, 2022. It is now read-only.

fix some bugs/issues of the training scripts #7

Merged
merged 1 commit into from
Nov 13, 2020

Conversation

qindazhu
Copy link
Collaborator

No description provided.

@@ -44,17 +46,7 @@
supervisions=manifests['supervisions']).compute_and_store_features(
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a warning here with the latest lhotse

UserWarning: ProcessPoolExecutor using a "fork" multiprocessing detected. In some circumstances this can crash or hang the program, e.g. when using  data augmentation libsox wrappers like WavAugment or torchaudio. We suggest passing an extra argument to initialize the executor: ProcessPoolExecutor(..., mp_context=multiprocessing.get_context("spawn"))
  warnings.warn('ProcessPoolExecutor using a "fork" multiprocessing detected. '

I believe it can be fixed with @pzelasko's PR https://github.com/k2-fsa/snowfall/pull/5/files

@danpovey
Copy link
Contributor

Thanks! Merging.

@qindazhu
Copy link
Collaborator Author

The scripts can now fully run with latest k2 and lhotse, but still have some issues:

  1. Sometimes start_frame + num_frames > T in K2SpeechRecognitionIterableDataset lhotse-speech/lhotse#125 causes we cannot run many batches successfully.
  2. The modle output is diverging, may because learning rate or model struct, we definitely need to look into this.
  3. Sometimes we may get Cuda out of memory, but because 1, we can not repro it.

@danpovey danpovey merged commit 27931f1 into k2-fsa:master Nov 13, 2020
@danpovey danpovey mentioned this pull request Jun 21, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants