Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug report #7192

Closed
iffthomas opened this issue Nov 3, 2023 · 1 comment
Closed

Bug report #7192

iffthomas opened this issue Nov 3, 2023 · 1 comment

Comments

@iffthomas
Copy link

iffthomas commented Nov 3, 2023

Hi all, I'm rather new to github and couldn't submit it as a bug, I've got a problem with 2 commands:

Will work

"python -m monai.apps.nnunet nnUNetV2Runner train --input_config "./input.yaml" --configs "2d" --trainer_class_name 'nnUNetTrainer_1epoch'"

Will not work

python -m monai.apps.nnunet nnUNetV2Runner train --input_config "./input.yaml" --configs "2d" --trainer_class_name 'nnUNetTrainer_250epoch'"

What I'm aimiing at is to run this command to save time, however it's not working as indended, I can let it train for 1000 epochs but not for 250

python -m monai.apps.nnunet nnUNetV2Runner run --input_config "./input.yaml" --configs "2d" --trainer_class_name 'nnUNetTrainer_250epoch'"

2023-11-03 09:13:00,858 - INFO - number of GPUs is 1, device ids are (0,)
Traceback (most recent call last):
File "/cluster/home/ifft/miniconda3/envs/venv-spine/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/cluster/home/ifft/miniconda3/envs/venv-spine/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/cluster/home/ifft/miniconda3/envs/venv-spine/lib/python3.10/site-packages/monai/apps/nnunet/main.py", line 20, in
fire.Fire({"nnUNetV2Runner": nnUNetV2Runner})
File "/cluster/home/ifft/miniconda3/envs/venv-spine/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/cluster/home/ifft/miniconda3/envs/venv-spine/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/cluster/home/ifft/miniconda3/envs/venv-spine/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/cluster/home/ifft/miniconda3/envs/venv-spine/lib/python3.10/site-packages/monai/apps/nnunet/nnunetv2_runner.py", line 589, in train
self.train_single_model(config=cfg, fold=_fold, gpu_id=gpu_id_for_all, **kwargs)
File "/cluster/home/ifft/miniconda3/envs/venv-spine/lib/python3.10/site-packages/monai/apps/nnunet/nnunetv2_runner.py", line 538, in train_single_model
run_training(
File "/cluster/home/ifft/miniconda3/envs/venv-spine/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 189, in run_training
nnunet_trainer = get_trainer_from_args(dataset_name_or_id, configuration, fold, trainer_class_name,
File "/cluster/home/ifft/miniconda3/envs/venv-spine/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 42, in get_trainer_from_args
raise RuntimeError(f'Could not find requested nnunet trainer {trainer_name} in '
RuntimeError: Could not find requested nnunet trainer nnUNetTrainer_250epoch in nnunetv2.training.nnUNetTrainer (/cluster/home/ifft/miniconda3/envs/venv-spine/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer). If it is located somewhere else, please move it there.

Expected behavior
Both command should work, one training only 1 epoch, the other one 250 epochs but for only the 2d configuration (also doesn't work with nnUNetV2Runner run command

Environment

python -c 'import monai; monai.config.print_debug_info()'

================================
Printing MONAI config...

MONAI version: 1.3.0
Numpy version: 1.26.0
Pytorch version: 2.0.1
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 865972f
MONAI file: /cluster/home//miniconda3/envs/venv-spine/lib/python3.10/site-packages/monai/init.py

Optional dependencies:
Pytorch Ignite version: NOT INSTALLED or UNKNOWN VERSION.
ITK version: 5.3.0
Nibabel version: 5.1.0
scikit-image version: 0.20.0
scipy version: 1.11.3
Pillow version: 10.0.1
Tensorboard version: NOT INSTALLED or UNKNOWN VERSION.
gdown version: NOT INSTALLED or UNKNOWN VERSION.
TorchVision version: 0.15.2
tqdm version: 4.66.1
lmdb version: NOT INSTALLED or UNKNOWN VERSION.
psutil version: 5.9.6
pandas version: 2.1.1
einops version: NOT INSTALLED or UNKNOWN VERSION.
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: NOT INSTALLED or UNKNOWN VERSION.
pynrrd version: NOT INSTALLED or UNKNOWN VERSION.
clearml version: NOT INSTALLED or UNKNOWN VERSION.

For details about installing the optional dependencies, please visit:
https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

================================
Printing system config...

System: Linux
Linux version: CentOS Linux 7 (Core)
Processor:
Machine: x86_64
Python version: 3.10.13
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Num physical CPUs: 20
Num logical CPUs: 40
Num usable CPUs: 2
CPU usage (%): [9.1, 3.5, 2.8, 3.7, 7.1, 4.7, 0.2, 0.2, 0.2, 0.0, 1.2, 0.5, 0.4, 0.3, 0.2, 0.0, 100.0, 100.0, 0.1, 0.5, 80.3, 1.4, 0.4, 0.0, 0.0, 0.0, 0.1, 5.6, 0.0, 0.0, 100.0, 0.1, 0.0, 0.0, 0.0, 100.0, 0.0, 0.0, 100.0, 5.8]
CPU freq. (MHz): 1763
Load avg. in last 1, 5, 15 mins (%): [20.2, 18.6, 18.2]
Disk usage (%): 4.9
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 249.6
Available memory (GB): 214.4
Used memory (GB): 30.7

================================
Printing GPU config...

Num GPUs: 1
Has CUDA: True
CUDA version: 11.8
cuDNN enabled: True
NVIDIA_TF32_OVERRIDE: None
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE: None
cuDNN version: 8700
Current device: 0
Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90', 'compute_37']
GPU 0 Name: NVIDIA GeForce GTX 1080 Ti
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 28
GPU 0 Total memory (GB): 10.9
GPU 0 CUDA capability (maj.min): 6.1

@iffthomas
Copy link
Author

Okay sooo what the problem was, was that the ReadMe was missleading: to specify
some were

nnUNetTrainer_250epochs

An s is missing in the tutorial here:
https://github.com/Project-MONAI/tutorials/tree/main/nnunet

wyli pushed a commit to Project-MONAI/tutorials that referenced this issue Nov 3, 2023
Fixes #Project-MONAI/MONAI#7192

### Checks
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [ ] Avoid including large-size files in the PR.
- [ ] Clean up long text outputs from code cells in the notebook.
- [ ] For security purposes, please check the contents and remove any
sensitive info such as user names and private key.
- [ ] Ensure (1) hyperlinks and markdown anchors are working (2) use
relative paths for tutorial repo files (3) put figure and graphs in the
`./figure` folder
- [ ] Notebook runs automatically `./runner.sh -t <path to .ipynb file>`

Signed-off-by: KumoLiu <yunl@nvidia.com>
@KumoLiu KumoLiu closed this as completed Nov 3, 2023
boneseva pushed a commit to boneseva/MONAI-tutorials that referenced this issue Apr 21, 2024
Fixes #Project-MONAI/MONAI#7192

### Checks
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [ ] Avoid including large-size files in the PR.
- [ ] Clean up long text outputs from code cells in the notebook.
- [ ] For security purposes, please check the contents and remove any
sensitive info such as user names and private key.
- [ ] Ensure (1) hyperlinks and markdown anchors are working (2) use
relative paths for tutorial repo files (3) put figure and graphs in the
`./figure` folder
- [ ] Notebook runs automatically `./runner.sh -t <path to .ipynb file>`

Signed-off-by: KumoLiu <yunl@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants