Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

hotfix to overcome xarray chunking error #900

Closed
6 tasks
AlexanderMeisinger opened this issue Oct 18, 2023 · 3 comments · Fixed by #901
Closed
6 tasks

hotfix to overcome xarray chunking error #900

AlexanderMeisinger opened this issue Oct 18, 2023 · 3 comments · Fixed by #901
Labels
bug Something isn't working

Comments

@AlexanderMeisinger
Copy link
Contributor

AlexanderMeisinger commented Oct 18, 2023 •
edited
Loading

Checklist

  • I tested my contribution locally and it seems to work fine.
  • Code and workflow changes are sufficiently documented.
  • Changed dependencies are added to envs/environment.yaml.
  • Changes in configuration options are added in all of config.default.yaml.
  • Changes in configuration options are also documented in doc/configtables/*.csv.
  • A release note doc/release_notes.rst is added.

Describe the Bug

Installation Problem Pypsa-Earth-Sec: Error in rule build_renewable_profiles after setting new environment via mamba env create -f envs/environment.yaml

Error Message

/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
INFO:distributed.scheduler:State start
INFO:distributed.scheduler:  Scheduler at:     tcp://127.0.0.1:34625
INFO:distributed.scheduler:  dashboard at:  http://127.0.0.1:8787/status
INFO:distributed.scheduler:Registering Worker plugin shuffle
INFO:distributed.nanny:        Start Nanny at: 'tcp://127.0.0.1:41765'
/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
INFO:distributed.worker:      Start worker at:      tcp://127.0.0.1:33721
INFO:distributed.worker:         Listening to:      tcp://127.0.0.1:33721
INFO:distributed.worker:          Worker name:                          0
INFO:distributed.worker:         dashboard at:            127.0.0.1:44759
INFO:distributed.worker:Waiting to connect to:      tcp://127.0.0.1:34625
INFO:distributed.worker:-------------------------------------------------
INFO:distributed.worker:              Threads:                          1
INFO:distributed.worker:               Memory:                  15.49 GiB
INFO:distributed.worker:      Local Directory: /tmp/dask-scratch-space/worker-gm756qca
INFO:distributed.worker:-------------------------------------------------
INFO:distributed.scheduler:Register worker <WorkerState 'tcp://127.0.0.1:33721', name: 0, status: init, memory: 0, processing: 0>
INFO:distributed.scheduler:Starting worker compute stream, tcp://127.0.0.1:33721
INFO:distributed.worker:Starting Worker plugin shuffle
INFO:distributed.worker:        Registered to:      tcp://127.0.0.1:34625
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:59124
INFO:distributed.worker:-------------------------------------------------
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:34625
INFO:distributed.scheduler:Receive client connection: Client-a565a91a-6803-11ee-9e51-00155dc9c55a
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:59134
/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/xarray/core/dataset.py:270: UserWarning: The specified chunks separate the stored chunks along dimension "time" starting at index 100. This could degrade performance. Instead, consider rechunking after loading.
  warnings.warn(
INFO:__main__:Calculate landuse availabilities...
/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
INFO:__main__:Completed availability calculation (38.89s)
INFO:atlite.convert:Convert and aggregate 'wind'.
INFO:atlite.convert:Convert and aggregate 'wind'.
Traceback (most recent call last):
  File "/mnt/c/Users/mea39219/Documents/H2Global_meets_Africa/Gitlab/pypsa-earth-sec/pypsa-earth/.snakemake/scripts/tmpo2gbuzs7.build_renewable_profiles.py", line 751, in <module>
    profile, capacities = func(
  File "/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/atlite/convert.py", line 521, in wind
    return cutout.convert_and_aggregate(
  File "/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/atlite/convert.py", line 174, in convert_and_aggregate
    results = aggregate_matrix(da, matrix=matrix, index=index)
  File "/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/atlite/aggregate.py", line 19, in aggregate_matrix
    return xr.apply_ufunc(
  File "/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/xarray/core/computation.py", line 1249, in apply_ufunc
    return apply_dataarray_vfunc(
  File "/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/xarray/core/computation.py", line 308, in apply_dataarray_vfunc
    result_var = func(*data_vars)
  File "/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/xarray/core/computation.py", line 754, in apply_variable_ufunc
    raise ValueError(
ValueError: dimension spatial on 0th function argument to apply_ufunc with dask='parallelized' consists of multiple chunks, but is also a core dimension. To fix, either rechunk into a single array chunk along this dimension, i.e., ``.chunk(dict(spatial=-1))``, or pass ``allow_rechunk=True`` in ``dask_gufunc_kwargs`` but beware that this may significantly increase memory usage.
INFO:distributed.nanny:Closing Nanny at 'tcp://127.0.0.1:41765'. Reason: nanny-close
INFO:distributed.nanny:Nanny asking worker to close. Reason: nanny-close
INFO:distributed.worker:Stopping worker at tcp://127.0.0.1:33721. Reason: nanny-close
INFO:distributed.core:Connection to tcp://127.0.0.1:34625 has been closed.
INFO:distributed.core:Received 'close-stream' from tcp://127.0.0.1:59124; closing.
INFO:distributed.scheduler:Remove worker <WorkerState 'tcp://127.0.0.1:33721', name: 0, status: closing, memory: 0, processing: 0> (stimulus_id='handle-worker-cleanup-1697007603.6887875')
INFO:distributed.scheduler:Lost all workers
INFO:distributed.scheduler:Scheduler closing due to unknown reason...
INFO:distributed.scheduler:Scheduler closing all comms
[Wed Oct 11 09:00:06 2023]
INFO:snakemake.logging:[Wed Oct 11 09:00:06 2023]
Error in rule build_renewable_profiles:
    jobid: 3
    input: resources/natura.tiff, data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, data/gebco/GEBCO_2021_TID.nc, resources/shapes/country_shapes.geojson, resources/shapes/offshore_shapes.geojson, data/hydro_capacities.csv, data/eia_hydro_annual_generation.csv, resources/powerplants.csv, resources/bus_regions/regions_onshore.geojson, cutouts/cutout-2013-era5.nc
    output: resources/renewable_profiles/profile_onwind.nc
    log: logs/build_renewable_profile_onwind.log (check log file(s) for error details)

ERROR:snakemake.logging:Error in rule build_renewable_profiles:
    jobid: 3
    input: resources/natura.tiff, data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, data/gebco/GEBCO_2021_TID.nc, resources/shapes/country_shapes.geojson, resources/shapes/offshore_shapes.geojson, data/hydro_capacities.csv, data/eia_hydro_annual_generation.csv, resources/powerplants.csv, resources/bus_regions/regions_onshore.geojson, cutouts/cutout-2013-era5.nc
    output: resources/renewable_profiles/profile_onwind.nc
    log: logs/build_renewable_profile_onwind.log (check log file(s) for error details)

RuleException:
CalledProcessError in file /mnt/c/Users/mea39219/Documents/H2Global_meets_Africa/Gitlab/pypsa-earth-sec/pypsa-earth/Snakefile, line 410:
Command 'set -euo pipefail;  /home/mea39219/anaconda3/envs/pypsa-earth/bin/python3.10 /mnt/c/Users/mea39219/Documents/H2Global_meets_Africa/Gitlab/pypsa-earth-sec/pypsa-earth/.snakemake/scripts/tmpo2gbuzs7.build_renewable_profiles.py' returned non-zero exit status 1.
  File "/mnt/c/Users/mea39219/Documents/H2Global_meets_Africa/Gitlab/pypsa-earth-sec/pypsa-earth/Snakefile", line 410, in __rule_build_renewable_profiles
  File "/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/concurrent/futures/thread.py", line 58, in run
ERROR:snakemake.logging:RuleException:
CalledProcessError in file /mnt/c/Users/mea39219/Documents/H2Global_meets_Africa/Gitlab/pypsa-earth-sec/pypsa-earth/Snakefile, line 410:
Command 'set -euo pipefail;  /home/mea39219/anaconda3/envs/pypsa-earth/bin/python3.10 /mnt/c/Users/mea39219/Documents/H2Global_meets_Africa/Gitlab/pypsa-earth-sec/pypsa-earth/.snakemake/scripts/tmpo2gbuzs7.build_renewable_profiles.py' returned non-zero exit status 1.
  File "/mnt/c/Users/mea39219/Documents/H2Global_meets_Africa/Gitlab/pypsa-earth-sec/pypsa-earth/Snakefile", line 410, in __rule_build_renewable_profiles
  File "/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message```
@AlexanderMeisinger AlexanderMeisinger added the bug Something isn't working label Oct 18, 2023
@ekatef
Copy link
Member

ekatef commented Oct 18, 2023

Hello @AlexanderMeisinger! Thank you so much for reporting it 🙂
Environment errors are usually the most annoying. So, that is really great to have it pinned!

Would you like to open a PR to fix it? You may add a restriction on xarray version, as it has been done in this PR of PyPSA-Eur.

@AlexanderMeisinger AlexanderMeisinger mentioned this issue Oct 18, 2023
Merged
4 tasks
@davide-f davide-f mentioned this issue Oct 18, 2023
Closed
@davide-f
Copy link
Member

Thanks @AlexanderMeisinger ! this is indeed an interesting temporary fix!
I mentioned the issue in atlite as well, as once #324 is fixed, this constraint should be removed.

@AlexanderMeisinger
Copy link
Contributor Author

Hey @davide-f ,
Thanks for the merge! Glad I could contribute to the community.
Best regards, Alex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Xarray version change AlexanderMeisinger/pypsa-earth
3 participants
@ekatef @davide-f @AlexanderMeisinger