Adjust for genno 1.6, pyam 1.0 #492

khaeru · 2021-07-06T08:55:26Z

As detected in Ensure years_active() uses sorted results #491:
- Improve performance of generating the full-scale MESSAGEix reporting graph.
  
  This was a performance regression introduced in genno 1.5.x (Make Key.dims order-insensitive khaeru/genno#42); now fixed by Improve performance of Computer.check_keys() and related khaeru/genno#47 / version 1.6.0, and it cause some tutorial notebooks to exceed the timeout limit of 10 seconds per cell.
- Use in rep not in rep.graph for more flexible genno.Computer.__contains__().
Adjust to pyam 1.0.

In particular, IamDataFrame.line_plot() was marked deprecated in 0.10 (Refactor to plotting API following pandas/matplotlib implementation IAMconsortium/pyam#473) and then removed as of 1.0, without an intervening major release. This broke the westeros_report.ipynb tutorial.

How to review

No review: internal/test suite only.

PR checklist

Continuous integration checks all ✅
Add or expand tests; coverage checks both ✅
~~Add, expand, or update documentation.~~
~~Update release notes.~~

Use "in rep" not "in rep.graph" for more flexible __contains__.

codecov · 2021-07-06T09:10:40Z

Codecov Report

Merging #492 (673af12) into main (56611ad) will not change coverage.
The diff coverage is 100.0%.

❗ Current head 673af12 differs from pull request most recent head 2df44f8. Consider uploading reports for the commit 2df44f8 to get more accurate results

@@          Coverage Diff          @@
##            main    #492   +/-   ##
=====================================
  Coverage   93.3%   93.3%           
=====================================
  Files         37      37           
  Lines       2819    2819           
=====================================
  Hits        2632    2632           
  Misses       187     187

Impacted Files	Coverage Δ
message_ix/reporting/computations.py	`100.0% <ø> (ø)`
message_ix/tests/test_reporting.py	`100.0% <100.0%> (ø)`

khaeru · 2021-07-06T15:05:45Z

I'm also seeing cell timeouts on this branch, which don't occur locally. The cause might be a performance regression in genno; will investigate further.

khaeru · 2021-07-07T11:32:26Z

At this point, timeouts occur:

macos-latest-py3.8 / message_ix/tests/test_tutorials.py::test_tutorial[R_austria_load_scenario]
macos-latest-py3.9 / message_ix/tests/test_tutorials.py::test_tutorial[R_austria]

These are both unrelated to genno/pyam, so they just seem to be flaky.

Former name was removed in pyam 1.0

khaeru · 2021-07-07T12:47:42Z

Again a timeout for:

macos-latest-py3.9 / message_ix/tests/test_tutorials.py::test_tutorial[R_austria]
…but not for the other.

I will try one more time to restart the tests, then exclude this case.

khaeru · 2021-07-07T13:12:39Z

Okay, now all pass/no timeouts.

commit 900a9f8 Author: Laura Wienpahl <57132039+LauWien@users.noreply.github.com> Date: Mon Aug 23 14:56:33 2021 +0200 Update documentation "Common issues" (iiasa#500) commit 38461e6 Author: Laura Wienpahl <57132039+LauWien@users.noreply.github.com> Date: Fri Aug 20 16:04:17 2021 +0200 Update installation documentation using Anaconda (iiasa#501) * Update installation documentation using Anaconda * Update conda CI checks accordingly commit a4ae7ff Author: OFR-IIASA <fricko@iiasa.ac.at> Date: Wed Aug 18 14:09:31 2021 +0200 Tutorial explaining historical parameters (iiasa#478) * Add tutorial for historical parameters * Update README * Update release notes commit 65b0767 Author: Laura Wienpahl <laura@wienpahl.de> Date: Tue Aug 17 14:04:04 2021 +0200 Update release notes commit 44c43d8 Author: Laura Wienpahl <laura@wienpahl.de> Date: Tue Aug 17 13:20:55 2021 +0200 Adjust test suite for pyam v1.1.0 compatibility commit f94304c Author: Laura Wienpahl <57132039+LauWien@users.noreply.github.com> Date: Tue Jul 20 10:23:28 2021 +0200 Adjust objective values for nightly test scenarios (iiasa#495) commit 6d19561 Merge: 6bf19c3 4eaef6f Author: Paul Natsuo Kishimoto <mail@paul.kishimoto.name> Date: Mon Jul 19 16:05:54 2021 +0200 Merge pull request iiasa#493 from iiasa/misc-2021-W28 Use np.isscalar; improve contributing.rst commit 4eaef6f Author: Paul Natsuo Kishimoto <mail@paul.kishimoto.name> Date: Sun Jul 18 15:40:09 2021 +0200 Update doc/contributing.rst commit 85d9cb7 Author: Paul Natsuo Kishimoto <mail@paul.kishimoto.name> Date: Sun Jul 18 15:20:57 2021 +0200 Use np.isscalar(), not deprecated ixmp.utils.isscalar commit 6bf19c3 Merge: 164cb4e ba4254b Author: Laura Wienpahl <57132039+LauWien@users.noreply.github.com> Date: Tue Jul 13 15:54:05 2021 +0200 Merge pull request iiasa#474 from volker-krey/soft_constraints_clean Adjust reference new capacity and activity for soft constraints commit ba4254b Author: OFR-IIASA <fricko@iiasa.ac.at> Date: Tue Jul 13 09:03:29 2021 +0200 Update RELEASE_NOTES.rst commit 7a87538 Author: OFR-IIASA <fricko@iiasa.ac.at> Date: Tue Jul 13 09:00:49 2021 +0200 Update RELEASE_NOTES.rst commit 440d78c Merge: 5fc853c 164cb4e Author: OFR-IIASA <fricko@iiasa.ac.at> Date: Tue Jul 13 08:45:53 2021 +0200 Merge branch 'main' into soft_constraints_clean commit 164cb4e Merge: 19eff0f 8849073 Author: Paul Natsuo Kishimoto <mail@paul.kishimoto.name> Date: Fri Jul 9 12:35:54 2021 +0200 Merge pull request iiasa#491 from OFR-IIASA/year_active_fix Ensure `years_active()` uses sorted results commit 8849073 Author: FRICKO Oliver <fricko@iiasa.ac.at> Date: Fri Jul 9 08:20:26 2021 +0200 Update release notes based on review comment commit 73e3c9c Author: FRICKO Oliver <fricko@iiasa.ac.at> Date: Mon Jul 5 12:07:21 2021 +0200 Apply black formatting commit 9fd6303 Author: FRICKO Oliver <fricko@iiasa.ac.at> Date: Mon Jul 5 12:05:50 2021 +0200 Add iiasa#491 to release notes commit f16b397 Author: FRICKO Oliver <fricko@iiasa.ac.at> Date: Mon Jul 5 11:55:08 2021 +0200 Modify years_active function to use sorted results commit 1889ba2 Author: FRICKO Oliver <fricko@iiasa.ac.at> Date: Mon Jul 5 11:53:43 2021 +0200 Add test to ensure years_active uses sorted results commit 19eff0f Merge: 56611ad 2df44f8 Author: Paul Natsuo Kishimoto <mail@paul.kishimoto.name> Date: Wed Jul 7 15:13:00 2021 +0200 Merge pull request iiasa#492 from iiasa/hotfix/genno-1.5.1 Adjust for genno 1.6, pyam 1.0 commit 2df44f8 Author: Paul Natsuo Kishimoto <mail@paul.kishimoto.name> Date: Wed Jul 7 14:08:11 2021 +0200 Add iiasa#492 to release notes commit a65f1e3 Author: Paul Natsuo Kishimoto <mail@paul.kishimoto.name> Date: Wed Jul 7 13:03:31 2021 +0200 Adjust use of pyam.IamDataFrame.line_plot() → plot() Former name was removed in pyam 1.0 commit 8d9ac4d Author: Paul Natsuo Kishimoto <mail@paul.kishimoto.name> Date: Wed Jul 7 12:43:33 2021 +0200 Git ignore prof/ directory (pytest-profiling) commit 136add2 Author: Paul Natsuo Kishimoto <mail@paul.kishimoto.name> Date: Tue Jul 6 16:32:01 2021 +0200 Correct a format string in .reporting.computations.plot_cumulative() commit e96819c Author: Paul Natsuo Kishimoto <mail@paul.kishimoto.name> Date: Tue Jul 6 10:53:39 2021 +0200 Adjust test_reporter_from_scenario() for genno 1.5 Use "in rep" not "in rep.graph" for more flexible __contains__. commit 5fc853c Author: FRICKO Oliver <fricko@iiasa.ac.at> Date: Fri Jun 18 08:43:57 2021 +0200 Descriptions corrected commit 6d7f44c Author: FRICKO Oliver <fricko@iiasa.ac.at> Date: Fri Jun 18 08:41:15 2021 +0200 Adjusted documentation of revised formulations commit 11e8bd4 Author: FRICKO Oliver <fricko@iiasa.ac.at> Date: Tue Jun 15 15:27:23 2021 +0200 Updated tutorial syntax commit c15e1ae Author: FRICKO Oliver <fricko@iiasa.ac.at> Date: Tue Jun 15 14:56:42 2021 +0200 Update tutorial descriptions commit 3e8d9b6 Author: FRICKO Oliver <fricko@iiasa.ac.at> Date: Tue Jun 15 14:56:10 2021 +0200 Added tests for soft constraints commit 339f727 Author: FRICKO Oliver <fricko@iiasa.ac.at> Date: Tue Jun 15 11:19:55 2021 +0200 Added tutorial for soft constraints commit 66fdef7 Author: FRICKO Oliver <fricko@iiasa.ac.at> Date: Mon Jun 14 12:46:46 2021 +0200 Updated release notes commit ff37498 Author: Oliver FRICKO <fricko@iiasa.ac.at> Date: Fri Aug 28 16:19:54 2020 +0200 adjusted the year to check for active technology for soft constraints commit ecd00d5 Author: Oliver FRICKO <fricko@iiasa.ac.at> Date: Fri Aug 28 11:16:01 2020 +0200 corrections made to GAMS code commit d86ed88 Author: Volker Krey <krey@iiasa.ac.at> Date: Thu Aug 27 14:37:52 2020 +0200 adjustment of reference new capacity and activity for soft constraints additional growth in soft constraints of new capacity and activity is now based on previous periods new capacity and activity, respectively, as for the regular growth constraints commit 56611ad Author: Paul Natsuo Kishimoto <mail@paul.kishimoto.name> Date: Tue Jun 15 10:46:05 2021 +0200 Also install 3rd party type stubs in 'lint' CI workflow

Adjust test_reporter_from_scenario() for genno 1.5

e96819c

Use "in rep" not "in rep.graph" for more flexible __contains__.

khaeru self-assigned this Jul 6, 2021

khaeru mentioned this pull request Jul 6, 2021

Ensure years_active() uses sorted results #491

Merged

3 tasks

Correct a format string in .reporting.computations.plot_cumulative()

136add2

khaeru mentioned this pull request Jul 7, 2021

Improve performance of Computer.check_keys() and related khaeru/genno#47

Merged

Git ignore prof/ directory (pytest-profiling)

8d9ac4d

khaeru force-pushed the hotfix/genno-1.5.1 branch from cb75e7a to 2502dc5 Compare July 7, 2021 11:05

khaeru changed the title ~~Adjust test_reporter_from_scenario() for genno 1.5~~ Adjust for genno 1.6, pyam 1.0 Jul 7, 2021

Adjust use of pyam.IamDataFrame.line_plot() → plot()

a65f1e3

Former name was removed in pyam 1.0

khaeru added a commit that referenced this pull request Jul 7, 2021

Add #492 to release notes

194031b

khaeru force-pushed the hotfix/genno-1.5.1 branch from 2502dc5 to 194031b Compare July 7, 2021 12:08

Add #492 to release notes

2df44f8

khaeru force-pushed the hotfix/genno-1.5.1 branch from 194031b to 2df44f8 Compare July 7, 2021 12:17

khaeru mentioned this pull request Jul 7, 2021

Add TimeSeries.transact(); clean up code tree iiasa/ixmp#422

Merged

4 tasks

khaeru added ci Continuous integration docs Documentation labels Jul 7, 2021

khaeru merged commit 19eff0f into main Jul 7, 2021

khaeru deleted the hotfix/genno-1.5.1 branch July 7, 2021 13:13

LauWien added this to the 3.4 milestone Jan 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adjust for genno 1.6, pyam 1.0 #492

Adjust for genno 1.6, pyam 1.0 #492

khaeru commented Jul 6, 2021 •

edited

Loading

codecov bot commented Jul 6, 2021 •

edited

Loading

khaeru commented Jul 6, 2021

khaeru commented Jul 7, 2021

khaeru commented Jul 7, 2021

khaeru commented Jul 7, 2021

Adjust for genno 1.6, pyam 1.0 #492

Adjust for genno 1.6, pyam 1.0 #492

Conversation

khaeru commented Jul 6, 2021 • edited Loading

How to review

PR checklist

codecov bot commented Jul 6, 2021 • edited Loading

Codecov Report

khaeru commented Jul 6, 2021

khaeru commented Jul 7, 2021

khaeru commented Jul 7, 2021

khaeru commented Jul 7, 2021

khaeru commented Jul 6, 2021 •

edited

Loading

codecov bot commented Jul 6, 2021 •

edited

Loading