Improve performance of pyam.concat()

[danielhuppmann]

Please confirm that this PR has done the following:

• Tests Added
• Documentation Added
• Name of contributors Added to AUTHORS.rst
• Description in RELEASE_NOTES.md Added

Description of PR

This PR refactors the implementation of pyam.concat() to use pd.concat() on the timeseries data rather than iteratively calling IamDataFrame.append().

closes #500

[danielhuppmann]

@pjuergens, it would be great if you could try out this implementation for your use cases and report any unexpected issues. Also, it would be interesting to hear about the performance improvements in a real-world application.

[codecov]

Codecov Report

Merging #510 (7c1b183) into main (ebceba1) will increase coverage by 0.0%.
The diff coverage is 100.0%.

@@          Coverage Diff          @@
##            main    #510   +/-   ##
=====================================
  Coverage   93.1%   93.1%           
=====================================
  Files         43      43           
  Lines       4907    4926   +19     
=====================================
+ Hits        4569    4588   +19     
  Misses       338     338           

Impacted Files	Coverage Δ
pyam/core.py	92.5% <100.0%> (+<0.1%)	⬆️
tests/conftest.py	100.0% <100.0%> (ø)
tests/test_core.py	100.0% <100.0%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ebceba1...7c1b183. Read the comment docs.

[pjuergens]

@danielhuppmann thanks for working on this issue. I'll try to test the implementation today.

Concerning performance-improvement in real-world-application: We have a code in which we aggregate a lot of hourly-resolved timeseries. The bottleneck there was the pyam.append-function. Switching from append to concat on major parts of the code with my original implementation I could reduce the calculation time from 24 to 15 minutes. I'll have a deeper look at it again today when I test the new implementation.

[pjuergens]

my script ran without any issues, thanks again!

[coroa]

LGTM. I can't find any faults in that implementation. Thanks for the nice speed-up!

The next low-hanging fruit on the speed-up quest is to add a (non-copying!?) fast-path to __init__ for a pd.Series in perfect condition, ie an index with the right names that is unique and maybe other conditions that one would have to extract from format_data to avoid having to reindex.

[danielhuppmann]

At some point, it will be so fast that you don't even have time for coffee while your scripts are running 😜...