Skip to content

Commit

Permalink
Refactor IIASA-connection-API and rework all related tests (#413)
Browse files Browse the repository at this point in the history
  • Loading branch information
@danielhuppmann
danielhuppmann authored Jul 8, 2020
1 parent bcf8293 commit 7f1a0e8
Show file tree
Hide file tree
Showing 6 changed files with 391 additions and 285 deletions.
11 changes: 11 additions & 0 deletions RELEASE_NOTES.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,16 @@

## API changes

PR [#413](https://github.com/IAMconsortium/pyam/pull/413) changed the
return type of `pyam.read_iiasa()` and `pyam.iiasa.Connection.query()`
to an `IamDataFrame` (instead of a `pandas.DataFrame`)
and loads meta-indicators by default.

Also, the following functions were deprecated for package consistency:
- `index()` replaces `scenario_list()` for an overview of all scenarios
- `meta_columns` (attribute) replaces `available_metadata()`
- `meta()` replaces `metadata()`

PR [#402](https://github.com/IAMconsortium/pyam/pull/402) changed the default
behaviour of `as_pandas()` to include all columns of `meta` in the returned
dataframe, or only merge columns given by the renamed argument `meta_cols`.
Expand All @@ -10,6 +20,7 @@ a utility function `pyam.plotting.mpl_args_to_meta_cols()`.

## Individual Updates

- [#413](https://github.com/IAMconsortium/pyam/pull/413) Refactor IIASA-connection-API and rework all related tests.
- [#412](https://github.com/IAMconsortium/pyam/pull/412) Add building the docs to GitHub Actions CI.
- [#411](https://github.com/IAMconsortium/pyam/pull/411) Add feature to pass an explicit weight dataframe to `downscale_region()`.
- [#410](https://github.com/IAMconsortium/pyam/pull/410) Activate tutorial tests on GitHub Actions CI (py3.8).
Expand Down
57 changes: 38 additions & 19 deletions doc/source/tutorials/iiasa_dbs.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"High-profile use cases include the [IAMC 1.5°C Scenario Explorer hosted by IIASA](https://data.ene.iiasa.ac.at/iamc-1.5c-explorer) supporting the *IPCC Special Report on Global Warming of 1.5°C* (SR15) and the Horizon 2020 project [CD-LINKS](https://data.ene.iiasa.ac.at/cd-links).\n",
"\n",
"IIASA's [modeling platform infrastructure](http://software.ene.iiasa.ac.at/ixmp-server) and the Scenario Explorer UI is not only a great resource on its own, but it also allows the underlying datasets to be directly queried.\n",
"**pyam** takes advantage of this ability to allow you to easily pull data and work with it."
"**pyam** takes advantage of this ability to allow you to easily pull data and work with it in your Python data processing and analysis workflow."
]
},
{
Expand All @@ -26,7 +26,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Accessing an explorer is done via a `Connection` object.\n",
"## Connecting to a data resource (aka the database API of a Scenario Explorer instance)\n",
"\n",
"Accessing a data resource is done via a **Connection** object.\n",
"By default, your can connect to all public scenario explorers instances. "
]
},
Expand All @@ -44,21 +46,26 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"If you have credentials to connect to a non-public or restricted database,\n",
"you can set this (in a separate Python console) using the following command:\n",
"If you have credentials to connect to a non-public or restricted Scenario Explorer instance,\n",
"you can store this information by running the following command in a separate Python console:\n",
"\n",
"```\n",
"import pyam\n",
"pyam.iiasa.set_config(<username>, <password>)\n",
"```\n",
"When initializing a new `Connection` instance, **pyam** will automatically search for the configuration in a known location."
"When initializing a new **Connection** instance, **pyam** will automatically search for the configuration in a known location."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this example, we will be pulling data from the Special Report on 1.5C explorer. This can be done either via the constructor:\n",
"In this example, we will be retrieving data from the *IAMC 1.5°C Scenario Explorer hosted by IIASA*\n",
"([link](https://data.ene.iiasa.ac.at/iamc-1.5c-explorer)),\n",
"which provides the quantiative scenario ensemble underpinning\n",
"the *IPCC Special Report on Global Warming of 1.5C* (SR15).\n",
"\n",
"This can be done either via the constructor:\n",
"\n",
"```\n",
"pyam.iiasa.Connection('iamc15')\n",
Expand All @@ -76,7 +83,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"We also provide some convenience functions to shorten the amount of code you have to write. Under the hood, `read_iiasa()` is just opening an connection to a database and making a query on that data.\n",
"We also provide some convenience functions to shorten the amount of code you have to write. Under the hood, `read_iiasa()` is just opening a connection to a database API and sends a query to the resource.\n",
"\n",
"In this tutorial, we will query specific subsets of data in a manner similar to `pyam.IamDataFrame.filter()`."
]
},
Expand All @@ -99,7 +107,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we pulled out all times series data for model(s) that start with 'MESSAGEix' that are in the 'World' region and associated with the two named variables. We also added the \"category\" metadata, which tells us the climate impact categorisation of each scenario as assessed in the IPCC SR15.\n",
"Here we pulled out all times series data for model(s) that start with 'MESSAGEix' that are in the 'World' region and associated with the two named variables. We also added the meta column \"category\", which tells us the climate impact categorisation of each scenario as assessed in the IPCC SR15.\n",
"\n",
"Let's plot CO2 emissions."
]
Expand Down Expand Up @@ -143,7 +151,7 @@
"source": [
"## Exploring the data resource\n",
"\n",
"If you're interested in what data is actually in the data source, you can use **pyam.iiasa.Connection** to do so."
"If you're interested in what data is available in the data source, you can use **pyam.iiasa.Connection** to do so."
]
},
{
Expand All @@ -159,7 +167,10 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"The `conn` object has a number of useful functions for listing what's in the dataset. A few of them are shown below."
"The **Connection** object has a number of useful functions for listing what's available in the data resource.\n",
"These functions follow the conventions of the **IamDataFrame** class (where possible).\n",
"\n",
"A few of them are shown below."
]
},
{
Expand Down Expand Up @@ -202,8 +213,11 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"A number of different kinds of indicators are available for model/scenario combinations.\n",
"We queried the \"category\" metadata in the above example, but there are many more. You can see them with"
"A number of different categorization and quantitative indicators are available for model/scenario combinations.\n",
"These are usually called `meta` indicators in **pyam**.\n",
"\n",
"We queried the meta-indicator \"category\" in the above example, but there are many more.\n",
"You can get a list with the following command:"
]
},
{
Expand All @@ -212,14 +226,14 @@
"metadata": {},
"outputs": [],
"source": [
"conn.available_metadata().head()"
"conn.meta_columns.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can directly query the **Connection**, which will give you a [pandas.DataFrame](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html)."
"You can directly query the **Connection**, which will return a **pyam.IamDataFrame**..."
]
},
{
Expand All @@ -232,15 +246,14 @@
" model='MESSAGEix*', \n",
" variable=['Emissions|CO2', 'Primary Energy|Coal'], \n",
" region='World'\n",
")\n",
"df.head()"
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And you can easily turn this into a **pyam.IamDataFrame** to continue your analysis."
"...so that you can directly continue with your analysis and visualization workflow using **pyam**!"
]
},
{
Expand All @@ -249,12 +262,18 @@
"metadata": {},
"outputs": [],
"source": [
"df = pyam.IamDataFrame(df)\n",
"ax = df.filter(variable='Primary Energy|Coal').line_plot(\n",
" color='scenario', \n",
" legend=dict(loc='center left', bbox_to_anchor=(1.0, 0.5))\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
Expand All @@ -273,7 +292,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.4"
"version": "3.7.7"
}
},
"nbformat": 4,
Expand Down
Loading

0 comments on commit 7f1a0e8

Please sign in to comment.