Skip to content

Commit

Permalink
Merge pull request #68 from pastas/dev
Browse files Browse the repository at this point in the history
Minor release for bug fix
  • Loading branch information
dbrakenhoff authored Jun 23, 2022
2 parents 3e6dd14 + 1165dd0 commit 24a536b
Show file tree
Hide file tree
Showing 8 changed files with 73 additions and 60 deletions.
22 changes: 14 additions & 8 deletions examples/notebooks/ex04_pastastore_yaml_interface.ipynb

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions pastastore/datasets.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
import pastastore as pst
from pastastore.base import BaseConnector


def example_pastastore(conn="DictConnector"):
"""Example dataset loaded into PastaStore.
Expand Down
5 changes: 3 additions & 2 deletions pastastore/plotting.py
Original file line number Diff line number Diff line change
Expand Up @@ -882,8 +882,9 @@ def stresslinks(self, kinds=None, model_names=None, color_lines=False,

stused = np.append(stused, s)

if labels:
self.add_labels(oseries.loc[struct['oseries'].unique()], ax, adjust=adjust)
if labels:
self.add_labels(
oseries.loc[struct['oseries'].unique()], ax, adjust=adjust)
self.add_labels(stresses.loc[np.unique(stused)], ax, adjust=adjust)

ax.scatter([x[1][0] for x in segments],
Expand Down
10 changes: 6 additions & 4 deletions pastastore/store.py
Original file line number Diff line number Diff line change
Expand Up @@ -173,12 +173,14 @@ def get_nearest_oseries(self, names: Optional[Union[list, str]] = None,

data = pd.DataFrame(columns=np.arange(n))

for series in distances.index:
others = distances.loc[series].dropna(
for series_name in distances.index:
others = distances.loc[series_name].dropna(
).sort_values().index.tolist()
# remove self
others.remove(series)
series = pd.Series(others[:n], name=series)
others.remove(series_name)
series = pd.DataFrame(index=[series_name],
columns=data.columns,
data=[others[:n]])
data = pd.concat([data, series], axis=0)
return data

Expand Down
2 changes: 1 addition & 1 deletion pastastore/version.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = "0.8.0"
__version__ = "0.8.1"
85 changes: 42 additions & 43 deletions readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,60 +10,59 @@ This module contains a tool to manage
[Pastas](https://pastas.readthedocs.io/en/latest/) timeseries and models in a
database.

Storing timeseries and models in a database gives the user
a simple way to manage Pastas projects with the added bonus of allowing the user
to pick upwhere they left off, without having to (re)load everything into memory.
Storing timeseries and models in a database gives the user a simple way to
manage Pastas projects with the added bonus of allowing the user to pick
upwhere they left off, without having to (re)load everything into memory.

The connection to database/disk/memory is managed by a connector object.
Currently, four connectors are included. The first implementation is an
in-memory connector. The other three store data on disk or in a database.
The PasConnector implementation writes human-readable JSON files to disk.
The ArcticConnector and PystoreConnector implementations are designed to have
fast read/write operations, while also compressing the stored data.
Currently, four connectors are included. The first implementation is an
in-memory connector. The other three store data on disk or in a database. The
PasConnector implementation writes human-readable JSON files to disk. The
ArcticConnector and PystoreConnector implementations are designed to have fast
read/write operations, while also compressing the stored data.

- In-memory: uses dictionaries to hold timeseries and pastas Models in-memory.
Does not require any additional packages to use.
Does not require any additional packages to use.

- Pastas: uses Pastas write and read methods to store data as JSON files on
disk. Does not require any additional packages to use.
disk. Does not require any additional packages to use.

- [Arctic](https://arctic.readthedocs.io/en/latest/) is a timeseries/dataframe
database that sits atop [MongoDB](https://www.mongodb.com). Arctic supports
pandas.DataFrames.
database that sits atop [MongoDB](https://www.mongodb.com). Arctic supports
pandas.DataFrames.

- [PyStore](https://github.com/ranaroussi/pystore) is a datastore (inspired
by Arctic) created for storing pandas dataframes (especially timeseries) on
disk. Data is stored using fastparquet and compressed with Snappy.
- [PyStore](https://github.com/ranaroussi/pystore) is a datastore (inspired by
Arctic) created for storing pandas dataframes (especially timeseries) on disk.
Data is stored using fastparquet and compressed with Snappy.

## Installation

Install the module by typing `pip install pastastore`.

For installing in development mode, clone the repository and install by
typing `pip install -e .` from the module root directory.
For installing in development mode, clone the repository and install by typing
`pip install -e .` from the module root directory.

For plotting backgroundmaps, the `contextily` and `pyproj` packages are required.
For a full install, including an optional dependency for plotting and labeling data
on maps, use: `pip install pastastore[full]` or `pip install .[full]` when on MacOS
or Linux. Windows users are asked to install `rasterio` themselves since it often
cannot be installed using `pip`. `rasterio` is a dependency of `contextily`.
Windows can install `pastastore` with the optional labeling package adjustText
using `pip install pastastore[adjusttext]` or `.[adjusttext]`

There are external dependencies when using the `pystore` or `arctic` connectors.
To install these dependencies read (see [Connector Dependencies section](#dependencies))!
since these are _not_ automatically installed.
For plotting backgroundmaps, the `contextily` and `pyproj` packages are
required. For a full install, including an optional dependency for plotting and
labeling data on maps, use: `pip install pastastore[full]` or `pip install
.[full]` when on MacOS or Linux. Windows users are asked to install `rasterio`
themselves since it often cannot be installed using `pip`. `rasterio` is a
dependency of `contextily`. Windows can install `pastastore` with the optional
labeling package adjustText using `pip install pastastore[adjusttext]` or
`.[adjusttext]`

There are external dependencies when using the `pystore` or `arctic`
connectors. To install these dependencies read (see [Connector Dependencies
section](#dependencies))! since these are _not_ automatically installed.

## Usage

The following snippets show typical usage. The general idea is to first define
the connector object. The next step is to pass that connector to
`PastaStore`.
the connector object. The next step is to pass that connector to `PastaStore`.

### Using in-memory dictionaries

This works out of the box after installing with `pip` without installing any
This works out of the box after installing with `pip` without installing any
additional Python dependencies or external software.

```python
Expand All @@ -78,8 +77,8 @@ store = pst.PastaStore("my_project", conn)

### Using Pastas read/load methods

Store data on disk as JSON files (with .pas extension) using Pastas read and
load methods. This works out of the box after installing with `pip` without
Store data on disk as JSON files (with .pas extension) using Pastas read and
load methods. This works out of the box after installing with `pip` without
installing any additional Python dependencies or external software.

```python
Expand Down Expand Up @@ -111,7 +110,7 @@ store = pst.PastaStore("my_project", conn)

### Using Pystore

Store data on disk as parquet files using compression. Only works if
Store data on disk as parquet files using compression. Only works if
`python-snappy` and `pystore` are installed.

```python
Expand All @@ -125,9 +124,9 @@ conn = pst.PystoreConnector("my_connector", path)
store = pst.PastaStore("my_project", conn)
```

The database read/write/delete methods can be accessed through the reference
to the connector object. For easy access, the
most common methods are registered to the `store` object. E.g.
The database read/write/delete methods can be accessed through the reference to
the connector object. For easy access, the most common methods are registered
to the `store` object. E.g.

```python
series = store.conn.get_oseries("my_oseries")
Expand All @@ -153,12 +152,12 @@ If using `ArcticConnector`:
([Windows](https://fastdl.mongodb.org/win32/mongodb-win32-x86_64-2012plus-4.2.1-signed.msi),
[MacOS](https://fastdl.mongodb.org/osx/mongodb-macos-x86_64-4.2.1.tgz)).

- OR, if you wish to use Docker for running MongoDB see the installation instructions [here](https://github.com/pastas/pastastore/tree/master/dockerfiles#running-mongodb-from-docker).
- OR, if you wish to use Docker for running MongoDB see the installation
instructions [here](https://github.com/pastas/pastastore/tree/master/dockerfiles#running-mongodb-from-docker).

If using `PystoreConnector`:

- PyStore uses [Snappy](http://google.github.io/snappy/), a fast and
efficient compression/decompression library from Google. You'll need to
install Snappy on your system before installing PyStore. See links for
installation instructions here:
<https://github.com/ranaroussi/pystore#dependencies>
- PyStore uses [Snappy](http://google.github.io/snappy/), a fast and efficient
compression/decompression library from Google. You'll need to install Snappy on
your system before installing PyStore. See links for installation instructions
here: <https://github.com/ranaroussi/pystore#dependencies>
6 changes: 6 additions & 0 deletions tests/test_003_pastastore.py
Original file line number Diff line number Diff line change
Expand Up @@ -272,3 +272,9 @@ def test_to_from_zip(pstore):
finally:
os.remove(zipname)
return store


def test_example_pastastore():
from pastastore.datasets import example_pastastore
pstore = example_pastastore()
return
2 changes: 0 additions & 2 deletions tests/test_005_maps_plots.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
import warnings

import matplotlib.pyplot as plt
import pytest
from pytest_dependency import depends
Expand Down

0 comments on commit 24a536b

Please sign in to comment.