Issue 482 add scheduling test for maximizing self consumption (#532)

Add scheduling test for maximizing self-consumption, and improve time series db queries for fixed tariffs (and other long-term constants). * Better documentation of flexibility model for storage in endpoint; refactor its parameters and handling within the code for readability Signed-off-by: Nicolas Höning <nicolas@seita.nl> * add changelog entry Signed-off-by: Nicolas Höning <nicolas@seita.nl> * make tests work, include updating older API versions, make prefer_charging_sooner part of storage specs & an optional parameter in API v3 Signed-off-by: Nicolas Höning <nicolas@seita.nl> * use storage_specs in CLI command, as well Signed-off-by: Nicolas Höning <nicolas@seita.nl> * remove default resolution of 15M, for now pass in what you want Signed-off-by: Nicolas Höning <nicolas@seita.nl> * various review comments Signed-off-by: Nicolas Höning <nicolas@seita.nl> * black Signed-off-by: Nicolas Höning <nicolas@seita.nl> * fix tests Signed-off-by: Nicolas Höning <nicolas@seita.nl> * always load sensor when checking storage specs Signed-off-by: Nicolas Höning <nicolas@seita.nl> * begin to handle source model and version during scheduling Signed-off-by: Nicolas Höning <nicolas@seita.nl> * we can get multiple sources from our query (in the old setting, when we use name, but also in the new setting, unless we always include the user_id) Signed-off-by: Nicolas Höning <nicolas@seita.nl> * give our two in-built schedulers an official __author__ and __version__ Signed-off-by: Nicolas Höning <nicolas@seita.nl> * review comments Signed-off-by: Nicolas Höning <nicolas@seita.nl> * refactor getting data source for a job to util function; use the actual data source ID for this lookup if possible Signed-off-by: Nicolas Höning <nicolas@seita.nl> * pass sensor to check_storage_specs, as we always have it already Signed-off-by: Nicolas Höning <nicolas@seita.nl> * wrap Scheduler in classes, unify data source handling a bit more Signed-off-by: Nicolas Höning <nicolas@seita.nl> * Support pandas 1.4 (#525) Add a pandas version check in initialize_index. * Use initialize_series util function Signed-off-by: F.N. Claessen <felix@seita.nl> * Update initialize_index for pandas>=1.4 Signed-off-by: F.N. Claessen <felix@seita.nl> * flake8 Signed-off-by: F.N. Claessen <felix@seita.nl> * Use initialize_index or initialize_series in all places where the closed keyword argument was used Signed-off-by: F.N. Claessen <felix@seita.nl> * flake8 Signed-off-by: F.N. Claessen <felix@seita.nl> * mypy: PEP 484 prohibits implicit Optional Signed-off-by: F.N. Claessen <felix@seita.nl> * black after mypy Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: F.N. Claessen <felix@seita.nl> * Stop requiring min/max SoC attributes, which have defaults: - Default min = 0 - Default max = the highest target value, or np.nan if there are no targets, which subsequently maps to infinity in our solver Signed-off-by: F.N. Claessen <felix@seita.nl> * Set up device constraint columns for efficiencies in Charge Point scheduler, too Signed-off-by: F.N. Claessen <felix@seita.nl> * Derive flow constraints for battery scheduling, too (copied from Charge Point scheduler) Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: rename BatteryScheduler to StorageScheduler Signed-off-by: F.N. Claessen <felix@seita.nl> * Warn for deprecation of schedule_battery and schedule_charging_station Signed-off-by: F.N. Claessen <felix@seita.nl> * Use StorageScheduler instead of ChargingStationScheduler Signed-off-by: F.N. Claessen <felix@seita.nl> * Deprecate ChargingStationScheduler Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: move StorageScheduler to dedicated module Signed-off-by: F.N. Claessen <felix@seita.nl> * Update docstring Signed-off-by: F.N. Claessen <felix@seita.nl> * fix test Signed-off-by: F.N. Claessen <felix@seita.nl> * flake8 Signed-off-by: F.N. Claessen <felix@seita.nl> * Lose the v in version strings; prefer versions showing up as 'version: 3' over 'version: v3'. Even though Scheduler versioning does not necessarily need to follow semantic versioning (see discussion here: semver/semver#235), the v is still redundant. Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: rename module Signed-off-by: F.N. Claessen <felix@seita.nl> * Deal with empty SoC targets Signed-off-by: F.N. Claessen <felix@seita.nl> * Stop wrapping DataFrame representations in logging Signed-off-by: F.N. Claessen <felix@seita.nl> * Log warning instead of raising UnknownForecastException, and assume zero power values for missing values Signed-off-by: F.N. Claessen <felix@seita.nl> * Workaround for GH #484 Signed-off-by: F.N. Claessen <felix@seita.nl> * Test maximizing self-consumption Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: a single parameterized test instead of 2 tests with a lot of duplicate code Signed-off-by: F.N. Claessen <felix@seita.nl> * Remove debug statements Signed-off-by: F.N. Claessen <felix@seita.nl> * black Signed-off-by: F.N. Claessen <felix@seita.nl> * Correct mistake while refactoring Signed-off-by: F.N. Claessen <felix@seita.nl> * Check default price sensor in both scenarios Signed-off-by: F.N. Claessen <felix@seita.nl> * Expand explanation of optimization context Signed-off-by: F.N. Claessen <felix@seita.nl> * Reorder assertions and add more comments Signed-off-by: F.N. Claessen <felix@seita.nl> * Upgrade timely-beliefs to partly resolve workaround Signed-off-by: F.N. Claessen <felix@seita.nl> * changelog entry Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: Nicolas Höning <nicolas@seita.nl> Signed-off-by: F.N. Claessen <felix@seita.nl> Co-authored-by: Nicolas Höning <nicolas@seita.nl>
FlexMeasures · Nov 21, 2022 · 3c95759 · 3c95759
1 parent 7715219
commit 3c95759
Show file tree

Hide file tree

Showing 6 changed files with 122 additions and 18 deletions.
diff --git a/documentation/changelog.rst b/documentation/changelog.rst
@@ -32,6 +32,7 @@ Infrastructure / Support
 * Plugins can save BeliefsSeries, too, instead of just BeliefsDataFrames [see `PR #523 <http://www.github.com/FlexMeasures/flexmeasures/pull/523>`_]
 * Improve documentation and code w.r.t. storage flexibility modelling ― prepare for handling other schedulers & merge battery and car charging schedulers [see `PR #511 <http://www.github.com/FlexMeasures/flexmeasures/pull/511>`_]
 * Revised strategy for removing unchanged beliefs when saving data: retain the oldest measurement (ex-post belief), too [see `PR #518 <http://www.github.com/FlexMeasures/flexmeasures/pull/518>`_]
+* Scheduling test for maximizing self-consumption, and improved time series db queries for fixed tariffs (and other long-term constants) [see `PR #532 <http://www.github.com/FlexMeasures/flexmeasures/pull/532>`_]
 
 
 v0.11.3 | November 2, 2022

diff --git a/flexmeasures/data/models/planning/tests/conftest.py b/flexmeasures/data/models/planning/tests/conftest.py
@@ -3,6 +3,7 @@
 from datetime import timedelta
 import pytest
 
+from timely_beliefs.sensors.func_store.knowledge_horizons import at_date
 import pandas as pd
 
 from flexmeasures.data.models.generic_assets import GenericAsset, GenericAssetType
@@ -18,6 +19,61 @@ def setup_planning_test_data(db, add_market_prices, add_charging_station_assets)
     print("Setting up data for planning tests on %s" % db.engine)
 
 
+@pytest.fixture(scope="module")
+def create_test_tariffs(db, setup_accounts, setup_sources) -> dict[str, Sensor]:
+    """Create a fixed consumption tariff and a fixed feed-in tariff that is lower."""
+
+    market_type = GenericAssetType(
+        name="tariff market",
+    )
+    db.session.add(market_type)
+    contract = GenericAsset(
+        name="supply contract",
+        generic_asset_type=market_type,
+        owner=setup_accounts["Supplier"],
+    )
+    db.session.add(contract)
+    consumption_price_sensor = Sensor(
+        name="fixed consumption tariff",
+        generic_asset=contract,
+        event_resolution=timedelta(hours=24 * 365),
+        unit="EUR/MWh",
+        knowledge_horizon=(at_date, {"knowledge_time": "2014-11-01T00:00+01:00"}),
+    )
+    db.session.add(consumption_price_sensor)
+    production_price_sensor = Sensor(
+        name="fixed feed-in tariff",
+        generic_asset=contract,
+        event_resolution=timedelta(hours=24 * 365),
+        unit="EUR/MWh",
+        knowledge_horizon=(at_date, {"knowledge_time": "2014-11-01T00:00+01:00"}),
+    )
+    db.session.add(production_price_sensor)
+
+    # Add prices
+    consumption_price = TimedBelief(
+        event_start="2015-01-01T00:00+01:00",
+        belief_time="2014-11-01T00:00+01:00",  # publication date
+        event_value=300 * 1.21,
+        source=setup_sources["Seita"],
+        sensor=consumption_price_sensor,
+    )
+    db.session.add(consumption_price)
+    production_price = TimedBelief(
+        event_start="2015-01-01T00:00+01:00",
+        belief_time="2014-11-01T00:00+01:00",  # publication date
+        event_value=300,
+        source=setup_sources["Seita"],
+        sensor=production_price_sensor,
+    )
+    db.session.add(production_price)
+    db.session.flush()  # make sure that prices are assigned to price sensors
+    return {
+        "consumption_price_sensor": consumption_price_sensor,
+        "production_price_sensor": production_price_sensor,
+    }
+
+
 @pytest.fixture(scope="module")
 def building(db, setup_accounts, setup_markets) -> GenericAsset:
     """

diff --git a/flexmeasures/data/models/planning/tests/test_solver.py b/flexmeasures/data/models/planning/tests/test_solver.py
@@ -272,25 +272,52 @@ def test_fallback_to_unsolvable_problem(target_soc, charging_station_name):
     )
 
 
+@pytest.mark.parametrize(
+    "market_scenario",
+    [
+        "dynamic contract",
+        "fixed contract",
+    ],
+)
 def test_building_solver_day_2(
     db,
     add_battery_assets,
     add_market_prices,
+    create_test_tariffs,
     add_inflexible_device_forecasts,
     inflexible_devices,
     flexible_devices,
+    market_scenario: str,
 ):
-    """Check battery scheduling results within the context of a building with PV, for day 2,
-    which is set up with 8 expensive, then 8 cheap, then again 8 expensive hours.
-    We expect the scheduler to:
-    - completely discharge within the first 8 hours
-    - completely charge within the next 8 hours
-    - completely discharge within the last 8 hours
+    """Check battery scheduling results within the context of a building with PV, for day 2, against the following market scenarios:
+    1) a dynamic tariff with equal consumption and feed-in tariffs, that is set up with 8 expensive, then 8 cheap, then again 8 expensive hours.
+    2) a fixed consumption tariff and a fixed feed-in tariff that is lower, which incentives to maximize self-consumption of PV power into the battery.
+    In the test data:
+    - Hours with net production coincide with low dynamic market prices.
+    - Hours with net consumption coincide with high dynamic market prices.
+    So when the prices are low (in scenario 1), we have net production, and when they are high, net consumption.
+    That means we have first net consumption, then net production, and then net consumption again.
+    In either scenario, we expect the scheduler to:
+    - completely discharge within the first 8 hours (either due to 1) high prices, or 2) net consumption)
+    - completely charge within the next 8 hours (either due to 1) low prices, or 2) net production)
+    - completely discharge within the last 8 hours (either due to 1) high prices, or 2) net consumption)
     """
-    epex_da = Sensor.query.filter(Sensor.name == "epex_da").one_or_none()
     battery = flexible_devices["battery power sensor"]
     building = battery.generic_asset
-    assert battery.get_attribute("market_id") == epex_da.id
+    default_consumption_price_sensor = Sensor.query.filter(
+        Sensor.name == "epex_da"
+    ).one_or_none()
+    assert battery.get_attribute("market_id") == default_consumption_price_sensor.id
+    if market_scenario == "dynamic contract":
+        consumption_price_sensor = default_consumption_price_sensor
+        production_price_sensor = consumption_price_sensor
+    elif market_scenario == "fixed contract":
+        consumption_price_sensor = create_test_tariffs["consumption_price_sensor"]
+        production_price_sensor = create_test_tariffs["production_price_sensor"]
+    else:
+        raise NotImplementedError(
+            f"Missing test case for market conditions '{market_scenario}'"
+        )
     tz = pytz.timezone("Europe/Amsterdam")
     start = tz.localize(datetime(2015, 1, 2))
     end = tz.localize(datetime(2015, 1, 3))
@@ -315,6 +342,8 @@ def test_building_solver_day_2(
         end,
         resolution,
         storage_specs=storage_specs,
+        consumption_price_sensor=consumption_price_sensor,
+        production_price_sensor=production_price_sensor,
         inflexible_device_sensors=inflexible_devices.values(),
     )
     soc_schedule = integrate_time_series(schedule, soc_at_start, decimal_precision=6)
@@ -353,14 +382,28 @@ def test_building_solver_day_2(
         assert soc >= max(soc_min, battery.get_attribute("min_soc_in_mwh"))
         assert soc <= battery.get_attribute("max_soc_in_mwh")
 
-    # Check whether the resulting soc schedule follows our expectations for 8 expensive, 8 cheap and 8 expensive hours
-    assert soc_schedule.iloc[-1] == max(
-        soc_min, battery.get_attribute("min_soc_in_mwh")
-    )  # Battery sold out at the end of its planning horizon
+    # Check whether the resulting soc schedule follows our expectations for.
+    # To recap, in scenario 1 and 2, the schedule should mainly be influenced by:
+    # 1) 8 expensive, 8 cheap and 8 expensive hours
+    # 2) 8 net-consumption, 8 net-production and 8 net-consumption hours
 
+    # Result after 8 hours
+    # 1) Sell what you begin with
+    # 2) The battery discharged as far as it could during the first 8 net-consumption hours
     assert soc_schedule.loc[start + timedelta(hours=8)] == max(
         soc_min, battery.get_attribute("min_soc_in_mwh")
-    )  # Sell what you begin with
+    )
+
+    # Result after second 8 hour-interval
+    # 1) Buy what you can to sell later, when prices will be high again
+    # 2) The battery charged with PV power as far as it could during the middle 8 net-production hours
     assert soc_schedule.loc[start + timedelta(hours=16)] == min(
         soc_max, battery.get_attribute("max_soc_in_mwh")
-    )  # Buy what you can to sell later
+    )
+
+    # Result at end of day
+    # 1) The battery sold out at the end of its planning horizon
+    # 2) The battery discharged as far as it could during the last 8 net-consumption hours
+    assert soc_schedule.iloc[-1] == max(
+        soc_min, battery.get_attribute("min_soc_in_mwh")
+    )
diff --git a/flexmeasures/data/models/time_series.py b/flexmeasures/data/models/time_series.py
@@ -652,8 +652,9 @@ def search(
             bdf = cls.search_session(
                 session=db.session,
                 sensor=sensor,
-                event_starts_after=event_starts_after,
-                event_ends_before=event_ends_before,
+                # Workaround (1st half) for https://github.com/FlexMeasures/flexmeasures/issues/484
+                event_ends_after=event_starts_after,
+                event_starts_before=event_ends_before,
                 beliefs_after=beliefs_after,
                 beliefs_before=beliefs_before,
                 horizons_at_least=horizons_at_least,
@@ -690,6 +691,9 @@ def search(
                 bdf = bdf.resample_events(
                     resolution, keep_only_most_recent_belief=most_recent_beliefs_only
                 )
+                # Workaround (2nd half) for https://github.com/FlexMeasures/flexmeasures/issues/484
+                bdf = bdf[bdf.event_starts >= event_starts_after]
+                bdf = bdf[bdf.event_ends <= event_ends_before]
             bdf_dict[bdf.sensor.name] = bdf
 
         if sum_multiple:

diff --git a/requirements/app.in b/requirements/app.in
@@ -28,7 +28,7 @@ tldextract
 pyomo>=5.6
 tabulate
 timetomodel>=0.7.1
-timely-beliefs[forecast]>=1.13
+timely-beliefs[forecast]>=1.14
 python-dotenv
 # a backport, not needed in Python3.8
 importlib_metadata

diff --git a/requirements/app.txt b/requirements/app.txt
@@ -314,7 +314,7 @@ tabulate==0.8.10
     # via -r requirements/app.in
 threadpoolctl==3.1.0
     # via scikit-learn
-timely-beliefs[forecast]==1.13.0
+timely-beliefs[forecast]==1.14.0
     # via -r requirements/app.in
 timetomodel==0.7.1
     # via -r requirements/app.in