Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tests for time conversions in tools package #2341

Open
wants to merge 43 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
9347f12
Add tests for tools.localize_to_utc
markcampanelli Dec 23, 2024
0638773
Add tests for datetime_to_djd and djd_to_datetime
markcampanelli Dec 23, 2024
c1df9a7
Update what's new
markcampanelli Dec 23, 2024
77c0f81
Appease the linter
markcampanelli Dec 23, 2024
6704d06
Fix pandas equality tests for Python 3.9
markcampanelli Dec 23, 2024
dbb1805
Fix pandas equality tests for Python 3.9 more
markcampanelli Dec 23, 2024
6750709
Fix pandas equality tests for Python 3.9 more more
markcampanelli Dec 23, 2024
1144106
Bump miniimum pandas to fix bad test failure
markcampanelli Dec 23, 2024
14715ed
Try alternative pandas test fix
markcampanelli Dec 23, 2024
545c196
Revert change in minimum pandas version
markcampanelli Dec 23, 2024
271fd97
Fix test
markcampanelli Dec 23, 2024
01263c2
Type Location's tz and pytz attributes as advertised
markcampanelli Dec 23, 2024
60a5d94
Add timezone type checks to Location init test
markcampanelli Dec 23, 2024
9ab2ecf
Don't parameterize repetitive tests
markcampanelli Dec 24, 2024
ddef8d1
Update whatsnew for Location bugfix
markcampanelli Dec 24, 2024
4f17f49
Update docstring
markcampanelli Dec 24, 2024
a3c3e03
Improve whatsnew formatting
markcampanelli Dec 24, 2024
5f59417
Support non-fractional int and float and pytz and zoneinfo time zones
markcampanelli Jan 9, 2025
c84801f
Appease the linter
markcampanelli Jan 9, 2025
195efbc
Use zoneinfo as single source of truth and tz as interface point
markcampanelli Jan 10, 2025
1a5efed
Add zoneinfo asserts in tests
markcampanelli Jan 10, 2025
e5af9ae
Try to fix asv ci
markcampanelli Jan 10, 2025
67e9844
See if newer asv works with newer conda
markcampanelli Jan 10, 2025
e35eb42
Remove comments no longer needed
markcampanelli Jan 10, 2025
a1a0261
Remove addition of zoneinfo attribute
markcampanelli Jan 10, 2025
8373ac4
Revise whatsnew bugfix
markcampanelli Jan 10, 2025
eee6f51
Revise whatsnew bugfix more
markcampanelli Jan 10, 2025
9662c1f
Spell my name correctly
markcampanelli Jan 10, 2025
32284ba
The linter strikes back again
markcampanelli Jan 10, 2025
01e4cfc
Merge branch 'main' into add_tools_tests
markcampanelli Jan 27, 2025
c09a328
Fix whatsnew after main merge
markcampanelli Jan 27, 2025
4ef4b69
Address Cliff's comment
markcampanelli Jan 28, 2025
7490792
Adjust Location documentation
markcampanelli Jan 28, 2025
a5f7646
Fix indent
markcampanelli Jan 28, 2025
1382e30
More docstring tweaks
markcampanelli Jan 28, 2025
059e35f
Try to fix bad parens
markcampanelli Jan 28, 2025
f9f07d7
Rearrange docstring
markcampanelli Jan 28, 2025
75db2aa
Appease the linter
markcampanelli Jan 28, 2025
1164c96
Document pytz attribute as read only
markcampanelli Jan 28, 2025
5f6ad14
Consistent read only
markcampanelli Jan 28, 2025
f691bb6
Update pvlib/location.py per review comment
markcampanelli Jan 28, 2025
7cfb170
Add breaking change to whatsnew and fix linting
markcampanelli Jan 28, 2025
ef5c60f
Clarify breaking change in whatsnew
markcampanelli Feb 5, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions docs/sphinx/source/whatsnew/v0.11.3.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,21 @@ Enhancements
~~~~~~~~~~~~


Bug Fixes
~~~~~~~~~
* Ensure proper tz and pytz types in pvlib.location.Location.
(:issue:`2340`, :pull:`2341`)


Documentation
~~~~~~~~~~~~~


Testing
~~~~~~~
* Add tests for timezone types in pvlib.location.Location.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Add tests for timezone types in pvlib.location.Location.
* Add tests for all input types for the pvlib.location.Location.tz attribute.

(:issue:`2340`, :pull:`2341`)
* Add tests for time conversions in pvlib.tools. (:issue:`2340`, :pull:`2341`)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Add tests for time conversions in pvlib.tools. (:issue:`2340`, :pull:`2341`)
* Add tests for time conversion functions in pvlib.tools. (:issue:`2340`, :pull:`2341`)



Requirements
Expand All @@ -26,5 +35,4 @@ Requirements

Contributors
~~~~~~~~~~~~


* Mark Campanellli (:ghuser:`markcampanelli`)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Mark Campanellli (:ghuser:`markcampanelli`)
* Mark Campanelli (:ghuser:`markcampanelli`)

11 changes: 7 additions & 4 deletions pvlib/location.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,11 +72,12 @@ def __init__(self, latitude, longitude, tz='UTC', altitude=None,
self.tz = 'UTC'
self.pytz = pytz.UTC
elif isinstance(tz, datetime.tzinfo):
# This includes pytz timezones.
self.tz = tz.zone
self.pytz = tz
self.pytz = pytz.timezone(tz.zone)
elif isinstance(tz, (int, float)):
self.tz = tz
self.pytz = pytz.FixedOffset(tz*60)
self.tz = f"Etc/GMT{int(-tz):+d}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm inclined to ask to revert this change. The docstring says tz can be a float. There are places that have a non-integer offset from UTC, e.g., the Marquesas are UTC−09:30UTC−09:30. localize_to_utc is changed in this PR to examine Location.pytz rather than Location.tz, so the error we discussed won't occur.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a point in the Location docstring that says this:

Location objects have two timezone attributes:
    * ``tz`` is a IANA timezone string.
    * ``pytz`` is a pytz timezone object.

While I appreciate my changes may break some things, I think this existing docstring suggests a much more sane approach. (Indeed, in the sequel using zoneinfo, I propose using one, and only one, internal representation of the timezone, instead of two, potentially with helper functions s to update this internal representation using int/float offsets, etc..)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not saying revert from storing a string, but revert the int portion. If tz is truncated to an integer, how are timezones such as UTC+05:30 represented?

I'm open to doing away with accepting int or float for tz.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cwhanse Sorry I misunderstood your meaning. I was aware that some locations are on half-hour offsets and such, but what is unclear to me is if/how that is represented in pytz (or zoneinfo for that matter). I do not see any fractional offsets in zoneinfo.available_timezones() and this also does not work for me: pytz.timezone("Etc/GMT+5.5").

I must be missing something here. Does something non-standard have to be done to support fractional offsets?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to doing away with accepting int or float for tz.

Hi again @cwhanse. It looks like the standard "non-integral-hour" offsets are only supported in the IANA standard through place names such as "Asia/Kathmandu" and "Asia/Yangon", but not through Etc/GMT+X.Y style offsets.

Given that, I think the most trouble-free thing to do is to remove the int and float options and be sure to discuss this issue in the documentation tutorial. (Also, the only supported tz strings would be the IANA ones.) Sounds like you are amenable to this.

Keeping both fields tz (only as a string) and pytz (as a pytz timezone type), then I think I should add setters that change one when the other is changed after the Location object has already been already initialized.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the most trouble-free thing to do is to remove the int and float options

I'm in favor of this change. @pvlib/pvlib-maintainer @pvlib/pvlib-triage your opinions?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No opinion.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TMY3 files represent timezone in numeric form, so this change may complicate some workflows. Outside of that, I never use this attribute (and don't know why I would), so I have no opinion either.

Remember that we usually have a deprecation period before outright removal.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did some more investigating on this yesterday, and I think I see an implementation that would better support the existing int and float offsets. However, this also raised some new questions about the existing implementation. I will try to push some concrete code here by end of day for people to consider, along with the new questions raised.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. I have added support back for int and non-fractional float being passed to tz in the Location constructor. I also test that pytz and zoneinfo timezones are likewise supported for initialization, and I beefed up the tests even more.

Note that the pytz Location attribute is now the single source of truth for the time zone in a Location object (which I recommend updating to a standard-library timezone.TimeZone object in #2342). self.tz simply returns the stringified self.pytz, but one can still directly set self.tz with any tz value supported by the Location intializer. Also, the tz and pytz attributes are kept in sync now, as changing one updates the other.

Ready for another review.

self.pytz = pytz.timezone(self.tz)
else:
raise TypeError('Invalid tz specification')

Expand All @@ -89,8 +90,10 @@ def __init__(self, latitude, longitude, tz='UTC', altitude=None,

def __repr__(self):
attrs = ['name', 'latitude', 'longitude', 'altitude', 'tz']
# Use None as getattr default in case __repr__ is called during
# initialization before all attributes have been assigned.
return ('Location: \n ' + '\n '.join(
f'{attr}: {getattr(self, attr)}' for attr in attrs))
f'{attr}: {getattr(self, attr, None)}' for attr in attrs))

@classmethod
def from_tmy(cls, tmy_metadata, tmy_data=None, **kwargs):
Expand Down
11 changes: 8 additions & 3 deletions pvlib/tests/test_location.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,16 @@ def test_location_all():


@pytest.mark.parametrize('tz', [
pytz.timezone('US/Arizona'), 'America/Phoenix', -7, -7.0,
datetime.timezone.utc
'America/Phoenix',
datetime.timezone.utc,
pytz.timezone('US/Arizona'),
-7,
-7.0,
])
def test_location_tz(tz):
Location(32.2, -111, tz)
loc = Location(32.2, -111, tz)
assert type(loc.tz) is str
assert isinstance(loc.pytz, pytz.tzinfo.BaseTzInfo)


def test_location_invalid_tz():
Expand Down
114 changes: 111 additions & 3 deletions pvlib/tests/test_tools.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
import pytest
from datetime import datetime
from zoneinfo import ZoneInfo

from pvlib import tools
import numpy as np
import pandas as pd
from numpy.testing import assert_allclose
import pandas as pd
import pytest

from pvlib import location, tools


@pytest.mark.parametrize('keys, input_dict, expected', [
Expand Down Expand Up @@ -144,3 +147,108 @@ def test_get_pandas_index(args, args_idx):
def test_normalize_max2one(data_in, expected):
result = tools.normalize_max2one(data_in)
assert_allclose(result, expected)


def test_localize_to_utc():
lat, lon = 43.2, -77.6
tz = "Etc/GMT+5"
loc = location.Location(lat, lon, tz=tz)
year, month, day, hour, minute, second = 1974, 6, 22, 18, 30, 15
hour_utc = hour + 5

# Test all combinations of supported inputs.
dt_time_aware_utc = datetime(
year, month, day, hour_utc, minute, second, tzinfo=ZoneInfo("UTC")
)
dt_time_aware = datetime(
year, month, day, hour, minute, second, tzinfo=ZoneInfo(tz)
)
assert tools.localize_to_utc(dt_time_aware, None) == dt_time_aware_utc
dt_time_naive = datetime(year, month, day, hour, minute, second)
assert tools.localize_to_utc(dt_time_naive, loc) == dt_time_aware_utc

# FIXME Derive timestamp strings from above variables.
dt_index_aware_utc = pd.DatetimeIndex(
[dt_time_aware_utc.strftime("%Y-%m-%dT%H:%M:%S")], tz=ZoneInfo("UTC")
)
dt_index_aware = pd.DatetimeIndex(
[dt_time_aware.strftime("%Y-%m-%dT%H:%M:%S")], tz=ZoneInfo(tz)
)
assert tools.localize_to_utc(dt_index_aware, None) == dt_index_aware_utc
dt_index_naive = pd.DatetimeIndex(
[dt_time_naive.strftime("%Y-%m-%dT%H:%M:%S")]
)
assert tools.localize_to_utc(dt_index_naive, loc) == dt_index_aware_utc

# Older pandas versions have wonky dtype equality check on timestamp
# index, so check the values as numpy.ndarray and indices one by one.
series_time_aware_utc_expected = pd.Series([24.42], dt_index_aware_utc)
series_time_aware = pd.Series([24.42], index=dt_index_aware)
series_time_aware_utc_got = tools.localize_to_utc(series_time_aware, None)
np.testing.assert_array_equal(
series_time_aware_utc_got.to_numpy(),
series_time_aware_utc_expected.to_numpy(),
)

for index_got, index_expected in zip(
series_time_aware_utc_got.index, series_time_aware_utc_expected.index
):
assert index_got == index_expected

series_time_naive = pd.Series([24.42], index=dt_index_naive)
series_time_naive_utc_got = tools.localize_to_utc(series_time_naive, loc)
np.testing.assert_array_equal(
series_time_naive_utc_got.to_numpy(),
series_time_aware_utc_expected.to_numpy(),
)

for index_got, index_expected in zip(
series_time_naive_utc_got.index, series_time_aware_utc_expected.index
):
assert index_got == index_expected

# Older pandas versions have wonky dtype equality check on timestamp
# index, so check the values as numpy.ndarray and indices one by one.
df_time_aware_utc_expected = pd.DataFrame([[24.42]], dt_index_aware)
df_time_naive = pd.DataFrame([[24.42]], index=dt_index_naive)
df_time_naive_utc_got = tools.localize_to_utc(df_time_naive, loc)
np.testing.assert_array_equal(
df_time_naive_utc_got.to_numpy(),
df_time_aware_utc_expected.to_numpy(),
)

for index_got, index_expected in zip(
df_time_naive_utc_got.index, df_time_aware_utc_expected.index
):
assert index_got == index_expected

df_time_aware = pd.DataFrame([[24.42]], index=dt_index_aware)
df_time_aware_utc_got = tools.localize_to_utc(df_time_aware, None)
np.testing.assert_array_equal(
df_time_aware_utc_got.to_numpy(),
df_time_aware_utc_expected.to_numpy(),
)

for index_got, index_expected in zip(
df_time_aware_utc_got.index, df_time_aware_utc_expected.index
):
assert index_got == index_expected


def test_datetime_to_djd():
expected = 27201.47934027778
dt_aware = datetime(1974, 6, 22, 18, 30, 15, tzinfo=ZoneInfo("Etc/GMT+5"))
assert tools.datetime_to_djd(dt_aware) == expected
dt_naive_utc = datetime(1974, 6, 22, 23, 30, 15)
assert tools.datetime_to_djd(dt_naive_utc) == expected


def test_djd_to_datetime():
djd = 27201.47934027778
tz = "Etc/GMT+5"

expected = datetime(1974, 6, 22, 18, 30, 15, tzinfo=ZoneInfo(tz))
assert tools.djd_to_datetime(djd, tz) == expected

expected = datetime(1974, 6, 22, 23, 30, 15, tzinfo=ZoneInfo("UTC"))
assert tools.djd_to_datetime(djd) == expected
8 changes: 4 additions & 4 deletions pvlib/tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -119,21 +119,21 @@ def atand(number):

def localize_to_utc(time, location):
"""
Converts or localizes a time series to UTC.
Converts time to UTC, localizing if necessary using location.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Converts time to UTC, localizing if necessary using location.
Converts ``time`` to UTC, localizing if necessary using location.


Parameters
----------
time : datetime.datetime, pandas.DatetimeIndex,
or pandas.Series/DataFrame with a DatetimeIndex.
location : pvlib.Location object
location : pvlib.Location object (unused if time is localized)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
location : pvlib.Location object (unused if time is localized)
location : pvlib.Location object (unused if ``time`` is localized)


Returns
-------
pandas object localized to UTC.
datetime.datetime or pandas object localized to UTC.
"""
if isinstance(time, dt.datetime):
if time.tzinfo is None:
time = pytz.timezone(location.tz).localize(time)
time = location.pytz.localize(time)
time_utc = time.astimezone(pytz.utc)
else:
try:
Expand Down
Loading