Two test failures in test_goes_imager_nc.py with Python 3.10

[gerritholl]

Describe the bug

Two tests fail on Python 3.10.

To Reproduce

pytest reader_tests/test_goes_imager_nc.py

Expected behavior

All tests pass.

Actual results

==================================================================================================================================================== test session starts ====================================================================================================================================================
platform linux -- Python 3.10.0, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /data/gholl/checkouts/satpy
plugins: cov-3.0.0, lazy-fixture-0.6.3
collected 20 items

test_goes_imager_nc.py ..F.....F...........                                                                                                                                                                                                                                                                           [100%]

========================================================================================================================================================= FAILURES ==========================================================================================================================================================
__________________________________________________________________________________________________________________________________________ GOESNCBaseFileHandlerTest.test_end_time __________________________________________________________________________________________________________________________________________

self = <satpy.tests.reader_tests.test_goes_imager_nc.GOESNCBaseFileHandlerTest testMethod=test_end_time>

    def test_end_time(self):
        """Test dataset end time stamp."""
        from satpy.readers.goes_imager_nc import FULL_DISC, SCAN_DURATION, UNKNOWN_SECTOR
        expected = {
            UNKNOWN_SECTOR: self.time,
            FULL_DISC: self.time + SCAN_DURATION[FULL_DISC]
        }
        for sector, end_time in expected.items():
            self.reader.sector = sector
>           self.assertEqual(self.reader.end_time, end_time)

test_goes_imager_nc.py:214:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../readers/goes_imager_nc.py:744: in end_time
    return self.start_time + SCAN_DURATION[self.sector]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <GOESNCBaseFileHandler: 'dummy'>

    @property
    def start_time(self):
        """Start timestamp of the dataset."""
        dt = self.nc['time'].dt
>       return datetime(year=dt.year, month=dt.month, day=dt.day,
                        hour=dt.hour, minute=dt.minute,
                        second=dt.second, microsecond=dt.microsecond)
E       TypeError: 'DataArray' object cannot be interpreted as an integer

../../readers/goes_imager_nc.py:736: TypeError
_________________________________________________________________________________________________________________________________________ GOESNCBaseFileHandlerTest.test_start_time _________________________________________________________________________________________________________________________________________

self = <satpy.tests.reader_tests.test_goes_imager_nc.GOESNCBaseFileHandlerTest testMethod=test_start_time>

    def test_start_time(self):
        """Test dataset start time stamp."""
>       self.assertEqual(self.reader.start_time, self.time)

test_goes_imager_nc.py:203:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <GOESNCBaseFileHandler: 'dummy'>

    @property
    def start_time(self):
        """Start timestamp of the dataset."""
        dt = self.nc['time'].dt
>       return datetime(year=dt.year, month=dt.month, day=dt.day,
                        hour=dt.hour, minute=dt.minute,
                        second=dt.second, microsecond=dt.microsecond)
E       TypeError: 'DataArray' object cannot be interpreted as an integer

../../readers/goes_imager_nc.py:736: TypeError
===================================================================================================================================================== warnings summary ======================================================================================================================================================
../../../../../mambaforge/envs/py310/lib/python3.10/site-packages/fsspec/spec.py:7
  /data/gholl/mambaforge/envs/py310/lib/python3.10/site-packages/fsspec/spec.py:7: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
    from distutils.version import LooseVersion

../../../../../mambaforge/envs/py310/lib/python3.10/site-packages/tornado/ioloop.py:263
../../../../../mambaforge/envs/py310/lib/python3.10/site-packages/tornado/ioloop.py:263
  /data/gholl/mambaforge/envs/py310/lib/python3.10/site-packages/tornado/ioloop.py:263: DeprecationWarning: There is no current event loop
    loop = asyncio.get_event_loop()

../../../../../mambaforge/envs/py310/lib/python3.10/site-packages/tornado/platform/asyncio.py:279
  /data/gholl/mambaforge/envs/py310/lib/python3.10/site-packages/tornado/platform/asyncio.py:279: DeprecationWarning: There is no current event loop
    super().initialize(asyncio.get_event_loop(), **kwargs)

satpy/tests/reader_tests/test_goes_imager_nc.py::GOESNCEUMFileHandlerRadianceTest::test_get_dataset_radiance
satpy/tests/reader_tests/test_goes_imager_nc.py::GOESNCEUMFileHandlerRadianceTest::test_get_dataset_radiance
satpy/tests/reader_tests/test_goes_imager_nc.py::GOESNCEUMFileHandlerRadianceTest::test_get_dataset_radiance
satpy/tests/reader_tests/test_goes_imager_nc.py::GOESNCEUMFileHandlerRadianceTest::test_get_dataset_radiance
satpy/tests/reader_tests/test_goes_imager_nc.py::GOESNCEUMFileHandlerReflectanceTest::test_get_dataset_reflectance
  /data/gholl/mambaforge/envs/py310/lib/python3.10/site-packages/xarray/core/dataarray.py:2360: PendingDeprecationWarning: dropping variables using `drop` will be deprecated; using drop_vars is encouraged.
    ds = self._to_temp_dataset().drop(labels, dim, errors=errors)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
================================================================================================================================================== short test summary info ==================================================================================================================================================
FAILED test_goes_imager_nc.py::GOESNCBaseFileHandlerTest::test_end_time - TypeError: 'DataArray' object cannot be interpreted as an integer
FAILED test_goes_imager_nc.py::GOESNCBaseFileHandlerTest::test_start_time - TypeError: 'DataArray' object cannot be interpreted as an integer
========================================================================================================================================= 2 failed, 18 passed, 9 warnings in 0.48s ==========================================================================================================================================

Environment Info:

• OS: openSUSE Leap 15.3
• Satpy Version: v0.32.0-20-g008b06d3
• Python Version: 3.10.0

Additional context

[djhoese]

@gerritholl Any idea what part of that is a DataArray:

        dt = self.nc['time'].dt
        return datetime(year=dt.year, month=dt.month, day=dt.day,
                        hour=dt.hour, minute=dt.minute,
                        second=dt.second, microsecond=dt.microsecond)

I would think that self.nc['time'].dt is the attribute right? Or maybe that's the coordinate variable so it is a DataArray. If it is the attribute then this should be updated to at least be self.nc['time'].attrs['dt'].

[gerritholl]

dt is <xarray.core.accessor_dt.DatetimeAccessor object at 0x7f81a70fb1f0>. dt.year, dt.month, etc. are all DataArray. self.nc["time"].attrs == {}.

[djhoese]

I believe @simonrp84 originally wrote this reader. Any ideas how this was supposed to work @simonrp84? Is dt supposed to be an accessor or is it an attribute/coordinate of the self.nc['time'] variable? My guess is we should either do .attrs['dt'] or .coords['dt'] depending on what it actually is.

@gerritholl Can you print out what self.nc['time'] is?

[gerritholl]

>>> print(self.nc["time"])
<xarray.DataArray 'time' (time: 1)>
array(['2018-08-16T16:07:00.000000000'], dtype='datetime64[ns]')
Coordinates:
  * time     (time) datetime64[ns] 2018-08-16T16:07:00

I could compare with Python 3.9, but I should work on other things right now.

[djhoese]

Not sure why this would change in Python 3.10, but it looks like the accessor excepts to be used on an array of datetime-like objects:

https://xarray.pydata.org/en/stable/generated/xarray.core.accessor_dt.DatetimeAccessor.html

Looking at the source for that accessor my guess is that this is a difference in xarray that is technically an unsupported use case (passing DataArrays to datetime and it automatically converting it to a scalar) or a change in xarray where things are completely based on pandas Series anymore (just a guess). I think it could be updated to either access the individual scalar value of each property of the dt accessor or the reader could get the string from .values[0] (I think) and do the strptime call directly.

[pnuu]

Closed by #1933 where I cast the xr.DataArray datetime values to integers.

[sfinkens]

Closed by @pnuu 🙂