Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix item failure when encountering null values in xml metadata #33

Merged
merged 4 commits into from
Apr 15, 2022
Merged

fix item failure when encountering null values in xml metadata #33

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ca34371
fix item failure when encountering null values in xml metadata
pjhartzell Apr 15, 2022
59efa95
correct typing
pjhartzell Apr 15, 2022
a730cca
typing fix
pjhartzell Apr 15, 2022
dc5ef44
update changelog
pjhartzell Apr 15, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.MD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).

### Fixed

- Remove the `unit`, `scale`, and `offset` properties from `raster:bands` when "NULL" values for `mult` and `add` are encountered in the metadata. This applies to Collection 2 Level-1 data only.
- Handle MultiPolygon geometries before applying the stactools antimeridian `fix_item` utility function ([#30](https://github.com/stactools-packages/landsat/pull/30))
- `eo:cloud_cover` is not added to Item properties when it was not computed (indicated by a value of -1) during scene processing ([#28](https://github.com/stactools-packages/landsat/pull/28))

Expand Down
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
- STAC extensions used:
- [classification](https://github.com/stac-extensions/classification)
- [eo](https://github.com/stac-extensions/eo)
- [landsat](https://landsat.usgs.gov/stac/landsat-extension/v1.1.0/schema.json)
- [landsat](https://landsat.usgs.gov/stac/landsat-extension/v1.1.1/schema.json)
- [proj](https://github.com/stac-extensions/projection/)
- [raster](https://github.com/stac-extensions/raster)
- [scientific](https://github.com/stac-extensions/scientific)
Expand Down
14 changes: 9 additions & 5 deletions src/stactools/landsat/fragments/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ class Fragments:
"""Class for accessing asset and extension data."""

def __init__(self, sensor: Sensor, satellite: int, base_href: str,
level1_radiance: Dict[str, Dict[str, float]]):
level1_radiance: Dict[str, Dict[str, Any]]):
gadomski marked this conversation as resolved.
Show resolved Hide resolved
"""Initialize a new group of fragments for the provided Sensor."""
self._sensor = sensor
self._satellite = satellite
Expand Down Expand Up @@ -165,10 +165,14 @@ def _update_mss_raster(self, mss_raster_dict: Dict[str,
for key, value in mss_raster_dict.items():
rad_key = value.pop("temp_name", None)
if rad_key:
mss_raster_dict[key]["scale"] = self.level1_radiance[rad_key][
"mult"]
mss_raster_dict[key]["offset"] = self.level1_radiance[rad_key][
"add"]
if not self.level1_radiance[rad_key][
"mult"] or not self.level1_radiance[rad_key]["add"]:
mss_raster_dict[key].pop("unit")
else:
mss_raster_dict[key]["scale"] = self.level1_radiance[
rad_key]["mult"]
mss_raster_dict[key]["offset"] = self.level1_radiance[
rad_key]["add"]

return mss_raster_dict

Expand Down
15 changes: 11 additions & 4 deletions src/stactools/landsat/mtl_metadata.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,11 @@ def _get_float(self, xpath: str) -> float:
def _get_int(self, xpath: str) -> int:
return int(self._get_text(xpath))

def _float_or_none(self, value: str) -> Optional[float]:
if value == "NULL":
return None
return float(value)

@property
def satellite_num(self) -> int:
"""Return the Landsat satellite number."""
Expand Down Expand Up @@ -317,11 +322,13 @@ def landsat_metadata(self) -> Dict[str, Any]:
return landsat_meta

@property
def level1_radiance(self) -> Dict[str, Any]:
def level1_radiance(self) -> Dict[str, Dict[str, Any]]:
"""Gets the scale (mult) and offset (add) values for generating TOA
radiance from Level-1 DNs.

This is relevant to MSS data, which is only processed to Level-1.
This is relevant to MSS data, which is only processed to Level-1. Sets
the scale and offset values to None if a NULL text string is
encountered.

Returns:
Dict[str, Any]: Dict of scale and offset dicts, keyed by band
Expand All @@ -333,10 +340,10 @@ def level1_radiance(self) -> Dict[str, Any]:
for item in node.element:
if item.tag.startswith("RADIANCE_MULT_BAND"):
band = f'B{item.tag.split("_")[-1]}'
mult_add[band]["mult"] = float(str(item.text))
mult_add[band]["mult"] = self._float_or_none(item.text)
elif item.tag.startswith("RADIANCE_ADD_BAND"):
band = f'B{item.tag.split("_")[-1]}'
mult_add[band]["add"] = float(str(item.text))
mult_add[band]["add"] = self._float_or_none(item.text)
return mult_add

@classmethod
Expand Down
167 changes: 167 additions & 0 deletions tests/data-files/mss/LM01_L1GS_007019_19771009_20200907_02_T2_MTL.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
<?xml version="1.0" encoding="UTF-8"?>
<LANDSAT_METADATA_FILE>
<PRODUCT_CONTENTS>
<ORIGIN>Image courtesy of the U.S. Geological Survey</ORIGIN>
<DIGITAL_OBJECT_IDENTIFIER>https://doi.org/10.5066/P9AF14YV</DIGITAL_OBJECT_IDENTIFIER>
<LANDSAT_PRODUCT_ID>LM01_L1GS_007019_19771009_20200907_02_T2</LANDSAT_PRODUCT_ID>
<PROCESSING_LEVEL>L1GS</PROCESSING_LEVEL>
<COLLECTION_NUMBER>02</COLLECTION_NUMBER>
<COLLECTION_CATEGORY>T2</COLLECTION_CATEGORY>
<OUTPUT_FORMAT>GEOTIFF</OUTPUT_FORMAT>
<FILE_NAME_BAND_4>LM01_L1GS_007019_19771009_20200907_02_T2_B4.TIF</FILE_NAME_BAND_4>
<FILE_NAME_BAND_5>LM01_L1GS_007019_19771009_20200907_02_T2_B5.TIF</FILE_NAME_BAND_5>
<FILE_NAME_BAND_6>LM01_L1GS_007019_19771009_20200907_02_T2_B6.TIF</FILE_NAME_BAND_6>
<FILE_NAME_BAND_7>LM01_L1GS_007019_19771009_20200907_02_T2_B7.TIF</FILE_NAME_BAND_7>
<FILE_NAME_QUALITY_L1_PIXEL>LM01_L1GS_007019_19771009_20200907_02_T2_QA_PIXEL.TIF</FILE_NAME_QUALITY_L1_PIXEL>
<FILE_NAME_QUALITY_L1_RADIOMETRIC_SATURATION>LM01_L1GS_007019_19771009_20200907_02_T2_QA_RADSAT.TIF</FILE_NAME_QUALITY_L1_RADIOMETRIC_SATURATION>
<FILE_NAME_METADATA_ODL>LM01_L1GS_007019_19771009_20200907_02_T2_MTL.txt</FILE_NAME_METADATA_ODL>
<FILE_NAME_METADATA_XML>LM01_L1GS_007019_19771009_20200907_02_T2_MTL.xml</FILE_NAME_METADATA_XML>
<DATA_TYPE_BAND_4>UINT8</DATA_TYPE_BAND_4>
<DATA_TYPE_BAND_5>UINT8</DATA_TYPE_BAND_5>
<DATA_TYPE_BAND_6>UINT8</DATA_TYPE_BAND_6>
<DATA_TYPE_BAND_7>UINT8</DATA_TYPE_BAND_7>
<DATA_TYPE_QUALITY_L1_PIXEL>UINT16</DATA_TYPE_QUALITY_L1_PIXEL>
<DATA_TYPE_QUALITY_L1_RADIOMETRIC_SATURATION>UINT16</DATA_TYPE_QUALITY_L1_RADIOMETRIC_SATURATION>
<PRESENT_BAND_4>M</PRESENT_BAND_4>
<PRESENT_BAND_5>Y</PRESENT_BAND_5>
<PRESENT_BAND_6>Y</PRESENT_BAND_6>
<PRESENT_BAND_7>Y</PRESENT_BAND_7>
</PRODUCT_CONTENTS>
<IMAGE_ATTRIBUTES>
<SPACECRAFT_ID>LANDSAT_1</SPACECRAFT_ID>
<SENSOR_ID>MSS</SENSOR_ID>
<WRS_TYPE>1</WRS_TYPE>
<WRS_PATH>007</WRS_PATH>
<WRS_ROW>019</WRS_ROW>
<DATE_ACQUIRED>1977-10-09</DATE_ACQUIRED>
<SCENE_CENTER_TIME>12:52:36.8530000Z</SCENE_CENTER_TIME>
<STATION_ID>GMD</STATION_ID>
<CLOUD_COVER>10.00</CLOUD_COVER>
<CLOUD_COVER_LAND>0.00</CLOUD_COVER_LAND>
<IMAGE_QUALITY>-1</IMAGE_QUALITY>
<SATURATION_BAND_4>N</SATURATION_BAND_4>
<SATURATION_BAND_5>Y</SATURATION_BAND_5>
<SATURATION_BAND_6>Y</SATURATION_BAND_6>
<SATURATION_BAND_7>N</SATURATION_BAND_7>
<SUN_AZIMUTH>139.16144300</SUN_AZIMUTH>
<SUN_ELEVATION>18.09490652</SUN_ELEVATION>
<EARTH_SUN_DISTANCE>0.9986936</EARTH_SUN_DISTANCE>
</IMAGE_ATTRIBUTES>
<PROJECTION_ATTRIBUTES>
<MAP_PROJECTION>UTM</MAP_PROJECTION>
<DATUM>WGS84</DATUM>
<ELLIPSOID>WGS84</ELLIPSOID>
<UTM_ZONE>21</UTM_ZONE>
<GRID_CELL_SIZE_REFLECTIVE>60.00</GRID_CELL_SIZE_REFLECTIVE>
<REFLECTIVE_LINES>3823</REFLECTIVE_LINES>
<REFLECTIVE_SAMPLES>3935</REFLECTIVE_SAMPLES>
<ORIENTATION>NORTH_UP</ORIENTATION>
<CORNER_UL_LAT_PRODUCT>59.58514</CORNER_UL_LAT_PRODUCT>
<CORNER_UL_LON_PRODUCT>-57.15228</CORNER_UL_LON_PRODUCT>
<CORNER_UR_LAT_PRODUCT>59.52341</CORNER_UR_LAT_PRODUCT>
<CORNER_UR_LON_PRODUCT>-52.97846</CORNER_UR_LON_PRODUCT>
<CORNER_LL_LAT_PRODUCT>57.52556</CORNER_LL_LAT_PRODUCT>
<CORNER_LL_LON_PRODUCT>-57.14360</CORNER_LL_LON_PRODUCT>
<CORNER_LR_LAT_PRODUCT>57.46859</CORNER_LR_LAT_PRODUCT>
<CORNER_LR_LON_PRODUCT>-53.20705</CORNER_LR_LON_PRODUCT>
<CORNER_UL_PROJECTION_X_PRODUCT>491400.000</CORNER_UL_PROJECTION_X_PRODUCT>
<CORNER_UL_PROJECTION_Y_PRODUCT>6605220.000</CORNER_UL_PROJECTION_Y_PRODUCT>
<CORNER_UR_PROJECTION_X_PRODUCT>727440.000</CORNER_UR_PROJECTION_X_PRODUCT>
<CORNER_UR_PROJECTION_Y_PRODUCT>6605220.000</CORNER_UR_PROJECTION_Y_PRODUCT>
<CORNER_LL_PROJECTION_X_PRODUCT>491400.000</CORNER_LL_PROJECTION_X_PRODUCT>
<CORNER_LL_PROJECTION_Y_PRODUCT>6375900.000</CORNER_LL_PROJECTION_Y_PRODUCT>
<CORNER_LR_PROJECTION_X_PRODUCT>727440.000</CORNER_LR_PROJECTION_X_PRODUCT>
<CORNER_LR_PROJECTION_Y_PRODUCT>6375900.000</CORNER_LR_PROJECTION_Y_PRODUCT>
</PROJECTION_ATTRIBUTES>
<LEVEL1_PROCESSING_RECORD>
<ORIGIN>Image courtesy of the U.S. Geological Survey</ORIGIN>
<DIGITAL_OBJECT_IDENTIFIER>https://doi.org/10.5066/P9AF14YV</DIGITAL_OBJECT_IDENTIFIER>
<REQUEST_ID>L2</REQUEST_ID>
<LANDSAT_SCENE_ID>LM10070191977282GMD03</LANDSAT_SCENE_ID>
<LANDSAT_PRODUCT_ID>LM01_L1GS_007019_19771009_20200907_02_T2</LANDSAT_PRODUCT_ID>
<PROCESSING_LEVEL>L1GS</PROCESSING_LEVEL>
<COLLECTION_CATEGORY>T2</COLLECTION_CATEGORY>
<OUTPUT_FORMAT>GEOTIFF</OUTPUT_FORMAT>
<DATE_PRODUCT_GENERATED>2020-09-07T05:35:06Z</DATE_PRODUCT_GENERATED>
<PROCESSING_SOFTWARE_VERSION>LPGS_15.3.1c</PROCESSING_SOFTWARE_VERSION>
<FILE_NAME_BAND_4>LM01_L1GS_007019_19771009_20200907_02_T2_B4.TIF</FILE_NAME_BAND_4>
<FILE_NAME_BAND_5>LM01_L1GS_007019_19771009_20200907_02_T2_B5.TIF</FILE_NAME_BAND_5>
<FILE_NAME_BAND_6>LM01_L1GS_007019_19771009_20200907_02_T2_B6.TIF</FILE_NAME_BAND_6>
<FILE_NAME_BAND_7>LM01_L1GS_007019_19771009_20200907_02_T2_B7.TIF</FILE_NAME_BAND_7>
<FILE_NAME_QUALITY_L1_PIXEL>LM01_L1GS_007019_19771009_20200907_02_T2_QA_PIXEL.TIF</FILE_NAME_QUALITY_L1_PIXEL>
<FILE_NAME_QUALITY_L1_RADIOMETRIC_SATURATION>LM01_L1GS_007019_19771009_20200907_02_T2_QA_RADSAT.TIF</FILE_NAME_QUALITY_L1_RADIOMETRIC_SATURATION>
<FILE_NAME_METADATA_ODL>LM01_L1GS_007019_19771009_20200907_02_T2_MTL.txt</FILE_NAME_METADATA_ODL>
<FILE_NAME_METADATA_XML>LM01_L1GS_007019_19771009_20200907_02_T2_MTL.xml</FILE_NAME_METADATA_XML>
<FILE_NAME_CPF>LM01CPF_19720723_19780107_02.01</FILE_NAME_CPF>
<EPHEMERIS_TYPE>PREDICTIVE</EPHEMERIS_TYPE>
</LEVEL1_PROCESSING_RECORD>
<LEVEL1_MIN_MAX_RADIANCE>
<RADIANCE_MAXIMUM_BAND_4>NULL</RADIANCE_MAXIMUM_BAND_4>
<RADIANCE_MINIMUM_BAND_4>NULL</RADIANCE_MINIMUM_BAND_4>
<RADIANCE_MAXIMUM_BAND_5>164.600</RADIANCE_MAXIMUM_BAND_5>
<RADIANCE_MINIMUM_BAND_5>-0.100</RADIANCE_MINIMUM_BAND_5>
<RADIANCE_MAXIMUM_BAND_6>165.600</RADIANCE_MAXIMUM_BAND_6>
<RADIANCE_MINIMUM_BAND_6>-0.100</RADIANCE_MINIMUM_BAND_6>
<RADIANCE_MAXIMUM_BAND_7>154.600</RADIANCE_MAXIMUM_BAND_7>
<RADIANCE_MINIMUM_BAND_7>0.000</RADIANCE_MINIMUM_BAND_7>
</LEVEL1_MIN_MAX_RADIANCE>
<LEVEL1_MIN_MAX_REFLECTANCE>
<REFLECTANCE_MAXIMUM_BAND_4>NULL</REFLECTANCE_MAXIMUM_BAND_4>
<REFLECTANCE_MINIMUM_BAND_4>NULL</REFLECTANCE_MINIMUM_BAND_4>
<REFLECTANCE_MAXIMUM_BAND_5>0.335560</REFLECTANCE_MAXIMUM_BAND_5>
<REFLECTANCE_MINIMUM_BAND_5>-0.000204</REFLECTANCE_MINIMUM_BAND_5>
<REFLECTANCE_MAXIMUM_BAND_6>0.407291</REFLECTANCE_MAXIMUM_BAND_6>
<REFLECTANCE_MINIMUM_BAND_6>-0.000246</REFLECTANCE_MINIMUM_BAND_6>
<REFLECTANCE_MAXIMUM_BAND_7>0.572400</REFLECTANCE_MAXIMUM_BAND_7>
<REFLECTANCE_MINIMUM_BAND_7>0.000000</REFLECTANCE_MINIMUM_BAND_7>
</LEVEL1_MIN_MAX_REFLECTANCE>
<LEVEL1_MIN_MAX_PIXEL_VALUE>
<QUANTIZE_CAL_MAX_BAND_4>NULL</QUANTIZE_CAL_MAX_BAND_4>
<QUANTIZE_CAL_MIN_BAND_4>NULL</QUANTIZE_CAL_MIN_BAND_4>
<QUANTIZE_CAL_MAX_BAND_5>255</QUANTIZE_CAL_MAX_BAND_5>
<QUANTIZE_CAL_MIN_BAND_5>1</QUANTIZE_CAL_MIN_BAND_5>
<QUANTIZE_CAL_MAX_BAND_6>255</QUANTIZE_CAL_MAX_BAND_6>
<QUANTIZE_CAL_MIN_BAND_6>1</QUANTIZE_CAL_MIN_BAND_6>
<QUANTIZE_CAL_MAX_BAND_7>255</QUANTIZE_CAL_MAX_BAND_7>
<QUANTIZE_CAL_MIN_BAND_7>1</QUANTIZE_CAL_MIN_BAND_7>
</LEVEL1_MIN_MAX_PIXEL_VALUE>
<LEVEL1_RADIOMETRIC_RESCALING>
<RADIANCE_MULT_BAND_4>NULL</RADIANCE_MULT_BAND_4>
<RADIANCE_MULT_BAND_5>6.4843E-01</RADIANCE_MULT_BAND_5>
<RADIANCE_MULT_BAND_6>6.5236E-01</RADIANCE_MULT_BAND_6>
<RADIANCE_MULT_BAND_7>6.0866E-01</RADIANCE_MULT_BAND_7>
<RADIANCE_ADD_BAND_4>NULL</RADIANCE_ADD_BAND_4>
<RADIANCE_ADD_BAND_5>-0.74843</RADIANCE_ADD_BAND_5>
<RADIANCE_ADD_BAND_6>-0.75236</RADIANCE_ADD_BAND_6>
<RADIANCE_ADD_BAND_7>-0.60866</RADIANCE_ADD_BAND_7>
<REFLECTANCE_MULT_BAND_4>NULL</REFLECTANCE_MULT_BAND_4>
<REFLECTANCE_MULT_BAND_5>1.3219E-03</REFLECTANCE_MULT_BAND_5>
<REFLECTANCE_MULT_BAND_6>1.6045E-03</REFLECTANCE_MULT_BAND_6>
<REFLECTANCE_MULT_BAND_7>2.2535E-03</REFLECTANCE_MULT_BAND_7>
<REFLECTANCE_ADD_BAND_4>NULL</REFLECTANCE_ADD_BAND_4>
<REFLECTANCE_ADD_BAND_5>-0.001526</REFLECTANCE_ADD_BAND_5>
<REFLECTANCE_ADD_BAND_6>-0.001850</REFLECTANCE_ADD_BAND_6>
<REFLECTANCE_ADD_BAND_7>-0.002254</REFLECTANCE_ADD_BAND_7>
</LEVEL1_RADIOMETRIC_RESCALING>
<LEVEL1_PROJECTION_PARAMETERS>
<MAP_PROJECTION>UTM</MAP_PROJECTION>
<DATUM>WGS84</DATUM>
<ELLIPSOID>WGS84</ELLIPSOID>
<UTM_ZONE>21</UTM_ZONE>
<GRID_CELL_SIZE_REFLECTIVE>60.00</GRID_CELL_SIZE_REFLECTIVE>
<ORIENTATION>NORTH_UP</ORIENTATION>
<RESAMPLING_OPTION>CUBIC_CONVOLUTION</RESAMPLING_OPTION>
<MAP_PROJECTION_L0RA>NA</MAP_PROJECTION_L0RA>
</LEVEL1_PROJECTION_PARAMETERS>
<PRODUCT_PARAMETERS>
<DATA_TYPE_L0RP>MSSR_L0RP</DATA_TYPE_L0RP>
<CORRECTION_GAIN_BAND_4>NA</CORRECTION_GAIN_BAND_4>
<CORRECTION_GAIN_BAND_5>CPF</CORRECTION_GAIN_BAND_5>
<CORRECTION_GAIN_BAND_6>CPF</CORRECTION_GAIN_BAND_6>
<CORRECTION_GAIN_BAND_7>CPF</CORRECTION_GAIN_BAND_7>
<GAIN_BAND_4>U</GAIN_BAND_4>
<GAIN_BAND_5>L</GAIN_BAND_5>
<GAIN_BAND_6>L</GAIN_BAND_6>
<GAIN_BAND_7>L</GAIN_BAND_7>
</PRODUCT_PARAMETERS>
</LANDSAT_METADATA_FILE>
Loading