SAOCOM L1A products are already calibrated #42

impresionista · 2022-08-08T21:12:32Z

I'm trying to read SAOCOM data without success.

Here is a minimal example to reproduce my error:

import os
from pathlib import Path

from eoreader.reader import Reader
from eoreader.bands import *


data_dir = os.path.abspath(os.path.join("..", "..", "data", "raw", "saocom"))
saocom_file = "S1A_OPER_SAR_EOSSP__CORE_L1A_OLVF_20220317T145008"

saocom_prod = Reader().open(data_dir, saocom_file)

print(saocom_prod)

eoreader.SaocomProduct 'S1A_OPER_SAR_EOSSP__CORE_L1A_OLVF_20220317T143007'
Attributes:
condensed_name: 20200801T103425_SAOCOM_VV_HH_VH_HV_SM_L1A
path: /root/data/raw/saocom
constellation: SAOCOM-1
sensor type: SAR
product type: L1A
default resolution: 10.0
acquisition datetime: 2020-08-01T10:34:25.550874
band mapping:
VV: VV
VV_DSPK: VV_DSPK
HH: HH
HH_DSPK: HH_DSPK
VH: VH
VH_DSPK: VH_DSPK
HV: HV
HV_DSPK: HV_DSPK
needs extraction: True
orbit direction: ASCENDING

It appears to correctly read the data/metadata, but it fails at getting some of the bands:

ok_bands = saocom_prod.get_existing_bands()
saocom_prod.load(bands=[VV])

Here is the traceback of the error:
https://pastebin.com/SwL9D9Ju

I need help to load the data as an xarray.

Thanks!

remi-braun · 2022-08-09T14:52:20Z

Hello !
Thanks for the feedback
I am currently in holidays so I will try to help you without access to my computer 😅

It seems this is a SNAP issue so I will propose a workaround in the meantime: could you modify the SAR Preprocess XML and remove the calibration step?
You can add this custom GPT Graph following the instructions on the SAR page of the documentation

Hope this will work...

I will investiguate further when I can 😉

remi-braun · 2022-08-09T15:02:56Z

But Absolute radiometric calibration has already been applied to the product isn't strange for L1A data ? Do you know if SAOCOM has changed its product specifications ?

impresionista · 2022-08-09T19:20:21Z

But Absolute radiometric calibration has already been applied to the product isn't strange for L1A data ? Do you know if SAOCOM has changed its product specifications ?

I've confirmation that SAOCOM L1A products are calibrated.

remi-braun · 2022-08-09T19:55:38Z

Ok so that's the issue, you need to remove the calibration step from the SNAP GPT XML

I will make the workaround as soon as I'm back, around the end of august 😉

remi-braun · 2022-08-09T19:56:36Z

The other issue is Why the SAOCOM product I'm using passes the tests 🤔

impresionista · 2022-08-09T19:57:18Z

Thanks! That'll be great! Let me know if I can help you in any way.

impresionista · 2022-08-09T20:02:41Z

The other issue is Why the SAOCOM product I'm using passes the tests thinking

What products are you using? I'm working inside docker, so i'm trying to understand if SNAP is not installed correctly inside it.
The error message wasn't enough for me to discern if there was an issue with my SNAP/GPT config or with the SAOCOM data.

Snap its installed using default configs:

RUN wget --no-verbose https://download.esa.int/step/snap/8.0/installers/esa-snap_all_unix_8_0.sh &&\
    chmod +x ./esa-snap_all_unix_8_0.sh &&\
    ./esa-snap_all_unix_8_0.sh -q &&\
    rm ./esa-snap_all_unix_8_0.sh

ENV PATH="/usr/local/snap/bin:$PATH"

Edit:
Just in case, I've executed in another cell (i'm working in a jupyter notebook)

!echo $PATH

/usr/local/snap/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

It appears to be ok. And:

!gpt

Returns gpt execution:

INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Incompatible GDAL 3.5.1 found on system. Internal GDAL 3.0.0 from distribution will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
Usage:
gpt | [options] [ ...]

Description:
This tool is used to execute SNAP raster data operators in batch-mode. The
operators can be used stand-alone or combined as a directed acyclic graph
(DAG). Processing graphs are represented using XML. More info about
processing graphs, the operator API, and the graph XML format can be found
in the SNAP documentation.

...

remi-braun · 2022-08-10T08:46:12Z

You can now use SNAP 9.0 😉
And if you are using the 8th version it is best to install all the updates if you can

I also am using everything in a docker, I could help you if needed

I am using the 4 polarisation product you can download on their website (among others)

impresionista · 2022-08-10T21:37:20Z

Well, I have news.

I've tried using a basic gpt graph without any preprocessing filter, only read an write, with relative success. I got a new error regarding missing CRS.

<graph id="Graph">
  <version>1.0</version>

  <node id="Read">
    <operator>Read</operator>
    <sources/>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <file>$file</file>
    </parameters>
  </node>

  <node id="Write">
    <operator>Write</operator>
    <sources>
      <sourceProduct refid="Read"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <file>$out</file>
      <formatName>BEAM-DIMAP</formatName>
    </parameters>
 </node>

</graph>

Python code:

saocom_vv = saocom_prod.load(bands=[VV])

Error message:

MissingCRS: CRS not found. Please set the CRS with 'rio.write_crs()'. Data variable: 20200614T103426_SAOCOM_VV_HH_VH_HV_SM_L1A_VV

I've tried adding a "Terrain-Correction" step to the preprocessing graph. The new error was about a division by cero.

<graph id="Graph">
  <version>1.0</version>

  <node id="Read">
    <operator>Read</operator>
    <sources/>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <file>$file</file>
    </parameters>
  </node>

  <node id="Terrain-Correction">
    <operator>Terrain-Correction</operator>
    <sources>
      <sourceProduct refid="Read"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <sourceBands/>
      <demName>SRTM 3Sec</demName>
      <externalDEMFile/>
      <externalDEMNoDataValue>0.0</externalDEMNoDataValue>
      <externalDEMApplyEGM>true</externalDEMApplyEGM>
      <demResamplingMethod>BILINEAR_INTERPOLATION</demResamplingMethod>
      <imgResamplingMethod>BILINEAR_INTERPOLATION</imgResamplingMethod>
      <pixelSpacingInMeter>8.46</pixelSpacingInMeter>
      <pixelSpacingInDegree>7.599747303651153E-5</pixelSpacingInDegree>
      <mapProjection>GEOGCS[&quot;WGS84(DD)&quot;,
        DATUM[&quot;WGS84&quot;,
          SPHEROID[&quot;WGS84&quot;, 6378137.0, 298.257223563]],
        PRIMEM[&quot;Greenwich&quot;, 0.0],
        UNIT[&quot;degree&quot;, 0.017453292519943295],
        AXIS[&quot;Geodetic longitude&quot;, EAST],
        AXIS[&quot;Geodetic latitude&quot;, NORTH]]</mapProjection>
      <alignToStandardGrid>false</alignToStandardGrid>
      <standardGridOriginX>0.0</standardGridOriginX>
      <standardGridOriginY>0.0</standardGridOriginY>
      <nodataValueAtSea>true</nodataValueAtSea>
      <saveDEM>false</saveDEM>
      <saveLatLon>false</saveLatLon>
      <saveIncidenceAngleFromEllipsoid>false</saveIncidenceAngleFromEllipsoid>
      <saveLocalIncidenceAngle>false</saveLocalIncidenceAngle>
      <saveProjectedLocalIncidenceAngle>false</saveProjectedLocalIncidenceAngle>
      <saveSelectedSourceBand>true</saveSelectedSourceBand>
      <saveLayoverShadowMask>false</saveLayoverShadowMask>
      <outputComplex>false</outputComplex>
      <applyRadiometricNormalization>false</applyRadiometricNormalization>
      <saveSigmaNought>false</saveSigmaNought>
      <saveGammaNought>false</saveGammaNought>
      <saveBetaNought>false</saveBetaNought>
      <incidenceAngleForSigma0>Use projected local incidence angle from DEM</incidenceAngleForSigma0>
      <incidenceAngleForGamma0>Use projected local incidence angle from DEM</incidenceAngleForGamma0>
      <auxFile>Latest Auxiliary File</auxFile>
      <externalAuxFile/>
    </parameters>
  </node>

  <node id="Write">
    <operator>Write</operator>
    <sources>
      <sourceProduct refid="Terrain-Correction"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <file>$out</file>
      <formatName>BEAM-DIMAP</formatName>
    </parameters>
 </node>

</graph>

Python code:

saocom_vv = saocom_prod.load(bands=[VV])

Error message:

Division by zero https://pastebin.com/tQz3zeVy

I've tried using a multilook filter with and without "Terrain-correction" with no success, the errors were the same as 1. and 2.

remi-braun · 2022-08-11T04:34:12Z

For L1A you must set the Terrain Correction step as EOReader only reads orthorectified data in UTM

Could you use the graph without calibration that is already existing: eoreader/data/cplx_no_calib_preprocess_default.xml

Did you add the updates to your SNAP 8 ?

impresionista · 2022-08-11T17:36:27Z

Great! I'll try eoreader/data/cplx_no_calib_preprocess_default.xml graph.

Did you add the updates to your SNAP 8 ?

I updated to SNAP 9.

impresionista · 2022-08-11T18:42:40Z

With eoreader/data/cplx_no_calib_preprocess_default.xml seems to be working perfectly!

remi-braun · 2022-08-11T18:45:50Z

OK so happy to hear that!
Could you try with the default graph just to see if it was SNAP version (without updates) that was the issue ?

impresionista · 2022-08-11T19:00:08Z

Yes, it seems that using SNAP 9 solved the issue.

I'll check with the SAR specialist in my team to find out if its working correctly, just in case there is something miss-configured.

remi-braun · 2022-08-11T19:04:00Z

I think that the updates of SNAP made the calibration step fail safe in some sort

I will add that in the documentation !

impresionista · 2022-08-18T20:37:23Z

I've checked with the team and it seams that the default filter is working correctly.
I'll close this issue.
Thanks for your help! :)

remi-braun · 2022-08-19T06:34:03Z

Thanks for the effort you put in helping me resolving this 😉
(I'm glad it was a SNAP issue 😅)

JaumePrimer · 2022-09-13T08:58:17Z

Hi, just jumping on to this issue, albeit closed.

I've had similar problems with SNAP 8.0 and SAOCOM. I'm using docker and all signs point to the problem being SNAP 8.0. I can't find any examples of a dockerfile for SNAP 9.0. Does an open source one exist?

remi-braun · 2022-09-13T09:07:01Z

Hello,

Here are the lines concerning the SNAP installation on my dockerfile.

# Install SNAP
# - download snap installer version 8.0 + TBX
# - change file execution rights for snap installer
# - install snap with gpt
# - link gpt so it can be used systemwide
ARG snap_version=9
ARG snap_update=snap_update.sh
ARG snap_script=esa-snap_sentinel_unix_"$snap_version"_0_0.sh
RUN wget http://step.esa.int/downloads/"$snap_version".0/installers/"$snap_script" \
&& chmod +x "$snap_script" \
&& ./"$snap_script" -q \
&& ln -s /usr/local/snap/bin/gpt /usr/bin/gpt \
# Update SNAP
&& chmod +x "$eeo_root""$snap_update"  \
&& "$eeo_root""$snap_update" \
# Add to line to GPT's VMoptions -> Bug (https://forum.step.esa.int/t/xmlfactory-error-using-snap-8/26566)
&& echo "-Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl" >> /usr/local/snap/bin/gpt.vmoptions \
# Remove heavy script
&& rm "$snap_script"

The update script is:

#!/bin/bash

snap --nosplash --nogui --modules --update-all 2>&1 | while read -r line; do
    echo "$line"
    [ "$line" = "updates=0" ] && sleep 2 && pkill -TERM -f "snap/jre/bin/java"
done

Basically the difference between SNAP 8.0 and 9.0 is very thin, the 8.0 script is called esa-snap_sentinel_unix_8_0.sh when the one for 9.0 is esa-snap_sentinel_unix_9_0_0.sh

JaumePrimer · 2022-09-14T08:38:39Z

Thanks @remi-braun , that's a big help!
I have most of the image building but running snap --nosplash --nogui --modules --update-all 2>&1 keeps on exiting with error code 33. Looking around, it sounds like it might be a problem with the server being busy. I've retried at different times of the day but the error is persisting. Is this a problem you've had? Am I just unlucky with my timing?

remi-braun · 2022-09-14T08:51:42Z

With SNAP 8.0 or 9.0 ? I've switched to 9.0 and I just reconstruted an image which worked just fine 😅

JaumePrimer · 2022-09-16T10:13:34Z

Working with SNAP 9.0.

I've just found the problem. Well, almost. I'm using a Mac M1 processor and couldn't install SNAP 9.0 in docker due to architectural differences. I instead used google cloud builds instead of building locally with docker as this way it will build on a Linux machine. This worked fine for the install, but failed on the update.

I ended up splitting the build in two:

install SNAP with google cloud build (on a Linux)
using the output from (1) as the base, update SNAP using docker build and a second Dockerfile locally on my Mac

I can't explain why the update didn't work on google, and the error codes gave no information. But in the end, it worked...

Thanks for the help, @remi-braun !

remi-braun · 2022-09-16T10:17:12Z

Happy to here that!
For now, it seems Mac M1 are still hard to work with

FYI: EOReader is currently not tested on Macs, so if you encounter any error outside your Docker I may not be able to correct them 😅

impresionista added the enhancement New feature or request label Aug 8, 2022

remi-braun changed the title ~~Issues reading SAOCOM data~~ SAOCOM L1A products are already calibrated Aug 9, 2022

impresionista closed this as completed Aug 18, 2022

remi-braun added a commit that referenced this issue Aug 22, 2022

- DOC: Add the need of using SNAP 8.0 up-to-date or SNAP 9.0 (#42)

c2f7f6a

bastiencyr pushed a commit to bastiencyr/eoreader that referenced this issue May 30, 2023

- DOC: Add the need of using SNAP 8.0 up-to-date or SNAP 9.0 (sertit#42)

93a51b5

bastiencyr pushed a commit that referenced this issue Feb 13, 2025

- DOC: Add the need of using SNAP 8.0 up-to-date or SNAP 9.0 (#42)

5d79a26

bastiencyr pushed a commit that referenced this issue Feb 13, 2025

- DOC: Add the need of using SNAP 8.0 up-to-date or SNAP 9.0 (#42)

c02d89c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SAOCOM L1A products are already calibrated #42

SAOCOM L1A products are already calibrated #42

impresionista commented Aug 8, 2022

remi-braun commented Aug 9, 2022

remi-braun commented Aug 9, 2022 •

edited

Loading

impresionista commented Aug 9, 2022

remi-braun commented Aug 9, 2022

remi-braun commented Aug 9, 2022

impresionista commented Aug 9, 2022

impresionista commented Aug 9, 2022 •

edited

Loading

remi-braun commented Aug 10, 2022

impresionista commented Aug 10, 2022

remi-braun commented Aug 11, 2022

impresionista commented Aug 11, 2022 •

edited

Loading

impresionista commented Aug 11, 2022

remi-braun commented Aug 11, 2022 •

edited

Loading

impresionista commented Aug 11, 2022

remi-braun commented Aug 11, 2022

impresionista commented Aug 18, 2022

remi-braun commented Aug 19, 2022

JaumePrimer commented Sep 13, 2022

remi-braun commented Sep 13, 2022 •

edited

Loading

JaumePrimer commented Sep 14, 2022

remi-braun commented Sep 14, 2022

JaumePrimer commented Sep 16, 2022 •

edited

Loading

remi-braun commented Sep 16, 2022

SAOCOM L1A products are already calibrated #42

SAOCOM L1A products are already calibrated #42

Comments

impresionista commented Aug 8, 2022

remi-braun commented Aug 9, 2022

remi-braun commented Aug 9, 2022 • edited Loading

impresionista commented Aug 9, 2022

remi-braun commented Aug 9, 2022

remi-braun commented Aug 9, 2022

impresionista commented Aug 9, 2022

impresionista commented Aug 9, 2022 • edited Loading

remi-braun commented Aug 10, 2022

impresionista commented Aug 10, 2022

remi-braun commented Aug 11, 2022

impresionista commented Aug 11, 2022 • edited Loading

impresionista commented Aug 11, 2022

remi-braun commented Aug 11, 2022 • edited Loading

impresionista commented Aug 11, 2022

remi-braun commented Aug 11, 2022

impresionista commented Aug 18, 2022

remi-braun commented Aug 19, 2022

JaumePrimer commented Sep 13, 2022

remi-braun commented Sep 13, 2022 • edited Loading

JaumePrimer commented Sep 14, 2022

remi-braun commented Sep 14, 2022

JaumePrimer commented Sep 16, 2022 • edited Loading

remi-braun commented Sep 16, 2022

remi-braun commented Aug 9, 2022 •

edited

Loading

impresionista commented Aug 9, 2022 •

edited

Loading

impresionista commented Aug 11, 2022 •

edited

Loading

remi-braun commented Aug 11, 2022 •

edited

Loading

remi-braun commented Sep 13, 2022 •

edited

Loading

JaumePrimer commented Sep 16, 2022 •

edited

Loading