`products/{identifier}` missing properties object in application/json response #155

jordanpadams · 2022-02-03T23:55:34Z

🐛 Describe the bug

missing properties object in application/json response (maybe others?)

📜 To Reproduce

When querying products/ endpoint, I get:

...
  "data": [
    {
      "id": "urn:nasa:pds:orex.ovirs::10.0",
      "type": "Product_Bundle",
      "title": "Origins, Spectral Interpretation, Resource Identification, Security, Regolith Explorer (OSIRIS-REx): Visible and InfraRed Spectrometer (OVIRS) Bundle",
      "start_date_time": "2016-09-08T23:05:00Z",
      "stop_date_time": "3000-01-01T00:00:00.000Z",
      "investigations": [
        {
          "id": "urn:nasa:pds:context:investigation:mission.orex",
          "href": "http://pds-gamma.jpl.nasa.gov/api/products/urn:nasa:pds:context:investigation:mission.orex"
        }
      ],
      "observing_system_components": [
        {
          "id": "urn:nasa:pds:context:instrument_host:spacecraft.orex",
          "href": "http://pds-gamma.jpl.nasa.gov/api/products/urn:nasa:pds:context:instrument_host:spacecraft.orex"
        },
        {
          "id": "urn:nasa:pds:context:instrument:ovirs.orex",
          "href": "http://pds-gamma.jpl.nasa.gov/api/products/urn:nasa:pds:context:instrument:ovirs.orex"
        }
      ],
      "targets": [
        {
          "id": "urn:nasa:pds:context:target:asteroid.101955_bennu",
          "href": "http://pds-gamma.jpl.nasa.gov/api/products/urn:nasa:pds:context:target:asteroid.101955_bennu"
        }
      ],
      "metadata": {
        "update_date_time": "2018-05-31T00:00:00Z",
        "version": "10.0",
        "label_url": "/data/pds4/test-data/registry/orex.ovirs/bundle_ovirs.xml"
      },
      "properties": {
        "pds:Modification_Detail.pds:description": [
          "Creation. This Product_Bundle label was written by OREX personnel, for the initial draft ovirs bundle.",
        ...
        ],
        "pds:Target_Identification.pds:alternate_designation": [
          "1999 RQ36"
        ],
...

But with products/urn:nasa:pds:orex.ovirs::10.0 (missing properties):

{
  "id": "urn:nasa:pds:orex.ovirs::10.0",
  "type": "Product_Bundle",
  "title": "Origins, Spectral Interpretation, Resource Identification, Security, Regolith Explorer (OSIRIS-REx): Visible and InfraRed Spectrometer (OVIRS) Bundle",
  "start_date_time": "2016-09-08T23:05:00Z",
  "stop_date_time": "3000-01-01T00:00:00.000Z",
  "investigations": [
    {
      "id": "urn:nasa:pds:context:investigation:mission.orex",
      "href": "http://pds-gamma.jpl.nasa.gov/api/products/urn:nasa:pds:context:investigation:mission.orex"
    }
  ],
  "observing_system_components": [
    {
      "id": "urn:nasa:pds:context:instrument_host:spacecraft.orex",
      "href": "http://pds-gamma.jpl.nasa.gov/api/products/urn:nasa:pds:context:instrument_host:spacecraft.orex"
    },
    {
      "id": "urn:nasa:pds:context:instrument:ovirs.orex",
      "href": "http://pds-gamma.jpl.nasa.gov/api/products/urn:nasa:pds:context:instrument:ovirs.orex"
    }
  ],
  "targets": [
    {
      "id": "urn:nasa:pds:context:target:asteroid.101955_bennu",
      "href": "http://pds-gamma.jpl.nasa.gov/api/products/urn:nasa:pds:context:target:asteroid.101955_bennu"
    }
  ],
  "metadata": {
    "update_date_time": "2018-05-31T00:00:00Z",
    "version": "10.0",
    "label_url": "/data/pds4/test-data/registry/orex.ovirs/bundle_ovirs.xml"
  }
}

🕵️ Expected behavior

properties block should be included in products/{identifier} response as well.

🦄 Related requirements

⚙️ Engineering Details

We should definitely test the other endpoints to make sure this is working correctly as well.

The text was updated successfully, but these errors were encountered:

al-niessner · 2022-02-07T17:46:16Z

@jordanpadams @tloubrieu-jpl

Testing with old version? When I test on PR NASA-PDS/registry-api#82 I get (hopefully I got cut-n-paste correct):

  "properties": {
    "pds:Modification_Detail.pds:description": [
      "Initial PDS4 version."
    ],
    "lid": [
      "urn:nasa:pds:izenberg_pdart14_meap:document:ns_inst"
    ],
    "pds:Modification_Detail.pds:modification_date": [
      "2016-03-16T00:00:00Z"
    ],
    "pds:Document.pds:publication_date": [
      "2016-03-16T00:00:00Z"
    ],
    "ref_lid_instrument": [
      "urn:nasa:pds:context:instrument:ns.mess"
    ],
    "ops:Label_File_Info.ops:md5_checksum": [
      "a8d09cca0a01728db50c15052c2736cf"
    ],
    "vid": [
      "1.0"
    ],
    "product_class": [
      "Product_Document"
    ],
    "pds:Document_File.pds:document_standard_id": [
      "PDF/A"
    ],
    "pds:Document_Edition.pds:edition_name": [
      "MESSENGER NS Instrument Description"
    ],
    "pds:Document_File.pds:file_name": [
      "ns_inst.pdf"
    ],
    "pds:Observing_System_Component.pds:type": [
      "Host",
      "Instrument"
    ],
    "ops:Label_File_Info.ops:file_ref": [
      "/var/local/harvest/archive/document/ns_inst.xml"
    ],
    "ops:Harvest_Info.ops:harvest_date_time": [
      "2022-01-24T20:45:05.175545Z"
    ],
    "ops:Data_File_Info.ops:mime_type": [
      "application/pdf"
    ],
    "pds:Target_Identification.pds:name": [
      "Mercury"
    ],
    "ops:Data_File_Info.ops:md5_checksum": [
      "8103f20c13a3c321dac4a193aba19d16"
    ],
    "pds:Identification_Area.pds:title": [
      "MESSENGER Neutron Spectrometer (NS) Description"
    ],
    "pds:Observing_System_Component.pds:name": [
      "MESSENGER",
      "NS"
    ],
    "pds:Investigation_Area.pds:name": [
      "MESSENGER"
    ],
    "ops:Label_File_Info.ops:creation_date_time": [
      "2022-01-24T20:08:23Z"
    ],
    "pds:Identification_Area.pds:product_class": [
      "Product_Document"
    ],
    "ref_lid_instrument_host": [
      "urn:nasa:pds:context:instrument_host:spacecraft.mess"
    ],
    "pds:Document.pds:description": [
      "Description of the MESSENGER Neutron Spectrometer (NS)."
    ],
    "description": [
      "Description of the MESSENGER Neutron Spectrometer (NS).",
      "Initial PDS4 version.",
      "Description of the MESSENGER Neutron Spectrometer (NS)."
    ],
    "pds:Citation_Information.pds:publication_year": [
      "2016"
    ],
    "ops:Harvest_Info.ops:node_name": [
      "PSA"
    ],
    "title": [
      "MESSENGER Neutron Spectrometer (NS) Description"
    ],
    "ops:Data_File_Info.ops:file_ref": [
      "/var/local/harvest/archive/document/ns_inst.pdf"
    ],
    "_package_id": [
      "2ef2bf5a-6afc-4bfa-9065-c2d4078825d1"
    ],
    "ops:Data_File_Info.ops:creation_date_time": [
      "2022-01-24T20:08:23Z"
    ],
    "pds:Identification_Area.pds:version_id": [
      "1.0"
    ],
    "pds:Citation_Information.pds:description": [
      "Description of the MESSENGER Neutron Spectrometer (NS)."
    ],
    "pds:Internal_Reference.pds:reference_type": [
      "document_to_investigation",
      "is_instrument_host",
      "is_instrument",
      "document_to_target"
    ],
    "pds:Identification_Area.pds:information_model_version": [
      "1.14.0.0"
    ],
    "pds:Internal_Reference.pds:lid_reference": [
      "urn:nasa:pds:context:investigation:mission.messenger",
      "urn:nasa:pds:context:instrument_host:spacecraft.mess",
      "urn:nasa:pds:context:instrument:ns.mess",
      "urn:nasa:pds:context:target:planet.mercury"
    ],
    "pds:Identification_Area.pds:logical_identifier": [
      "urn:nasa:pds:izenberg_pdart14_meap:document:ns_inst"
    ],
    "ops:Data_File_Info.ops:file_size": [
      "138172"
    ],
    "pds:Observing_System.pds:name": [
      "MESSENGER"
    ],
    "ref_lid_investigation": [
      "urn:nasa:pds:context:investigation:mission.messenger"
    ],
    "lidvid": [
      "urn:nasa:pds:izenberg_pdart14_meap:document:ns_inst::1.0"
    ],
    "pds:Modification_Detail.pds:version_id": [
      "1.0"
    ],
    "ops:Label_File_Info.ops:file_size": [
      "3589"
    ],
    "ops:Data_File_Info.ops:file_name": [
      "ns_inst.pdf"
    ],
    "ops:Label_File_Info.ops:file_name": [
      "ns_inst.xml"
    ],
    "pds:Document_Edition.pds:files": [
      "1"
    ],
    "pds:Target_Identification.pds:type": [
      "Planet"
    ],
    "ref_lid_target": [
      "urn:nasa:pds:context:target:planet.mercury"
    ],
    "pds:Document_Edition.pds:language": [
      "English"
    ],
    "pds:Investigation_Area.pds:type": [
      "Mission"
    ]
  }

jordanpadams · 2022-02-07T17:53:37Z

@al-niessner copy that. I will wait to see if @tloubrieu-jpl @ramesh-maddegoda or @tdddblog can test and confirm this?

al-niessner · 2022-02-07T17:55:57Z

If it is, we can just add it to addressed in issues in the PR.

ramesh-maddegoda · 2022-02-08T01:46:49Z

@jordanpadams and @al-niessner, in the docker compose related work, I am directly pulling the following image of the Registry API.

# Docker image of the Registry API
REG_API_IMAGE=nasapds/registry-api-service

I get the following output with that. May be nasapds/registry-api-service image does not have the latest version.

http://localhost:8080/products/urn%3Anasa%3Apds%3Ainsight_rad%3A%3A2.1

JSON as text
{
  "id" : "urn:nasa:pds:insight_rad::2.1",
  "type" : "Product_Bundle",
  "title" : "Mars InSight Lander Radiometer Data Archive",
  "start_date_time" : "2018-05-05T00:00:00Z",
  "stop_date_time" : "3000-01-01T00:00:00.000Z",
  "investigations" : [ {
    "id" : "urn:nasa:pds:context:investigation:mission.insight",
    "href" : "http://localhost:8080/products/urn:nasa:pds:context:investigation:mission.insight"
  } ],
  "observing_system_components" : [ {
    "id" : "urn:nasa:pds:context:instrument_host:spacecraft.insight",
    "href" : "http://localhost:8080/products/urn:nasa:pds:context:instrument_host:spacecraft.insight"
  }, {
    "id" : "urn:nasa:pds:context:instrument:radiometer.insight",
    "href" : "http://localhost:8080/products/urn:nasa:pds:context:instrument:radiometer.insight"
  } ],
  "targets" : [ {
    "id" : "urn:nasa:pds:context:target:planet.mars",
    "href" : "http://localhost:8080/products/urn:nasa:pds:context:target:planet.mars"
  } ],
  "metadata" : {
    "update_date_time" : "2018-02-01T00:00:00Z",
    "version" : "2.1",
    "label_url" : "/data/test-data/bundle_insight_rad.xml"
  }
}

jordanpadams · 2022-02-08T02:50:36Z

@ramesh-maddegoda so that docker image is from what is actually in DockerHub? If so, then I think that is an older version than the current main branch.

ramesh-maddegoda · 2022-02-08T02:57:40Z

@jordanpadams , that is correct. The docker compose is expected to pull from our DockerHub images (when the image is not available locally), to simply the deployment, without having to build locally.

jordanpadams · 2022-02-08T03:04:21Z

@ramesh-maddegoda copy. could you try running the testing use case we are talking about here using one of the other docker compose use cases, where we want to deploy everything but deploy the API built from source. I think this is currently called something like api-dev? @tloubrieu-jpl can probably clarify since I know he uses it.

ramesh-maddegoda · 2022-02-08T17:34:40Z

@jordanpadams and @al-niessner, I tried to build the the main branch of registry-api with mvn clean install, but it fails with the following errors.

Caused by: com.github.jknack.handlebars.HandlebarsException: /handlebars/JavaSpring/pojo.mustache:2:6: java.lang.reflect.InaccessibleObjectException: Unable to make public boolean java.util.Collections$EmptyMap.isEmpty() accessible: module java.base does not "opens java.util" to unnamed module @553da911
    /handlebars/JavaSpring/pojo.mustache:2:6

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] gov.nasa.pds:registry-api-model 0.5.0-SNAPSHOT ..... FAILURE [  1.265 s]
[INFO] PDS API Search Query Lexer 1.1.0-SNAPSHOT .......... SKIPPED
[INFO] gov.nasa.pds.api.registry-service 0.5.0-SNAPSHOT ... SKIPPED
[INFO] gov.nasa.pds:registry 0.5.0-SNAPSHOT ............... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.794 s
[INFO] Finished at: 2022-02-08T12:25:30-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.swagger.codegen.v3:swagger-codegen-maven-plugin:3.0.20:generate (generate-swagger-javacode) on project registry-api-model: Code generation failed. See above for the full exception. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

Therefore, I tried to use the jar file available at registry-api v0.5.0-SNAPSHOT. This is the latest jar file available at https://github.com/NASA-PDS/registry-api/releases .

Note: I am using the test data downloaded from the following URL.

# URL to download the test data to Harvest (only required, if executing with test data)
TEST_DATA_URL=https://pds-gamma.jpl.nasa.gov/data/pds4/test-data/registry/urn-nasa-pds-insight_rad.tar.gz

But with above v0.5.0-SNAPSHOT jar file also, it shows a similar output as follows.

JSON as text
{
  "id" : "urn:nasa:pds:insight_rad::2.1",
  "type" : "Product_Bundle",
  "title" : "Mars InSight Lander Radiometer Data Archive",
  "start_date_time" : "2018-05-05T00:00:00Z",
  "stop_date_time" : "3000-01-01T00:00:00.000Z",
  "investigations" : [ {
    "id" : "urn:nasa:pds:context:investigation:mission.insight",
    "href" : "http://localhost:8080/products/urn:nasa:pds:context:investigation:mission.insight"
  } ],
  "observing_system_components" : [ {
    "id" : "urn:nasa:pds:context:instrument_host:spacecraft.insight",
    "href" : "http://localhost:8080/products/urn:nasa:pds:context:instrument_host:spacecraft.insight"
  }, {
    "id" : "urn:nasa:pds:context:instrument:radiometer.insight",
    "href" : "http://localhost:8080/products/urn:nasa:pds:context:instrument:radiometer.insight"
  } ],
  "targets" : [ {
    "id" : "urn:nasa:pds:context:target:planet.mars",
    "href" : "http://localhost:8080/products/urn:nasa:pds:context:target:planet.mars"
  } ],
  "metadata" : {
    "update_date_time" : "2018-02-01T00:00:00Z",
    "version" : "2.1",
    "label_url" : "/data/bundle_insight_rad.xml"
  }
}

jordanpadams · 2022-02-08T20:49:30Z

thanks @ramesh-maddegoda so looks like this may have been fixed per NASA-PDS/registry-api#82. will update that PR comment to make sure it is tested.

tloubrieu-jpl · 2022-02-08T21:00:57Z

@ramesh-maddegoda I was able to build. I need to check what local setting might impact you or me.

al-niessner · 2022-02-08T21:10:25Z

@jordanpadams and @al-niessner, I tried to build the the main branch of registry-api with mvn clean install, but it fails with the following errors.

Caused by: com.github.jknack.handlebars.HandlebarsException: /handlebars/JavaSpring/pojo.mustache:2:6: java.lang.reflect.InaccessibleObjectException: Unable to make public boolean java.util.Collections$EmptyMap.isEmpty() accessible: module java.base does not "opens java.util" to unnamed module @553da911
    /handlebars/JavaSpring/pojo.mustache:2:6

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] gov.nasa.pds:registry-api-model 0.5.0-SNAPSHOT ..... FAILURE [  1.265 s]
[INFO] PDS API Search Query Lexer 1.1.0-SNAPSHOT .......... SKIPPED
[INFO] gov.nasa.pds.api.registry-service 0.5.0-SNAPSHOT ... SKIPPED
[INFO] gov.nasa.pds:registry 0.5.0-SNAPSHOT ............... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.794 s
[INFO] Finished at: 2022-02-08T12:25:30-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.swagger.codegen.v3:swagger-codegen-maven-plugin:3.0.20:generate (generate-swagger-javacode) on project registry-api-model: Code generation failed. See above for the full exception. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

@ramesh-maddegoda

The build failure worries me. If the main branch fails like that, then how can we trust the latest built JAR file? I do docker builds of this stuff all the time, and mvn clean install works perfectly every time. I am using Dockerfile.local to give you something to compare.

al-niessner · 2022-02-08T21:21:40Z

@tloubrieu-jpl @ramesh-maddegoda @jordanpadams

[INFO] gov.nasa.pds:registry-api-model 0.5.0-SNAPSHOT ..... FAILURE [ 1.265 s]

I believe that error is because nobody has run maven in NASA-PDS/registry-api to install the artifacts in the maven repo sonatype.org. They did get renamed in one of my various branches.

jordanpadams · 2022-02-09T14:48:02Z

@al-niessner @ramesh-maddegoda @tloubrieu-jpl I just removed my local maven repo and did a fresh mvn clean install and everything installed correctly on main?

@ramesh-maddegoda what version of Java and Maven are you running? Maybe it was just some funky thing with sonatype when you ran it yesterday?

ramesh-maddegoda · 2022-02-09T17:02:40Z

@jordanpadams,

Java version:

openjdk 11.0.12 2021-07-20
OpenJDK Runtime Environment Homebrew (build 11.0.12+0)
OpenJDK 64-Bit Server VM Homebrew (build 11.0.12+0, mixed mode)

Maven version:

Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
Maven home: /usr/local/Cellar/maven/3.8.4/libexec
Java version: 17.0.1, vendor: Homebrew, runtime: /usr/local/Cellar/openjdk/17.0.1_1/libexec/openjdk.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "11.6.2", arch: "x86_64", family: "mac"

tloubrieu-jpl · 2022-02-09T20:12:43Z

Hi @ramesh-maddegoda ,

that worked for me as well with a fresh local maven repository (~/.m2).

java -version    
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)

% mvn --version
Apache Maven 3.8.3 (ff8e977a158738155dc465c6a97ffaf31982d739)
Maven home: /Users/loubrieu/tools/apache-maven-3.8.3
Java version: 11.0.2, vendor: Oracle Corporation, runtime: /Users/loubrieu/tools/jdk-11.0.2.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.16", arch: "x86_64", family: "mac"

ramesh-maddegoda · 2022-02-09T21:35:35Z

Hi @tloubrieu-jpl, I think, then it can be a specific issue on my laptop. I got both JDK and maven from Homebrew. Will troubleshoot that.

To check the specific issue on this ticket, is it possible to push the latest docker image of registry-api to Dockerhub?

jordanpadams added bug Something isn't working B12.1 s.high labels Feb 3, 2022

jordanpadams assigned al-niessner and tloubrieu-jpl Feb 3, 2022

jordanpadams mentioned this issue Feb 8, 2022

Issue 81: touch up XML some NASA-PDS/registry-api#82

Merged

tloubrieu-jpl closed this as completed in NASA-PDS/registry-api#82 Feb 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`products/{identifier}` missing properties object in application/json response #155

`products/{identifier}` missing properties object in application/json response #155

jordanpadams commented Feb 3, 2022

al-niessner commented Feb 7, 2022

jordanpadams commented Feb 7, 2022

al-niessner commented Feb 7, 2022

ramesh-maddegoda commented Feb 8, 2022

jordanpadams commented Feb 8, 2022

ramesh-maddegoda commented Feb 8, 2022

jordanpadams commented Feb 8, 2022

ramesh-maddegoda commented Feb 8, 2022

jordanpadams commented Feb 8, 2022

tloubrieu-jpl commented Feb 8, 2022

al-niessner commented Feb 8, 2022

al-niessner commented Feb 8, 2022

jordanpadams commented Feb 9, 2022

ramesh-maddegoda commented Feb 9, 2022

tloubrieu-jpl commented Feb 9, 2022

ramesh-maddegoda commented Feb 9, 2022

products/{identifier} missing properties object in application/json response #155

products/{identifier} missing properties object in application/json response #155

Comments

jordanpadams commented Feb 3, 2022

🐛 Describe the bug

📜 To Reproduce

🕵️ Expected behavior

🦄 Related requirements

⚙️ Engineering Details

al-niessner commented Feb 7, 2022

jordanpadams commented Feb 7, 2022

al-niessner commented Feb 7, 2022

ramesh-maddegoda commented Feb 8, 2022

jordanpadams commented Feb 8, 2022

ramesh-maddegoda commented Feb 8, 2022

jordanpadams commented Feb 8, 2022

ramesh-maddegoda commented Feb 8, 2022

jordanpadams commented Feb 8, 2022

tloubrieu-jpl commented Feb 8, 2022

al-niessner commented Feb 8, 2022

al-niessner commented Feb 8, 2022

jordanpadams commented Feb 9, 2022

ramesh-maddegoda commented Feb 9, 2022

tloubrieu-jpl commented Feb 9, 2022

ramesh-maddegoda commented Feb 9, 2022

`products/{identifier}` missing properties object in application/json response #155

`products/{identifier}` missing properties object in application/json response #155