Gen3 Monthly Release 2023.04 gen3.datacommons.io 1680087945

[PlanXCyborg]

Applying version 2023.04 to gen3.datacommons.io

[PlanXCyborg]

gen3.datacommons.io

❌ etlMapping.yaml - gitops.json mismatch

• Field error: Field [node_id] in explorerConfig.charts not found in etlMapping

[PlanXCyborg]

gen3.datacommons.io/manifest.json

Deployment changes

• fence
  • Requires a Fence DB migration (PXP-10541 Client credentials rotation fence#1068)
• metadata
  • Envs using AggMDS before must run the AggMDS sync job after updating to this version to migrate AggMDS reocrds syntax (HP-1044 Feat/aggmds vlmd metadata-service#90)
• peregrine
  • The pidgin service can be removed. Its functionalities are available at peregrine's /coremetadata endpoint (PXP-10708 Add coremetadata endpoint peregrine#198)
  • ASK AN ADMINISTRATOR TO DELETE THE PIDGIN DEPLOYMENT if you remove pidgin. This is NOT automatic (PXP-10708 Add coremetadata endpoint peregrine#198)
  • Peregrine does not rely on configuration files in the cloud-automation repository anymore. The Peregrine configuration files are in the Peregrine repository. (PXP-10163 Python 3.9 peregrine#191)
• pidgin
  • If you are seeing this message, you are deploying the Pidgin service after it has been deprecated. If you are using Peregrine version 3.2.0/2023.04 or newer, the Pidgin service can and should be removed. (PXP-10708 Deployment notes for release automation pidgin#48)
• sheepdog
  • Sheepdog does not rely on configuration files in the cloud-automation repository anymore. The Sheepdog configuration files are in the Sheepdog repository. (PXP-6245 PXP-9633 Poetry + Python 3.9 sheepdog#387)
• sower.container.image.pelican-export
  • The Pelican export job should now be deployed alongside the metadata-delete-expired-objects cronjob: add "metadata-delete-expired-objects": "quay.io/cdis/metadata-delete-expired-objects:<version>" to the versions block of the manifest, run gen3 kube-setup-metadata-delete-expired-objects-cronjob and grant the metadata-delete-expired-objects-job client access to (resource=/mds_gateway, method=access, service=mds_gateway) and (resource=/programs, method=delete, service=fence) in the user.yaml. (PXP-10558 Pelican export: store PFB file's _expires_at in MDS pelican#69)

Breaking changes

• indexd
  • Ensure you are using Fence>=2021.10. Remove deprecated GA4GH access endpoints. these have existed (and been used) from the Fence microservice since last year. See cloud automation change here (Revert "Revert "fix(migrate): improve acl->authz migration, remove de… indexd#338)
• metadata
  • Data format of response returned from /aggregate/metadata/guid/{guid} has changed, please refer to the PR description and/or API docs for details (HP-1044 Feat/aggmds vlmd metadata-service#90)
  • Data format of response returned from /aggregate/metadata/{name} has changed, please refer to the PR description and/or API docs for details (HP-1044 Feat/aggmds vlmd metadata-service#90)
• sower.container.image.download-indexd-manifest
  • update batch export job for MDS v3 AggMDS response syntax (fix: update for mds 3.0.0 sower-jobs#43)
• sower.container.image.get-dbgap-metadata
  • update batch export job for MDS v3 AggMDS response syntax (fix: update for mds 3.0.0 sower-jobs#43)
• sower.container.image.manifest-indexing
  • update batch export job for MDS v3 AggMDS response syntax (fix: update for mds 3.0.0 sower-jobs#43)
• sower.container.image.manifest-merging
  • update batch export job for MDS v3 AggMDS response syntax (fix: update for mds 3.0.0 sower-jobs#43)
• sower.container.image.metadata-manifest-ingestion
  • update batch export job for MDS v3 AggMDS response syntax (fix: update for mds 3.0.0 sower-jobs#43)
• sower.container.image.pelican-export
  • The Pelican export job now requires access to submit data in MDS. In cloud-automation setups, run kubectl delete secrets pelicanservice-g3auto, rm Gen3Secrets/g3auto/pelicanservice/config.json and gen3 kube-setup-pelicanjob. Then, grant the pelican-export-job client access to (resource=/mds_gateway, method=access, service=mds_gateway) in the user.yaml. (PXP-10558 Pelican export: store PFB file's _expires_at in MDS pelican#69)

[PlanXCyborg]

gen3.datacommons.io

❌ etlMapping.yaml - gitops.json mismatch

• Field error: Field [node_id] in explorerConfig.charts not found in etlMapping

[PlanXCyborg]

gen3.datacommons.io/manifest.json

Deployment changes

• fence
  • Requires a Fence DB migration (PXP-10541 Client credentials rotation fence#1068)
• metadata
  • Envs using AggMDS before must run the AggMDS sync job after updating to this version to migrate AggMDS reocrds syntax (HP-1044 Feat/aggmds vlmd metadata-service#90)
• peregrine
  • The pidgin service can be removed. Its functionalities are available at peregrine's /coremetadata endpoint (PXP-10708 Add coremetadata endpoint peregrine#198)
  • ASK AN ADMINISTRATOR TO DELETE THE PIDGIN DEPLOYMENT if you remove pidgin. This is NOT automatic (PXP-10708 Add coremetadata endpoint peregrine#198)
  • Peregrine does not rely on configuration files in the cloud-automation repository anymore. The Peregrine configuration files are in the Peregrine repository. (PXP-10163 Python 3.9 peregrine#191)
• pidgin
  • If you are seeing this message, you are deploying the Pidgin service after it has been deprecated. If you are using Peregrine version 3.2.0/2023.04 or newer, the Pidgin service can and should be removed. (PXP-10708 Deployment notes for release automation pidgin#48)
• sheepdog
  • Sheepdog does not rely on configuration files in the cloud-automation repository anymore. The Sheepdog configuration files are in the Sheepdog repository. (PXP-6245 PXP-9633 Poetry + Python 3.9 sheepdog#387)
• sower.container.image.pelican-export
  • The Pelican export job should now be deployed alongside the metadata-delete-expired-objects cronjob: add "metadata-delete-expired-objects": "quay.io/cdis/metadata-delete-expired-objects:<version>" to the versions block of the manifest, run gen3 kube-setup-metadata-delete-expired-objects-cronjob and grant the metadata-delete-expired-objects-job client access to (resource=/mds_gateway, method=access, service=mds_gateway) and (resource=/programs, method=delete, service=fence) in the user.yaml. (PXP-10558 Pelican export: store PFB file's _expires_at in MDS pelican#69)

Breaking changes

• indexd
  • Ensure you are using Fence>=2021.10. Remove deprecated GA4GH access endpoints. these have existed (and been used) from the Fence microservice since last year. See cloud automation change here (Revert "Revert "fix(migrate): improve acl->authz migration, remove de… indexd#338)
• metadata
  • Data format of response returned from /aggregate/metadata/guid/{guid} has changed, please refer to the PR description and/or API docs for details (HP-1044 Feat/aggmds vlmd metadata-service#90)
  • Data format of response returned from /aggregate/metadata/{name} has changed, please refer to the PR description and/or API docs for details (HP-1044 Feat/aggmds vlmd metadata-service#90)
• sower.container.image.download-indexd-manifest
  • update batch export job for MDS v3 AggMDS response syntax (fix: update for mds 3.0.0 sower-jobs#43)
• sower.container.image.get-dbgap-metadata
  • update batch export job for MDS v3 AggMDS response syntax (fix: update for mds 3.0.0 sower-jobs#43)
• sower.container.image.manifest-indexing
  • update batch export job for MDS v3 AggMDS response syntax (fix: update for mds 3.0.0 sower-jobs#43)
• sower.container.image.manifest-merging
  • update batch export job for MDS v3 AggMDS response syntax (fix: update for mds 3.0.0 sower-jobs#43)
• sower.container.image.metadata-manifest-ingestion
  • update batch export job for MDS v3 AggMDS response syntax (fix: update for mds 3.0.0 sower-jobs#43)
• sower.container.image.pelican-export
  • The Pelican export job now requires access to submit data in MDS. In cloud-automation setups, run kubectl delete secrets pelicanservice-g3auto, rm Gen3Secrets/g3auto/pelicanservice/config.json and gen3 kube-setup-pelicanjob. Then, grant the pelican-export-job client access to (resource=/mds_gateway, method=access, service=mds_gateway) in the user.yaml. (PXP-10558 Pelican export: store PFB file's _expires_at in MDS pelican#69)