Gen3 Monthly Release 2022.10 dcf-interop.kidsfirstdrc.org 1666320713

[PlanXCyborg]

Applying version 2022.10 to dcf-interop.kidsfirstdrc.org

[PlanXCyborg]

dcf-interop.kidsfirstdrc.org/manifest.json

⚠️ Services on branch

• arranger-dashboard
• arranger-adminapi

Deployment changes

• arborist
  • Replace Travis with a GitHub workflow for CI (Fix lint CI + replace travis with GH workflow arborist#150)
  • Needs Fence later than 4.22.2 or 2020.08 (which introduces the new JWT scopes claim). ((PXP-6617): remove scopes from jwt aud claim arborist#130)
  • Assuming any commons running Arborist < 2.3 was using user sync only as AuthZ provider (chore(authz): set authz_provider in migration arborist#111)
  • One-off: Update authz_provider to the correct values (Feat/authz provider arborist#109)
  • Database deployment (feat/db: add database persistence arborist#42)
  • See migrations folder for database schema and setup utilities (feat/db: add database persistence arborist#42)
• fence
  • Configure the ALLOWED_DATA_UPLOAD_BUCKETS setting to allow users to upload to buckets other than DATA_UPLOAD_BUCKET (PXP-10361 Allow specifying the upload bucket fence#1051)
  • B/c the default session expiration changed from 30 minutes to 15 minutes to comply with FedRAMP, you MUST ensure that an instance that should comply with FedRAMP does NOT override this fence configuration (e.g. it should be 15 minutes) SESSION_TIMEOUT: 900 (feat(session): change default to 15 minutes [PPS-122] fence#1040)
  • Requires a Fence DB migration (Client.redirect_uri is now optional) (PXP-10113 Client credentials grant fence#1033)
  • Requires database migration for a new table to Fence, google Access expiration to add new expires column, and to create iss_sub_pair_to_user table Update manifest.json #968 updated BRAIN production data model to 1.0.0 #963 feat(stage-tiered-access): Updated portal versions to fix tiered acce… #973 (Feat/passport fence#964)
  • Requires Arborist >= 2022.04 OR >=3.4.0 (PXP-9480): Support merge param for PUT /resource endpoint arborist#143 (Feat/passport fence#964)
  • To enable Passports -> DRS in an environment you must update Fence Configuration to at a minimum specify GA4GH_PASSPORTS_TO_DRS_ENABLED: true (only do this after coordinating with the environment owner as this has several requirements before it can be enabled, including the need to use Indexd's authz field in ALL records rather than acl) (Feat/passport fence#964)
  • Requires a Fence DB migration (chore(migration): add authz column to project fence#1003)
  • Requires a database migration to create iss_sub_pair_to_user table ((PXP-8682): Add visa validation and sync for data download fence#973)
  • Google Access expiration requires a database migration to add new expires column! (feat(storage_expiration): add expires to db for Google access, create… fence#968)
  • Requires database migration. Adding a new table to Fence. ((PXP-8679): Add new db table and update login to add entries fence#963)
  • Requires these cloud-automation updates to revproxy: (PXP-7565): Exclude Fence register-user blueprint from CSRF check cloud-automation#1580 ((PXP-7565): User registration, plus CSRF changes fence#906)
  • By default Fence will allow only visas issued by RAS staging, RAS prod, and itself; if a different set of allowed visa issuers is needed, edit the GA4GH_VISA_ISSUER_ALLOWLIST config value. Otherwise, no action needed. ((PXP-6339): RAS visa validation fence#933)
  • /link/google will DENY requests with "redirect" param values that aren't in the "allowed redirects" for OAuth2/OIDC Clients in Fence (You have to add any valid redirects to the client before deploying this) ((PXP-8286): fix(google): add redirect validation to /link/google fence#927)
  • For existing deployments with "ENABLE_AUDIT_LOGS" turned ON, to DISABLE pushing audit logs to an AWS SQS, the configuration "PUSH_AUDIT_LOGS_CONFIG.type" must be changed to "api" (default: "aws_sqs"). If audit-service has never been deployed, no action required. (PXP-7805 Push audit logs to an AWS SQS fence#923)
  • To ENABLE pushing audit logs to an AWS SQS (REQUIRES audit-service 1.0.0 or more recent), create/update the configuration file's "PUSH_AUDIT_LOGS_CONFIG" field ("type: aws_sqs" (default), and the SQS URL and region can be copied from the audit-service configuration - or run gen3 sqs info $(gen3 api safe-name audit-sqs)), see https://github.com/uc-cdis/fence/blob/fec3dd937d68384e86b2ed6f9b53daf0026c2df9/fence/config-default.yaml#L632-L636. Then run kubectl delete secret fence-config and gen3 kube-setup-fence (PXP-7805 Push audit logs to an AWS SQS fence#923)
  • This commit includes patches to allow old style refresh tokens and API keys to pass new style validation, while issuing new style refresh tokens and API keys. Since refresh tokens and API keys have a default TTL of 30 days, stay on this tag for 30 days if possible. (PXP-6617 Remove scopes from aud claim in tokens fence#839)
  • Old access tokens will NOT work once this is deployed so active sessions may be temporarily interrupted. (PXP-6617 Remove scopes from aud claim in tokens fence#839)
  • Audit logs for user logins and data downloads can be enabled by deploying the audit-service and adding setting ENABLE_AUDIT_LOGS to the fence config (see https://github.com/uc-cdis/fence/blob/1c850d07e54f8527dbde182fdd9dd593d18f660e/fence/config-default.yaml#L586-L588) (PXP-7447 audit-service integration fence#885)
  • REQUIRES A FENCE DB MIGRATION, since with this update there is now an additional column in the authorization_code table "refresh_token_expires_in" (Feat/refresh expires in fence#848)
  • Commons configured with RAS staging login must add the "discovery_url" fence config, because it now defaults to RAS prod (Fix/ras url fence#824)
  • If the fence config already has [CLIENT_, USER_, SESSION_]ALLOWED_SCOPES fields, and GA4GH AAI integration is desired, then add ga4gh_passport_v1 to all those lists. If default config values are being used then no change required. (PXP-6406 Feat/ga4gh scope fence#799)
  • To use the new Cognito integration, apply in fence config the changes made to config-default.yaml. Otherwise no action required. (PXP-5613 Feat/cognito fence#767)
  • In fence-config, the region field of S3_BUCKETS is now required: if not specified, Fence will print a warning and try to call GetBucketLocation. Fence will error at startup if that doesn't work. (Fix/make region required for s3 bucket config fence#749)
  • Fence does NOT handle c999 the same as before. If using dbgap sync w/ consent code parsing and indexd records have acls with just phsid to represent objects someone w/ c999 should be able to see, you will need to ensure that the records also have the phsid.consent_code as well or users will lose access (Feat/dbgap options fence#702)
  • Fence has a new configuration to support more dbGaP syncing options: if you require "Common Exchange Area" data access, review the new configuration and supply necessary values (Feat/dbgap options fence#702)
  • Fence config ENABLED_IDENTITY_PROVIDERS section deprecated and replaced by LOGIN_OPTIONS section to allow enabling several login options for the same provider (backwards compatible) (Feat/incommon updates fence#703)
  • Requires manual work in Arborist DB to fill in the empty authz_provider fields (Synapse Integration fence#696)
  • new cloud automation job for database migration feat(jobs): fence db create job cloud-automation#977 (feat(fence-create): new migrate command to migrate the database fence#694)
  • configuration added for requester pays Google Buckets. if this is needed, you'll need to update configuration. default config does not support requester pays bucket access (Feat/requester pays fence#668)
• indexd
  • Include example secrets to load PostgreSQL settings for indexd (Add files for psql ssl support indexd#317)
  • When running gen3authz, backoff package needed separate poetry run pip install (PXP-6824: return 401 errors from Arborist indexd#304)
  • Indexd's blank record endpoints now allow direct iteration from a user with their access token but requires the file upload arborist role to be modified so the service is not just fence but instead * so indexd works too. see https://github.com/uc-cdis/commons-users/pull/908/files for example (Update blueprint.py indexd#275)
  • Add script to run for filling out records' authz fields based on the acl fields. (chore(authz-migration): add migration script indexd#218)
• peregrine
  • ARBORIST_URL environment variable replaces broken default ARBORIST config variable (fix(arborist-url) peregrine#142)
  • Optional configuration variable ARBORIST (arborist base URL) (Feat/arborist peregrine#138)
  • Requires Arborist to be deployed (Feat/arborist peregrine#138)
  • look for INDEX_CLIENT instead of SIGNPOST in app.config (Remove signpost peregrine#135)
  • get INDEX_CLIENT_HOST env var instead of SIGNPOST_HOST (Remove signpost peregrine#135)
• pidgin
  • added required configuration variable "API_HEALTH_URL" for status endpoint (Fix/_status pidgin#31)
  • use the latest version of the python 3 base image (Chore/bump image pidgin#29)
  • The Dockerfile now uses an Alpine image (feat(alpine-docker): use alpine image pidgin#24)
• portal
  • Added new optional field stridesPortalURL in portal config which can display a link to the STRIDES admin portal on workspace page (HP-708 Feat/multiple paymodels data-portal#978)
  • For decoupled workspace, must work with Hatchery 1.0.0+ (HP-708 Feat/multiple paymodels data-portal#978)
  • See gitops configuration options for tabbed metadata detail vs traditional (Feat/disco tabs data-portal#983)
  • The pay model support depends on this PR: PXP-8779 HP-347 Feat/ecs hatchery#38 (PXP-8520 Feat/paymodels data-portal#900)
  • set "showExternalLoginsOnProfile": true for showing external logins on the profile page (Feat/profile fed auth data-portal#934)
  • Discovery: introduced a new config section discoveryConfig.features.search.tagSearchDropdown, see portal-config.md for more info (HP-555 Feat/disco tag dropdown viewer data-portal#927)
  • To enable this check, set features.exportToWorkspace.verifyExternalLogins flag to true in gitops (Check all WTS OIDCs are signed in before download/export to WS data-portal#921)
  • Misc: add option for bundle name: ecosystem, which is the same as heal bundle under the hood (Chore/discovery for more bundles data-portal#918)
  • Requires loginForDownload flag set to true in gitops (PXP-8448 - Register users for download data-portal#912)
  • User registration before download feature depends on (PXP-7565): User registration, plus CSRF changes fence#906 (PXP-8448 - Register users for download data-portal#912)
  • Requires the configuration of the batch-export sower job: Batch export sower job sower-jobs#29 in the environment's manifest. This job also has its own setup in cloud automation: Feat/batch export cloud-automation#1708 (Add UI components and polling for zip downloads data-portal#906)
  • 2 new fields in gitops.json: discoveryconfig.features.exportToWorkspace.[enableDownloadZip (bool) downloadZipButtonText (string)] (Add UI components and polling for zip downloads data-portal#906)
  • Requires PXP-8364 Fix/portal csp cloud-automation#1687, otherwsie Indexing page may not work properly (PXP-8364 fix: strict csp data-portal#893)
  • Portal config field discoveryConfig.features.exportToWorkspaceBETA has been renamed to discoveryConfig.features.exportToWorkspace (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • Note that there are several other new configurations introduced for the Discovery page, please refer to this diff (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • New GEN3_BUNDLE option heal introduced for buliding a HEAL bundle (HP-255 HEAL UI updates with adaptations to ecosystems UI data-portal#876)
  • To enable DD RUM, must create a portal-datadog-config k8s secret with datadog_application_id and datadog_client_token in it, see DEVOPS-62 feat: add datadog to portal cloud-automation#1651 (DEVOPS-62 Feat/datadog data-portal#881)
  • envs with customized color schemes no longer need to overwrite dictionary button color in their gitops.css ((PXP-6570): refactor/dictionaryBtnColor: change dictionary btn to use defined global colors data-portal#822)
  • New optional config property, tierAccessLevel, in each tab of the explorerConfig block in the portal config. Used to enable index-scoped tiered-access, which is only available starting with Guppy server version 0.11.0. See README for more information. (Feat/index scoped tiered access data-portal#802)
  • To deploy Discovery page: ((PXP-7402) Discovery page data-portal#803)
  • "featureFlags": { "discovery": true } must be set in the portal config. ((PXP-7402) Discovery page data-portal#803)
  • If discoveryConfig.features.authorization.enabled: true, "useArboristUI": true must be set in the portal config. ((PXP-7402) Discovery page data-portal#803)
  • "discoveryConfig": <DiscoveryConfig> must be set in the portal config. DiscoveryConfig documentation can be found at https://github.com/uc-cdis/data-portal/blob/master/docs/portal_config.md ((PXP-7402) Discovery page data-portal#803)
  • To upload data to Discovery page, data must be uploaded to environment's metadata-service instance with guid_type: "discovery_metadata". The metadata must all be under the top-level property gen3_discovery. (No documentation on this process yet, see this gist for right now https://gist.github.com/Avantol13/a53593837503fede2881f315198247c4) ((PXP-7402) Discovery page data-portal#803)
  • Added new entry GWASApp for analysisTools for this new GWAS app for VA. Beaware not to confuse with the old "GWAS PoC" app, which bears the entry vaGWAS (PXP-7296 PXP-7180 Feat/va gwas app data-portal#777)
  • antd modal now by default will use g3-color__base-blue as header background color, this is consistant with our color schemes for other UI components. If an env has customized color scheme, please override .ant-modal-header in gitops.css (PXP-7296 PXP-7180 Feat/va gwas app data-portal#777)
  • The new feature of Portal requires Guppy v0.10.0 ((PXP-6540): Feat/explorer download formats data-portal#780)
  • Added new button types in the Portal config. See portal_config.md for an example. ((PXP-6540): Feat/explorer download formats data-portal#780)
  • At time of writing, qa-midrc is only env using the new feature flag. These changes should be deployed alongside each other to production, but nothing will break if done separately. ((PXP-7097) Add optional alternate TopBar profile menu data-portal#759)
  • New value for GEN3_BUNDLE env var GEN3_BUNDLE=nct (PXP-6895 NIAID CT bundle data-portal#742)
  • New optional portal config variable injectDAPTag (default value is false). If set to true, will inject DAP tag script into page header (this is required for NCT PROD env only) (PXP-6895 NIAID CT bundle data-portal#742)
  • All envs should check if the dataExplorerConfig field in its portal config has ONLY arrangerConfig field but NO guppyConfig field (it is okay to have both). If so, please update dataExplorerConfig to use Guppy (PXP-6409 Remove arranger data-portal#743)
  • A new Export to PFB button can be optionally enabled in the config.json. It is disabled by default. The button requires at least pelican:2020.10 release. (Feat/export pfb to workspace data-portal#681)
  • Add GEN3_BUNDLE=covid19 as portal env var to enable covid bundle (Feat/covid bundle data-portal#737)
  • Limited File PFB Export requires Pelican >= 0.5.0, Tube >= 0.4.1 ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • Limited File PFB Export is enabled by setting fileExplorerConfig.enableLimitedFilePFBExport: { sourceNodeField: "source_node" } and by adding buttons of buttonType export-files (Export to Terra), export-files-to-pfb (Download PFB), or export-files-to-seven-bridges (Export to Seven Bridges) to fileExplorerConfig.buttons. Example: ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "fileExplorerConfig": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ... ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "buttons": [ ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • .... ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "enabled": true, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "type": "export-files-to-pfb", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "title": "Export All to PFB", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "rightIcon": "external-link", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "tooltipText": "You have not selected any cases to export. Please use the checkboxes on the left to select specific cases you would like to export." ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "enabled": true, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "type": "export-files", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "title": "Export All to Terra", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "rightIcon": "external-link", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "tooltipText": "You have not selected any cases to export. Please use the checkboxes on the left to select specific cases you would like to export." ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ], ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "enableLimitedFilePFBExport": {"sourceNodeField": "source_node"}, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • Limited File PFB Export requires the source_node property to be ETL'd -- source_node should be added to props in the file section of etlMapping.yaml. Example: ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • name: mpingram_file ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • doc_type: file ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • type: collector ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • root: None ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • category: data_file ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • props: ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • name: object_id ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • name: md5sum ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ... ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • name: source_node ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • Limited File PFB Export requires a new export-files job block to be added to the Sower config in manifest.json. The $ROOT_NODE environment variable must be set to "file", or the name of the Guppy file index if it isn't "file". For BioDataCatalyst, the EXTRA_NODES environment variable must be set to "". (This is for backwards compatibility: pelican-export includes reference_file by default on all BDCat PFB exports unless $EXTRA_NODES is set to an empty string). Example sower config: ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "pelican-export-files", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "action": "export-files", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "container": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "job-task", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "image": "quay.io/cdis/pelican-export:0.5.0", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "pull_policy": "Always", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "env": [ ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "DICTIONARY_URL", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "valueFrom": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "configMapKeyRef": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "manifest-global", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "key": "dictionary_url" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "GEN3_HOSTNAME", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "valueFrom": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "configMapKeyRef": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "manifest-global", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "key": "hostname" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "ROOT_NODE", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "value": "file" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "EXTRA_NODES", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "value": "" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ], ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "volumeMounts": [ ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "pelican-creds-volume", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "readOnly": true, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "mountPath": "/pelican-creds.json", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "subPath": "config.json" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "peregrine-creds-volume", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "readOnly": true, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "mountPath": "/peregrine-creds.json", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "subPath": "creds.json" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ], ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "cpu-limit": "1", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "memory-limit": "4Gi" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "volumes": [ ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "pelican-creds-volume", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "secret": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "secretName": "pelicanservice-g3auto" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "name": "peregrine-creds-volume", ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "secret": { ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "secretName": "peregrine-creds" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • } ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • ], ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • "restart_policy": "Never" ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • }, ((PXP-6544) Limited PFB Export from the Files Tab data-portal#729)
  • Portal should be updated to include this fix if Indexd is above 2.12.0 or 2020.07 and user want to use Arborist UI (fix: data upload auth check data-portal#702)
  • New optional config field in the index section of gitops.json called homepageChartNodeChunkSize. Sets the value for paritioning requests to Peregrine for the homepage chart. Defaults to 15 if not provided. (Feat/peregrine request chunk data-portal#699)
  • New explorerConfig which can take configs for multiple tabs, backward compatiable with current dataExplorerConfig and fileExplorerConfig (COV-78 Feat/multi tab explorer data-portal#676)
  • The Terra template attribute in gitops.json now expects a list of strings, not just a single string. (Chore/update terra template data-portal#663)
  • Added footer link configuration to portal config: components.footer.links is an array of { text: <string, optional>, href: <string> }. ((PXP-5571) Add footer links data-portal#661)
  • Added a new property to gitops.json: components.login.contact_link: { text: string, href: string }. If login.contact_link is present, it will be displayed instead of login.email. Email links can be represented in contact_link using "mailto:email@address" as the "href". ((PXP-5571) Allow contact to be a link on login page data-portal#660)
  • Updated tiered access logic requires Guppy server >= 0.4.0 (PXP-4794 PXP-4534 Fix/tiered access data-portal#650)
  • New optional title parameter in Terra Export Warning configuration: terraExportWarning: { subjectThreshold: int, title: string(optional), message: string(optional) } (Make Terra Export Warning title configurable data-portal#656)
  • Added terraExportWarning: { subjectThreshold: <number>, message: <string, optional> } to portal config. (Feat/terra export warning data-portal#642)
  • New optional portal config flag: dataExplorerConfig.terraTemplate. The value of this parameter is placed in the Terra export URL as the value of the query parameter 'template'. (Chore/update terra url data-portal#645)
  • InCommon login needs fence 4.8.0 (Feat/incommon login data-portal#595)
  • Need Guppy 0.3.7+ to handle large file size (PXP-4372 Fix/file size data-portal#603)
  • if deployed with "useArboristUI": true, needs Arborist's GET /auth/mapping endpoint to be exposed via revproxy. (useArboristUI false by default) (Arborist UI integration data-portal#592)
  • New optional portal config field called adminAppliedPreFilters. Goes in data/config/*.json. (Feat/explorer project filters data-portal#576)
  • For a "cert" if the required document is a quiz, then hasCorrectAnswers should be true, otherwise false. (Make quiz configurable data-portal#571)
  • Windmill lists unmapped files by checking Indexd. Before centralized auth's deployment, a file was determined to be unmapped by checking if the acl field was null. After centralized auth, we use the authz field instead. The new manifest configuration setting use_indexd_authz in the global block specifies which of the two fields should be used. This field defaults to false. (feat: use indexd authz config setting data-portal#570)
  • Deployed in https://qa-canine.planx-pla.net with fence version < 3, index authz field set to false (feat: use indexd authz config setting data-portal#570)
  • (by default). (feat: use indexd authz config setting data-portal#570)
  • Deployed in https://qa-brain.planx-pla.net with fence version > 3, index authz field set to true. (feat: use indexd authz config setting data-portal#570)
  • New portal configuration and images for AnVIL (New AnVIL styles data-portal#569)
  • Requires a url to be specified in dataExplorerConfig called terraExportURL (Export to Terra using PFB data-portal#568)
  • This should be bvdp-saturn-${env}.appspot.com/#import-data, where ${env} is dev, staging, or prod (Export to Terra using PFB data-portal#568)
  • Needs a version of cloud-automation that exposes arborist's auth/proxy endpoint (Feat/button auth data-portal#565)
  • fixed DAT requires Guppy >= 0.3.1 (Fix/dat data-portal#563)
  • DAT requires Guppy >= 0.3.0 (Feat/dat data-portal#561)
  • This change requires sower version >= 0.3.0. (PXP-3754: "action" field for sower data-portal#562)
  • This PR requires feat/support is not op for filter guppy#45 (0.2.1+) if want to use Guppy for NIAID (fix/HIV app and Guppy data-portal#558)
  • require fileExplorerConfig block with buttons (Add download/export buttons for file explorer data-portal#554)
  • New manifest.json variable: logout_inactive_users. Defaults to true if not provided. (Fix login logout issues - PXP-3000 data-portal#526)
  • Feat/logout user opt out cloud-automation#818 (Fix login logout issues - PXP-3000 data-portal#526)
  • enable source map tool so that developers could have better debugging under dev mode (fix(srcmap): add source map for better debugging under dev mode data-portal#514)
• sheepdog
  • The environment's user.yaml needs to be updated to add the new Sheepdog admin policy/resources/roles; for each user that has admin: true, grant them the new policy. See https://github.com/uc-cdis/commons-users/pull/810 for an example. (PXP-4102 Feat/arb admin sheepdog#319)
  • Fix Travis coverage reports (combine the results of all 3 tests into one coverage report). (PXP-3408 Update Sheepdog to Python3 sheepdog#314)
  • ARBORIST_URL environment variable replaces broken default ARBORIST config variable (fix(arborist-url) sheepdog#304)
  • Optional configuration variable ARBORIST (arborist base URL) (Feat/arborist2 sheepdog#298)
  • Requires Arborist to be deployed (Feat/arborist2 sheepdog#298)
  • look for INDEX_CLIENT instead of SIGNPOST in app.config (remove signpost from sheepdog deps sheepdog#291)
  • get INDEX_CLIENT_HOST env var instead of SIGNPOST_HOST (remove signpost from sheepdog deps sheepdog#291)
  • Sheepdog will now create (or replace) the missing indexes during migration (PXD-2916: implement uniqueKeys sheepdog#259)
  • New unique indexes have been added to all nodes PXD-2916: add unique indexes for uniqueKeys gen3datamodel#12 (PXD-2916: implement uniqueKeys sheepdog#259)
• tube
  • Requires pelican-export >= 0.5.x (fix(project_id): ensure project_id exists tube#131)
  • Breaking change adding underscore to all {subject,case,file}_id fields. Requires migrating gitops.json and etlMapping.yaml following these instructions: 1. Make a list of all the doc_type fields in etlMapping.yaml. 2. Change any instance of {doc_type}_id to _{doc_type}_id in gitops.json and etlMapping.yaml. (E.g. if doc_type is subject, replace all instances of subject_id in etlMapping.yaml and gitops.json with _subject_id). 3. If there is no change to the etlMapping in the PR, manually add a comment to the etlMapping or change the order of two props in order to force gitops-sync to run the ETL. (fix(project_id): ensure project_id exists tube#131)

Breaking changes

• arborist
  • The GET /auth/mapping endpoint does not accept username as a query parameter anymore. It only supports parsing the username from the provided JWT. To specify a username, use the POST /auth/mapping endpoint. (PXP-10229 Client auth mapping arborist#153)
  • In step with Fence changes to issued JWTs, in which scopes are moved into a dedicated scope claim (Fence 4.22.2/2020.08) and out of the aud claim (Fence 5.0.0/2021.07), this commit changes Arborist JWT validation logic so that it does the same validation it was doing on the aud field, but does it on the scope field instead. This Arborist version must therefore be paired with a Fence later than 4.22.2/2020.08. Conversely, Fence 5.0.0/2021.07 will require Arborist later than this commit. ((PXP-6617): remove scopes from jwt aud claim arborist#130)
• fence
  • Default session expiration changed from 30 minutes to 15 minutes to comply with FedRAMP (feat(session): change default to 15 minutes [PPS-122] fence#1040)
  • URL Signing when no_force_sign query param is provided: Previously Fence would make a decision based off if the data was public and no_force_sign provided. Fence will now NEVER sign if no_force_sign is provided (since the concept of "public" data has been abstracted out of Fence. Data access - public or not - is the sole responsibility of the policy engine). In other words, if no_force_sign is provided at the API level, Fence will respect that regardless of whether the resulting URL will actually work. The default Fence behavior should remain the same. chore(jupyter-nde1.1.0) #988 (Feat/passport fence#964)
  • URL Signing when no_force_sign query param is provided: Previously Fence would make a decision based off if the data was public and no_force_sign provided. Fence will now NEVER sign if no_force_sign is provided (since the concept of "public" data has been abstracted out of Fence. Data access - public or not - is the sole responsibility of the policy engine). In other words, if no_force_sign is provided at the API level, Fence will respect that regardless of whether the resulting URL will actually work. The default Fence behavior should remain the same. (feat(passport-exp): handle expiration edge cases [PXP-9235] fence#988)
  • /link/google will DENY requests with "redirect" param values that aren't in the "allowed redirects" for OAuth2/OIDC Clients in Fence ((PXP-8286): fix(google): add redirect validation to /link/google fence#927)
  • Remove scopes from "aud" claim in Fence-issued tokens. This means that in general, old tokens (minted by previous Fence releases) will not pass validation by this Fence. (PXP-6617 Remove scopes from aud claim in tokens fence#839)
  • Access tokens will still include scopes for compatibility with other microservices, but scopes are no longer guaranteed and should no longer be expected in the aud claim from this version on. ID tokens, refresh tokens, and API keys will not have scopes in the aud claim. (PXP-6617 Remove scopes from aud claim in tokens fence#839)
  • Add custom "scope" claim in tokens and put scopes there instead. Custom validation for scopes claim added in Authutils 6.0.1 (PXP-6617 Remove scopes from aud claim in tokens fence#839)
  • Add issuer to "aud" claim (alongside other aud values--so when there is a Fence client involved, this means the aud claim will include both Fence's BASE_URL and the client_id) (PXP-6617 Remove scopes from aud claim in tokens fence#839)
  • Remove custom "aud" claim validation and use normal validation instead (by bumping to Authutils 6.0.1). This means checking that the aud claim contains a value that identifies the consuming service, i.e. Fence's BASE_URL. (PXP-6617 Remove scopes from aud claim in tokens fence#839)
  • This Nginx status endpoint is a hard dependency for the nginx-prometheus-exporter side car. Any older fence image that is used with the Kubernetes deployment descriptor that is declared with the sidecar below will fail to come up. (chore(monitoring): Adopt new pybase img version. fence#817)
  • name: nginx-prometheus-exporter (chore(monitoring): Adopt new pybase img version. fence#817)
  • image: registry.hub.docker.com/nginx/nginx-prometheus-exporter:0.8.0 (chore(monitoring): Adopt new pybase img version. fence#817)
  • args: ["-nginx.scrape-uri", "http://127.0.0.1/nginx_status"] (chore(monitoring): Adopt new pybase img version. fence#817)
  • ports: (chore(monitoring): Adopt new pybase img version. fence#817)
  • containerPort: 9113 (chore(monitoring): Adopt new pybase img version. fence#817)
  • Usersync will no longer populate user's email field in Fence db with the user's username, UNLESS the username is an email address, in which case previous behavior applies (email field is populated with the username). In other words, nothing should change except in cases where usersync was putting invalid emails into user email fields anyway. (PXP-2802 feat(usync-emails) fence#788)
  • c999 is no longer converted to just phsid when parsing consent codes (which causes issues when switching the configuration on/off in terms of data access). Now, when consent code parsing is on, usersync handles c999 by providing access to all consents explicitly (including itself, .c999, to represent the study-specific exchange area) (Feat/dbgap options fence#702)
  • projects field (user access) removed from id_token for implicit client flow (Feat/id token projects fence#684)
  • Python 3 instead of python 2 (Chore/python3 fence#646)
  • remove --verbose option to fence-create CLI, prefers checking the DEBUG value from config now to determine whether or not to output debug logs (verbose isn't playing well with log level setting based on cfg, remov… fence#647)
  • Users' RBAC policies are now owned by arborist, not fence, so the tables are removed from fence/userdatamodel. (feat/arborist fence#608)
  • The usersync is updated to work specifically with the new arborist version. (feat/arborist fence#608)
  • In general, this and following versions of fence should be deployed only with arborist>=2.0.0 (feat/arborist fence#608)
  • Intended to work only with arborist >= 2.0 (using new database). Won't work with older arborist API. (feat/arborist-sync fence#613)
  • Remove (unused) RBAC blueprint (feat(download-auth): use arborist for data download check fence#606)
  • Questions: (feat(download-auth): use arborist for data download check fence#606)
  • stick with "read-storage"/"write-storage" ? (feat(download-auth): use arborist for data download check fence#606)
• indexd
  • Ensure you are using Fence>=2021.10. Remove deprecated GA4GH access endpoints. these have existed (and been used) from the Fence microservice since last year. See cloud automation change here (fix(migrate): improve acl->authz migration, remove deprecated endpoints indexd#336)
  • indexd blank record endpoints no longer require uploader field (fix(blank): uploader is no longer technically required since arborist… indexd#277)
  • Python 3 instead of python 2 (Chore/python3 indexd#220)
• peregrine
  • Python 3 Support (not expected to be backwards compatible) (Update to Python3  peregrine#154)
  • Use Arborist + gen3authz for auth requests (Feat/arborist peregrine#138)
  • removed redundant building steps (Feat/base peregrine peregrine#110)
• portal
  • Replaced loading the data in iframes in Workspace page with Redirect component. (Add Workspace Registration to Data-Portal data-portal#1067)
  • Deprecating the use of default-workspaces i. e an individual commons will no longer be able to deploy standalone jupyterhub images, they may use hatchery service instead. (Add Workspace Registration to Data-Portal data-portal#1067)
  • studyRegistrationConfig in gitops.json must now be placed within the regsitrationConfigs.features block [Refer this PR] (Add Workspace Registration to Data-Portal data-portal#1067)
  • Rewrite some logic to get rid of deprecated React component lifecycle hooks (PXP-7942 Feat/newer react data-portal#848)
  • Removed react-slick and slick-carousel, use antd Carousel component instead (PXP-7942 Feat/newer react data-portal#848)
  • Arranger is removed (PXP-6409 Remove arranger data-portal#743)
  • DataExplorer is removed (PXP-6409 Remove arranger data-portal#743)
  • ExplorerPage is removed (PXP-6409 Remove arranger data-portal#743)
  • file page 'download' button will always render if useArboristUI is false (it no longer respects Fence authz). If useArboristUI is true, it will render if user has read-storage. Previously, it checked user.project_access (from Fence userinfo) and rendered if user had read-storage. (Fix/permissions PXP-4524 data-portal#610)
  • New sower requests. (PXP-3754: "action" field for sower data-portal#562)
• sheepdog
  • Sheepdog now validates formats. If your dictionary uses formats and you have existing data or submission pipelines that do not respect these formats, this might be a breaking change (PXP-9192 Validate format sheepdog#363)
  • Note that this PR will cause Sheepdog behavior to change slightly during a data submission flow in which the acl and/or authz field are previously populated in the corresponding Indexd record and the Indexd uploader field is not null and the Sheepdog submitter does not match the Indexd uploader. While the existing behavior will cause data submission to fail and report an error message to the user, this PR’s changes will mean that data submission can still succeed in this case of populated acl/authz and uploader mismatch. I believe this change in behavior is not problematic in that it would be unexpected in any commons for uploader and (acl and/or authz) to both be populated. ((PXP-6580): fix data submission by only writing acl and authz iff both are empty sheepdog#330)
  • Having admin: true in user.yaml no longer allows a user to do program/project CRUD in Sheepdog. Users now need to have the Sheepdog admin policy in Arborist. (PXP-4102 Feat/arb admin sheepdog#319)
  • Requires Python 3 (PXP-3408 Update Sheepdog to Python3 sheepdog#314)
  • Updates Sheepdog to use new Indexd aliases endpoint rather than deprecated aliases endpoint (Feat/index record alias api indexd#248) (PXP-3408 Update Sheepdog to Python3 sheepdog#314)
  • Use Arborist + gen3authz for auth requests (Feat/arborist2 sheepdog#298)
  • looks for INDEX_CLIENT instead of SIGNPOST in app.config and gets INDEX_CLIENT_HOST env var instead of SIGNPOST_HOST--you must therefore use the latest cloud automation or manually recreate your sheepdog secrets (remove signpost from sheepdog deps sheepdog#291)
  • removes the USE_SIGNPOST backwards-compatibility workarounds and testing, since GDC forked sheepdog and signpost is dead (remove signpost from sheepdog deps sheepdog#291)
  • deprecate "related cases" caching--not a problem if CASE_CACHE was False (deprecate caching sheepdog#290)
  • removed redundant building steps (Feat/base sheepdog sheepdog#263)
  • changed uwsgi socket location (Feat/base sheepdog sheepdog#263)
• tube
  • python 3! (Chore/py3 tube#119)