Skip to content

Commit

Permalink
Develop (#52)
Browse files Browse the repository at this point in the history
* BT-732 Checksum validation for blobs read by engine (broadinstitute#6838)

* Draft support for optional FileHash

* Draft getMd5 for BlobPath

* Resolve non-parallel IO to fix tests

* Checksum validation for BlobPath

* Nicer error message

* Test for missing Blob hash

* Break attr acquisition into separate method

* Cleanup, comments

* In-progress tests of blob hash command

* Remove test

* Remove unused import

* BT-711 Refresh SAS token for filesystem on expiry (broadinstitute#6831)

* BT-711 Refresh SAS token for filesystem on expiry

* Rough cut of token refresh using exceptions

* Ignore tests, and minor cleanup

* Remove stray line

* Draft of manager class for handling expiring file systems

* Style fixes

* Refactor of blobfilesystemManager and tests covering its functionality

* Refined tests to validate close filesystem as separate unit

* Ignore connected tests

* Clean up of some things

* Refactor BlobFileSystemManager to separate file, and some other cleanup

* Some additional scala-ifying

* Small cleanup

* Correcting imports

* trigger tests

* trigger tests

* Batch 1 of scala steward updates (broadinstitute#6903)

* Batch 1 of scala steward updates

* Rollback snakeYAML

* Attempt 3, with only the passing dependancies

* Revert google API and Big Query udpates

* Winding back other google deps

* rollback remaining google updates

* trigger tests

* trigger tests

* [BW-1398] Migrate PKs to BIGINT (broadinstitute#6907)

* BT-745  Batch 2 of scala steward updates (broadinstitute#6906)

* Update SBT to 2.0.0

* Fix sbt-git import

* Update mouse to 1.0.11

* Update rhino 1.7.14

* SUP-692 Retry with more memory after RC 137 (broadinstitute#6912)

* Reorder execution result checks so 137 can retry with more memory

* Test for memory retry after 137 RC

* Fix test expectations

* Make memory retry checks consistent

* Revert changes to existing test

* Rename retryWithMoreMemory to outOfMemoryDetected

* Scala steward updates batch 3 (broadinstitute#6913)

* Scala steward updates batch 3

* WX-745 Batch 4 scala steward updates (broadinstitute#6916)

* WX-746 Localize all DRS inputs in a single Action (broadinstitute#6914)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-755 Build all images instead of just Cromwell (broadinstitute#6919)

* WX-755 Add `isRelease` option for Docker builds (broadinstitute#6923)

* WX-755 Cromwell/CromIAM automatically board train (broadinstitute#6924)

* WX-755 Fix environment variable syntax (broadinstitute#6926)

* WX-743 Enable TES task creation with BlobPaths (broadinstitute#6921)

* Give blob SAS tokens write permission

* Case class wrapper for subscription id

* Resolve duplicate container name in absolute BlobPath

* Ignored test demonstrating correct absolute path generation

* Update filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobPathBuilderSpec.scala

Co-authored-by: Brian Reilly <breilly@broadinstitute.org>

* PR feedback

Co-authored-by: Brian Reilly <breilly@broadinstitute.org>

* [WX-765] Update snakeyaml to 1.33 (broadinstitute#6927)

* update snakeyaml to 1.33

* Don't use deprecated no-arg Constructor constructor

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WM-1414 Refactoring WesRunLog to omit Cromwell's "workflowLog" object (broadinstitute#6925)

* Upgrade Postgres to 42.4.1 (broadinstitute#6932)

* WX-735 Fix incorrect and/or nondeterministic filesystem ordering (broadinstitute#6930)

* WX-772 Update Scala to 2.13.9 (broadinstitute#6928)

* Update Scala to 2.13.9

* Try updating sbt-scoverage

* Does this version exist anywhere we can see?

* This version actually exists

* Update library version to remove conflict

* Codegen version

* Fix fun new 2.13.9 compiler errors

* Resolve warnings

* Newest Scala?

* I guess not

* Does this please Travis?

* force ci

* Back out changes to generated code

Co-authored-by: Adam Nichols <aednichols@gmail.com>

* WX-781 Bump jackson-databind in /CromwellRefdiskManifestCreator (broadinstitute#6935)

Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.13.2.2 to 2.13.4.1.
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)

---
updated-dependencies:
- dependency-name: com.fasterxml.jackson.core:jackson-databind
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* WX-808 Host allowlist for HTTP imports (broadinstitute#6938)

* `hostAllowlist` that allows everything

* Refactor

* Stick allow list in HttpResolver

* Better default config

* Allow list tests

* Make it build

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Update commons text to 1.10.0 (broadinstitute#6937)

* WX-751 Token refresh signal for monitoring (broadinstitute#6939)

* Log messages

* `DEBUG` -> `INFO`

* WX-744 Optionally rewrite blob paths to appear as local paths (broadinstitute#6941)

* Modify blob paths for TES

* Make blob transformation configurable

* Update supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Apply PR feedback in second place

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Update changelog for wdl http allow list (broadinstitute#6944)

* WM-1491 Fixing Cromwell-client (broadinstitute#6943)

* More updated client for use in cbas

* Removing excess code

* Fix client build script (broadinstitute#6945)

* WX-837: Remove CWL references from documentation (broadinstitute#6949)

* wx-837 removed cwl references in markdown doc files

* wx-837 removed cwlParsingOverview.md, updated mkdocs.yml

* wx-837 updated cromwell.yaml, generated new RESTAPI file

* WX-728 Add configurable WSM client to Cromwell (broadinstitute#6948)

* Dependencies

* Compiles but no tests

* Formatting

* Moar exclusions

* Update to latest WSM

* Add additional dependency

* We need some UUID here to make the request

* Formatting

* Clarify what is fake

* Formatting

* Use our own version of Jersey and Jackson stuff

* Port-in Khalid's changes (thank you!)

Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>

* Test longevity

Don't break the test if someone decides to add a cert to `ws.org`

* Cleanup

* Cleanup

* Cleanup

* Adjust TES config file for CI

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>

* CROM-6554: Removed PAPIv1 references from doc (broadinstitute#6950)

* crom-6554 removed references to PAPI v1 from doc

* crom-6554 pr feedback, reworded doc to use example conf as a starting point

* WX-833 Real Azure DRS Credentials (broadinstitute#6952)

* Remove B2C reference from name

* Get token for current user rather than getting from KeyVault

* Remove KeyVault config for engine

* Remove KeyVault config for DRSLocalizer

* Remove KeyVault dependency

* Remove KeyVault support from localizer repo template

* Cleaned up and working Azure token acquisition for engine

* Collapse localizer's AccessTokenStrategy into DrsCredentials

* Cleanup

* WX-853 Remove most CWL (broadinstitute#6955)

* WX-696 Enable getting SAS token from WSM (broadinstitute#6954)

* WX-696 Enable getting SAS token from WSM

* Wire container resource id from config

* Move resource-container-id config path

* First pass at config for WSM

* Remove unused singleton config

* Tests for new config

* Fix config parsing

* Modified b2c token to be provided each time

* Remove singletonConfig arg from factory

* Restore types to factory configs

* Clean up comments and empty token default

* Default to config b2c before searching environment

* Fix token default on api client

* Fix test

* Refactor error handling for when there is no token

* Remove token constructor arg for clientProvider

* Move configs to global singleton config

* Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala

* default -> override

* Add override token to test

* Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Parentheses

* Reduce token timeout

* Move AzureCredentials to separate file

* Make AzureCredentials an object

* WSM token cleanup

* Config refactor (broadinstitute#6960)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Initial blob token documentation

* Refine language in BlobSasTokenGenerator

* Update comment and formatting

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-853 Remove CWL language factory, Centaur runner (broadinstitute#6961)

* WX-842 Add Pact Dependency for Cromwell (broadinstitute#6962)

* WX-842 Add Pact Dependency for Cromwell

* Remove incomplete test spec

* Initial Pact Test

* Fix pact so it compiles

* Add breadcrumb comment and clean up

* ID-125 Add support for drshub, rename all the things (broadinstitute#6959)

* Add support for drshub, rename all the things

* fallback to martha if resolver is not in config

* WX-867 Translate crc32c hashes to b64 for getm (broadinstitute#6970)

* Translate crc32c hashes to b64 for getm

* Update tests

* Remove obsolete b64 handling for md5, centralize hex validation

* Restore old test, fix other test

* WX-843 Workflow failure reason should accurately indicate issues opening blob filesystem (broadinstitute#6965)

* WX-859 Accept workflow execution identity in config (broadinstitute#6967)

* WX-892 Trim down `ValueStore` logging to prevent OOMs (broadinstitute#6981)

* Add Nirvana 3.18.1 reference image test, minor cleanup [VS-705] (broadinstitute#6975)

* WX-863 Turn off Azure NIO logging (broadinstitute#6982)

* Turn off Azure NIO logging

* Poke Travis

* WM-1616: Allow repeating attempts at initialization (take 2) (broadinstitute#6985)

* WX-878 Single shared BlobFileSystemManager (broadinstitute#6986)

* Make BlobFileSystemManager shared across all BlobPathBuilders

* Update TES conf file to reflect new singleton config

* Shell escape reference image  files [VS-796] [WX-910] (broadinstitute#6989)

* WX-769 `disks` compatibility for TES backend (broadinstitute#6991)

* Update FiveMinuteIntro.md (broadinstitute#6994)

* WX-906 Sbt Unit Tests as Github Actions (broadinstitute#6992)

* WX-926 Support falling back to OCI Manifest Format (broadinstitute#7003)

* WX-926 Support falling back to OCI Manifest Forma

* Only mount reference disks if requested [WX-925] (broadinstitute#7001)

* [WM-1646] Add missing fields for `WorkflowDescription` for WomTool /describe endpoint to Swagger (broadinstitute#7004)

* WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status (broadinstitute#6980)

* WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status

* Address feedback

* Address feedback (broadinstitute#6997)

* Address additional feedback (broadinstitute#7000)

* Fix copy/paste error (broadinstitute#7005)

* Address additional feedback

* Fix copy/paste error

* Trigger CI

---------

Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Centaur reference image test should validate symlinks [VS-796] (broadinstitute#6996)

* WX-903 Pre-GHA test suite disablement

* WX-877 Update CHANGELOG for release 85 (broadinstitute#7011)

* Update cromwell version from 85 to 86

* WX-905 (broadinstitute#7012)

Co-authored-by: Thomas Wiseman <twiseman@wm275-315.lan>
Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-719 modernize `cromwell-publish` image (broadinstitute#7013)

* WX-930 Add `CODEOWNERS` file (broadinstitute#7015)

* Update FiveMinuteIntro: Java 11 requirement (broadinstitute#6830)

Co-authored-by: Adam Nichols <aednichols@gmail.com>

* WX-952 Fix Cromwell version update in Helm chart (broadinstitute#7014)

* Reference Disk Manifest Builder App Test (broadinstitute#7017)

* [WM-1696] Update Womtool (broadinstitute#7019)

Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org>

* WX-958 write_map() should write its last entry with a newline (broadinstitute#7022)

* Centaur slurm (broadinstitute#7083)

* Remove Deploy Key (broadinstitute#7084)

* WX-950 Set user agent in Java client (broadinstitute#7087)

* WX-950 Upgrade Azure libs to probably fix extraneous log (broadinstitute#7088)

* WX-984 Revert "WX-950 Upgrade Azure libs to probably fix extraneous log" (broadinstitute#7090)

* Add cron run integrations each day of the week (broadinstitute#7089)

* 85 release (#28)

* Update cromwell version from 83 to 84

* BW-1255 Implement POST /runs endpoint (broadinstitute#6779)

* Adding route

* Fixing HTTP method error

* All formFields made optional

* A compliling state

* Saving

* Saving

* All three endpoints functioning as expected; updated RESTAPI.md

* Updated response for submission from 200 to 201 to pass tests

* Test submission response

* Moved updated submission response to askSubmit

* test

* updating RESTAPI.md

* saving

* Adding utility file for submitRequest

* cleanup

* Update awssdkv from 2.17.152 to 2.17.194 (broadinstitute#6814)

* BW-1305 Swagger Update (broadinstitute#6818)

* Properly documenting metadataArchiveStatus in WorkflowQueryResult model

* Update docs

* BT-710 Add configs for BlobPathBuilderFactory (broadinstitute#6817)

BT-710 Add configs for BlobPathBuilderFactory

* BW-1305 Make "name" optional in workflow query response (broadinstitute#6821)

* BT-724 Fix BlobPathBuilder failing on retrieving existing filesystem (broadinstitute#6816)

Modify blobPathBuilder to fallback to creating a filesystem if one is not found

* Logging updates: (broadinstitute#6813)

* [BT-698] first pass on BlobTokenGenerator with E2E test (broadinstitute#6824)

* first pass on BlobTokenGenerator with E2E test

* update BlobPathBuilder constructor args in test

* account -> container level client

* [BT-687] specify correct types (broadinstitute#6829)

* specify correct types

* fix test with new type

* remove type declarations in function call

* remove unnecessary sas-token config

* BW-1206 - Combine all Wes Endpoints & add Tests (broadinstitute#6833)

* Add tests, getting frid of WesRunRoutes.scala

* wesWorkflowId fix, ec implicits errors gone

* Refactoring path for GET /runs

* Indentation fix

* Commit to rollback

* Revert "Indentation fix"

This reverts commit 63fc484.

* PR trigger

* Optimize imports

* Missed import

* BW-1354 - Porting CBAS preliminary step (broadinstitute#6837)

* Getting rid of shared utility file; Adding/Updating WES version of submit.

* Edit spec file

* Adding Wes-like error

* BW-1378 Addl CromIAM user enablement checks (broadinstitute#6826)

* Update cromwell version from 84 to 85

* BW-1393 Release doc updates (broadinstitute#6839)

* BT-732 Checksum validation for blobs read by engine (broadinstitute#6838)

* Draft support for optional FileHash

* Draft getMd5 for BlobPath

* Resolve non-parallel IO to fix tests

* Checksum validation for BlobPath

* Nicer error message

* Test for missing Blob hash

* Break attr acquisition into separate method

* Cleanup, comments

* In-progress tests of blob hash command

* Remove test

* Remove unused import

* BT-711 Refresh SAS token for filesystem on expiry (broadinstitute#6831)

* BT-711 Refresh SAS token for filesystem on expiry

* Rough cut of token refresh using exceptions

* Ignore tests, and minor cleanup

* Remove stray line

* Draft of manager class for handling expiring file systems

* Style fixes

* Refactor of blobfilesystemManager and tests covering its functionality

* Refined tests to validate close filesystem as separate unit

* Ignore connected tests

* Clean up of some things

* Refactor BlobFileSystemManager to separate file, and some other cleanup

* Some additional scala-ifying

* Small cleanup

* Correcting imports

* trigger tests

* trigger tests

* Batch 1 of scala steward updates (broadinstitute#6903)

* Batch 1 of scala steward updates

* Rollback snakeYAML

* Attempt 3, with only the passing dependancies

* Revert google API and Big Query udpates

* Winding back other google deps

* rollback remaining google updates

* trigger tests

* trigger tests

* [BW-1398] Migrate PKs to BIGINT (broadinstitute#6907)

* BT-745  Batch 2 of scala steward updates (broadinstitute#6906)

* Update SBT to 2.0.0

* Fix sbt-git import

* Update mouse to 1.0.11

* Update rhino 1.7.14

* SUP-692 Retry with more memory after RC 137 (broadinstitute#6912)

* Reorder execution result checks so 137 can retry with more memory

* Test for memory retry after 137 RC

* Fix test expectations

* Make memory retry checks consistent

* Revert changes to existing test

* Rename retryWithMoreMemory to outOfMemoryDetected

* Scala steward updates batch 3 (broadinstitute#6913)

* Scala steward updates batch 3

* WX-745 Batch 4 scala steward updates (broadinstitute#6916)

* WX-746 Localize all DRS inputs in a single Action (broadinstitute#6914)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-755 Build all images instead of just Cromwell (broadinstitute#6919)

* WX-755 Add `isRelease` option for Docker builds (broadinstitute#6923)

* WX-755 Cromwell/CromIAM automatically board train (broadinstitute#6924)

* WX-755 Fix environment variable syntax (broadinstitute#6926)

* WX-743 Enable TES task creation with BlobPaths (broadinstitute#6921)

* Give blob SAS tokens write permission

* Case class wrapper for subscription id

* Resolve duplicate container name in absolute BlobPath

* Ignored test demonstrating correct absolute path generation

* Update filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobPathBuilderSpec.scala

Co-authored-by: Brian Reilly <breilly@broadinstitute.org>

* PR feedback

Co-authored-by: Brian Reilly <breilly@broadinstitute.org>

* [WX-765] Update snakeyaml to 1.33 (broadinstitute#6927)

* update snakeyaml to 1.33

* Don't use deprecated no-arg Constructor constructor

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WM-1414 Refactoring WesRunLog to omit Cromwell's "workflowLog" object (broadinstitute#6925)

* Upgrade Postgres to 42.4.1 (broadinstitute#6932)

* WX-735 Fix incorrect and/or nondeterministic filesystem ordering (broadinstitute#6930)

* WX-772 Update Scala to 2.13.9 (broadinstitute#6928)

* Update Scala to 2.13.9

* Try updating sbt-scoverage

* Does this version exist anywhere we can see?

* This version actually exists

* Update library version to remove conflict

* Codegen version

* Fix fun new 2.13.9 compiler errors

* Resolve warnings

* Newest Scala?

* I guess not

* Does this please Travis?

* force ci

* Back out changes to generated code

Co-authored-by: Adam Nichols <aednichols@gmail.com>

* WX-781 Bump jackson-databind in /CromwellRefdiskManifestCreator (broadinstitute#6935)

Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.13.2.2 to 2.13.4.1.
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)

---
updated-dependencies:
- dependency-name: com.fasterxml.jackson.core:jackson-databind
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* WX-808 Host allowlist for HTTP imports (broadinstitute#6938)

* `hostAllowlist` that allows everything

* Refactor

* Stick allow list in HttpResolver

* Better default config

* Allow list tests

* Make it build

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Update commons text to 1.10.0 (broadinstitute#6937)

* WX-751 Token refresh signal for monitoring (broadinstitute#6939)

* Log messages

* `DEBUG` -> `INFO`

* WX-744 Optionally rewrite blob paths to appear as local paths (broadinstitute#6941)

* Modify blob paths for TES

* Make blob transformation configurable

* Update supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Apply PR feedback in second place

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Update changelog for wdl http allow list (broadinstitute#6944)

* WM-1491 Fixing Cromwell-client (broadinstitute#6943)

* More updated client for use in cbas

* Removing excess code

* Fix client build script (broadinstitute#6945)

* WX-837: Remove CWL references from documentation (broadinstitute#6949)

* wx-837 removed cwl references in markdown doc files

* wx-837 removed cwlParsingOverview.md, updated mkdocs.yml

* wx-837 updated cromwell.yaml, generated new RESTAPI file

* WX-728 Add configurable WSM client to Cromwell (broadinstitute#6948)

* Dependencies

* Compiles but no tests

* Formatting

* Moar exclusions

* Update to latest WSM

* Add additional dependency

* We need some UUID here to make the request

* Formatting

* Clarify what is fake

* Formatting

* Use our own version of Jersey and Jackson stuff

* Port-in Khalid's changes (thank you!)

Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>

* Test longevity

Don't break the test if someone decides to add a cert to `ws.org`

* Cleanup

* Cleanup

* Cleanup

* Adjust TES config file for CI

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>

* CROM-6554: Removed PAPIv1 references from doc (broadinstitute#6950)

* crom-6554 removed references to PAPI v1 from doc

* crom-6554 pr feedback, reworded doc to use example conf as a starting point

* WX-833 Real Azure DRS Credentials (broadinstitute#6952)

* Remove B2C reference from name

* Get token for current user rather than getting from KeyVault

* Remove KeyVault config for engine

* Remove KeyVault config for DRSLocalizer

* Remove KeyVault dependency

* Remove KeyVault support from localizer repo template

* Cleaned up and working Azure token acquisition for engine

* Collapse localizer's AccessTokenStrategy into DrsCredentials

* Cleanup

* WX-853 Remove most CWL (broadinstitute#6955)

* WX-696 Enable getting SAS token from WSM (broadinstitute#6954)

* WX-696 Enable getting SAS token from WSM

* Wire container resource id from config

* Move resource-container-id config path

* First pass at config for WSM

* Remove unused singleton config

* Tests for new config

* Fix config parsing

* Modified b2c token to be provided each time

* Remove singletonConfig arg from factory

* Restore types to factory configs

* Clean up comments and empty token default

* Default to config b2c before searching environment

* Fix token default on api client

* Fix test

* Refactor error handling for when there is no token

* Remove token constructor arg for clientProvider

* Move configs to global singleton config

* Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala

* default -> override

* Add override token to test

* Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Parentheses

* Reduce token timeout

* Move AzureCredentials to separate file

* Make AzureCredentials an object

* WSM token cleanup

* Config refactor (broadinstitute#6960)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Initial blob token documentation

* Refine language in BlobSasTokenGenerator

* Update comment and formatting

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-853 Remove CWL language factory, Centaur runner (broadinstitute#6961)

* WX-842 Add Pact Dependency for Cromwell (broadinstitute#6962)

* WX-842 Add Pact Dependency for Cromwell

* Remove incomplete test spec

* Initial Pact Test

* Fix pact so it compiles

* Add breadcrumb comment and clean up

* ID-125 Add support for drshub, rename all the things (broadinstitute#6959)

* Add support for drshub, rename all the things

* fallback to martha if resolver is not in config

* WX-867 Translate crc32c hashes to b64 for getm (broadinstitute#6970)

* Translate crc32c hashes to b64 for getm

* Update tests

* Remove obsolete b64 handling for md5, centralize hex validation

* Restore old test, fix other test

* WX-843 Workflow failure reason should accurately indicate issues opening blob filesystem (broadinstitute#6965)

* WX-859 Accept workflow execution identity in config (broadinstitute#6967)

* WX-892 Trim down `ValueStore` logging to prevent OOMs (broadinstitute#6981)

* Add Nirvana 3.18.1 reference image test, minor cleanup [VS-705] (broadinstitute#6975)

* WX-863 Turn off Azure NIO logging (broadinstitute#6982)

* Turn off Azure NIO logging

* Poke Travis

* WM-1616: Allow repeating attempts at initialization (take 2) (broadinstitute#6985)

* WX-878 Single shared BlobFileSystemManager (broadinstitute#6986)

* Make BlobFileSystemManager shared across all BlobPathBuilders

* Update TES conf file to reflect new singleton config

* Shell escape reference image  files [VS-796] [WX-910] (broadinstitute#6989)

* WX-769 `disks` compatibility for TES backend (broadinstitute#6991)

* Update FiveMinuteIntro.md (broadinstitute#6994)

* WX-906 Sbt Unit Tests as Github Actions (broadinstitute#6992)

* WX-926 Support falling back to OCI Manifest Format (broadinstitute#7003)

* WX-926 Support falling back to OCI Manifest Forma

* Only mount reference disks if requested [WX-925] (broadinstitute#7001)

* [WM-1646] Add missing fields for `WorkflowDescription` for WomTool /describe endpoint to Swagger (broadinstitute#7004)

* WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status (broadinstitute#6980)

* WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status

* Address feedback

* Address feedback (broadinstitute#6997)

* Address additional feedback (broadinstitute#7000)

* Fix copy/paste error (broadinstitute#7005)

* Address additional feedback

* Fix copy/paste error

* Trigger CI

---------

Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Centaur reference image test should validate symlinks [VS-796] (broadinstitute#6996)

* WX-903 Pre-GHA test suite disablement

* WX-877 Update CHANGELOG for release 85 (broadinstitute#7011)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Katrina P <68349264+kpierre13@users.noreply.github.com>
Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com>
Co-authored-by: Christian Freitas <christian.j.freitas@gmail.com>
Co-authored-by: Saloni Shah <salonipshah11@gmail.com>
Co-authored-by: kshakir <kshakir@broadinstitute.org>
Co-authored-by: mspector <michael.l.spector@gmail.com>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>
Co-authored-by: Brian Reilly <breilly@broadinstitute.org>
Co-authored-by: Adam Nichols <aednichols@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Justin Variath Thomas <JVThomas@users.noreply.github.com>
Co-authored-by: Christian Freitas <cfreitas@broadinstitute.org>
Co-authored-by: Trevyn Langsford <tlangs@broadinstitute.org>
Co-authored-by: Miguel Covarrubias <mcovarr@users.noreply.github.com>
Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com>
Co-authored-by: Tom Wiseman <tom.h.wiseman@gmail.com>
Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>

* Remove Yaml (broadinstitute#7098)

* Azure blob read/write integration test (broadinstitute#7024)

Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org>

* Goodbye Travis (broadinstitute#7100)

* Single workflow runner (broadinstitute#7106)

* WX-975 Revert special TES path munging (broadinstitute#7103)

* Centaur blob filesystem (broadinstitute#7104)

* WX-1010 Update Github Actions set output syntax (broadinstitute#7111)

* WX-1001 Upgrade JSON Small and Fast Parser (broadinstitute#7116)

* WX-1015 Override glob behavior for TES to use absolute path (broadinstitute#7120)

* WX-990 Make TES request backoff behavior configurable (broadinstitute#7122)

* WX-1103 Revert CI to old TES polling behavior (broadinstitute#7126)

* Develop aws (#29)

* stuck on globbing

* efs works, no callcaching

* update readme

* extended EFS support

* fix for globbing in nested scatters

* updated config for globbing, to prevent issues with empty folders

* WX-1103 Accelerate TES CI (broadinstitute#7130)

* Fix Horicromtal Deadlock Test (broadinstitute#7131)

* WM-1963: Validate PRs begin with Jira tags (broadinstitute#7127)

* WX-1106 Add logging for failed docker manifest pulls (broadinstitute#7135)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* [WX-1108] Disable drs tests (broadinstitute#7145)

* ID-377 Setup pact for consumer contract testing. (broadinstitute#7123)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-1092 Support `size` engine function for public HTTP files (broadinstitute#7128)

* WX-1105 Fix interpretation of full http blob paths (broadinstitute#7138)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Revert "ID-377 Setup pact for consumer contract testing." (broadinstitute#7146)

* WX-966 Add Cascades, remove Directory from Biscayne (broadinstitute#7105)

* WX-966 Post-merge branch cleanup + test fix (broadinstitute#7149)

* WX-1122 Enable Azure ApplicationInsights (broadinstitute#7143)

* [WX-1108] Add tests back (broadinstitute#7148)

* [WX-1136] Docker Build Test (broadinstitute#7151)

* [WX-1136] Self Hosted (broadinstitute#7152)

* WX-1145 Fix minor regression introduced in WDL 1.1 foundation (broadinstitute#7153)

* ID-377 Setup pact for consumer contract testing. Retry (broadinstitute#7147)

Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org>
Co-authored-by: Ivan <ichang@broadinstitute.org>

* WX-1133 Shorten TES localized blob paths (broadinstitute#7150)

* WX-1154 Fix Slack Messaging  (broadinstitute#7159)

* efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs (#32)

* WX-1160 Fix bad MariaDB version assumptions (broadinstitute#7160)

* WX-1126 Upgrade to modern Python, 3.8 not available in package repo anymore (broadinstitute#7164)

* WX-1122 Use legacy AppInsights to get better control over logging (broadinstitute#7157)

* WX-1137 Replace 4-byte unicode chars in PAPI event descriptions (broadinstitute#7166)

* WX-1101 Update library versions to support batch (broadinstitute#7155)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-1195 Update Azure identity libs (broadinstitute#7173)

* WX-1114 Initial inclusion of Azure NIO 'fork' (broadinstitute#7168)

* WX-1110[risk=low] Added endpoint to fetch failed tasks by root workflow id (broadinstitute#7165)

* WX-1210 Added JIRA ID for Cromwhelm auto commit message (broadinstitute#7180)

* WX-1179 DRS image build updates, remove self-hosted runners (broadinstitute#7179)

* WX-1210-action-fix Use PR title to find JIRA ID for cromwhelm commit message (broadinstitute#7184)

* WX-1110-query-fix Corrected Query to pull in attributes outside of executionStatus and backendStatus (broadinstitute#7185)

* WX-1179 GCP Batch new base (broadinstitute#7177)

Co-authored-by: dspeck <dspeck@burwood.com>
Co-authored-by: jarroyo28 <johnarroyo45@gmail.com>
Co-authored-by: Jacob Jennings <53196131+jakejennings@users.noreply.github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org>
Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com>
Co-authored-by: Christian Freitas <cfreitas@broadinstitute.org>
Co-authored-by: Tristan Garwood <tristan.garwood@gmail.com>

* WM-2100: Update terra-helmfile on publish (broadinstitute#7187)

* WM-2100: Correct commit and PR description of terra-helmfile auto update (broadinstitute#7188)

* WX-1230 Ignore `invalidate_bad_caches_use_good_local.test` (broadinstitute#7197)

* WX-1078 ACR support (broadinstitute#7192)

* WX-1179 Enable GCP Batch Integration Tests (broadinstitute#7199)

Co-authored-by: Adam Nichols <aednichols@gmail.com>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-1179 GCP Batch Docs Update (broadinstitute#7196)

Co-authored-by: Jacob Jennings <53196131+jakejennings@users.noreply.github.com>
Co-authored-by: Adam Nichols <aednichols@gmail.com>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* ID-734 Increase Timeout for DRSHub Communication (broadinstitute#7198)

* [WX-1156] internal_path_prefix for TES 4.4 (broadinstitute#7190)

* [WM-2184] Remove brackets from Jira ID (broadinstitute#7206)

* WX-1153 Azure blob read md5 from metadata for large files (broadinstitute#7204)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-1156 Fix internal_path_prefix (broadinstitute#7208)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-1256 Temporarily turn off engine hashing for blob files (broadinstitute#7209)

Co-authored-by: Adam Nichols <aednichols@gmail.com>

* WX-1173 Reopen filesystem for blob storage outside workspace (broadinstitute#7178)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org>
Co-authored-by: Adam Nichols <aednichols@gmail.com>

* WX-1174 Adjust NIO Copy functionality (broadinstitute#7207)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* [WX-1168] TES Log Paths (broadinstitute#7210)

* Develop aws (#34)

* efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs

* add deployment manual, fix issue with empty disks

* update documentation

* update documentation

* update documentation

* WX-1264 Don't expire an unexpirable filesystem (broadinstitute#7216)

* [WX-495] DRS Parallel Downloads (broadinstitute#7214)

* WX-1225 Print TES error messages to job logger (broadinstitute#7220)

* WX-1217 Workflow completion callback (broadinstitute#7213)

Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com>

* [WM-2199] Changeset for modifying ownership of tables for WORKFLOWS app (broadinstitute#7218)

Co-authored-by: Chris Llanwarne <cjllanwarne@gmail.com>

* [WX-1234] Update CHANGELOG.md (broadinstitute#7227)

Merging past CI since this is doc only.

* WM-2252: Configurable metadata write role (broadinstitute#7225)

Co-authored-by: dvoet <dvoet@users.noreply.github.com>

* Update cromwell version from 86 to 87

* WX-1282 Update failedJobs Query to use `lo_get` instead of INNER JOIN against pg_largeobject (broadinstitute#7228)

* [WX-1234] Update Release Process with Docker Instructions (broadinstitute#7231)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WM-2294: Allow role-setter action to run on change (broadinstitute#7233)

* WM-2296: Callback should supply fully qualified output names (broadinstitute#7234)

* [WX-499] DRS Parallel Downloads Follow-up (broadinstitute#7229)

* WX-1318 gcp batch: Add GPU driver install (broadinstitute#7235)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-1232 Include useful workflow ids in TES tags (broadinstitute#7221)

* Options to publish status only (#36)

* add options to publish status only

* updated readme.md

---------

Co-authored-by: quekx <quekx@gene.com>

* WX-1307 Azure E2E test (broadinstitute#7239)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Fix aws unit tests (#39)

* checkpoint

* fix ecr and batch tests

* fix AwsBatchJobSpec.scala

---------

Co-authored-by: quekx <quekx@gene.com>

* return bucket directly instead of listing and checking it (#38)

Co-authored-by: quekx <quekx@gene.com>

* WX-1340 GCP Batch: Mount with extra colon issue and multiple zones support (broadinstitute#7240)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-1339 Make `throwExceptionOnExecuteError` false for PAPI aborts (broadinstitute#7245)

* WX-1338 Fix cron invocation of E2E test (broadinstitute#7244)

* WX-1341 Better logging when a runner stops picking up new workflows (broadinstitute#7246)

* Add evaluteOnExit for aws batch retry (#40)

Co-authored-by: quekx <quekx@gene.com>

* [WX-1260] Acquire sas token from task runner (broadinstitute#7241)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* [WX-1183] Ice Lake (broadinstitute#7252)

* WX-1333 Improve logging visibility for load management (broadinstitute#7253)

* [WX-1391] Fix Bash Bug (broadinstitute#7326)

* [WX-1393] Add Content Length to Curl request (broadinstitute#7328)

* [WX-1346] Scalafmt  (broadinstitute#7257)

* WX-1351 Speed up `Centaur Horicromtal PapiV2 Beta` (broadinstitute#7329)

* WX-1351 CI CWL cleanup (broadinstitute#7327)

* WX-1351 Remove slow/misbehaving localization test (broadinstitute#7330)

* WX-1351 Split out restart tests (broadinstitute#7333)

* [WX-1345] Automatic Token Acquisition for TES Config (broadinstitute#7256)

* WX-1387 Remove unused case classes (broadinstitute#7334)

* [WX-1394] Upgrade Logback Core to address CVE-2023-6378 (broadinstitute#7332)

* WX-1351 Another round of CI cleanup (broadinstitute#7336)

* [WX-1395] Scalafmt GitHub Action (broadinstitute#7337)

* WX-1351 SBT compile speedup (broadinstitute#7339)

* WX-1407 Fix contract tests (broadinstitute#7340)

Co-authored-by: Ivan <ichang@broadinstitute.org>

* WX-1409 Java 17 (broadinstitute#7342)

* Revert "WX-1409 Java 17 (broadinstitute#7342)" (broadinstitute#7343)

* WX-1409 Restore Java 17 (broadinstitute#7342) (broadinstitute#7344)

* Revert "WX-1409 Java 17" (broadinstitute#7346)

* [WX-1395] Scala formatter Github Action (broadinstitute#7341)

Co-authored-by: ScalaFmt Fixer <ScalaFmtFixer@fakeDomain.com>
Co-authored-by: Broad Bot <broadbot@broadinstitute.org>

* Improved tagging support (#37)

* efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs

* add deployment manual, fix issue with empty disks

* update documentation

* update documentation

* update documentation

* support for tagging instances and volumes used in jobs, some support for spaces in file paths

* corrected workflow id in tagging

* redirect exit code 137 to retry-with-more-memory routine

---------

Co-authored-by: Henrique Ribeiro <henriqueribeiro@users.noreply.github.com>

* add gpu count (#41)

* add gpu count

* fix typo

---------

Co-authored-by: quekx <quekx@gene.com>
Co-authored-by: Henrique Ribeiro <henriqueribeiro@users.noreply.github.com>

* WX-1385 Reject blob URLs with external SAS tokens as unparsable (broadinstitute#7347)

* WX-1411 Require preinstalled `jq` and `curl` for just-in-time SAS feature (broadinstitute#7350)

* WX-1396 Cromwell client allow additional properties (broadinstitute#7352)

* WM-2428: Include full error context when failing to abort TES jobs (broadinstitute#7354)

* WX-1420 Fix GCP Batch label regex restriction (broadinstitute#7355)

Co-authored-by: Beibei Chen <beibei@formbio.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-1385 Remove SAS tokens from TES input log printouts (broadinstitute#7358)

* [WX-1184] PostgreSQL Docker Image for Local Cromwell (broadinstitute#7172)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-1444 Use MySQL LTS in DBMS tests (broadinstitute#7360)

* WX-1449 Add `latest` Docker tag (broadinstitute#7362)

* WX-767 Upgrade Cloud SDK to 461.0.0 (broadinstitute#7361)

* WX-1443 Adopt `gcloud storage` for localization only (broadinstitute#7359)

* WX-964 suffix() (broadinstitute#7363)

* WX-1417 New database role strategy (broadinstitute#7366)

* [WM-2291] Callback API contract tests between Cromwell and CBAS (broadinstitute#7251)

Co-authored-by: Chris Llanwarne <cjllanwarne@gmail.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-1445 Update docker image regex to handle python:3 correctly (broadinstitute#7367)

* WX-1252 Runtime attributes cleanup – CWL runtime attributes (broadinstitute#7370)

* WX-1485 Upgrade postgres lib (broadinstitute#7371)

* WM-2454: Private GitHub support on describe api (broadinstitute#7365)

* WX-1252 Runtime attributes cleanup – CWL runtime environment (broadinstitute#7369)

* WX-1462 POSIX-flavored sub() (broadinstitute#7374)

* [WX-963] Unzip Engine Function  (broadinstitute#7368)

* WX-1489 Hopefully reduce CI flakiness by modernizing deadlock test (broadinstitute#7376)

* [WX-965] quote() and squote() engine functions. (broadinstitute#7375)

Co-authored-by: Ryan Saperstein <rsaperst@wmd55-b47.hsd1.ma.comcast.net>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* [WX-1317] Remove Akka 'server' header from all HTTP responses (broadinstitute#7379)

* WM-2461] Add support for running private workflows on Azure (broadinstitute#7373)

* WX-1252 Per-backend runtime attributes (broadinstitute#7380)

* ID-347 Remove Martha References (broadinstitute#7384)

* WX-1461 Remove `womtool upgrade` command (broadinstitute#7382)

* WX-1488 Supply cloud platform when making DRS requests (broadinstitute#7381)

* WX-757 Fix workflow stuck in aborting after WDL type error (broadinstitute#7385)

* [WX-1506] Add jobLogger output for subworkflow running (with parent and subworkflow ID) (broadinstitute#7388)

Co-authored-by: jlester-msft <47044104+jlester-msft@users.noreply.github.com>

* WX-757 Fix `stdout`, `stderr` in workflow body causing crashes (broadinstitute#7386)

* WX-1519 Stop leaking DRS Localizer images on every CI run (broadinstitute#7390)

* [WX-1460] WDL 1.1 Struct Literal Parsing (broadinstitute#7391)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* [WM-2500][WM-2502] Fetch Github token from ECM for importing and running private workflows (broadinstitute#7392)

* WX-1542 Fix workflow cancellation (broadinstitute#7398)

* WX-1550 Don't pointlessly continue running integration tests (broadinstitute#7400)

* WX-1557 Fix default values from Configuring.md and Scaling.md (broadinstitute#7393)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-1557 Fix actor-factory in Batch101.md (broadinstitute#7377)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-1557 Add more tests to the GCP Batch backend (broadinstitute#7394)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* [WX-1531] Struct Literal Type Checking (broadinstitute#7402)

* [WX-1468] Implement `returnCodes` runtime attribute (broadinstitute#7389)

Co-authored-by: Ryan Saperstein <rsaperst@wmd55-b47.hsd1.ma.comcast.net>
Co-authored-by: Ryan Saperstein <rsaperst@wmd55-b47.broadinstitute.org>

* [WM-2555] Cromwell -> ECM contract test (broadinstitute#7405)

* [WX-1568] Bump Akka max-response-reason-length limit (broadinstitute#7406)

* WX-1530 Strip pesky URL bits when creating local paths for HTTP inputs (broadinstitute#7404)

* WX-1538 Add more tests to the GCP Batch backend (broadinstitute#7410)

* [WX-1605] Fix Codecov Report Uploads (broadinstitute#7418)

* [WX-1410] Sanitize 4 byte UTF-8 characters before inserting into METADATA_ENTRY (broadinstitute#7414)

* WX-1566 Special Docker build for debugging (broadinstitute#7417)

Run `sbt -Dproject.isDebug=true server/docker`

* [WX-1448] Add verbose logging and timeout for getm (broadinstitute#7416)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-1566 Fix Morgan's call cache file hash CPU thrash Cromwell crash (broadinstitute#7419)

* WX-1566 Fix RTD so this ticket's docs get published (broadinstitute#7420)

* fix womFileType coercion (#47)

Co-authored-by: quekx <quekx@gene.com>

* PR #44: Multi-aserisk globbing by @kevinRenaersBio.  Reapplied selection of changes on aws_develop branch

* PR #45 reapplied to aws_develop: support for sharedMemory. skipped EFS fixes for now while waiting for feedback

* upstream PR broadinstitute#7488 : fix compilation error due to swagger2markup

* Fix issue with caching on EFS by replacing touch with existOrThrow routine

* make sure that md5 sibling file of EFS data is newer than main file to be valid

* revise sharedmemory attribute to take reguluar GB/MB specification

* updated AWS README file

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Christian Freitas <christian.j.freitas@gmail.com>
Co-authored-by: Saloni Shah <salonipshah11@gmail.com>
Co-authored-by: Brian Reilly <breilly@broadinstitute.org>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>
Co-authored-by: Katrina P <68349264+kpierre13@users.noreply.github.com>
Co-authored-by: Adam Nichols <aednichols@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com>
Co-authored-by: Justin Variath Thomas <JVThomas@users.noreply.github.com>
Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>
Co-authored-by: Christian Freitas <cfreitas@broadinstitute.org>
Co-authored-by: Trevyn Langsford <tlangs@broadinstitute.org>
Co-authored-by: Miguel Covarrubias <mcovarr@users.noreply.github.com>
Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com>
Co-authored-by: Tom Wiseman <tom.h.wiseman@gmail.com>
Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>
Co-authored-by: Thomas Wiseman <twiseman@wm275-315.lan>
Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org>
Co-authored-by: Stephen Fleming <sjfleming@users.noreply.github.com>
Co-authored-by: Henrique Ribeiro <henriqueribeiro@users.noreply.github.com>
Co-authored-by: mspector <michael.l.spector@gmail.com>
Co-authored-by: Tristan Garwood <tristan.garwood@gmail.com>
Co-authored-by: Ivan <ichang@broadinstitute.org>
Co-authored-by: dspeck1 <dspeck@burwood.com>
Co-authored-by: jarroyo28 <johnarroyo45@gmail.com>
Co-authored-by: Jacob Jennings <53196131+jakejennings@users.noreply.github.com>
Co-authored-by: Dillon Scott <67511512+dillydally414@users.noreply.github.com>
Co-authored-by: Chris Llanwarne <cjllanwarne@gmail.com>
Co-authored-by: dvoet <dvoet@users.noreply.github.com>
Co-authored-by: xquek <46759870+xquek@users.noreply.github.com>
Co-authored-by: quekx <quekx@gene.com>
Co-authored-by: ScalaFmt Fixer <ScalaFmtFixer@fakeDomain.com>
Co-authored-by: Broad Bot <broadbot@broadinstitute.org>
Co-authored-by: Beibei Chen <beibei@formbio.com>
Co-authored-by: Maria Yazykova <95451512+zykonda@users.noreply.github.com>
Co-authored-by: rsaperst <156102201+rsaperst@users.noreply.github.com>
Co-authored-by: Ryan Saperstein <rsaperst@wmd55-b47.hsd1.ma.comcast.net>
Co-authored-by: jlester-msft <47044104+jlester-msft@users.noreply.github.com>
Co-authored-by: Alexis Hernandez <github@alexitc.com>
Co-authored-by: dkj <david.jackson@sanger.ac.uk>
Co-authored-by: Ryan Saperstein <rsaperst@wmd55-b47.broadinstitute.org>
Co-authored-by: Alexis Hernandez <alexis22229@gmail.com>
  • Loading branch information
Show file tree
Hide file tree
Showing 1,896 changed files with 84,337 additions and 39,214 deletions.
174 changes: 0 additions & 174 deletions .circleci/config.yml

This file was deleted.

1 change: 0 additions & 1 deletion .gitattributes
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,3 @@
*.MD text
*.java text
*.html text
docs/api/RESTAPI.md linguist-generated=true
117 changes: 117 additions & 0 deletions .github/workflows/azure_e2e_run_workflow.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
name: 'Azure e2e - Run Workflow'
on:
schedule:
- cron: '0 16 * * *' # UTC 4pm, EST 11am, EDT 12pm
workflow_dispatch:

env:
BROADBOT_TOKEN: '${{ secrets.BROADBOT_GITHUB_TOKEN }}' # github token for access to kick off a job in the private repo
RUN_NAME_SUFFIX: '${{ github.event.repository.name }}-${{ github.run_id }}-${{ github.run_attempt }}'

jobs:

# This job provisions useful parameters for e2e tests
params-gen:
runs-on: ubuntu-latest
permissions:
contents: 'read'
id-token: 'write'
outputs:
project-name: ${{ steps.gen.outputs.project_name }}
bee-name: '${{ github.event.repository.name }}-${{ github.run_id }}-${{ github.run_attempt}}-dev'
steps:
- name: Generate a random billing project name
id: 'gen'
run: |
project_name=$(echo "tmp-billing-project-$(uuidgen)" | cut -c -30)
echo "project_name=${project_name}" >> $GITHUB_OUTPUT
create-bee-workflow:
runs-on: ubuntu-latest
needs: [params-gen]
permissions:
contents: 'read'
id-token: 'write'
steps:
- name: Dispatch to terra-github-workflows
uses: broadinstitute/workflow-dispatch@v3
with:
workflow: bee-create
repo: broadinstitute/terra-github-workflows
ref: refs/heads/main
token: ${{ env.BROADBOT_TOKEN }}
# NOTE: Opting to use "prod" instead of custom tag since I specifically want to test against the current prod state
# NOTE: For testing/development purposes I'm using dev
inputs: '{ "bee-name": "${{ needs.params-gen.outputs.bee-name }}", "version-template": "dev", "bee-template-name": "rawls-e2e-azure-tests"}'

create-and-attach-billing-project-to-landing-zone-workflow:
runs-on: ubuntu-latest
needs: [create-bee-workflow, params-gen]
steps:
- name: dispatch to terra-github-workflows
uses: broadinstitute/workflow-dispatch@v3
with:
workflow: attach-billing-project-to-landing-zone.yaml
repo: broadinstitute/terra-github-workflows
ref: refs/heads/main
token: ${{ env.BROADBOT_TOKEN }}
inputs: '{
"run-name": "attach-billing-project-to-landing-zone-${{ env.RUN_NAME_SUFFIX }}",
"bee-name": "${{ needs.params-gen.outputs.bee-name }}",
"billing-project": "${{ needs.params-gen.outputs.project-name }}",
"service-account": "firecloud-qa@broad-dsde-qa.iam.gserviceaccount.com" }'

run-cromwell-az-e2e:
needs: [params-gen, create-and-attach-billing-project-to-landing-zone-workflow]
permissions:
contents: read
id-token: write
uses: "broadinstitute/dsp-reusable-workflows/.github/workflows/cromwell-az-e2e-test.yaml@main"
with:
bee-name: "${{ needs.params-gen.outputs.bee-name }}"
billing-project-name: "${{ needs.params-gen.outputs.project-name }}"

delete-billing-project-v2-from-bee-workflow:
continue-on-error: true
runs-on: ubuntu-latest
needs: [run-cromwell-az-e2e, create-and-attach-billing-project-to-landing-zone-workflow, params-gen]
if: always()
steps:
- name: dispatch to terra-github-workflows
uses: broadinstitute/workflow-dispatch@v3
with:
workflow: .github/workflows/delete-billing-project-v2-from-bee.yaml
repo: broadinstitute/terra-github-workflows
ref: refs/heads/main
token: ${{ env.BROADBOT_TOKEN }}
inputs: '{
"run-name": "delete-billing-project-v2-from-bee-${{ env.RUN_NAME_SUFFIX }}",
"bee-name": "${{ needs.params-gen.outputs.bee-name }}",
"billing-project": "${{ needs.params-gen.outputs.project-name }}",
"service-account": "firecloud-qa@broad-dsde-qa.iam.gserviceaccount.com",
"silent-on-failure": "false" }'

destroy-bee-workflow:
runs-on: ubuntu-latest
needs: [params-gen, create-bee-workflow, delete-billing-project-v2-from-bee-workflow]
if: always()
permissions:
contents: 'read'
id-token: 'write'
steps:
- name: dispatch to terra-github-workflows
uses: broadinstitute/workflow-dispatch@v3
with:
workflow: bee-destroy.yaml
repo: broadinstitute/terra-github-workflows
ref: refs/heads/main
token: ${{ env.BROADBOT_TOKEN }}
inputs: '{ "bee-name": "${{ needs.params-gen.outputs.bee-name }}" }'
wait-for-completion: true

report-workflow:
uses: broadinstitute/sherlock/.github/workflows/client-report-workflow.yaml@main
with:
notify-slack-channels-upon-workflow-failure: "#cromwell_jenkins_ci_errors"
permissions:
id-token: write
62 changes: 56 additions & 6 deletions .github/workflows/chart_update_on_merge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,14 @@ jobs:
chart-update:
name: Cromwhelm Chart Auto Updater
if: github.event.pull_request.merged == true
runs-on: self-hosted # Faster machines; see https://github.com/broadinstitute/cromwell/settings/actions/runners
runs-on: ubuntu-latest
steps:
- name: Fetch Jira ID from the commit message
id: fetch-jira-id
run: |
JIRA_ID=$(echo '${{ github.event.pull_request.title }}' | grep -Eo '[A-Z][A-Z]+-[0-9]+' | xargs echo -n | tr '[:space:]' ',')
[[ -z "$JIRA_ID" ]] && { echo "No Jira ID found in $1" ; exit 1; }
echo "JIRA_ID=$JIRA_ID" >> $GITHUB_OUTPUT
- name: Clone Cromwell
uses: actions/checkout@v2
with:
Expand Down Expand Up @@ -49,11 +55,12 @@ jobs:
username: dsdejenkins
password: ${{ secrets.DSDEJENKINS_PASSWORD }}
# Build & push `cromwell`, `womtool`, `cromiam`, and `cromwell-drs-localizer`
# This step is validated in the GHA 'docker_build_test.yml' without the accompanying docker push
- name: Build Cromwell Docker
run: |
set -e
cd cromwell
sbt -Dproject.isSnapshot=false -Dproject.isRelease=false dockerBuildAndPush
sbt -Dproject.isSnapshot=false dockerBuildAndPush
- name: Deploy to dev and board release train (Cromwell)
uses: broadinstitute/repository-dispatch@master
with:
Expand All @@ -68,18 +75,61 @@ jobs:
repository: broadinstitute/terra-helmfile
event-type: update-service
client-payload: '{"service": "cromiam", "version": "${{ env.CROMWELL_VERSION }}", "dev_only": false}'
- name: Edit & push chart
- name: Edit & push cromwhelm chart
env:
BROADBOT_GITHUB_TOKEN: ${{ secrets.BROADBOT_GITHUB_TOKEN }}
run: |
set -e
cd cromwhelm
git checkout main
ls -la
sed -i "s/appVersion.*/appVersion: \"$CROMWELL_VERSION\"/" cromwell-helm/Chart.yaml
sed -i "s/image: broadinstitute\/cromwell.*/image: broadinstitute\/cromwell:$CROMWELL_VERSION/" cromwell-helm/templates/cromwell.yaml
sed -i "s|image: broadinstitute/cromwell:.*|image: broadinstitute/cromwell:$CROMWELL_VERSION|" terra-batch-libchart/values.yaml
git diff
git config --global user.name "broadbot"
git config --global user.email "broadbot@broadinstitute.org"
git commit -am "Auto update to Cromwell $CROMWELL_VERSION"
git commit -am "${{ steps.fetch-jira-id.outputs.JIRA_ID }}: Auto update to Cromwell $CROMWELL_VERSION"
git push https://broadbot:$BROADBOT_GITHUB_TOKEN@github.com/broadinstitute/cromwhelm.git main
cd -
- name: Clone terra-helmfile
uses: actions/checkout@v3
with:
repository: broadinstitute/terra-helmfile
token: ${{ secrets.BROADBOT_GITHUB_TOKEN }} # Has to be set at checkout AND later when pushing to work
path: terra-helmfile

- name: Update workflows-app in terra-helmfile
run: |
set -e
cd terra-helmfile
sed -i "s|image: broadinstitute/cromwell:.*|image: broadinstitute/cromwell:$CROMWELL_VERSION|" charts/workflows-app/values.yaml
cd -
- name: Update cromwell-runner-app in terra-helmfile
run: |
set -e
cd terra-helmfile
sed -i "s|image: broadinstitute/cromwell:.*|image: broadinstitute/cromwell:$CROMWELL_VERSION|" charts/cromwell-runner-app/values.yaml
cd -
- name: Make PR in terra-helmfile
env:
BROADBOT_TOKEN: ${{ secrets.BROADBOT_GITHUB_TOKEN }}
GH_TOKEN: ${{ secrets.BROADBOT_GITHUB_TOKEN }}
run: |
set -e
JIRA_ID=${{ steps.fetch-jira-id.outputs.JIRA_ID }}
if [[ $JIRA_ID == "missing" ]]; then
echo "JIRA_ID missing, PR to terra-helmfile will not be created"
exit 0;
fi
cd terra-helmfile
git checkout -b ${JIRA_ID}-cromwell-update-$CROMWELL_VERSION
git config --global user.name "broadbot"
git config --global user.email "broadbot@broadinstitute.org"
git commit -am "${JIRA_ID}: Auto update Cromwell to $CROMWELL_VERSION in workflows-app and cromwell-runner-app"
git push -u origin ${JIRA_ID}-cromwell-update-$CROMWELL_VERSION
gh pr create --title "${JIRA_ID}: auto update Cromwell version to $CROMWELL_VERSION in workflows-app and cromwell-runner-app" --body "${JIRA_ID} helm chart update" --label "automerge"
cd -
Loading

0 comments on commit 9d22c1a

Please sign in to comment.