WM-2461] Add support for running private workflows on Azure #7373

salonishah11 · 2024-02-23T22:56:48Z

Tested changes on BEE by importing a private workflow and running it

Closes https://broadworkbench.atlassian.net/browse/WM-2461

salonishah11 · 2024-02-23T23:06:13Z

.../cromwell/engine/workflow/lifecycle/materialization/MaterializeWorkflowDescriptorActor.scala

@@ -201,6 +223,7 @@ class MaterializeWorkflowDescriptorActor(serviceRegistryActor: ActorRef,

  val iOExecutionContext = context.system.dispatchers.lookup("akka.dispatchers.io-dispatcher")
  implicit val ec = context.dispatcher
+  implicit val timeout = Timeout(5.seconds)


Question for reviewers - What should the value for timeout be?

Since this is a non-synchronous activity, I would lean higher here (like maybe 60 seconds)? But not a strong opinion and happy to be talked down from that

TOL can we provide this value explicitly where it's used? Or rename it something less generic? In the context of the materialize workflow actor, a global value called "timeout" to refer specifically to github auth lookup is a little odd

This is coming from the GithubAuthVendingSupport trait. I am not sure how to provide this value where it is used since its coming from the trait? 🤔

cjllanwarne · 2024-02-27T18:50:33Z

core/src/main/resources/reference.conf

+private-workflows {
+  enabled = false
+  auth {
+    azure = false
+  }
+}


Should we combine this with the token access config? Is there any additional "turning off" that is necessary here if the github auth lookup service is turned off?

Also this is called "private workflows" but seems very specific to github authing?

+1 to combining this with the token access config.

I had added this here as there wasn't an enabled config value in GithubAuthVending initially. But now I can combine it. Do we want the enabling of GithubAuthVending to be only for Azure environment? Or would the value for GithubAuthVending.config.enabled be always false but overriden in Azure helm charts in future?

Actually we might need some way of figuring out which environment Cromwell is running in or not because we can only do AzureCredentials.getToken() here only if Cromwell is running on Azure

Maybe we can incorporate the auth.azure config key into the other config, and always fall back to it if there isn't a token attached to the incoming request?

In practice we need two different auth strategies to support what we're trying to do on Azure, but that's due to the specific way we've set up these multiple instances. It would be nice to hide that difference from the configurer, and them think of it in terms of: Cromwell will use a token attached to an incoming request when it can, and will otherwise get a token from .

I can move auth.azure inside GithubAuthVending.config 👍

Cromwell will use a token attached to an incoming request when it can

For workflow submissions, if we extract tokens from the request, we would need to store that token in the database or in some form in memory no? Because when workflow is being materialized it doesn't have access to the original request and its headers as the workflow upon submission is store in WorkflowStore table? If my understanding is right then in this case we would always need to get the token using AzureCredentials.getToken()

Oh, by "when it can" I mean in the parts of the process where that token is already available :) Not suggesting we make it available in places it isn't already.

cjllanwarne · 2024-02-27T20:26:14Z

.../cromwell/engine/workflow/lifecycle/materialization/MaterializeWorkflowDescriptorActor.scala

+  def getImportAuthProviders(conf: Config,
+                             authProviderFunc: String => ImportAuthProvider
+  ): ErrorOr[List[ImportAuthProvider]] = {
+    val isPrivateWorkflowsEnabled = conf.as[Boolean]("private-workflows.enabled")


I think there's a philosophical question here about whether

The materializer should have a config that decides whether or not it opts-in to the auth provider that's configured

The materializer always opts in to the auth provider, which can be configured whether or not to supply auth tokens (and if so, what type)

I think my personal preference would be for 2, but it might be worth considering the pros and cons of each

FWIW:

I think the auth vending service is already configurable to opt-out of doing any auth lookups

It would be nice if the describe and execution were configured in the same place and in the same way

It might make the logic here a little simpler?

I like the idea of the describe and execution functions referencing the same config, and of all the config checking happening inside GithubAuthVendingSupport.

The materializer always opts in to the auth provider, which can be configured whether or not to supply auth tokens (and if so, what type)

So in the MaterializeWorkflowDescriptorActor it would always pass in the GithAuthProvider to the HttpResolver and it GithAuthVending service will be opting in or out of doing auth lookups based on the config. Is my understanding right?

jgainerdewar · 2024-02-27T21:25:34Z

core/src/main/resources/reference.conf

+private-workflows {
+  enabled = false
+  auth {
+    azure = false
+  }
+}


+1 to combining this with the token access config.

jgainerdewar · 2024-02-27T21:34:23Z

.../cromwell/engine/workflow/lifecycle/materialization/MaterializeWorkflowDescriptorActor.scala

+  def getImportAuthProviders(conf: Config,
+                             authProviderFunc: String => ImportAuthProvider
+  ): ErrorOr[List[ImportAuthProvider]] = {
+    val isPrivateWorkflowsEnabled = conf.as[Boolean]("private-workflows.enabled")


I like the idea of the describe and execution functions referencing the same config, and of all the config checking happening inside GithubAuthVendingSupport.

jgainerdewar · 2024-02-27T21:41:16Z

.../cromwell/engine/workflow/lifecycle/materialization/MaterializeWorkflowDescriptorActor.scala

+    } else false
+
+    if (isPrivateWorkflowsEnabled && isAuthAzure) {
+      val azureToken = AzureCredentials.getAccessToken()


Do you think we need to worry at all about this token expiring? How long does the longest workflow materialization take?

I would think that the time between this token is fetched and workflow is materialized isn't very long, and the token shouldn't expire between that period. Or do you think there is a possibility that token might expire after it was fetched? But it is however possible that Cromwell wasn't able to refresh the token or received a null token (based on this) and in that case the materialization will fail.

jgainerdewar · 2024-02-27T21:43:01Z

.../cromwell/engine/workflow/lifecycle/materialization/MaterializeWorkflowDescriptorActor.scala

+    } else false
+
+    if (isPrivateWorkflowsEnabled && isAuthAzure) {
+      val azureToken = AzureCredentials.getAccessToken()


I know we talked about how WDL materialization includes caching, so if we are running the same workflow 1000 times we don't do all the work of materializing it each time. Does this token generation benefit from that caching, or will we be generating it for each workflow, then often not using it? (Do we pull the workflow from the cache before we create this actor, or after?)

I am actually not sure where that WDL caching happens. 🤔

then often not using it

also just to note, it is possible we don't end up using the token even when the workflow is a public workflow.

So for Azure, we generate this token once for every workflow we run, except potentially cached ones? Do you know whether it happens for each subworkflow as well?

It can take a while (like a couple seconds) to generate this token, I don't know that it's required for this PR but I think we'll eventually want some caching. It would be good to confirm whether we're already benefiting from the WDL caching, just so we understand the behavior. If we are, I think this is fine as-is.

I can look into where WDL caching is happening. But for the code as is, if the MaterializeWorkflowActor is created for each sub-workflow then it would try to get the token each time.

I agree with you that some form of token caching in the future is a good idea.

One thing to be wary of is that private workflows only get cached for the same user. Otherwise we have a potential data leak situation possible

It seems that WDL caching is happening after the workflow url is resolved in the MaterializeDescriptorActor.

Workflow source/url resolution happens here

once it is resolved it calls buildValidatedNamespace(...) here

buildValidatedNamespace calls validateNamespace and each WDL language has its own implementation. Inside validateNamespace it calls retrieveOrCalculate which does the WDL caching

So it seems that upon running 100 same workflows in a single Cromwell Runner in Azure, it would fetch Azure token each time to resolve the workflow url and then the WDL caching will happening during WOM generation.

jgainerdewar · 2024-02-27T21:43:41Z

.../cromwell/engine/workflow/lifecycle/materialization/MaterializeWorkflowDescriptorActor.scala

                                         workflowId: WorkflowId,
                                         cromwellBackends: => CromwellBackends,
                                         importLocalFilesystem: Boolean,
                                         ioActorProxy: ActorRef,
                                         hogGroup: HogGroup
 ) extends LoggingFSM[MaterializeWorkflowDescriptorActorState, Unit]
    with StrictLogging
-    with WorkflowLogging {
+    with WorkflowLogging
+    with GithubAuthVendingSupport {


Love being able to just hook this in!

jgainerdewar

This looks great, nice job!

jgainerdewar · 2024-03-05T15:25:25Z

services/src/main/scala/cromwell/services/auth/GithubAuthVendingSupport.scala

 import scala.concurrent.{ExecutionContext, Future}

 trait GithubAuthVendingSupport extends AskSupport with StrictLogging {

  def serviceRegistryActor: ActorRef

-  implicit val timeout: Timeout


I don't necessarily have an objection, just curious - why is the timeout implicit rather than overridden by classes that extend this trait?

The reason I removed it from here and put it as implicit parameter for each method was to address this comment.

But in regards to your specific question, I am not really sure. I have seen this pattern for timeout being implicit in other actor classes and my understanding was making them implicit means it could be extracted from the context indirectly (as long as it is in the context) when needed instead of each class having to supply it. But my understanding is basic. Maybe @cjllanwarne knows more?

cjllanwarne

LGTM

@kevinRenaersBio

* BT-732 Checksum validation for blobs read by engine (broadinstitute#6838) * Draft support for optional FileHash * Draft getMd5 for BlobPath * Resolve non-parallel IO to fix tests * Checksum validation for BlobPath * Nicer error message * Test for missing Blob hash * Break attr acquisition into separate method * Cleanup, comments * In-progress tests of blob hash command * Remove test * Remove unused import * BT-711 Refresh SAS token for filesystem on expiry (broadinstitute#6831) * BT-711 Refresh SAS token for filesystem on expiry * Rough cut of token refresh using exceptions * Ignore tests, and minor cleanup * Remove stray line * Draft of manager class for handling expiring file systems * Style fixes * Refactor of blobfilesystemManager and tests covering its functionality * Refined tests to validate close filesystem as separate unit * Ignore connected tests * Clean up of some things * Refactor BlobFileSystemManager to separate file, and some other cleanup * Some additional scala-ifying * Small cleanup * Correcting imports * trigger tests * trigger tests * Batch 1 of scala steward updates (broadinstitute#6903) * Batch 1 of scala steward updates * Rollback snakeYAML * Attempt 3, with only the passing dependancies * Revert google API and Big Query udpates * Winding back other google deps * rollback remaining google updates * trigger tests * trigger tests * [BW-1398] Migrate PKs to BIGINT (broadinstitute#6907) * BT-745 Batch 2 of scala steward updates (broadinstitute#6906) * Update SBT to 2.0.0 * Fix sbt-git import * Update mouse to 1.0.11 * Update rhino 1.7.14 * SUP-692 Retry with more memory after RC 137 (broadinstitute#6912) * Reorder execution result checks so 137 can retry with more memory * Test for memory retry after 137 RC * Fix test expectations * Make memory retry checks consistent * Revert changes to existing test * Rename retryWithMoreMemory to outOfMemoryDetected * Scala steward updates batch 3 (broadinstitute#6913) * Scala steward updates batch 3 * WX-745 Batch 4 scala steward updates (broadinstitute#6916) * WX-746 Localize all DRS inputs in a single Action (broadinstitute#6914) Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * WX-755 Build all images instead of just Cromwell (broadinstitute#6919) * WX-755 Add `isRelease` option for Docker builds (broadinstitute#6923) * WX-755 Cromwell/CromIAM automatically board train (broadinstitute#6924) * WX-755 Fix environment variable syntax (broadinstitute#6926) * WX-743 Enable TES task creation with BlobPaths (broadinstitute#6921) * Give blob SAS tokens write permission * Case class wrapper for subscription id * Resolve duplicate container name in absolute BlobPath * Ignored test demonstrating correct absolute path generation * Update filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobPathBuilderSpec.scala Co-authored-by: Brian Reilly <breilly@broadinstitute.org> * PR feedback Co-authored-by: Brian Reilly <breilly@broadinstitute.org> * [WX-765] Update snakeyaml to 1.33 (broadinstitute#6927) * update snakeyaml to 1.33 * Don't use deprecated no-arg Constructor constructor Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * WM-1414 Refactoring WesRunLog to omit Cromwell's "workflowLog" object (broadinstitute#6925) * Upgrade Postgres to 42.4.1 (broadinstitute#6932) * WX-735 Fix incorrect and/or nondeterministic filesystem ordering (broadinstitute#6930) * WX-772 Update Scala to 2.13.9 (broadinstitute#6928) * Update Scala to 2.13.9 * Try updating sbt-scoverage * Does this version exist anywhere we can see? * This version actually exists * Update library version to remove conflict * Codegen version * Fix fun new 2.13.9 compiler errors * Resolve warnings * Newest Scala? * I guess not * Does this please Travis? * force ci * Back out changes to generated code Co-authored-by: Adam Nichols <aednichols@gmail.com> * WX-781 Bump jackson-databind in /CromwellRefdiskManifestCreator (broadinstitute#6935) Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.13.2.2 to 2.13.4.1. - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) --- updated-dependencies: - dependency-name: com.fasterxml.jackson.core:jackson-databind dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * WX-808 Host allowlist for HTTP imports (broadinstitute#6938) * `hostAllowlist` that allows everything * Refactor * Stick allow list in HttpResolver * Better default config * Allow list tests * Make it build Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * Update commons text to 1.10.0 (broadinstitute#6937) * WX-751 Token refresh signal for monitoring (broadinstitute#6939) * Log messages * `DEBUG` -> `INFO` * WX-744 Optionally rewrite blob paths to appear as local paths (broadinstitute#6941) * Modify blob paths for TES * Make blob transformation configurable * Update supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * Apply PR feedback in second place Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * Update changelog for wdl http allow list (broadinstitute#6944) * WM-1491 Fixing Cromwell-client (broadinstitute#6943) * More updated client for use in cbas * Removing excess code * Fix client build script (broadinstitute#6945) * WX-837: Remove CWL references from documentation (broadinstitute#6949) * wx-837 removed cwl references in markdown doc files * wx-837 removed cwlParsingOverview.md, updated mkdocs.yml * wx-837 updated cromwell.yaml, generated new RESTAPI file * WX-728 Add configurable WSM client to Cromwell (broadinstitute#6948) * Dependencies * Compiles but no tests * Formatting * Moar exclusions * Update to latest WSM * Add additional dependency * We need some UUID here to make the request * Formatting * Clarify what is fake * Formatting * Use our own version of Jersey and Jackson stuff * Port-in Khalid's changes (thank you!) Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org> * Test longevity Don't break the test if someone decides to add a cert to `ws.org` * Cleanup * Cleanup * Cleanup * Adjust TES config file for CI Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org> * CROM-6554: Removed PAPIv1 references from doc (broadinstitute#6950) * crom-6554 removed references to PAPI v1 from doc * crom-6554 pr feedback, reworded doc to use example conf as a starting point * WX-833 Real Azure DRS Credentials (broadinstitute#6952) * Remove B2C reference from name * Get token for current user rather than getting from KeyVault * Remove KeyVault config for engine * Remove KeyVault config for DRSLocalizer * Remove KeyVault dependency * Remove KeyVault support from localizer repo template * Cleaned up and working Azure token acquisition for engine * Collapse localizer's AccessTokenStrategy into DrsCredentials * Cleanup * WX-853 Remove most CWL (broadinstitute#6955) * WX-696 Enable getting SAS token from WSM (broadinstitute#6954) * WX-696 Enable getting SAS token from WSM * Wire container resource id from config * Move resource-container-id config path * First pass at config for WSM * Remove unused singleton config * Tests for new config * Fix config parsing * Modified b2c token to be provided each time * Remove singletonConfig arg from factory * Restore types to factory configs * Clean up comments and empty token default * Default to config b2c before searching environment * Fix token default on api client * Fix test * Refactor error handling for when there is no token * Remove token constructor arg for clientProvider * Move configs to global singleton config * Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala * default -> override * Add override token to test * Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * Parentheses * Reduce token timeout * Move AzureCredentials to separate file * Make AzureCredentials an object * WSM token cleanup * Config refactor (broadinstitute#6960) Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * Initial blob token documentation * Refine language in BlobSasTokenGenerator * Update comment and formatting Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * WX-853 Remove CWL language factory, Centaur runner (broadinstitute#6961) * WX-842 Add Pact Dependency for Cromwell (broadinstitute#6962) * WX-842 Add Pact Dependency for Cromwell * Remove incomplete test spec * Initial Pact Test * Fix pact so it compiles * Add breadcrumb comment and clean up * ID-125 Add support for drshub, rename all the things (broadinstitute#6959) * Add support for drshub, rename all the things * fallback to martha if resolver is not in config * WX-867 Translate crc32c hashes to b64 for getm (broadinstitute#6970) * Translate crc32c hashes to b64 for getm * Update tests * Remove obsolete b64 handling for md5, centralize hex validation * Restore old test, fix other test * WX-843 Workflow failure reason should accurately indicate issues opening blob filesystem (broadinstitute#6965) * WX-859 Accept workflow execution identity in config (broadinstitute#6967) * WX-892 Trim down `ValueStore` logging to prevent OOMs (broadinstitute#6981) * Add Nirvana 3.18.1 reference image test, minor cleanup [VS-705] (broadinstitute#6975) * WX-863 Turn off Azure NIO logging (broadinstitute#6982) * Turn off Azure NIO logging * Poke Travis * WM-1616: Allow repeating attempts at initialization (take 2) (broadinstitute#6985) * WX-878 Single shared BlobFileSystemManager (broadinstitute#6986) * Make BlobFileSystemManager shared across all BlobPathBuilders * Update TES conf file to reflect new singleton config * Shell escape reference image files [VS-796] [WX-910] (broadinstitute#6989) * WX-769 `disks` compatibility for TES backend (broadinstitute#6991) * Update FiveMinuteIntro.md (broadinstitute#6994) * WX-906 Sbt Unit Tests as Github Actions (broadinstitute#6992) * WX-926 Support falling back to OCI Manifest Format (broadinstitute#7003) * WX-926 Support falling back to OCI Manifest Forma * Only mount reference disks if requested [WX-925] (broadinstitute#7001) * [WM-1646] Add missing fields for `WorkflowDescription` for WomTool /describe endpoint to Swagger (broadinstitute#7004) * WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status (broadinstitute#6980) * WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status * Address feedback * Address feedback (broadinstitute#6997) * Address additional feedback (broadinstitute#7000) * Fix copy/paste error (broadinstitute#7005) * Address additional feedback * Fix copy/paste error * Trigger CI --------- Co-authored-by: Blair Murri <BMurri@users.noreply.github.com> Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * Centaur reference image test should validate symlinks [VS-796] (broadinstitute#6996) * WX-903 Pre-GHA test suite disablement * WX-877 Update CHANGELOG for release 85 (broadinstitute#7011) * Update cromwell version from 85 to 86 * WX-905 (broadinstitute#7012) Co-authored-by: Thomas Wiseman <twiseman@wm275-315.lan> Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org> Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * WX-719 modernize `cromwell-publish` image (broadinstitute#7013) * WX-930 Add `CODEOWNERS` file (broadinstitute#7015) * Update FiveMinuteIntro: Java 11 requirement (broadinstitute#6830) Co-authored-by: Adam Nichols <aednichols@gmail.com> * WX-952 Fix Cromwell version update in Helm chart (broadinstitute#7014) * Reference Disk Manifest Builder App Test (broadinstitute#7017) * [WM-1696] Update Womtool (broadinstitute#7019) Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org> * WX-958 write_map() should write its last entry with a newline (broadinstitute#7022) * Centaur slurm (broadinstitute#7083) * Remove Deploy Key (broadinstitute#7084) * WX-950 Set user agent in Java client (broadinstitute#7087) * WX-950 Upgrade Azure libs to probably fix extraneous log (broadinstitute#7088) * WX-984 Revert "WX-950 Upgrade Azure libs to probably fix extraneous log" (broadinstitute#7090) * Add cron run integrations each day of the week (broadinstitute#7089) * 85 release (#28) * Update cromwell version from 83 to 84 * BW-1255 Implement POST /runs endpoint (broadinstitute#6779) * Adding route * Fixing HTTP method error * All formFields made optional * A compliling state * Saving * Saving * All three endpoints functioning as expected; updated RESTAPI.md * Updated response for submission from 200 to 201 to pass tests * Test submission response * Moved updated submission response to askSubmit * test * updating RESTAPI.md * saving * Adding utility file for submitRequest * cleanup * Update awssdkv from 2.17.152 to 2.17.194 (broadinstitute#6814) * BW-1305 Swagger Update (broadinstitute#6818) * Properly documenting metadataArchiveStatus in WorkflowQueryResult model * Update docs * BT-710 Add configs for BlobPathBuilderFactory (broadinstitute#6817) BT-710 Add configs for BlobPathBuilderFactory * BW-1305 Make "name" optional in workflow query response (broadinstitute#6821) * BT-724 Fix BlobPathBuilder failing on retrieving existing filesystem (broadinstitute#6816) Modify blobPathBuilder to fallback to creating a filesystem if one is not found * Logging updates: (broadinstitute#6813) * [BT-698] first pass on BlobTokenGenerator with E2E test (broadinstitute#6824) * first pass on BlobTokenGenerator with E2E test * update BlobPathBuilder constructor args in test * account -> container level client * [BT-687] specify correct types (broadinstitute#6829) * specify correct types * fix test with new type * remove type declarations in function call * remove unnecessary sas-token config * BW-1206 - Combine all Wes Endpoints & add Tests (broadinstitute#6833) * Add tests, getting frid of WesRunRoutes.scala * wesWorkflowId fix, ec implicits errors gone * Refactoring path for GET /runs * Indentation fix * Commit to rollback * Revert "Indentation fix" This reverts commit 63fc484. * PR trigger * Optimize imports * Missed import * BW-1354 - Porting CBAS preliminary step (broadinstitute#6837) * Getting rid of shared utility file; Adding/Updating WES version of submit. * Edit spec file * Adding Wes-like error * BW-1378 Addl CromIAM user enablement checks (broadinstitute#6826) * Update cromwell version from 84 to 85 * BW-1393 Release doc updates (broadinstitute#6839) * BT-732 Checksum validation for blobs read by engine (broadinstitute#6838) * Draft support for optional FileHash * Draft getMd5 for BlobPath * Resolve non-parallel IO to fix tests * Checksum validation for BlobPath * Nicer error message * Test for missing Blob hash * Break attr acquisition into separate method * Cleanup, comments * In-progress tests of blob hash command * Remove test * Remove unused import * BT-711 Refresh SAS token for filesystem on expiry (broadinstitute#6831) * BT-711 Refresh SAS token for filesystem on expiry * Rough cut of token refresh using exceptions * Ignore tests, and minor cleanup * Remove stray line * Draft of manager class for handling expiring file systems * Style fixes * Refactor of blobfilesystemManager and tests covering its functionality * Refined tests to validate close filesystem as separate unit * Ignore connected tests * Clean up of some things * Refactor BlobFileSystemManager to separate file, and some other cleanup * Some additional scala-ifying * Small cleanup * Correcting imports * trigger tests * trigger tests * Batch 1 of scala steward updates (broadinstitute#6903) * Batch 1 of scala steward updates * Rollback snakeYAML * Attempt 3, with only the passing dependancies * Revert google API and Big Query udpates * Winding back other google deps * rollback remaining google updates * trigger tests * trigger tests * [BW-1398] Migrate PKs to BIGINT (broadinstitute#6907) * BT-745 Batch 2 of scala steward updates (broadinstitute#6906) * Update SBT to 2.0.0 * Fix sbt-git import * Update mouse to 1.0.11 * Update rhino 1.7.14 * SUP-692 Retry with more memory after RC 137 (broadinstitute#6912) * Reorder execution result checks so 137 can retry with more memory * Test for memory retry after 137 RC * Fix test expectations * Make memory retry checks consistent * Revert changes to existing test * Rename retryWithMoreMemory to outOfMemoryDetected * Scala steward updates batch 3 (broadinstitute#6913) * Scala steward updates batch 3 * WX-745 Batch 4 scala steward updates (broadinstitute#6916) * WX-746 Localize all DRS inputs in a single Action (broadinstitute#6914) Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * WX-755 Build all images instead of just Cromwell (broadinstitute#6919) * WX-755 Add `isRelease` option for Docker builds (broadinstitute#6923) * WX-755 Cromwell/CromIAM automatically board train (broadinstitute#6924) * WX-755 Fix environment variable syntax (broadinstitute#6926) * WX-743 Enable TES task creation with BlobPaths (broadinstitute#6921) * Give blob SAS tokens write permission * Case class wrapper for subscription id * Resolve duplicate container name in absolute BlobPath * Ignored test demonstrating correct absolute path generation * Update filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobPathBuilderSpec.scala Co-authored-by: Brian Reilly <breilly@broadinstitute.org> * PR feedback Co-authored-by: Brian Reilly <breilly@broadinstitute.org> * [WX-765] Update snakeyaml to 1.33 (broadinstitute#6927) * update snakeyaml to 1.33 * Don't use deprecated no-arg Constructor constructor Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * WM-1414 Refactoring WesRunLog to omit Cromwell's "workflowLog" object (broadinstitute#6925) * Upgrade Postgres to 42.4.1 (broadinstitute#6932) * WX-735 Fix incorrect and/or nondeterministic filesystem ordering (broadinstitute#6930) * WX-772 Update Scala to 2.13.9 (broadinstitute#6928) * Update Scala to 2.13.9 * Try updating sbt-scoverage * Does this version exist anywhere we can see? * This version actually exists * Update library version to remove conflict * Codegen version * Fix fun new 2.13.9 compiler errors * Resolve warnings * Newest Scala? * I guess not * Does this please Travis? * force ci * Back out changes to generated code Co-authored-by: Adam Nichols <aednichols@gmail.com> * WX-781 Bump jackson-databind in /CromwellRefdiskManifestCreator (broadinstitute#6935) Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.13.2.2 to 2.13.4.1. - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) --- updated-dependencies: - dependency-name: com.fasterxml.jackson.core:jackson-databind dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * WX-808 Host allowlist for HTTP imports (broadinstitute#6938) * `hostAllowlist` that allows everything * Refactor * Stick allow list in HttpResolver * Better default config * Allow list tests * Make it build Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * Update commons text to 1.10.0 (broadinstitute#6937) * WX-751 Token refresh signal for monitoring (broadinstitute#6939) * Log messages * `DEBUG` -> `INFO` * WX-744 Optionally rewrite blob paths to appear as local paths (broadinstitute#6941) * Modify blob paths for TES * Make blob transformation configurable * Update supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * Apply PR feedback in second place Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * Update changelog for wdl http allow list (broadinstitute#6944) * WM-1491 Fixing Cromwell-client (broadinstitute#6943) * More updated client for use in cbas * Removing excess code * Fix client build script (broadinstitute#6945) * WX-837: Remove CWL references from documentation (broadinstitute#6949) * wx-837 removed cwl references in markdown doc files * wx-837 removed cwlParsingOverview.md, updated mkdocs.yml * wx-837 updated cromwell.yaml, generated new RESTAPI file * WX-728 Add configurable WSM client to Cromwell (broadinstitute#6948) * Dependencies * Compiles but no tests * Formatting * Moar exclusions * Update to latest WSM * Add additional dependency * We need some UUID here to make the request * Formatting * Clarify what is fake * Formatting * Use our own version of Jersey and Jackson stuff * Port-in Khalid's changes (thank you!) Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org> * Test longevity Don't break the test if someone decides to add a cert to `ws.org` * Cleanup * Cleanup * Cleanup * Adjust TES config file for CI Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org> * CROM-6554: Removed PAPIv1 references from doc (broadinstitute#6950) * crom-6554 removed references to PAPI v1 from doc * crom-6554 pr feedback, reworded doc to use example conf as a starting point * WX-833 Real Azure DRS Credentials (broadinstitute#6952) * Remove B2C reference from name * Get token for current user rather than getting from KeyVault * Remove KeyVault config for engine * Remove KeyVault config for DRSLocalizer * Remove KeyVault dependency * Remove KeyVault support from localizer repo template * Cleaned up and working Azure token acquisition for engine * Collapse localizer's AccessTokenStrategy into DrsCredentials * Cleanup * WX-853 Remove most CWL (broadinstitute#6955) * WX-696 Enable getting SAS token from WSM (broadinstitute#6954) * WX-696 Enable getting SAS token from WSM * Wire container resource id from config * Move resource-container-id config path * First pass at config for WSM * Remove unused singleton config * Tests for new config * Fix config parsing * Modified b2c token to be provided each time * Remove singletonConfig arg from factory * Restore types to factory configs * Clean up comments and empty token default * Default to config b2c before searching environment * Fix token default on api client * Fix test * Refactor error handling for when there is no token * Remove token constructor arg for clientProvider * Move configs to global singleton config * Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala * default -> override * Add override token to test * Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * Parentheses * Reduce token timeout * Move AzureCredentials to separate file * Make AzureCredentials an object * WSM token cleanup * Config refactor (broadinstitute#6960) Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * Initial blob token documentation * Refine language in BlobSasTokenGenerator * Update comment and formatting Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * WX-853 Remove CWL language factory, Centaur runner (broadinstitute#6961) * WX-842 Add Pact Dependency for Cromwell (broadinstitute#6962) * WX-842 Add Pact Dependency for Cromwell * Remove incomplete test spec * Initial Pact Test * Fix pact so it compiles * Add breadcrumb comment and clean up * ID-125 Add support for drshub, rename all the things (broadinstitute#6959) * Add support for drshub, rename all the things * fallback to martha if resolver is not in config * WX-867 Translate crc32c hashes to b64 for getm (broadinstitute#6970) * Translate crc32c hashes to b64 for getm * Update tests * Remove obsolete b64 handling for md5, centralize hex validation * Restore old test, fix other test * WX-843 Workflow failure reason should accurately indicate issues opening blob filesystem (broadinstitute#6965) * WX-859 Accept workflow execution identity in config (broadinstitute#6967) * WX-892 Trim down `ValueStore` logging to prevent OOMs (broadinstitute#6981) * Add Nirvana 3.18.1 reference image test, minor cleanup [VS-705] (broadinstitute#6975) * WX-863 Turn off Azure NIO logging (broadinstitute#6982) * Turn off Azure NIO logging * Poke Travis * WM-1616: Allow repeating attempts at initialization (take 2) (broadinstitute#6985) * WX-878 Single shared BlobFileSystemManager (broadinstitute#6986) * Make BlobFileSystemManager shared across all BlobPathBuilders * Update TES conf file to reflect new singleton config * Shell escape reference image files [VS-796] [WX-910] (broadinstitute#6989) * WX-769 `disks` compatibility for TES backend (broadinstitute#6991) * Update FiveMinuteIntro.md (broadinstitute#6994) * WX-906 Sbt Unit Tests as Github Actions (broadinstitute#6992) * WX-926 Support falling back to OCI Manifest Format (broadinstitute#7003) * WX-926 Support falling back to OCI Manifest Forma * Only mount reference disks if requested [WX-925] (broadinstitute#7001) * [WM-1646] Add missing fields for `WorkflowDescription` for WomTool /describe endpoint to Swagger (broadinstitute#7004) * WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status (broadinstitute#6980) * WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status * Address feedback * Address feedback (broadinstitute#6997) * Address additional feedback (broadinstitute#7000) * Fix copy/paste error (broadinstitute#7005) * Address additional feedback * Fix copy/paste error * Trigger CI --------- Co-authored-by: Blair Murri <BMurri@users.noreply.github.com> Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * Centaur reference image test should validate symlinks [VS-796] (broadinstitute#6996) * WX-903 Pre-GHA test suite disablement * WX-877 Update CHANGELOG for release 85 (broadinstitute#7011) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> Co-authored-by: Katrina P <68349264+kpierre13@users.noreply.github.com> Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com> Co-authored-by: Christian Freitas <christian.j.freitas@gmail.com> Co-authored-by: Saloni Shah <salonipshah11@gmail.com> Co-authored-by: kshakir <kshakir@broadinstitute.org> Co-authored-by: mspector <michael.l.spector@gmail.com> Co-authored-by: Adam Nichols <anichols@broadinstitute.org> Co-authored-by: Brian Reilly <breilly@broadinstitute.org> Co-authored-by: Adam Nichols <aednichols@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Justin Variath Thomas <JVThomas@users.noreply.github.com> Co-authored-by: Christian Freitas <cfreitas@broadinstitute.org> Co-authored-by: Trevyn Langsford <tlangs@broadinstitute.org> Co-authored-by: Miguel Covarrubias <mcovarr@users.noreply.github.com> Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com> Co-authored-by: Tom Wiseman <tom.h.wiseman@gmail.com> Co-authored-by: Blair Murri <BMurri@users.noreply.github.com> * Remove Yaml (broadinstitute#7098) * Azure blob read/write integration test (broadinstitute#7024) Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org> * Goodbye Travis (broadinstitute#7100) * Single workflow runner (broadinstitute#7106) * WX-975 Revert special TES path munging (broadinstitute#7103) * Centaur blob filesystem (broadinstitute#7104) * WX-1010 Update Github Actions set output syntax (broadinstitute#7111) * WX-1001 Upgrade JSON Small and Fast Parser (broadinstitute#7116) * WX-1015 Override glob behavior for TES to use absolute path (broadinstitute#7120) * WX-990 Make TES request backoff behavior configurable (broadinstitute#7122) * WX-1103 Revert CI to old TES polling behavior (broadinstitute#7126) * Develop aws (#29) * stuck on globbing * efs works, no callcaching * update readme * extended EFS support * fix for globbing in nested scatters * updated config for globbing, to prevent issues with empty folders * WX-1103 Accelerate TES CI (broadinstitute#7130) * Fix Horicromtal Deadlock Test (broadinstitute#7131) * WM-1963: Validate PRs begin with Jira tags (broadinstitute#7127) * WX-1106 Add logging for failed docker manifest pulls (broadinstitute#7135) Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * [WX-1108] Disable drs tests (broadinstitute#7145) * ID-377 Setup pact for consumer contract testing. (broadinstitute#7123) Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * WX-1092 Support `size` engine function for public HTTP files (broadinstitute#7128) * WX-1105 Fix interpretation of full http blob paths (broadinstitute#7138) Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * Revert "ID-377 Setup pact for consumer contract testing." (broadinstitute#7146) * WX-966 Add Cascades, remove Directory from Biscayne (broadinstitute#7105) * WX-966 Post-merge branch cleanup + test fix (broadinstitute#7149) * WX-1122 Enable Azure ApplicationInsights (broadinstitute#7143) * [WX-1108] Add tests back (broadinstitute#7148) * [WX-1136] Docker Build Test (broadinstitute#7151) * [WX-1136] Self Hosted (broadinstitute#7152) * WX-1145 Fix minor regression introduced in WDL 1.1 foundation (broadinstitute#7153) * ID-377 Setup pact for consumer contract testing. Retry (broadinstitute#7147) Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org> Co-authored-by: Ivan <ichang@broadinstitute.org> * WX-1133 Shorten TES localized blob paths (broadinstitute#7150) * WX-1154 Fix Slack Messaging (broadinstitute#7159) * efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs (#32) * WX-1160 Fix bad MariaDB version assumptions (broadinstitute#7160) * WX-1126 Upgrade to modern Python, 3.8 not available in package repo anymore (broadinstitute#7164) * WX-1122 Use legacy AppInsights to get better control over logging (broadinstitute#7157) * WX-1137 Replace 4-byte unicode chars in PAPI event descriptions (broadinstitute#7166) * WX-1101 Update library versions to support batch (broadinstitute#7155) Co-authored-by: Adam Nichols <anichols@broadinstitute.org> Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * WX-1195 Update Azure identity libs (broadinstitute#7173) * WX-1114 Initial inclusion of Azure NIO 'fork' (broadinstitute#7168) * WX-1110[risk=low] Added endpoint to fetch failed tasks by root workflow id (broadinstitute#7165) * WX-1210 Added JIRA ID for Cromwhelm auto commit message (broadinstitute#7180) * WX-1179 DRS image build updates, remove self-hosted runners (broadinstitute#7179) * WX-1210-action-fix Use PR title to find JIRA ID for cromwhelm commit message (broadinstitute#7184) * WX-1110-query-fix Corrected Query to pull in attributes outside of executionStatus and backendStatus (broadinstitute#7185) * WX-1179 GCP Batch new base (broadinstitute#7177) Co-authored-by: dspeck <dspeck@burwood.com> Co-authored-by: jarroyo28 <johnarroyo45@gmail.com> Co-authored-by: Jacob Jennings <53196131+jakejennings@users.noreply.github.com> Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org> Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com> Co-authored-by: Christian Freitas <cfreitas@broadinstitute.org> Co-authored-by: Tristan Garwood <tristan.garwood@gmail.com> * WM-2100: Update terra-helmfile on publish (broadinstitute#7187) * WM-2100: Correct commit and PR description of terra-helmfile auto update (broadinstitute#7188) * WX-1230 Ignore `invalidate_bad_caches_use_good_local.test` (broadinstitute#7197) * WX-1078 ACR support (broadinstitute#7192) * WX-1179 Enable GCP Batch Integration Tests (broadinstitute#7199) Co-authored-by: Adam Nichols <aednichols@gmail.com> Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * WX-1179 GCP Batch Docs Update (broadinstitute#7196) Co-authored-by: Jacob Jennings <53196131+jakejennings@users.noreply.github.com> Co-authored-by: Adam Nichols <aednichols@gmail.com> Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * ID-734 Increase Timeout for DRSHub Communication (broadinstitute#7198) * [WX-1156] internal_path_prefix for TES 4.4 (broadinstitute#7190) * [WM-2184] Remove brackets from Jira ID (broadinstitute#7206) * WX-1153 Azure blob read md5 from metadata for large files (broadinstitute#7204) Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * WX-1156 Fix internal_path_prefix (broadinstitute#7208) Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * WX-1256 Temporarily turn off engine hashing for blob files (broadinstitute#7209) Co-authored-by: Adam Nichols <aednichols@gmail.com> * WX-1173 Reopen filesystem for blob storage outside workspace (broadinstitute#7178) Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org> Co-authored-by: Adam Nichols <aednichols@gmail.com> * WX-1174 Adjust NIO Copy functionality (broadinstitute#7207) Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * [WX-1168] TES Log Paths (broadinstitute#7210) * Develop aws (#34) * efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs * add deployment manual, fix issue with empty disks * update documentation * update documentation * update documentation * WX-1264 Don't expire an unexpirable filesystem (broadinstitute#7216) * [WX-495] DRS Parallel Downloads (broadinstitute#7214) * WX-1225 Print TES error messages to job logger (broadinstitute#7220) * WX-1217 Workflow completion callback (broadinstitute#7213) Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com> * [WM-2199] Changeset for modifying ownership of tables for WORKFLOWS app (broadinstitute#7218) Co-authored-by: Chris Llanwarne <cjllanwarne@gmail.com> * [WX-1234] Update CHANGELOG.md (broadinstitute#7227) Merging past CI since this is doc only. * WM-2252: Configurable metadata write role (broadinstitute#7225) Co-authored-by: dvoet <dvoet@users.noreply.github.com> * Update cromwell version from 86 to 87 * WX-1282 Update failedJobs Query to use `lo_get` instead of INNER JOIN against pg_largeobject (broadinstitute#7228) * [WX-1234] Update Release Process with Docker Instructions (broadinstitute#7231) Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * WM-2294: Allow role-setter action to run on change (broadinstitute#7233) * WM-2296: Callback should supply fully qualified output names (broadinstitute#7234) * [WX-499] DRS Parallel Downloads Follow-up (broadinstitute#7229) * WX-1318 gcp batch: Add GPU driver install (broadinstitute#7235) Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * WX-1232 Include useful workflow ids in TES tags (broadinstitute#7221) * Options to publish status only (#36) * add options to publish status only * updated readme.md --------- Co-authored-by: quekx <quekx@gene.com> * WX-1307 Azure E2E test (broadinstitute#7239) Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * Fix aws unit tests (#39) * checkpoint * fix ecr and batch tests * fix AwsBatchJobSpec.scala --------- Co-authored-by: quekx <quekx@gene.com> * return bucket directly instead of listing and checking it (#38) Co-authored-by: quekx <quekx@gene.com> * WX-1340 GCP Batch: Mount with extra colon issue and multiple zones support (broadinstitute#7240) Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * WX-1339 Make `throwExceptionOnExecuteError` false for PAPI aborts (broadinstitute#7245) * WX-1338 Fix cron invocation of E2E test (broadinstitute#7244) * WX-1341 Better logging when a runner stops picking up new workflows (broadinstitute#7246) * Add evaluteOnExit for aws batch retry (#40) Co-authored-by: quekx <quekx@gene.com> * [WX-1260] Acquire sas token from task runner (broadinstitute#7241) Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * [WX-1183] Ice Lake (broadinstitute#7252) * WX-1333 Improve logging visibility for load management (broadinstitute#7253) * [WX-1391] Fix Bash Bug (broadinstitute#7326) * [WX-1393] Add Content Length to Curl request (broadinstitute#7328) * [WX-1346] Scalafmt (broadinstitute#7257) * WX-1351 Speed up `Centaur Horicromtal PapiV2 Beta` (broadinstitute#7329) * WX-1351 CI CWL cleanup (broadinstitute#7327) * WX-1351 Remove slow/misbehaving localization test (broadinstitute#7330) * WX-1351 Split out restart tests (broadinstitute#7333) * [WX-1345] Automatic Token Acquisition for TES Config (broadinstitute#7256) * WX-1387 Remove unused case classes (broadinstitute#7334) * [WX-1394] Upgrade Logback Core to address CVE-2023-6378 (broadinstitute#7332) * WX-1351 Another round of CI cleanup (broadinstitute#7336) * [WX-1395] Scalafmt GitHub Action (broadinstitute#7337) * WX-1351 SBT compile speedup (broadinstitute#7339) * WX-1407 Fix contract tests (broadinstitute#7340) Co-authored-by: Ivan <ichang@broadinstitute.org> * WX-1409 Java 17 (broadinstitute#7342) * Revert "WX-1409 Java 17 (broadinstitute#7342)" (broadinstitute#7343) * WX-1409 Restore Java 17 (broadinstitute#7342) (broadinstitute#7344) * Revert "WX-1409 Java 17" (broadinstitute#7346) * [WX-1395] Scala formatter Github Action (broadinstitute#7341) Co-authored-by: ScalaFmt Fixer <ScalaFmtFixer@fakeDomain.com> Co-authored-by: Broad Bot <broadbot@broadinstitute.org> * Improved tagging support (#37) * efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs * add deployment manual, fix issue with empty disks * update documentation * update documentation * update documentation * support for tagging instances and volumes used in jobs, some support for spaces in file paths * corrected workflow id in tagging * redirect exit code 137 to retry-with-more-memory routine --------- Co-authored-by: Henrique Ribeiro <henriqueribeiro@users.noreply.github.com> * add gpu count (#41) * add gpu count * fix typo --------- Co-authored-by: quekx <quekx@gene.com> Co-authored-by: Henrique Ribeiro <henriqueribeiro@users.noreply.github.com> * WX-1385 Reject blob URLs with external SAS tokens as unparsable (broadinstitute#7347) * WX-1411 Require preinstalled `jq` and `curl` for just-in-time SAS feature (broadinstitute#7350) * WX-1396 Cromwell client allow additional properties (broadinstitute#7352) * WM-2428: Include full error context when failing to abort TES jobs (broadinstitute#7354) * WX-1420 Fix GCP Batch label regex restriction (broadinstitute#7355) Co-authored-by: Beibei Chen <beibei@formbio.com> Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * WX-1385 Remove SAS tokens from TES input log printouts (broadinstitute#7358) * [WX-1184] PostgreSQL Docker Image for Local Cromwell (broadinstitute#7172) Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * WX-1444 Use MySQL LTS in DBMS tests (broadinstitute#7360) * WX-1449 Add `latest` Docker tag (broadinstitute#7362) * WX-767 Upgrade Cloud SDK to 461.0.0 (broadinstitute#7361) * WX-1443 Adopt `gcloud storage` for localization only (broadinstitute#7359) * WX-964 suffix() (broadinstitute#7363) * WX-1417 New database role strategy (broadinstitute#7366) * [WM-2291] Callback API contract tests between Cromwell and CBAS (broadinstitute#7251) Co-authored-by: Chris Llanwarne <cjllanwarne@gmail.com> Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * WX-1445 Update docker image regex to handle python:3 correctly (broadinstitute#7367) * WX-1252 Runtime attributes cleanup – CWL runtime attributes (broadinstitute#7370) * WX-1485 Upgrade postgres lib (broadinstitute#7371) * WM-2454: Private GitHub support on describe api (broadinstitute#7365) * WX-1252 Runtime attributes cleanup – CWL runtime environment (broadinstitute#7369) * WX-1462 POSIX-flavored sub() (broadinstitute#7374) * [WX-963] Unzip Engine Function (broadinstitute#7368) * WX-1489 Hopefully reduce CI flakiness by modernizing deadlock test (broadinstitute#7376) * [WX-965] quote() and squote() engine functions. (broadinstitute#7375) Co-authored-by: Ryan Saperstein <rsaperst@wmd55-b47.hsd1.ma.comcast.net> Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * [WX-1317] Remove Akka 'server' header from all HTTP responses (broadinstitute#7379) * WM-2461] Add support for running private workflows on Azure (broadinstitute#7373) * WX-1252 Per-backend runtime attributes (broadinstitute#7380) * ID-347 Remove Martha References (broadinstitute#7384) * WX-1461 Remove `womtool upgrade` command (broadinstitute#7382) * WX-1488 Supply cloud platform when making DRS requests (broadinstitute#7381) * WX-757 Fix workflow stuck in aborting after WDL type error (broadinstitute#7385) * [WX-1506] Add jobLogger output for subworkflow running (with parent and subworkflow ID) (broadinstitute#7388) Co-authored-by: jlester-msft <47044104+jlester-msft@users.noreply.github.com> * WX-757 Fix `stdout`, `stderr` in workflow body causing crashes (broadinstitute#7386) * WX-1519 Stop leaking DRS Localizer images on every CI run (broadinstitute#7390) * [WX-1460] WDL 1.1 Struct Literal Parsing (broadinstitute#7391) Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> * [WM-2500][WM-2502] Fetch Github token from ECM for importing and running private workflows (broadinstitute#7392) * WX-1542 Fix workflow cancellation (broadinstitute#7398) * WX-1550 Don't pointlessly continue running integration tests (broadinstitute#7400) * WX-1557 Fix default values from Configuring.md and Scaling.md (broadinstitute#7393) Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * WX-1557 Fix actor-factory in Batch101.md (broadinstitute#7377) Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * WX-1557 Add more tests to the GCP Batch backend (broadinstitute#7394) Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * [WX-1531] Struct Literal Type Checking (broadinstitute#7402) * [WX-1468] Implement `returnCodes` runtime attribute (broadinstitute#7389) Co-authored-by: Ryan Saperstein <rsaperst@wmd55-b47.hsd1.ma.comcast.net> Co-authored-by: Ryan Saperstein <rsaperst@wmd55-b47.broadinstitute.org> * [WM-2555] Cromwell -> ECM contract test (broadinstitute#7405) * [WX-1568] Bump Akka max-response-reason-length limit (broadinstitute#7406) * WX-1530 Strip pesky URL bits when creating local paths for HTTP inputs (broadinstitute#7404) * WX-1538 Add more tests to the GCP Batch backend (broadinstitute#7410) * [WX-1605] Fix Codecov Report Uploads (broadinstitute#7418) * [WX-1410] Sanitize 4 byte UTF-8 characters before inserting into METADATA_ENTRY (broadinstitute#7414) * WX-1566 Special Docker build for debugging (broadinstitute#7417) Run `sbt -Dproject.isDebug=true server/docker` * [WX-1448] Add verbose logging and timeout for getm (broadinstitute#7416) Co-authored-by: Adam Nichols <anichols@broadinstitute.org> * WX-1566 Fix Morgan's call cache file hash CPU thrash Cromwell crash (broadinstitute#7419) * WX-1566 Fix RTD so this ticket's docs get published (broadinstitute#7420) * fix womFileType coercion (#47) Co-authored-by: quekx <quekx@gene.com> * PR #44: Multi-aserisk globbing by @kevinRenaersBio. Reapplied selection of changes on aws_develop branch * PR #45 reapplied to aws_develop: support for sharedMemory. skipped EFS fixes for now while waiting for feedback * upstream PR broadinstitute#7488 : fix compilation error due to swagger2markup * Fix issue with caching on EFS by replacing touch with existOrThrow routine * make sure that md5 sibling file of EFS data is newer than main file to be valid * revise sharedmemory attribute to take reguluar GB/MB specification * updated AWS README file --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org> Co-authored-by: Christian Freitas <christian.j.freitas@gmail.com> Co-authored-by: Saloni Shah <salonipshah11@gmail.com> Co-authored-by: Brian Reilly <breilly@broadinstitute.org> Co-authored-by: Adam Nichols <anichols@broadinstitute.org> Co-authored-by: Katrina P <68349264+kpierre13@users.noreply.github.com> Co-authored-by: Adam Nichols <aednichols@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com> Co-authored-by: Justin Variath Thomas <JVThomas@users.noreply.github.com> Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org> Co-authored-by: Christian Freitas <cfreitas@broadinstitute.org> Co-authored-by: Trevyn Langsford <tlangs@broadinstitute.org> Co-authored-by: Miguel Covarrubias <mcovarr@users.noreply.github.com> Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com> Co-authored-by: Tom Wiseman <tom.h.wiseman@gmail.com> Co-authored-by: Blair Murri <BMurri@users.noreply.github.com> Co-authored-by: Thomas Wiseman <twiseman@wm275-315.lan> Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org> Co-authored-by: Stephen Fleming <sjfleming@users.noreply.github.com> Co-authored-by: Henrique Ribeiro <henriqueribeiro@users.noreply.github.com> Co-authored-by: mspector <michael.l.spector@gmail.com> Co-authored-by: Tristan Garwood <tristan.garwood@gmail.com> Co-authored-by: Ivan <ichang@broadinstitute.org> Co-authored-by: dspeck1 <dspeck@burwood.com> Co-authored-by: jarroyo28 <johnarroyo45@gmail.com> Co-authored-by: Jacob Jennings <53196131+jakejennings@users.noreply.github.com> Co-authored-by: Dillon Scott <67511512+dillydally414@users.noreply.github.com> Co-authored-by: Chris Llanwarne <cjllanwarne@gmail.com> Co-authored-by: dvoet <dvoet@users.noreply.github.com> Co-authored-by: xquek <46759870+xquek@users.noreply.github.com> Co-authored-by: quekx <quekx@gene.com> Co-authored-by: ScalaFmt Fixer <ScalaFmtFixer@fakeDomain.com> Co-authored-by: Broad Bot <broadbot@broadinstitute.org> Co-authored-by: Beibei Chen <beibei@formbio.com> Co-authored-by: Maria Yazykova <95451512+zykonda@users.noreply.github.com> Co-authored-by: rsaperst <156102201+rsaperst@users.noreply.github.com> Co-authored-by: Ryan Saperstein <rsaperst@wmd55-b47.hsd1.ma.comcast.net> Co-authored-by: jlester-msft <47044104+jlester-msft@users.noreply.github.com> Co-authored-by: Alexis Hernandez <github@alexitc.com> Co-authored-by: dkj <david.jackson@sanger.ac.uk> Co-authored-by: Ryan Saperstein <rsaperst@wmd55-b47.broadinstitute.org> Co-authored-by: Alexis Hernandez <alexis22229@gmail.com>

port changes to this branch

6ba6ec1

salonishah11 requested a review from a team as a code owner February 23, 2024 22:56

salonishah11 commented Feb 23, 2024

View reviewed changes

salonishah11 added 2 commits February 27, 2024 10:11

Merge branch 'develop' into sps_run_private_workflows

16d47a2

Merge branch 'develop' into sps_run_private_workflows

21e8180

cjllanwarne self-requested a review February 27, 2024 20:24

cjllanwarne reviewed Feb 27, 2024

View reviewed changes

jgainerdewar reviewed Feb 27, 2024

View reviewed changes

salonishah11 added 3 commits March 4, 2024 17:29

move importAuthProvider to GithubAuthVendingSupport

57eb0fd

Merge branch 'develop' into sps_run_private_workflows

2d3f09c

remove extra newline

81bad06

salonishah11 requested review from jgainerdewar and cjllanwarne March 5, 2024 14:38

jgainerdewar approved these changes Mar 5, 2024

View reviewed changes

Merge branch 'develop' into sps_run_private_workflows

b9adb74

cjllanwarne approved these changes Mar 5, 2024

View reviewed changes

salonishah11 merged commit 79c2bff into develop Mar 7, 2024
34 checks passed

salonishah11 deleted the sps_run_private_workflows branch March 7, 2024 19:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WM-2461] Add support for running private workflows on Azure #7373

WM-2461] Add support for running private workflows on Azure #7373

salonishah11 commented Feb 23, 2024 •

edited

Loading

salonishah11 Feb 23, 2024

cjllanwarne Feb 27, 2024

cjllanwarne Feb 27, 2024

salonishah11 Feb 27, 2024

cjllanwarne Feb 27, 2024

cjllanwarne Feb 27, 2024

jgainerdewar Feb 27, 2024

salonishah11 Feb 27, 2024 •

edited

Loading

salonishah11 Feb 27, 2024

jgainerdewar Feb 29, 2024

salonishah11 Feb 29, 2024

jgainerdewar Mar 4, 2024

cjllanwarne Feb 27, 2024

cjllanwarne Feb 27, 2024

jgainerdewar Feb 27, 2024

salonishah11 Feb 27, 2024

jgainerdewar Feb 27, 2024

jgainerdewar Feb 27, 2024

jgainerdewar Feb 27, 2024

salonishah11 Feb 27, 2024

jgainerdewar Feb 27, 2024

salonishah11 Feb 27, 2024

jgainerdewar Feb 29, 2024

salonishah11 Feb 29, 2024

cjllanwarne Mar 4, 2024

salonishah11 Mar 4, 2024

jgainerdewar Feb 27, 2024

jgainerdewar left a comment

jgainerdewar Mar 5, 2024

salonishah11 Mar 5, 2024

cjllanwarne left a comment

WM-2461] Add support for running private workflows on Azure #7373

WM-2461] Add support for running private workflows on Azure #7373

Conversation

salonishah11 commented Feb 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

salonishah11 Feb 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jgainerdewar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjllanwarne left a comment

Choose a reason for hiding this comment

salonishah11 commented Feb 23, 2024 •

edited

Loading

salonishah11 Feb 27, 2024 •

edited

Loading