Skip to content
This repository has been archived by the owner on Aug 13, 2024. It is now read-only.

Make Factotum go as far through the DAG as it can (fail fast should be off by default) #61

Closed
alexanderdean opened this issue Aug 26, 2016 · 0 comments
Closed

Make Factotum go as far through the DAG as it can (fail fast should be off by default) #61

alexanderdean opened this issue Aug 26, 2016 · 0 comments
Assignees
@ninjabear
Labels
code-complete
Milestone
Version 0.3.0

Comments

@alexanderdean
Copy link
Contributor

Consider this DAG:

{
    "schema": "iglu:com.snowplowanalytics.factotum/factfile/jsonschema/1-0-0",
    "data": {
        "name": "snplow2",
        "tasks": [
            {
                "name": "clojure-emr-etl-runner",
                "executor": "shell",
                "command": "/opt/mt-scripts/common/scripts/r77/emr-etl-runner-r77.sh ",
                "arguments": [ "a-clojure" ],
                "dependsOn": [],
                "onResult": {
                    "terminateJobWithSuccess": [ 3, 4 ],
                    "continueJob": [ 0 ]
                }
            },
            {
                "name": "cloudfront-emr-etl-runner",
                "executor": "shell",
                "command": "/opt/mt-scripts/common/scripts/r77/emr-etl-runner-r77.sh ",
                "arguments": [ "a-cloudfront" ],
                "dependsOn": [],
                "onResult": {
                    "terminateJobWithSuccess": [ 3, 4 ],
                    "continueJob": [ 0 ]
                }
            },
            {
                "name": "clojure-storage-loader",
                "executor": "shell",
                "command": "/opt/mt-scripts/common/scripts/r77/storage-loader-r77.sh",
                "arguments": [ "a-clojure" ],
                "dependsOn": [ "clojure-emr-etl-runner" ],
                "onResult": {
                    "terminateJobWithSuccess": [],
                    "continueJob": [ 0 ]
                }
            },
            {
                "name": "cloudfront-storage-loader",
                "executor": "shell",
                "command": "/opt/mt-scripts/common/scripts/r77/storage-loader-r77.sh",
                "arguments": [ "a-cloudfront" ],
                "dependsOn": [ "cloudfront-emr-etl-runner", "clojure-storage-loader" ],
                "onResult": {
                    "terminateJobWithSuccess": [],
                    "continueJob": [ 0 ]
                }
            }
        ]
    }
}

If clojure-emr-etl-runner succeeds but cloudfront-emr-etl-runner fails, then current behavior is that the bottom two tasks won't run, even though clojure-storage-loader does not depend on cloudfront-emr-etl-runner.

In other words, current behavior is "fail fast", which is non-obvious and very difficult to reason about.
@alexanderdean alexanderdean added this to the Version 0.3.0 milestone Aug 26, 2016
@alexanderdean alexanderdean changed the title Factotum does not attempt to get as far through the DAG as it can (fail fast should be off by default) Make Factotum go as far through the DAG as it can (fail fast should be off by default) Sep 16, 2016
ninjabear added a commit that referenced this issue Sep 29, 2016
@ninjabear
Add --dry-run switch to run command (closes #60)
06b405d 
This also does a lot of refactoring, primarily around decoupling
execution strategies from actual execution. This includes adding
a layer over a dag to model which tasks can be run together,
and keep a log of which tasks have been run. This puts the foundation
down for #59 and #61.
ninjabear added a commit that referenced this issue Sep 30, 2016
@ninjabear
Add --dry-run switch to run command (closes #60)
2d59750 
This also does a lot of refactoring, primarily around decoupling
execution strategies from actual execution. This includes adding
a layer over a dag to model which tasks can be run together,
and keep a log of which tasks have been run. This puts the foundation
down for #59 and #61.
ninjabear added a commit that referenced this issue Oct 10, 2016
@ninjabear
Add better task failure handling (closes #61)
9f39971 
"Fail fast" is now off - tasks that fail have their dependencies skipped,
but all unrelated tasks will continue to execute. For example:

            A
           / \
          B   C
          |
          D

prior to this commit, if C failed D would not run, after this commit it will.
ninjabear added a commit that referenced this issue Oct 10, 2016
@ninjabear
Add better task failure handling (closes #61)
2d59d23 
"Fail fast" is now off - tasks that fail have their dependencies skipped,
but all unrelated tasks will continue to execute. For example:

            A
           / \
          B   C
          |
          D

prior to this commit, if C failed D would not run, after this commit it will.
ninjabear added a commit that referenced this issue Oct 10, 2016
@ninjabear
Add better task failure handling (closes #61)
bd00a30 
"Fail fast" is now off - tasks that fail have their dependencies skipped,
but all unrelated tasks will continue to execute. For example:

            A
           / \
          B   C
          |
          D

prior to this commit, if C failed D would not run, after this commit it will.
ninjabear added a commit that referenced this issue Oct 10, 2016
@ninjabear
Add better task failure handling (closes #61)
5f8f6de 
"Fail fast" is now off - tasks that fail have their dependencies skipped,
but all unrelated tasks will continue to execute. For example:

            A
           / \
          B   C
          |
          D

prior to this commit, if C failed D would not run, after this commit it will.
ninjabear added a commit that referenced this issue Oct 11, 2016
@ninjabear
Add better task failure handling (closes #61)
e6dbc20 
"Fail fast" is now off - tasks that fail have their dependencies skipped,
but all unrelated tasks will continue to execute. For example:

            A
           / \
          B   C
          |
          D

prior to this commit, if C failed D would not run, after this commit it will.
ninjabear added a commit that referenced this issue Oct 11, 2016
@ninjabear
Add better task failure handling (closes #61)
1a6f6c6 
"Fail fast" is now off - tasks that fail have their dependencies skipped,
but all unrelated tasks will continue to execute. For example:

            A
           / \
          B   C
          |
          D

prior to this commit, if C failed D would not run, after this commit it will.
@ninjabear ninjabear assigned ninjabear and unassigned ninjabear Oct 21, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@ninjabear ninjabear

Labels
code-complete
Projects
None yet
Milestone
Version 0.3.0
Development

No branches or pull requests
2 participants
@alexanderdean @ninjabear