[FlyteCtl] Update LaunchPlan #394

brucearctor · 2020-07-09T03:21:12Z

No description provided.

pmahindrakar-oss · 2021-06-04T09:04:11Z

Taken care of using this #919

* Add bigquery plugin Signed-off-by: Kevin Su <pingsutw@apache.org> * rerun tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Use go 1.16 Signed-off-by: Kevin Su <pingsutw@apache.org>

Signed-off-by: Nastya Rusina <nastya@union.ai>

* Add bigquery plugin Signed-off-by: Kevin Su <pingsutw@apache.org> * rerun tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Use go 1.16 Signed-off-by: Kevin Su <pingsutw@apache.org>

Signed-off-by: Daniel Rammer <daniel@union.ai>

* Add elastic config args to pytorch proto Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> * Add elastic config message type for torchrun training Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> --------- Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com> Co-authored-by: Ketan Umare <ketan.umare@gmail.com>

* Add elastic config args to pytorch proto Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> * Add elastic config message type for torchrun training Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> --------- Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com> Co-authored-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Add elastic config args to pytorch proto Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> * Add elastic config message type for torchrun training Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> --------- Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com> Co-authored-by: Ketan Umare <ketan.umare@gmail.com>

…teorg#394) * Fix docs Signed-off-by: Ankit Goyal <ankit3goyal@gmail.com> * add generated docs Signed-off-by: Ankit Goyal <angoyal@linkedin.com> --------- Signed-off-by: Ankit Goyal <ankit3goyal@gmail.com> Signed-off-by: Ankit Goyal <angoyal@linkedin.com>

This change adds a new `FastTaskAssignment` message type to fast task protos, which is used to send actor assignment info to admin via task events. While here I also hardened linting on our protos and updates existing protos to conform. This matches the restrictions we have in place in unionai/cloud/idl - [x] Add and run unittests - [x] Run locally and verify worker assignment info comes through in task events ``` ❯ curl -s http://localhost:8088/api/v1/task_executions/flytesnacks/development/f0dcb5e7cae6d47acaf3/n0\?limit\=100 | jq '.taskExecutions[0].closure.metadata.externalResources' [ { "externalId": "", "index": 0, "retryAttempt": 0, "phase": "UNDEFINED", "cacheStatus": "CACHE_DISABLED", "logs": [], "customInfo": { "assignedWorker": "example-2b377271ee80", "environmentId": "flytesnacks_development_example_c183cb6c9868d96" } } ] ❯ curl -s http://localhost:8088/api/v1/task_executions/flytesnacks/development/f0dcb5e7cae6d47acaf3/n1\?limit\=100 | jq '.taskExecutions[0].closure.metadata.externalResources' [ { "externalId": "", "index": 0, "retryAttempt": 0, "phase": "UNDEFINED", "cacheStatus": "CACHE_DISABLED", "logs": [], "customInfo": { "assignedWorker": "example-2b377271ee80", "environmentId": "flytesnacks_development_example_c183cb6c9868d96" } } ] ``` One merged, bring to cloud. Will be a no-op initially, but we can use this in a few dependent tasks to surface logs or look up other fast task worker info. Should this change be upstreamed to OSS (flyteorg/flyte)? If not, please uncheck this box, which is used for auditing. Note, it is the responsibility of each developer to actually upstream their changes. See [this guide](https://unionai.atlassian.net/wiki/spaces/ENG/pages/447610883/Flyte+-+Union+Cloud+Development+Runbook/#When-are-versions-updated%3F). - [ ] To be upstreamed to OSS ref COR-1581 Add custom info field to ExternalResourceInfo Adds a custom info field to `ExternalResourceInfo`. This is an extension of the [ExternalId](https://github.com/unionai/flyte/blob/2b6dfcaedab7ed6a7606d1434211086d064c6560/flyteidl/protos/flyteidl/event/event.proto#L269) field and can be used to pass plugin specific context or identifiers. We will use this to pass fast task worker assignment. N/A Just merge, not used yet Should this change be upstreamed to OSS (flyteorg/flyte)? If not, please uncheck this box, which is used for auditing. Note, it is the responsibility of each developer to actually upstream their changes. See [this guide](https://unionai.atlassian.net/wiki/spaces/ENG/pages/447610883/Flyte+-+Union+Cloud+Development+Runbook/#When-are-versions-updated%3F). - [x] To be upstreamed to OSS ref COR-1581 Closes #393

This change adds a new `FastTaskAssignment` message type to fast task protos, which is used to send actor assignment info to admin via task events. While here I also hardened linting on our protos and updates existing protos to conform. This matches the restrictions we have in place in unionai/cloud/idl - [x] Add and run unittests - [x] Run locally and verify worker assignment info comes through in task events ``` ❯ curl -s http://localhost:8088/api/v1/task_executions/flytesnacks/development/f0dcb5e7cae6d47acaf3/n0\?limit\=100 | jq '.taskExecutions[0].closure.metadata.externalResources' [ { "externalId": "", "index": 0, "retryAttempt": 0, "phase": "UNDEFINED", "cacheStatus": "CACHE_DISABLED", "logs": [], "customInfo": { "assignedWorker": "example-2b377271ee80", "environmentId": "flytesnacks_development_example_c183cb6c9868d96" } } ] ❯ curl -s http://localhost:8088/api/v1/task_executions/flytesnacks/development/f0dcb5e7cae6d47acaf3/n1\?limit\=100 | jq '.taskExecutions[0].closure.metadata.externalResources' [ { "externalId": "", "index": 0, "retryAttempt": 0, "phase": "UNDEFINED", "cacheStatus": "CACHE_DISABLED", "logs": [], "customInfo": { "assignedWorker": "example-2b377271ee80", "environmentId": "flytesnacks_development_example_c183cb6c9868d96" } } ] ``` One merged, bring to cloud. Will be a no-op initially, but we can use this in a few dependent tasks to surface logs or look up other fast task worker info. Should this change be upstreamed to OSS (flyteorg/flyte)? If not, please uncheck this box, which is used for auditing. Note, it is the responsibility of each developer to actually upstream their changes. See [this guide](https://unionai.atlassian.net/wiki/spaces/ENG/pages/447610883/Flyte+-+Union+Cloud+Development+Runbook/#When-are-versions-updated%3F). - [ ] To be upstreamed to OSS ref COR-1581 Add custom info field to ExternalResourceInfo Adds a custom info field to `ExternalResourceInfo`. This is an extension of the [ExternalId](https://github.com/unionai/flyte/blob/2b6dfcaedab7ed6a7606d1434211086d064c6560/flyteidl/protos/flyteidl/event/event.proto#L269) field and can be used to pass plugin specific context or identifiers. We will use this to pass fast task worker assignment. N/A Just merge, not used yet Should this change be upstreamed to OSS (flyteorg/flyte)? If not, please uncheck this box, which is used for auditing. Note, it is the responsibility of each developer to actually upstream their changes. See [this guide](https://unionai.atlassian.net/wiki/spaces/ENG/pages/447610883/Flyte+-+Union+Cloud+Development+Runbook/#When-are-versions-updated%3F). - [x] To be upstreamed to OSS ref COR-1581 Closes #393 Signed-off-by: Daniel Rammer <hamersaw@protonmail.com>

* Send actor worker assignment in task events (#394)(#393) This change adds a new `FastTaskAssignment` message type to fast task protos, which is used to send actor assignment info to admin via task events. While here I also hardened linting on our protos and updates existing protos to conform. This matches the restrictions we have in place in unionai/cloud/idl - [x] Add and run unittests - [x] Run locally and verify worker assignment info comes through in task events ``` ❯ curl -s http://localhost:8088/api/v1/task_executions/flytesnacks/development/f0dcb5e7cae6d47acaf3/n0\?limit\=100 | jq '.taskExecutions[0].closure.metadata.externalResources' [ { "externalId": "", "index": 0, "retryAttempt": 0, "phase": "UNDEFINED", "cacheStatus": "CACHE_DISABLED", "logs": [], "customInfo": { "assignedWorker": "example-2b377271ee80", "environmentId": "flytesnacks_development_example_c183cb6c9868d96" } } ] ❯ curl -s http://localhost:8088/api/v1/task_executions/flytesnacks/development/f0dcb5e7cae6d47acaf3/n1\?limit\=100 | jq '.taskExecutions[0].closure.metadata.externalResources' [ { "externalId": "", "index": 0, "retryAttempt": 0, "phase": "UNDEFINED", "cacheStatus": "CACHE_DISABLED", "logs": [], "customInfo": { "assignedWorker": "example-2b377271ee80", "environmentId": "flytesnacks_development_example_c183cb6c9868d96" } } ] ``` One merged, bring to cloud. Will be a no-op initially, but we can use this in a few dependent tasks to surface logs or look up other fast task worker info. Should this change be upstreamed to OSS (flyteorg/flyte)? If not, please uncheck this box, which is used for auditing. Note, it is the responsibility of each developer to actually upstream their changes. See [this guide](https://unionai.atlassian.net/wiki/spaces/ENG/pages/447610883/Flyte+-+Union+Cloud+Development+Runbook/#When-are-versions-updated%3F). - [ ] To be upstreamed to OSS ref COR-1581 Add custom info field to ExternalResourceInfo Adds a custom info field to `ExternalResourceInfo`. This is an extension of the [ExternalId](https://github.com/unionai/flyte/blob/2b6dfcaedab7ed6a7606d1434211086d064c6560/flyteidl/protos/flyteidl/event/event.proto#L269) field and can be used to pass plugin specific context or identifiers. We will use this to pass fast task worker assignment. N/A Just merge, not used yet Should this change be upstreamed to OSS (flyteorg/flyte)? If not, please uncheck this box, which is used for auditing. Note, it is the responsibility of each developer to actually upstream their changes. See [this guide](https://unionai.atlassian.net/wiki/spaces/ENG/pages/447610883/Flyte+-+Union+Cloud+Development+Runbook/#When-are-versions-updated%3F). - [x] To be upstreamed to OSS ref COR-1581 Closes #393 Signed-off-by: Daniel Rammer <hamersaw@protonmail.com> * removed unused time import Signed-off-by: Daniel Rammer <hamersaw@protonmail.com> --------- Signed-off-by: Daniel Rammer <hamersaw@protonmail.com> Co-authored-by: Andrew Dye <andrewwdye@gmail.com>

brucearctor added enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers labels Jul 9, 2020

kumare3 added good first issue Good for newcomers help wanted Extra attention is needed and removed untriaged This issues has not yet been looked at by the Maintainers labels Jul 9, 2020

kumare3 added flytectl Issues related to flytectl -Flytes CLI and removed help wanted Extra attention is needed labels Feb 9, 2021

pmahindrakar-oss closed this as completed Jun 4, 2021

eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022

chore: cleanup of unused variables (flyteorg#394)

3acb21d

Signed-off-by: Nastya Rusina <nastya@union.ai>

eapolinario pushed a commit that referenced this issue Sep 6, 2023

Updating boilerplate to lock golangci-lint version (#394)

e1a4d0e

Signed-off-by: Daniel Rammer <daniel@union.ai>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FlyteCtl] Update LaunchPlan #394

[FlyteCtl] Update LaunchPlan #394

brucearctor commented Jul 9, 2020

pmahindrakar-oss commented Jun 4, 2021

[FlyteCtl] Update LaunchPlan #394

[FlyteCtl] Update LaunchPlan #394

Comments

brucearctor commented Jul 9, 2020

pmahindrakar-oss commented Jun 4, 2021