Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Backend][Plugin] RawContainer Support #297

Closed
1 of 5 tasks
kumare3 opened this issue Apr 25, 2020 · 0 comments · Fixed by #358
Closed
1 of 5 tasks

[Backend][Plugin] RawContainer Support #297

kumare3 opened this issue Apr 25, 2020 · 0 comments · Fixed by #358
Assignees
@kumare3
Labels
enhancement New feature or request
Milestone
0.5.0

Comments

@kumare3
Copy link
Contributor

kumare3 commented Apr 25, 2020 •
edited
Loading

Why would this plugin be helpful to the Flyte community
Today to run every container as a task in Flyte that produces an output the container needs to contain some mechanism to talk with the blob store and the marshalling and unmarshalling logic from FlyteIDL.
We could provide a plugin that would allow running containers without needing these bits.
This plugin may be generalized in the future for all plugins.

The detailed proposal can be seen here
https://docs.google.com/document/d/1ZsCDOZ5ZJBPWzCNc45FhNtYQOxYHz0PAu9lrtDVnUpw/edit#

Type of Plugin

  • Web Service (e.g. AWS Sagemaker, GCP DataFlow, Qubole etc...)
  • Kubernetes Operator (e.g. TfOperator, SparkOperator, FlinkK8sOperator, etc...)
  • Customized Plugin using native kubernetes constructs
  • Other

Can you help us with the implementation?

  • [ x ] Yes
  • No

Additional context
The expentation would be, that I am able to run a shell script and for example sed replace some values in a string. This is a contrived example, but would enable to drive the right implementation
@kumare3 kumare3 added the untriaged This issues has not yet been looked at by the Maintainers label Apr 25, 2020
@kumare3 kumare3 self-assigned this Apr 25, 2020
@kumare3 kumare3 added enhancement New feature or request task labels Apr 25, 2020
@kumare3 kumare3 added this to the 0.4.0 milestone Apr 25, 2020
@kumare3 kumare3 mentioned this issue Apr 25, 2020
Closed
@kumare3 kumare3 removed the untriaged This issues has not yet been looked at by the Maintainers label Apr 30, 2020
This was referenced May 12, 2020
Merged
Merged
Merged
This was referenced May 19, 2020
Merged
Merged
Merged
@kumare3 kumare3 mentioned this issue May 27, 2020
Merged
8 tasks
@EngHabu EngHabu modified the milestones: 0.4.0, 0.5.0 May 28, 2020
@kumare3 kumare3 mentioned this issue Jun 4, 2020
Merged
8 tasks
This was referenced Jun 17, 2020
Merged
Merged
@kumare3 kumare3 linked a pull request Jun 24, 2020 that will close this issue
feature; automated data loading, raw container, co-pilot pre-alpha release #358
Merged
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 6, 2022
Bump plugins to pick up ephemeral storage changes (flyteorg#297)
dfafd9a 
Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 6, 2022
@EngHabu
Add nil checks and avoid nil returns (flyteorg#297)
e79dbf7
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
@fg91
Apply default pod template to PytorchJob pods (flyteorg#297)
428b7a4 
* Merge pod template spec with pod spec in separate func

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Merge pod template spec with pod spec in separate func

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Apply pod template to pytorch job pod spec

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Handle nil podspecs before merging

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Pass both default and primare container name to MergePodSpecs

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Move podSpec.DeepCopy into MergePodSpecs

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Add tests

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Merge pod template into tfjob and mpijob

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Lint

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Correct usage of default and primate container (template) name

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Override mpi default container name

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Carry over ObjectMeta from pod template

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Remove old `TestBuildPodWithSpec` test

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>
Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
@flyte-bot
Update Boilerplate (flyteorg#297)
4d93abf 
Signed-off-by: Flyte-Bot <admin@flyte.org>

Co-authored-by: flyte-bot <flyte-bot@users.noreply.github.com>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
@pmahindrakar-oss
[Doc] Updated flytectl uprade command (flyteorg#297)
44e0175
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 9, 2023
Bump plugins to pick up ephemeral storage changes (flyteorg#297)
65bfdd6 
Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 21, 2023
@EngHabu
Add nil checks and avoid nil returns (flyteorg#297)
5c7e7b4
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 21, 2023
@fg91
Apply default pod template to PytorchJob pods (flyteorg#297)
968edb3 
* Merge pod template spec with pod spec in separate func

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Merge pod template spec with pod spec in separate func

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Apply pod template to pytorch job pod spec

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Handle nil podspecs before merging

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Pass both default and primare container name to MergePodSpecs

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Move podSpec.DeepCopy into MergePodSpecs

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Add tests

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Merge pod template into tfjob and mpijob

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Lint

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Correct usage of default and primate container (template) name

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Override mpi default container name

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Carry over ObjectMeta from pod template

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

* Remove old `TestBuildPodWithSpec` test

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>

Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com>
Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Apr 30, 2024
@pmahindrakar-oss
[Doc] Updated flytectl uprade command (flyteorg#297)
308ddc8
austin362667 pushed a commit to austin362667/flyte that referenced this issue May 7, 2024
@pmahindrakar-oss @austin362667
[Doc] Updated flytectl uprade command (flyteorg#297)
d17767a
robert-ulbrich-mercedes-benz pushed a commit to robert-ulbrich-mercedes-benz/flyte that referenced this issue Jul 2, 2024
@pmahindrakar-oss @robert-ulbrich-mercedes-benz
[Doc] Updated flytectl uprade command (flyteorg#297)
3a8f20e
troychiu pushed a commit that referenced this issue Jul 8, 2024
@andrewwdye
Stop logging auth metadata (#297)
d498c2f 
## Overview
Stop logging grpc auth metadata

## Test Plan
N/A

## Rollout Plan (if applicable)
Simple logic change, but will plan to take out quickly

## Upstream Changes
Should this change be upstreamed to OSS (flyteorg/flyte)? If so, please check this box for auditing. Note, this is the responsibility of each developer. See [this guide](https://unionai.atlassian.net/wiki/spaces/ENG/pages/447610883/Flyte+-+Union+Cloud+Development+Runbook/#When-are-versions-updated%3F).
- [x] To be upstreamed

## Issue
ref CLD-943
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kumare3 kumare3

Labels
enhancement New feature or request
Projects
None yet
Milestone
0.5.0
Development

Successfully merging a pull request may close this issue.

feature; automated data loading, raw container, co-pilot pre-alpha release flyteorg/flyte
2 participants
@EngHabu @kumare3