This repository has been archived by the owner on Oct 9, 2023. It is now read-only.
[WIP] Typed Input and Outputs without flytekit/flyteidl. Raw container support #31
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 11 commits
November 16, 2019 17:55
Support for Containers without flytekit. This PR creates special binary that can be used sans runtime. It will download and upload inputs and outputs using a shared volume
- Using success file as the signal
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Today FlytePropeller passes inputs and outputs using a blob store (managed using Stow). It is the responsibility of the container to read the inputs (unmarshal them) and then finally write the outputs in the same flyteidl defined protobuf format.
With this feature, containers will be able to read inputs from local filesystem in raw form or as a Json/Yaml and write outputs to the local filesystem. Flyte Engine will be responsible for marshaling/unmarshling to and from protobuf, storing them to the right place so that Propeller can access and maintaining Type safety.
Proposal:
https://docs.google.com/document/d/1ZsCDOZ5ZJBPWzCNc45FhNtYQOxYHz0PAu9lrtDVnUpw/edit#
Related Issue
flyteorg/flyte#297
This code should be moved to FlytePlugins.