SDK/Components - Added /data to the generated file paths #663
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
6c0ad33 to
8b07d6a
Compare
This is needed for the future storage system based on volume mounts: If outputs were written to files in the same dir (e.g. /outputs/out1.txt and /outputs/out2.txt), then we cannot separate them and mount to the downstream task containers independently.
8b07d6a to
433bb82
Compare
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Ark-kun, gaoning777 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
1 similar comment
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Ark-kun, gaoning777 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Linchin
pushed a commit
to Linchin/pipelines
that referenced
this pull request
Apr 11, 2023
* Remove references to blueprints; this should not have been committed. We use a separate script to cleanup blueprints.
magdalenakuhn17
pushed a commit
to magdalenakuhn17/pipelines
that referenced
this pull request
Oct 22, 2023
* delete isvc-tensorflow after testing * add predict
HumairAK
pushed a commit
to red-hat-data-services/data-science-pipelines
that referenced
this pull request
Mar 11, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is needed for the future storage system based on volume mounts:
If outputs were written to files in the same dir (e.g.
/outputs/out1.txtand/outputs/out2.txt), then we cannot separate them and mount to the downstream task containers independently.Writing outputs to
/outputs/out1/dataand/outputs/out2/datasolves this problem.The PR is just 3 lines, but here is a more detailed explanation on why this improves the storage story:
In future we want to use mounted volumes to pass the artifacts between components.
Each input needs to be placed into its own directory.
Multiple inputs are stored in different locations on the storage volume.
Only one volume can be mounted to a container's local directory.
Thus, to mount multiple inputs from different locations we a directory for each input to mount the corresponding volume.
Each output needs to be placed into its own directory.
Output volume can only be mounted to a directory.
If all outputs are placed in the same local directory, then it would be impossible to mount just one of them to the downstream container. Also, having files from multiple outputs in a same directory requires them to have different names which breaks the next item.
Thus every output should be placed into its own directory.
Output files should have same names in storage (different directories, but same name e.g. "data").
When we pass outputs of some upstream task to the input of downstream task, the downstream program needs to know the full input file path. The input volume mount locations can generated from the downstream component definition alone (e.g.
/inputs/Downstream_input_1/). But to generate the full path, the system also needs to know the file name in the mounted volume. If that name is always the same, then the problem is solved. Otherwise, the system needs information that's taken from the upstream task. This complicates the system and can be very hard to do in cases of caching or recursion.So, when two tasks are connected, the name of the file that's being passed should be predictable by downstream task without knowing anything about the upstream task.
Thus output file names must be same everywhere.
This change is