-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SDK] Fix withItem loop #2572
[SDK] Fix withItem loop #2572
Conversation
There might be a problem if the user has non-Kubernetes keys in their dictionaries since the |
sdk/python/kfp/compiler/compiler.py
Outdated
task['withItems'] = sub_group.loop_args.to_list_for_task_yaml() | ||
# Need to sanitize the dict keys for consistency. | ||
loop_tasks = sub_group.loop_args.to_list_for_task_yaml() | ||
isinstance(loop_tasks[0], dict) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure this line does anything.
That's true. So shall we alert the user whenever they want to access a non-Kubernetes keys in their ContainerOp, for example, try to use |
sdk/python/kfp/compiler/compiler.py
Outdated
loop_tasks = sub_group.loop_args.to_list_for_task_yaml() | ||
isinstance(loop_tasks[0], dict) | ||
sanitized_tasks = [] | ||
for item in loop_tasks: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: for argument_set in loop_tasks:
/lgtm |
That would be great. But there is a problem - python requires underscores while kubernetes requires dashes. So the names will be converted often. Update. I just remember that this is not Kubernetes limitation, but rather Argo limitation. And Argo lifted it in 2.3.0. So maybe we should just stop converting underscores to dashes. Then the problem will mostly go away. |
/lgtm |
Thanks for the info! If that's the case, I'd prefer the latter way, because as discussed in the related issue, current fix cannot fix the problem with withParams. I think it would be better off if we just do not use sanitized name in this case. Let me try to improve it a little bit and will let you know when I'm done. |
It's fine to do that in another PR though. It might require some sizable amount of work and testing. |
Okay. |
/approve |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Ark-kun The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Partially fix #2570
Question: I can imagine for withparams we have the same problem if user provides arbitrary names in the loop variable's runtime value. How to prevent that from happening? Shall we bypass k8s name sanitization in this case?
This change is