Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SDK] Fix withItem loop #2572

Merged
merged 8 commits into from
Nov 8, 2019
Merged

Conversation

numerology
Copy link

@numerology numerology commented Nov 7, 2019

Partially fix #2570

Question: I can imagine for withparams we have the same problem if user provides arbitrary names in the loop variable's runtime value. How to prevent that from happening? Shall we bypass k8s name sanitization in this case?


This change is Reviewable

@Ark-kun
Copy link
Contributor

Ark-kun commented Nov 7, 2019

we have the same problem if user provides arbitrary names in the loop variable's runtime value

There might be a problem if the user has non-Kubernetes keys in their dictionaries since the {{item.*}} placeholders might be converted. The runtime values won't be a problem. But the placeholder might not be able to access them (e.g. you cannot access keys with spaces).

task['withItems'] = sub_group.loop_args.to_list_for_task_yaml()
# Need to sanitize the dict keys for consistency.
loop_tasks = sub_group.loop_args.to_list_for_task_yaml()
isinstance(loop_tasks[0], dict)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this line does anything.

@numerology
Copy link
Author

we have the same problem if user provides arbitrary names in the loop variable's runtime value

There might be a problem if the user has non-Kubernetes keys in their dictionaries since the {{item.*}} placeholders might be converted. The runtime values won't be a problem. But the placeholder might not be able to access them (e.g. you cannot access keys with spaces).

That's true. So shall we alert the user whenever they want to access a non-Kubernetes keys in their ContainerOp, for example, try to use item.a_a, which will be converted to a-a under the hood?

loop_tasks = sub_group.loop_args.to_list_for_task_yaml()
isinstance(loop_tasks[0], dict)
sanitized_tasks = []
for item in loop_tasks:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: for argument_set in loop_tasks:

@Ark-kun
Copy link
Contributor

Ark-kun commented Nov 7, 2019

/lgtm

@k8s-ci-robot k8s-ci-robot removed the lgtm label Nov 7, 2019
@Ark-kun
Copy link
Contributor

Ark-kun commented Nov 7, 2019

So shall we alert the user whenever they want to access a non-Kubernetes keys in their ContainerOp, for example, try to use item.a_a, which will be converted to a-a under the hood?

That would be great. But there is a problem - python requires underscores while kubernetes requires dashes. So the names will be converted often.

Update. I just remember that this is not Kubernetes limitation, but rather Argo limitation. And Argo lifted it in 2.3.0. So maybe we should just stop converting underscores to dashes. Then the problem will mostly go away.

@Ark-kun
Copy link
Contributor

Ark-kun commented Nov 7, 2019

/lgtm

@numerology
Copy link
Author

So shall we alert the user whenever they want to access a non-Kubernetes keys in their ContainerOp, for example, try to use item.a_a, which will be converted to a-a under the hood?

That would be great. But there is a problem - python requires underscores while kubernetes requires dashes. So the names will be converted often.

Update. I just remember that this is not Kubernetes limitation, but rather Argo limitation. And Argo lifted it in 2.3.0. So maybe we should just stop converting underscores to dashes. Then the problem will mostly go away.

Thanks for the info! If that's the case, I'd prefer the latter way, because as discussed in the related issue, current fix cannot fix the problem with withParams. I think it would be better off if we just do not use sanitized name in this case.

Let me try to improve it a little bit and will let you know when I'm done.

@numerology numerology changed the title [SDK] Fix withItem loop [WIP][SDK] Fix withItem loop Nov 7, 2019
@Ark-kun
Copy link
Contributor

Ark-kun commented Nov 7, 2019

Let me try to improve it a little bit and will let you know when I'm done.

It's fine to do that in another PR though. It might require some sizable amount of work and testing.

@numerology
Copy link
Author

Let me try to improve it a little bit and will let you know when I'm done.

It's fine to do that in another PR though. It might require some sizable amount of work and testing.

Okay.

@numerology numerology changed the title [WIP][SDK] Fix withItem loop [SDK] Fix withItem loop Nov 7, 2019
@Ark-kun
Copy link
Contributor

Ark-kun commented Nov 8, 2019

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Ark-kun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit ead912c into kubeflow:master Nov 8, 2019
@numerology numerology deleted the fix-dsl-par-loop branch December 3, 2019 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ParallelFor parameter parsing does not retain user input format
3 participants