Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vscode decorator for the dynamic task #2994

Merged
merged 38 commits into from
Dec 18, 2024
Merged

vscode decorator for the dynamic task #2994

merged 38 commits into from
Dec 18, 2024

Conversation

pingsutw
Copy link
Member

Tracking issue

NA

Why are the changes needed?

To support vscode decorator for the dynamic tasks.

What changes were proposed in this pull request?

Set the correct resolver in the current context, so flytekit can compile the vscode task

How was this patch tested?

from flytekit import task, workflow, ImageSpec, Resources, dynamic
from flytekit.interactive import vscode

new_flytekit = "git+https://github.com/flyteorg/flytekit@cedb8188f6fa86481064fb9022b16c9f5d369f4d"
image_spec = ImageSpec(registry="ghcr.io/unionai-oss",
                       packages=[new_flytekit, "kubernetes", "mypy"],
                       apt_packages=["git"])


@task(container_image=image_spec, enable_deck=True, requests=Resources(cpu="2", mem="2000Mi"))
def train():
    print("forward", flush=True)
    print("backwarddddf")


@dynamic(container_image=image_spec)
@vscode
def d1():
    print("dynamic", flush=True)
    train()


@workflow
def wf():
    train()
    d1(

Setup process

Screenshots

Screenshot 2024-12-10 at 12 48 29 PM

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Docs link

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
@pingsutw pingsutw changed the title Debug dynamic vscode decorator for the dynamic task Dec 11, 2024
Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
@eapolinario eapolinario enabled auto-merge (squash) December 18, 2024 14:55
@eapolinario eapolinario disabled auto-merge December 18, 2024 14:59
Comment on lines +738 to +740
from flytekit.core.python_auto_container import default_task_resolver

return CompilationState(prefix=prefix, task_resolver=default_task_resolver)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need this if the correct task resolver is being set in compile in flytekit/core/workflow.py?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I set the task resolver in the ctx.with_compilation_state(...), but there are some places where new_compilation_state() get called, potentially causing the loss of the resolver if it's not set.

@pingsutw pingsutw merged commit b92e911 into master Dec 18, 2024
102 checks passed
shuyingliang pushed a commit to shuyingliang/flytekit that referenced this pull request Dec 20, 2024
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
eapolinario added a commit that referenced this pull request Dec 23, 2024
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
eapolinario added a commit that referenced this pull request Dec 26, 2024
* vscode decorator for the dynamic task (#2994)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Use correct name in flyteagent test (#3011)

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* fix: Improve error details and logging config handling (#3012)

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Add python interpreter into vscode settings (#3014)

Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>

* add support for toggling data mode for array node (#2940)

* add support for toggling data mode for array node

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* clean up

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* clean up

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* cleanup

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* Bump flyteidl lower-bound to 1.14.1

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Add import of FlyteLaunchPlan back

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

---------

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Store protos in local cache (#3022)

* Store proto obj instead of model Literal in local cache

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Remove unused file

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

---------

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

---------

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Co-authored-by: Kevin Su <pingsutw@apache.org>
Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: Thomas J. Fan <thomasjpfan@gmail.com>
Co-authored-by: Paul Dittamo <37558497+pvditt@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants