Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Git storage with use_ssh does not work with Prefect image #4538

Closed
zangell44 opened this issue May 17, 2021 · 0 comments · Fixed by #4539
Closed

Git storage with use_ssh does not work with Prefect image #4538

zangell44 opened this issue May 17, 2021 · 0 comments · Fixed by #4539
Assignees

Comments

@zangell44
Copy link
Collaborator

Description

Git storage does not work when using prefecthq/prefect:latest image.

When trying to retrieve a flow using ssh Git storage, you'll see a run time error [Errno 2] No such file or directory: 'ssh': 'ssh'.

This is my fault, I overlooked this when writing Git storage.

Expected Behavior

The storage class works smoothly when ssh keys are mounted in to the container directly.

An example of mounting ssh keys using the prefect image should also be documented.

Reproduction

  1. Create Flow
import os
from prefect import Flow, Parameter, task
from prefect.run_configs import KubernetesRun
from prefect.storage import Git
@task(log_stdout=True)
def say_hello(name):
    print("Hello, {}!".format(name))
with Flow("Hello World") as flow:
    thename = Parameter("name")
    say_hello(thename)
storage = Git(
    repo="my/repo",
    flow_path="src/flows/hello_world.py",
    branch_name="my/branch-name",
    use_ssh=True,
)
storage.add_flow(flow) # the Flow object
storage.get_flow(flow.name) # try to load the flow from git storage
  1. Publish Flow to GitHub
  2. Run Docker Image
    $ docker run --rm -it --entrypoint sh -v path/to/flow:/src -v ~/.ssh:/root/.ssh prefecthq/prefect:latest
  3. Debug Flow
# python /src/my_test_file.py
Traceback (most recent call last):
  File "/src/TestCloningRepoWithSSH.py", line 26, in <module>
    storage.get_flow(flow.name) # try to load the flow from git storage
  File "/usr/local/lib/python3.7/site-packages/prefect/storage/git.py", line 122, in get_flow
    clone_depth=self.clone_depth,
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/git.py", line 48, in __enter__
    source=self.git_clone_url, target=self.temp_dir.name, depth=self.clone_depth
  File "/usr/local/lib/python3.7/site-packages/dulwich/porcelain.py", line 476, in clone
    **kwargs
  File "/usr/local/lib/python3.7/site-packages/dulwich/porcelain.py", line 1559, in fetch
    fetch_result = client.fetch(path, r, progress=errstream.write, depth=depth)
  File "/usr/local/lib/python3.7/site-packages/dulwich/client.py", line 528, in fetch
    depth=depth,
  File "/usr/local/lib/python3.7/site-packages/dulwich/client.py", line 1009, in fetch_pack
    proto, can_read, stderr = self._connect(b"upload-pack", path)
  File "/usr/local/lib/python3.7/site-packages/dulwich/client.py", line 1659, in _connect
    self.host, argv, port=self.port, username=self.username, **kwargs
  File "/usr/local/lib/python3.7/site-packages/dulwich/client.py", line 1522, in run_command
    stderr=subprocess.PIPE,
  File "/usr/local/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'ssh': 'ssh'
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/weakref.py", line 648, in _exitfunc
    f()
  File "/usr/local/lib/python3.7/weakref.py", line 572, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/usr/local/lib/python3.7/tempfile.py", line 797, in _cleanup
    _shutil.rmtree(name)
  File "/usr/local/lib/python3.7/shutil.py", line 485, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/usr/local/lib/python3.7/shutil.py", line 483, in rmtree
    orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp9goeajxv'

Environment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant