Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Git storage + SSH documentation #4539

Merged
merged 5 commits into from
Jun 15, 2021
Merged

Add Git storage + SSH documentation #4539

merged 5 commits into from
Jun 15, 2021

Conversation

zangell44
Copy link
Collaborator

@zangell44 zangell44 commented May 17, 2021

Summary

Adds ssh agent to Dockerfile, allowing the use of ssh cloning when using the image for Git storage.

Changes

  • Adds ssh-agent to Dockerfile
  • Minor documentation update

Importance

Resolves #4538.

Checklist

This PR:

  • adds new tests (if appropriate)
  • adds a change file in the changes/ directory (if appropriate)
  • updates docstrings for any new functions or function arguments, including docs/outline.toml for API reference docs (if appropriate)

@zangell44 zangell44 requested a review from jcrist as a code owner May 17, 2021 14:35
@zangell44 zangell44 requested a review from zanieb May 17, 2021 14:37
@TylerWanner
Copy link
Contributor

TylerWanner commented May 17, 2021

I don't see a compelling reason to add ssh clients to our base Docker image and would oppose doing so until convinced otherwise. Is there any way to just install at runtime when using git storage? Putting ssh clients in all flow containers feels heavy, though I can't make a complete argument that it's prohibitively insecure

@zangell44
Copy link
Collaborator Author

Because of security concerns, we've decided not to add ssh-agent to our base image.

I'll update this PR to provide documentation on adding ssh-agent to the base image and configuring ssh keys.

@zanieb zanieb marked this pull request as draft May 25, 2021 21:19
@zangell44 zangell44 removed the request for review from jcrist June 14, 2021 13:18
@zangell44 zangell44 marked this pull request as ready for review June 14, 2021 15:06
@zangell44 zangell44 changed the title Add ssh agent to Dockerfile Add Git storage + SSH documentation Jun 14, 2021
@zangell44 zangell44 requested a review from TylerWanner June 14, 2021 15:07
kubectl create secret generic my-ssh-key --from-file=<ssh-key-name>=/path/to/<ssh-key-name> --from-file=known_hosts=/path/to/known_hosts
```

Next, create a custom job template to mount the secret volume to `/root/.ssh`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's kinda weird that the example is not a full "job template" even though it is minimally viable to be properly merged with the default job template. this story continues to haunt me but this is good imo

Finally, [configure the agent or flow to use the custom job template](https://docs.prefect.io/orchestration/agents/kubernetes.html#custom-job-template).


Creating a [Kubernetes service account](https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/) to permission the Secret properly is recommended. Once configured in Kubernetes, service account can be set either [on agent start or on the run config](https://docs.prefect.io/orchestration/agents/kubernetes.html#service-account).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the RBAC config is a must for this to work

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zangell44 Does this need a change before merge?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@madkinsz this is okay to merge as is, I've followed these instructions successfully myself
I have my eye on this documentation to update it as we get feedback

@TylerWanner
Copy link
Contributor

New docs look good. Definitely an improvement

@zanieb zanieb merged commit b1c2e0e into master Jun 15, 2021
@zanieb zanieb deleted the git-ssh-prefect-image branch June 15, 2021 20:40
@zanieb zanieb mentioned this pull request Jun 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Git storage with use_ssh does not work with Prefect image
3 participants