Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Do I need to be running kubeflow on GCP to use pipelines? #661

Closed
govindKAG opened this issue Jan 10, 2019 · 3 comments
Closed

Do I need to be running kubeflow on GCP to use pipelines? #661

govindKAG opened this issue Jan 10, 2019 · 3 comments
Assignees
@paveldournov

Comments

@govindKAG
Copy link

The articles on pipelines seem to suggest that pipelines are in some way dependent on GCP. I could not find anything definitive on this front. Is it possible to fully utilize kubeflow pipelines without GCP? if not, is it by design?
@paveldournov
Copy link
Contributor

KubeFlow pipeline at its core is a multi-cloud platform and can be deployed on-prem as well, however some of the specific pipeline examples (e.g. TFX Taxi sample and Spark sample pipelines) require GCP services such as the GCS storage, DataFlow, and DataProc. We will be publishing more cloud neutral samples going forward and welcome contributions to the sample pipelines that are GCP independent.

@paveldournov paveldournov self-assigned this Jan 20, 2019
@gaoning777
Copy link
Contributor

Another aspect is that we will provide more kubernetes interfaces in the DSL language. For example, image_pull_secret will be exposed in the DSL such that images can be fetched from other registries. The goal is to make Kubeflow pipeline a kubernetes native(cross-platform) product.

@govindKAG
Copy link
Author

that's good to know!

Linchin pushed a commit to Linchin/pipelines that referenced this issue Apr 11, 2023
@jlewi
Fix auto-deployments reconciler. (kubeflow#661)
1fb9197 
* The reconciler is running amok because in kubeflow#657 we changed the code
  to skip over deployments with no manifest field because we couldn't get
  the zone. It turns out the lack of the manifest is an indication that
  the deployment failed. By not matching these deployments we end
  up retrying the auto-deployment. This is causing a cascading failure
  because we end up using up all our DM quota which in turn causes auto
  deployments to fail which in turn causes us to try again and eat up
  more deployments.

* This PR fixes that by matching deployments even if the manifest field
  is missing. This should prevent us from repeatedly retrying.

* Add exponential backoff to delete deployments. Because of the above
  error we are bumping to write quota limits per day which is impacting
  our ability to GC deployments.
magdalenakuhn17 pushed a commit to magdalenakuhn17/pipelines that referenced this issue Oct 22, 2023
@pugangxa
Add pugangxa as reviewer (kubeflow#661)
1fff911
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@paveldournov paveldournov

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@gaoning777 @paveldournov @govindKAG