how to connect kfp.Client to standalone KFP v0.2 deployment on GCP?

[schmidt-jake]

Sorry for the noob question...

I successfully stood up my KFP v0.2 standalone deployment on a GKE cluster. I can access the UI at ....pipelines.googleusercontent.com. But I can't seem to get my kfp.Client to connect:

import kfp
client = kfp.Client(host='....pipelines.googleusercontent.com/pipeline', client_id='....pipelines.googleusercontent.com')
print(client.create_run_from_pipeline_func(...))

yields

Traceback (most recent call last):
  File "rig_state/pipeline.py", line 109, in <module>
    experiment_name='rig_state'
  File "/mapr/houmaprdev/user/jschmidt2/.conda/envs/pip_py36/lib/python3.6/site-packages/kfp/_client.py", line 372, in create_run_from_pipeline_func
    return self.create_run_from_pipeline_package(pipeline_package_path, arguments, run_name, experiment_name, namespace)
  File "/mapr/houmaprdev/user/jschmidt2/.conda/envs/pip_py36/lib/python3.6/site-packages/kfp/_client.py", line 414, in create_run_from_pipeline_package
    return RunPipelineResult(self, run_info)
  File "/mapr/houmaprdev/user/jschmidt2/.conda/envs/pip_py36/lib/python3.6/site-packages/kfp/_client.py", line 394, in __init__
    self.run_id = run_info.id
AttributeError: 'NoneType' object has no attribute 'id'

I know I'm doing something wrong that would be simple to fix...

[schmidt-jake]

A call to client.list_experiments yields

{'experiments': None, 'next_page_token': None, 'total_size': None}

even though I have created a couple experiments in the UI.

[schmidt-jake]

Figured it out. Removing host=... works, now with a warning about not using a service account. I'm familiar with how to enable Oauth for a full kubeflow deployment but I could use a little guidance for how to do it for kfp standalone... in https://iap.googleapis.com/v1/oauth/clientIds/<CLIENT_ID>:handleRedirect do I set CLIENT_ID to the result of kubectl describe configmap inverse-proxy-config -n kubeflow | grep googleusercontent.com?

[parthmishra]

I remember having the same issue but I was able to solve it by removing /pipeline from the host URL

[numerology]

@JakeTheWise
If you are attempting to connect to KFP standalone from your local environment (laptop/desktop, for example), you can first run gauth login in your terminal to authenticate, then

client = kfp.Client(host='....pipelines.googleusercontent.com')

should work. Note that you need to get rid of the /pipeline

[parthmishra]

@numerology Wait I'm experiencing this after upgrading to 0.2.1. I think in #2896 there was a change that caused the type of the token to change from str to bytes and that is causing it? I reverted these 2 lines to how it was in 0.1.40 and I can connect.

cc @Bobgy is this intended?

[Bobgy]

@parthmishra Sorry, that's not intended (I am not familiar with python, might be a mistake.), would you mind submitting a fix?

[numerology]

@Bobgy @parthmishra
#2986 is the fix.

[parthmishra]

Looks like #2986 fixed it so that will probably close this

[rmgogogo]

The fix #2986 is in 0.2.2, please retry. Here close. feel free to reopen if still has issue.

[schmidt-jake]

@rmgogogo This is broken again with kubeflow pipelines version 0.4.

[Bobgy]

/reopen
@JakeTheWise Can you share the detailed error message? Is it the same as mentioned above?

/assign @rmgogogo @numerology

[k8s-ci-robot]

@Bobgy: Reopened this issue.

In response to this:

/reopen
@JakeTheWise Can you share the detailed error message? Is it the same as mentioned above?

/assign @rmgogogo @numerology

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[schmidt-jake]

kfp_server_api.rest.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Content-Length': '1451', 'Content-Type': 'text/html; charset=utf-8', 'Date': 'Thu, 23 Apr 2020 02:05:41 GMT', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'SAMEORIGIN', 'X-Powered-By': 'Express', 'X-Xss-Protection': '0', 'Set-Cookie': 'S=cloud_datalab_tunnel=ARRK8GwzhjnEVBSJtocu-IU_0vB4hAjY; Path=/; Max-Age=3600'})
HTTP response body: 
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 400 (Bad Request)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/logos/errorpage/error_logo-150x54.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/logos/errorpage/error_logo-150x54-2x.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/logos/errorpage/error_logo-150x54-2x.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/logos/errorpage/error_logo-150x54-2x.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>400.</b> <ins>That’s an error.</ins>
  <p>  <ins>That’s all we know.</ins>

[Bobgy]

@JakeTheWise that seems different from this issue. If that's true, can you open a separate issue for it?

[schmidt-jake]

Well, the issue is that I can't seem to connect to my KFP Standalone deployment from python using the kfp sdk. Now I'm getting this error:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='35.239.148.234', port=443): Max retries exceeded with url: /api/v1/namespaces/kubeflow/services/ml-pipeline:http/proxy//apis/v1beta1/experiments (Caused by SSLError(SSLError("bad handshake: SysCallError(104, 'ECONNRESET')",),))

[schmidt-jake]

I'm trying to run this code from a python script:

client = kfp.Client(
        client_id='...-dot-us-central2.pipelines.googleusercontent.com'
    )
    client.create_run_from_pipeline_func(...)

[schmidt-jake]

Sorry, didn't mean to close the issue

[Bobgy]

I see, makes sense.
/assign @numerology @rmgogogo for investigation
I don't have much expertise on these

[k8s-ci-robot]

@Bobgy: GitHub didn't allow me to assign the following users: for, investigation.

Note that only kubeflow members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

I see, makes sense.
/assign @numerology @rmgogogo for investigation
I don't have much expertise on these

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[numerology]

client_id

Hi @JakeTheWise I believe you need to set the host instead of client_id in kfp.Client(). Do you mind trying again?

[schmidt-jake]

Yes, that causes the kfp_server_api.rest.ApiException I mentioned earlier.

[numerology]

Yes, that causes the kfp_server_api.rest.ApiException I mentioned earlier.

Oh I see.

So I think there are two possible reasons here:

1. The KFP deployment itself is temporarily unavailable for some reason (auto-upgrading/scaling etc.), in which case I think you cannot access the UI either; or
2. There is a mismatch b/w the kfp-api-server on the client side vs. that on the server side. So can you confirm which KFP SDK version you are using? And which KFP deployment version you are using?

If you are using SDK version > 0.4.0 it's likely incompatible with a KFP deployment < 0.4.0

[schmidt-jake]

I can access the UI just fine. The server and kfp package are v0.4. However the kfp-api-server was v0.3 - it doesn't seem there is a v0.4 on PyPI. Upgrading kfp-api-server to 0.5 also failed.
I also should mention that before all this, I tried to spin up the v0.5 standalone pipelines deployment and it failed.
Is there a better way to discuss minor issues like this? Can someone add me to the slack channel?

[rmgogogo]

I just did these and it works.

virtualenv -p python3 ~/kfpcli
source ~/kfpcli/bin/activate
pip install kfp
# It installs kfp 0.4.0 and kfp-server-api-0.3.0 (it's expected as if no API changes, we don't bump version for kfp-server-api)

gcloud auth login
kfp --endpoint 1e18af0c54f57e18-dot-us-central2.pipelines.googleusercontent.com pipeline list
# It's expected to see a list of pipelines here.

python
# Now in python shell
>>> import kfp
>>> client = kfp.Client(host='1e18af0c54f57e18-dot-us-central2.pipelines.googleusercontent.com')
>>> client.list_pipelines()
# It's expected to see a list of pipelines here.

So it passes test my side. You may double check whether it's reproduceable. I used virtualenv to avoid hitting some Python env issue.

[Bobgy]

@JakeTheWise Note, 0.5.0 is being released in progress, maybe some of the dependencies haven't been ready yet.

[schmidt-jake]

I had no idea that there was a CLI. Where was this feature announced, and where is it documented?

kfp --endpoint CLIENT pipeline list worked, and yet in python kfp.Client(client_id=CLIENT).list_pipelines() doesn't. Using kfp==0.4, python==3.6.8.

[arshyasharifian]

I just did these and it works.

virtualenv -p python3 ~/kfpcli
source ~/kfpcli/bin/activate
pip install kfp
# It installs kfp 0.4.0 and kfp-server-api-0.3.0 (it's expected as if no API changes, we don't bump version for kfp-server-api)

gcloud auth login
kfp --endpoint 1e18af0c54f57e18-dot-us-central2.pipelines.googleusercontent.com pipeline list
# It's expected to see a list of pipelines here.

python
# Now in python shell
>>> import kfp
>>> client = kfp.Client(host='1e18af0c54f57e18-dot-us-central2.pipelines.googleusercontent.com')
>>> client.list_pipelines()
# It's expected to see a list of pipelines here.

So it passes test my side. You may double check whether it's reproduceable. I used virtualenv to avoid hitting some Python env issue.

where did you get/find the host name, 1e18af0c54f57e18-dot-us-central2.pipelines.googleusercontent.com?

[numerology]

Hi @arshyasharifian

where did you get/find the host name, 1e18af0c54f57e18-dot-us-central2.pipelines.googleusercontent.com?

You can find it when accessing the KFP UI, it's just the web address. If you're using inverse-proxy (which is true if you deployed using standalone deployment, or from GKE marketplace), it should be of the format something-dot-region.pipelines.googleusercontent.com

[arshyasharifian]

I've deployed a Kubernetes cluster which has an envoy-ingress service with this endpoint: https://pskfp-1.endpoints.bright-voltage-248522.cloud.goog/

Is this what you are referring to?

When I use this as the host for initializing the kfp client with all the parameter fields, I get an invalid_client 401 error.

Here are my parameters for intializing the kfp client:

here are my user credentials in GCP (web application and other type users were created):

I feel like the authentication pipes are not configured properly, but I don't know.

error:
ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'X-Goog-IAP-Generated-Response': 'true', 'Content-Length': '36', 'Content-Type': 'text/html; charset=UTF-8', 'Date': 'Mon, 04 May 2020 20:11:12 GMT', 'Alt-Svc': 'clear'})
HTTP response body: Invalid IAP credentials: empty token

[numerology]

/assign @chensun
/assign @Bobgy
Perhaps you can take a look for the IAP authentication part? Thanks!

[arshyasharifian]

/assign @chensun
/assign @Bobgy
Perhaps you can take a look for the IAP authentication part? Thanks!

im following these instructions once more:
https://www.kubeflow.org/docs/gke/deploy/oauth-setup/

[chensun]

Here are my parameters for intializing the kfp client:

here are my user credentials in GCP (web application and other type users were created):

Hi @arshyasharifian, my_client_id should be the ID of client kubeflow_ps, i.e. 1092944325593-k4e..., whereas my_other_client_id should be the ID of client kfpclient, i.e. 1092944325593-hik... (and of course my_other_client_secret should be the secret of client kfpclient.

Besides, I think the the host should be https://<host>/pipeline

[arshyasharifian]

okay some progress.

I am being prompted to enter an authorization code after I enter the following params:

But I am still running into an authentication error when I try to use the client. For example, I want to get info about a pipeline I know exists in my UI

:
client.get_pipeline(pipeline_id='394aa68a-fa05-4a41-a568-b7ff674c0d05')

I get this error:

[chensun]

I am being prompted to enter an authorization code after I enter the following params:

Replace http with https in my_host, and see if it works?

[arshyasharifian]

works!!!!!!

Thank you so much.

Final KFP client params that works on local jupyter notebook:
import kfp

my_host = 'https://pskfp-1.endpoints.bright-voltage-248522.cloud.goog/pipeline'
#GCP client ID for web application: kubeflow_ps
my_client_id = '1092944325593-k4el4nl41b2cnjkftrm70l3umvjmbkno.apps.googleusercontent.com'
my_namespace = 'kubeflow-arshyasharifian'
#GCP client ID for Other, name: kfpclient
my_other_client_id ='1092944325593-hik3fqg7f3quact1p1t4fli8i3cmvl2f.apps.googleusercontent.com'
my_other_client_secret = 'EdCY8xdoisKdp9yQCka5JWYS'

#test script to see if client works (change pipeline ID to one that exists in your kubeflow UI):
client.get_pipeline(pipeline_id='')

[chensun]

Cool.
JFYI, we store the refresh token in a local file, so that you only get asked for the authorization code the first time you initialize a KFP client. In case you want to login with a different user on the same machine, simply delete the local file, and you will be prompted for login again.

[arshyasharifian]

Great, thanks.

[Bobgy]

@arshyasharifian Note, be careful, you are sharing your credentials publically. Please regenerate your credentials.