Pipeline notebook samples : "Install Pipeline SDK " error

[yylin1]

I‘m trying to test sample for "KubeFlow Pipeline Using TFX OSS Components", but have error for Collecting https://storage.googleapis.com/ml-pipeline/release/0.1.4/kfp.tar.gz.

Environment:

• kubeflow_v0.3.5
• ksonnet version: 0.13.0

JupyterHub Spawner :

• Image : gcr.io/kubeflow-images-public/tensorflow-1.8.0-notebook-gpu:v0.3.1

# Install Pipeline SDK
!pip3 install $KFP_PACKAGE --upgrade

Collecting https://storage.googleapis.com/ml-pipeline/release/0.1.4/kfp.tar.gz
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fc2d951f8d0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /ml-pipeline/release/0.1.4/kfp.tar.gz
...
...
...
Could not install packages due to an EnvironmentError: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /ml-pipeline/release/0.1.4/kfp.tar.gz (Caused by NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fc2d94d8240>: Failed to establish a new connection: [Errno -2] Name or service not known',))

[xiaozhouX]

Maybe there are some network problem between your jupyter pod and storage.googleapis.com. You can check it in your pod.

[yylin1]

@xiaozhouX how can I check jupyter pod ?

But I find it in my (ns kubeflow) get pod have problem in ml-pipelines-load-samples status Error，when in new deploy.

ml-pipeline-b59b58dd6-cwsb6                               1/1       Running   1          6h
ml-pipeline-persistenceagent-9ff99498c-p7qjz              1/1       Running   7          6h
ml-pipeline-scheduledworkflow-78794fd86f-xhd4g            1/1       Running   0          6h
ml-pipeline-ui-9884fd997-22v92                            1/1       Running   0          6h
ml-pipelines-load-samples-zp9hr                           0/1       Error     0          6h

Get logs :

$ kubectl logs -f ml-pipelines-load-samples-zp9hr -n kubeflow

F0103 08:49:21.808166       1 error.go:296] dial tcp 10.109.143.171:3306: connect: connection timed out
goroutine 1 [running]:
github.com/kubeflow/pipelines/vendor/github.com/golang/glog.stacks(0xc000135f00, 0xc000618000, 0x68, 0x9b)
	/go/src/github.com/kubeflow/pipelines/vendor/github.com/golang/glog/glog.go:769 +0xd4
github.com/kubeflow/pipelines/vendor/github.com/golang/glog.(*loggingT).output(0x263bee0, 0xc000000003, 0xc0005c8840, 0x239a6b6, 0x8, 0x128, 0x0)
	/go/src/github.com/kubeflow/pipelines/vendor/github.com/golang/glog/glog.go:720 +0x329
github.com/kubeflow/pipelines/vendor/github.com/golang/glog.(*loggingT).printf(0x263bee0, 0x3, 0x15d6e4d, 0x2, 0xc000671a70, 0x1, 0x1)
	/go/src/github.com/kubeflow/pipelines/vendor/github.com/golang/glog/glog.go:655 +0x14b
github.com/kubeflow/pipelines/vendor/github.com/golang/glog.Fatalf(0x15d6e4d, 0x2, 0xc000671a70, 0x1, 0x1)
	/go/src/github.com/kubeflow/pipelines/vendor/github.com/golang/glog/glog.go:1148 +0x67
github.com/kubeflow/pipelines/backend/src/common/util.TerminateIfError(0x17814e0, 0xc0000b6230)
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:296 +0x79
main.initMysql(0xc0003474c8, 0x5, 0x53d1ac1000, 0x0, 0x0)
	/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/client_manager.go:204 +0x2c6
main.initDBClient(0x53d1ac1000, 0x15)
	/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/client_manager.go:137 +0x54c
main.(*ClientManager).init(0xc000671d50)
	/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/client_manager.go:103 +0x80
main.newClientManager(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/client_manager.go:242 +0x90
main.main()
	/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/main.go:53 +0x88

[jlewi]

Duplicate of #603

[jlewi]

Actually this looks like a different issue than #603.

#603 complains about creating a foreign key. Where'as here there is a connection time out.

[hamedhsn]

+1 Having the same problem. v0.4.0

[IronPan]

@hamedhsn what problem did you see in v0.4.0 it looks like there are multiple issues mentioned in this issues and some of them are fixed.

[hamedhsn]

exactly like what @yylin1 reported..pods related to pipelines are all up except ml-pipelines-load-samples-xxx with same error log.
and when I open the UI no experiment is loaded, I suppose it is because of that issue.

[yylin1]

@hamedhsn You are new deploy on v0.4.0 ？ get pod same problem ml-pipelines-load-samples-xxxxx is Error?

I can try to change the version for v0.4.0. (testing)

[hamedhsn]

yes, it is a new deployment(testing 0.4.0) and getting the same error.

[gaoning777]

The load sample issue should not lead to the SDK installation error. Are you still having problems in installing the SDK in the notebook? If so, "pip3 install https://storage.googleapis.com/ml-pipeline/release/0.1.4/kfp.tar.gz --upgrade" should work fine in any networked computers and you could try running this in your own machine to see if it works. My guess is that the SDK installation problem is mostly about network failure.

[hamedhsn]

@gaoning777 I can install the pip package with no problem.
It is probably a different issue. I created a new issue #676

[vicaire]

Resolving since inactive for 2 months. Please reopen if needed.

[wdhorton]

I think I'm hitting a similar issue, seeing connection timed out in the ml-pipeline pod:

$ kubectl -n kubeflow logs ml-pipeline-cb898579d-rmlw5 -p
I1008 14:55:57.976245       6 client_manager.go:123] Initializing client manager
F1008 15:02:29.361177       6 error.go:296] dial tcp 10.100.159.188:3306: connect: connection timed out
goroutine 1 [running]:
github.com/golang/glog.stacks(0xc000015500, 0xc000524c80, 0x68, 0x9b)
	external/com_github_golang_glog/glog.go:769 +0xd4
github.com/golang/glog.(*loggingT).output(0x295a520, 0xc000000003, 0xc0005c8fd0, 0x283c0ab, 0x8, 0x128, 0x0)
	external/com_github_golang_glog/glog.go:720 +0x329
github.com/golang/glog.(*loggingT).printf(0x295a520, 0x3, 0x1a79b1d, 0x2, 0xc0001959b0, 0x1, 0x1)
	external/com_github_golang_glog/glog.go:655 +0x14b
github.com/golang/glog.Fatalf(0x1a79b1d, 0x2, 0xc0001959b0, 0x1, 0x1)
	external/com_github_golang_glog/glog.go:1148 +0x67
github.com/kubeflow/pipelines/backend/src/common/util.TerminateIfError(0x1c2cd80, 0xc0000ce1e0)
	backend/src/common/util/error.go:296 +0x79
main.initMysql(0xc000047c9a, 0x5, 0x53d1ac1000, 0x0, 0x0)
	backend/src/apiserver/client_manager.go:260 +0x37d
main.initDBClient(0x53d1ac1000, 0x15)
	backend/src/apiserver/client_manager.go:190 +0x5c0
main.(*ClientManager).init(0xc000195cd8)
	backend/src/apiserver/client_manager.go:125 +0x80
main.newClientManager(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	backend/src/apiserver/client_manager.go:300 +0x7b
main.main()
	backend/src/apiserver/main.go:56 +0x5e