From 440f7dcae8b9e84aea4ff37aa373bfca72d43c0a Mon Sep 17 00:00:00 2001 From: numerology Date: Wed, 21 Aug 2019 20:42:06 -0700 Subject: [PATCH] Squashed commit of the following: commit 41d39c17a2bea8bce4a5a779f105ad7ff923b44a Author: Kirin Patel Date: Wed Aug 21 19:38:31 2019 -0700 Add run with json data as input within fixed-data.ts for UI testing and development (#1895) * Added run with json data as input * Changed run and uid to not be duplicates of hello-world-runtime commit 851e7c80d256ee1a586804606c566b8ad068d4f2 Author: Kirin Patel Date: Wed Aug 21 19:04:31 2019 -0700 Replace codemirror editor react component with react-ace editor component (#1890) * Replaced CodeMirror with Editor in PipelineDetails.tsx * Replaced codemirror in DetailsTable with react-ace * Removed codemirror * Updated unit tests for Editor.tsx to test placeholder and value in simplified manner * Updated DetailsTable.test.tsx.snap to reflect changes made in DetailsTable.tsx * Updated PipelineDetails test snapshot * Changed width of Editor in DetailsTable to be 100% instead of 300px * Revert "Updated unit tests for Editor.tsx to test placeholder and value in simplified manner" This reverts commit 40103f2e019fc3a20329204d2de23f2d3916f740. commit 8c3d6fe1213da85ce3196668834553e359b03a5d Author: Kirin Patel Date: Wed Aug 21 18:30:33 2019 -0700 Add visualization-server service to lightweight deployment (#1844) * Add visualization-server service to lightweight deployment * Addressed PR suggestions * Added field to determine if visualization service is active and fixed unit tests for visualization_server.go * Additional small fixes * port change from 88888 -> 8888 * version change from 0.1.15 -> 0.1.26 * removed visualization-server from base/kustomization.yaml * Fixed visualization_server_test.go to reflect new changes * Changed implementation to be fail fast * Changed host name to be constant provided by environment * Added retry and extracted isVisualizationServiceAlive logic to function * Fixed deployment.yaml file * Fixed serviceURL configuration issuse serviceURL is now properly obtained from the environment, the service ip address and port are used rather than service name and namespace * Added log message to indicate when visualization service is unreachable * Addressed PR comments * Removed _HTTP commit ad307db5b96c5965295d59c85e0225f21bd5b550 Author: Eterna2 Date: Thu Aug 22 08:52:32 2019 +0800 [Bug Fix] Delete ResourceOp should not have output parameters (#1822) * Fix bug where delete resource op should not have success_condition, failure_condition, and output parameters * remove unnecessary whitespace * compiler test for delete resource ops should retrieve templates from spec instead of root commit 593f25a5aaf54cc369c04f7bce91471c92fde606 Author: Alexey Volkov Date: Wed Aug 21 17:16:33 2019 -0700 Collecting coverage when running python tests (#898) * Collecting coiverage when running python tests * Added coveralls to python unit tests * Try removing the PATH modification * Specifying coverage run --source * Using the installed package * Try getting the correct coverage paths commit 553885ffb12209e9323cdda63e334b8f428f32bb Author: Alexey Volkov Date: Wed Aug 21 16:38:12 2019 -0700 SDK - Components - Fixed ModelBase comparison bug (#1874) commit 2622c674ea81dd287d94bbd63dc7ea257767faba Author: IronPan Date: Wed Aug 21 16:37:53 2019 -0700 cleanup test dir (#1914) commit 203307dbaf7b8fb9576345feddc2fd7350da718e Author: Alexey Volkov Date: Wed Aug 21 16:37:21 2019 -0700 SDK - Lightweight - Fixed custom types in multi-output case (#1875) The type was mistakenly serialized as `_ForwardRef('CustomType')`. The input parameter types and single-output types were not affected. commit 2e7f2d48160dfb9b6911afcc244e434adf7410e4 Author: IronPan Date: Wed Aug 21 16:36:35 2019 -0700 Add cloud sql and gcs connection for pipeline-lite deployment (#1910) * restructure * working example * working example * move mysql * moving minio and mysql out * add gcp * add files * fix test commit 9adf16301d940217ccde977b0e2b551c348bce6b Author: Alexey Volkov Date: Wed Aug 21 16:29:54 2019 -0700 SDK - Airflow - Fixed bug in airflow op creation (#1911) This PR fixes a bug in AirFlow op creation. The `_run_airflow_op` helper function was not captured along with the `_run_airflow_op_closure` function, because they belong to different modules (`_run_airflow_op_closure` was module-less). This was not discovered during the notebook testing of the code since in that environment the `_run_airflow_op` was also module-less as it was defined in a notebook (not in .py file). commit 7ec56978b6de4f071e47411a3af67a565077ba75 Author: hongye-sun <43763191+hongye-sun@users.noreply.github.com> Date: Wed Aug 21 16:06:31 2019 -0700 Release 151c5349f13bea9d626c988563c04c0a86210c21 (#1916) * Updated component images to version 151c5349f13bea9d626c988563c04c0a86210c21 * Updated components to version a97f1d0ad0e7b92203f35c5b0b9af3a314952e05 * Update setup.py * Update setup.py commit 7e062cea77fe74f49a576f8f9bc035402975638e Author: IronPan Date: Wed Aug 21 15:14:31 2019 -0700 Update README.md commit 8e1e823139d212849a37ee9898da2ba02c29bbe1 Author: Christian Clauss Date: Thu Aug 22 00:04:31 2019 +0200 Lint Python code for undefined names (#1721) * Lint Python code for undefined names * Lint Python code for undefined names * Exclude tfdv.py to workaround an overzealous pytest * Fixup for tfdv.py * Fixup for tfdv.py * Fixup for tfdv.py commit 23993486c5c6c13a238587f7af48f5f73c9919f7 Author: dushyanthsc <43390008+dushyanthsc@users.noreply.github.com> Date: Wed Aug 21 13:44:31 2019 -0700 apiserver: Remove TFX output artifact recording to metadatastore (#1904) commit 151c5349f13bea9d626c988563c04c0a86210c21 Author: Alexey Volkov Date: Wed Aug 21 12:56:33 2019 -0700 Simplified the build_docker_image function (#1887) * Simplified the build_docker_image function * Extracted 'Dockerfile' into a variable Addressed PR feedback * Fixed conflict with updated master. * Addressed the feedback commit 3b7340f25851055c320fc391fe350727f8737fb4 Author: IronPan Date: Wed Aug 21 12:28:31 2019 -0700 Change the type of resource reference payload column (#1905) Gorm doesn't automatically change the type of a column. This changes introduced a column type change which might not be effective for an existing cluster doing upgrade. https://github.com/kubeflow/pipelines/commit/4e43750c9dcebe37b75aedadd7b1c829b409caaa#diff-c4afa92d7e54eecff0a482cf57490aa8R40 /assign @hongye-sun commit a4991fd81ae0c8a4e1b24f743eb8c2774acd2f91 Author: Kirin Patel Date: Wed Aug 21 11:56:31 2019 -0700 Enable error propagation from nbconvert to frontend (#1909) commit 94aa0788d5523de92c803407abb470ea053421f5 Author: Ning Date: Wed Aug 21 10:22:31 2019 -0700 remove tfx notebook sample in favor of the TFX OSS sample (#1908) commit 93e3121644614be84199b0813b91b6bb9797b8f3 Author: Ning Date: Wed Aug 21 09:20:39 2019 -0700 remove kubeflow training (#1902) commit 2592307cceb72fdb61be2673f67d7b4a4bd12023 Author: Christian Clauss Date: Wed Aug 21 04:15:19 2019 +0200 Undefined name 'e' in openvino (#1876) Discovered in #1721 ``` ./contrib/components/openvino/ovms-deployer/containers/evaluate.py:62:16: F821 undefined name 'e' except e: ^ ./contrib/components/openvino/ovms-deployer/containers/evaluate.py:63:50: F821 undefined name 'e' print("Can not read the image file", e) ^ ``` Your review please @Ark-kun commit 7b442f4a54dc6effecc223752c199ce9f48e70a5 Author: Kirin Patel Date: Tue Aug 20 18:27:19 2019 -0700 Created extensible code editor based on react-ace (#1855) * Created extensible code editor based on react-ace * Installed dependencies * Updated unit tests for Editor.tsx to test placeholder and value in simplified manner * Updated Editor unit tests to use snapshot testing where applicable commit d11fae78d8a49e9945760d10bbffc5427c3741d4 Author: Yuan (Bob) Gong Date: Wed Aug 21 08:25:20 2019 +0800 Use KFP lite deployment for presubmit tests (#1808) * Refactor presubmit-tests-with-pipeline-deployment.sh so that it can be run from a different project * Simplify getting service account from cluster. * Migrate presubmit-tests-with-pipeline-deployment.sh to use kfp lightweight deployment. * Add option to cache built images to make debugging faster. * Fix cluster set up * Copy image builder image instead of granting permission * Add missed yes command * fix stuff * Let other usages of image-builder image become configurable * let test workflow use image builder image * Fix permission issue * Hide irrelevant error logs * Use shared service account key instead * Move test manifest to test folder * Move build-images.sh to a different script file * Update README.md * add cluster info dump * Use the same cluster resources as kubeflow deployment * Remove cluster info dump * Add timing to test log * cleaned up code * fix tests * address cr comments * Address cr comments * Enable image caching to improve retest speed commit 0864fafbbc967435574a784e9bb4587ccae02ee4 Author: IronPan Date: Tue Aug 20 14:09:19 2019 -0700 Use single part as default (#1893) The data stored in artifact storage are usually small. Using multi-part is not strictly a requirement. Change the default to true to better support more platform out of box. commit 6284dc10c6a5db491d55a829c8e1e702b7f62dc5 Author: Christian Clauss Date: Tue Aug 20 22:01:18 2019 +0200 IBM Watson samples: from six.moves import xrange (#1877) Discovered in #1721 __xrange()__ was removed in Python 3 in favor of an improved version of __range()__. This PR ensures equivalent functionality in both Python 2 and Python 3. ``` ./samples/contrib/ibm-samples/watson/source/model-source-code/tf-model/input_data.py:100:40: F821 undefined name 'xrange' fake_image = [1.0 for _ in xrange(784)] ^ ./samples/contrib/ibm-samples/watson/source/model-source-code/tf-model/input_data.py:102:41: F821 undefined name 'xrange' return [fake_image for _ in xrange(batch_size)], [ ^ ./samples/contrib/ibm-samples/watson/source/model-source-code/tf-model/input_data.py:103:37: F821 undefined name 'xrange' fake_label for _ in xrange(batch_size)] ^ ``` @gaoning777 @Ark-kun Your reviews please. commit 79c7bdabaf37dffb59608ec2661378867f022667 Author: Ning Date: Tue Aug 20 09:24:56 2019 -0700 fix unit tests and address some comments (#1892) commit 6a7b28f35d8ec06258946f40b62ed97aa2ae688c Author: Aakash Bajaj Date: Tue Aug 20 14:07:33 2019 +0530 gcp cred bug fix for multiple credentials in single pipeline (#1384) * gcp cred bug fix for multiple credentials in single pipeline * squash to remove conflict commit 60fd70c093f4c40d6119d940d184004e374ef09a Author: Andy Wei Date: Tue Aug 20 15:33:33 2019 +0800 Let backend apiserver mysql dbname configurable (#1714) * let mysql dbname configurable * solve conflict * move declaration out to fix scope commit 101a346c0e989d115c1f9856f9f4908e191869da Author: olegchorny Date: Tue Aug 20 08:33:47 2019 +0300 'core' folder included to parameters related On-Premise cluster (#1751) * 'core' folder included to parameters related On-Premise cluster Update is required because this sample was migrated to the samples/core folder * Update README.md commit 4c5d34fe714e76cc7f82b55724c17ccc59258d29 Author: dushyanthsc <43390008+dushyanthsc@users.noreply.github.com> Date: Mon Aug 19 18:57:33 2019 -0700 test/project-cleanup: Support to cleanup gke-clusters in test project (#1857) Change to add base framework for cleaning up resources in a GCP project. The resource specification is specified declaratively using a YAML file. As per current requirements this change only adds cleaning up of GKE clusters. commit 0ed5819ae99fa39a70cff8ce389b2353d72f2031 Author: Yuan (Bob) Gong Date: Tue Aug 20 08:13:31 2019 +0800 Refactor presubmit-tests-with-pipeline-deployment.sh to run in other projects (#1732) * Refactor presubmit-tests-with-pipeline-deployment.sh so that it can be run from a different project * Simplify getting service account from cluster. * Copy image builder image instead of granting permission * Add missed yes command * fix stuff * Let other usages of image-builder image become configurable * let test workflow use image builder image commit 0369a4c382f23c238f28a4236c234f21764ed6ec Author: Ajay Gopinathan Date: Fri Aug 16 21:02:07 2019 -0700 Update manifests to point to 0.26 release. (#1870) commit 2b246bc356c54d70495aff28d86b388aad50857d Author: Alexey Volkov Date: Fri Aug 16 19:54:07 2019 -0700 SDK - Tests - Improved the "ContainerOp.set_retry" test (#1843) Properly testing the feature isntead of just comparing with golden data. commit d66508d486d0a36579deffef5a4b301f55325336 Author: Ajay Gopinathan Date: Fri Aug 16 18:42:48 2019 -0700 Update changelog for 0.1.26 (#1872) * Update changelog for 0.1.26 * Update changelog. commit 1d997048ba7c26c0ff578e9f8608afa53bbaf4f0 Author: Alexey Volkov Date: Fri Aug 16 18:24:09 2019 -0700 SDK - Fixed string comparisons (#1756) commit 6d5ffa28dc6d95cc8c94bd6c05171f231927fe42 Author: Ajay Gopinathan Date: Fri Aug 16 16:57:33 2019 -0700 Remove copying of tfx data for cloudbuild release steps. (#1871) commit de538aa3e3560d9237181b76675481513e5f1940 Author: Ning Date: Fri Aug 16 15:53:07 2019 -0700 add compile step in the samples to generate zip files (#1866) * add compile step in the samples to generate zip files commit 5bcc665928aa521ae4720bf1f201f9acaacb6e0c Author: Ajay Gopinathan Date: Fri Aug 16 13:58:23 2019 -0700 Update Python SDK versions for release. (#1860) commit 0d898cb40f2bf30b711fce07c3d8769a9dc819b9 Author: Ajay Gopinathan Date: Fri Aug 16 13:16:11 2019 -0700 Release 0517114dc2b365a4a6d95424af6157ead774eff3 (#1859) * Updated component images to version 0517114dc2b365a4a6d95424af6157ead774eff3 * Updated components to version 48dd338c8ab328084633c51704cda77db79ac8c2 commit 7dbca1afce27e7f6d038a64d5b24bfcb2fc1b4ba Author: Ning Date: Fri Aug 16 12:06:37 2019 -0700 update gcloud ml-engine to ai-platform (#1863) commit e849f22a28b545402512dbd4a8bff0187336e30a Author: Ryan Dawson Date: Fri Aug 16 19:21:23 2019 +0100 Seldon examples (#1405) commit a6f3f40e5e5efb3e971f51487d0c47ec2ee567c9 Author: sina chavoshi Date: Fri Aug 16 09:56:09 2019 -0700 Adding a sample for serving component (#1830) * Adding a sample for serving component * removed typo / updated based on PR feedback * fixing the jupyter rendering issue * adding pip3 for tensorflow * Fixed spelling error in VERSION * fix indentation based on review feedback commit 54ff3e66142a3043a82bbd91f217734305a30c93 Author: Alexey Volkov Date: Fri Aug 16 01:22:31 2019 -0700 SDK - Cleanup - Serialized PipelineParamTuple does not need value or type (#1469) * SDK - Refactoring - Serialized PipelineParam does not need type Only the types in non-serialized PipelineParams are ever used. * SDK - Refactoring - Serialized PipelineParam does not need value Default values are only relevant when PipelineParam is used in the pipeline function signature and even in this case compiler captures them explicitly from the pipelineParam objects in the signature. There is no other uses for them. commit d2e94e4e04a3f130dbfc4516f75b8eeaf4b708e1 Author: Riley Bauer <34456002+rileyjbauer@users.noreply.github.com> Date: Thu Aug 15 19:52:34 2019 -0700 Fix run duration bug (#1827) * Allows durations >=24h and renames 'showLink' in RunList * Update, fix tests commit d8eaeaad95288709fecd3b4ef6aac34ce7e0c324 Author: Alexey Volkov Date: Thu Aug 15 17:25:59 2019 -0700 SDK - Preserving the pipeline input information in the compiled Workflow (#1381) * SDK - Preserving the pipeline metadata in the compiled Workflow * Stabilizing the DSL compiler tests commit afe8a694f6367b72698ee7fb375f5995278849ba Author: Kirin Patel Date: Thu Aug 15 17:00:28 2019 -0700 Reduce API usage by utilizing reference name in reference resource API (#1824) * Regenerated run api for frontend * Added support for reference name to resource reference API in frontend * Revert "Regenerated run api for frontend" * Addressed PR comments * Removed extra if statement by setting default value of parameter * Removed the whole comment * Addressed PR feedback * Addressed PR feedback * Simplified logic after offline discussion commit ea67c998b647522e183ad52d3cd36c675d973939 Author: Kirin Patel Date: Thu Aug 15 16:18:35 2019 -0700 Change how Variables are Provided to Visualizations (#1754) * Changed way visualization variables are passed from request to NotebookNode Visualization variables are now saved to a json file and loaded by a NotebookNode upon execution. * Updated roc_curve visualization to reflect changes made to dependency injection * Fixed bug where checking if is_generated is provided to roc_curve visualization would crash visualizaiton Also changed ' -> " * Changed text_exporter to always sort variables by key for testing * Addressed PR suggestions commit d238bef1fe93fe7c783fdc43e685bb8b9da451c6 Author: Jiaxiao Zheng Date: Thu Aug 15 12:55:29 2019 -0700 Add back coveralls. (#1849) * Remove redundant import. * Simplify sample_test.yaml by using withItem syntax. * Simplify sample_test.yaml by using withItem syntax. * Change dict to str in withItems. * Add back coveralls. commit 0517114dc2b365a4a6d95424af6157ead774eff3 Author: Riley Bauer <34456002+rileyjbauer@users.noreply.github.com> Date: Thu Aug 15 12:28:35 2019 -0700 Reduce getPipeline calls in RunList (#1852) * Skips calling getPipeline in RunList if the pipeline name is in the pipeline_spec * Update fixed data to include pipeline names in pipeline specs * Remove redundant getRuns call commit 39e5840f2f4b9fde62e4cacec680f4b1ba93f62e Author: IronPan Date: Thu Aug 15 11:04:34 2019 -0700 Add retry button in Pipeline UI (#1782) * add retry button * add retry button * add retry button * address comments * fix tests * fix tests * update image * Update StatusUtils.test.tsx * Update RunDetails.test.tsx * Update Buttons.ts * update test * update frontend * update * update * addrerss comments * update test --- .release.cloudbuild.yaml | 5 - .travis.yml | 21 +- BUILD.bazel | 3 - CHANGELOG.md | 99 ++- WORKSPACE | 178 ++++- backend/src/apiserver/BUILD.bazel | 5 - backend/src/apiserver/client_manager.go | 63 +- backend/src/apiserver/config/config.json | 3 +- backend/src/apiserver/main.go | 13 +- backend/src/apiserver/metadata/BUILD.bazel | 29 - .../src/apiserver/metadata/metadata_store.go | 135 ---- .../apiserver/metadata/metadata_store_test.go | 357 --------- backend/src/apiserver/resource/BUILD.bazel | 3 - .../apiserver/resource/client_manager_fake.go | 18 +- .../apiserver/resource/resource_manager.go | 39 +- .../resource/resource_manager_test.go | 4 + backend/src/apiserver/server/BUILD.bazel | 1 + .../apiserver/server/visualization_server.go | 45 +- .../server/visualization_server_test.go | 60 +- backend/src/apiserver/storage/BUILD.bazel | 4 - backend/src/apiserver/storage/run_store.go | 35 +- .../src/apiserver/storage/run_store_test.go | 2 +- .../src/apiserver/visualization/exporter.py | 22 +- .../src/apiserver/visualization/roc_curve.py | 24 +- backend/src/apiserver/visualization/server.py | 59 +- .../snapshots/snap_test_exporter.py | 89 ++- .../apiserver/visualization/test_exporter.py | 39 +- .../apiserver/visualization/test_server.py | 11 + backend/src/apiserver/visualization/tfdv.py | 2 +- .../python/arena/_arena_distributed_tf_op.py | 33 +- .../emr/create_cluster/src/create_cluster.py | 6 + .../src/submit_pyspark_job.py | 5 + .../submit_spark_job/src/submit_spark_job.py | 5 + .../batch_transform/src/batch_transform.py | 6 + components/dataflow/predict/component.yaml | 2 +- components/dataflow/tfdv/component.yaml | 2 +- components/dataflow/tfma/component.yaml | 2 +- components/dataflow/tft/component.yaml | 2 +- components/gcp/bigquery/query/README.md | 2 +- components/gcp/bigquery/query/component.yaml | 2 +- components/gcp/bigquery/query/sample.ipynb | 2 +- .../google/ml_engine/_create_version.py | 2 +- .../container/component_sdk/python/setup.py | 47 +- .../gcp/dataflow/launch_python/README.md | 2 +- .../gcp/dataflow/launch_python/component.yaml | 2 +- .../gcp/dataflow/launch_python/sample.ipynb | 2 +- .../gcp/dataflow/launch_template/README.md | 2 +- .../dataflow/launch_template/component.yaml | 2 +- .../gcp/dataflow/launch_template/sample.ipynb | 2 +- .../gcp/dataproc/create_cluster/README.md | 2 +- .../dataproc/create_cluster/component.yaml | 2 +- .../gcp/dataproc/create_cluster/sample.ipynb | 2 +- .../gcp/dataproc/delete_cluster/README.md | 2 +- .../dataproc/delete_cluster/component.yaml | 2 +- .../gcp/dataproc/delete_cluster/sample.ipynb | 2 +- .../gcp/dataproc/submit_hadoop_job/README.md | 2 +- .../dataproc/submit_hadoop_job/component.yaml | 2 +- .../dataproc/submit_hadoop_job/sample.ipynb | 2 +- .../gcp/dataproc/submit_hive_job/README.md | 2 +- .../dataproc/submit_hive_job/component.yaml | 2 +- .../gcp/dataproc/submit_hive_job/sample.ipynb | 2 +- .../gcp/dataproc/submit_pig_job/README.md | 2 +- .../dataproc/submit_pig_job/component.yaml | 2 +- .../gcp/dataproc/submit_pig_job/sample.ipynb | 2 +- .../gcp/dataproc/submit_pyspark_job/README.md | 2 +- .../submit_pyspark_job/component.yaml | 2 +- .../dataproc/submit_pyspark_job/sample.ipynb | 2 +- .../gcp/dataproc/submit_spark_job/README.md | 2 +- .../dataproc/submit_spark_job/component.yaml | 2 +- .../dataproc/submit_spark_job/sample.ipynb | 2 +- .../dataproc/submit_sparksql_job/README.md | 2 +- .../submit_sparksql_job/component.yaml | 2 +- .../dataproc/submit_sparksql_job/sample.ipynb | 2 +- .../gcp/ml_engine/batch_predict/README.md | 2 +- .../ml_engine/batch_predict/component.yaml | 2 +- .../gcp/ml_engine/batch_predict/sample.ipynb | 2 +- components/gcp/ml_engine/deploy/README.md | 2 +- .../gcp/ml_engine/deploy/component.yaml | 2 +- components/gcp/ml_engine/deploy/sample.ipynb | 2 +- components/gcp/ml_engine/train/README.md | 2 +- components/gcp/ml_engine/train/component.yaml | 2 +- components/gcp/ml_engine/train/sample.ipynb | 2 +- components/kubeflow/deployer/component.yaml | 2 +- components/kubeflow/dnntrainer/component.yaml | 2 +- .../launcher/kubeflow_tfjob_launcher_op.py | 2 +- .../kubeflow/launcher/src/train.template.yaml | 6 +- .../local/confusion_matrix/component.yaml | 2 +- components/local/roc/component.yaml | 2 +- .../ovms-deployer/containers/evaluate.py | 4 +- frontend/mock-backend/fixed-data.ts | 63 +- frontend/mock-backend/json-runtime.ts | 99 +++ frontend/package-lock.json | 96 ++- frontend/package.json | 5 +- frontend/src/apis/experiment/api.ts | 32 +- frontend/src/apis/filter/api.ts | 35 +- frontend/src/apis/job/api.ts | 60 +- frontend/src/apis/pipeline/api.ts | 14 +- frontend/src/apis/run/api.ts | 88 ++- frontend/src/components/DetailsTable.tsx | 28 +- frontend/src/components/Editor.test.tsx | 52 ++ frontend/src/components/Editor.tsx | 45 ++ .../__snapshots__/DetailsTable.test.tsx.snap | 245 ++++-- .../__snapshots__/Editor.test.tsx.snap | 7 + frontend/src/lib/Buttons.ts | 27 + frontend/src/lib/RunUtils.ts | 32 +- frontend/src/lib/Utils.test.ts | 24 +- frontend/src/lib/Utils.ts | 2 +- frontend/src/lib/WorkflowParser.ts | 10 +- frontend/src/pages/ExperimentDetails.tsx | 10 +- frontend/src/pages/NewRun.tsx | 4 +- frontend/src/pages/PipelineDetails.tsx | 24 +- frontend/src/pages/RunDetails.tsx | 16 +- frontend/src/pages/RunList.test.tsx | 19 +- frontend/src/pages/RunList.tsx | 58 +- .../PipelineDetails.test.tsx.snap | 46 +- .../__snapshots__/RunDetails.test.tsx.snap | 6 +- .../pages/__snapshots__/RunList.test.tsx.snap | 75 +- go.mod | 29 +- go.sum | 139 ++++ manifests/kustomize/README.md | 11 +- .../kustomize/base/argo/kustomization.yaml | 1 + .../minio-artifact-secret.yaml | 0 manifests/kustomize/base/kustomization.yaml | 20 +- .../base/pipeline/kustomization.yaml | 2 + .../ml-pipeline-visualization-deployment.yaml | 21 + .../ml-pipeline-visualization-service.yaml | 12 + .../kustomize/env/dev/kustomization.yaml | 16 + .../dev}/minio/kustomization.yaml | 1 - .../dev}/minio/minio-deployment.yaml | 2 +- .../{base => env/dev}/minio/minio-pvc.yaml | 0 .../dev}/minio/minio-service.yaml | 0 .../dev}/mysql/kustomization.yaml | 0 .../dev}/mysql/mysql-deployment.yaml | 0 .../dev}/mysql/mysql-pv-claim.yaml | 0 .../dev}/mysql/mysql-service.yaml | 0 manifests/kustomize/env/gcp/.gitignore | 2 + .../kustomize/env/gcp/kustomization.yaml | 26 + .../env/gcp/minio/kustomization.yaml | 6 + .../minio/minio-gcs-gateway-deployment.yaml | 37 + .../gcp/minio/minio-gcs-gateway-service.yaml | 11 + ...l-pipeline-apiserver-deployment-patch.yaml | 16 + .../gcp/mysql/cloudsql-proxy-deployment.yaml | 42 + .../env/gcp/mysql/kustomization.yaml | 6 + .../env/gcp/mysql/mysql-service.yaml | 10 + manifests/kustomize/namespaced-install.yaml | 20 +- .../kustomize/namespaced-install/README.md | 2 - .../kustomization.yaml | 4 +- .../kustomizeconfig/namespace.yaml | 0 ...ine-persistenceagent-deployment-patch.yaml | 0 ...ne-scheduledworkflow-deployment-patch.yaml | 0 .../ml-pipeline-ui-deployment-patch.yaml | 0 ...-pipeline-viewer-crd-deployment-patch.yaml | 0 .../namespace.yaml | 0 .../workflow-controller-configmap.yaml | 0 proxy/get_proxy_url.py | 5 + .../mini-image-classification-pipeline.py | 2 +- .../model-source-code/tf-model/input_data.py | 3 +- .../resnet-cmle/resnet-train-pipeline.py | 7 +- samples/contrib/seldon/README.md | 10 + samples/contrib/seldon/iris_storagebucket.py | 67 ++ samples/contrib/seldon/mabdeploy_seldon.py | 152 ++++ samples/contrib/seldon/mnist_tf.py | 235 ++++++ samples/contrib/seldon/mnist_tf_volume.py | 180 +++++ .../ai-platform/Chicago Crime Pipeline.ipynb | 19 +- .../artifact_location/artifact_location.py | 4 + samples/core/condition/condition.py | 1 - .../DSL Static Type Checking.ipynb | 31 +- .../core/imagepullsecrets/imagepullsecrets.py | 4 + ...ow Pipeline Using TFX OSS Components.ipynb | 745 ------------------ .../kubeflow_tf_serving.ipynb | 297 +++++++ .../README.md | 47 -- .../kubeflow_training_classification.py | 91 --- samples/core/recursion/recursion.py | 2 +- samples/core/resource_ops/resourceop_basic.py | 11 +- samples/core/sidecar/sidecar.py | 5 +- samples/core/tfx_cab_classification/README.md | 16 +- .../tfx_cab_classification.py | 16 +- samples/core/volume_ops/volumeop.py | 8 +- .../volume_snapshot_ops/volume_snapshot_op.py | 9 +- samples/core/xgboost_training_cm/README.md | 4 +- .../xgboost_training_cm.py | 16 +- sdk/python/kfp/compiler/_component_builder.py | 40 +- sdk/python/kfp/compiler/_container_builder.py | 4 +- sdk/python/kfp/compiler/compiler.py | 4 + sdk/python/kfp/components/_airflow_op.py | 1 + sdk/python/kfp/components/_dynamic.py | 8 +- sdk/python/kfp/components/_python_op.py | 6 +- sdk/python/kfp/components/modelbase.py | 2 +- sdk/python/kfp/dsl/_pipeline.py | 6 +- sdk/python/kfp/dsl/_pipeline_param.py | 27 +- sdk/python/kfp/dsl/_resource_op.py | 11 + sdk/python/kfp/gcp.py | 10 +- sdk/python/setup.py | 14 +- sdk/python/tests/compiler/compiler_tests.py | 53 +- .../tests/compiler/container_builder_test.py | 5 +- .../tests/compiler/testdata/add_pod_env.yaml | 3 + .../compiler/testdata/artifact_location.yaml | 5 + sdk/python/tests/compiler/testdata/basic.yaml | 2 + sdk/python/tests/compiler/testdata/coin.yaml | 2 + .../tests/compiler/testdata/compose.yaml | 2 + .../compiler/testdata/default_value.yaml | 2 + .../compiler/testdata/imagepullsecrets.yaml | 2 + .../compiler/testdata/param_op_transform.yaml | 6 + .../testdata/param_substitutions.yaml | 3 + .../compiler/testdata/pipelineparams.yaml | 5 + .../testdata/preemptible_tpu_gpu.yaml | 2 + .../compiler/testdata/recursive_do_while.yaml | 3 + .../compiler/testdata/recursive_while.yaml | 4 + .../compiler/testdata/resourceop_basic.yaml | 5 + sdk/python/tests/compiler/testdata/retry.py | 42 - sdk/python/tests/compiler/testdata/retry.yaml | 55 -- .../tests/compiler/testdata/sidecar.yaml | 3 + .../tests/compiler/testdata/timeout.yaml | 4 + .../tests/compiler/testdata/volume.yaml | 3 + .../testdata/volume_snapshotop_rokurl.yaml | 4 + .../volume_snapshotop_sequential.yaml | 4 + .../compiler/testdata/volumeop_basic.yaml | 4 + .../tests/compiler/testdata/volumeop_dag.yaml | 3 + .../compiler/testdata/volumeop_parallel.yaml | 3 + .../testdata/volumeop_sequential.yaml | 3 + .../tests/components/test_components.py | 1 - .../tests/components/test_graph_components.py | 1 - sdk/python/tests/components/test_python_op.py | 75 ++ .../components/test_structure_model_base.py | 15 + sdk/python/tests/dsl/component_tests.py | 2 +- sdk/python/tests/dsl/pipeline_param_tests.py | 6 +- test/.gitignore | 2 + test/README.md | 20 + test/build-images.sh | 59 ++ test/build_image.yaml | 4 +- test/check-argo-status.sh | 5 +- test/component_test.yaml | 4 +- test/deploy-cluster.sh | 73 ++ test/deploy-kubeflow.sh | 59 -- test/deploy-pipeline-lite.sh | 47 ++ test/deploy-pipeline.sh | 64 -- test/e2e_test_gke_v2.yaml | 4 +- test/install-argo.sh | 27 +- test/manifests/kustomization.yaml | 7 + test/minikube/OWNERS | 6 - .../install_and_start_minikube_without_vm.sh | 46 -- test/minikube/install_docker.sh | 34 - test/presubmit-tests-gce-minikube.sh | 87 -- ...resubmit-tests-with-pipeline-deployment.sh | 44 +- test/sample-test/run_sample_test.py | 26 +- test/sample-test/run_test.sh | 60 +- test/sample_test.yaml | 6 +- test/test-prep.sh | 6 +- test/tools/project-cleaner/main.go | 164 ++++ .../project-cleaner/project_cleaner.sh} | 15 +- .../project-cleaner/resource_spec.yaml} | 19 +- 251 files changed, 3813 insertions(+), 2769 deletions(-) delete mode 100644 backend/src/apiserver/metadata/BUILD.bazel delete mode 100644 backend/src/apiserver/metadata/metadata_store.go delete mode 100644 backend/src/apiserver/metadata/metadata_store_test.go create mode 100644 frontend/mock-backend/json-runtime.ts create mode 100644 frontend/src/components/Editor.test.tsx create mode 100644 frontend/src/components/Editor.tsx create mode 100644 frontend/src/components/__snapshots__/Editor.test.tsx.snap rename manifests/kustomize/base/{minio => argo}/minio-artifact-secret.yaml (100%) create mode 100644 manifests/kustomize/base/pipeline/ml-pipeline-visualization-deployment.yaml create mode 100644 manifests/kustomize/base/pipeline/ml-pipeline-visualization-service.yaml create mode 100644 manifests/kustomize/env/dev/kustomization.yaml rename manifests/kustomize/{base => env/dev}/minio/kustomization.yaml (82%) rename manifests/kustomize/{base => env/dev}/minio/minio-deployment.yaml (90%) rename manifests/kustomize/{base => env/dev}/minio/minio-pvc.yaml (100%) rename manifests/kustomize/{base => env/dev}/minio/minio-service.yaml (100%) rename manifests/kustomize/{base => env/dev}/mysql/kustomization.yaml (100%) rename manifests/kustomize/{base => env/dev}/mysql/mysql-deployment.yaml (100%) rename manifests/kustomize/{base => env/dev}/mysql/mysql-pv-claim.yaml (100%) rename manifests/kustomize/{base => env/dev}/mysql/mysql-service.yaml (100%) create mode 100644 manifests/kustomize/env/gcp/.gitignore create mode 100644 manifests/kustomize/env/gcp/kustomization.yaml create mode 100644 manifests/kustomize/env/gcp/minio/kustomization.yaml create mode 100644 manifests/kustomize/env/gcp/minio/minio-gcs-gateway-deployment.yaml create mode 100644 manifests/kustomize/env/gcp/minio/minio-gcs-gateway-service.yaml create mode 100644 manifests/kustomize/env/gcp/ml-pipeline-apiserver-deployment-patch.yaml create mode 100644 manifests/kustomize/env/gcp/mysql/cloudsql-proxy-deployment.yaml create mode 100644 manifests/kustomize/env/gcp/mysql/kustomization.yaml create mode 100644 manifests/kustomize/env/gcp/mysql/mysql-service.yaml delete mode 100644 manifests/kustomize/namespaced-install/README.md rename manifests/kustomize/{namespaced-install => namespaced}/kustomization.yaml (94%) rename manifests/kustomize/{namespaced-install => namespaced}/kustomizeconfig/namespace.yaml (100%) rename manifests/kustomize/{namespaced-install => namespaced}/ml-pipeline-persistenceagent-deployment-patch.yaml (100%) rename manifests/kustomize/{namespaced-install => namespaced}/ml-pipeline-scheduledworkflow-deployment-patch.yaml (100%) rename manifests/kustomize/{namespaced-install => namespaced}/ml-pipeline-ui-deployment-patch.yaml (100%) rename manifests/kustomize/{namespaced-install => namespaced}/ml-pipeline-viewer-crd-deployment-patch.yaml (100%) rename manifests/kustomize/{namespaced-install => namespaced}/namespace.yaml (100%) rename manifests/kustomize/{namespaced-install => namespaced}/workflow-controller-configmap.yaml (100%) create mode 100644 samples/contrib/seldon/README.md create mode 100644 samples/contrib/seldon/iris_storagebucket.py create mode 100644 samples/contrib/seldon/mabdeploy_seldon.py create mode 100644 samples/contrib/seldon/mnist_tf.py create mode 100644 samples/contrib/seldon/mnist_tf_volume.py delete mode 100644 samples/core/kubeflow_pipeline_using_TFX_OSS_components/KubeFlow Pipeline Using TFX OSS Components.ipynb create mode 100644 samples/core/kubeflow_tf_serving/kubeflow_tf_serving.ipynb delete mode 100644 samples/core/kubeflow_training_classification/README.md delete mode 100755 samples/core/kubeflow_training_classification/kubeflow_training_classification.py delete mode 100755 sdk/python/tests/compiler/testdata/retry.py delete mode 100644 sdk/python/tests/compiler/testdata/retry.yaml create mode 100644 test/.gitignore create mode 100755 test/build-images.sh create mode 100755 test/deploy-cluster.sh delete mode 100755 test/deploy-kubeflow.sh create mode 100755 test/deploy-pipeline-lite.sh delete mode 100755 test/deploy-pipeline.sh create mode 100644 test/manifests/kustomization.yaml delete mode 100644 test/minikube/OWNERS delete mode 100755 test/minikube/install_and_start_minikube_without_vm.sh delete mode 100755 test/minikube/install_docker.sh delete mode 100755 test/presubmit-tests-gce-minikube.sh create mode 100644 test/tools/project-cleaner/main.go rename test/{minikube/install_argo_client.sh => tools/project-cleaner/project_cleaner.sh} (72%) rename test/{minikube/install_docker_minikube_argo.sh => tools/project-cleaner/resource_spec.yaml} (56%) mode change 100755 => 100644 diff --git a/.release.cloudbuild.yaml b/.release.cloudbuild.yaml index 50b3f0431de..51941447aff 100644 --- a/.release.cloudbuild.yaml +++ b/.release.cloudbuild.yaml @@ -124,11 +124,6 @@ steps: id: 'copyPythonComponentSDKToLatest' waitFor: ['copyPythonComponentSDKLocal'] -# Copy the TFX sample data to the ml-pipeline-playground bucket -- name: 'gcr.io/cloud-builders/gsutil' - args: ['cp', '-r', '/workspace/samples/tfx/taxi-cab-classification/*', 'gs://ml-pipeline-playground/tfx/taxi-cab-classification/'] - id: 'copyTFXData' - images: - 'gcr.io/ml-pipeline/frontend:$TAG_NAME' - 'gcr.io/ml-pipeline/frontend:$COMMIT_SHA' diff --git a/.travis.yml b/.travis.yml index adf2ef33f6f..d86e882c566 100644 --- a/.travis.yml +++ b/.travis.yml @@ -24,7 +24,8 @@ matrix: - cd $TRAVIS_BUILD_DIR/frontend - node -v - npm i - - npm run test -- --coverage + # Comment out next line if coverall has an ongoing outage to unblock. + - npm run test:coveralls - language: generic env: - BAZEL_URL="https://github.com/bazelbuild/bazel/releases/download/0.23.0/bazel-0.23.0-installer-linux-x86_64.sh" @@ -65,13 +66,15 @@ matrix: env: TOXENV=py35 script: &1 # Additional dependencies - - pip3 install jsonschema==3.0.1 + - pip3 install coverage coveralls jsonschema==3.0.1 # DSL tests - cd $TRAVIS_BUILD_DIR/sdk/python - - python3 setup.py install - - python3 tests/dsl/main.py - - python3 tests/compiler/main.py - - $TRAVIS_BUILD_DIR/sdk/python/tests/run_tests.sh + - python3 setup.py develop + - cd $TRAVIS_BUILD_DIR # Changing the current directory to the repo root for correct coverall paths + - coverage run --source=kfp --append sdk/python/tests/dsl/main.py + - coverage run --source=kfp --append sdk/python/tests/compiler/main.py + - coverage run --source=kfp --append -m unittest discover --verbose --start-dir sdk/python/tests --top-level-directory=sdk/python + - coveralls # Visualization test - cd $TRAVIS_BUILD_DIR/backend/src/apiserver/visualization @@ -93,5 +96,9 @@ matrix: - language: python python: "3.7" env: TOXENV=py37 - dist: xenial # required for Python >= 3.7 script: *1 + - name: "Lint Python code with flake8" + language: python + python: "3.7" + install: pip install flake8 + script: flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics diff --git a/BUILD.bazel b/BUILD.bazel index 0e9906756fd..94797ba522d 100644 --- a/BUILD.bazel +++ b/BUILD.bazel @@ -4,8 +4,5 @@ load("@bazel_gazelle//:def.bzl", "gazelle") # gazelle:resolve proto protoc-gen-swagger/options/annotations.proto @com_github_grpc_ecosystem_grpc_gateway//protoc-gen-swagger/options:options_proto # gazelle:resolve proto go protoc-gen-swagger/options/annotations.proto @com_github_grpc_ecosystem_grpc_gateway//protoc-gen-swagger/options:go_default_library # gazelle:resolve go github.com/kubeflow/pipelines/backend/api/go_client //backend/api:go_default_library -# gazelle:resolve go ml_metadata/metadata_store/mlmetadata @google_ml_metadata//ml_metadata/metadata_store:metadata_store_go -# gazelle:resolve go ml_metadata/proto/metadata_store_go_proto @google_ml_metadata//ml_metadata/proto:metadata_store_go_proto -# gazelle:resolve go ml_metadata/proto/metadata_store_service_go_proto @google_ml_metadata//ml_metadata/proto:metadata_store_service_go_proto # gazelle:exclude vendor/ gazelle(name = "gazelle") diff --git a/CHANGELOG.md b/CHANGELOG.md index 0ed264a3485..29a5465670c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,102 @@ # Change Log +## [0.1.26](https://github.com/kubeflow/pipelines/tree/0.1.26) (2019-08-16) +[Full Changelog](https://github.com/kubeflow/pipelines/compare/0.1.25...0.1.26) + +**Merged pull requests:** + +- update gcloud ml-engine to ai-platform [\#1863](https://github.com/kubeflow/pipelines/pull/1863) ([gaoning777](https://github.com/gaoning777)) +- Release 0517114dc2b365a4a6d95424af6157ead774eff3 [\#1859](https://github.com/kubeflow/pipelines/pull/1859) ([neuromage](https://github.com/neuromage)) +- Reduce getPipeline calls in RunList [\#1852](https://github.com/kubeflow/pipelines/pull/1852) ([rileyjbauer](https://github.com/rileyjbauer)) +- Add back coveralls. [\#1849](https://github.com/kubeflow/pipelines/pull/1849) ([numerology](https://github.com/numerology)) +- Propagate pipeline name in pipeline spec [\#1842](https://github.com/kubeflow/pipelines/pull/1842) ([IronPan](https://github.com/IronPan)) +- Create composite indexes \[ResourceType, ReferenceUUID, ReferenceType\] [\#1836](https://github.com/kubeflow/pipelines/pull/1836) ([IronPan](https://github.com/IronPan)) +- Improve sql efficiency for getting the run [\#1835](https://github.com/kubeflow/pipelines/pull/1835) ([IronPan](https://github.com/IronPan)) +- Adding a sample for serving component [\#1830](https://github.com/kubeflow/pipelines/pull/1830) ([SinaChavoshi](https://github.com/SinaChavoshi)) +- Update for sample repo restructuring [\#1828](https://github.com/kubeflow/pipelines/pull/1828) ([zanedurante](https://github.com/zanedurante)) +- Fix run duration bug [\#1827](https://github.com/kubeflow/pipelines/pull/1827) ([rileyjbauer](https://github.com/rileyjbauer)) +- Reduce API usage by utilizing reference name in reference resource API [\#1824](https://github.com/kubeflow/pipelines/pull/1824) ([ajchili](https://github.com/ajchili)) +- Update npm test to not use coverall [\#1819](https://github.com/kubeflow/pipelines/pull/1819) ([IronPan](https://github.com/IronPan)) +- Add subprocess pip install example in lightweight component example notebook [\#1817](https://github.com/kubeflow/pipelines/pull/1817) ([Bobgy](https://github.com/Bobgy)) +- Build - Fix CloudBuild bug [\#1816](https://github.com/kubeflow/pipelines/pull/1816) ([Ark-kun](https://github.com/Ark-kun)) +- Refactors toolbar buttons to use a map rather than an array [\#1812](https://github.com/kubeflow/pipelines/pull/1812) ([rileyjbauer](https://github.com/rileyjbauer)) +- Disable flaky tests temporarily [\#1809](https://github.com/kubeflow/pipelines/pull/1809) ([Bobgy](https://github.com/Bobgy)) +- Fix test loophole for loading samples during KFP startup [\#1807](https://github.com/kubeflow/pipelines/pull/1807) ([IronPan](https://github.com/IronPan)) +- Container builder default gcr [\#1806](https://github.com/kubeflow/pipelines/pull/1806) ([gaoning777](https://github.com/gaoning777)) +- Fix the broken sample path in API [\#1805](https://github.com/kubeflow/pipelines/pull/1805) ([IronPan](https://github.com/IronPan)) +- Garbage collect the completed workflow after persisted to database [\#1802](https://github.com/kubeflow/pipelines/pull/1802) ([IronPan](https://github.com/IronPan)) +- Fix github security alert. [\#1798](https://github.com/kubeflow/pipelines/pull/1798) ([hongye-sun](https://github.com/hongye-sun)) +- ContainerBuilder loading kube config [\#1795](https://github.com/kubeflow/pipelines/pull/1795) ([gaoning777](https://github.com/gaoning777)) +- Move TF installation to notebooks [\#1793](https://github.com/kubeflow/pipelines/pull/1793) ([numerology](https://github.com/numerology)) +- Move argo installation to dockerfile from bash script. [\#1792](https://github.com/kubeflow/pipelines/pull/1792) ([numerology](https://github.com/numerology)) +- fix sample reference link [\#1789](https://github.com/kubeflow/pipelines/pull/1789) ([gaoning777](https://github.com/gaoning777)) +- skip storing log to files [\#1788](https://github.com/kubeflow/pipelines/pull/1788) ([IronPan](https://github.com/IronPan)) +- Remove yebrahim from approvers/reviewers [\#1787](https://github.com/kubeflow/pipelines/pull/1787) ([yebrahim](https://github.com/yebrahim)) +- update owner files in samples and test [\#1785](https://github.com/kubeflow/pipelines/pull/1785) ([gaoning777](https://github.com/gaoning777)) +- Fixed Dockerfile used for buildVisualizationServer in Cloud Build [\#1783](https://github.com/kubeflow/pipelines/pull/1783) ([ajchili](https://github.com/ajchili)) +- Add retry button in Pipeline UI [\#1782](https://github.com/kubeflow/pipelines/pull/1782) ([IronPan](https://github.com/IronPan)) +- add reference name to resource reference API proto [\#1781](https://github.com/kubeflow/pipelines/pull/1781) ([IronPan](https://github.com/IronPan)) +- Update images, bug fixes, clean up code [\#1778](https://github.com/kubeflow/pipelines/pull/1778) ([carolynwang](https://github.com/carolynwang)) +- Container builder [\#1774](https://github.com/kubeflow/pipelines/pull/1774) ([gaoning777](https://github.com/gaoning777)) +- fix api server sort test [\#1769](https://github.com/kubeflow/pipelines/pull/1769) ([IronPan](https://github.com/IronPan)) +- SDK - Containers - Returning image name with digest [\#1768](https://github.com/kubeflow/pipelines/pull/1768) ([Ark-kun](https://github.com/Ark-kun)) +- Move imagepullsecrets sample to samples/core [\#1767](https://github.com/kubeflow/pipelines/pull/1767) ([numerology](https://github.com/numerology)) +- Not return error if run update doesn't change DB entry [\#1765](https://github.com/kubeflow/pipelines/pull/1765) ([IronPan](https://github.com/IronPan)) +- remove copying the samples since we are not releasing the samples in the GCS [\#1764](https://github.com/kubeflow/pipelines/pull/1764) ([gaoning777](https://github.com/gaoning777)) +- Backend - Docker build should fail on sample compilation failures [\#1760](https://github.com/kubeflow/pipelines/pull/1760) ([Ark-kun](https://github.com/Ark-kun)) +- Move samples to the correct location [\#1759](https://github.com/kubeflow/pipelines/pull/1759) ([gaoning777](https://github.com/gaoning777)) +- Change how Variables are Provided to Visualizations [\#1754](https://github.com/kubeflow/pipelines/pull/1754) ([ajchili](https://github.com/ajchili)) +- Add preemtptible gpu sample [\#1749](https://github.com/kubeflow/pipelines/pull/1749) ([numerology](https://github.com/numerology)) +- Revert "Backend - Updated the version of the ml metadata package" [\#1747](https://github.com/kubeflow/pipelines/pull/1747) ([Ark-kun](https://github.com/Ark-kun)) +- Revert "Backend - Starting the api-server container build from scratch" [\#1742](https://github.com/kubeflow/pipelines/pull/1742) ([Ark-kun](https://github.com/Ark-kun)) +- Refactor aws samples to match new folder structure [\#1741](https://github.com/kubeflow/pipelines/pull/1741) ([carolynwang](https://github.com/carolynwang)) +- Components - Added the pymongo license URL [\#1740](https://github.com/kubeflow/pipelines/pull/1740) ([Ark-kun](https://github.com/Ark-kun)) +- Add Visualization Server to Cloud Build yaml Files [\#1738](https://github.com/kubeflow/pipelines/pull/1738) ([ajchili](https://github.com/ajchili)) +- Update Watson Machine Learning auth with IAM [\#1737](https://github.com/kubeflow/pipelines/pull/1737) ([Tomcli](https://github.com/Tomcli)) +- Delete KFP component before reinstalling again [\#1736](https://github.com/kubeflow/pipelines/pull/1736) ([IronPan](https://github.com/IronPan)) +- Moving samples to match the new folder structure [\#1734](https://github.com/kubeflow/pipelines/pull/1734) ([SinaChavoshi](https://github.com/SinaChavoshi)) +- fix cloudbuild failure [\#1733](https://github.com/kubeflow/pipelines/pull/1733) ([gaoning777](https://github.com/gaoning777)) +- Refactor sample tests configuration to reduce the efforts of adding samples. [\#1730](https://github.com/kubeflow/pipelines/pull/1730) ([numerology](https://github.com/numerology)) +- SDK - Lightweight - Fixed regression for components without outputs [\#1726](https://github.com/kubeflow/pipelines/pull/1726) ([Ark-kun](https://github.com/Ark-kun)) +- Backend - Updated the version of the ml metadata package [\#1725](https://github.com/kubeflow/pipelines/pull/1725) ([Ark-kun](https://github.com/Ark-kun)) +- Add API to rerun the pipeline [\#1720](https://github.com/kubeflow/pipelines/pull/1720) ([IronPan](https://github.com/IronPan)) +- Remove outdated argo install instruction [\#1719](https://github.com/kubeflow/pipelines/pull/1719) ([Bobgy](https://github.com/Bobgy)) +- SDK - ContainerOp.set\_display\_name should return self to enable chaining [\#1718](https://github.com/kubeflow/pipelines/pull/1718) ([Ark-kun](https://github.com/Ark-kun)) +- Rename InputPath -\> Source for Visualization API definition [\#1717](https://github.com/kubeflow/pipelines/pull/1717) ([ajchili](https://github.com/ajchili)) +- Add SageMaker create workteam and Ground Truth components, sample demo pipeline, other minor updates [\#1716](https://github.com/kubeflow/pipelines/pull/1716) ([carolynwang](https://github.com/carolynwang)) +- Support Single part PutFile [\#1713](https://github.com/kubeflow/pipelines/pull/1713) ([nirsagi](https://github.com/nirsagi)) +- Fixes cloning of recurring runs [\#1712](https://github.com/kubeflow/pipelines/pull/1712) ([rileyjbauer](https://github.com/rileyjbauer)) +- Restructure samples [\#1710](https://github.com/kubeflow/pipelines/pull/1710) ([gaoning777](https://github.com/gaoning777)) +- Simplify sample\_test.yaml [\#1709](https://github.com/kubeflow/pipelines/pull/1709) ([numerology](https://github.com/numerology)) +- add jxzheng to the reviewers for samples [\#1705](https://github.com/kubeflow/pipelines/pull/1705) ([gaoning777](https://github.com/gaoning777)) +- Component build fix [\#1703](https://github.com/kubeflow/pipelines/pull/1703) ([gaoning777](https://github.com/gaoning777)) +- Allows creation of jobs without experiments [\#1702](https://github.com/kubeflow/pipelines/pull/1702) ([rileyjbauer](https://github.com/rileyjbauer)) +- Backend - Starting the api-server container build from scratch [\#1699](https://github.com/kubeflow/pipelines/pull/1699) ([Ark-kun](https://github.com/Ark-kun)) +- Moving component\_sdk to components/gcp/ [\#1698](https://github.com/kubeflow/pipelines/pull/1698) ([Ark-kun](https://github.com/Ark-kun)) +- SDK - Lightweight - Added support for complex default values [\#1696](https://github.com/kubeflow/pipelines/pull/1696) ([Ark-kun](https://github.com/Ark-kun)) +- Changelog 0.1.25 [\#1695](https://github.com/kubeflow/pipelines/pull/1695) ([jingzhang36](https://github.com/jingzhang36)) +- Move kustomize manifests a dedicate directory [\#1690](https://github.com/kubeflow/pipelines/pull/1690) ([IronPan](https://github.com/IronPan)) +- Clears the workflow's name in GetWorkflowSpec and uses it for the GenerateName [\#1689](https://github.com/kubeflow/pipelines/pull/1689) ([rileyjbauer](https://github.com/rileyjbauer)) +- API - Updated swagger-codegen-cli version [\#1686](https://github.com/kubeflow/pipelines/pull/1686) ([Ark-kun](https://github.com/Ark-kun)) +- Update SageMaker components and sample pipeline [\#1682](https://github.com/kubeflow/pipelines/pull/1682) ([carolynwang](https://github.com/carolynwang)) +- Basic component build sample [\#1681](https://github.com/kubeflow/pipelines/pull/1681) ([SinaChavoshi](https://github.com/SinaChavoshi)) +- Separate codegen from containerbuild 2 [\#1680](https://github.com/kubeflow/pipelines/pull/1680) ([gaoning777](https://github.com/gaoning777)) +- Separate codegen from containerbuild [\#1679](https://github.com/kubeflow/pipelines/pull/1679) ([gaoning777](https://github.com/gaoning777)) +- Add new PlotType to Allow for Visualization Creation [\#1677](https://github.com/kubeflow/pipelines/pull/1677) ([ajchili](https://github.com/ajchili)) +- Container op mount secret sample [\#1676](https://github.com/kubeflow/pipelines/pull/1676) ([SinaChavoshi](https://github.com/SinaChavoshi)) +- SDK/Lightweight - Updated default image to tensorflow:1.13.2-py3 [\#1671](https://github.com/kubeflow/pipelines/pull/1671) ([Ark-kun](https://github.com/Ark-kun)) +- Adding a sample for explicitly defining the execution order [\#1668](https://github.com/kubeflow/pipelines/pull/1668) ([SinaChavoshi](https://github.com/SinaChavoshi)) +- Adding multiple outputs into sdk with sample [\#1667](https://github.com/kubeflow/pipelines/pull/1667) ([zanedurante](https://github.com/zanedurante)) +- SDK - Removed the build\_image parameter from build\_python\_component function [\#1657](https://github.com/kubeflow/pipelines/pull/1657) ([Ark-kun](https://github.com/Ark-kun)) +- update kaniko executor version to speed up image build [\#1652](https://github.com/kubeflow/pipelines/pull/1652) ([gaoning777](https://github.com/gaoning777)) +- Add code for python visualization service [\#1651](https://github.com/kubeflow/pipelines/pull/1651) ([ajchili](https://github.com/ajchili)) +- SDK/Client - Added the create\_run\_from\_pipeline\_package method [\#1523](https://github.com/kubeflow/pipelines/pull/1523) ([Ark-kun](https://github.com/Ark-kun)) +- SDK - Using Airflow ops in Pipelines [\#1483](https://github.com/kubeflow/pipelines/pull/1483) ([Ark-kun](https://github.com/Ark-kun)) +- SDK - Cleanup - Serialized PipelineParamTuple does not need value or type [\#1469](https://github.com/kubeflow/pipelines/pull/1469) ([Ark-kun](https://github.com/Ark-kun)) +- Reorganize ResourceOp samples [\#1433](https://github.com/kubeflow/pipelines/pull/1433) ([elikatsis](https://github.com/elikatsis)) +- add default value type checking [\#1407](https://github.com/kubeflow/pipelines/pull/1407) ([gaoning777](https://github.com/gaoning777)) +- Seldon examples [\#1405](https://github.com/kubeflow/pipelines/pull/1405) ([ryandawsonuk](https://github.com/ryandawsonuk)) + ## [0.1.25](https://github.com/kubeflow/pipelines/tree/0.1.25) (2019-07-26) [Full Changelog](https://github.com/kubeflow/pipelines/compare/0.1.24...0.1.25) @@ -1394,4 +1491,4 @@ -\* *This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)* \ No newline at end of file +\* *This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)* diff --git a/WORKSPACE b/WORKSPACE index 21f64e10e45..0ddcd7fa87b 100644 --- a/WORKSPACE +++ b/WORKSPACE @@ -48,20 +48,6 @@ tf_workspace() load("@bazel_tools//tools/build_defs/repo:git.bzl", "new_git_repository") -go_repository( - name = "google_ml_metadata", - commit = "0fb82dc56ff7", - importpath = "github.com/google/ml-metadata", -) - -new_git_repository( - name = "libmysqlclient", - build_file = "@google_ml_metadata//ml_metadata:libmysqlclient.BUILD", - remote = "https://github.com/MariaDB/mariadb-connector-c.git", - tag = "v3.0.8-release", - workspace_file = "@google_ml_metadata//ml_metadata:libmysqlclient.WORKSPACE", -) - go_repository( name = "io_k8s_client_go", build_file_proto_mode = "disable_global", @@ -152,8 +138,8 @@ http_archive( go_repository( name = "co_honnef_go_tools", - commit = "88497007e858", importpath = "honnef.co/go/tools", + tag = "v0.0.1-2019.2.2", ) go_repository( @@ -369,19 +355,19 @@ go_repository( go_repository( name = "com_github_golang_mock", importpath = "github.com/golang/mock", - tag = "v1.1.1", + tag = "v1.3.1", ) go_repository( name = "com_github_google_btree", - commit = "e89373fe6b4a", importpath = "github.com/google/btree", + tag = "v1.0.0", ) go_repository( name = "com_github_google_go_cmp", importpath = "github.com/google/go-cmp", - tag = "v0.2.0", + tag = "v0.3.1", ) go_repository( @@ -404,8 +390,8 @@ go_repository( go_repository( name = "com_github_hashicorp_golang_lru", - commit = "0fb14efe8c47", importpath = "github.com/hashicorp/golang-lru", + tag = "v0.5.3", ) go_repository( @@ -705,7 +691,7 @@ go_repository( go_repository( name = "com_google_cloud_go", importpath = "cloud.google.com/go", - tag = "v0.26.0", + tag = "v0.44.3", ) go_repository( @@ -716,7 +702,7 @@ go_repository( go_repository( name = "in_gopkg_check_v1", - commit = "20d25e280405", + commit = "788fd7840127", importpath = "gopkg.in/check.v1", ) @@ -765,12 +751,12 @@ go_repository( go_repository( name = "org_golang_google_appengine", importpath = "google.golang.org/appengine", - tag = "v1.1.0", + tag = "v1.6.1", ) go_repository( name = "org_golang_google_genproto", - commit = "ae2f86662275", + commit = "fa694d86fc64", importpath = "google.golang.org/genproto", ) @@ -778,61 +764,61 @@ go_repository( name = "org_golang_google_grpc", build_file_proto_mode = "disable_global", importpath = "google.golang.org/grpc", - tag = "v1.16.0", + tag = "v1.23.0", ) go_repository( name = "org_golang_x_crypto", - commit = "505ab145d0a9", + commit = "4def268fd1a4", importpath = "golang.org/x/crypto", ) go_repository( name = "org_golang_x_lint", - commit = "06c8688daad7", + commit = "959b441ac422", importpath = "golang.org/x/lint", ) go_repository( name = "org_golang_x_net", build_file_proto_mode = "disable_global", - commit = "351d144fa1fc", + commit = "74dc4d7220e7", importpath = "golang.org/x/net", ) go_repository( name = "org_golang_x_oauth2", - commit = "d2e6202438be", + commit = "0f29369cfe45", importpath = "golang.org/x/oauth2", ) go_repository( name = "org_golang_x_sync", - commit = "42b317875d0f", + commit = "112230192c58", importpath = "golang.org/x/sync", ) go_repository( name = "org_golang_x_sys", - commit = "a5c9d58dba9a", + commit = "fde4db37ae7a", importpath = "golang.org/x/sys", ) go_repository( name = "org_golang_x_text", importpath = "golang.org/x/text", - tag = "v0.3.0", + tag = "v0.3.2", ) go_repository( name = "org_golang_x_time", - commit = "fbb02b2291d2", + commit = "9d24e82272b4", importpath = "golang.org/x/time", ) go_repository( name = "org_golang_x_tools", - commit = "6cd1fcedba52", + commit = "922a4ee32d1a", importpath = "golang.org/x/tools", ) @@ -857,7 +843,7 @@ go_repository( go_repository( name = "com_github_golang_protobuf", importpath = "github.com/golang/protobuf", - tag = "v1.2.0", + tag = "v1.3.2", ) go_repository( @@ -916,7 +902,7 @@ go_repository( go_repository( name = "com_github_google_pprof", - commit = "3ea8567a2e57", + commit = "34ac40c74b70", importpath = "github.com/google/pprof", ) @@ -959,5 +945,125 @@ go_repository( go_repository( name = "com_github_gorilla_websocket", importpath = "github.com/gorilla/websocket", - tag = "v1.2.0", + tag = "v1.4.0", +) + +go_repository( + name = "com_github_burntsushi_xgb", + commit = "27f122750802", + importpath = "github.com/BurntSushi/xgb", +) + +go_repository( + name = "com_github_creack_pty", + importpath = "github.com/creack/pty", + tag = "v1.1.7", +) + +go_repository( + name = "com_github_google_martian", + importpath = "github.com/google/martian", + tag = "v2.1.0", +) + +go_repository( + name = "com_github_google_renameio", + importpath = "github.com/google/renameio", + tag = "v0.1.0", +) + +go_repository( + name = "com_github_googleapis_gax_go_v2", + importpath = "github.com/googleapis/gax-go/v2", + tag = "v2.0.5", +) + +go_repository( + name = "com_github_jstemmer_go_junit_report", + commit = "af01ea7f8024", + importpath = "github.com/jstemmer/go-junit-report", +) + +go_repository( + name = "com_github_kr_pretty", + importpath = "github.com/kr/pretty", + tag = "v0.1.0", +) + +go_repository( + name = "com_github_kr_pty", + importpath = "github.com/kr/pty", + tag = "v1.1.8", +) + +go_repository( + name = "com_github_kr_text", + importpath = "github.com/kr/text", + tag = "v0.1.0", +) + +go_repository( + name = "com_github_rogpeppe_go_internal", + importpath = "github.com/rogpeppe/go-internal", + tag = "v1.3.0", +) + +go_repository( + name = "com_google_cloud_go_datastore", + importpath = "cloud.google.com/go/datastore", + tag = "v1.0.0", +) + +go_repository( + name = "in_gopkg_errgo_v2", + importpath = "gopkg.in/errgo.v2", + tag = "v2.1.0", +) + +go_repository( + name = "io_opencensus_go", + importpath = "go.opencensus.io", + tag = "v0.22.0", +) + +go_repository( + name = "io_rsc_binaryregexp", + importpath = "rsc.io/binaryregexp", + tag = "v0.2.0", +) + +go_repository( + name = "org_golang_google_api", + importpath = "google.golang.org/api", + tag = "v0.8.0", +) + +go_repository( + name = "org_golang_x_exp", + commit = "ec7cb31e5a56", + importpath = "golang.org/x/exp", +) + +go_repository( + name = "org_golang_x_image", + commit = "cff245a6509b", + importpath = "golang.org/x/image", +) + +go_repository( + name = "org_golang_x_mobile", + commit = "e8b3e6111d02", + importpath = "golang.org/x/mobile", +) + +go_repository( + name = "org_golang_x_mod", + importpath = "golang.org/x/mod", + tag = "v0.1.0", +) + +go_repository( + name = "org_golang_x_xerrors", + commit = "a985d3407aa7", + importpath = "golang.org/x/xerrors", ) diff --git a/backend/src/apiserver/BUILD.bazel b/backend/src/apiserver/BUILD.bazel index 93d54a3b450..3ac3cfab7e4 100644 --- a/backend/src/apiserver/BUILD.bazel +++ b/backend/src/apiserver/BUILD.bazel @@ -13,7 +13,6 @@ go_library( deps = [ "//backend/api:go_default_library", "//backend/src/apiserver/client:go_default_library", - "//backend/src/apiserver/metadata:go_default_library", "//backend/src/apiserver/model:go_default_library", "//backend/src/apiserver/resource:go_default_library", "//backend/src/apiserver/server:go_default_library", @@ -24,15 +23,11 @@ go_library( "@com_github_cenkalti_backoff//:go_default_library", "@com_github_fsnotify_fsnotify//:go_default_library", "@com_github_golang_glog//:go_default_library", - "@com_github_golang_protobuf//proto:go_default_library", "@com_github_grpc_ecosystem_grpc_gateway//runtime:go_default_library", "@com_github_jinzhu_gorm//:go_default_library", "@com_github_jinzhu_gorm//dialects/sqlite:go_default_library", "@com_github_minio_minio_go//:go_default_library", "@com_github_spf13_viper//:go_default_library", - "@google_ml_metadata//ml_metadata/metadata_store:metadata_store_go", # keep - "@google_ml_metadata//ml_metadata/proto:metadata_store_go_proto", # keep - "@google_ml_metadata//ml_metadata/proto:metadata_store_service_go_proto", # keep "@io_k8s_client_go//kubernetes/typed/core/v1:go_default_library", "@org_golang_google_grpc//:go_default_library", "@org_golang_google_grpc//reflection:go_default_library", diff --git a/backend/src/apiserver/client_manager.go b/backend/src/apiserver/client_manager.go index 7004857380c..5c241a90691 100644 --- a/backend/src/apiserver/client_manager.go +++ b/backend/src/apiserver/client_manager.go @@ -19,38 +19,36 @@ import ( "fmt" v1 "k8s.io/client-go/kubernetes/typed/core/v1" "os" - "strconv" "time" workflowclient "github.com/argoproj/argo/pkg/client/clientset/versioned/typed/workflow/v1alpha1" "github.com/cenkalti/backoff" "github.com/golang/glog" - "github.com/golang/protobuf/proto" + "github.com/jinzhu/gorm" _ "github.com/jinzhu/gorm/dialects/sqlite" "github.com/kubeflow/pipelines/backend/src/apiserver/client" - "github.com/kubeflow/pipelines/backend/src/apiserver/metadata" "github.com/kubeflow/pipelines/backend/src/apiserver/model" "github.com/kubeflow/pipelines/backend/src/apiserver/storage" "github.com/kubeflow/pipelines/backend/src/common/util" scheduledworkflowclient "github.com/kubeflow/pipelines/backend/src/crd/pkg/client/clientset/versioned/typed/scheduledworkflow/v1beta1" "github.com/minio/minio-go" - - "ml_metadata/metadata_store/mlmetadata" - mlpb "ml_metadata/proto/metadata_store_go_proto" ) const ( - minioServiceHost = "MINIO_SERVICE_SERVICE_HOST" - minioServicePort = "MINIO_SERVICE_SERVICE_PORT" - mysqlServiceHost = "MYSQL_SERVICE_HOST" - mysqlUser = "DBConfig.User" - mysqlPassword = "DBConfig.Password" - mysqlServicePort = "MYSQL_SERVICE_PORT" + minioServiceHost = "MINIO_SERVICE_SERVICE_HOST" + minioServicePort = "MINIO_SERVICE_SERVICE_PORT" + mysqlServiceHost = "MYSQL_SERVICE_HOST" + mysqlServicePort = "MYSQL_SERVICE_PORT" + mysqlUser = "DBConfig.User" + mysqlPassword = "DBConfig.Password" + mysqlDBName = "DBConfig.DBName" podNamespace = "POD_NAMESPACE" - dbName = "mlpipeline" initConnectionTimeout = "InitConnectionTimeout" + + visualizationServiceHost = "ML_PIPELINE_VISUALIZATIONSERVER_SERVICE_HOST" + visualizationServicePort = "ML_PIPELINE_VISUALIZATIONSERVER_SERVICE_PORT" ) // Container for all service clients @@ -69,8 +67,6 @@ type ClientManager struct { podClient v1.PodInterface time util.TimeInterface uuid util.UUIDGeneratorInterface - - MetadataStore *mlmetadata.Store } func (c *ClientManager) ExperimentStore() storage.ExperimentStoreInterface { @@ -154,8 +150,7 @@ func (c *ClientManager) init() { c.podClient = client.CreatePodClientOrFatal( getStringConfig(podNamespace), getDurationConfig(initConnectionTimeout)) - metadataStore := initMetadataStore() - runStore := storage.NewRunStore(db, c.time, metadataStore) + runStore := storage.NewRunStore(db, c.time) c.runStore = runStore glog.Infof("Client manager initialized successfully") @@ -165,31 +160,6 @@ func (c *ClientManager) Close() { c.db.Close() } -func initMetadataStore() *metadata.Store { - port, err := strconv.Atoi(getStringConfig(mysqlServicePort)) - if err != nil { - glog.Fatalf("Failed to parse valid MySQL service port from %q: %v", getStringConfig(mysqlServicePort), err) - } - - cfg := &mlpb.ConnectionConfig{ - Config: &mlpb.ConnectionConfig_Mysql{ - &mlpb.MySQLDatabaseConfig{ - Host: proto.String(getStringConfig(mysqlServiceHost)), - Port: proto.Uint32(uint32(port)), - Database: proto.String("mlmetadata"), - User: proto.String(getStringConfigWithDefault(mysqlUser, "root")), - Password: proto.String(getStringConfigWithDefault(mysqlPassword, "")), - }, - }, - } - - mlmdStore, err := mlmetadata.NewStore(cfg) - if err != nil { - glog.Fatalf("Failed to create ML Metadata store: %v", err) - } - return metadata.NewStore(mlmdStore) -} - func initDBClient(initConnectionTimeout time.Duration) *storage.DB { driverName := getStringConfig("DBConfig.DriverName") var arg string @@ -220,6 +190,12 @@ func initDBClient(initConnectionTimeout time.Duration) *storage.DB { if response.Error != nil { glog.Fatalf("Failed to initialize the databases.") } + + response = db.Model(&model.ResourceReference{}).ModifyColumn("Payload", "longtext") + if response.Error != nil { + glog.Fatalf("Failed to update the resource reference payload type. Error: %s", response.Error) + } + response = db.Model(&model.RunMetric{}). AddForeignKey("RunUUID", "run_details(UUID)", "CASCADE" /* onDelete */, "CASCADE" /* update */) if response.Error != nil { @@ -255,6 +231,7 @@ func initMysql(driverName string, initConnectionTimeout time.Duration) string { util.TerminateIfError(err) // Create database if not exist + dbName := getStringConfig(mysqlDBName) operation = func() error { _, err = db.Exec(fmt.Sprintf("CREATE DATABASE IF NOT EXISTS %s", dbName)) if err != nil { @@ -280,7 +257,7 @@ func initMinioClient(initConnectionTimeout time.Duration) storage.ObjectStoreInt accessKey := getStringConfig("ObjectStoreConfig.AccessKey") secretKey := getStringConfig("ObjectStoreConfig.SecretAccessKey") bucketName := getStringConfig("ObjectStoreConfig.BucketName") - disableMultipart := getBoolConfigWithDefault("ObjectStoreConfig.Multipart.Disable", false) + disableMultipart := getBoolConfigWithDefault("ObjectStoreConfig.Multipart.Disable", true) minioClient := client.CreateMinioClientOrFatal(minioServiceHost, minioServicePort, accessKey, secretKey, initConnectionTimeout) diff --git a/backend/src/apiserver/config/config.json b/backend/src/apiserver/config/config.json index c8f9c530506..00ee8eb121c 100644 --- a/backend/src/apiserver/config/config.json +++ b/backend/src/apiserver/config/config.json @@ -1,7 +1,8 @@ { "DBConfig": { "DriverName": "mysql", - "DataSourceName": "" + "DataSourceName": "", + "DBName": "mlpipeline" }, "ObjectStoreConfig":{ "AccessKey": "minio", diff --git a/backend/src/apiserver/main.go b/backend/src/apiserver/main.go index 7bdd6243e57..e7bac0ed9e3 100644 --- a/backend/src/apiserver/main.go +++ b/backend/src/apiserver/main.go @@ -34,10 +34,6 @@ import ( "github.com/kubeflow/pipelines/backend/src/apiserver/server" "google.golang.org/grpc" "google.golang.org/grpc/reflection" - - _ "ml_metadata/metadata_store/mlmetadata" - _ "ml_metadata/proto/metadata_store_go_proto" - _ "ml_metadata/proto/metadata_store_service_go_proto" ) var ( @@ -83,7 +79,14 @@ func startRpcServer(resourceManager *resource.ResourceManager) { api.RegisterRunServiceServer(s, server.NewRunServer(resourceManager)) api.RegisterJobServiceServer(s, server.NewJobServer(resourceManager)) api.RegisterReportServiceServer(s, server.NewReportServer(resourceManager)) - api.RegisterVisualizationServiceServer(s, server.NewVisualizationServer(resourceManager)) + api.RegisterVisualizationServiceServer( + s, + server.NewVisualizationServer( + resourceManager, + getStringConfig(visualizationServiceHost), + getStringConfig(visualizationServicePort), + getDurationConfig(initConnectionTimeout), + )) // Register reflection service on gRPC server. reflection.Register(s) diff --git a/backend/src/apiserver/metadata/BUILD.bazel b/backend/src/apiserver/metadata/BUILD.bazel deleted file mode 100644 index fa019b6763c..00000000000 --- a/backend/src/apiserver/metadata/BUILD.bazel +++ /dev/null @@ -1,29 +0,0 @@ -load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test") - -go_library( - name = "go_default_library", - srcs = ["metadata_store.go"], - importpath = "github.com/kubeflow/pipelines/backend/src/apiserver/metadata", - visibility = ["//visibility:public"], - deps = [ - "//backend/src/common/util:go_default_library", - "@com_github_argoproj_argo//pkg/apis/workflow/v1alpha1:go_default_library", - "@com_github_golang_protobuf//jsonpb:go_default_library_gen", - "@com_github_golang_protobuf//proto:go_default_library", - "@google_ml_metadata//ml_metadata/metadata_store:metadata_store_go", - "@google_ml_metadata//ml_metadata/proto:metadata_store_go_proto", - ], -) - -go_test( - name = "go_default_test", - srcs = ["metadata_store_test.go"], - embed = [":go_default_library"], - deps = [ - "@com_github_argoproj_argo//pkg/apis/workflow/v1alpha1:go_default_library", - "@com_github_golang_protobuf//proto:go_default_library", - "@com_github_google_go_cmp//cmp:go_default_library", - "@google_ml_metadata//ml_metadata/metadata_store:metadata_store_go", - "@google_ml_metadata//ml_metadata/proto:metadata_store_go_proto", - ], -) diff --git a/backend/src/apiserver/metadata/metadata_store.go b/backend/src/apiserver/metadata/metadata_store.go deleted file mode 100644 index 086bfa4634f..00000000000 --- a/backend/src/apiserver/metadata/metadata_store.go +++ /dev/null @@ -1,135 +0,0 @@ -package metadata - -import ( - "encoding/json" - "ml_metadata/metadata_store/mlmetadata" - mlpb "ml_metadata/proto/metadata_store_go_proto" - "strings" - - argoWorkflow "github.com/argoproj/argo/pkg/apis/workflow/v1alpha1" - "github.com/golang/protobuf/jsonpb" - "github.com/golang/protobuf/proto" - "github.com/kubeflow/pipelines/backend/src/common/util" -) - -// Store encapsulates a ML Metadata store. -type Store struct { - mlmdStore *mlmetadata.Store -} - -// NewStore creates a new Store, using mlmdStore as the backing ML Metadata -// store. -func NewStore(mlmdStore *mlmetadata.Store) *Store { - return &Store{mlmdStore: mlmdStore} -} - -// RecordOutputArtifacts records metadata on artifacts as parsed from the Argo -// output parameters in currentManifest. storedManifest represents the currently -// stored manifest for the run with id runID, and is used to ensure metadata is -// recorded at most once per artifact. -func (s *Store) RecordOutputArtifacts(runID, storedManifest, currentManifest string) error { - storedWorkflow := &argoWorkflow.Workflow{} - if err := json.Unmarshal([]byte(storedManifest), storedWorkflow); err != nil { - return util.NewInternalServerError(err, "unmarshaling workflow failed") - } - - currentWorkflow := &argoWorkflow.Workflow{} - if err := json.Unmarshal([]byte(currentManifest), currentWorkflow); err != nil { - return util.NewInternalServerError(err, "unmarshaling workflow failed") - } - - completed := make(map[string]bool) - for _, n := range storedWorkflow.Status.Nodes { - if n.Completed() { - completed[n.ID] = true - } - } - - for _, n := range currentWorkflow.Status.Nodes { - if n.Completed() && !completed[n.ID] { - // Newly completed node. Record output ml-metadata artifacts. - if n.Outputs != nil { - for _, output := range n.Outputs.Parameters { - if output.ValueFrom == nil || output.Value == nil || !strings.HasPrefix(output.ValueFrom.Path, "/output/ml_metadata/") { - continue - } - - artifacts, err := parseTFXMetadata(*output.Value) - if err != nil { - return util.NewInvalidInputError("metadata parsing failure: %v", err) - } - - if err := s.storeArtifacts(artifacts); err != nil { - return util.NewInvalidInputError("artifact storing failure: %v", err) - } - } - } - } - } - - return nil -} - -func (s *Store) storeArtifacts(artifacts artifactStructs) error { - for _, a := range artifacts { - id, err := s.mlmdStore.PutArtifactType( - a.ArtifactType, &mlmetadata.PutTypeOptions{AllFieldsMustMatch: true}) - if err != nil { - return util.NewInternalServerError(err, "failed to register artifact type") - } - a.Artifact.TypeId = proto.Int64(int64(id)) - _, err = s.mlmdStore.PutArtifacts([]*mlpb.Artifact{a.Artifact}) - if err != nil { - return util.NewInternalServerError(err, "failed to record artifact") - } - } - return nil -} - -type artifactStruct struct { - ArtifactType *mlpb.ArtifactType `json:"artifact_type"` - Artifact *mlpb.Artifact `json:"artifact"` -} - -func (a *artifactStruct) UnmarshalJSON(b []byte) error { - errorF := func(err error) error { - return util.NewInvalidInputError("JSON Unmarshal failure: %v", err) - } - - jsonMap := make(map[string]*json.RawMessage) - if err := json.Unmarshal(b, &jsonMap); err != nil { - return errorF(err) - } - - if _, ok := jsonMap["artifact_type"]; !ok { - return util.NewInvalidInputError("JSON Unmarshal failure: missing 'artifact_type' field") - } - - if _, ok := jsonMap["artifact"]; !ok { - return util.NewInvalidInputError("JSON Unmarshal failure: missing 'artifact_type' field") - } - - a.ArtifactType = &mlpb.ArtifactType{} - a.Artifact = &mlpb.Artifact{} - - if err := jsonpb.UnmarshalString(string(*jsonMap["artifact_type"]), a.ArtifactType); err != nil { - return errorF(err) - } - - if err := jsonpb.UnmarshalString(string(*jsonMap["artifact"]), a.Artifact); err != nil { - return errorF(err) - } - - return nil -} - -type artifactStructs []*artifactStruct - -func parseTFXMetadata(value string) (artifactStructs, error) { - var tfxArtifacts artifactStructs - - if err := json.Unmarshal([]byte(value), &tfxArtifacts); err != nil { - return nil, util.NewInternalServerError(err, "parse TFX metadata failure") - } - return tfxArtifacts, nil -} diff --git a/backend/src/apiserver/metadata/metadata_store_test.go b/backend/src/apiserver/metadata/metadata_store_test.go deleted file mode 100644 index 5ca9691be1a..00000000000 --- a/backend/src/apiserver/metadata/metadata_store_test.go +++ /dev/null @@ -1,357 +0,0 @@ -package metadata - -import ( - "encoding/json" - "fmt" - "ml_metadata/metadata_store/mlmetadata" - mlpb "ml_metadata/proto/metadata_store_go_proto" - "testing" - - argo "github.com/argoproj/argo/pkg/apis/workflow/v1alpha1" - "github.com/golang/protobuf/proto" - "github.com/google/go-cmp/cmp" -) - -func (as artifactStructs) String() string { - var s string - for _, a := range as { - s += fmt.Sprintf("%+v\n", a) - } - return s -} - -func TestParseValidTFXMetadata(t *testing.T) { - tests := []struct { - input string - want artifactStructs - }{ - { - `[{ - "artifact_type": { - "name": "Artifact", - "properties": {"state": "STRING", "span": "INT" } }, - "artifact": { - "uri": "/location", - "properties": { - "state": {"stringValue": "complete"}, - "span": {"intValue": 10} } - } - }]`, - []*artifactStruct{ - &artifactStruct{ - ArtifactType: &mlpb.ArtifactType{ - Name: proto.String("Artifact"), - Properties: map[string]mlpb.PropertyType{ - "state": mlpb.PropertyType_STRING, - "span": mlpb.PropertyType_INT, - }, - }, - Artifact: &mlpb.Artifact{ - Uri: proto.String("/location"), - Properties: map[string]*mlpb.Value{ - "state": &mlpb.Value{Value: &mlpb.Value_StringValue{"complete"}}, - "span": &mlpb.Value{Value: &mlpb.Value_IntValue{10}}, - }, - }, - }, - }, - }, - { - `[{ - "artifact_type": { - "name": "Artifact 1", - "properties": {"state": "STRING", "span": "INT" } }, - "artifact": { - "uri": "/location 1", - "properties": { - "state": {"stringValue": "complete"}, - "span": {"intValue": 10} } - } - }, - { - "artifact_type": { - "name": "Artifact 2", - "properties": {"state": "STRING", "span": "INT" } }, - "artifact": { - "uri": "/location 2", - "properties": { - "state": {"stringValue": "complete"}, - "span": {"intValue": 10} } - } - }]`, - []*artifactStruct{ - &artifactStruct{ - ArtifactType: &mlpb.ArtifactType{ - Name: proto.String("Artifact 1"), - Properties: map[string]mlpb.PropertyType{ - "state": mlpb.PropertyType_STRING, - "span": mlpb.PropertyType_INT, - }, - }, - Artifact: &mlpb.Artifact{ - Uri: proto.String("/location 1"), - Properties: map[string]*mlpb.Value{ - "state": &mlpb.Value{Value: &mlpb.Value_StringValue{"complete"}}, - "span": &mlpb.Value{Value: &mlpb.Value_IntValue{10}}, - }, - }, - }, - &artifactStruct{ - ArtifactType: &mlpb.ArtifactType{ - Name: proto.String("Artifact 2"), - Properties: map[string]mlpb.PropertyType{ - "state": mlpb.PropertyType_STRING, - "span": mlpb.PropertyType_INT, - }, - }, - Artifact: &mlpb.Artifact{ - Uri: proto.String("/location 2"), - Properties: map[string]*mlpb.Value{ - "state": &mlpb.Value{Value: &mlpb.Value_StringValue{"complete"}}, - "span": &mlpb.Value{Value: &mlpb.Value_IntValue{10}}, - }, - }, - }, - }, - }, - } - - for _, test := range tests { - got, err := parseTFXMetadata(test.input) - if err != nil || !cmp.Equal(got, test.want) { - t.Errorf("parseTFXMetadata(%q)\nGot:\n<%+v, %+v>\nWant:\n%+v, nil error\nDiff:\n%s", test.input, got, err, test.want, cmp.Diff(test.want, got)) - } - } -} - -func TestParseInvalidTFXMetadata(t *testing.T) { - tests := []struct { - desc string - input string - }{ - { - "no artifact type", - `[{ - "artifact": { - "uri": "/location", - "properties": { - "state": {"stringValue": "complete"}, - "span": {"intValue": 10} } - } - }]`, - }, - { - "no artifact", - `[{ - "artifact_type": { - "name": "Artifact", - "properties": {"state": "STRING", "span": "INT" } }, - }]`, - }, - { - "empty string", - "", - }, - } - - for _, test := range tests { - _, err := parseTFXMetadata(test.input) - if err == nil { - t.Errorf("Test: %q", test.desc) - t.Errorf("parseTFXMetadata(%q)\nGot non-nil error. Want error.", test.input) - } - } -} - -func fakeMLMDStore(t *testing.T) *mlmetadata.Store { - cfg := &mlpb.ConnectionConfig{ - Config: &mlpb.ConnectionConfig_FakeDatabase{&mlpb.FakeDatabaseConfig{}}, - } - - mlmdStore, err := mlmetadata.NewStore(cfg) - if err != nil { - t.Fatalf("Failed to create ML Metadata Store: %v", err) - } - return mlmdStore -} - -func TestRecordOutputArtifacts(t *testing.T) { - - tests := []struct { - desc string - stored *argo.Workflow - current *argo.Workflow - wantTypes []*mlpb.ArtifactType - wantArtifacts []*mlpb.Artifact - }{ - { - desc: "nothing is recorded when node is still running", - stored: &argo.Workflow{Status: argo.WorkflowStatus{ - Nodes: map[string]argo.NodeStatus{ - "step_1": argo.NodeStatus{ - ID: "step_1_node_id", - Phase: argo.NodeRunning, - Outputs: &argo.Outputs{ - Parameters: []argo.Parameter{ - argo.Parameter{ - ValueFrom: &argo.ValueFrom{Path: "/output/ml_metadata/output1"}, - Value: proto.String(`[{ - "artifact_type": { "name": "step_1_artifact_type", - "properties": {"state": "STRING" } }, - "artifact": { "uri": "/location 2", - "properties": { - "state": {"stringValue": "complete"} } } - }]`), - }}}}}}}, - current: &argo.Workflow{Status: argo.WorkflowStatus{ - Nodes: map[string]argo.NodeStatus{ - "step_1": argo.NodeStatus{ - ID: "step_1_node_id", - Phase: argo.NodeRunning, - Outputs: &argo.Outputs{ - Parameters: []argo.Parameter{ - argo.Parameter{ - ValueFrom: &argo.ValueFrom{Path: "/output/ml_metadata/output1"}, - Value: proto.String(`[{ - "artifact_type": { "name": "step_1_artifact_type", - "properties": {"state": "STRING" } }, - "artifact": { "uri": "/step_1_location", - "properties": { - "state": {"stringValue": "complete"} } } - }]`), - }}}}}}}, - wantTypes: nil, - wantArtifacts: nil, - }, - { - desc: "artifacts are recorded when node transitions to Complete", - stored: &argo.Workflow{Status: argo.WorkflowStatus{ - Nodes: map[string]argo.NodeStatus{ - "step_1": argo.NodeStatus{ - ID: "step_1_node_id", - Phase: argo.NodeRunning, - Outputs: &argo.Outputs{ - Parameters: []argo.Parameter{ - argo.Parameter{ - ValueFrom: &argo.ValueFrom{Path: "/output/ml_metadata/output1"}, - Value: proto.String(`[{ - "artifact_type": { "name": "step_1_artifact_type", - "properties": {"state": "STRING" } }, - "artifact": { "uri": "/location 2", - "properties": { - "state": {"stringValue": "complete"} } } - }]`), - }}}}}}}, - current: &argo.Workflow{Status: argo.WorkflowStatus{ - Nodes: map[string]argo.NodeStatus{ - "step_1": argo.NodeStatus{ - ID: "step_1_node_id", - Phase: argo.NodeSucceeded, - Outputs: &argo.Outputs{ - Parameters: []argo.Parameter{ - argo.Parameter{ - ValueFrom: &argo.ValueFrom{Path: "/output/ml_metadata/output1"}, - Value: proto.String(`[{ - "artifact_type": { "name": "step_1_artifact_type", - "properties": {"state": "STRING" } }, - "artifact": { "uri": "/step_1_location", - "properties": { - "state": {"stringValue": "complete"} } } - }]`), - }}}}}}}, - wantTypes: []*mlpb.ArtifactType{ - &mlpb.ArtifactType{ - Id: proto.Int64(1), - Name: proto.String("step_1_artifact_type"), - Properties: map[string]mlpb.PropertyType{"state": mlpb.PropertyType_STRING}, - }}, - wantArtifacts: []*mlpb.Artifact{ - &mlpb.Artifact{ - Id: proto.Int64(1), - TypeId: proto.Int64(1), - Uri: proto.String("/step_1_location"), - Properties: map[string]*mlpb.Value{ - "state": &mlpb.Value{Value: &mlpb.Value_StringValue{"complete"}}}, - }}, - }, - { - desc: "Records artifacts only from Node with output parameter specified in ValueFrom Path", - stored: &argo.Workflow{Status: argo.WorkflowStatus{ - Nodes: map[string]argo.NodeStatus{ - "step_1": argo.NodeStatus{ - ID: "step_1_node_id", - Phase: argo.NodeRunning, - Outputs: &argo.Outputs{ - Parameters: []argo.Parameter{ - argo.Parameter{ - Value: proto.String(`[{ - "artifact_type": { "name": "step_1_artifact_type", - "properties": {"state": "STRING" } }, - "artifact": { "uri": "/location 2", - "properties": { - "state": {"stringValue": "complete"} } } - }]`), - }}}}}}}, - current: &argo.Workflow{Status: argo.WorkflowStatus{ - Nodes: map[string]argo.NodeStatus{ - "step_1": argo.NodeStatus{ - ID: "step_1_node_id", - Phase: argo.NodeSucceeded, - Outputs: &argo.Outputs{ - Parameters: []argo.Parameter{ - argo.Parameter{ - Value: proto.String(`[{ - "artifact_type": { "name": "step_1_artifact_type", - "properties": {"state": "STRING" } }, - "artifact": { "uri": "/step_1_location", - "properties": { - "state": {"stringValue": "complete"} } } - }]`), - }}}}}}}, - wantTypes: nil, - wantArtifacts: nil, - }, - } - - for _, test := range tests { - mlmdStore := fakeMLMDStore(t) - store := Store{mlmdStore: mlmdStore} - - current, err := json.Marshal(test.current) - if err != nil { - t.Errorf("Test: %q", test.desc) - t.Errorf("json.Marshal(%v) = %v", test.current, err) - continue - } - - stored, err := json.Marshal(test.stored) - if err != nil { - t.Errorf("Test: %q", test.desc) - t.Errorf("json.Marshal(%v) = %v", test.stored, err) - continue - } - - if err := store.RecordOutputArtifacts("", string(stored), string(current)); err != nil { - t.Errorf("Test: %q", test.desc) - t.Errorf("store.RecordOutputArtifacts(%q, %q) = %v\nWant non-nil error\n", current, stored, err) - continue - } - - gotTypes, err := mlmdStore.GetArtifactTypesByID([]mlmetadata.ArtifactTypeID{1}) - if !cmp.Equal(gotTypes, test.wantTypes) { - t.Errorf("Test: %q", test.desc) - t.Errorf("mlmdStore.GetArtifactTypes()\n") - t.Errorf("Got artifact types:\n%v\nWant\n%v\n", gotTypes, test.wantTypes) - t.Errorf("Got error:\n%v\nWant nil error\n", err) - } - - gotArtifacts, err := mlmdStore.GetArtifacts() - if !cmp.Equal(gotArtifacts, test.wantArtifacts) { - t.Errorf("Test: %q", test.desc) - t.Errorf("mlmdStore.GetArtifacts()\n") - t.Errorf("Got artifacts:\n%v\nWant\n%v\n", gotArtifacts, test.wantArtifacts) - t.Errorf("Got error:\n%v\nWant nil error\n", err) - } - } - -} diff --git a/backend/src/apiserver/resource/BUILD.bazel b/backend/src/apiserver/resource/BUILD.bazel index fb004e94e43..c477c49e6aa 100644 --- a/backend/src/apiserver/resource/BUILD.bazel +++ b/backend/src/apiserver/resource/BUILD.bazel @@ -17,7 +17,6 @@ go_library( "//backend/api:go_default_library", "//backend/src/apiserver/common:go_default_library", "//backend/src/apiserver/list:go_default_library", - "//backend/src/apiserver/metadata:go_default_library", "//backend/src/apiserver/model:go_default_library", "//backend/src/apiserver/storage:go_default_library", "//backend/src/common/util:go_default_library", @@ -29,8 +28,6 @@ go_library( "@com_github_cenkalti_backoff//:go_default_library", "@com_github_golang_glog//:go_default_library", "@com_github_pkg_errors//:go_default_library", - "@google_ml_metadata//ml_metadata/metadata_store:metadata_store_go", - "@google_ml_metadata//ml_metadata/proto:metadata_store_go_proto", "@io_k8s_api//core/v1:go_default_library", "@io_k8s_api//policy/v1beta1:go_default_library", "@io_k8s_apimachinery//pkg/api/errors:go_default_library", diff --git a/backend/src/apiserver/resource/client_manager_fake.go b/backend/src/apiserver/resource/client_manager_fake.go index eb42b5b20dc..a0e62ce9add 100644 --- a/backend/src/apiserver/resource/client_manager_fake.go +++ b/backend/src/apiserver/resource/client_manager_fake.go @@ -15,12 +15,8 @@ package resource import ( - "ml_metadata/metadata_store/mlmetadata" - mlpb "ml_metadata/proto/metadata_store_go_proto" - workflowclient "github.com/argoproj/argo/pkg/client/clientset/versioned/typed/workflow/v1alpha1" "github.com/golang/glog" - "github.com/kubeflow/pipelines/backend/src/apiserver/metadata" "github.com/kubeflow/pipelines/backend/src/apiserver/storage" "github.com/kubeflow/pipelines/backend/src/common/util" scheduledworkflowclient "github.com/kubeflow/pipelines/backend/src/crd/pkg/client/clientset/versioned/typed/scheduledworkflow/v1beta1" @@ -71,7 +67,7 @@ func NewFakeClientManager(time util.TimeInterface, uuid util.UUIDGeneratorInterf experimentStore: storage.NewExperimentStore(db, time, uuid), pipelineStore: storage.NewPipelineStore(db, time, uuid), jobStore: storage.NewJobStore(db, time), - runStore: storage.NewRunStore(db, time, initFakeMetadataStore()), + runStore: storage.NewRunStore(db, time), workflowClientFake: NewWorkflowClientFake(), resourceReferenceStore: storage.NewResourceReferenceStore(db), dBStatusStore: storage.NewDBStatusStore(db), @@ -84,18 +80,6 @@ func NewFakeClientManager(time util.TimeInterface, uuid util.UUIDGeneratorInterf }, nil } -func initFakeMetadataStore() *metadata.Store { - cfg := &mlpb.ConnectionConfig{ - Config: &mlpb.ConnectionConfig_FakeDatabase{&mlpb.FakeDatabaseConfig{}}, - } - - mlmdStore, err := mlmetadata.NewStore(cfg) - if err != nil { - glog.Fatalf("Failed to create ML Metadata store: %v", err) - } - return metadata.NewStore(mlmdStore) -} - func NewFakeClientManagerOrFatal(time util.TimeInterface) *FakeClientManager { uuid := util.NewFakeUUIDGeneratorOrFatal(DefaultFakeUUID, nil) fakeStore, err := NewFakeClientManager(time, uuid) diff --git a/backend/src/apiserver/resource/resource_manager.go b/backend/src/apiserver/resource/resource_manager.go index e1c4fe3fef1..e7f17542ded 100644 --- a/backend/src/apiserver/resource/resource_manager.go +++ b/backend/src/apiserver/resource/resource_manager.go @@ -369,23 +369,38 @@ func (r *ResourceManager) RetryRun(runId string) error { return util.NewInternalServerError(err, "Retry run failed. Failed to clean up the failed pods from previous run.") } - _, updateErr := r.workflowClient.Update(newWorkflow.Workflow) - if updateErr == nil { - return nil + // First try to update workflow + updateError := r.updateWorkflow(newWorkflow) + if updateError != nil { + // Remove resource version + newWorkflow.ResourceVersion = "" + newCreatedWorkflow, createError := r.workflowClient.Create(newWorkflow.Workflow) + if createError != nil { + return util.NewInternalServerError(createError, + "Retry run failed. Failed to create or update the run. Update Error: %s, Create Error: %s", + updateError.Error(), createError.Error()) + } + newWorkflow = util.NewWorkflow(newCreatedWorkflow) } - - // Remove resource version - newWorkflow.ResourceVersion = "" - _, createError := r.workflowClient.Create(newWorkflow.Workflow) - if createError != nil { - return util.NewInternalServerError(createError, - "Retry run failed. Failed to create or update the run. Update Error: %s, Create Error: %s", - updateErr.Error(), createError.Error()) + err = r.runStore.UpdateRun(runId, newWorkflow.Condition(), 0, newWorkflow.ToStringForStore()) + if err != nil { + return util.NewInternalServerError(err, "Failed to update the database entry.") } - return nil } +func (r *ResourceManager) updateWorkflow(newWorkflow *util.Workflow) error { + // If fail to get the workflow, return error. + latestWorkflow, err := r.workflowClient.Get(newWorkflow.Name, v1.GetOptions{}) + if err != nil { + return err + } + // Update the workflow's resource version to latest. + newWorkflow.ResourceVersion = latestWorkflow.ResourceVersion + _, err = r.workflowClient.Update(newWorkflow.Workflow) + return err +} + func (r *ResourceManager) GetJob(id string) (*model.Job, error) { return r.jobStore.GetJob(id) } diff --git a/backend/src/apiserver/resource/resource_manager_test.go b/backend/src/apiserver/resource/resource_manager_test.go index 99f182d1356..c693e6e316a 100644 --- a/backend/src/apiserver/resource/resource_manager_test.go +++ b/backend/src/apiserver/resource/resource_manager_test.go @@ -632,6 +632,10 @@ func TestRetryRun(t *testing.T) { err = manager.RetryRun(runDetail.UUID) assert.Nil(t, err) + + actualRunDetail, err = manager.GetRun(runDetail.UUID) + assert.Nil(t, err) + assert.Contains(t, actualRunDetail.WorkflowRuntimeManifest, "Running") } func TestRetryRun_RunNotExist(t *testing.T) { diff --git a/backend/src/apiserver/server/BUILD.bazel b/backend/src/apiserver/server/BUILD.bazel index 83c307b0b8a..3ffdcea7483 100644 --- a/backend/src/apiserver/server/BUILD.bazel +++ b/backend/src/apiserver/server/BUILD.bazel @@ -27,6 +27,7 @@ go_library( "//backend/src/common/util:go_default_library", "//backend/src/crd/pkg/apis/scheduledworkflow/v1beta1:go_default_library", "@com_github_argoproj_argo//pkg/apis/workflow/v1alpha1:go_default_library", + "@com_github_cenkalti_backoff//:go_default_library", "@com_github_golang_glog//:go_default_library", "@com_github_golang_protobuf//jsonpb:go_default_library_gen", "@com_github_robfig_cron//:go_default_library", diff --git a/backend/src/apiserver/server/visualization_server.go b/backend/src/apiserver/server/visualization_server.go index bcaf519f95a..9c4006dcbc3 100644 --- a/backend/src/apiserver/server/visualization_server.go +++ b/backend/src/apiserver/server/visualization_server.go @@ -4,6 +4,8 @@ import ( "context" "encoding/json" "fmt" + "github.com/cenkalti/backoff" + "github.com/golang/glog" "github.com/kubeflow/pipelines/backend/api/go_client" "github.com/kubeflow/pipelines/backend/src/apiserver/resource" "github.com/kubeflow/pipelines/backend/src/common/util" @@ -11,11 +13,13 @@ import ( "net/http" "net/url" "strings" + "time" ) type VisualizationServer struct { - resourceManager *resource.ResourceManager - serviceURL string + resourceManager *resource.ResourceManager + serviceURL string + isServiceAvailable bool } func (s *VisualizationServer) CreateVisualization(ctx context.Context, request *go_client.CreateVisualizationRequest) (*go_client.Visualization, error) { @@ -56,6 +60,12 @@ func (s *VisualizationServer) validateCreateVisualizationRequest(request *go_cli // service to generate HTML visualizations from a request. // It returns the generated HTML as a string and any error that is encountered. func (s *VisualizationServer) generateVisualizationFromRequest(request *go_client.CreateVisualizationRequest) ([]byte, error) { + if !s.isServiceAvailable { + return nil, util.NewInternalServerError( + fmt.Errorf("service not available"), + "Service not available", + ) + } visualizationType := strings.ToLower(go_client.Visualization_Type_name[int32(request.Visualization.Type)]) arguments := fmt.Sprintf("--type %s --source %s --arguments '%s'", visualizationType, request.Visualization.Source, request.Visualization.Arguments) resp, err := http.PostForm(s.serviceURL, url.Values{"arguments": {arguments}}) @@ -73,6 +83,33 @@ func (s *VisualizationServer) generateVisualizationFromRequest(request *go_clien return body, nil } -func NewVisualizationServer(resourceManager *resource.ResourceManager) *VisualizationServer { - return &VisualizationServer{resourceManager: resourceManager, serviceURL: "http://visualization-service.kubeflow"} +func isVisualizationServiceAlive(serviceURL string, initConnectionTimeout time.Duration) bool { + var operation = func() error { + _, err := http.Get(serviceURL) + if err != nil { + glog.Error("Unable to verify visualization service is alive!", err) + return err + } + return nil + } + b := backoff.NewExponentialBackOff() + b.MaxElapsedTime = initConnectionTimeout + err := backoff.Retry(operation, b) + return err == nil +} + +func NewVisualizationServer(resourceManager *resource.ResourceManager, serviceHost string, servicePort string, initConnectionTimeout time.Duration) *VisualizationServer { + serviceURL := fmt.Sprintf("http://%s:%s", serviceHost, servicePort) + isServiceAvailable := isVisualizationServiceAlive(serviceURL, initConnectionTimeout) + return &VisualizationServer{ + resourceManager: resourceManager, + serviceURL: serviceURL, + // TODO: isServiceAvailable is used to determine if the new visualization + // service is alive. If this is true, then the service is alive and + // requests can be made to it. Otherwise, if it is false, the service is + // not alive and requests should not be made. This prevents timeouts and + // counteracts current instabilities with the service. This should be + // removed after the visualization service is deemed stable. + isServiceAvailable: isServiceAvailable, + } } diff --git a/backend/src/apiserver/server/visualization_server_test.go b/backend/src/apiserver/server/visualization_server_test.go index 5edfd664841..82c39e27012 100644 --- a/backend/src/apiserver/server/visualization_server_test.go +++ b/backend/src/apiserver/server/visualization_server_test.go @@ -11,7 +11,10 @@ import ( func TestValidateCreateVisualizationRequest(t *testing.T) { clients, manager, _ := initWithExperiment(t) defer clients.Close() - server := NewVisualizationServer(manager) + server := &VisualizationServer{ + resourceManager: manager, + isServiceAvailable: false, + } visualization := &go_client.Visualization{ Type: go_client.Visualization_ROC_CURVE, Source: "gs://ml-pipeline/roc/data.csv", @@ -27,7 +30,10 @@ func TestValidateCreateVisualizationRequest(t *testing.T) { func TestValidateCreateVisualizationRequest_ArgumentsAreEmpty(t *testing.T) { clients, manager, _ := initWithExperiment(t) defer clients.Close() - server := NewVisualizationServer(manager) + server := &VisualizationServer{ + resourceManager: manager, + isServiceAvailable: false, + } visualization := &go_client.Visualization{ Type: go_client.Visualization_ROC_CURVE, Source: "gs://ml-pipeline/roc/data.csv", @@ -43,7 +49,10 @@ func TestValidateCreateVisualizationRequest_ArgumentsAreEmpty(t *testing.T) { func TestValidateCreateVisualizationRequest_SourceIsEmpty(t *testing.T) { clients, manager, _ := initWithExperiment(t) defer clients.Close() - server := NewVisualizationServer(manager) + server := &VisualizationServer{ + resourceManager: manager, + isServiceAvailable: false, + } visualization := &go_client.Visualization{ Type: go_client.Visualization_ROC_CURVE, Source: "", @@ -59,7 +68,10 @@ func TestValidateCreateVisualizationRequest_SourceIsEmpty(t *testing.T) { func TestValidateCreateVisualizationRequest_ArgumentsNotValidJSON(t *testing.T) { clients, manager, _ := initWithExperiment(t) defer clients.Close() - server := NewVisualizationServer(manager) + server := &VisualizationServer{ + resourceManager: manager, + isServiceAvailable: false, + } visualization := &go_client.Visualization{ Type: go_client.Visualization_ROC_CURVE, Source: "gs://ml-pipeline/roc/data.csv", @@ -80,7 +92,11 @@ func TestGenerateVisualization(t *testing.T) { rw.Write([]byte("roc_curve")) })) defer httpServer.Close() - server := &VisualizationServer{resourceManager: manager, serviceURL: httpServer.URL} + server := &VisualizationServer{ + resourceManager: manager, + serviceURL: httpServer.URL, + isServiceAvailable: true, + } visualization := &go_client.Visualization{ Type: go_client.Visualization_ROC_CURVE, Source: "gs://ml-pipeline/roc/data.csv", @@ -90,8 +106,34 @@ func TestGenerateVisualization(t *testing.T) { Visualization: visualization, } body, err := server.generateVisualizationFromRequest(request) - assert.Equal(t, []byte("roc_curve"), body) assert.Nil(t, err) + assert.Equal(t, []byte("roc_curve"), body) +} + +func TestGenerateVisualization_ServiceNotAvailableError(t *testing.T) { + clients, manager, _ := initWithExperiment(t) + defer clients.Close() + httpServer := httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) { + assert.Equal(t, "/", req.URL.String()) + rw.WriteHeader(500) + })) + defer httpServer.Close() + server := &VisualizationServer{ + resourceManager: manager, + serviceURL: httpServer.URL, + isServiceAvailable: false, + } + visualization := &go_client.Visualization{ + Type: go_client.Visualization_ROC_CURVE, + Source: "gs://ml-pipeline/roc/data.csv", + Arguments: "{}", + } + request := &go_client.CreateVisualizationRequest{ + Visualization: visualization, + } + body, err := server.generateVisualizationFromRequest(request) + assert.Nil(t, body) + assert.Equal(t, "InternalServerError: Service not available: service not available", err.Error()) } func TestGenerateVisualization_ServerError(t *testing.T) { @@ -102,7 +144,11 @@ func TestGenerateVisualization_ServerError(t *testing.T) { rw.WriteHeader(500) })) defer httpServer.Close() - server := &VisualizationServer{resourceManager: manager, serviceURL: httpServer.URL} + server := &VisualizationServer{ + resourceManager: manager, + serviceURL: httpServer.URL, + isServiceAvailable: true, + } visualization := &go_client.Visualization{ Type: go_client.Visualization_ROC_CURVE, Source: "gs://ml-pipeline/roc/data.csv", diff --git a/backend/src/apiserver/storage/BUILD.bazel b/backend/src/apiserver/storage/BUILD.bazel index adcf90ab634..26def425b38 100644 --- a/backend/src/apiserver/storage/BUILD.bazel +++ b/backend/src/apiserver/storage/BUILD.bazel @@ -25,7 +25,6 @@ go_library( "//backend/api:go_default_library", "//backend/src/apiserver/common:go_default_library", "//backend/src/apiserver/list:go_default_library", - "//backend/src/apiserver/metadata:go_default_library", "//backend/src/apiserver/model:go_default_library", "//backend/src/common/util:go_default_library", "@com_github_argoproj_argo//pkg/apis/workflow/v1alpha1:go_default_library", @@ -38,9 +37,6 @@ go_library( "@com_github_minio_minio_go//:go_default_library", "@com_github_pkg_errors//:go_default_library", "@com_github_vividcortex_mysqlerr//:go_default_library", - "@google_ml_metadata//ml_metadata/metadata_store:metadata_store_go", # keep - "@google_ml_metadata//ml_metadata/proto:metadata_store_go_proto", # keep - "@google_ml_metadata//ml_metadata/proto:metadata_store_service_go_proto", # keep "@io_k8s_apimachinery//pkg/util/json:go_default_library", ], ) diff --git a/backend/src/apiserver/storage/run_store.go b/backend/src/apiserver/storage/run_store.go index e18fb1e39e5..36dd3385241 100644 --- a/backend/src/apiserver/storage/run_store.go +++ b/backend/src/apiserver/storage/run_store.go @@ -26,7 +26,6 @@ import ( api "github.com/kubeflow/pipelines/backend/api/go_client" "github.com/kubeflow/pipelines/backend/src/apiserver/common" "github.com/kubeflow/pipelines/backend/src/apiserver/list" - "github.com/kubeflow/pipelines/backend/src/apiserver/metadata" "github.com/kubeflow/pipelines/backend/src/apiserver/model" "github.com/kubeflow/pipelines/backend/src/common/util" "k8s.io/apimachinery/pkg/util/json" @@ -71,7 +70,6 @@ type RunStore struct { db *DB resourceReferenceStore *ResourceReferenceStore time util.TimeInterface - metadataStore *metadata.Store } // Runs two SQL queries in a transaction to return a list of matching runs, as well as their @@ -374,29 +372,6 @@ func (s *RunStore) UpdateRun(runID string, condition string, finishedAtInSec int return util.NewInternalServerError(err, "transaction creation failed") } - // Lock the row for update, so we ensure no other update of the same run - // happens while we're parsing it for metadata. We rely on per-row updates - // being synchronous, so metadata can be recorded at most once. Right now, - // persistence agent will call UpdateRun all the time, even if there is nothing - // new in the status of an Argo manifest. This means we need to keep track - // manually here on what the previously updated state of the run is, to ensure - // we do not add duplicate metadata. Hence the locking below. - query := "SELECT WorkflowRuntimeManifest FROM run_details WHERE UUID = ?" - query = s.db.SelectForUpdate(query) - - row := tx.QueryRow(query, runID) - var storedManifest string - if err := row.Scan(&storedManifest); err != nil { - tx.Rollback() - return util.NewInvalidInputError("Failed to update run %s. Row not found. Error: %s", runID, err.Error()) - } - - if err := s.metadataStore.RecordOutputArtifacts(runID, storedManifest, workflowRuntimeManifest); err != nil { - // Metadata storage failed. Log the error here, but continue to allow the run - // to be updated as per usual. - glog.Errorf("Failed to record output artifacts: %+v", err) - } - sql, args, err := sq. Update("run_details"). SetMap(sq.Eq{ @@ -425,7 +400,11 @@ func (s *RunStore) UpdateRun(runID string, condition string, finishedAtInSec int if r > 1 { tx.Rollback() return util.NewInternalServerError(errors.New("Failed to update run"), "Failed to update run %s. More than 1 rows affected", runID) + } else if r == 0 { + tx.Rollback() + return util.NewInternalServerError(errors.New("Failed to update run"), "Failed to update run %s. Row not found", runID) } + if err := tx.Commit(); err != nil { return util.NewInternalServerError(err, "failed to commit transaction") } @@ -570,14 +549,12 @@ func (s *RunStore) toRunMetadatas(models []model.ListableDataModel) []model.Run return runMetadatas } -// NewRunStore creates a new RunStore. If metadataStore is non-nil, it will be -// used to record artifact metadata. -func NewRunStore(db *DB, time util.TimeInterface, metadataStore *metadata.Store) *RunStore { +// NewRunStore creates a new RunStore. +func NewRunStore(db *DB, time util.TimeInterface) *RunStore { return &RunStore{ db: db, resourceReferenceStore: NewResourceReferenceStore(db), time: time, - metadataStore: metadataStore, } } diff --git a/backend/src/apiserver/storage/run_store_test.go b/backend/src/apiserver/storage/run_store_test.go index ddb39e310d5..8ea50479dac 100644 --- a/backend/src/apiserver/storage/run_store_test.go +++ b/backend/src/apiserver/storage/run_store_test.go @@ -33,7 +33,7 @@ func initializeRunStore() (*DB, *RunStore) { expStore.CreateExperiment(&model.Experiment{Name: "exp1"}) expStore = NewExperimentStore(db, util.NewFakeTimeForEpoch(), util.NewFakeUUIDGeneratorOrFatal(defaultFakeExpIdTwo, nil)) expStore.CreateExperiment(&model.Experiment{Name: "exp2"}) - runStore := NewRunStore(db, util.NewFakeTimeForEpoch(), nil) + runStore := NewRunStore(db, util.NewFakeTimeForEpoch()) run1 := &model.RunDetail{ Run: model.Run{ diff --git a/backend/src/apiserver/visualization/exporter.py b/backend/src/apiserver/visualization/exporter.py index abb81921c8a..19f02b6efef 100644 --- a/backend/src/apiserver/visualization/exporter.py +++ b/backend/src/apiserver/visualization/exporter.py @@ -17,7 +17,6 @@ # See the License for the specific language governing permissions and # limitations under the License. -import argparse from enum import Enum import json from pathlib import Path @@ -86,31 +85,22 @@ def __init__( self.km.start_kernel() self.ep = ExecutePreprocessor( timeout=self.timeout, - kernel_name='python3' + kernel_name='python3', + allow_errors=True ) @staticmethod - def create_cell_from_args(args: argparse.Namespace) -> NotebookNode: - """Creates a NotebookNode object with provided arguments as variables. + def create_cell_from_args(variables: dict) -> NotebookNode: + """Creates NotebookNode object containing dict of provided variables. Args: - args: Arguments that need to be injected into a NotebookNode. + variables: Arguments that need to be injected into a NotebookNode. Returns: NotebookNode with provided arguments as variables. """ - variables = "" - args = json.loads(args) - for key in sorted(args.keys()): - # Check type of variable to maintain type when converting from JSON - # to notebook cell - if args[key] is None or isinstance(args[key], bool): - variables += "{} = {}\n".format(key, args[key]) - else: - variables += '{} = "{}"\n'.format(key, args[key]) - - return new_code_cell(variables) + return new_code_cell("variables = {}".format(repr(variables))) @staticmethod def create_cell_from_file(filepath: Text) -> NotebookNode: diff --git a/backend/src/apiserver/visualization/roc_curve.py b/backend/src/apiserver/visualization/roc_curve.py index 581914d9f84..a9142cca17f 100644 --- a/backend/src/apiserver/visualization/roc_curve.py +++ b/backend/src/apiserver/visualization/roc_curve.py @@ -12,6 +12,8 @@ # See the License for the specific language governing permissions and # limitations under the License. +# flake8: noqa TODO + import json from pathlib import Path from bokeh.layouts import row @@ -34,13 +36,13 @@ # trueclass # true_score_column -if is_generated is False: +if not variables.get("is_generated"): # Create data from specified csv file(s). # The schema file provides column names for the csv file that will be used # to generate the roc curve. - schema_file = Path(source) / 'schema.json' + schema_file = Path(source) / "schema.json" schema = json.loads(file_io.read_file_to_string(schema_file)) - names = [x['name'] for x in schema] + names = [x["name"] for x in schema] dfs = [] files = file_io.get_matching_files(source) @@ -48,25 +50,25 @@ dfs.append(pd.read_csv(f, names=names)) df = pd.concat(dfs) - if target_lambda: - df['target'] = df.apply(eval(target_lambda), axis=1) + if variables["target_lambda"]: + df["target"] = df.apply(eval(variables["target_lambda"]), axis=1) else: - df['target'] = df['target'].apply(lambda x: 1 if x == trueclass else 0) - fpr, tpr, thresholds = roc_curve(df['target'], df[true_score_column]) - source = pd.DataFrame({'fpr': fpr, 'tpr': tpr, 'thresholds': thresholds}) + df["target"] = df["target"].apply(lambda x: 1 if x == variables["trueclass"] else 0) + fpr, tpr, thresholds = roc_curve(df["target"], df[variables["true_score_column"]]) + df = pd.DataFrame({"fpr": fpr, "tpr": tpr, "thresholds": thresholds}) else: # Load data from generated csv file. - source = pd.read_csv( + df = pd.read_csv( source, header=None, - names=['fpr', 'tpr', 'thresholds'] + names=["fpr", "tpr", "thresholds"] ) # Create visualization. output_notebook() p = figure(tools="pan,wheel_zoom,box_zoom,reset,hover,previewsave") -p.line('fpr', 'tpr', line_width=2, source=source) +p.line("fpr", "tpr", line_width=2, source=df) hover = p.select(dict(type=HoverTool)) hover.tooltips = [("Threshold", "@thresholds")] diff --git a/backend/src/apiserver/visualization/server.py b/backend/src/apiserver/visualization/server.py index f894326d001..fe02c6820d6 100644 --- a/backend/src/apiserver/visualization/server.py +++ b/backend/src/apiserver/visualization/server.py @@ -13,7 +13,9 @@ # limitations under the License. import argparse +from argparse import Namespace import importlib +import json from pathlib import Path from typing import Text import shlex @@ -73,48 +75,47 @@ def initialize(self): help="JSON string of arguments to be provided to visualizations." ) - def get_arguments_from_body(self) -> argparse.Namespace: - """Converts arguments from post request to argparser.Namespace format. + def get_arguments_from_body(self) -> Namespace: + """Converts arguments from post request to Namespace format. This is done because arguments, by default are provided in the x-www-form-urlencoded format. This format is difficult to parse compared - to argparser.Namespace, which is a dict. + to Namespace, which is a dict. Returns: - Arguments provided from post request as arparser.Namespace object. + Arguments provided from post request as a Namespace object. """ split_arguments = shlex.split(self.get_body_argument("arguments")) return self.requestParser.parse_args(split_arguments) - def is_valid_request_arguments(self, arguments: argparse.Namespace) -> bool: - """Validates arguments from post request and sends error if invalid. + def is_valid_request_arguments(self, arguments: Namespace): + """Validates arguments from post request and raises error if invalid. Args: - arguments: x-www-form-urlencoded formatted arguments - - Returns: - Boolean value representing if provided arguments are valid. + arguments: Namespace formatted arguments """ if arguments.type is None: - self.send_error(400, reason="No type specified.") - return False + raise Exception("No type specified.") if arguments.source is None: - self.send_error(400, reason="No source specified.") - return False + raise Exception("No source specified.") + try: + json.loads(arguments.arguments) + except json.JSONDecodeError: + raise Exception("Invalid JSON provided as arguments.") return True def generate_notebook_from_arguments( self, - arguments: argparse.Namespace, + arguments: dict, source: Text, visualization_type: Text ) -> NotebookNode: """Generates a NotebookNode from provided arguments. Args: - arguments: x-www-form-urlencoded formatted arguments. - input_path: Path or path pattern to be used as data reference for + arguments: JSON object containing provided arguments. + source: Path or path pattern to be used as data reference for visualization. visualization_type: Name of visualization to be generated. @@ -139,16 +140,20 @@ def post(self): # Parse arguments from request. request_arguments = self.get_arguments_from_body() # Validate arguments from request. - if self.is_valid_request_arguments(request_arguments): - # Create notebook with arguments from request. - nb = self.generate_notebook_from_arguments( - request_arguments.arguments, - request_arguments.source, - request_arguments.type - ) - # Generate visualization (output for notebook). - html = _exporter.generate_html_from_notebook(nb) - self.write(html) + try: + self.is_valid_request_arguments(request_arguments) + except Exception as e: + return self.send_error(400, reason=str(e)) + + # Create notebook with arguments from request. + nb = self.generate_notebook_from_arguments( + json.loads(request_arguments.arguments), + request_arguments.source, + request_arguments.type + ) + # Generate visualization (output for notebook). + html = _exporter.generate_html_from_notebook(nb) + self.write(html) if __name__ == "__main__": diff --git a/backend/src/apiserver/visualization/snapshots/snap_test_exporter.py b/backend/src/apiserver/visualization/snapshots/snap_test_exporter.py index 8a43b6a1b11..9db40f393e1 100644 --- a/backend/src/apiserver/visualization/snapshots/snap_test_exporter.py +++ b/backend/src/apiserver/visualization/snapshots/snap_test_exporter.py @@ -7,16 +7,82 @@ snapshots = Snapshot() -snapshots['TestExporterMethods::test_create_cell_from_args_with_multiple_args 1'] = '''source = "gs://ml-pipeline/data.csv" -target_lambda = "lambda x: (x[\'target\'] > x[\'fare\'] * 0.2)" +snapshots['TestExporterMethods::test_create_cell_from_args_with_multiple_args 1'] = ''' +
+
+ + +
+ +
+ + +
+
['gs://ml-pipeline/data.csv', "lambda x: (x['target'] > x['fare'] * 0.2)"]
+
+
+
+ +
+
+ + + ''' -snapshots['TestExporterMethods::test_create_cell_from_args_with_no_args 1'] = '' +snapshots['TestExporterMethods::test_create_cell_from_args_with_no_args 1'] = ''' +
+
+ + +
+ +
+ + +
+
{}
+
+
+
+ +
+
+ + -snapshots['TestExporterMethods::test_create_cell_from_args_with_one_arg 1'] = '''source = "gs://ml-pipeline/data.csv" ''' -snapshots['TestExporterMethods::test_create_cell_from_file 1'] = '''# Copyright 2019 Google LLC +snapshots['TestExporterMethods::test_create_cell_from_args_with_one_arg 1'] = ''' +
+
+ + +
+ +
+ + +
+
['gs://ml-pipeline/data.csv']
+
+
+
+ +
+
+ + + +''' + +snapshots['TestExporterMethods::test_create_cell_from_file 1'] = '''""" +test.py is used for test_server.py as a way to test the tornado web server +without having a reliance on the validity of visualizations. It does not serve +as a valid visualization and is only used for testing purposes. +""" + +# Copyright 2019 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -30,17 +96,8 @@ # See the License for the specific language governing permissions and # limitations under the License. -import tensorflow_data_validation as tfdv - -# The following variables are provided through dependency injection. These -# variables come from the specified input path and arguments provided by the -# API post request. -# -# source - -train_stats = tfdv.generate_statistics_from_csv(data_location=source) - -tfdv.visualize_statistics(train_stats) +x = 2 +print(x) ''' snapshots['TestExporterMethods::test_generate_html_from_notebook 1'] = ''' diff --git a/backend/src/apiserver/visualization/test_exporter.py b/backend/src/apiserver/visualization/test_exporter.py index 52e4dced1fe..f7447b932e4 100644 --- a/backend/src/apiserver/visualization/test_exporter.py +++ b/backend/src/apiserver/visualization/test_exporter.py @@ -28,36 +28,45 @@ def setUp(self): def test_create_cell_from_args_with_no_args(self): self.maxDiff = None - args = "{}" - cell = self.exporter.create_cell_from_args(args) - self.assertMatchSnapshot(cell.source) + nb = new_notebook() + args = {} + nb.cells.append(self.exporter.create_cell_from_args(args)) + nb.cells.append(new_code_cell("print(variables)")) + html = self.exporter.generate_html_from_notebook(nb) + self.assertMatchSnapshot(html) def test_create_cell_from_args_with_one_arg(self): self.maxDiff = None - args = '{"source": "gs://ml-pipeline/data.csv"}' - cell = self.exporter.create_cell_from_args(args) - self.assertMatchSnapshot(cell.source) + nb = new_notebook() + args = {"source": "gs://ml-pipeline/data.csv"} + nb.cells.append(self.exporter.create_cell_from_args(args)) + nb.cells.append(new_code_cell("print([variables[key] for key in sorted(variables.keys())])")) + html = self.exporter.generate_html_from_notebook(nb) + self.assertMatchSnapshot(html) def test_create_cell_from_args_with_multiple_args(self): self.maxDiff = None - args = ( - '{"source": "gs://ml-pipeline/data.csv", ' - "\"target_lambda\": \"lambda x: (x['target'] > x['fare'] * 0.2)\"}" - ) - cell = self.exporter.create_cell_from_args(args) - self.assertMatchSnapshot(cell.source) + nb = new_notebook() + args = { + "source": "gs://ml-pipeline/data.csv", + "target_lambda": "lambda x: (x['target'] > x['fare'] * 0.2)" + } + nb.cells.append(self.exporter.create_cell_from_args(args)) + nb.cells.append(new_code_cell("print([variables[key] for key in sorted(variables.keys())])")) + html = self.exporter.generate_html_from_notebook(nb) + self.assertMatchSnapshot(html) def test_create_cell_from_file(self): self.maxDiff = None - cell = self.exporter.create_cell_from_file("tfdv.py") + cell = self.exporter.create_cell_from_file("test.py") self.assertMatchSnapshot(cell.source) def test_generate_html_from_notebook(self): self.maxDiff = None nb = new_notebook() - args = '{"x": 2}' + args = {"x": 2} nb.cells.append(self.exporter.create_cell_from_args(args)) - nb.cells.append(new_code_cell("print(x)")) + nb.cells.append(new_code_cell("print(variables['x'])")) html = self.exporter.generate_html_from_notebook(nb) self.assertMatchSnapshot(html) diff --git a/backend/src/apiserver/visualization/test_server.py b/backend/src/apiserver/visualization/test_server.py index a70e2a8e126..222090ddc88 100644 --- a/backend/src/apiserver/visualization/test_server.py +++ b/backend/src/apiserver/visualization/test_server.py @@ -70,6 +70,17 @@ def test_create_visualization_fails_when_missing_input_path(self): response.body ) + def test_create_visualization_fails_when_invalid_json_is_provided(self): + response = self.fetch( + "/", + method="POST", + body='arguments=--type test --source gs://ml-pipeline/data.csv --arguments "{"') + self.assertEqual(400, response.code) + self.assertEqual( + wrap_error_in_html("400: Invalid JSON provided as arguments."), + response.body + ) + def test_create_visualization(self): response = self.fetch( "/", diff --git a/backend/src/apiserver/visualization/tfdv.py b/backend/src/apiserver/visualization/tfdv.py index 0190f434e6d..b01f4336fa5 100644 --- a/backend/src/apiserver/visualization/tfdv.py +++ b/backend/src/apiserver/visualization/tfdv.py @@ -20,6 +20,6 @@ # # source -train_stats = tfdv.generate_statistics_from_csv(data_location=source) +train_stats = tfdv.generate_statistics_from_csv(data_location=source) # noqa: F821 tfdv.visualize_statistics(train_stats) diff --git a/components/arena/python/arena/_arena_distributed_tf_op.py b/components/arena/python/arena/_arena_distributed_tf_op.py index e98d01a38e1..f67175cb20f 100644 --- a/components/arena/python/arena/_arena_distributed_tf_op.py +++ b/components/arena/python/arena/_arena_distributed_tf_op.py @@ -13,20 +13,21 @@ # See the License for the specific language governing permissions and # limitations under the License. +# flake8: noqa TODO import kfp.dsl as dsl import datetime import logging -def estimator_op(name, image, command, +def estimator_op(name, image, command, chief_cpu_limit, chief_memory_limit, chief_port, workers, worker_image, worker_cpu_limit, worker_memory_limit, - parameter_servers, ps_image, ps_cpu_limit, ps_memory_limit, ps_port, - gpus, rdma, - tensorboard, + parameter_servers, ps_image, ps_cpu_limit, ps_memory_limit, ps_port, + gpus, rdma, + tensorboard, worker_port, annotations=[], - evaluator=False, evaluator_cpu_limit='0', evaluator_memory_limit='0', + evaluator=False, evaluator_cpu_limit='0', evaluator_memory_limit='0', env=[], data=[], sync_source=None, metrics=['Train-accuracy:PERCENTAGE'], arena_image='cheyang/arena_launcher:v0.7', @@ -44,11 +45,11 @@ def estimator_op(name, image, command, workers=workers, worker_image=worker_image, worker_cpu_limit=worker_cpu_limit, worker_memory_limit=worker_memory, parameter_servers=parameter_servers, ps_image=ps_image, ps_cpu_limit=ps_cpu_limit, ps_memory_limit=ps_memory_limit, gpus=gpus, rdma=rdma, - chief=True, + chief=True, chief_cpu_limit=chief_cpu_limit, worker_port=worker_port, ps_port=ps_port, - tensorboard=tensorboard, + tensorboard=tensorboard, metrics=metrics, arena_image=arena_image, timeout_hours=timeout_hours) @@ -58,9 +59,9 @@ def estimator_op(name, image, command, def parameter_servers_op(name, image, command, env, data, sync_source, annotations, workers, worker_image, worker_cpu_limit, worker_memory, parameter_servers, ps_image, ps_cpu_limit, ps_memory_limit, - gpus, rdma, - tensorboard, - worker_port, ps_port, + gpus, rdma, + tensorboard, + worker_port, ps_port, metrics=['Train-accuracy:PERCENTAGE'], arena_image='cheyang/arena_launcher:v0.7', timeout_hours=240): @@ -76,10 +77,10 @@ def parameter_servers_op(name, image, command, env, data, sync_source, annotatio return distributed_tf_op(name=name, image=image, command=command, envs=envs, data=data, sync_source=sync_source, workers=workers, worker_image=worker_image, worker_cpu_limit=worker_cpu_limit, worker_memory_limit=worker_memory, parameter_servers=parameter_servers, ps_image=ps_image, ps_cpu_limit=ps_cpu_limit, ps_memory_limit=ps_memory_limit, - gpus=gpus, rdma=rdma, + gpus=gpus, rdma=rdma, worker_port=worker_port, ps_port=ps_port, - tensorboard=tensorboard, + tensorboard=tensorboard, metrics=metrics, arena_image=arena_image, timeout_hours=timeout_hours) @@ -87,15 +88,15 @@ def parameter_servers_op(name, image, command, env, data, sync_source, annotatio def distributed_tf_op(name, image, command, env=[], data=[], sync_source=None, - chief=False, chief_cpu_limit='0', chief_memory_limit='0', + chief=False, chief_cpu_limit='0', chief_memory_limit='0', workers=0, worker_image=None, worker_cpu_limit='0', worker_memory_limit='0', parameter_servers=0, ps_image=None, ps_cpu_limit='0', ps_memory_limit='0', - evaluator=False, evaluator_cpu_limit='0', evaluator_memory_limit='0', - gpus=0, rdma=False, + evaluator=False, evaluator_cpu_limit='0', evaluator_memory_limit='0', + gpus=0, rdma=False, chief_port=22222, worker_port=22222, ps_port=22224, - tensorboard=False, + tensorboard=False, metrics=['Train-accuracy:PERCENTAGE'], arena_image='cheyang/arena_launcher:v0.7', timeout_hours=240): diff --git a/components/aws/emr/create_cluster/src/create_cluster.py b/components/aws/emr/create_cluster/src/create_cluster.py index 9885158464a..a570042db47 100644 --- a/components/aws/emr/create_cluster/src/create_cluster.py +++ b/components/aws/emr/create_cluster/src/create_cluster.py @@ -17,6 +17,12 @@ from common import _utils +try: + unicode +except NameError: + unicode = str + + def main(argv=None): parser = argparse.ArgumentParser(description='Create EMR Cluster') parser.add_argument('--region', type=str, help='EMR Cluster region.') diff --git a/components/aws/emr/submit_pyspark_job/src/submit_pyspark_job.py b/components/aws/emr/submit_pyspark_job/src/submit_pyspark_job.py index a3a668ff158..c4a5acdc0e3 100644 --- a/components/aws/emr/submit_pyspark_job/src/submit_pyspark_job.py +++ b/components/aws/emr/submit_pyspark_job/src/submit_pyspark_job.py @@ -29,6 +29,11 @@ from common import _utils +try: + unicode +except NameError: + unicode = str + def main(argv=None): parser = argparse.ArgumentParser(description='Submit PySpark Job') diff --git a/components/aws/emr/submit_spark_job/src/submit_spark_job.py b/components/aws/emr/submit_spark_job/src/submit_spark_job.py index 3ed852f0676..0d6630031e3 100644 --- a/components/aws/emr/submit_spark_job/src/submit_spark_job.py +++ b/components/aws/emr/submit_spark_job/src/submit_spark_job.py @@ -30,6 +30,11 @@ from common import _utils +try: + unicode +except NameError: + unicode = str + def main(argv=None): parser = argparse.ArgumentParser(description='Submit Spark Job') diff --git a/components/aws/sagemaker/batch_transform/src/batch_transform.py b/components/aws/sagemaker/batch_transform/src/batch_transform.py index f75bb65996c..f5c6b8f6fd2 100644 --- a/components/aws/sagemaker/batch_transform/src/batch_transform.py +++ b/components/aws/sagemaker/batch_transform/src/batch_transform.py @@ -16,6 +16,12 @@ from common import _utils +try: + unicode +except NameError: + unicode = str + + def main(argv=None): parser = argparse.ArgumentParser(description='SageMaker Batch Transformation Job') parser.add_argument('--region', type=str.strip, required=True, help='The region where the cluster launches.') diff --git a/components/dataflow/predict/component.yaml b/components/dataflow/predict/component.yaml index 846e085cc1b..6fae9083fb3 100644 --- a/components/dataflow/predict/component.yaml +++ b/components/dataflow/predict/component.yaml @@ -15,7 +15,7 @@ outputs: - {name: Predictions dir, type: GCSPath, description: 'GCS or local directory.'} #Will contain prediction_results-* and schema.json files; TODO: Split outputs and replace dir with single file # type: {GCSPath: {path_type: Directory}} implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-dataflow-tf-predict:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-dataflow-tf-predict:151c5349f13bea9d626c988563c04c0a86210c21 command: [python2, /ml/predict.py] args: [ --data, {inputValue: Data file pattern}, diff --git a/components/dataflow/tfdv/component.yaml b/components/dataflow/tfdv/component.yaml index 009ebd3a03b..b72a358fa73 100644 --- a/components/dataflow/tfdv/component.yaml +++ b/components/dataflow/tfdv/component.yaml @@ -18,7 +18,7 @@ outputs: - {name: Validation result, type: String, description: Indicates whether anomalies were detected or not.} implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-dataflow-tfdv:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-dataflow-tfdv:151c5349f13bea9d626c988563c04c0a86210c21 command: [python2, /ml/validate.py] args: [ --csv-data-for-inference, {inputValue: Inference data}, diff --git a/components/dataflow/tfma/component.yaml b/components/dataflow/tfma/component.yaml index 940b72bee6d..bc865d6d4c0 100644 --- a/components/dataflow/tfma/component.yaml +++ b/components/dataflow/tfma/component.yaml @@ -17,7 +17,7 @@ outputs: - {name: Analysis results dir, type: GCSPath, description: GCS or local directory where the analysis results should were written.} # type: {GCSPath: {path_type: Directory}} implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-dataflow-tfma:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-dataflow-tfma:151c5349f13bea9d626c988563c04c0a86210c21 command: [python2, /ml/model_analysis.py] args: [ --model, {inputValue: Model}, diff --git a/components/dataflow/tft/component.yaml b/components/dataflow/tft/component.yaml index f004defee54..c77ea6886ba 100644 --- a/components/dataflow/tft/component.yaml +++ b/components/dataflow/tft/component.yaml @@ -12,7 +12,7 @@ outputs: - {name: Transformed data dir, type: GCSPath} # type: {GCSPath: {path_type: Directory}} implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-dataflow-tft:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-dataflow-tft:151c5349f13bea9d626c988563c04c0a86210c21 command: [python2, /ml/transform.py] args: [ --train, {inputValue: Training data file pattern}, diff --git a/components/gcp/bigquery/query/README.md b/components/gcp/bigquery/query/README.md index da0458d47f0..8e9331f013a 100644 --- a/components/gcp/bigquery/query/README.md +++ b/components/gcp/bigquery/query/README.md @@ -89,7 +89,7 @@ KFP_PACKAGE = 'https://storage.googleapis.com/ml-pipeline/release/0.1.14/kfp.tar import kfp.components as comp bigquery_query_op = comp.load_component_from_url( - 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/bigquery/query/component.yaml') + 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/bigquery/query/component.yaml') help(bigquery_query_op) ``` diff --git a/components/gcp/bigquery/query/component.yaml b/components/gcp/bigquery/query/component.yaml index 8c93e658ac0..07f29549b40 100644 --- a/components/gcp/bigquery/query/component.yaml +++ b/components/gcp/bigquery/query/component.yaml @@ -57,7 +57,7 @@ outputs: type: GCSPath implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-gcp:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-gcp:151c5349f13bea9d626c988563c04c0a86210c21 args: [ kfp_component.google.bigquery, query, --query, {inputValue: query}, diff --git a/components/gcp/bigquery/query/sample.ipynb b/components/gcp/bigquery/query/sample.ipynb index 6ddb4ce483b..98b6e75ad07 100644 --- a/components/gcp/bigquery/query/sample.ipynb +++ b/components/gcp/bigquery/query/sample.ipynb @@ -108,7 +108,7 @@ "import kfp.components as comp\n", "\n", "bigquery_query_op = comp.load_component_from_url(\n", - " 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/bigquery/query/component.yaml')\n", + " 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/bigquery/query/component.yaml')\n", "help(bigquery_query_op)" ] }, diff --git a/components/gcp/container/component_sdk/python/kfp_component/google/ml_engine/_create_version.py b/components/gcp/container/component_sdk/python/kfp_component/google/ml_engine/_create_version.py index d9f9fda0bd6..8054b8d3aca 100644 --- a/components/gcp/container/component_sdk/python/kfp_component/google/ml_engine/_create_version.py +++ b/components/gcp/container/component_sdk/python/kfp_component/google/ml_engine/_create_version.py @@ -181,7 +181,7 @@ def _dump_metadata(self): ### GCloud command ```bash -gcloud ml-engine predict --model {} \ +gcloud ai-platform predict --model {} \ --version {} \ --json-instances instances.json ``` diff --git a/components/gcp/container/component_sdk/python/setup.py b/components/gcp/container/component_sdk/python/setup.py index ba988792bf2..32947a265b5 100644 --- a/components/gcp/container/component_sdk/python/setup.py +++ b/components/gcp/container/component_sdk/python/setup.py @@ -14,8 +14,8 @@ from setuptools import setup -PACKAGE_NAME = "kfp-component" -VERSION = '0.1.25' +PACKAGE_NAME = 'kfp-component' +VERSION = '0.1.27' setup( name=PACKAGE_NAME, @@ -23,33 +23,30 @@ description='KubeFlow Pipelines Component SDK', author='google', install_requires=[ - 'kubernetes >= 8.0.1', - 'urllib3>=1.15,<1.25', - 'fire == 0.1.3', - 'google-api-python-client == 1.7.8', - 'google-cloud-storage == 1.14.0', - 'google-cloud-bigquery == 1.9.0' + 'kubernetes >= 8.0.1', 'urllib3>=1.15,<1.25', 'fire == 0.1.3', + 'google-api-python-client == 1.7.8', 'google-cloud-storage == 1.14.0', + 'google-cloud-bigquery == 1.9.0' ], packages=[ - 'kfp_component', + 'kfp_component', ], classifiers=[ - 'Intended Audience :: Developers', - 'Intended Audience :: Education', - 'Intended Audience :: Science/Research', - 'License :: OSI Approved :: Apache Software License', - 'Programming Language :: Python', - 'Programming Language :: Python :: 2', - 'Programming Language :: Python :: 2.7', - 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.5', - 'Programming Language :: Python :: 3.6', - 'Programming Language :: Python :: 3.7', - 'Topic :: Scientific/Engineering', - 'Topic :: Scientific/Engineering :: Artificial Intelligence', - 'Topic :: Software Development', - 'Topic :: Software Development :: Libraries', - 'Topic :: Software Development :: Libraries :: Python Modules', + 'Intended Audience :: Developers', + 'Intended Audience :: Education', + 'Intended Audience :: Science/Research', + 'License :: OSI Approved :: Apache Software License', + 'Programming Language :: Python', + 'Programming Language :: Python :: 2', + 'Programming Language :: Python :: 2.7', + 'Programming Language :: Python :: 3', + 'Programming Language :: Python :: 3.5', + 'Programming Language :: Python :: 3.6', + 'Programming Language :: Python :: 3.7', + 'Topic :: Scientific/Engineering', + 'Topic :: Scientific/Engineering :: Artificial Intelligence', + 'Topic :: Software Development', + 'Topic :: Software Development :: Libraries', + 'Topic :: Software Development :: Libraries :: Python Modules', ], include_package_data=True, ) diff --git a/components/gcp/dataflow/launch_python/README.md b/components/gcp/dataflow/launch_python/README.md index 855cab82970..8279bd18ed7 100644 --- a/components/gcp/dataflow/launch_python/README.md +++ b/components/gcp/dataflow/launch_python/README.md @@ -77,7 +77,7 @@ KFP_PACKAGE = 'https://storage.googleapis.com/ml-pipeline/release/0.1.14/kfp.tar import kfp.components as comp dataflow_python_op = comp.load_component_from_url( - 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/dataflow/launch_python/component.yaml') + 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/dataflow/launch_python/component.yaml') help(dataflow_python_op) ``` diff --git a/components/gcp/dataflow/launch_python/component.yaml b/components/gcp/dataflow/launch_python/component.yaml index f3c6baa3203..920408aa48d 100644 --- a/components/gcp/dataflow/launch_python/component.yaml +++ b/components/gcp/dataflow/launch_python/component.yaml @@ -51,7 +51,7 @@ outputs: type: String implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-gcp:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-gcp:151c5349f13bea9d626c988563c04c0a86210c21 args: [ kfp_component.google.dataflow, launch_python, --python_file_path, {inputValue: python_file_path}, diff --git a/components/gcp/dataflow/launch_python/sample.ipynb b/components/gcp/dataflow/launch_python/sample.ipynb index 8f8a0e2efcf..db277bae853 100644 --- a/components/gcp/dataflow/launch_python/sample.ipynb +++ b/components/gcp/dataflow/launch_python/sample.ipynb @@ -95,7 +95,7 @@ "import kfp.components as comp\n", "\n", "dataflow_python_op = comp.load_component_from_url(\n", - " 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/dataflow/launch_python/component.yaml')\n", + " 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/dataflow/launch_python/component.yaml')\n", "help(dataflow_python_op)" ] }, diff --git a/components/gcp/dataflow/launch_template/README.md b/components/gcp/dataflow/launch_template/README.md index 0b23a149d76..92a0f9771f1 100644 --- a/components/gcp/dataflow/launch_template/README.md +++ b/components/gcp/dataflow/launch_template/README.md @@ -67,7 +67,7 @@ KFP_PACKAGE = 'https://storage.googleapis.com/ml-pipeline/release/0.1.14/kfp.tar import kfp.components as comp dataflow_template_op = comp.load_component_from_url( - 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/dataflow/launch_template/component.yaml') + 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/dataflow/launch_template/component.yaml') help(dataflow_template_op) ``` diff --git a/components/gcp/dataflow/launch_template/component.yaml b/components/gcp/dataflow/launch_template/component.yaml index 5eb56f3732b..09f29d0fab9 100644 --- a/components/gcp/dataflow/launch_template/component.yaml +++ b/components/gcp/dataflow/launch_template/component.yaml @@ -61,7 +61,7 @@ outputs: type: String implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-gcp:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-gcp:151c5349f13bea9d626c988563c04c0a86210c21 args: [ kfp_component.google.dataflow, launch_template, --project_id, {inputValue: project_id}, diff --git a/components/gcp/dataflow/launch_template/sample.ipynb b/components/gcp/dataflow/launch_template/sample.ipynb index 461a548521b..e5dd3eb1c0c 100644 --- a/components/gcp/dataflow/launch_template/sample.ipynb +++ b/components/gcp/dataflow/launch_template/sample.ipynb @@ -85,7 +85,7 @@ "import kfp.components as comp\n", "\n", "dataflow_template_op = comp.load_component_from_url(\n", - " 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/dataflow/launch_template/component.yaml')\n", + " 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/dataflow/launch_template/component.yaml')\n", "help(dataflow_template_op)" ] }, diff --git a/components/gcp/dataproc/create_cluster/README.md b/components/gcp/dataproc/create_cluster/README.md index f65e04cb763..94ceb6d7d85 100644 --- a/components/gcp/dataproc/create_cluster/README.md +++ b/components/gcp/dataproc/create_cluster/README.md @@ -74,7 +74,7 @@ KFP_PACKAGE = 'https://storage.googleapis.com/ml-pipeline/release/0.1.14/kfp.tar import kfp.components as comp dataproc_create_cluster_op = comp.load_component_from_url( - 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/dataproc/create_cluster/component.yaml') + 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/dataproc/create_cluster/component.yaml') help(dataproc_create_cluster_op) ``` diff --git a/components/gcp/dataproc/create_cluster/component.yaml b/components/gcp/dataproc/create_cluster/component.yaml index 6a567356375..c5cad047204 100644 --- a/components/gcp/dataproc/create_cluster/component.yaml +++ b/components/gcp/dataproc/create_cluster/component.yaml @@ -68,7 +68,7 @@ outputs: type: String implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-gcp:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-gcp:151c5349f13bea9d626c988563c04c0a86210c21 args: [ kfp_component.google.dataproc, create_cluster, --project_id, {inputValue: project_id}, diff --git a/components/gcp/dataproc/create_cluster/sample.ipynb b/components/gcp/dataproc/create_cluster/sample.ipynb index e5b2df61e64..5c22e8731b8 100644 --- a/components/gcp/dataproc/create_cluster/sample.ipynb +++ b/components/gcp/dataproc/create_cluster/sample.ipynb @@ -92,7 +92,7 @@ "import kfp.components as comp\n", "\n", "dataproc_create_cluster_op = comp.load_component_from_url(\n", - " 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/dataproc/create_cluster/component.yaml')\n", + " 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/dataproc/create_cluster/component.yaml')\n", "help(dataproc_create_cluster_op)" ] }, diff --git a/components/gcp/dataproc/delete_cluster/README.md b/components/gcp/dataproc/delete_cluster/README.md index 83e524cb333..c9167e81ea9 100644 --- a/components/gcp/dataproc/delete_cluster/README.md +++ b/components/gcp/dataproc/delete_cluster/README.md @@ -56,7 +56,7 @@ KFP_PACKAGE = 'https://storage.googleapis.com/ml-pipeline/release/0.1.14/kfp.tar import kfp.components as comp dataproc_delete_cluster_op = comp.load_component_from_url( - 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/dataproc/delete_cluster/component.yaml') + 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/dataproc/delete_cluster/component.yaml') help(dataproc_delete_cluster_op) ``` diff --git a/components/gcp/dataproc/delete_cluster/component.yaml b/components/gcp/dataproc/delete_cluster/component.yaml index f164bb56bbf..15654a8f24e 100644 --- a/components/gcp/dataproc/delete_cluster/component.yaml +++ b/components/gcp/dataproc/delete_cluster/component.yaml @@ -36,7 +36,7 @@ inputs: type: Integer implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-gcp:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-gcp:151c5349f13bea9d626c988563c04c0a86210c21 args: [ kfp_component.google.dataproc, delete_cluster, --project_id, {inputValue: project_id}, diff --git a/components/gcp/dataproc/delete_cluster/sample.ipynb b/components/gcp/dataproc/delete_cluster/sample.ipynb index 8105ea6015b..db162b43eb4 100644 --- a/components/gcp/dataproc/delete_cluster/sample.ipynb +++ b/components/gcp/dataproc/delete_cluster/sample.ipynb @@ -75,7 +75,7 @@ "import kfp.components as comp\n", "\n", "dataproc_delete_cluster_op = comp.load_component_from_url(\n", - " 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/dataproc/delete_cluster/component.yaml')\n", + " 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/dataproc/delete_cluster/component.yaml')\n", "help(dataproc_delete_cluster_op)" ] }, diff --git a/components/gcp/dataproc/submit_hadoop_job/README.md b/components/gcp/dataproc/submit_hadoop_job/README.md index a567880a2f2..23487186c16 100644 --- a/components/gcp/dataproc/submit_hadoop_job/README.md +++ b/components/gcp/dataproc/submit_hadoop_job/README.md @@ -72,7 +72,7 @@ KFP_PACKAGE = 'https://storage.googleapis.com/ml-pipeline/release/0.1.14/kfp.tar import kfp.components as comp dataproc_submit_hadoop_job_op = comp.load_component_from_url( - 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/dataproc/submit_hadoop_job/component.yaml') + 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/dataproc/submit_hadoop_job/component.yaml') help(dataproc_submit_hadoop_job_op) ``` diff --git a/components/gcp/dataproc/submit_hadoop_job/component.yaml b/components/gcp/dataproc/submit_hadoop_job/component.yaml index 617a9053862..8e73c7a4d7a 100644 --- a/components/gcp/dataproc/submit_hadoop_job/component.yaml +++ b/components/gcp/dataproc/submit_hadoop_job/component.yaml @@ -78,7 +78,7 @@ outputs: type: String implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-gcp:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-gcp:151c5349f13bea9d626c988563c04c0a86210c21 args: [ kfp_component.google.dataproc, submit_hadoop_job, --project_id, {inputValue: project_id}, diff --git a/components/gcp/dataproc/submit_hadoop_job/sample.ipynb b/components/gcp/dataproc/submit_hadoop_job/sample.ipynb index 10992967e9c..7f3bdf98e2d 100644 --- a/components/gcp/dataproc/submit_hadoop_job/sample.ipynb +++ b/components/gcp/dataproc/submit_hadoop_job/sample.ipynb @@ -90,7 +90,7 @@ "import kfp.components as comp\n", "\n", "dataproc_submit_hadoop_job_op = comp.load_component_from_url(\n", - " 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/dataproc/submit_hadoop_job/component.yaml')\n", + " 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/dataproc/submit_hadoop_job/component.yaml')\n", "help(dataproc_submit_hadoop_job_op)" ] }, diff --git a/components/gcp/dataproc/submit_hive_job/README.md b/components/gcp/dataproc/submit_hive_job/README.md index a9c2197fc1b..5d55816e3b0 100644 --- a/components/gcp/dataproc/submit_hive_job/README.md +++ b/components/gcp/dataproc/submit_hive_job/README.md @@ -63,7 +63,7 @@ KFP_PACKAGE = 'https://storage.googleapis.com/ml-pipeline/release/0.1.14/kfp.tar import kfp.components as comp dataproc_submit_hive_job_op = comp.load_component_from_url( - 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/dataproc/submit_hive_job/component.yaml') + 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/dataproc/submit_hive_job/component.yaml') help(dataproc_submit_hive_job_op) ``` diff --git a/components/gcp/dataproc/submit_hive_job/component.yaml b/components/gcp/dataproc/submit_hive_job/component.yaml index 9f0063f927a..3140de61005 100644 --- a/components/gcp/dataproc/submit_hive_job/component.yaml +++ b/components/gcp/dataproc/submit_hive_job/component.yaml @@ -73,7 +73,7 @@ outputs: type: String implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-gcp:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-gcp:151c5349f13bea9d626c988563c04c0a86210c21 args: [ kfp_component.google.dataproc, submit_hive_job, --project_id, {inputValue: project_id}, diff --git a/components/gcp/dataproc/submit_hive_job/sample.ipynb b/components/gcp/dataproc/submit_hive_job/sample.ipynb index fe860941fa7..81c8f46225e 100644 --- a/components/gcp/dataproc/submit_hive_job/sample.ipynb +++ b/components/gcp/dataproc/submit_hive_job/sample.ipynb @@ -81,7 +81,7 @@ "import kfp.components as comp\n", "\n", "dataproc_submit_hive_job_op = comp.load_component_from_url(\n", - " 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/dataproc/submit_hive_job/component.yaml')\n", + " 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/dataproc/submit_hive_job/component.yaml')\n", "help(dataproc_submit_hive_job_op)" ] }, diff --git a/components/gcp/dataproc/submit_pig_job/README.md b/components/gcp/dataproc/submit_pig_job/README.md index 087d9935be3..10800068599 100644 --- a/components/gcp/dataproc/submit_pig_job/README.md +++ b/components/gcp/dataproc/submit_pig_job/README.md @@ -66,7 +66,7 @@ KFP_PACKAGE = 'https://storage.googleapis.com/ml-pipeline/release/0.1.14/kfp.tar import kfp.components as comp dataproc_submit_pig_job_op = comp.load_component_from_url( - 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/dataproc/submit_pig_job/component.yaml') + 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/dataproc/submit_pig_job/component.yaml') help(dataproc_submit_pig_job_op) ``` diff --git a/components/gcp/dataproc/submit_pig_job/component.yaml b/components/gcp/dataproc/submit_pig_job/component.yaml index 80c80dc165b..52e314afe1c 100644 --- a/components/gcp/dataproc/submit_pig_job/component.yaml +++ b/components/gcp/dataproc/submit_pig_job/component.yaml @@ -73,7 +73,7 @@ outputs: type: String implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-gcp:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-gcp:151c5349f13bea9d626c988563c04c0a86210c21 args: [ kfp_component.google.dataproc, submit_pig_job, --project_id, {inputValue: project_id}, diff --git a/components/gcp/dataproc/submit_pig_job/sample.ipynb b/components/gcp/dataproc/submit_pig_job/sample.ipynb index 5ae5118be40..000c798d5d8 100644 --- a/components/gcp/dataproc/submit_pig_job/sample.ipynb +++ b/components/gcp/dataproc/submit_pig_job/sample.ipynb @@ -84,7 +84,7 @@ "import kfp.components as comp\n", "\n", "dataproc_submit_pig_job_op = comp.load_component_from_url(\n", - " 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/dataproc/submit_pig_job/component.yaml')\n", + " 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/dataproc/submit_pig_job/component.yaml')\n", "help(dataproc_submit_pig_job_op)" ] }, diff --git a/components/gcp/dataproc/submit_pyspark_job/README.md b/components/gcp/dataproc/submit_pyspark_job/README.md index 4db980d4e1c..ab550f7a7c1 100644 --- a/components/gcp/dataproc/submit_pyspark_job/README.md +++ b/components/gcp/dataproc/submit_pyspark_job/README.md @@ -67,7 +67,7 @@ KFP_PACKAGE = 'https://storage.googleapis.com/ml-pipeline/release/0.1.14/kfp.tar import kfp.components as comp dataproc_submit_pyspark_job_op = comp.load_component_from_url( - 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/dataproc/submit_pyspark_job/component.yaml') + 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/dataproc/submit_pyspark_job/component.yaml') help(dataproc_submit_pyspark_job_op) ``` diff --git a/components/gcp/dataproc/submit_pyspark_job/component.yaml b/components/gcp/dataproc/submit_pyspark_job/component.yaml index 000958a6595..30036ea18c0 100644 --- a/components/gcp/dataproc/submit_pyspark_job/component.yaml +++ b/components/gcp/dataproc/submit_pyspark_job/component.yaml @@ -67,7 +67,7 @@ outputs: type: String implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-gcp:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-gcp:151c5349f13bea9d626c988563c04c0a86210c21 args: [ kfp_component.google.dataproc, submit_pyspark_job, --project_id, {inputValue: project_id}, diff --git a/components/gcp/dataproc/submit_pyspark_job/sample.ipynb b/components/gcp/dataproc/submit_pyspark_job/sample.ipynb index be999d5eec1..0500ee12f9b 100644 --- a/components/gcp/dataproc/submit_pyspark_job/sample.ipynb +++ b/components/gcp/dataproc/submit_pyspark_job/sample.ipynb @@ -86,7 +86,7 @@ "import kfp.components as comp\n", "\n", "dataproc_submit_pyspark_job_op = comp.load_component_from_url(\n", - " 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/dataproc/submit_pyspark_job/component.yaml')\n", + " 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/dataproc/submit_pyspark_job/component.yaml')\n", "help(dataproc_submit_pyspark_job_op)" ] }, diff --git a/components/gcp/dataproc/submit_spark_job/README.md b/components/gcp/dataproc/submit_spark_job/README.md index 85b78e3ac05..742384c807a 100644 --- a/components/gcp/dataproc/submit_spark_job/README.md +++ b/components/gcp/dataproc/submit_spark_job/README.md @@ -80,7 +80,7 @@ KFP_PACKAGE = 'https://storage.googleapis.com/ml-pipeline/release/0.1.14/kfp.tar import kfp.components as comp dataproc_submit_spark_job_op = comp.load_component_from_url( - 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/dataproc/submit_spark_job/component.yaml') + 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/dataproc/submit_spark_job/component.yaml') help(dataproc_submit_spark_job_op) ``` diff --git a/components/gcp/dataproc/submit_spark_job/component.yaml b/components/gcp/dataproc/submit_spark_job/component.yaml index d3f47741b3b..d585b97e0af 100644 --- a/components/gcp/dataproc/submit_spark_job/component.yaml +++ b/components/gcp/dataproc/submit_spark_job/component.yaml @@ -74,7 +74,7 @@ outputs: type: String implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-gcp:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-gcp:151c5349f13bea9d626c988563c04c0a86210c21 args: [ kfp_component.google.dataproc, submit_spark_job, --project_id, {inputValue: project_id}, diff --git a/components/gcp/dataproc/submit_spark_job/sample.ipynb b/components/gcp/dataproc/submit_spark_job/sample.ipynb index 677f6a35670..b4344e3fe16 100644 --- a/components/gcp/dataproc/submit_spark_job/sample.ipynb +++ b/components/gcp/dataproc/submit_spark_job/sample.ipynb @@ -99,7 +99,7 @@ "import kfp.components as comp\n", "\n", "dataproc_submit_spark_job_op = comp.load_component_from_url(\n", - " 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/dataproc/submit_spark_job/component.yaml')\n", + " 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/dataproc/submit_spark_job/component.yaml')\n", "help(dataproc_submit_spark_job_op)" ] }, diff --git a/components/gcp/dataproc/submit_sparksql_job/README.md b/components/gcp/dataproc/submit_sparksql_job/README.md index 87f40b8cead..f8ad799e700 100644 --- a/components/gcp/dataproc/submit_sparksql_job/README.md +++ b/components/gcp/dataproc/submit_sparksql_job/README.md @@ -62,7 +62,7 @@ KFP_PACKAGE = 'https://storage.googleapis.com/ml-pipeline/release/0.1.14/kfp.tar import kfp.components as comp dataproc_submit_sparksql_job_op = comp.load_component_from_url( - 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/dataproc/submit_sparksql_job/component.yaml') + 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/dataproc/submit_sparksql_job/component.yaml') help(dataproc_submit_sparksql_job_op) ``` diff --git a/components/gcp/dataproc/submit_sparksql_job/component.yaml b/components/gcp/dataproc/submit_sparksql_job/component.yaml index f599655b001..119e8592d35 100644 --- a/components/gcp/dataproc/submit_sparksql_job/component.yaml +++ b/components/gcp/dataproc/submit_sparksql_job/component.yaml @@ -73,7 +73,7 @@ outputs: type: String implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-gcp:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-gcp:151c5349f13bea9d626c988563c04c0a86210c21 args: [ kfp_component.google.dataproc, submit_sparksql_job, --project_id, {inputValue: project_id}, diff --git a/components/gcp/dataproc/submit_sparksql_job/sample.ipynb b/components/gcp/dataproc/submit_sparksql_job/sample.ipynb index 39c3aa7afec..bfdd7e83a08 100644 --- a/components/gcp/dataproc/submit_sparksql_job/sample.ipynb +++ b/components/gcp/dataproc/submit_sparksql_job/sample.ipynb @@ -81,7 +81,7 @@ "import kfp.components as comp\n", "\n", "dataproc_submit_sparksql_job_op = comp.load_component_from_url(\n", - " 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/dataproc/submit_sparksql_job/component.yaml')\n", + " 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/dataproc/submit_sparksql_job/component.yaml')\n", "help(dataproc_submit_sparksql_job_op)" ] }, diff --git a/components/gcp/ml_engine/batch_predict/README.md b/components/gcp/ml_engine/batch_predict/README.md index 39f2fa28ac4..6020191e4fd 100644 --- a/components/gcp/ml_engine/batch_predict/README.md +++ b/components/gcp/ml_engine/batch_predict/README.md @@ -94,7 +94,7 @@ KFP_PACKAGE = 'https://storage.googleapis.com/ml-pipeline/release/0.1.14/kfp.tar import kfp.components as comp mlengine_batch_predict_op = comp.load_component_from_url( - 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/ml_engine/batch_predict/component.yaml') + 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/ml_engine/batch_predict/component.yaml') help(mlengine_batch_predict_op) ``` diff --git a/components/gcp/ml_engine/batch_predict/component.yaml b/components/gcp/ml_engine/batch_predict/component.yaml index a4a723b1ff4..0a3bbf1fb36 100644 --- a/components/gcp/ml_engine/batch_predict/component.yaml +++ b/components/gcp/ml_engine/batch_predict/component.yaml @@ -67,7 +67,7 @@ outputs: type: String implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-gcp:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-gcp:151c5349f13bea9d626c988563c04c0a86210c21 args: [ kfp_component.google.ml_engine, batch_predict, --project_id, {inputValue: project_id}, diff --git a/components/gcp/ml_engine/batch_predict/sample.ipynb b/components/gcp/ml_engine/batch_predict/sample.ipynb index 41f7b81868a..e8de2f10656 100644 --- a/components/gcp/ml_engine/batch_predict/sample.ipynb +++ b/components/gcp/ml_engine/batch_predict/sample.ipynb @@ -112,7 +112,7 @@ "import kfp.components as comp\n", "\n", "mlengine_batch_predict_op = comp.load_component_from_url(\n", - " 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/ml_engine/batch_predict/component.yaml')\n", + " 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/ml_engine/batch_predict/component.yaml')\n", "help(mlengine_batch_predict_op)" ] }, diff --git a/components/gcp/ml_engine/deploy/README.md b/components/gcp/ml_engine/deploy/README.md index d7a3a4ca57c..c41fabe8bd8 100644 --- a/components/gcp/ml_engine/deploy/README.md +++ b/components/gcp/ml_engine/deploy/README.md @@ -110,7 +110,7 @@ KFP_PACKAGE = 'https://storage.googleapis.com/ml-pipeline/release/0.1.14/kfp.tar import kfp.components as comp mlengine_deploy_op = comp.load_component_from_url( - 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/ml_engine/deploy/component.yaml') + 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/ml_engine/deploy/component.yaml') help(mlengine_deploy_op) ``` diff --git a/components/gcp/ml_engine/deploy/component.yaml b/components/gcp/ml_engine/deploy/component.yaml index 400145bd895..5b4280cefdb 100644 --- a/components/gcp/ml_engine/deploy/component.yaml +++ b/components/gcp/ml_engine/deploy/component.yaml @@ -93,7 +93,7 @@ outputs: type: String implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-gcp:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-gcp:151c5349f13bea9d626c988563c04c0a86210c21 args: [ kfp_component.google.ml_engine, deploy, --model_uri, {inputValue: model_uri}, diff --git a/components/gcp/ml_engine/deploy/sample.ipynb b/components/gcp/ml_engine/deploy/sample.ipynb index 06bafaf47b0..2037bd2416b 100644 --- a/components/gcp/ml_engine/deploy/sample.ipynb +++ b/components/gcp/ml_engine/deploy/sample.ipynb @@ -128,7 +128,7 @@ "import kfp.components as comp\n", "\n", "mlengine_deploy_op = comp.load_component_from_url(\n", - " 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/ml_engine/deploy/component.yaml')\n", + " 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/ml_engine/deploy/component.yaml')\n", "help(mlengine_deploy_op)" ] }, diff --git a/components/gcp/ml_engine/train/README.md b/components/gcp/ml_engine/train/README.md index 93807939b7a..2e843f38f0a 100644 --- a/components/gcp/ml_engine/train/README.md +++ b/components/gcp/ml_engine/train/README.md @@ -86,7 +86,7 @@ KFP_PACKAGE = 'https://storage.googleapis.com/ml-pipeline/release/0.1.14/kfp.tar import kfp.components as comp mlengine_train_op = comp.load_component_from_url( - 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/ml_engine/train/component.yaml') + 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/ml_engine/train/component.yaml') help(mlengine_train_op) ``` diff --git a/components/gcp/ml_engine/train/component.yaml b/components/gcp/ml_engine/train/component.yaml index 8dddf112464..0f6dd63adcc 100644 --- a/components/gcp/ml_engine/train/component.yaml +++ b/components/gcp/ml_engine/train/component.yaml @@ -101,7 +101,7 @@ outputs: type: GCSPath implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-gcp:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-gcp:151c5349f13bea9d626c988563c04c0a86210c21 args: [ kfp_component.google.ml_engine, train, --project_id, {inputValue: project_id}, diff --git a/components/gcp/ml_engine/train/sample.ipynb b/components/gcp/ml_engine/train/sample.ipynb index c6f13849112..c36b84602aa 100644 --- a/components/gcp/ml_engine/train/sample.ipynb +++ b/components/gcp/ml_engine/train/sample.ipynb @@ -104,7 +104,7 @@ "import kfp.components as comp\n", "\n", "mlengine_train_op = comp.load_component_from_url(\n", - " 'https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/gcp/ml_engine/train/component.yaml')\n", + " 'https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/gcp/ml_engine/train/component.yaml')\n", "help(mlengine_train_op)" ] }, diff --git a/components/kubeflow/deployer/component.yaml b/components/kubeflow/deployer/component.yaml index 4cc031608f1..583615f702f 100644 --- a/components/kubeflow/deployer/component.yaml +++ b/components/kubeflow/deployer/component.yaml @@ -11,7 +11,7 @@ inputs: # - {name: Endppoint URI, type: Serving URI, description: 'URI of the deployed prediction service..'} implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-kubeflow-deployer:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-kubeflow-deployer:151c5349f13bea9d626c988563c04c0a86210c21 command: [/bin/deploy.sh] args: [ --model-export-path, {inputValue: Model dir}, diff --git a/components/kubeflow/dnntrainer/component.yaml b/components/kubeflow/dnntrainer/component.yaml index 8bbd0007538..70b5a25a9ee 100644 --- a/components/kubeflow/dnntrainer/component.yaml +++ b/components/kubeflow/dnntrainer/component.yaml @@ -15,7 +15,7 @@ outputs: - {name: Training output dir, type: GCSPath, description: 'GCS or local directory.'} # type: {GCSPath: {path_type: Directory}} implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-kubeflow-tf-trainer:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-kubeflow-tf-trainer:151c5349f13bea9d626c988563c04c0a86210c21 command: [python2, -m, trainer.task] args: [ --transformed-data-dir, {inputValue: Transformed data dir}, diff --git a/components/kubeflow/launcher/kubeflow_tfjob_launcher_op.py b/components/kubeflow/launcher/kubeflow_tfjob_launcher_op.py index 64a524c8a4d..be10cdc8115 100644 --- a/components/kubeflow/launcher/kubeflow_tfjob_launcher_op.py +++ b/components/kubeflow/launcher/kubeflow_tfjob_launcher_op.py @@ -17,7 +17,7 @@ def kubeflow_tfjob_launcher_op(container_image, command, number_of_workers: int, number_of_parameter_servers: int, tfjob_timeout_minutes: int, output_dir=None, step_name='TFJob-launcher'): return dsl.ContainerOp( name = step_name, - image = 'gcr.io/ml-pipeline/ml-pipeline-kubeflow-tf:fe639f41661d8e17fcda64ff8242127620b80ba0', + image = 'gcr.io/ml-pipeline/ml-pipeline-kubeflow-tf:151c5349f13bea9d626c988563c04c0a86210c21', arguments = [ '--workers', number_of_workers, '--pss', number_of_parameter_servers, diff --git a/components/kubeflow/launcher/src/train.template.yaml b/components/kubeflow/launcher/src/train.template.yaml index 46ae538d0a3..4e8d660e72b 100644 --- a/components/kubeflow/launcher/src/train.template.yaml +++ b/components/kubeflow/launcher/src/train.template.yaml @@ -26,7 +26,7 @@ spec: spec: containers: - name: tensorflow - image: gcr.io/ml-pipeline/ml-pipeline-kubeflow-tf-trainer:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-kubeflow-tf-trainer:151c5349f13bea9d626c988563c04c0a86210c21 command: - python - -m @@ -49,7 +49,7 @@ spec: spec: containers: - name: tensorflow - image: gcr.io/ml-pipeline/ml-pipeline-kubeflow-tf-trainer:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-kubeflow-tf-trainer:151c5349f13bea9d626c988563c04c0a86210c21 command: - python - -m @@ -72,7 +72,7 @@ spec: spec: containers: - name: tensorflow - image: gcr.io/ml-pipeline/ml-pipeline-kubeflow-tf-trainer:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-kubeflow-tf-trainer:151c5349f13bea9d626c988563c04c0a86210c21 command: - python - -m diff --git a/components/local/confusion_matrix/component.yaml b/components/local/confusion_matrix/component.yaml index 439d0029913..12d21d15e10 100644 --- a/components/local/confusion_matrix/component.yaml +++ b/components/local/confusion_matrix/component.yaml @@ -9,7 +9,7 @@ inputs: # - {name: Metrics, type: Metrics} implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-local-confusion-matrix:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-local-confusion-matrix:151c5349f13bea9d626c988563c04c0a86210c21 command: [python2, /ml/confusion_matrix.py] args: [ --predictions, {inputValue: Predictions}, diff --git a/components/local/roc/component.yaml b/components/local/roc/component.yaml index 452dbe2a40a..4110ca34ee0 100644 --- a/components/local/roc/component.yaml +++ b/components/local/roc/component.yaml @@ -11,7 +11,7 @@ inputs: # - {name: Metrics, type: Metrics} implementation: container: - image: gcr.io/ml-pipeline/ml-pipeline-local-confusion-matrix:fe639f41661d8e17fcda64ff8242127620b80ba0 + image: gcr.io/ml-pipeline/ml-pipeline-local-confusion-matrix:151c5349f13bea9d626c988563c04c0a86210c21 command: [python2, /ml/roc.py] args: [ --predictions, {inputValue: Predictions dir}, diff --git a/contrib/components/openvino/ovms-deployer/containers/evaluate.py b/contrib/components/openvino/ovms-deployer/containers/evaluate.py index 1f49a0acffd..ad934474439 100755 --- a/contrib/components/openvino/ovms-deployer/containers/evaluate.py +++ b/contrib/components/openvino/ovms-deployer/containers/evaluate.py @@ -59,7 +59,7 @@ def getJpeg(path, size, path_prefix): img = img.astype('float32') img = img.transpose(2,0,1).reshape(1,3,size,size) print(path, img.shape, "; data range:",np.amin(img),":",np.amax(img)) - except e: + except Exception as e: print("Can not read the image file", e) img = None else: @@ -138,4 +138,4 @@ def getJpeg(path, size, path_prefix): json.dump(metrics, f) f.close print("\nOverall accuracy=",matched/i*100,"%") -print("Average latency=",latency,"ms") \ No newline at end of file +print("Average latency=",latency,"ms") diff --git a/frontend/mock-backend/fixed-data.ts b/frontend/mock-backend/fixed-data.ts index 9a4f010f7b8..8f5e273091d 100644 --- a/frontend/mock-backend/fixed-data.ts +++ b/frontend/mock-backend/fixed-data.ts @@ -17,6 +17,7 @@ import helloWorldWithStepsRun from './hello-world-with-steps-runtime'; import coinflipRun from './mock-coinflip-runtime'; import errorRun from './mock-error-runtime'; import xgboostRun from './mock-xgboost-runtime'; +import jsonRun from './json-runtime'; import { ApiExperiment } from '../src/apis/experiment'; import { ApiJob } from '../src/apis/job'; import { ApiPipeline } from '../src/apis/pipeline'; @@ -136,6 +137,7 @@ const jobs: ApiJob[] = [ } ], pipeline_id: pipelines[0].id, + pipeline_name: pipelines[0].name, }, resource_references: [{ key: { @@ -177,6 +179,7 @@ const jobs: ApiJob[] = [ } ], pipeline_id: pipelines[1].id, + pipeline_name: pipelines[1].name, }, resource_references: [{ key: { @@ -221,6 +224,7 @@ const jobs: ApiJob[] = [ } ], pipeline_id: pipelines[2].id, + pipeline_name: pipelines[2].name, }, resource_references: [{ key: { @@ -269,6 +273,7 @@ const runs: ApiRunDetail[] = [ run: { created_at: new Date('2018-03-17T20:58:23.000Z'), description: 'A recursive coinflip run', + finished_at: new Date('2018-03-18T21:01:23.000Z'), id: '3308d0ec-f1b3-4488-a2d3-8ad0f91e88e7', metrics: [ { @@ -290,7 +295,8 @@ const runs: ApiRunDetail[] = [ { name: 'paramName1', value: 'paramVal1' }, { name: 'paramName2', value: 'paramVal2' }, ], - pipeline_id: '8fbe3bd6-a01f-11e8-98d0-529269fb1459', + pipeline_id: pipelines[0].id, + pipeline_name: pipelines[0].name, }, resource_references: [{ key: { @@ -319,6 +325,7 @@ const runs: ApiRunDetail[] = [ run: { created_at: new Date('2018-04-17T21:00:00.000Z'), description: 'A coinflip run with an error. No metrics', + finished_at: new Date('2018-04-17T21:00:33.000Z'), id: '47a3d09c-7db4-4788-ac55-3f8d908574aa', metrics: [], name: 'coinflip-error-nklng2', @@ -327,7 +334,8 @@ const runs: ApiRunDetail[] = [ { name: 'paramName1', value: 'paramVal1' }, { name: 'paramName2', value: 'paramVal2' }, ], - pipeline_id: '8fbe3bd6-a01f-11e8-98d0-529269fb1459', + pipeline_id: pipelines[0].id, + pipeline_name: pipelines[0].name, }, resource_references: [{ key: { @@ -340,6 +348,40 @@ const runs: ApiRunDetail[] = [ status: 'Error', }, }, + { + pipeline_runtime: { + workflow_manifest: JSON.stringify(jsonRun), + }, + run: { + created_at: new Date('2018-05-17T21:58:23.000Z'), + description: 'A simple run with json input', + id: '183ac01f-dc26-4ebf-b817-7b3f96fdc3ac', + metrics: [{ + format: RunMetricFormat.PERCENTAGE, + name: 'accuracy', + node_id: 'json-12abc', + number_value: 0.5423, + }], + name: 'json-12abc', + pipeline_spec: { + parameters: [ + { name: 'paramName1', value: 'paramVal1' }, + { name: 'paramName2', value: 'paramVal2' }, + ], + pipeline_id: pipelines[2].id, + pipeline_name: pipelines[2].name, + }, + resource_references: [{ + key: { + id: 'a4d4f8c6-ce9c-4200-a92e-c48ec759b733', + type: ApiResourceType.EXPERIMENT, + }, + relationship: ApiRelationship.OWNER, + }], + scheduled_at: new Date('2018-05-17T21:58:23.000Z'), + status: 'Running', + } + }, { pipeline_runtime: { workflow_manifest: JSON.stringify(helloWorldRun), @@ -360,7 +402,8 @@ const runs: ApiRunDetail[] = [ { name: 'paramName1', value: 'paramVal1' }, { name: 'paramName2', value: 'paramVal2' }, ], - pipeline_id: '8fbe41b2-a01f-11e8-98d0-529269fb1459', + pipeline_id: pipelines[2].id, + pipeline_name: pipelines[2].name, }, resource_references: [{ key: { @@ -380,6 +423,7 @@ const runs: ApiRunDetail[] = [ run: { created_at: new Date('2018-06-17T22:58:23.000Z'), description: 'A simple hello world run, but with steps. Not attached to any experiment', + finished_at: new Date('2018-06-18T21:00:33.000Z'), id: '21afb688-7597-47e9-b6c3-35d3145fe5e1', metrics: [{ format: RunMetricFormat.PERCENTAGE, @@ -393,7 +437,8 @@ const runs: ApiRunDetail[] = [ { name: 'paramName1', value: 'paramVal1' }, { name: 'paramName2', value: 'paramVal2' }, ], - pipeline_id: '8fbe42f2-a01f-11e8-98d0-529269fb1459', + pipeline_id: pipelines[3].id, + pipeline_name: pipelines[3].name, }, scheduled_at: new Date('2018-06-17T22:58:23.000Z'), status: 'Failed', @@ -439,7 +484,8 @@ const runs: ApiRunDetail[] = [ { name: 'paramName1', value: 'paramVal1' }, { name: 'paramName2', value: 'paramVal2' }, ], - pipeline_id: '8fbe3f78-a01f-11e8-98d0-529269fb1459', + pipeline_id: pipelines[1].id, + pipeline_name: pipelines[1].name, }, resource_references: [{ key: { @@ -466,6 +512,7 @@ const runs: ApiRunDetail[] = [ + ' Ut nec dapibus eros, vitae iaculis nunc. In aliquet accumsan rhoncus. Donec vitae' + ' ipsum a tellus fermentum pharetra in in neque. Pellentesque consequat felis non est' + ' vulputate pellentesque. Aliquam eget cursus enim.', + finished_at: new Date('2018-08-20T21:01:23.000Z'), id: '7fc01714-4a13-4c05-8044-a8a72c14253b', metrics: [ { @@ -488,7 +535,8 @@ const runs: ApiRunDetail[] = [ { name: 'paramName1', value: 'paramVal1' }, { name: 'paramName2', value: 'paramVal2' }, ], - pipeline_id: '8fbe3f78-a01f-11e8-98d0-529269fb1459', + pipeline_id: pipelines[1].id, + pipeline_name: pipelines[1].name, }, resource_references: [{ key: { @@ -508,6 +556,7 @@ const runs: ApiRunDetail[] = [ run: { created_at: new Date('2018-08-18T20:58:23.000Z'), description: 'simple run with pipeline spec embedded in it.', + finished_at: new Date('2018-08-18T21:01:23.000Z'), id: '7fc01715-4a93-4c00-8044-a8a72c14253b', metrics: [ { @@ -575,6 +624,7 @@ function generateNRuns(): ApiRunDetail[] { run: { created_at: new Date('2018-02-12T20:' + padStartTwoZeroes(i.toString()) + ':23.000Z'), description: 'The description of a dummy run', + finished_at: new Date('2018-02-12T20:' + padStartTwoZeroes(((2 * i) % 60).toString()) + ':25.000Z'), id: 'Some-run-id-' + i, metrics: [ { @@ -603,6 +653,7 @@ function generateNRuns(): ApiRunDetail[] { { name: 'paramName2', value: 'paramVal2' }, ], pipeline_id: 'Some-pipeline-id-' + i, + pipeline_name: 'Kubeflow Pipeline number ' + i, }, resource_references: [{ key: { diff --git a/frontend/mock-backend/json-runtime.ts b/frontend/mock-backend/json-runtime.ts new file mode 100644 index 00000000000..f4f8351e107 --- /dev/null +++ b/frontend/mock-backend/json-runtime.ts @@ -0,0 +1,99 @@ +// Copyright 2019 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// tslint:disable:object-literal-sort-keys +export default { + metadata: { + name: 'json-12abc', + generateName: 'json-', + namespace: 'default', + selfLink: '/apis/argoproj.io/v1alpha1/namespaces/default/workflows/json-7sm94', + uid: 'dfc82af5-c5cb-43b1-822b-52487cb872d2', + resourceVersion: '1322', + creationTimestamp: '2018-06-06T00:04:49Z', + labels: { + 'workflows.argoproj.io/completed': 'true', + 'workflows.argoproj.io/phase': 'Succeeded' + } + }, + spec: { + templates: [ + { + name: 'whalesay1', + inputs: {}, + outputs: {}, + metadata: {}, + container: { + name: '', + image: 'docker/whalesay:latest', + command: [ + 'cowsay' + ], + args: [ + '{{workflow.parameters.message}}' + ], + resources: {} + } + } + ], + entrypoint: 'whalesay1', + arguments: { + parameters: [ + { + name: 'message', + value: 'hello world' + } + ] + } + }, + status: { + phase: 'Succeeded', + startedAt: '2018-06-06T00:04:49Z', + finishedAt: '2018-06-06T00:05:23Z', + nodes: { + 'json-12abc': { + id: 'json-12abc', + name: 'json-12abc', + displayName: 'json-12abc', + type: 'Pod', + templateName: 'whalesay1', + phase: 'Succeeded', + startedAt: '2018-06-06T00:04:49Z', + finishedAt: '2018-06-06T00:05:23Z', + inputs: { + parameters: [ + { + name: 'JSON Data', + value: JSON.stringify({ + 'string1': 'a', + 'string2': 'b', + 'number1': 1, + 'number2': 2.2, + 'object': { + 'string': 'a', + 'number': 2 + }, + 'array': [ + 'a', + 'b', + 'c' + ] + }) + } + ] + } + } + } + } +}; diff --git a/frontend/package-lock.json b/frontend/package-lock.json index 1bac6456b88..d6eec28a170 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -34,6 +34,27 @@ } } }, + "@babel/polyfill": { + "version": "7.4.4", + "resolved": "https://registry.npmjs.org/@babel/polyfill/-/polyfill-7.4.4.tgz", + "integrity": "sha512-WlthFLfhQQhh+A2Gn5NSFl0Huxz36x86Jn+E9OW7ibK8edKPq+KLy4apM1yDpQ8kJOVi1OVjpP4vSDLdrI04dg==", + "requires": { + "core-js": "^2.6.5", + "regenerator-runtime": "^0.13.2" + }, + "dependencies": { + "core-js": { + "version": "2.6.9", + "resolved": "https://registry.npmjs.org/core-js/-/core-js-2.6.9.tgz", + "integrity": "sha512-HOpZf6eXmnl7la+cUdMnLvUxKNqLUzJvgIziQ0DiF3JwSImNphIqdGqzj6hIKyX04MmV0poclQ7+wjWvxQyR2A==" + }, + "regenerator-runtime": { + "version": "0.13.3", + "resolved": "https://registry.npmjs.org/regenerator-runtime/-/regenerator-runtime-0.13.3.tgz", + "integrity": "sha512-naKIZz2GQ8JWh///G7L3X6LaQUAMp2lvb1rvwwsURe/VXwD6VMfr+/1NuNw3ag8v2kY1aQ/go5SNn79O9JU7yw==" + } + } + }, "@babel/runtime": { "version": "7.2.0", "resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.2.0.tgz", @@ -139,12 +160,6 @@ "integrity": "sha512-fOM/Jhv51iyugY7KOBZz2ThfT1gwvsGCfWxpLpZDgkGjpEO4Le9cld07OdskikLjDUQJ43dzDaVRSFwQlpdqVg==", "dev": true }, - "@types/codemirror": { - "version": "0.0.60", - "resolved": "https://registry.npmjs.org/@types/codemirror/-/codemirror-0.0.60.tgz", - "integrity": "sha512-fhuwXmN81SeDz/33kFruCcLgEOAz+asgWNkuszUlm5W1YIjn/qEVU+dwV1qC12rJ6duLBTWWumlDaxtckZpFXw==", - "dev": true - }, "@types/connect": { "version": "3.4.32", "resolved": "https://registry.npmjs.org/@types/connect/-/connect-3.4.32.tgz", @@ -2255,6 +2270,11 @@ } } }, + "brace": { + "version": "0.11.1", + "resolved": "https://registry.npmjs.org/brace/-/brace-0.11.1.tgz", + "integrity": "sha1-SJb8ydVE7vRfS7dmDbMg07N5/lg=" + }, "brace-expansion": { "version": "1.1.11", "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.11.tgz", @@ -2998,11 +3018,6 @@ "resolved": "https://registry.npmjs.org/code-point-at/-/code-point-at-1.1.0.tgz", "integrity": "sha1-DQcLTQQ6W+ozovGkDi7bPZpMz3c=" }, - "codemirror": { - "version": "5.42.2", - "resolved": "https://registry.npmjs.org/codemirror/-/codemirror-5.42.2.tgz", - "integrity": "sha512-Tkv6im39VuhduFMsDA3MlXcC/kKas3Z0PI1/8N88QvFQbtOeiiwnfFJE4juGyC8/a4sb1BSxQlzsil8XLQdxRw==" - }, "collection-visit": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/collection-visit/-/collection-visit-1.0.0.tgz", @@ -4105,6 +4120,11 @@ "resolved": "https://registry.npmjs.org/diff/-/diff-3.5.0.tgz", "integrity": "sha512-A46qtFgd+g7pDZinpnwiRJtxbC1hpgf0uzP3iG89scHk0AUC7A1TGxf5OiiOUv/JMZR8GOt8hL900hV0bOy5xA==" }, + "diff-match-patch": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/diff-match-patch/-/diff-match-patch-1.0.4.tgz", + "integrity": "sha512-Uv3SW8bmH9nAtHKaKSanOQmj2DnlH65fUpcrMdfdaOxUG02QQ4YGZ8AE7kKOMisF7UqvOlGKVYWRvezdncW9lg==" + }, "diffie-hellman": { "version": "5.0.3", "resolved": "https://registry.npmjs.org/diffie-hellman/-/diffie-hellman-5.0.3.tgz", @@ -5319,7 +5339,8 @@ }, "ansi-regex": { "version": "2.1.1", - "bundled": true + "bundled": true, + "optional": true }, "aproba": { "version": "1.2.0", @@ -5684,7 +5705,8 @@ }, "safe-buffer": { "version": "5.1.1", - "bundled": true + "bundled": true, + "optional": true }, "safer-buffer": { "version": "2.1.2", @@ -5732,6 +5754,7 @@ "strip-ansi": { "version": "3.0.1", "bundled": true, + "optional": true, "requires": { "ansi-regex": "^2.0.0" } @@ -5770,11 +5793,13 @@ }, "wrappy": { "version": "1.0.2", - "bundled": true + "bundled": true, + "optional": true }, "yallist": { "version": "3.0.2", - "bundled": true + "bundled": true, + "optional": true } } }, @@ -8418,6 +8443,11 @@ "integrity": "sha1-+wMJF/hqMTTlvJvsDWngAT3f7bI=", "dev": true }, + "lodash.get": { + "version": "4.4.2", + "resolved": "https://registry.npmjs.org/lodash.get/-/lodash.get-4.4.2.tgz", + "integrity": "sha1-LRd/ZS+jHpObRDjVNBSZ36OCXpk=" + }, "lodash.isarguments": { "version": "3.1.0", "resolved": "https://registry.npmjs.org/lodash.isarguments/-/lodash.isarguments-3.1.0.tgz", @@ -8433,8 +8463,7 @@ "lodash.isequal": { "version": "4.5.0", "resolved": "https://registry.npmjs.org/lodash.isequal/-/lodash.isequal-4.5.0.tgz", - "integrity": "sha1-QVxEePK8wwEgwizhDtMib30+GOA=", - "dev": true + "integrity": "sha1-QVxEePK8wwEgwizhDtMib30+GOA=" }, "lodash.isfunction": { "version": "3.0.9", @@ -11194,10 +11223,35 @@ "scheduler": "^0.12.0" } }, - "react-codemirror2": { - "version": "5.1.0", - "resolved": "https://registry.npmjs.org/react-codemirror2/-/react-codemirror2-5.1.0.tgz", - "integrity": "sha512-Cksbgbviuf2mJfMyrKmcu7ycK6zX/ukuQO8dvRZdFWqATf5joalhjFc6etnBdGCcPA2LbhIwz+OPnQxLN/j1Fw==" + "react-ace": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/react-ace/-/react-ace-7.0.2.tgz", + "integrity": "sha512-+TFuO1nO6dme/q+qEHjb7iOuWI8jRDzeALs9JyH8HoyHb9+A2bC8WHuJyNU3pmPo8623bytgAgzEJAzDMkzjlw==", + "requires": { + "@babel/polyfill": "^7.4.4", + "brace": "^0.11.1", + "diff-match-patch": "^1.0.4", + "lodash.get": "^4.4.2", + "lodash.isequal": "^4.5.0", + "prop-types": "^15.7.2" + }, + "dependencies": { + "prop-types": { + "version": "15.7.2", + "resolved": "https://registry.npmjs.org/prop-types/-/prop-types-15.7.2.tgz", + "integrity": "sha512-8QQikdH7//R2vurIJSutZ1smHYTcLpRWEOlHnzcWHmBYrOGUysKwSsrC89BCiFj3CbrfJ/nXFdJepOVrY1GCHQ==", + "requires": { + "loose-envify": "^1.4.0", + "object-assign": "^4.1.1", + "react-is": "^16.8.1" + } + }, + "react-is": { + "version": "16.9.0", + "resolved": "https://registry.npmjs.org/react-is/-/react-is-16.9.0.tgz", + "integrity": "sha512-tJBzzzIgnnRfEm046qRcURvwQnZVXmuCbscxUO5RWrGTXpon2d4c8mI0D8WE6ydVIm29JiLB6+RslkIvym9Rjw==" + } + } }, "react-dev-utils": { "version": "5.0.3", diff --git a/frontend/package.json b/frontend/package.json index ee01ec5ea47..4c35fedc052 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -6,7 +6,7 @@ "@material-ui/core": "3.7.1", "@material-ui/icons": "^3.0.1", "@types/js-yaml": "^3.11.2", - "codemirror": "^5.40.2", + "brace": "^0.11.1", "d3": "^5.7.0", "d3-dsv": "^1.0.10", "dagre": "^0.8.2", @@ -18,7 +18,7 @@ "portable-fetch": "^3.0.0", "re-resizable": "^4.9.0", "react": "^16.7.0", - "react-codemirror2": "^5.1.0", + "react-ace": "^7.0.2", "react-dom": "^16.5.2", "react-dropzone": "^5.1.0", "react-router-dom": "^4.3.1", @@ -51,7 +51,6 @@ "vr-test": "ts-node -O '{\"module\": \"commonjs\"}' backstop.ts" }, "devDependencies": { - "@types/codemirror": "0.0.60", "@types/d3": "^5.0.0", "@types/d3-dsv": "^1.0.33", "@types/dagre": "^0.7.40", diff --git a/frontend/src/apis/experiment/api.ts b/frontend/src/apis/experiment/api.ts index 9024ba5e42d..7e85bbca732 100644 --- a/frontend/src/apis/experiment/api.ts +++ b/frontend/src/apis/experiment/api.ts @@ -117,19 +117,19 @@ export interface ApiExperiment { */ export interface ApiListExperimentsResponse { /** - * + * A list of experiments returned. * @type {Array<ApiExperiment>} * @memberof ApiListExperimentsResponse */ experiments?: Array; /** - * + * The total number of experiments for the given query. * @type {number} * @memberof ApiListExperimentsResponse */ total_size?: number; /** - * + * The token to list the next page of experiments. * @type {string} * @memberof ApiListExperimentsResponse */ @@ -169,7 +169,7 @@ export interface ApiStatus { */ export interface ProtobufAny { /** - * A URL/resource name that uniquely identifies the type of the serialized protocol buffer message. This string must contain at least one \"/\" character. The last segment of the URL's path must represent the fully qualified name of the type (as in `path/google.protobuf.Duration`). The name should be in a canonical form (e.g., leading \".\" is not accepted). In practice, teams usually precompile into the binary all types that they expect it to use in the context of Any. However, for URLs which use the scheme `http`, `https`, or no scheme, one can optionally set up a type server that maps type URLs to message definitions as follows: * If no scheme is provided, `https` is assumed. * An HTTP GET on the URL must yield a [google.protobuf.Type][] value in binary format, or produce an error. * Applications are allowed to cache lookup results based on the URL, or have them precompiled into a binary to avoid any lookup. Therefore, binary compatibility needs to be preserved on changes to types. (Use versioned type names to manage breaking changes.) Note: this functionality is not currently available in the official protobuf release, and it is not used for type URLs beginning with type.googleapis.com. Schemes other than `http`, `https` (or the empty scheme) might be used with implementation specific semantics. + * A URL/resource name that uniquely identifies the type of the serialized protocol buffer message. The last segment of the URL's path must represent the fully qualified name of the type (as in `path/google.protobuf.Duration`). The name should be in a canonical form (e.g., leading \".\" is not accepted). In practice, teams usually precompile into the binary all types that they expect it to use in the context of Any. However, for URLs which use the scheme `http`, `https`, or no scheme, one can optionally set up a type server that maps type URLs to message definitions as follows: * If no scheme is provided, `https` is assumed. * An HTTP GET on the URL must yield a [google.protobuf.Type][] value in binary format, or produce an error. * Applications are allowed to cache lookup results based on the URL, or have them precompiled into a binary to avoid any lookup. Therefore, binary compatibility needs to be preserved on changes to types. (Use versioned type names to manage breaking changes.) Note: this functionality is not currently available in the official protobuf release, and it is not used for type URLs beginning with type.googleapis.com. Schemes other than `http`, `https` (or the empty scheme) might be used with implementation specific semantics. * @type {string} * @memberof ProtobufAny */ @@ -191,7 +191,7 @@ export const ExperimentServiceApiFetchParamCreator = function (configuration?: C return { /** * - * @param {ApiExperiment} body + * @param {ApiExperiment} body The experiment to be created * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -230,7 +230,7 @@ export const ExperimentServiceApiFetchParamCreator = function (configuration?: C }, /** * - * @param {string} id + * @param {string} id The ID of the experiment to be deleted. * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -266,7 +266,7 @@ export const ExperimentServiceApiFetchParamCreator = function (configuration?: C }, /** * - * @param {string} id + * @param {string} id The ID of the experiment to be retrieved * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -361,7 +361,7 @@ export const ExperimentServiceApiFp = function(configuration?: Configuration) { return { /** * - * @param {ApiExperiment} body + * @param {ApiExperiment} body The experiment to be created * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -379,7 +379,7 @@ export const ExperimentServiceApiFp = function(configuration?: Configuration) { }, /** * - * @param {string} id + * @param {string} id The ID of the experiment to be deleted. * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -397,7 +397,7 @@ export const ExperimentServiceApiFp = function(configuration?: Configuration) { }, /** * - * @param {string} id + * @param {string} id The ID of the experiment to be retrieved * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -445,7 +445,7 @@ export const ExperimentServiceApiFactory = function (configuration?: Configurati return { /** * - * @param {ApiExperiment} body + * @param {ApiExperiment} body The experiment to be created * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -454,7 +454,7 @@ export const ExperimentServiceApiFactory = function (configuration?: Configurati }, /** * - * @param {string} id + * @param {string} id The ID of the experiment to be deleted. * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -463,7 +463,7 @@ export const ExperimentServiceApiFactory = function (configuration?: Configurati }, /** * - * @param {string} id + * @param {string} id The ID of the experiment to be retrieved * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -494,7 +494,7 @@ export const ExperimentServiceApiFactory = function (configuration?: Configurati export class ExperimentServiceApi extends BaseAPI { /** * - * @param {} body + * @param {} body The experiment to be created * @param {*} [options] Override http request option. * @throws {RequiredError} * @memberof ExperimentServiceApi @@ -505,7 +505,7 @@ export class ExperimentServiceApi extends BaseAPI { /** * - * @param {} id + * @param {} id The ID of the experiment to be deleted. * @param {*} [options] Override http request option. * @throws {RequiredError} * @memberof ExperimentServiceApi @@ -516,7 +516,7 @@ export class ExperimentServiceApi extends BaseAPI { /** * - * @param {} id + * @param {} id The ID of the experiment to be retrieved * @param {*} [options] Override http request option. * @throws {RequiredError} * @memberof ExperimentServiceApi diff --git a/frontend/src/apis/filter/api.ts b/frontend/src/apis/filter/api.ts index e324b0ab50d..65f3eee5cea 100644 --- a/frontend/src/apis/filter/api.ts +++ b/frontend/src/apis/filter/api.ts @@ -5,13 +5,14 @@ * No description provided (generated by Swagger Codegen https://github.com/swagger-api/swagger-codegen) * * OpenAPI spec version: version not set - * + * * * NOTE: This class is auto generated by the swagger code generator program. * https://github.com/swagger-api/swagger-codegen.git * Do not edit the class manually. */ + import * as portableFetch from "portable-fetch"; import { Configuration } from "./configuration"; @@ -38,7 +39,7 @@ export interface FetchAPI { } /** - * + * * @export * @interface FetchArgs */ @@ -48,7 +49,7 @@ export interface FetchArgs { } /** - * + * * @export * @class BaseAPI */ @@ -64,7 +65,7 @@ export class BaseAPI { }; /** - * + * * @export * @class RequiredError * @extends {Error} @@ -91,13 +92,13 @@ export interface ApiFilter { } /** - * + * * @export * @interface ApiIntValues */ export interface ApiIntValues { /** - * + * * @type {Array<number>} * @memberof ApiIntValues */ @@ -105,13 +106,13 @@ export interface ApiIntValues { } /** - * + * * @export * @interface ApiLongValues */ export interface ApiLongValues { /** - * + * * @type {Array<string>} * @memberof ApiLongValues */ @@ -125,31 +126,31 @@ export interface ApiLongValues { */ export interface ApiPredicate { /** - * + * * @type {PredicateOp} * @memberof ApiPredicate */ op?: PredicateOp; /** - * + * * @type {string} * @memberof ApiPredicate */ key?: string; /** - * + * * @type {number} * @memberof ApiPredicate */ int_value?: number; /** - * + * * @type {string} * @memberof ApiPredicate */ long_value?: string; /** - * + * * @type {string} * @memberof ApiPredicate */ @@ -167,13 +168,13 @@ export interface ApiPredicate { */ int_values?: ApiIntValues; /** - * + * * @type {ApiLongValues} * @memberof ApiPredicate */ long_values?: ApiLongValues; /** - * + * * @type {ApiStringValues} * @memberof ApiPredicate */ @@ -181,13 +182,13 @@ export interface ApiPredicate { } /** - * + * * @export * @interface ApiStringValues */ export interface ApiStringValues { /** - * + * * @type {Array<string>} * @memberof ApiStringValues */ diff --git a/frontend/src/apis/job/api.ts b/frontend/src/apis/job/api.ts index c8481583e8f..f278e2b7db5 100644 --- a/frontend/src/apis/job/api.ts +++ b/frontend/src/apis/job/api.ts @@ -183,7 +183,7 @@ export interface ApiJob { */ error?: string; /** - * + * Input. Whether the job is enabled or not. * @type {boolean} * @memberof ApiJob */ @@ -197,7 +197,7 @@ export interface ApiJob { */ export interface ApiListJobsResponse { /** - * + * A list of jobs returned. * @type {Array<ApiJob>} * @memberof ApiListJobsResponse */ @@ -274,6 +274,12 @@ export interface ApiPipelineSpec { * @memberof ApiPipelineSpec */ pipeline_id?: string; + /** + * Optional output field. The name of the pipeline. Not empty if the pipeline id is not empty. + * @type {string} + * @memberof ApiPipelineSpec + */ + pipeline_name?: string; /** * Optional input field. The marshalled raw argo JSON workflow. This will be deprecated when pipeline_manifest is in use. * @type {string} @@ -337,6 +343,12 @@ export interface ApiResourceReference { * @memberof ApiResourceReference */ key?: ApiResourceKey; + /** + * The name of the resource that referred to. + * @type {string} + * @memberof ApiResourceReference + */ + name?: string; /** * Required field. The relationship from referred resource to the object. * @type {ApiRelationship} @@ -383,7 +395,7 @@ export interface ApiStatus { } /** - * + * Trigger defines what starts a pipeline run. * @export * @interface ApiTrigger */ @@ -420,7 +432,7 @@ export enum JobMode { */ export interface ProtobufAny { /** - * A URL/resource name that uniquely identifies the type of the serialized protocol buffer message. This string must contain at least one \"/\" character. The last segment of the URL's path must represent the fully qualified name of the type (as in `path/google.protobuf.Duration`). The name should be in a canonical form (e.g., leading \".\" is not accepted). In practice, teams usually precompile into the binary all types that they expect it to use in the context of Any. However, for URLs which use the scheme `http`, `https`, or no scheme, one can optionally set up a type server that maps type URLs to message definitions as follows: * If no scheme is provided, `https` is assumed. * An HTTP GET on the URL must yield a [google.protobuf.Type][] value in binary format, or produce an error. * Applications are allowed to cache lookup results based on the URL, or have them precompiled into a binary to avoid any lookup. Therefore, binary compatibility needs to be preserved on changes to types. (Use versioned type names to manage breaking changes.) Note: this functionality is not currently available in the official protobuf release, and it is not used for type URLs beginning with type.googleapis.com. Schemes other than `http`, `https` (or the empty scheme) might be used with implementation specific semantics. + * A URL/resource name that uniquely identifies the type of the serialized protocol buffer message. The last segment of the URL's path must represent the fully qualified name of the type (as in `path/google.protobuf.Duration`). The name should be in a canonical form (e.g., leading \".\" is not accepted). In practice, teams usually precompile into the binary all types that they expect it to use in the context of Any. However, for URLs which use the scheme `http`, `https`, or no scheme, one can optionally set up a type server that maps type URLs to message definitions as follows: * If no scheme is provided, `https` is assumed. * An HTTP GET on the URL must yield a [google.protobuf.Type][] value in binary format, or produce an error. * Applications are allowed to cache lookup results based on the URL, or have them precompiled into a binary to avoid any lookup. Therefore, binary compatibility needs to be preserved on changes to types. (Use versioned type names to manage breaking changes.) Note: this functionality is not currently available in the official protobuf release, and it is not used for type URLs beginning with type.googleapis.com. Schemes other than `http`, `https` (or the empty scheme) might be used with implementation specific semantics. * @type {string} * @memberof ProtobufAny */ @@ -442,7 +454,7 @@ export const JobServiceApiFetchParamCreator = function (configuration?: Configur return { /** * - * @param {ApiJob} body + * @param {ApiJob} body The job to be created * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -481,7 +493,7 @@ export const JobServiceApiFetchParamCreator = function (configuration?: Configur }, /** * - * @param {string} id + * @param {string} id The ID of the job to be deleted * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -517,7 +529,7 @@ export const JobServiceApiFetchParamCreator = function (configuration?: Configur }, /** * - * @param {string} id + * @param {string} id The ID of the job to be disabled * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -553,7 +565,7 @@ export const JobServiceApiFetchParamCreator = function (configuration?: Configur }, /** * - * @param {string} id + * @param {string} id The ID of the job to be enabled * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -589,7 +601,7 @@ export const JobServiceApiFetchParamCreator = function (configuration?: Configur }, /** * - * @param {string} id + * @param {string} id The ID of the job to be retrieved * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -694,7 +706,7 @@ export const JobServiceApiFp = function(configuration?: Configuration) { return { /** * - * @param {ApiJob} body + * @param {ApiJob} body The job to be created * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -712,7 +724,7 @@ export const JobServiceApiFp = function(configuration?: Configuration) { }, /** * - * @param {string} id + * @param {string} id The ID of the job to be deleted * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -730,7 +742,7 @@ export const JobServiceApiFp = function(configuration?: Configuration) { }, /** * - * @param {string} id + * @param {string} id The ID of the job to be disabled * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -748,7 +760,7 @@ export const JobServiceApiFp = function(configuration?: Configuration) { }, /** * - * @param {string} id + * @param {string} id The ID of the job to be enabled * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -766,7 +778,7 @@ export const JobServiceApiFp = function(configuration?: Configuration) { }, /** * - * @param {string} id + * @param {string} id The ID of the job to be retrieved * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -816,7 +828,7 @@ export const JobServiceApiFactory = function (configuration?: Configuration, fet return { /** * - * @param {ApiJob} body + * @param {ApiJob} body The job to be created * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -825,7 +837,7 @@ export const JobServiceApiFactory = function (configuration?: Configuration, fet }, /** * - * @param {string} id + * @param {string} id The ID of the job to be deleted * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -834,7 +846,7 @@ export const JobServiceApiFactory = function (configuration?: Configuration, fet }, /** * - * @param {string} id + * @param {string} id The ID of the job to be disabled * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -843,7 +855,7 @@ export const JobServiceApiFactory = function (configuration?: Configuration, fet }, /** * - * @param {string} id + * @param {string} id The ID of the job to be enabled * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -852,7 +864,7 @@ export const JobServiceApiFactory = function (configuration?: Configuration, fet }, /** * - * @param {string} id + * @param {string} id The ID of the job to be retrieved * @param {*} [options] Override http request option. * @throws {RequiredError} */ @@ -885,7 +897,7 @@ export const JobServiceApiFactory = function (configuration?: Configuration, fet export class JobServiceApi extends BaseAPI { /** * - * @param {} body + * @param {} body The job to be created * @param {*} [options] Override http request option. * @throws {RequiredError} * @memberof JobServiceApi @@ -896,7 +908,7 @@ export class JobServiceApi extends BaseAPI { /** * - * @param {} id + * @param {} id The ID of the job to be deleted * @param {*} [options] Override http request option. * @throws {RequiredError} * @memberof JobServiceApi @@ -907,7 +919,7 @@ export class JobServiceApi extends BaseAPI { /** * - * @param {} id + * @param {} id The ID of the job to be disabled * @param {*} [options] Override http request option. * @throws {RequiredError} * @memberof JobServiceApi @@ -918,7 +930,7 @@ export class JobServiceApi extends BaseAPI { /** * - * @param {} id + * @param {} id The ID of the job to be enabled * @param {*} [options] Override http request option. * @throws {RequiredError} * @memberof JobServiceApi @@ -929,7 +941,7 @@ export class JobServiceApi extends BaseAPI { /** * - * @param {} id + * @param {} id The ID of the job to be retrieved * @param {*} [options] Override http request option. * @throws {RequiredError} * @memberof JobServiceApi diff --git a/frontend/src/apis/pipeline/api.ts b/frontend/src/apis/pipeline/api.ts index 2d2ba1b78d4..a11484a2272 100644 --- a/frontend/src/apis/pipeline/api.ts +++ b/frontend/src/apis/pipeline/api.ts @@ -145,37 +145,37 @@ export interface ApiParameter { */ export interface ApiPipeline { /** - * + * Output. Unique pipeline ID. Generated by API server. * @type {string} * @memberof ApiPipeline */ id?: string; /** - * + * Output. The time this pipeline is created. * @type {Date} * @memberof ApiPipeline */ created_at?: Date; /** - * + * Optional input field. Pipeline name provided by user. If not specified, file name is used as pipeline name. * @type {string} * @memberof ApiPipeline */ name?: string; /** - * + * Optional input field. Describing the purpose of the job. * @type {string} * @memberof ApiPipeline */ description?: string; /** - * + * Output. The input parameters for this pipeline. * @type {Array<ApiParameter>} * @memberof ApiPipeline */ parameters?: Array; /** - * + * The URL to the source of the pipeline. This is required when creating the pipeine through CreatePipeline API. * @type {ApiUrl} * @memberof ApiPipeline */ @@ -235,7 +235,7 @@ export interface ApiUrl { */ export interface ProtobufAny { /** - * A URL/resource name that uniquely identifies the type of the serialized protocol buffer message. This string must contain at least one \"/\" character. The last segment of the URL's path must represent the fully qualified name of the type (as in `path/google.protobuf.Duration`). The name should be in a canonical form (e.g., leading \".\" is not accepted). In practice, teams usually precompile into the binary all types that they expect it to use in the context of Any. However, for URLs which use the scheme `http`, `https`, or no scheme, one can optionally set up a type server that maps type URLs to message definitions as follows: * If no scheme is provided, `https` is assumed. * An HTTP GET on the URL must yield a [google.protobuf.Type][] value in binary format, or produce an error. * Applications are allowed to cache lookup results based on the URL, or have them precompiled into a binary to avoid any lookup. Therefore, binary compatibility needs to be preserved on changes to types. (Use versioned type names to manage breaking changes.) Note: this functionality is not currently available in the official protobuf release, and it is not used for type URLs beginning with type.googleapis.com. Schemes other than `http`, `https` (or the empty scheme) might be used with implementation specific semantics. + * A URL/resource name that uniquely identifies the type of the serialized protocol buffer message. The last segment of the URL's path must represent the fully qualified name of the type (as in `path/google.protobuf.Duration`). The name should be in a canonical form (e.g., leading \".\" is not accepted). In practice, teams usually precompile into the binary all types that they expect it to use in the context of Any. However, for URLs which use the scheme `http`, `https`, or no scheme, one can optionally set up a type server that maps type URLs to message definitions as follows: * If no scheme is provided, `https` is assumed. * An HTTP GET on the URL must yield a [google.protobuf.Type][] value in binary format, or produce an error. * Applications are allowed to cache lookup results based on the URL, or have them precompiled into a binary to avoid any lookup. Therefore, binary compatibility needs to be preserved on changes to types. (Use versioned type names to manage breaking changes.) Note: this functionality is not currently available in the official protobuf release, and it is not used for type URLs beginning with type.googleapis.com. Schemes other than `http`, `https` (or the empty scheme) might be used with implementation specific semantics. * @type {string} * @memberof ProtobufAny */ diff --git a/frontend/src/apis/run/api.ts b/frontend/src/apis/run/api.ts index 36a8650ecf6..e75829739e9 100644 --- a/frontend/src/apis/run/api.ts +++ b/frontend/src/apis/run/api.ts @@ -156,6 +156,12 @@ export interface ApiPipelineSpec { * @memberof ApiPipelineSpec */ pipeline_id?: string; + /** + * Optional output field. The name of the pipeline. Not empty if the pipeline id is not empty. + * @type {string} + * @memberof ApiPipelineSpec + */ + pipeline_name?: string; /** * Optional input field. The marshalled raw argo JSON workflow. This will be deprecated when pipeline_manifest is in use. * @type {string} @@ -267,6 +273,12 @@ export interface ApiResourceReference { * @memberof ApiResourceReference */ key?: ApiResourceKey; + /** + * The name of the resource that referred to. + * @type {string} + * @memberof ApiResourceReference + */ + name?: string; /** * Required field. The relationship from referred resource to the object. * @type {ApiRelationship} @@ -341,7 +353,7 @@ export interface ApiRun { */ scheduled_at?: Date; /** - * Output. When this run is finished. + * Output. The time this run is finished. * @type {Date} * @memberof ApiRun */ @@ -838,6 +850,42 @@ export const RunServiceApiFetchParamCreator = function (configuration?: Configur options: localVarRequestOptions, }; }, + /** + * + * @param {string} run_id + * @param {*} [options] Override http request option. + * @throws {RequiredError} + */ + retryRun(run_id: string, options: any = {}): FetchArgs { + // verify required parameter 'run_id' is not null or undefined + if (run_id === null || run_id === undefined) { + throw new RequiredError('run_id','Required parameter run_id was null or undefined when calling retryRun.'); + } + const localVarPath = `/apis/v1beta1/runs/{run_id}/retry` + .replace(`{${"run_id"}}`, encodeURIComponent(String(run_id))); + const localVarUrlObj = url.parse(localVarPath, true); + const localVarRequestOptions = Object.assign({ method: 'POST' }, options); + const localVarHeaderParameter = {} as any; + const localVarQueryParameter = {} as any; + + // authentication Bearer required + if (configuration && configuration.apiKey) { + const localVarApiKeyValue = typeof configuration.apiKey === 'function' + ? configuration.apiKey("authorization") + : configuration.apiKey; + localVarHeaderParameter["authorization"] = localVarApiKeyValue; + } + + localVarUrlObj.query = Object.assign({}, localVarUrlObj.query, localVarQueryParameter, options.query); + // fix override query string Detail: https://stackoverflow.com/a/7517673/1077943 + delete localVarUrlObj.search; + localVarRequestOptions.headers = Object.assign({}, localVarHeaderParameter, options.headers); + + return { + url: url.format(localVarUrlObj), + options: localVarRequestOptions, + }; + }, /** * * @param {string} run_id @@ -1054,6 +1102,24 @@ export const RunServiceApiFp = function(configuration?: Configuration) { }); }; }, + /** + * + * @param {string} run_id + * @param {*} [options] Override http request option. + * @throws {RequiredError} + */ + retryRun(run_id: string, options?: any): (fetch?: FetchAPI, basePath?: string) => Promise { + const localVarFetchArgs = RunServiceApiFetchParamCreator(configuration).retryRun(run_id, options); + return (fetch: FetchAPI = portableFetch, basePath: string = BASE_PATH) => { + return fetch(basePath + localVarFetchArgs.url, localVarFetchArgs.options).then((response) => { + if (response.status >= 200 && response.status < 300) { + return response.json(); + } else { + throw response; + } + }); + }; + }, /** * * @param {string} run_id @@ -1171,6 +1237,15 @@ export const RunServiceApiFactory = function (configuration?: Configuration, fet reportRunMetrics(run_id: string, body: ApiReportRunMetricsRequest, options?: any) { return RunServiceApiFp(configuration).reportRunMetrics(run_id, body, options)(fetch, basePath); }, + /** + * + * @param {string} run_id + * @param {*} [options] Override http request option. + * @throws {RequiredError} + */ + retryRun(run_id: string, options?: any) { + return RunServiceApiFp(configuration).retryRun(run_id, options)(fetch, basePath); + }, /** * * @param {string} run_id @@ -1285,6 +1360,17 @@ export class RunServiceApi extends BaseAPI { return RunServiceApiFp(this.configuration).reportRunMetrics(run_id, body, options)(this.fetch, this.basePath); } + /** + * + * @param {} run_id + * @param {*} [options] Override http request option. + * @throws {RequiredError} + * @memberof RunServiceApi + */ + public retryRun(run_id: string, options?: any) { + return RunServiceApiFp(this.configuration).retryRun(run_id, options)(this.fetch, this.basePath); + } + /** * * @param {} run_id diff --git a/frontend/src/components/DetailsTable.tsx b/frontend/src/components/DetailsTable.tsx index 4d0ce73b8f4..3f43b955c5b 100644 --- a/frontend/src/components/DetailsTable.tsx +++ b/frontend/src/components/DetailsTable.tsx @@ -14,17 +14,16 @@ * limitations under the License. */ -import 'codemirror/lib/codemirror.css'; -import 'codemirror/mode/javascript/javascript.js'; import * as React from 'react'; import { stylesheet } from 'typestyle'; import { color, spacing, commonCss } from '../Css'; -import { UnControlled as CodeMirror } from 'react-codemirror2'; +import Editor from './Editor'; +import 'brace'; +import 'brace/ext/language_tools'; +import 'brace/mode/json'; +import 'brace/theme/github'; export const css = stylesheet({ - codeMirrorGutter: { - width: 6, - }, key: { color: color.strong, flex: '0 0 50%', @@ -57,25 +56,16 @@ export default (props: DetailsTableProps) => { try { const parsedJson = JSON.parse(f[1]); // Nulls, booleans, strings, and numbers can all be parsed as JSON, but we don't care - // about rendering those using CodeMirror. Note that `typeOf null` returns 'object' + // about rendering. Note that `typeOf null` returns 'object' if (parsedJson === null || typeof parsedJson !== 'object') { throw new Error('Parsed JSON was neither an array nor an object. Using default renderer'); } return (
{f[0]} - editor.refresh()} - options={{ - gutters: [css.codeMirrorGutter], - lineWrapping: true, - mode: 'application/json', - readOnly: true, - theme: 'default', - }} - /> +
); } catch (err) { diff --git a/frontend/src/components/Editor.test.tsx b/frontend/src/components/Editor.test.tsx new file mode 100644 index 00000000000..2245fb6ba66 --- /dev/null +++ b/frontend/src/components/Editor.test.tsx @@ -0,0 +1,52 @@ +/* + * Copyright 2019 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import * as React from 'react'; +import { mount } from 'enzyme'; +import Editor from './Editor'; + +/* + These tests mimic https://github.com/securingsincity/react-ace/blob/master/tests/src/ace.spec.js + to ensure that editor properties (placeholder and value) can be properly + tested. +*/ + +describe('Editor', () => { + it('renders without a placeholder and value', () => { + const tree = mount(); + expect(tree.html()).toMatchSnapshot(); + }); + + it('renders with a placeholder', () => { + const placeholder = 'I am a placeholder.'; + const tree = mount(); + expect(tree.html()).toMatchSnapshot(); + }); + + it ('renders a placeholder that contains HTML', () => { + const placeholder = 'I am a placeholder with HTML.'; + const tree = mount(); + expect(tree.html()).toMatchSnapshot(); + }); + + it('has its value set to the provided value', () => { + const value = 'I am a value.'; + const tree = mount(); + expect(tree).not.toBeNull(); + const editor = (tree.instance() as any).editor; + expect(editor.getValue()).toBe(value); + }); +}); \ No newline at end of file diff --git a/frontend/src/components/Editor.tsx b/frontend/src/components/Editor.tsx new file mode 100644 index 00000000000..11dec1c990d --- /dev/null +++ b/frontend/src/components/Editor.tsx @@ -0,0 +1,45 @@ +/* + * Copyright 2019 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import AceEditor from 'react-ace'; + +// Modified AceEditor that supports HTML within provided placeholder. This is +// important because it allows for the usage of multi-line placeholders. +class Editor extends AceEditor { + public updatePlaceholder(): void { + const editor = this.editor; + const { placeholder } = this.props; + + const showPlaceholder = !editor.session.getValue().length; + let node = editor.renderer.placeholderNode; + if (!showPlaceholder && node) { + editor.renderer.scroller.removeChild(editor.renderer.placeholderNode); + editor.renderer.placeholderNode = null; + } else if (showPlaceholder && !node) { + node = editor.renderer.placeholderNode = document.createElement('div'); + node.innerHTML = placeholder || ''; + node.className = 'ace_comment ace_placeholder'; + node.style.padding = '0 9px'; + node.style.position = 'absolute'; + node.style.zIndex = '3'; + editor.renderer.scroller.appendChild(node); + } else if (showPlaceholder && node) { + node.innerHTML = placeholder; + } + } +} + +export default Editor; \ No newline at end of file diff --git a/frontend/src/components/__snapshots__/DetailsTable.test.tsx.snap b/frontend/src/components/__snapshots__/DetailsTable.test.tsx.snap index 6ba6b42d71f..fd6e8640512 100644 --- a/frontend/src/components/__snapshots__/DetailsTable.test.tsx.snap +++ b/frontend/src/components/__snapshots__/DetailsTable.test.tsx.snap @@ -115,21 +115,44 @@ exports[`DetailsTable does render arrays as JSON 1`] = ` > key - @@ -148,21 +171,44 @@ exports[`DetailsTable does render arrays as JSON 2`] = ` > key - @@ -208,25 +254,48 @@ exports[`DetailsTable shows key and JSON value in row 1`] = ` > key - @@ -312,25 +381,48 @@ exports[`DetailsTable shows keys and values for multiple rows 1`] = ` > key2 -
key5 -
I am a placeholder with HTML.
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
"`; + +exports[`Editor renders with a placeholder 1`] = `"
I am a placeholder.
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
"`; + +exports[`Editor renders without a placeholder and value 1`] = `"
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
"`; diff --git a/frontend/src/lib/Buttons.ts b/frontend/src/lib/Buttons.ts index b321f72e5a0..df8956d6b85 100644 --- a/frontend/src/lib/Buttons.ts +++ b/frontend/src/lib/Buttons.ts @@ -28,6 +28,7 @@ export enum ButtonKeys { ARCHIVE = 'archive', CLONE_RUN = 'cloneRun', CLONE_RECURRING_RUN = 'cloneRecurringRun', + RETRY = 'retry', COLLAPSE = 'collapse', COMPARE = 'compare', DELETE_RUN = 'deleteRun', @@ -99,6 +100,19 @@ export default class Buttons { return this; } + public retryRun(getSelectedIds: () => string[], useCurrentResource: boolean, + callback: (selectedIds: string[], success: boolean) => void): Buttons { + this._map[ButtonKeys.RETRY] = { + action: () => this._retryRun(getSelectedIds(), useCurrentResource, callback), + disabled: !useCurrentResource, + disabledTitle: useCurrentResource ? undefined : 'Select at least one resource to retry', + id: 'retryBtn', + title: 'Retry', + tooltip: 'Retry', + }; + return this; + } + public collapseSections(action: () => void): Buttons { this._map[ButtonKeys.COLLAPSE] = { action, @@ -292,6 +306,19 @@ export default class Buttons { } } + private _retryRun(selectedIds: string[], useCurrent: boolean, + callback: (selectedIds: string[], success: boolean) => void): void { + this._dialogActionHandler( + selectedIds, + 'Retry this run?', + useCurrent, + id => Apis.runServiceApi.retryRun(id), + callback, + 'Retry', + 'run' + ); + } + private _archive(selectedIds: string[], useCurrent: boolean, callback: (selectedIds: string[], success: boolean) => void): void { this._dialogActionHandler( diff --git a/frontend/src/lib/RunUtils.ts b/frontend/src/lib/RunUtils.ts index b7c26ccef15..eb696df8378 100644 --- a/frontend/src/lib/RunUtils.ts +++ b/frontend/src/lib/RunUtils.ts @@ -29,6 +29,11 @@ export interface MetricMetadata { name: string; } +export interface ExperimentInfo { + displayName?: string; + id: string; +} + function getParametersFromRun(run: ApiRunDetail): ApiParameter[] { return getParametersFromRuntime(run.pipeline_runtime); } @@ -51,20 +56,26 @@ function getPipelineId(run?: ApiRun | ApiJob): string | null { return (run && run.pipeline_spec && run.pipeline_spec.pipeline_id) || null; } -function getPipelineSpec(run?: ApiRun | ApiJob): string | null { +function getPipelineName(run?: ApiRun | ApiJob): string | null { + return (run && run.pipeline_spec && run.pipeline_spec.pipeline_name) || null; +} + +function getWorkflowManifest(run?: ApiRun | ApiJob): string | null { return (run && run.pipeline_spec && run.pipeline_spec.workflow_manifest) || null; } +function getFirstExperimentReference(run?: ApiRun | ApiJob): ApiResourceReference | null { + return getAllExperimentReferences(run)[0] || null; +} + function getFirstExperimentReferenceId(run?: ApiRun | ApiJob): string | null { - if (run) { - const reference = getAllExperimentReferences(run)[0]; - return reference && reference.key && reference.key.id || null; - } - return null; + const reference = getFirstExperimentReference(run); + return reference && reference.key && reference.key.id || null; } -function getFirstExperimentReference(run?: ApiRun | ApiJob): ApiResourceReference | null { - return run && getAllExperimentReferences(run)[0] || null; +function getFirstExperimentReferenceName(run?: ApiRun | ApiJob): string | null { + const reference = getFirstExperimentReference(run); + return reference && reference.name || null; } function getAllExperimentReferences(run?: ApiRun | ApiJob): ApiResourceReference[] { @@ -123,15 +134,18 @@ function getRecurringRunId(run?: ApiRun): string { return ''; } +// TODO: This file needs tests export default { extractMetricMetadata, getAllExperimentReferences, getFirstExperimentReference, getFirstExperimentReferenceId, + getFirstExperimentReferenceName, getParametersFromRun, getParametersFromRuntime, getPipelineId, - getPipelineSpec, + getPipelineName, getRecurringRunId, + getWorkflowManifest, runsToMetricMetadataMap, }; diff --git a/frontend/src/lib/Utils.test.ts b/frontend/src/lib/Utils.test.ts index 83760a5aab8..d61751b9a95 100644 --- a/frontend/src/lib/Utils.test.ts +++ b/frontend/src/lib/Utils.test.ts @@ -114,7 +114,7 @@ describe('Utils', () => { it('computes seconds', () => { const run = { - created_at: new Date(2018, 1, 2, 3, 55, 30).toISOString(), + created_at: new Date(2018, 1, 3, 3, 55, 30).toISOString(), finished_at: new Date(2018, 1, 3, 3, 56, 25).toISOString(), status: NodePhase.SUCCEEDED, } as any; @@ -123,29 +123,29 @@ describe('Utils', () => { it('computes minutes/seconds', () => { const run = { - created_at: new Date(2018, 1, 2, 3, 55, 10).toISOString(), + created_at: new Date(2018, 1, 3, 3, 55, 10).toISOString(), finished_at: new Date(2018, 1, 3, 3, 59, 25).toISOString(), status: NodePhase.SUCCEEDED, } as any; expect(getRunDuration(run)).toBe('0:04:15'); }); - it('computes days/minutes/seconds', () => { + it('computes hours/minutes/seconds', () => { const run = { created_at: new Date(2018, 1, 2, 3, 55, 10).toISOString(), finished_at: new Date(2018, 1, 3, 4, 55, 10).toISOString(), status: NodePhase.SUCCEEDED, } as any; - expect(getRunDuration(run)).toBe('1:00:00'); + expect(getRunDuration(run)).toBe('25:00:00'); }); - it('computes padded days/minutes/seconds', () => { + it('computes padded hours/minutes/seconds', () => { const run = { created_at: new Date(2018, 1, 2, 3, 55, 10).toISOString(), finished_at: new Date(2018, 1, 3, 4, 56, 11).toISOString(), status: NodePhase.SUCCEEDED, } as any; - expect(getRunDuration(run)).toBe('1:01:01'); + expect(getRunDuration(run)).toBe('25:01:01'); }); it('shows negative sign if start date is after end date', () => { @@ -190,7 +190,7 @@ describe('Utils', () => { const workflow = { status: { finishedAt: new Date(2018, 1, 3, 3, 56, 25).toISOString(), - startedAt: new Date(2018, 1, 2, 3, 55, 30).toISOString(), + startedAt: new Date(2018, 1, 3, 3, 55, 30).toISOString(), } } as any; expect(getRunDurationFromWorkflow(workflow)).toBe('0:00:55'); @@ -200,27 +200,27 @@ describe('Utils', () => { const workflow = { status: { finishedAt: new Date(2018, 1, 3, 3, 59, 25).toISOString(), - startedAt: new Date(2018, 1, 2, 3, 55, 10).toISOString(), + startedAt: new Date(2018, 1, 3, 3, 55, 10).toISOString(), } } as any; expect(getRunDurationFromWorkflow(workflow)).toBe('0:04:15'); }); - it('computes days/minutes/seconds run time if status is provided', () => { + it('computes hours/minutes/seconds run time if status is provided', () => { const workflow = { status: { finishedAt: new Date(2018, 1, 3, 4, 55, 10).toISOString(), - startedAt: new Date(2018, 1, 2, 3, 55, 10).toISOString(), + startedAt: new Date(2018, 1, 3, 3, 55, 10).toISOString(), } } as any; expect(getRunDurationFromWorkflow(workflow)).toBe('1:00:00'); }); - it('computes padded days/minutes/seconds run time if status is provided', () => { + it('computes padded hours/minutes/seconds run time if status is provided', () => { const workflow = { status: { finishedAt: new Date(2018, 1, 3, 4, 56, 11).toISOString(), - startedAt: new Date(2018, 1, 2, 3, 55, 10).toISOString(), + startedAt: new Date(2018, 1, 3, 3, 55, 10).toISOString(), } } as any; expect(getRunDurationFromWorkflow(workflow)).toBe('1:01:01'); diff --git a/frontend/src/lib/Utils.ts b/frontend/src/lib/Utils.ts index fde46328a15..d6e5fdfdf51 100644 --- a/frontend/src/lib/Utils.ts +++ b/frontend/src/lib/Utils.ts @@ -71,7 +71,7 @@ function getDuration(start: Date, end: Date): string { const seconds = ('0' + Math.floor((diff / SECOND) % 60).toString()).slice(-2); const minutes = ('0' + Math.floor((diff / MINUTE) % 60).toString()).slice(-2); // Hours are the largest denomination, so we don't pad them - const hours = Math.floor((diff / HOUR) % 24).toString(); + const hours = Math.floor(diff / HOUR).toString(); return `${sign}${hours}:${minutes}:${seconds}`; } diff --git a/frontend/src/lib/WorkflowParser.ts b/frontend/src/lib/WorkflowParser.ts index 60d5bee1357..fa060f811af 100644 --- a/frontend/src/lib/WorkflowParser.ts +++ b/frontend/src/lib/WorkflowParser.ts @@ -120,8 +120,11 @@ export default class WorkflowParser { Object.keys(workflowNodes) .forEach((nodeId) => { if (workflowNodes[nodeId].children) { - workflowNodes[nodeId].children.forEach((childNodeId) => - g.setEdge(nodeId, childNodeId)); + workflowNodes[nodeId].children.forEach((childNodeId) => { + if (workflowNodes[childNodeId]) { + g.setEdge(nodeId, childNodeId); + } + }); } }); @@ -130,6 +133,7 @@ export default class WorkflowParser { .forEach((nodeId) => { // Many nodes have the Argo root node as a boundaryID, and we can discard these. if (workflowNodes[nodeId].boundaryID && + workflowNodes[workflowNodes[nodeId].boundaryID] && (!g.inEdges(nodeId) || !g.inEdges(nodeId)!.length) && workflowNodes[nodeId].boundaryID !== workflowName) { // BoundaryIDs point from children to parents. @@ -339,4 +343,4 @@ export default class WorkflowParser { return ''; } } -} +} \ No newline at end of file diff --git a/frontend/src/pages/ExperimentDetails.tsx b/frontend/src/pages/ExperimentDetails.tsx index defc7cc320a..3f0d87c12ea 100644 --- a/frontend/src/pages/ExperimentDetails.tsx +++ b/frontend/src/pages/ExperimentDetails.tsx @@ -222,7 +222,11 @@ class ExperimentDetails extends Page<{}, ExperimentDetailsState> { } public async refresh(): Promise { - return this.load(); + await this.load(); + if (this._runlistRef.current) { + await this._runlistRef.current.refresh(); + } + return; } public async componentDidMount(): Promise { @@ -272,10 +276,6 @@ class ExperimentDetails extends Page<{}, ExperimentDetailsState> { await this.showPageError(`Error: failed to retrieve experiment: ${experimentId}.`, err); logger.error(`Error loading experiment: ${experimentId}`, err); } - - if (this._runlistRef.current) { - this._runlistRef.current.refresh(); - } } private _selectionChanged(selectedIds: string[]): void { diff --git a/frontend/src/pages/NewRun.tsx b/frontend/src/pages/NewRun.tsx index 0bc7e8f50a0..ef5a1d5607b 100644 --- a/frontend/src/pages/NewRun.tsx +++ b/frontend/src/pages/NewRun.tsx @@ -489,7 +489,7 @@ class NewRun extends Page<{}, NewRunState> { try { runWithEmbeddedPipeline = await Apis.runServiceApi.getRun(embeddedPipelineRunId); - embeddedPipelineSpec = RunUtils.getPipelineSpec(runWithEmbeddedPipeline.run); + embeddedPipelineSpec = RunUtils.getWorkflowManifest(runWithEmbeddedPipeline.run); } catch (err) { await this.showPageError( `Error: failed to retrieve the specified run: ${embeddedPipelineRunId}.`, err); @@ -538,7 +538,7 @@ class NewRun extends Page<{}, NewRunState> { const referencePipelineId = RunUtils.getPipelineId(originalRun); // This corresponds to a run where the pipeline has not been uploaded, such as runs started from // the CLI or notebooks - const embeddedPipelineSpec = RunUtils.getPipelineSpec(originalRun); + const embeddedPipelineSpec = RunUtils.getWorkflowManifest(originalRun); if (referencePipelineId) { try { pipeline = await Apis.pipelineServiceApi.getPipeline(referencePipelineId); diff --git a/frontend/src/pages/PipelineDetails.tsx b/frontend/src/pages/PipelineDetails.tsx index 15f8642a6ab..c39f02fcb87 100644 --- a/frontend/src/pages/PipelineDetails.tsx +++ b/frontend/src/pages/PipelineDetails.tsx @@ -14,8 +14,6 @@ * limitations under the License. */ -import 'codemirror/lib/codemirror.css'; -import 'codemirror/mode/yaml/yaml.js'; import * as JsYaml from 'js-yaml'; import * as React from 'react'; import * as StaticGraphParser from '../lib/StaticGraphParser'; @@ -35,11 +33,15 @@ import { Page } from './Page'; import { RoutePage, RouteParams, QUERY_PARAMS } from '../components/Router'; import { ToolbarProps } from '../components/Toolbar'; import { URLParser } from '../lib/URLParser'; -import { UnControlled as CodeMirror } from 'react-codemirror2'; import { Workflow } from '../../third_party/argo-ui/argo_template'; import { classes, stylesheet } from 'typestyle'; +import Editor from '../components/Editor'; import { color, commonCss, padding, fontsize, fonts, zIndex } from '../Css'; import { logger, formatDateString } from '../lib/Utils'; +import 'brace'; +import 'brace/ext/language_tools'; +import 'brace/mode/yaml'; +import 'brace/theme/github'; interface PipelineDetailsState { graph?: dagre.graphlib.Graph; @@ -67,6 +69,7 @@ export const css = stylesheet({ }, }, background: '#f7f7f7', + height: '100%', }, footer: { background: color.graphBg, @@ -213,16 +216,11 @@ class PipelineDetails extends Page<{}, PipelineDetailsState> { } {selectedTab === 1 && !!templateString &&
- editor.refresh()} - options={{ - lineNumbers: true, - lineWrapping: true, - mode: 'text/yaml', - readOnly: true, - theme: 'default', - }} + height='100%' width='100%' mode='yaml' theme='github' + editorProps={{ $blockScrolling: true }} + readOnly={true} highlightActiveLine={true} showGutter={true} />
} @@ -258,7 +256,7 @@ class PipelineDetails extends Page<{}, PipelineDetailsState> { // Convert the run's pipeline spec to YAML to be displayed as the pipeline's source. try { - const pipelineSpec = JSON.parse(RunUtils.getPipelineSpec(runDetails.run) || '{}'); + const pipelineSpec = JSON.parse(RunUtils.getWorkflowManifest(runDetails.run) || '{}'); try { templateString = JsYaml.safeDump(pipelineSpec); } catch (err) { diff --git a/frontend/src/pages/RunDetails.tsx b/frontend/src/pages/RunDetails.tsx index ed6adc81aa0..de1a4d1687f 100644 --- a/frontend/src/pages/RunDetails.tsx +++ b/frontend/src/pages/RunDetails.tsx @@ -152,6 +152,9 @@ class RunDetails extends Page { const buttons = new Buttons(this.props, this.refresh.bind(this)); return { actions: buttons + .retryRun( + () => this.state.runMetadata? [this.state.runMetadata!.id!] : [], + true, () => this.retry()) .cloneRun(() => this.state.runMetadata ? [this.state.runMetadata!.id!] : [], true) .terminateRun( () => this.state.runMetadata ? [this.state.runMetadata!.id!] : [], @@ -375,6 +378,15 @@ class RunDetails extends Page { this.clearBanner(); } + public async retry(): Promise{ + const runFinished = false; + this.setStateSafe({ + runFinished, + }); + + await this._startAutoRefresh(); + } + public async refresh(): Promise { await this.load(); } @@ -458,7 +470,9 @@ class RunDetails extends Page { buttons.archive(idGetter, true, () => this.refresh()); const actions = buttons.getToolbarActionMap(); actions[ButtonKeys.TERMINATE_RUN].disabled = - (runMetadata.status as NodePhase) === NodePhase.TERMINATING || runFinished; + (runMetadata.status as NodePhase) === NodePhase.TERMINATING || runFinished; + actions[ButtonKeys.RETRY].disabled = + (runMetadata.status as NodePhase) !== NodePhase.FAILED && (runMetadata.status as NodePhase) !== NodePhase.ERROR ; this.props.updateToolbar({ actions, breadcrumbs, pageTitle, pageTitleTooltip: runMetadata.name }); this.setStateSafe({ diff --git a/frontend/src/pages/RunList.test.tsx b/frontend/src/pages/RunList.test.tsx index 1bdc2bd9f82..0d7b5af9b87 100644 --- a/frontend/src/pages/RunList.test.tsx +++ b/frontend/src/pages/RunList.test.tsx @@ -310,7 +310,16 @@ describe('RunList', () => { }); it('shows pipeline name', async () => { - mockNRuns(1, { run: { pipeline_spec: { pipeline_id: 'test-pipeline-id' } } }); + mockNRuns(1, { run: { pipeline_spec: { pipeline_id: 'test-pipeline-id', pipeline_name: 'pipeline name' } } }); + const props = generateProps(); + tree = shallow(); + await (tree.instance() as RunListTest)._loadRuns({}); + expect(props.onError).not.toHaveBeenCalled(); + expect(tree).toMatchSnapshot(); + }); + + it('retrieves pipeline from backend to display name if not in spec', async () => { + mockNRuns(1, { run: { pipeline_spec: { pipeline_id: 'test-pipeline-id' /* no pipeline_name */ } } }); getPipelineSpy.mockImplementationOnce(() => ({ name: 'test pipeline' })); const props = generateProps(); tree = shallow(); @@ -372,19 +381,19 @@ describe('RunList', () => { }); it('renders pipeline name as link to its details page', () => { - expect(getMountedInstance()._pipelineCustomRenderer({ value: { displayName: 'test pipeline', id: 'pipeline-id', showLink: false }, id: 'run-id' })).toMatchSnapshot(); + expect(getMountedInstance()._pipelineCustomRenderer({ value: { displayName: 'test pipeline', id: 'pipeline-id', usePlaceholder: false }, id: 'run-id' })).toMatchSnapshot(); }); it('handles no pipeline id given', () => { - expect(getMountedInstance()._pipelineCustomRenderer({ value: { displayName: 'test pipeline', showLink: false }, id: 'run-id' })).toMatchSnapshot(); + expect(getMountedInstance()._pipelineCustomRenderer({ value: { displayName: 'test pipeline', usePlaceholder: false }, id: 'run-id' })).toMatchSnapshot(); }); it('shows "View pipeline" button if pipeline is embedded in run', () => { - expect(getMountedInstance()._pipelineCustomRenderer({ value: { displayName: 'test pipeline', id: 'pipeline-id', showLink: true }, id: 'run-id' })).toMatchSnapshot(); + expect(getMountedInstance()._pipelineCustomRenderer({ value: { displayName: 'test pipeline', id: 'pipeline-id', usePlaceholder: true }, id: 'run-id' })).toMatchSnapshot(); }); it('handles no pipeline name', () => { - expect(getMountedInstance()._pipelineCustomRenderer({ value: { /* no displayName */ showLink: true }, id: 'run-id' })).toMatchSnapshot(); + expect(getMountedInstance()._pipelineCustomRenderer({ value: { /* no displayName */ usePlaceholder: true }, id: 'run-id' })).toMatchSnapshot(); }); it('renders pipeline name as link to its details page', () => { diff --git a/frontend/src/pages/RunList.tsx b/frontend/src/pages/RunList.tsx index 711a02cb664..4f1bff25e3e 100644 --- a/frontend/src/pages/RunList.tsx +++ b/frontend/src/pages/RunList.tsx @@ -17,7 +17,7 @@ import * as React from 'react'; import CustomTable, { Column, Row, CustomRendererProps } from '../components/CustomTable'; import Metric from '../components/Metric'; -import RunUtils, { MetricMetadata } from '../../src/lib/RunUtils'; +import RunUtils, { MetricMetadata, ExperimentInfo } from '../../src/lib/RunUtils'; import { ApiRun, ApiResourceType, ApiRunMetric, RunStorageState, ApiRunDetail } from '../../src/apis/run'; import { Apis, RunSortKeys, ListRequest } from '../lib/Apis'; import { Link, RouteComponentProps } from 'react-router-dom'; @@ -29,16 +29,11 @@ import { commonCss, color } from '../Css'; import { formatDateString, logger, errorToMessage, getRunDuration } from '../lib/Utils'; import { statusToIcon } from './Status'; -interface ExperimentInfo { - displayName?: string; - id: string; -} - interface PipelineInfo { displayName?: string; id?: string; runId?: string; - showLink: boolean; + usePlaceholder: boolean; } interface RecurringRunInfo { @@ -191,17 +186,17 @@ class RunList extends React.PureComponent { public _pipelineCustomRenderer: React.FC> = (props: CustomRendererProps) => { // If the getPipeline call failed or a run has no pipeline, we display a placeholder. - if (!props.value || (!props.value.showLink && !props.value.id)) { + if (!props.value || (!props.value.usePlaceholder && !props.value.id)) { return
-
; } const search = new URLParser(this.props).build({ [QUERY_PARAMS.fromRunId]: props.id }); - const url = props.value.showLink ? + const url = props.value.usePlaceholder ? RoutePage.PIPELINE_DETAILS.replace(':' + RouteParams.pipelineId + '?', '') + search : RoutePage.PIPELINE_DETAILS.replace(':' + RouteParams.pipelineId, props.value.id || ''); return ( e.stopPropagation()} to={url}> - {props.value.showLink ? '[View pipeline]' : props.value.displayName} + {props.value.usePlaceholder ? '[View pipeline]' : props.value.displayName} ); } @@ -357,15 +352,23 @@ class RunList extends React.PureComponent { private async _getAndSetPipelineNames(displayRun: DisplayRun): Promise { const pipelineId = RunUtils.getPipelineId(displayRun.run); if (pipelineId) { - try { - const pipeline = await Apis.pipelineServiceApi.getPipeline(pipelineId); - displayRun.pipeline = { displayName: pipeline.name || '', id: pipelineId, showLink: false }; - } catch (err) { - // This could be an API exception, or a JSON parse exception. - displayRun.error = 'Failed to get associated pipeline: ' + await errorToMessage(err); + let pipelineName = RunUtils.getPipelineName(displayRun.run); + if (!pipelineName) { + try { + const pipeline = await Apis.pipelineServiceApi.getPipeline(pipelineId); + pipelineName = pipeline.name || ''; + } catch (err) { + displayRun.error = 'Failed to get associated pipeline: ' + await errorToMessage(err); + return; + } } - } else if (!!RunUtils.getPipelineSpec(displayRun.run)) { - displayRun.pipeline = { showLink: true }; + displayRun.pipeline = { + displayName: pipelineName, + id: pipelineId, + usePlaceholder: false + }; + } else if (!!RunUtils.getWorkflowManifest(displayRun.run)) { + displayRun.pipeline = { usePlaceholder: true }; } } @@ -378,13 +381,20 @@ class RunList extends React.PureComponent { private async _getAndSetExperimentNames(displayRun: DisplayRun): Promise { const experimentId = RunUtils.getFirstExperimentReferenceId(displayRun.run); if (experimentId) { - try { - const experiment = await Apis.experimentServiceApi.getExperiment(experimentId); - displayRun.experiment = { displayName: experiment.name || '', id: experimentId }; - } catch (err) { - // This could be an API exception, or a JSON parse exception. - displayRun.error = 'Failed to get associated experiment: ' + await errorToMessage(err); + let experimentName = RunUtils.getFirstExperimentReferenceName(displayRun.run); + if (!experimentName) { + try { + const experiment = await Apis.experimentServiceApi.getExperiment(experimentId); + experimentName = experiment.name || ''; + } catch (err) { + displayRun.error = 'Failed to get associated experiment: ' + await errorToMessage(err); + return; + } } + displayRun.experiment = { + displayName: experimentName, + id: experimentId + }; } } } diff --git a/frontend/src/pages/__snapshots__/PipelineDetails.test.tsx.snap b/frontend/src/pages/__snapshots__/PipelineDetails.test.tsx.snap index f086dcaa941..f31dbf4e947 100644 --- a/frontend/src/pages/__snapshots__/PipelineDetails.test.tsx.snap +++ b/frontend/src/pages/__snapshots__/PipelineDetails.test.tsx.snap @@ -780,18 +780,48 @@ exports[`PipelineDetails shows pipeline source code when config tab is clicked 1
-
diff --git a/frontend/src/pages/__snapshots__/RunDetails.test.tsx.snap b/frontend/src/pages/__snapshots__/RunDetails.test.tsx.snap index 0a34a052fa5..c5c8bbe1f31 100644 --- a/frontend/src/pages/__snapshots__/RunDetails.test.tsx.snap +++ b/frontend/src/pages/__snapshots__/RunDetails.test.tsx.snap @@ -1835,7 +1835,7 @@ exports[`RunDetails shows run config fields - handles no description 1`] = ` ], Array [ "Duration", - "1:01:01", + "25:01:01", ], ] } @@ -1896,7 +1896,7 @@ exports[`RunDetails shows run config fields - handles no metadata 1`] = ` ], Array [ "Duration", - "1:01:01", + "25:01:01", ], ] } @@ -1957,7 +1957,7 @@ exports[`RunDetails shows run config fields 1`] = ` ], Array [ "Duration", - "1:01:01", + "25:01:01", ], ] } diff --git a/frontend/src/pages/__snapshots__/RunList.test.tsx.snap b/frontend/src/pages/__snapshots__/RunList.test.tsx.snap index 074546ab29a..e80c1cb38ab 100644 --- a/frontend/src/pages/__snapshots__/RunList.test.tsx.snap +++ b/frontend/src/pages/__snapshots__/RunList.test.tsx.snap @@ -7745,6 +7745,77 @@ exports[`RunList renders the empty experience 1`] = ` `; +exports[`RunList retrieves pipeline from backend to display name if not in spec 1`] = ` +
+ +
+`; + exports[`RunList shows "View pipeline" button if pipeline is embedded in run 1`] = ` :9000` op = dsl.ContainerOp(name="foo", image="busybox:%s" % tag) + +if __name__ == '__main__': + kfp.compiler.Compiler().compile(custom_artifact_location, __file__ + '.zip') diff --git a/samples/core/condition/condition.py b/samples/core/condition/condition.py index 62125c20f37..7bcec319a1b 100755 --- a/samples/core/condition/condition.py +++ b/samples/core/condition/condition.py @@ -13,7 +13,6 @@ # See the License for the specific language governing permissions and # limitations under the License. - import kfp from kfp import dsl diff --git a/samples/core/dsl_static_type_checking/DSL Static Type Checking.ipynb b/samples/core/dsl_static_type_checking/DSL Static Type Checking.ipynb index f37ebc01c0f..64ef3b6bb9c 100644 --- a/samples/core/dsl_static_type_checking/DSL Static Type Checking.ipynb +++ b/samples/core/dsl_static_type_checking/DSL Static Type Checking.ipynb @@ -259,7 +259,7 @@ " a = task_factory_a(field_l=12)\n", " b = task_factory_b(field_x=a.outputs['field_n'], field_y=a.outputs['field_o'], field_z=a.outputs['field_m'])\n", "\n", - "compiler.Compiler().compile(pipeline_a, 'pipeline_a.tar.gz', type_check=True)" + "compiler.Compiler().compile(pipeline_a, 'pipeline_a.zip', type_check=True)" ] }, { @@ -365,7 +365,7 @@ " b = task_factory_b(field_x=a.outputs['field_n'], field_y=a.outputs['field_o'], field_z=a.outputs['field_m'])\n", "\n", "try:\n", - " compiler.Compiler().compile(pipeline_b, 'pipeline_b.tar.gz', type_check=True)\n", + " compiler.Compiler().compile(pipeline_b, 'pipeline_b.zip', type_check=True)\n", "except InconsistentTypeException as e:\n", " print(e)" ] @@ -384,7 +384,7 @@ "outputs": [], "source": [ "# Disable the type_check\n", - "compiler.Compiler().compile(pipeline_b, 'pipeline_b.tar.gz', type_check=False)" + "compiler.Compiler().compile(pipeline_b, 'pipeline_b.zip', type_check=False)" ] }, { @@ -474,7 +474,7 @@ " a = task_factory_a(field_l=12)\n", " b = task_factory_b(field_x=a.outputs['field_n'], field_y=a.outputs['field_o'], field_z=a.outputs['field_m'])\n", "\n", - "compiler.Compiler().compile(pipeline_c, 'pipeline_c.tar.gz', type_check=True)" + "compiler.Compiler().compile(pipeline_c, 'pipeline_c.zip', type_check=True)" ] }, { @@ -572,7 +572,7 @@ " b = task_factory_b(field_x=a.outputs['field_n'], field_y=a.outputs['field_o'], field_z=a.outputs['field_m'])\n", "\n", "try:\n", - " compiler.Compiler().compile(pipeline_d, 'pipeline_d.tar.gz', type_check=True)\n", + " compiler.Compiler().compile(pipeline_d, 'pipeline_d.zip', type_check=True)\n", "except InconsistentTypeException as e:\n", " print(e)" ] @@ -597,7 +597,7 @@ " a = task_factory_a(field_l=12)\n", " # For each of the arguments, authors can also ignore the types by calling ignore_type function.\n", " b = task_factory_b(field_x=a.outputs['field_n'], field_y=a.outputs['field_o'], field_z=a.outputs['field_m'].ignore_type())\n", - "compiler.Compiler().compile(pipeline_d, 'pipeline_d.tar.gz', type_check=True)" + "compiler.Compiler().compile(pipeline_d, 'pipeline_d.zip', type_check=True)" ] }, { @@ -684,7 +684,7 @@ " a = task_factory_a(field_l=12)\n", " b = task_factory_b(field_x=a.outputs['field_n'], field_y=a.outputs['field_o'], field_z=a.outputs['field_m'])\n", "\n", - "compiler.Compiler().compile(pipeline_e, 'pipeline_e.tar.gz', type_check=True)" + "compiler.Compiler().compile(pipeline_e, 'pipeline_e.zip', type_check=True)" ] }, { @@ -707,7 +707,7 @@ " a = task_factory_a(field_l=12)\n", " b = task_factory_b(a.outputs['field_n'], a.outputs['field_o'], field_z=a.outputs['field_m'])\n", "\n", - "compiler.Compiler().compile(pipeline_f, 'pipeline_f.tar.gz', type_check=True)" + "compiler.Compiler().compile(pipeline_f, 'pipeline_f.zip', type_check=True)" ] }, { @@ -750,7 +750,7 @@ " task_factory_a(field_m=a, field_o=b)\n", "\n", "try:\n", - " compiler.Compiler().compile(pipeline_g, 'pipeline_g.tar.gz', type_check=True)\n", + " compiler.Compiler().compile(pipeline_g, 'pipeline_g.zip', type_check=True)\n", "except InconsistentTypeException as e:\n", " print(e)" ] @@ -769,7 +769,7 @@ "outputs": [], "source": [ "from pathlib import Path\n", - "for p in Path(\".\").glob(\"pipeline_[a-g].tar.gz\"):\n", + "for p in Path(\".\").glob(\"pipeline_[a-g].zip\"):\n", " p.unlink()" ] } @@ -792,8 +792,17 @@ "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.5" + }, + "pycharm": { + "stem_cell": { + "cell_type": "raw", + "source": [], + "metadata": { + "collapsed": false + } + } } }, "nbformat": 4, "nbformat_minor": 2 -} +} \ No newline at end of file diff --git a/samples/core/imagepullsecrets/imagepullsecrets.py b/samples/core/imagepullsecrets/imagepullsecrets.py index 880ab9ab2ad..d31e030dbca 100644 --- a/samples/core/imagepullsecrets/imagepullsecrets.py +++ b/samples/core/imagepullsecrets/imagepullsecrets.py @@ -15,6 +15,7 @@ container registry. """ +import kfp import kfp.dsl as dsl from kubernetes import client as k8s_client @@ -52,3 +53,6 @@ def save_most_frequent_word(message: str): # Call set_image_pull_secrets after get_pipeline_conf(). dsl.get_pipeline_conf()\ .set_image_pull_secrets([k8s_client.V1ObjectReference(name="secretA")]) + +if __name__ == '__main__': + kfp.compiler.Compiler().compile(save_most_frequent_word, __file__ + '.zip') diff --git a/samples/core/kubeflow_pipeline_using_TFX_OSS_components/KubeFlow Pipeline Using TFX OSS Components.ipynb b/samples/core/kubeflow_pipeline_using_TFX_OSS_components/KubeFlow Pipeline Using TFX OSS Components.ipynb deleted file mode 100644 index 9f97d75c393..00000000000 --- a/samples/core/kubeflow_pipeline_using_TFX_OSS_components/KubeFlow Pipeline Using TFX OSS Components.ipynb +++ /dev/null @@ -1,745 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# KubeFlow Pipeline Using TFX OSS Components\n", - "\n", - "In this notebook, we will demo: \n", - "\n", - "* Defining a KubeFlow pipeline with Python DSL\n", - "* Submiting it to Pipelines System\n", - "* Customize a step in the pipeline\n", - "\n", - "We will use a pipeline that includes some TFX OSS components such as [TFDV](https://github.com/tensorflow/data-validation), [TFT](https://github.com/tensorflow/transform), [TFMA](https://github.com/tensorflow/model-analysis)." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Setup" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": { - "tags": [ - "parameters" - ] - }, - "outputs": [], - "source": [ - "# Set your output and project. !!!Must Do before you can proceed!!!\n", - "EXPERIMENT_NAME = 'demo'\n", - "OUTPUT_DIR = 'Your-Gcs-Path' # Such as gs://bucket/objact/path\n", - "PROJECT_NAME = 'Your-Gcp-Project-Name'\n", - "BASE_IMAGE='gcr.io/%s/pusherbase:dev' % PROJECT_NAME\n", - "TARGET_IMAGE='gcr.io/%s/pusher:dev' % PROJECT_NAME\n", - "TARGET_IMAGE_TWO='gcr.io/%s/pusher_two:dev' % PROJECT_NAME\n", - "TRAIN_DATA = 'gs://ml-pipeline-playground/tfx/taxi-cab-classification/train.csv'\n", - "EVAL_DATA = 'gs://ml-pipeline-playground/tfx/taxi-cab-classification/eval.csv'\n", - "HIDDEN_LAYER_SIZE = '1500'\n", - "STEPS = 3000\n", - "DATAFLOW_TFDV_IMAGE = 'gcr.io/ml-pipeline/ml-pipeline-dataflow-tfdv:fe639f41661d8e17fcda64ff8242127620b80ba0'\n", - "DATAFLOW_TFT_IMAGE = 'gcr.io/ml-pipeline/ml-pipeline-dataflow-tft:fe639f41661d8e17fcda64ff8242127620b80ba0'\n", - "DATAFLOW_TFMA_IMAGE = 'gcr.io/ml-pipeline/ml-pipeline-dataflow-tfma:fe639f41661d8e17fcda64ff8242127620b80ba0'\n", - "DATAFLOW_TF_PREDICT_IMAGE = 'gcr.io/ml-pipeline/ml-pipeline-dataflow-tf-predict:fe639f41661d8e17fcda64ff8242127620b80ba0'\n", - "KUBEFLOW_TF_TRAINER_IMAGE = 'gcr.io/ml-pipeline/ml-pipeline-kubeflow-tf-trainer:fe639f41661d8e17fcda64ff8242127620b80ba0'\n", - "KUBEFLOW_TF_TRAINER_GPU_IMAGE = 'gcr.io/ml-pipeline/ml-pipeline-kubeflow-tf-trainer-gpu:fe639f41661d8e17fcda64ff8242127620b80ba0'\n", - "KUBEFLOW_DEPLOYER_IMAGE = 'gcr.io/ml-pipeline/ml-pipeline-kubeflow-deployer:fe639f41661d8e17fcda64ff8242127620b80ba0'\n", - "DEPLOYER_MODEL = 'notebook_tfx_taxi'\n", - "DEPLOYER_VERSION_DEV = 'dev'\n", - "DEPLOYER_VERSION_PROD = 'prod'\n", - "DEPLOYER_VERSION_PROD_TWO = 'prodtwo'" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "scrolled": false - }, - "outputs": [], - "source": [ - "# Install Pipeline SDK and tensorflow\n", - "!pip3 install kfp --upgrade\n", - "!pip3 install tensorflow==1.8.0" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Create an Experiment in the Pipeline System\n", - "\n", - "Pipeline system requires an \"Experiment\" to group pipeline runs. You can create a new experiment, or call client.list_experiments() to get existing ones." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Note that this notebook should be running in JupyterHub in the same cluster as the pipeline system.\n", - "# Otherwise it will fail to talk to the pipeline system.\n", - "import kfp\n", - "from kfp import compiler\n", - "import kfp.dsl as dsl\n", - "import kfp.notebook\n", - "import kfp.gcp as gcp\n", - "\n", - "# If you are using Kubeflow JupyterHub, then no need to set host in Client() constructor.\n", - "# But if you are using your local Jupyter instance, and have a kubectl connection to the cluster,\n", - "# Then do:\n", - "# client = kfp.Client('127.0.0.1:8080/pipeline')\n", - "client = kfp.Client()\n", - "exp = client.create_experiment(name=EXPERIMENT_NAME)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Test Run a Pipeline" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Download a pipeline package\n", - "!gsutil cp gs://ml-pipeline-playground/coin.tar.gz ." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "run = client.run_pipeline(exp.id, 'coin', 'coin.tar.gz')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Define a Pipeline\n", - "\n", - "Authoring a pipeline is just like authoring a normal Python function. The pipeline function describes the topology of the pipeline. Each step in the pipeline is typically a ContainerOp --- a simple class or function describing how to interact with a docker container image. In the below pipeline, all the container images referenced in the pipeline are already built. The pipeline starts with a TFDV step which is used to infer the schema of the data. Then it uses TFT to transform the data for training. After a single node training step, it analyze the test data predictions and generate a feature slice metrics view using a TFMA component. At last, it deploys the model to TF-Serving inside the same cluster." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Below are a list of helper functions to wrap the components to provide a simpler interface for pipeline function.\n", - "def dataflow_tf_data_validation_op(inference_data: 'GcsUri', validation_data: 'GcsUri', column_names: 'GcsUri[text/json]', key_columns, project: 'GcpProject', mode, validation_output: 'GcsUri[Directory]', step_name='validation'):\n", - " return dsl.ContainerOp(\n", - " name = step_name,\n", - " image = DATAFLOW_TFDV_IMAGE,\n", - " arguments = [\n", - " '--csv-data-for-inference', inference_data,\n", - " '--csv-data-to-validate', validation_data,\n", - " '--column-names', column_names,\n", - " '--key-columns', key_columns,\n", - " '--project', project,\n", - " '--mode', mode,\n", - " '--output', validation_output,\n", - " ],\n", - " file_outputs = {\n", - " 'schema': '/schema.txt',\n", - " }\n", - " )\n", - "\n", - "def dataflow_tf_transform_op(train_data: 'GcsUri', evaluation_data: 'GcsUri', schema: 'GcsUri[text/json]', project: 'GcpProject', preprocess_mode, preprocess_module: 'GcsUri[text/code/python]', transform_output: 'GcsUri[Directory]', step_name='preprocess'):\n", - " return dsl.ContainerOp(\n", - " name = step_name,\n", - " image = DATAFLOW_TFT_IMAGE,\n", - " arguments = [\n", - " '--train', train_data,\n", - " '--eval', evaluation_data,\n", - " '--schema', schema,\n", - " '--project', project,\n", - " '--mode', preprocess_mode,\n", - " '--preprocessing-module', preprocess_module,\n", - " '--output', transform_output,\n", - " ],\n", - " file_outputs = {'transformed': '/output.txt'}\n", - " )\n", - "\n", - "\n", - "def tf_train_op(transformed_data_dir, schema: 'GcsUri[text/json]', learning_rate: float, hidden_layer_size: int, steps: int, target: str, preprocess_module: 'GcsUri[text/code/python]', training_output: 'GcsUri[Directory]', step_name='training', use_gpu=False):\n", - " tf_train_op = dsl.ContainerOp(\n", - " name = step_name,\n", - " image = KUBEFLOW_TF_TRAINER_IMAGE,\n", - " arguments = [\n", - " '--transformed-data-dir', transformed_data_dir,\n", - " '--schema', schema,\n", - " '--learning-rate', learning_rate,\n", - " '--hidden-layer-size', hidden_layer_size,\n", - " '--steps', steps,\n", - " '--target', target,\n", - " '--preprocessing-module', preprocess_module,\n", - " '--job-dir', training_output,\n", - " ],\n", - " file_outputs = {'train': '/output.txt'}\n", - " )\n", - " if use_gpu:\n", - " tf_train_op.image = KUBEFLOW_TF_TRAINER_GPU_IMAGE\n", - " tf_train_op.set_gpu_limit(1)\n", - " \n", - " return tf_train_op\n", - "\n", - "def dataflow_tf_model_analyze_op(model: 'TensorFlow model', evaluation_data: 'GcsUri', schema: 'GcsUri[text/json]', project: 'GcpProject', analyze_mode, analyze_slice_column, analysis_output: 'GcsUri', step_name='analysis'):\n", - " return dsl.ContainerOp(\n", - " name = step_name,\n", - " image = DATAFLOW_TFMA_IMAGE,\n", - " arguments = [\n", - " '--model', model,\n", - " '--eval', evaluation_data,\n", - " '--schema', schema,\n", - " '--project', project,\n", - " '--mode', analyze_mode,\n", - " '--slice-columns', analyze_slice_column,\n", - " '--output', analysis_output,\n", - " ],\n", - " file_outputs = {'analysis': '/output.txt'}\n", - " )\n", - "\n", - "\n", - "def dataflow_tf_predict_op(evaluation_data: 'GcsUri', schema: 'GcsUri[text/json]', target: str, model: 'TensorFlow model', predict_mode, project: 'GcpProject', prediction_output: 'GcsUri', step_name='prediction'):\n", - " return dsl.ContainerOp(\n", - " name = step_name,\n", - " image = DATAFLOW_TF_PREDICT_IMAGE,\n", - " arguments = [\n", - " '--data', evaluation_data,\n", - " '--schema', schema,\n", - " '--target', target,\n", - " '--model', model,\n", - " '--mode', predict_mode,\n", - " '--project', project,\n", - " '--output', prediction_output,\n", - " ],\n", - " file_outputs = {'prediction': '/output.txt'}\n", - " )\n", - "\n", - "def kubeflow_deploy_op(model: 'TensorFlow model', tf_server_name, step_name='deploy'):\n", - " return dsl.ContainerOp(\n", - " name = step_name,\n", - " image = KUBEFLOW_DEPLOYER_IMAGE,\n", - " arguments = [\n", - " '--model-export-path', '%s/export/export' % model,\n", - " '--server-name', tf_server_name\n", - " ]\n", - " )\n", - "\n", - "\n", - "# The pipeline definition\n", - "@dsl.pipeline(\n", - " name='TFX Taxi Cab Classification Pipeline Example',\n", - " description='Example pipeline that does classification with model analysis based on a public BigQuery dataset.'\n", - ")\n", - "def taxi_cab_classification(\n", - " output,\n", - " project,\n", - " column_names='gs://ml-pipeline-playground/tfx/taxi-cab-classification/column-names.json',\n", - " key_columns='trip_start_timestamp',\n", - " train=TRAIN_DATA,\n", - " evaluation=EVAL_DATA,\n", - " validation_mode='local',\n", - " preprocess_mode='local',\n", - " preprocess_module: dsl.PipelineParam='gs://ml-pipeline-playground/tfx/taxi-cab-classification/preprocessing.py',\n", - " target='tips',\n", - " learning_rate=0.1,\n", - " hidden_layer_size=HIDDEN_LAYER_SIZE,\n", - " steps=STEPS,\n", - " predict_mode='local',\n", - " analyze_mode='local',\n", - " analyze_slice_column='trip_start_hour'):\n", - "\n", - " # set the flag to use GPU trainer\n", - " use_gpu = False\n", - " \n", - " validation_output = '%s/{{workflow.name}}/validation' % output\n", - " transform_output = '%s/{{workflow.name}}/transformed' % output\n", - " training_output = '%s/{{workflow.name}}/train' % output\n", - " analysis_output = '%s/{{workflow.name}}/analysis' % output\n", - " prediction_output = '%s/{{workflow.name}}/predict' % output\n", - " tf_server_name = 'taxi-cab-classification-model-{{workflow.name}}'\n", - "\n", - " validation = dataflow_tf_data_validation_op(train, evaluation, column_names, key_columns, project, validation_mode, validation_output).apply(gcp.use_gcp_secret('user-gcp-sa'))\n", - "\n", - " preprocess = dataflow_tf_transform_op(train, evaluation, validation.outputs['schema'], project, preprocess_mode, preprocess_module, transform_output).apply(gcp.use_gcp_secret('user-gcp-sa'))\n", - " training = tf_train_op(preprocess.output, validation.outputs['schema'], learning_rate, hidden_layer_size, steps, target, preprocess_module, training_output, use_gpu=use_gpu).apply(gcp.use_gcp_secret('user-gcp-sa'))\n", - " analysis = dataflow_tf_model_analyze_op(training.output, evaluation, validation.outputs['schema'], project, analyze_mode, analyze_slice_column, analysis_output).apply(gcp.use_gcp_secret('user-gcp-sa'))\n", - " prediction = dataflow_tf_predict_op(evaluation, validation.outputs['schema'], target, training.output, predict_mode, project, prediction_output).apply(gcp.use_gcp_secret('user-gcp-sa'))\n", - " deploy = kubeflow_deploy_op(training.output, tf_server_name).apply(gcp.use_gcp_secret('user-gcp-sa'))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Submit the run" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Compile it into a tar package.\n", - "compiler.Compiler().compile(taxi_cab_classification, 'tfx.zip')\n", - "\n", - "# Submit a run.\n", - "run = client.run_pipeline(exp.id, 'tfx', 'tfx.zip',\n", - " params={'output': OUTPUT_DIR,\n", - " 'project': PROJECT_NAME})" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Customize a step in the above pipeline\n", - "\n", - "Let's say I got the pipeline source code from github, and I want to modify the pipeline a little bit by swapping the last deployer step with my own deployer. Instead of tf-serving deployer, I want to deploy it to Cloud ML Engine service." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Create and test a python function for the new deployer" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# in order to run it locally we need a python package\n", - "!pip3 install google-api-python-client" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "@dsl.python_component(\n", - " name='cmle_deployer',\n", - " description='deploys a model to GCP CMLE',\n", - " base_image=BASE_IMAGE\n", - ")\n", - "def deploy_model(model_name: str, version_name: str, model_path: str, gcp_project: str, runtime: str):\n", - "\n", - " from googleapiclient import discovery\n", - " from tensorflow.python.lib.io import file_io\n", - " import os\n", - " \n", - " model_path = file_io.get_matching_files(os.path.join(model_path, 'export', 'export', '*'))[0]\n", - " api = discovery.build('ml', 'v1')\n", - " body = {'name': model_name}\n", - " parent = 'projects/%s' % gcp_project\n", - " try:\n", - " api.projects().models().create(body=body, parent=parent).execute()\n", - " except Exception as e:\n", - " # If the error is to create an already existing model. Ignore it.\n", - " print(str(e))\n", - " pass\n", - "\n", - " import time\n", - "\n", - " body = {\n", - " 'name': version_name,\n", - " 'deployment_uri': model_path,\n", - " 'runtime_version': runtime\n", - " }\n", - "\n", - " full_mode_name = 'projects/%s/models/%s' % (gcp_project, model_name)\n", - " response = api.projects().models().versions().create(body=body, parent=full_mode_name).execute()\n", - " \n", - " while True:\n", - " response = api.projects().operations().get(name=response['name']).execute()\n", - " if 'done' not in response or response['done'] is not True:\n", - " time.sleep(10)\n", - " print('still deploying...')\n", - " else:\n", - " if 'error' in response:\n", - " print(response['error'])\n", - " else:\n", - " print('Done.')\n", - " break" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "metadata": { - "scrolled": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "still deploying...\n", - "still deploying...\n", - "still deploying...\n", - "still deploying...\n", - "still deploying...\n", - "still deploying...\n", - "still deploying...\n", - "still deploying...\n", - "still deploying...\n", - "still deploying...\n", - "still deploying...\n", - "Done.\n" - ] - } - ], - "source": [ - "# Test the function and make sure it works.\n", - "path = 'gs://ml-pipeline-playground/sampledata/taxi/train'\n", - "deploy_model(DEPLOYER_MODEL, DEPLOYER_VERSION_DEV, path, PROJECT_NAME, '1.9')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Build a Pipeline Step With the Above Function(Note: run either of the two options below)\n", - "#### Option One: Specify the dependency directly\n", - "Now that we've tested the function locally, we want to build a component that can run as a step in the pipeline. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# The return value \"DeployerOp\" represents a step that can be used directly in a pipeline function\n", - "DeployerOp = compiler.build_python_component(\n", - " component_func=deploy_model,\n", - " staging_gcs_path=OUTPUT_DIR,\n", - " dependency=[kfp.compiler.VersionedDependency(name='google-api-python-client', version='1.7.0')],\n", - " base_image='tensorflow/tensorflow:1.12.0-py3',\n", - " target_image=TARGET_IMAGE)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Option Two: build a base docker container image with both tensorflow and google api client packages" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%%docker {BASE_IMAGE} {OUTPUT_DIR}\n", - "FROM tensorflow/tensorflow:1.10.0-py3\n", - "RUN pip3 install google-api-python-client" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Once the base docker container image is built, we can build a \"target\" container image that is base_image plus the python function as entry point. The target container image can be used as a step in a pipeline." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# The return value \"DeployerOp\" represents a step that can be used directly in a pipeline function\n", - "DeployerOp = compiler.build_python_component(\n", - " component_func=deploy_model,\n", - " staging_gcs_path=OUTPUT_DIR,\n", - " target_image=TARGET_IMAGE)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Modify the pipeline with the new deployer" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# My New Pipeline. It's almost the same as the original one with the last step deployer replaced.\n", - "@dsl.pipeline(\n", - " name='TFX Taxi Cab Classification Pipeline Example',\n", - " description='Example pipeline that does classification with model analysis based on a public BigQuery dataset.'\n", - ")\n", - "def my_taxi_cab_classification(\n", - " output,\n", - " project,\n", - " model,\n", - " version,\n", - " column_names='gs://ml-pipeline-playground/tfx/taxi-cab-classification/column-names.json',\n", - " key_columns='trip_start_timestamp',\n", - " train=TRAIN_DATA,\n", - " evaluation=EVAL_DATA,\n", - " validation_mode='local',\n", - " preprocess_mode='local',\n", - " preprocess_module='gs://ml-pipeline-playground/tfx/taxi-cab-classification/preprocessing.py',\n", - " target='tips',\n", - " learning_rate=0.1,\n", - " hidden_layer_size=HIDDEN_LAYER_SIZE,\n", - " steps=STEPS,\n", - " predict_mode='local',\n", - " analyze_mode='local',\n", - " analyze_slice_column='trip_start_hour'):\n", - " \n", - " \n", - " validation_output = '%s/{{workflow.name}}/validation' % output\n", - " transform_output = '%s/{{workflow.name}}/transformed' % output\n", - " training_output = '%s/{{workflow.name}}/train' % output\n", - " analysis_output = '%s/{{workflow.name}}/analysis' % output\n", - " prediction_output = '%s/{{workflow.name}}/predict' % output\n", - "\n", - " validation = dataflow_tf_data_validation_op(\n", - " train, evaluation, column_names, key_columns, project,\n", - " validation_mode, validation_output).apply(gcp.use_gcp_secret('user-gcp-sa'))\n", - " preprocess = dataflow_tf_transform_op(\n", - " train, evaluation, validation.outputs['schema'], project, preprocess_mode,\n", - " preprocess_module, transform_output).apply(gcp.use_gcp_secret('user-gcp-sa'))\n", - " training = tf_train_op(\n", - " preprocess.output, validation.outputs['schema'], learning_rate, hidden_layer_size,\n", - " steps, target, preprocess_module, training_output).apply(gcp.use_gcp_secret('user-gcp-sa'))\n", - " analysis = dataflow_tf_model_analyze_op(\n", - " training.output, evaluation, validation.outputs['schema'], project,\n", - " analyze_mode, analyze_slice_column, analysis_output).apply(gcp.use_gcp_secret('user-gcp-sa'))\n", - " prediction = dataflow_tf_predict_op(\n", - " evaluation, validation.outputs['schema'], target, training.output,\n", - " predict_mode, project, prediction_output).apply(gcp.use_gcp_secret('user-gcp-sa'))\n", - " \n", - " # The new deployer. Note that the DeployerOp interface is similar to the function \"deploy_model\".\n", - " deploy = DeployerOp(\n", - " gcp_project=project, model_name=model, version_name=version, runtime='1.9',\n", - " model_path=training.output).apply(gcp.use_gcp_secret('user-gcp-sa'))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Submit a new job" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "compiler.Compiler().compile(my_taxi_cab_classification, 'my-tfx.zip')\n", - "\n", - "run = client.run_pipeline(exp.id, 'my-tfx', 'my-tfx.zip',\n", - " params={'output': OUTPUT_DIR,\n", - " 'project': PROJECT_NAME,\n", - " 'model': DEPLOYER_MODEL,\n", - " 'version': DEPLOYER_VERSION_PROD})\n", - "\n", - "result = client.wait_for_run_completion(run.id, timeout=600)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Customize a step in Python2\n", - "Let's reuse the deploy_model function defined above. However, this time we will use python2 instead of the default python3." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# The return value \"DeployerOp\" represents a step that can be used directly in a pipeline function\n", - "#TODO: demonstrate the python2 support in another sample.\n", - "DeployerOp = compiler.build_python_component(\n", - " component_func=deploy_model,\n", - " staging_gcs_path=OUTPUT_DIR,\n", - " dependency=[kfp.compiler.VersionedDependency(name='google-api-python-client', version='1.7.0')],\n", - " base_image='tensorflow/tensorflow:1.12.0',\n", - " target_image=TARGET_IMAGE_TWO,\n", - " python_version='python2')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Modify the pipeline with the new deployer" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# My New Pipeline. It's almost the same as the original one with the last step deployer replaced.\n", - "@dsl.pipeline(\n", - " name='TFX Taxi Cab Classification Pipeline Example',\n", - " description='Example pipeline that does classification with model analysis based on a public BigQuery dataset.'\n", - ")\n", - "def my_taxi_cab_classification(\n", - " output,\n", - " project,\n", - " model,\n", - " version,\n", - " column_names='gs://ml-pipeline-playground/tfx/taxi-cab-classification/column-names.json',\n", - " key_columns='trip_start_timestamp',\n", - " train=TRAIN_DATA,\n", - " evaluation=EVAL_DATA,\n", - " validation_mode='local',\n", - " preprocess_mode='local',\n", - " preprocess_module='gs://ml-pipeline-playground/tfx/taxi-cab-classification/preprocessing.py',\n", - " target='tips',\n", - " learning_rate=0.1,\n", - " hidden_layer_size=HIDDEN_LAYER_SIZE,\n", - " steps=STEPS,\n", - " predict_mode='local',\n", - " analyze_mode='local',\n", - " analyze_slice_column='trip_start_hour'):\n", - " \n", - " \n", - " validation_output = '%s/{{workflow.name}}/validation' % output\n", - " transform_output = '%s/{{workflow.name}}/transformed' % output\n", - " training_output = '%s/{{workflow.name}}/train' % output\n", - " analysis_output = '%s/{{workflow.name}}/analysis' % output\n", - " prediction_output = '%s/{{workflow.name}}/predict' % output\n", - "\n", - " validation = dataflow_tf_data_validation_op(\n", - " train, evaluation, column_names, key_columns, project,\n", - " validation_mode, validation_output).apply(gcp.use_gcp_secret('user-gcp-sa'))\n", - " preprocess = dataflow_tf_transform_op(\n", - " train, evaluation, validation.outputs['schema'], project, preprocess_mode,\n", - " preprocess_module, transform_output).apply(gcp.use_gcp_secret('user-gcp-sa'))\n", - " training = tf_train_op(\n", - " preprocess.output, validation.outputs['schema'], learning_rate, hidden_layer_size,\n", - " steps, target, preprocess_module, training_output).apply(gcp.use_gcp_secret('user-gcp-sa'))\n", - " analysis = dataflow_tf_model_analyze_op(\n", - " training.output, evaluation, validation.outputs['schema'], project,\n", - " analyze_mode, analyze_slice_column, analysis_output).apply(gcp.use_gcp_secret('user-gcp-sa'))\n", - " prediction = dataflow_tf_predict_op(\n", - " evaluation, validation.outputs['schema'], target, training.output,\n", - " predict_mode, project, prediction_output).apply(gcp.use_gcp_secret('user-gcp-sa'))\n", - " \n", - " # The new deployer. Note that the DeployerOp interface is similar to the function \"deploy_model\".\n", - " deploy = DeployerOp(\n", - " gcp_project=project, model_name=model, version_name=version, runtime='1.9',\n", - " model_path=training.output).apply(gcp.use_gcp_secret('user-gcp-sa'))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Submit a new job" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "compiler.Compiler().compile(my_taxi_cab_classification, 'my-tfx-two.tar.gz')\n", - "\n", - "run = client.run_pipeline(exp.id, 'my-tfx-two', 'my-tfx-two.tar.gz',\n", - " params={'output': OUTPUT_DIR,\n", - " 'project': PROJECT_NAME,\n", - " 'model': DEPLOYER_MODEL,\n", - " 'version': DEPLOYER_VERSION_PROD_TWO})\n", - "\n", - "result = client.wait_for_run_completion(run.id, timeout=600)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Clean up" - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Activated service account credentials for: [kubeflow3-user@bradley-playground.iam.gserviceaccount.com]\n", - "Deleting version [prod]......done. \n", - "Deleting version [dev]......done. \n", - "Deleting model [notebook_tfx_taxi]...done. \n" - ] - } - ], - "source": [ - "# the step is only needed if you are using an in-cluster JupyterHub instance.\n", - "!gcloud auth activate-service-account --key-file /var/run/secrets/sa/user-gcp-sa.json\n", - "\n", - "\n", - "!gcloud ml-engine versions delete $DEPLOYER_VERSION_PROD --model $DEPLOYER_MODEL -q\n", - "!gcloud ml-engine versions delete $DEPLOYER_VERSION_PROD_TWO --model $DEPLOYER_MODEL -q\n", - "!gcloud ml-engine versions delete $DEPLOYER_VERSION_DEV --model $DEPLOYER_MODEL -q\n", - "!gcloud ml-engine models delete $DEPLOYER_MODEL -q" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.5" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/samples/core/kubeflow_tf_serving/kubeflow_tf_serving.ipynb b/samples/core/kubeflow_tf_serving/kubeflow_tf_serving.ipynb new file mode 100644 index 00000000000..25e98ce2233 --- /dev/null +++ b/samples/core/kubeflow_tf_serving/kubeflow_tf_serving.ipynb @@ -0,0 +1,297 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "# Copyright 2019 Google Inc. All Rights Reserved.\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# http://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Install Pipeline SDK - This only needs to be ran once in the enviroment. \n", + "!pip3 install kfp --upgrade\n", + "!pip3 install tensorflow==1.14" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## KubeFlow Pipelines Serving Component\n", + "In this notebook, we will demo:\n", + "\n", + "* Saving a Keras model in a format compatible with TF Serving\n", + "* Creating a pipeline to serve a trained model within a KubeFlow cluster\n", + "\n", + "Reference documentation:\n", + "* https://www.tensorflow.org/tfx/serving/architecture\n", + "* https://www.tensorflow.org/beta/guide/keras/saving_and_serializing\n", + "* https://www.kubeflow.org/docs/components/serving/tfserving_new/" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Setup\n" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "tags": [ + "parameters" + ] + }, + "outputs": [], + "source": [ + "# Set your output and project. !!!Must Do before you can proceed!!!\n", + "PROJECT_NAME = 'Your-Gcp-Project-ID' #'Your-GCP-Project-ID'\n", + "MODEL_NAME = 'Model-Name' # Model name matching TF_serve naming requirements \n", + "EXPERIMENT_NAME = 'serving_component'\n", + "MODEL_VERSION = '1' # A number representing the version model \n", + "OUTPUT_BUCKET = 'gs://%s-serving-component' % PROJECT_NAME # A GCS bucket for asset outputs\n", + "KUBEFLOW_DEPLOYER_IMAGE = 'gcr.io/ml-pipeline/ml-pipeline-kubeflow-deployer:151c5349f13bea9d626c988563c04c0a86210c21'\n", + "MODEL_PATH = '%s/%s' % (OUTPUT_BUCKET,MODEL_NAME) \n", + "MODEL_VERSION_PATH = '%s/%s/%s' % (OUTPUT_BUCKET,MODEL_NAME,MODEL_VERSION)" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Creating gs://chavoshi-dev-2-serving-component/...\n"] + } + ], + "source": [ + "!gsutil mb {OUTPUT_BUCKET}" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "#Get or create an experiment and submit a pipeline run\n", + "import kfp\n", + "client = kfp.Client()\n", + "\n", + "try:\n", + " experiment = client.get_experiment(experiment_name=EXPERIMENT_NAME)\n", + "except:\n", + " experiment = client.create_experiment(EXPERIMENT_NAME)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Load a Keras Model \n", + "Loading a pretrained Keras model to use as an example. " + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], + "source": [ + "import tensorflow as tf" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING: Logging before flag parsing goes to stderr.\n", + "W0813 22:25:44.771417 139829630277440 deprecation.py:506] From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", + "Instructions for updating:\n", + "Call initializer instance with the dtype argument instead of passing it to the constructor\n" + ] + } + ], + "source": [ + "model = tf.keras.applications.NASNetMobile(input_shape=None,\n", + " include_top=True,\n", + " weights='imagenet',\n", + " input_tensor=None,\n", + " pooling=None,\n", + " classes=1000)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Saved the Model for TF-Serve\n", + "Save the model using keras export_saved_model function. Note that specifically for TF-Serve the output directory should be structure as model_name/model_version/saved_model." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "W0813 22:26:30.974738 139829630277440 deprecation.py:506] From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:97: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", + "Instructions for updating:\n", + "Call initializer instance with the dtype argument instead of passing it to the constructor\n", + "W0813 22:26:30.992686 139829630277440 deprecation.py:506] From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:97: calling Ones.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", + "Instructions for updating:\n", + "Call initializer instance with the dtype argument instead of passing it to the constructor\n", + "W0813 22:26:31.067510 139829630277440 deprecation.py:506] From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:97: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", + "Instructions for updating:\n", + "Call initializer instance with the dtype argument instead of passing it to the constructor\n", + "W0813 22:27:34.353115 139829630277440 deprecation.py:323] From /opt/conda/lib/python3.6/site-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.\n", + "Instructions for updating:\n", + "This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.\n" + ] + } + ], + "source": [ + "tf.keras.experimental.export_saved_model(model,MODEL_VERSION_PATH)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Create a pipeline using KFP TF-Serve component\n" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [], + "source": [ + "def kubeflow_deploy_op():\n", + " return dsl.ContainerOp(\n", + " name = 'deploy',\n", + " image = KUBEFLOW_DEPLOYER_IMAGE,\n", + " arguments = [\n", + " '--model-export-path', MODEL_PATH,\n", + " '--server-name', MODEL_NAME,\n", + " ]\n", + " )" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [], + "source": [ + "import kfp\n", + "import kfp.dsl as dsl\n", + "from kfp.gcp import use_gcp_secret\n", + "\n", + "# The pipeline definition\n", + "@dsl.pipeline(\n", + " name='Sample model deployer',\n", + " description='Sample for deploying models using KFP model serving component'\n", + ")\n", + "def model_server():\n", + " deploy = kubeflow_deploy_op().apply(use_gcp_secret('user-gcp-sa'))" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [], + "source": [ + "pipeline_func = model_server\n", + "pipeline_filename = pipeline_func.__name__ + '.pipeline.zip'\n", + "\n", + "import kfp.compiler as compiler\n", + "compiler.Compiler().compile(pipeline_func, pipeline_filename)" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "Run link here" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "#Specify pipeline argument values\n", + "arguments = {}\n", + "\n", + "#Submit a pipeline run\n", + "run_name = pipeline_func.__name__ + ' run'\n", + "run_result = client.run_pipeline(experiment.id, run_name, pipeline_filename, arguments)\n", + "\n", + "#This link leads to the run information page. \n", + "#Note: There is a bug in JupyterLab that modifies the URL and makes the link stop working" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.7" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/samples/core/kubeflow_training_classification/README.md b/samples/core/kubeflow_training_classification/README.md deleted file mode 100644 index a551605772e..00000000000 --- a/samples/core/kubeflow_training_classification/README.md +++ /dev/null @@ -1,47 +0,0 @@ -## Overview - -The `kubeflow-training-classification.py` pipeline creates a TensorFlow model on structured data and image URLs (Google Cloud Storage). It works for both classification and regression. -Everything runs inside the pipeline cluster (Kubeflow). The only possible dependency is Google Cloud DataFlow if you enable the "*cloud*" mode for -the preprocessing or prediction step. - -## The requirements - -By default, the preprocessing and prediction steps use the "*local*" mode and run inside the cluster. If you specify the value of "*preprocess_mode*" as "*cloud*", you must enable the -[DataFlow API](https://cloud.google.com/endpoints/docs/openapi/enable-api) for the given GCP project so that the preprocessing step -can use Cloud DataFlow. - -Note: The trainer depends on Kubeflow API version v1alpha2. - -## Compiling the pipeline template - -Follow the guide to [building a pipeline](https://www.kubeflow.org/docs/guides/pipelines/build-pipeline/) to install the Kubeflow Pipelines SDK, then run the following command to compile the sample Python into a workflow specification. The specification takes the form of a YAML file compressed into a `.tar.gz` file. - -```bash -dsl-compile --py kubeflow-training-classification.py --output kubeflow-training-classification.tar.gz -``` - -## Deploying the pipeline - -Open the Kubeflow pipelines UI. Create a new pipeline, and then upload the compiled specification (`.tar.gz` file) as a new pipeline template. - -The pipeline requires one argument: - -1. An output directory in a Google Cloud Storage bucket, of the form `gs:///`. - -## Components source - -Preprocessing: - [source code](https://github.com/kubeflow/pipelines/tree/master/components/dataflow/tft/src), - [container](https://github.com/kubeflow/pipelines/tree/master/components/dataflow/tft) - -Training: - [source code](https://github.com/kubeflow/pipelines/tree/master/components/kubeflow/launcher/src), - [container](https://github.com/kubeflow/pipelines/tree/master/components/kubeflow/launcher) - -Prediction: - [source code](https://github.com/kubeflow/pipelines/tree/master/components/dataflow/predict/src), - [container](https://github.com/kubeflow/pipelines/tree/master/components/dataflow/predict) - -Confusion Matrix: - [source code](https://github.com/kubeflow/pipelines/tree/master/components/local/confusion_matrix/src), - [container](https://github.com/kubeflow/pipelines/tree/master/components/local/confusion_matrix) diff --git a/samples/core/kubeflow_training_classification/kubeflow_training_classification.py b/samples/core/kubeflow_training_classification/kubeflow_training_classification.py deleted file mode 100755 index a8d43f89997..00000000000 --- a/samples/core/kubeflow_training_classification/kubeflow_training_classification.py +++ /dev/null @@ -1,91 +0,0 @@ -#!/usr/bin/env python3 -# Copyright 2019 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - - -import kfp -from kfp import components -from kfp import dsl -from kfp import gcp - -dataflow_tf_transform_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/dataflow/tft/component.yaml') -kubeflow_tf_training_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/kubeflow/dnntrainer/component.yaml') -dataflow_tf_predict_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/dataflow/predict/component.yaml') -confusion_matrix_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/local/confusion_matrix/component.yaml') - -@dsl.pipeline( - name='TF training and prediction pipeline', - description='' -) -def kubeflow_training(output, project, - evaluation='gs://ml-pipeline-playground/flower/eval100.csv', - train='gs://ml-pipeline-playground/flower/train200.csv', - schema='gs://ml-pipeline-playground/flower/schema.json', - learning_rate=0.1, - hidden_layer_size='100,50', - steps=2000, - target='label', - workers=0, - pss=0, - preprocess_mode='local', - predict_mode='local', -): - output_template = str(output) + '/{{workflow.uid}}/{{pod.name}}/data' - - # set the flag to use GPU trainer - use_gpu = False - - preprocess = dataflow_tf_transform_op( - training_data_file_pattern=train, - evaluation_data_file_pattern=evaluation, - schema=schema, - gcp_project=project, - run_mode=preprocess_mode, - preprocessing_module='', - transformed_data_dir=output_template - ).apply(gcp.use_gcp_secret('user-gcp-sa')) - - training = kubeflow_tf_training_op( - transformed_data_dir=preprocess.output, - schema=schema, - learning_rate=learning_rate, - hidden_layer_size=hidden_layer_size, - steps=steps, - target=target, - preprocessing_module='', - training_output_dir=output_template - ).apply(gcp.use_gcp_secret('user-gcp-sa')) - - if use_gpu: - training.image = 'gcr.io/ml-pipeline/ml-pipeline-kubeflow-tf-trainer-gpu:fe639f41661d8e17fcda64ff8242127620b80ba0', - training.set_gpu_limit(1) - - prediction = dataflow_tf_predict_op( - data_file_pattern=evaluation, - schema=schema, - target_column=target, - model=training.output, - run_mode=predict_mode, - gcp_project=project, - predictions_dir=output_template - ).apply(gcp.use_gcp_secret('user-gcp-sa')) - - confusion_matrix = confusion_matrix_op( - predictions=prediction.output, - output_dir=output_template - ).apply(gcp.use_gcp_secret('user-gcp-sa')) - - -if __name__ == '__main__': - kfp.compiler.Compiler().compile(kubeflow_training, __file__ + '.zip') diff --git a/samples/core/recursion/recursion.py b/samples/core/recursion/recursion.py index a5f219dac3f..0d57e6994dd 100644 --- a/samples/core/recursion/recursion.py +++ b/samples/core/recursion/recursion.py @@ -66,4 +66,4 @@ def flipcoin(): if __name__ == '__main__': - kfp.compiler.Compiler().compile(flipcoin, __file__ + '.tar.gz') + kfp.compiler.Compiler().compile(flipcoin, __file__ + '.zip') diff --git a/samples/core/resource_ops/resourceop_basic.py b/samples/core/resource_ops/resourceop_basic.py index 3079379cbdb..8593006d81d 100644 --- a/samples/core/resource_ops/resourceop_basic.py +++ b/samples/core/resource_ops/resourceop_basic.py @@ -18,10 +18,9 @@ It is not a good practice to put password as a pipeline argument, since it will be visible on KFP UI. """ - -from kubernetes import client as k8s_client +import kfp import kfp.dsl as dsl - +from kubernetes import client as k8s_client @dsl.pipeline( name="ResourceOp Basic", @@ -54,7 +53,5 @@ def resourceop_basic(username, password): pvolumes={"/etc/secret-volume": secret} ) - -if __name__ == "__main__": - import kfp.compiler as compiler - compiler.Compiler().compile(resourceop_basic, __file__ + ".tar.gz") +if __name__ == '__main__': + kfp.compiler.Compiler().compile(resourceop_basic, __file__ + '.zip') diff --git a/samples/core/sidecar/sidecar.py b/samples/core/sidecar/sidecar.py index 620040a6001..096ec475036 100644 --- a/samples/core/sidecar/sidecar.py +++ b/samples/core/sidecar/sidecar.py @@ -13,9 +13,9 @@ # See the License for the specific language governing permissions and # limitations under the License. +import kfp import kfp.dsl as dsl - @dsl.pipeline( name="pipeline_with_sidecar", description="A pipeline that demonstrates how to add a sidecar to an operation." @@ -47,3 +47,6 @@ def pipeline_with_sidecar(sleep_ms: int = 10): command=["sh", "-c"], arguments=["echo %s" % op1.output], # print out content of op1 output ) + +if __name__ == '__main__': + kfp.compiler.Compiler().compile(pipeline_with_sidecar, __file__ + '.zip') \ No newline at end of file diff --git a/samples/core/tfx_cab_classification/README.md b/samples/core/tfx_cab_classification/README.md index 6c27b45f0a9..bdfc1c6236c 100644 --- a/samples/core/tfx_cab_classification/README.md +++ b/samples/core/tfx_cab_classification/README.md @@ -56,20 +56,14 @@ Open the Kubeflow pipelines UI. Create a new pipeline, and then upload the compi 1. The name of a GCP project. 2. An output directory in a Google Cloud Storage bucket, of the form `gs:///`. - On-Premise - For On-Premise cluster, the pipeline will create a Persistent Volume Claim (PVC), and download - automatically the - [source data](https://github.com/kubeflow/pipelines/tree/master/samples/core/tfx_cab_classification/taxi-cab-classification) - to the PVC. + For On-Premise cluster, the pipeline will create a Persistent Volume Claim (PVC), and download automatically the [source date](https://github.com/kubeflow/pipelines/tree/master/samples/core/tfx_cab_classification/taxi-cab-classification) to the PVC. 1. The `output` is PVC mount point for the containers, can be set to `/mnt`. 2. The `project` can be set to `taxi-cab-classification-pipeline-onprem`. 3. If the PVC mounted to `/mnt`, the value of below parameters need to be set as following: - - `column-names`: ` -/mnt/pipelines/samples/tfx/taxi-cab-classification/column-names.json` - - `train`: `/mnt/pipelines/samples/tfx/taxi-cab-classification/train.csv` - - `evaluation`: `/mnt/pipelines/samples/tfx/taxi-cab-classification/eval.csv` - - `preprocess-module`: `/mnt/pipelines/samples/tfx/taxi-cab-classification/preprocessing.py` - - + - `column-names`: `/mnt/pipelines/samples/core/tfx_cab_classification/taxi-cab-classification/column-names.json` + - `train`: `/mnt/pipelines/samples/core/tfx_cab_classification/taxi-cab-classification/train.csv` + - `evaluation`: `/mnt/pipelines/samples/core/tfx_cab_classification/taxi-cab-classification/eval.csv` + - `preprocess-module`: `/mnt/pipelines/samples/core/tfx_cab_classification/taxi-cab-classification/preprocessing.py` ## Components source diff --git a/samples/core/tfx_cab_classification/tfx_cab_classification.py b/samples/core/tfx_cab_classification/tfx_cab_classification.py index 69258bcc0a4..db47320b5fb 100755 --- a/samples/core/tfx_cab_classification/tfx_cab_classification.py +++ b/samples/core/tfx_cab_classification/tfx_cab_classification.py @@ -22,16 +22,16 @@ platform = 'GCP' -dataflow_tf_data_validation_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/dataflow/tfdv/component.yaml') -dataflow_tf_transform_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/dataflow/tft/component.yaml') -tf_train_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/kubeflow/dnntrainer/component.yaml') -dataflow_tf_model_analyze_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/dataflow/tfma/component.yaml') -dataflow_tf_predict_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/dataflow/predict/component.yaml') +dataflow_tf_data_validation_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/dataflow/tfdv/component.yaml') +dataflow_tf_transform_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/dataflow/tft/component.yaml') +tf_train_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/kubeflow/dnntrainer/component.yaml') +dataflow_tf_model_analyze_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/dataflow/tfma/component.yaml') +dataflow_tf_predict_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/dataflow/predict/component.yaml') -confusion_matrix_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/local/confusion_matrix/component.yaml') -roc_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/local/roc/component.yaml') +confusion_matrix_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/local/confusion_matrix/component.yaml') +roc_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/local/roc/component.yaml') -kubeflow_deploy_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/kubeflow/deployer/component.yaml') +kubeflow_deploy_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/kubeflow/deployer/component.yaml') @dsl.pipeline( name='TFX Taxi Cab Classification Pipeline Example', diff --git a/samples/core/volume_ops/volumeop.py b/samples/core/volume_ops/volumeop.py index babf12db6d1..5d91a3d1867 100644 --- a/samples/core/volume_ops/volumeop.py +++ b/samples/core/volume_ops/volumeop.py @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. - +import kfp import kfp.dsl as dsl @@ -36,7 +36,5 @@ def volumeop_basic(size): pvolumes={"/mnt": vop.volume} ) - -if __name__ == "__main__": - import kfp.compiler as compiler - compiler.Compiler().compile(volumeop_basic, __file__ + ".tar.gz") +if __name__ == '__main__': + kfp.compiler.Compiler().compile(volumeop_basic, __file__ + '.zip') \ No newline at end of file diff --git a/samples/core/volume_snapshot_ops/volume_snapshot_op.py b/samples/core/volume_snapshot_ops/volume_snapshot_op.py index 2b8500ec963..621e55cf3c0 100644 --- a/samples/core/volume_snapshot_ops/volume_snapshot_op.py +++ b/samples/core/volume_snapshot_ops/volume_snapshot_op.py @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. - +import kfp import kfp.dsl as dsl @@ -80,8 +80,5 @@ def volume_snapshotop_sequential(url): pvolumes={"/data": step3.pvolume} ) - -if __name__ == "__main__": - import kfp.compiler as compiler - compiler.Compiler().compile(volume_snapshotop_sequential, - __file__ + ".tar.gz") +if __name__ == '__main__': + kfp.compiler.Compiler().compile(volume_snapshotop_sequential, __file__ + '.zip') \ No newline at end of file diff --git a/samples/core/xgboost_training_cm/README.md b/samples/core/xgboost_training_cm/README.md index 14759859324..2e093c63090 100644 --- a/samples/core/xgboost_training_cm/README.md +++ b/samples/core/xgboost_training_cm/README.md @@ -14,11 +14,11 @@ Preprocessing uses Google Cloud DataProc. Therefore, you must enable the [DataPr ## Compile -Follow the guide to [building a pipeline](https://www.kubeflow.org/docs/guides/pipelines/build-pipeline/) to install the Kubeflow Pipelines SDK and compile the sample Python into a workflow specification. The specification takes the form of a YAML file compressed into a `.tar.gz` file. +Follow the guide to [building a pipeline](https://www.kubeflow.org/docs/guides/pipelines/build-pipeline/) to install the Kubeflow Pipelines SDK and compile the sample Python into a workflow specification. The specification takes the form of a YAML file compressed into a `.zip` file. ## Deploy -Open the Kubeflow pipelines UI. Create a new pipeline, and then upload the compiled specification (`.tar.gz` file) as a new pipeline template. +Open the Kubeflow pipelines UI. Create a new pipeline, and then upload the compiled specification (`.zip` file) as a new pipeline template. ## Run diff --git a/samples/core/xgboost_training_cm/xgboost_training_cm.py b/samples/core/xgboost_training_cm/xgboost_training_cm.py index 5a1f28936b8..f581eae7623 100755 --- a/samples/core/xgboost_training_cm/xgboost_training_cm.py +++ b/samples/core/xgboost_training_cm/xgboost_training_cm.py @@ -20,8 +20,8 @@ from kfp import dsl from kfp import gcp -confusion_matrix_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/local/confusion_matrix/component.yaml') -roc_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/0b07e456b1f319d8b7a7301274f55c00fda9f537/components/local/roc/component.yaml') +confusion_matrix_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/local/confusion_matrix/component.yaml') +roc_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/a97f1d0ad0e7b92203f35c5b0b9af3a314952e05/components/local/roc/component.yaml') # ! Please do not forget to enable the Dataproc API in your cluster https://console.developers.google.com/apis/api/dataproc.googleapis.com/overview @@ -36,7 +36,7 @@ def dataproc_create_cluster_op( ): return dsl.ContainerOp( name='Dataproc - Create cluster', - image='gcr.io/ml-pipeline/ml-pipeline-dataproc-create-cluster:fe639f41661d8e17fcda64ff8242127620b80ba0', + image='gcr.io/ml-pipeline/ml-pipeline-dataproc-create-cluster:151c5349f13bea9d626c988563c04c0a86210c21', arguments=[ '--project', project, '--region', region, @@ -56,7 +56,7 @@ def dataproc_delete_cluster_op( ): return dsl.ContainerOp( name='Dataproc - Delete cluster', - image='gcr.io/ml-pipeline/ml-pipeline-dataproc-delete-cluster:fe639f41661d8e17fcda64ff8242127620b80ba0', + image='gcr.io/ml-pipeline/ml-pipeline-dataproc-delete-cluster:151c5349f13bea9d626c988563c04c0a86210c21', arguments=[ '--project', project, '--region', region, @@ -76,7 +76,7 @@ def dataproc_analyze_op( ): return dsl.ContainerOp( name='Dataproc - Analyze', - image='gcr.io/ml-pipeline/ml-pipeline-dataproc-analyze:fe639f41661d8e17fcda64ff8242127620b80ba0', + image='gcr.io/ml-pipeline/ml-pipeline-dataproc-analyze:151c5349f13bea9d626c988563c04c0a86210c21', arguments=[ '--project', project, '--region', region, @@ -103,7 +103,7 @@ def dataproc_transform_op( ): return dsl.ContainerOp( name='Dataproc - Transform', - image='gcr.io/ml-pipeline/ml-pipeline-dataproc-transform:fe639f41661d8e17fcda64ff8242127620b80ba0', + image='gcr.io/ml-pipeline/ml-pipeline-dataproc-transform:151c5349f13bea9d626c988563c04c0a86210c21', arguments=[ '--project', project, '--region', region, @@ -141,7 +141,7 @@ def dataproc_train_op( return dsl.ContainerOp( name='Dataproc - Train XGBoost model', - image='gcr.io/ml-pipeline/ml-pipeline-dataproc-train:fe639f41661d8e17fcda64ff8242127620b80ba0', + image='gcr.io/ml-pipeline/ml-pipeline-dataproc-train:151c5349f13bea9d626c988563c04c0a86210c21', arguments=[ '--project', project, '--region', region, @@ -174,7 +174,7 @@ def dataproc_predict_op( ): return dsl.ContainerOp( name='Dataproc - Predict with XGBoost model', - image='gcr.io/ml-pipeline/ml-pipeline-dataproc-predict:fe639f41661d8e17fcda64ff8242127620b80ba0', + image='gcr.io/ml-pipeline/ml-pipeline-dataproc-predict:151c5349f13bea9d626c988563c04c0a86210c21', arguments=[ '--project', project, '--region', region, diff --git a/sdk/python/kfp/compiler/_component_builder.py b/sdk/python/kfp/compiler/_component_builder.py index 52daf20de1a..ece5dd6362a 100644 --- a/sdk/python/kfp/compiler/_component_builder.py +++ b/sdk/python/kfp/compiler/_component_builder.py @@ -128,18 +128,18 @@ def _generate_dockerfile(filename, base_image, entrypoint_filename, python_versi raise ValueError('python_version has to be either python2 or python3') with open(filename, 'w') as f: f.write('FROM ' + base_image + '\n') - if python_version is 'python3': + if python_version == 'python3': f.write('RUN apt-get update -y && apt-get install --no-install-recommends -y -q python3 python3-pip python3-setuptools\n') else: f.write('RUN apt-get update -y && apt-get install --no-install-recommends -y -q python python-pip python-setuptools\n') if requirement_filename is not None: f.write('ADD ' + requirement_filename + ' /ml/requirements.txt\n') - if python_version is 'python3': + if python_version == 'python3': f.write('RUN pip3 install -r /ml/requirements.txt\n') else: f.write('RUN pip install -r /ml/requirements.txt\n') f.write('ADD ' + entrypoint_filename + ' /ml/main.py\n') - if python_version is 'python3': + if python_version == 'python3': f.write('ENTRYPOINT ["python3", "-u", "/ml/main.py"]') else: f.write('ENTRYPOINT ["python", "-u", "/ml/main.py"]') @@ -282,23 +282,6 @@ def __init__(self, gcs_staging, target_image, namespace): self._arc_requirement_filename = 'requirements.txt' self._container_builder = ContainerBuilder(gcs_staging, gcr_image_tag=target_image, namespace=namespace) - def _prepare_files(self, local_dir, docker_filename, python_filename=None, requirement_filename=None): - """ _prepare_buildfiles generates the tarball with all the build files - Args: - local_dir (dir): a directory that stores all the build files - docker_filename (str): docker filename - python_filename (str): python filename - requirement_filename (str): requirement filename - """ - dst_docker_filepath = os.path.join(local_dir, self._arc_docker_filename) - shutil.copyfile(docker_filename, dst_docker_filepath) - if python_filename is not None: - dst_python_filepath = os.path.join(local_dir, self._arc_python_filename) - shutil.copyfile(python_filename, dst_python_filepath) - if requirement_filename is not None: - dst_requirement_filepath = os.path.join(local_dir, self._arc_requirement_filename) - shutil.copyfile(requirement_filename, dst_requirement_filepath) - def build_image_from_func(self, component_func, base_image, timeout, dependency, python_version='python3'): """ build_image builds an image for the given python function args: @@ -325,12 +308,6 @@ def build_image_from_func(self, component_func, base_image, timeout, dependency, logging.info('Generate build files.') return self._container_builder.build(local_build_dir, self._arc_docker_filename, timeout=timeout) - def build_image_from_dockerfile(self, docker_filename, timeout): - """ build_image_from_dockerfile builds an image based on the dockerfile """ - with tempfile.TemporaryDirectory() as local_build_dir: - self._prepare_files(local_build_dir, docker_filename) - return self._container_builder.build(local_build_dir, self._arc_docker_filename, timeout=timeout) - def _configure_logger(logger): """ _configure_logger configures the logger such that the info level logs go to the stdout and the error(or above) level logs go to the stderr. @@ -455,7 +432,14 @@ def build_docker_image(staging_gcs_path, target_image, dockerfile_path, timeout= namespace (str): the namespace within which to run the kubernetes kaniko job, default is "kubeflow" """ _configure_logger(logging.getLogger()) - builder = ComponentBuilder(gcs_staging=staging_gcs_path, target_image=target_image, namespace=namespace) - image_name_with_digest = builder.build_image_from_dockerfile(docker_filename=dockerfile_path, timeout=timeout) + + with tempfile.TemporaryDirectory() as local_build_dir: + dockerfile_rel_path = 'Dockerfile' + dst_dockerfile_path = os.path.join(local_build_dir, dockerfile_rel_path) + shutil.copyfile(dockerfile_path, dst_dockerfile_path) + + container_builder = ContainerBuilder(staging_gcs_path, target_image, namespace=namespace) + image_name_with_digest = container_builder.build(local_build_dir, dockerfile_rel_path, timeout) + logging.info('Build image complete.') return image_name_with_digest diff --git a/sdk/python/kfp/compiler/_container_builder.py b/sdk/python/kfp/compiler/_container_builder.py index b4d16318828..d8a9f785d12 100644 --- a/sdk/python/kfp/compiler/_container_builder.py +++ b/sdk/python/kfp/compiler/_container_builder.py @@ -41,7 +41,7 @@ def __init__(self, gcs_staging=None, gcr_image_tag=None, namespace=None): try: gcs_bucket = self._get_project_id() except: - raise ValueError('Please provide the gcr_staging.') + raise ValueError('Cannot get the Google Cloud project ID, please specify the gcs_staging argument.') self._gcs_staging = 'gs://' + gcs_bucket + '/' + GCS_STAGING_BLOB_DEFAULT_PREFIX else: from pathlib import PurePath @@ -160,7 +160,7 @@ def build(self, local_dir, docker_filename, target_image=None, timeout=1000): target_image (str): the target image tag to push the final image. timeout (int): time out in seconds. Default: 1000 """ - target_image = self._gcr_image_tag if target_image is None else target_image + target_image = target_image or self._gcr_image_tag # Prepare build context with tempfile.TemporaryDirectory() as local_build_dir: from ._gcs_helper import GCSHelper diff --git a/sdk/python/kfp/compiler/compiler.py b/sdk/python/kfp/compiler/compiler.py index 30d71870424..92d8658dcd6 100644 --- a/sdk/python/kfp/compiler/compiler.py +++ b/sdk/python/kfp/compiler/compiler.py @@ -655,6 +655,10 @@ def _compile(self, pipeline_func): op_transformers = [add_pod_env] op_transformers.extend(p.conf.op_transformers) workflow = self._create_pipeline_workflow(args_list_with_defaults, p, op_transformers) + + import json + workflow.setdefault('metadata', {}).setdefault('annotations', {})['pipelines.kubeflow.org/pipeline_spec'] = json.dumps(pipeline_meta.to_dict(), sort_keys=True) + return workflow def compile(self, pipeline_func, package_path, type_check=True): diff --git a/sdk/python/kfp/components/_airflow_op.py b/sdk/python/kfp/components/_airflow_op.py index 5b1fdf9e047..92244e60e73 100644 --- a/sdk/python/kfp/components/_airflow_op.py +++ b/sdk/python/kfp/components/_airflow_op.py @@ -70,6 +70,7 @@ def _create_component_spec_from_airflow_op( variables_output_names = variables_to_output or [] xcoms_output_names = xcoms_to_output or [] modules_to_capture = modules_to_capture or [op_class.__module__] + modules_to_capture.append(_run_airflow_op.__module__) output_names = [] if result_output_name is not None: diff --git a/sdk/python/kfp/components/_dynamic.py b/sdk/python/kfp/components/_dynamic.py index f35a693d19d..1a1086bd535 100644 --- a/sdk/python/kfp/components/_dynamic.py +++ b/sdk/python/kfp/components/_dynamic.py @@ -28,9 +28,9 @@ def create_function_from_parameter_names(func: Callable[[Mapping[str, Any]], Any def create_function_from_parameters(func: Callable[[Mapping[str, Any]], Any], parameters: Sequence[Parameter], documentation=None, func_name=None, func_filename=None): new_signature = Signature(parameters) # Checks the parameter consistency - + def pass_locals(): - return dict_func(locals()) + return dict_func(locals()) # noqa: F821 TODO code = pass_locals.__code__ mod_co_argcount = len(parameters) @@ -59,10 +59,10 @@ def pass_locals(): mod_co_firstlineno, code.co_lnotab ) - + default_arg_values = tuple( p.default for p in parameters if p.default != Parameter.empty ) #!argdefs "starts from the right"/"is right-aligned" modified_func = types.FunctionType(modified_code, {'dict_func': func, 'locals': locals}, name=func_name, argdefs=default_arg_values) modified_func.__doc__ = documentation modified_func.__signature__ = new_signature - + return modified_func diff --git a/sdk/python/kfp/components/_python_op.py b/sdk/python/kfp/components/_python_op.py index e73bbb52fa1..2771501d1bf 100644 --- a/sdk/python/kfp/components/_python_op.py +++ b/sdk/python/kfp/components/_python_op.py @@ -23,6 +23,7 @@ import inspect from pathlib import Path +import typing from typing import TypeVar, Generic, List T = TypeVar('T') @@ -146,8 +147,9 @@ def annotation_to_type_struct(annotation): return None if isinstance(annotation, type): return str(annotation.__name__) - else: - return str(annotation) + if hasattr(annotation, '__forward_arg__'): # Handling typing.ForwardRef('Type_name') (the name was _ForwardRef in python 3.5-3.6) + return str(annotation.__forward_arg__) # It can only be string + return str(annotation) for parameter in parameters: type_struct = annotation_to_type_struct(parameter.annotation) diff --git a/sdk/python/kfp/components/modelbase.py b/sdk/python/kfp/components/modelbase.py index a3b2e3c92e9..7748bdc9936 100644 --- a/sdk/python/kfp/components/modelbase.py +++ b/sdk/python/kfp/components/modelbase.py @@ -282,7 +282,7 @@ def __repr__(self): return self.__class__.__name__ + '(' + ', '.join(param + '=' + repr(getattr(self, param)) for param in self._get_field_names()) + ')' def __eq__(self, other): - return self.__class__ == other.__class__ and {k: getattr(self, k) for k in self._get_field_names()} == {k: getattr(self, k) for k in other._get_field_names()} + return self.__class__ == other.__class__ and {k: getattr(self, k) for k in self._get_field_names()} == {k: getattr(other, k) for k in other._get_field_names()} def __ne__(self, other): return not self == other \ No newline at end of file diff --git a/sdk/python/kfp/dsl/_pipeline.py b/sdk/python/kfp/dsl/_pipeline.py index 902fc0286f7..12216c2a8dd 100644 --- a/sdk/python/kfp/dsl/_pipeline.py +++ b/sdk/python/kfp/dsl/_pipeline.py @@ -47,7 +47,7 @@ def _pipeline(func): if _pipeline_decorator_handler: return _pipeline_decorator_handler(func) or func else: - return func + return func return _pipeline @@ -88,7 +88,7 @@ def set_ttl_seconds_after_finished(self, seconds: int): seconds: number of seconds for the workflow to be garbage collected after it is finished. """ self.ttl_seconds_after_finished = seconds - return self + return self def set_artifact_location(self, artifact_location): """Configures the pipeline level artifact location. @@ -243,7 +243,7 @@ def _set_metadata(self, metadata): Args: metadata (ComponentMeta): component metadata ''' - if not isinstance(metadata, PipelineMeta): + if not isinstance(metadata, PipelineMeta): # noqa: F821 TODO raise ValueError('_set_medata is expecting PipelineMeta.') self._metadata = metadata diff --git a/sdk/python/kfp/dsl/_pipeline_param.py b/sdk/python/kfp/dsl/_pipeline_param.py index c17be58b4ab..3d4875e8689 100644 --- a/sdk/python/kfp/dsl/_pipeline_param.py +++ b/sdk/python/kfp/dsl/_pipeline_param.py @@ -22,7 +22,7 @@ # TODO: Move this to a separate class # For now, this identifies a condition with only "==" operator supported. ConditionOperator = namedtuple('ConditionOperator', 'operator operand1 operand2') -PipelineParamTuple = namedtuple('PipelineParamTuple', 'name op value type pattern') +PipelineParamTuple = namedtuple('PipelineParamTuple', 'name op pattern') def sanitize_k8s_name(name): @@ -40,26 +40,13 @@ def match_serialized_pipelineparam(payload: str): Returns: PipelineParamTuple """ - matches = re.findall(r'{{pipelineparam:op=([\w\s_-]*);name=([\w\s_-]+);value=(.*?);type=(.*?);}}', payload) - if len(matches) == 0: - matches = re.findall(r'{{pipelineparam:op=([\w\s_-]*);name=([\w\s_-]+);value=(.*?)}}', payload) + matches = re.findall(r'{{pipelineparam:op=([\w\s_-]*);name=([\w\s_-]+)}}', payload) param_tuples = [] for match in matches: - if len(match) == 3: - pattern = '{{pipelineparam:op=%s;name=%s;value=%s}}' % (match[0], match[1], match[2]) + pattern = '{{pipelineparam:op=%s;name=%s}}' % (match[0], match[1]) param_tuples.append(PipelineParamTuple( name=sanitize_k8s_name(match[1]), op=sanitize_k8s_name(match[0]), - value=match[2], - type='', - pattern=pattern)) - elif len(match) == 4: - pattern = '{{pipelineparam:op=%s;name=%s;value=%s;type=%s;}}' % (match[0], match[1], match[2], match[3]) - param_tuples.append(PipelineParamTuple( - name=sanitize_k8s_name(match[1]), - op=sanitize_k8s_name(match[0]), - value=match[2], - type=match[3], pattern=pattern)) return param_tuples @@ -81,8 +68,6 @@ def _extract_pipelineparams(payloads: str or List[str]): for param_tuple in list(set(param_tuples)): pipeline_params.append(PipelineParam(param_tuple.name, param_tuple.op, - param_tuple.value, - TypeMeta.deserialize(param_tuple.type), pattern=param_tuple.pattern)) return pipeline_params @@ -205,11 +190,7 @@ def __str__(self): # return str(self.value) op_name = self.op_name if self.op_name else '' - value = self.value if self.value else '' - if self.param_type is None: - return '{{pipelineparam:op=%s;name=%s;value=%s}}' % (op_name, self.name, value) - else: - return '{{pipelineparam:op=%s;name=%s;value=%s;type=%s;}}' % (op_name, self.name, value, self.param_type.serialize()) + return '{{pipelineparam:op=%s;name=%s}}' % (op_name, self.name) def __repr__(self): return str({self.__class__.__name__: self.__dict__}) diff --git a/sdk/python/kfp/dsl/_resource_op.py b/sdk/python/kfp/dsl/_resource_op.py index b07207662bf..2e69fd3f8f3 100644 --- a/sdk/python/kfp/dsl/_resource_op.py +++ b/sdk/python/kfp/dsl/_resource_op.py @@ -105,6 +105,10 @@ def __init__(self, if merge_strategy and action != "apply": ValueError("You can't set merge_strategy when action != 'apply'") + + # if action is delete, there should not be any outputs, success_condition, and failure_condition + if action == "delete" and (success_condition or failure_condition or attribute_outputs): + ValueError("You can't set success_condition, failure_condition, or attribute_outputs when action == 'delete'") init_resource = { "action": action, @@ -117,6 +121,13 @@ def __init__(self, self.k8s_resource = k8s_resource + # if action is delete, there should not be any outputs, success_condition, and failure_condition + if action == "delete": + self.attribute_outputs = {} + self.outputs = {} + self.output = None + return + # Set attribute_outputs extra_attribute_outputs = \ attribute_outputs if attribute_outputs else {} diff --git a/sdk/python/kfp/gcp.py b/sdk/python/kfp/gcp.py index 6e1cf30273d..c8eab7529d5 100644 --- a/sdk/python/kfp/gcp.py +++ b/sdk/python/kfp/gcp.py @@ -14,8 +14,7 @@ from kubernetes.client import V1Toleration - -def use_gcp_secret(secret_name='user-gcp-sa', secret_file_path_in_volume='/user-gcp-sa.json', volume_name=None, secret_volume_mount_path='/secret/gcp-credentials'): +def use_gcp_secret(secret_name='user-gcp-sa', secret_file_path_in_volume=None, volume_name=None, secret_volume_mount_path='/secret/gcp-credentials'): """An operator that configures the container to use GCP service account. The user-gcp-sa secret is created as part of the kubeflow deployment that @@ -32,8 +31,13 @@ def use_gcp_secret(secret_name='user-gcp-sa', secret_file_path_in_volume='/user- service account access permission. """ + # permitted values for secret_name = ['admin-gcp-sa', 'user-gcp-sa'] + if secret_file_path_in_volume is None: + secret_file_path_in_volume = '/' + secret_name + '.json' + if volume_name is None: volume_name = 'gcp-credentials-' + secret_name + else: import warnings warnings.warn('The volume_name parameter is deprecated and will be removed in next release. The volume names are now generated automatically.', DeprecationWarning) @@ -107,4 +111,4 @@ def _set_preemptible(task): task.add_node_selector_constraint("cloud.google.com/gke-preemptible", "true") return task - return _set_preemptible \ No newline at end of file + return _set_preemptible diff --git a/sdk/python/setup.py b/sdk/python/setup.py index cf0945ff1b0..f2cc3f65f95 100644 --- a/sdk/python/setup.py +++ b/sdk/python/setup.py @@ -15,10 +15,10 @@ from setuptools import setup NAME = 'kfp' -VERSION = '0.1.25' +VERSION = '0.1.27' REQUIRES = [ - 'urllib3>=1.15,<1.25', #Fixing the version conflict with the "requests" package + 'urllib3>=1.15,<1.25', #Fixing the version conflict with the "requests" package 'six >= 1.10', 'certifi', 'python-dateutil', @@ -30,7 +30,7 @@ 'google-auth>=1.6.1', 'requests_toolbelt>=0.8.0', 'cloudpickle', - 'kfp-server-api >= 0.1.18, <= 0.1.25', #Update the upper version whenever a new version of the kfp-server-api package is released. Update the lower version when there is a breaking change in kfp-server-api. + 'kfp-server-api >= 0.1.18, <= 0.1.25', #Update the upper version whenever a new version of the kfp-server-api package is released. Update the lower version when there is a breaking change in kfp-server-api. 'argo-models == 2.2.1a', #2.2.1a is equivalent to argo 2.2.1 'jsonschema >= 3.0.1', 'tabulate == 0.8.3', @@ -70,6 +70,8 @@ ], python_requires='>=3.5.3', include_package_data=True, - entry_points={'console_scripts': [ - 'dsl-compile = kfp.compiler.main:main', - 'kfp=kfp.__main__:main']}) + entry_points={ + 'console_scripts': [ + 'dsl-compile = kfp.compiler.main:main', 'kfp=kfp.__main__:main' + ] + }) diff --git a/sdk/python/tests/compiler/compiler_tests.py b/sdk/python/tests/compiler/compiler_tests.py index 181e5b8d950..b1afb93e177 100644 --- a/sdk/python/tests/compiler/compiler_tests.py +++ b/sdk/python/tests/compiler/compiler_tests.py @@ -31,6 +31,14 @@ from kubernetes.client import V1Toleration +def some_op(): + return dsl.ContainerOp( + name='sleep', + image='busybox', + command=['sleep 1'], + ) + + class TestCompiler(unittest.TestCase): # Define the places of samples covered by unit tests. core_sample_path = os.path.join(os.path.dirname(__file__), '..', '..', '..', @@ -89,7 +97,7 @@ def test_operator_to_template(self): 'inputs': {'parameters': [ {'name': 'msg1'}, - {'name': 'msg2', 'value': 'value2'}, + {'name': 'msg2'}, ]}, 'name': 'echo', 'outputs': { @@ -337,7 +345,16 @@ def test_py_volume(self): def test_py_retry(self): """Test retry functionality.""" - self._test_py_compile_yaml('retry') + number_of_retries = 137 + def my_pipeline(): + some_op().set_retry(number_of_retries) + + workflow = kfp.compiler.Compiler()._compile(my_pipeline) + name_to_template = {template['name']: template for template in workflow['spec']['templates']} + main_dag_tasks = name_to_template[workflow['spec']['entrypoint']]['dag']['tasks'] + template = name_to_template[main_dag_tasks[0]['template']] + + self.assertEqual(template['retryStrategy']['limit'], number_of_retries) def test_py_image_pull_secrets(self): """Test pipeline imagepullsecret.""" @@ -616,3 +633,35 @@ def init_container_pipeline(): init_container = init_containers[0] self.assertEqual(init_container, {'image':'alpine:latest', 'command': ['echo', 'bye'], 'name': 'echo'}) + + def test_delete_resource_op(self): + """Test a pipeline with a delete resource operation.""" + from kubernetes import client as k8s + + @dsl.pipeline() + def some_pipeline(): + # create config map object with k6 load test script + config_map = k8s.V1ConfigMap( + api_version="v1", + data={"foo": "bar"}, + kind="ConfigMap", + metadata=k8s.V1ObjectMeta( + name="foo-bar-cm", + namespace="default" + ) + ) + # delete the config map in k8s + dsl.ResourceOp( + name="delete-config-map", + action="delete", + k8s_resource=config_map + ) + + workflow_dict = kfp.compiler.Compiler()._compile(some_pipeline) + delete_op_template = [template for template in workflow_dict['spec']['templates'] if template['name'] == 'delete-config-map'][0] + + # delete resource operation should not have success condition, failure condition or output parameters. + # See https://github.com/argoproj/argo/blob/5331fc02e257266a4a5887dfe6277e5a0b42e7fc/cmd/argoexec/commands/resource.go#L30 + self.assertIsNone(delete_op_template.get("successCondition")) + self.assertIsNone(delete_op_template.get("failureCondition")) + self.assertDictEqual(delete_op_template.get("outputs"), {}) diff --git a/sdk/python/tests/compiler/container_builder_test.py b/sdk/python/tests/compiler/container_builder_test.py index d72bb16575c..3bf7509bb49 100644 --- a/sdk/python/tests/compiler/container_builder_test.py +++ b/sdk/python/tests/compiler/container_builder_test.py @@ -21,6 +21,7 @@ from kfp.compiler._component_builder import ContainerBuilder GCS_BASE = 'gs://kfp-testing/' +GCR_IMAGE_TAG = 'gcr.io/kfp-testing/image' @mock.patch('kfp.compiler._gcs_helper.GCSHelper') class TestContainerBuild(unittest.TestCase): @@ -39,7 +40,7 @@ def test_wrap_dir_in_tarball(self, mock_gcshelper): f.write('temporary file two content') # check - builder = ContainerBuilder(gcs_staging=GCS_BASE, namespace='') + builder = ContainerBuilder(gcs_staging=GCS_BASE, gcr_image_tag=GCR_IMAGE_TAG, namespace='') builder._wrap_dir_in_tarball(temp_tarball, test_data_dir) self.assertTrue(os.path.exists(temp_tarball)) with tarfile.open(temp_tarball) as temp_tarball_handle: @@ -57,7 +58,7 @@ def test_generate_kaniko_yaml(self, mock_gcshelper): test_data_dir = os.path.join(os.path.dirname(__file__), 'testdata') # check - builder = ContainerBuilder(gcs_staging=GCS_BASE, namespace='default') + builder = ContainerBuilder(gcs_staging=GCS_BASE, gcr_image_tag=GCR_IMAGE_TAG, namespace='default') generated_yaml = builder._generate_kaniko_spec(docker_filename='dockerfile', context='gs://mlpipeline/kaniko_build.tar.gz', target_image='gcr.io/mlpipeline/kaniko_image:latest') with open(os.path.join(test_data_dir, 'kaniko.basic.yaml'), 'r') as f: diff --git a/sdk/python/tests/compiler/testdata/add_pod_env.yaml b/sdk/python/tests/compiler/testdata/add_pod_env.yaml index fb4b114587e..bfcdc1a41a5 100644 --- a/sdk/python/tests/compiler/testdata/add_pod_env.yaml +++ b/sdk/python/tests/compiler/testdata/add_pod_env.yaml @@ -1,6 +1,9 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "Test adding pod env", + "inputs": [], "name": "Test adding pod env"}' generateName: test-adding-pod-env- spec: arguments: diff --git a/sdk/python/tests/compiler/testdata/artifact_location.yaml b/sdk/python/tests/compiler/testdata/artifact_location.yaml index 0bd8432ff8c..c6c369b10de 100644 --- a/sdk/python/tests/compiler/testdata/artifact_location.yaml +++ b/sdk/python/tests/compiler/testdata/artifact_location.yaml @@ -14,6 +14,11 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "hello world", "inputs": + [{"default": null, "description": "", "name": "tag", "type": ""}, {"default": + "kubeflow", "description": "", "name": "namespace", "type": ""}, {"default": + "foobar", "description": "", "name": "bucket", "type": ""}], "name": "foo"}' generateName: foo- spec: arguments: diff --git a/sdk/python/tests/compiler/testdata/basic.yaml b/sdk/python/tests/compiler/testdata/basic.yaml index 3a88a734f5a..03b8e27880a 100644 --- a/sdk/python/tests/compiler/testdata/basic.yaml +++ b/sdk/python/tests/compiler/testdata/basic.yaml @@ -14,6 +14,8 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "Get Most Frequent Word and Save to GCS", "inputs": [{"default": null, "description": "", "name": "message", "type": ""}, {"default": null, "description": "", "name": "outputpath", "type": ""}], "name": "Save Most Frequent"}' generateName: save-most-frequent- spec: arguments: diff --git a/sdk/python/tests/compiler/testdata/coin.yaml b/sdk/python/tests/compiler/testdata/coin.yaml index 5a044b747ff..c70e31294eb 100644 --- a/sdk/python/tests/compiler/testdata/coin.yaml +++ b/sdk/python/tests/compiler/testdata/coin.yaml @@ -14,6 +14,8 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "shows how to use dsl.Condition.", "inputs": [], "name": "pipeline flip coin"}' generateName: pipeline-flip-coin- spec: arguments: diff --git a/sdk/python/tests/compiler/testdata/compose.yaml b/sdk/python/tests/compiler/testdata/compose.yaml index 645eb5d2018..59dc3513dba 100644 --- a/sdk/python/tests/compiler/testdata/compose.yaml +++ b/sdk/python/tests/compiler/testdata/compose.yaml @@ -14,6 +14,8 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "Download and Get Most Frequent Word and Save to GCS", "inputs": [{"default": null, "description": "", "name": "url", "type": ""}, {"default": null, "description": "", "name": "outputpath", "type": ""}], "name": "Download and Save Most Frequent"}' generateName: download-and-save-most-frequent- spec: arguments: diff --git a/sdk/python/tests/compiler/testdata/default_value.yaml b/sdk/python/tests/compiler/testdata/default_value.yaml index daa6140c978..a03a0803463 100644 --- a/sdk/python/tests/compiler/testdata/default_value.yaml +++ b/sdk/python/tests/compiler/testdata/default_value.yaml @@ -14,6 +14,8 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "A pipeline with parameter and default value.", "inputs": [{"default": "gs://ml-pipeline/shakespeare1.txt", "description": "", "name": "url", "type": ""}], "name": "Default Value"}' generateName: default-value- spec: arguments: diff --git a/sdk/python/tests/compiler/testdata/imagepullsecrets.yaml b/sdk/python/tests/compiler/testdata/imagepullsecrets.yaml index b7a5a91c8d3..3ea58057fac 100644 --- a/sdk/python/tests/compiler/testdata/imagepullsecrets.yaml +++ b/sdk/python/tests/compiler/testdata/imagepullsecrets.yaml @@ -1,6 +1,8 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "Get Most Frequent Word and Save to GCS", "inputs": [{"default": null, "description": "", "name": "message", "type": ""}], "name": "Save Most Frequent"}' generateName: save-most-frequent- spec: arguments: diff --git a/sdk/python/tests/compiler/testdata/param_op_transform.yaml b/sdk/python/tests/compiler/testdata/param_op_transform.yaml index a9f800c2acd..7a394c755e2 100644 --- a/sdk/python/tests/compiler/testdata/param_op_transform.yaml +++ b/sdk/python/tests/compiler/testdata/param_op_transform.yaml @@ -1,6 +1,12 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "Test that parameters used + in Op transformation functions as pod labels would be correcly identified and + set as arguments in he generated yaml", "inputs": [{"default": null, "description": + "", "name": "param", "type": ""}], "name": "Parameters in Op transformation + functions"}' generateName: parameters-in-op-transformation-functions- spec: arguments: diff --git a/sdk/python/tests/compiler/testdata/param_substitutions.yaml b/sdk/python/tests/compiler/testdata/param_substitutions.yaml index b29d212be15..14ef651d68f 100644 --- a/sdk/python/tests/compiler/testdata/param_substitutions.yaml +++ b/sdk/python/tests/compiler/testdata/param_substitutions.yaml @@ -1,6 +1,9 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "Test the same PipelineParam + getting substituted in multiple places", "inputs": [], "name": "Param Substitutions"}' generateName: param-substitutions- spec: arguments: diff --git a/sdk/python/tests/compiler/testdata/pipelineparams.yaml b/sdk/python/tests/compiler/testdata/pipelineparams.yaml index f5c5c30222a..2f145d3ba2e 100644 --- a/sdk/python/tests/compiler/testdata/pipelineparams.yaml +++ b/sdk/python/tests/compiler/testdata/pipelineparams.yaml @@ -14,6 +14,11 @@ --- apiVersion: argoproj.io/v1alpha1 metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "A pipeline with multiple + pipeline params.", "inputs": [{"default": "latest", "description": "", "name": + "tag", "type": ""}, {"default": 10, "description": "", "name": "sleep_ms", "type": + ""}], "name": "PipelineParams"}' generateName: pipelineparams- spec: entrypoint: pipelineparams diff --git a/sdk/python/tests/compiler/testdata/preemptible_tpu_gpu.yaml b/sdk/python/tests/compiler/testdata/preemptible_tpu_gpu.yaml index 172adfff15a..48d426336d6 100644 --- a/sdk/python/tests/compiler/testdata/preemptible_tpu_gpu.yaml +++ b/sdk/python/tests/compiler/testdata/preemptible_tpu_gpu.yaml @@ -1,6 +1,8 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "shows how to use dsl.Condition.", "inputs": [], "name": "pipeline flip coin"}' generateName: pipeline-flip-coin- spec: arguments: diff --git a/sdk/python/tests/compiler/testdata/recursive_do_while.yaml b/sdk/python/tests/compiler/testdata/recursive_do_while.yaml index f3f18980b57..9da2c3c65c0 100644 --- a/sdk/python/tests/compiler/testdata/recursive_do_while.yaml +++ b/sdk/python/tests/compiler/testdata/recursive_do_while.yaml @@ -1,6 +1,9 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "shows how to use graph_component.", + "inputs": [], "name": "pipeline flip coin"}' generateName: pipeline-flip-coin- spec: arguments: diff --git a/sdk/python/tests/compiler/testdata/recursive_while.yaml b/sdk/python/tests/compiler/testdata/recursive_while.yaml index d07fa97722b..3ec59455a18 100644 --- a/sdk/python/tests/compiler/testdata/recursive_while.yaml +++ b/sdk/python/tests/compiler/testdata/recursive_while.yaml @@ -1,6 +1,10 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "shows how to use dsl.Condition.", + "inputs": [{"default": 12, "description": "", "name": "maxVal", "type": ""}], + "name": "pipeline flip coin"}' generateName: pipeline-flip-coin- spec: arguments: diff --git a/sdk/python/tests/compiler/testdata/resourceop_basic.yaml b/sdk/python/tests/compiler/testdata/resourceop_basic.yaml index 2da6dfd2188..a12332fff6f 100644 --- a/sdk/python/tests/compiler/testdata/resourceop_basic.yaml +++ b/sdk/python/tests/compiler/testdata/resourceop_basic.yaml @@ -1,6 +1,11 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "A Basic Example on ResourceOp + Usage.", "inputs": [{"default": null, "description": "", "name": "username", + "type": ""}, {"default": null, "description": "", "name": "password", "type": + ""}], "name": "ResourceOp Basic"}' generateName: resourceop-basic- spec: arguments: diff --git a/sdk/python/tests/compiler/testdata/retry.py b/sdk/python/tests/compiler/testdata/retry.py deleted file mode 100755 index 030fc1ddc49..00000000000 --- a/sdk/python/tests/compiler/testdata/retry.py +++ /dev/null @@ -1,42 +0,0 @@ -#!/usr/bin/env python3 -# Copyright 2018 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - - -import kfp.dsl as dsl - - -class RandomFailure1Op(dsl.ContainerOp): - """A component that fails randomly.""" - - def __init__(self, exit_codes): - super(RandomFailure1Op, self).__init__( - name='random_failure', - image='python:alpine3.6', - command=['python', '-c'], - arguments=["import random; import sys; exit_code = random.choice([%s]); print(exit_code); sys.exit(exit_code)" % exit_codes]) - - -@dsl.pipeline( - name='pipeline includes two steps which fail randomly.', - description='shows how to use ContainerOp set_retry().' -) -def retry_sample_pipeline(): - op1 = RandomFailure1Op('0,1,2,3').set_retry(100) - op2 = RandomFailure1Op('0,1').set_retry(50) - - -if __name__ == '__main__': - import kfp.compiler as compiler - compiler.Compiler().compile(retry_sample_pipeline, __file__ + '.tar.gz') diff --git a/sdk/python/tests/compiler/testdata/retry.yaml b/sdk/python/tests/compiler/testdata/retry.yaml deleted file mode 100644 index 8e9c927a1a0..00000000000 --- a/sdk/python/tests/compiler/testdata/retry.yaml +++ /dev/null @@ -1,55 +0,0 @@ -apiVersion: argoproj.io/v1alpha1 -kind: Workflow -metadata: - generateName: pipeline-includes-two-steps-which-fail-randomly- -spec: - arguments: - parameters: [] - entrypoint: pipeline-includes-two-steps-which-fail-randomly - serviceAccountName: pipeline-runner - templates: - - dag: - tasks: - - name: random-failure - template: random-failure - - name: random-failure-2 - template: random-failure-2 - name: pipeline-includes-two-steps-which-fail-randomly - - container: - args: - - import random; import sys; exit_code = random.choice([0,1,2,3]); print(exit_code); - sys.exit(exit_code) - command: - - python - - -c - image: python:alpine3.6 - name: random-failure - outputs: - artifacts: - - name: mlpipeline-ui-metadata - path: /mlpipeline-ui-metadata.json - optional: true - - name: mlpipeline-metrics - path: /mlpipeline-metrics.json - optional: true - retryStrategy: - limit: 100 - - container: - args: - - import random; import sys; exit_code = random.choice([0,1]); print(exit_code); - sys.exit(exit_code) - command: - - python - - -c - image: python:alpine3.6 - name: random-failure-2 - outputs: - artifacts: - - name: mlpipeline-ui-metadata - path: /mlpipeline-ui-metadata.json - optional: true - - name: mlpipeline-metrics - path: /mlpipeline-metrics.json - optional: true - retryStrategy: - limit: 50 diff --git a/sdk/python/tests/compiler/testdata/sidecar.yaml b/sdk/python/tests/compiler/testdata/sidecar.yaml index be267bfb038..924a382244d 100644 --- a/sdk/python/tests/compiler/testdata/sidecar.yaml +++ b/sdk/python/tests/compiler/testdata/sidecar.yaml @@ -13,6 +13,9 @@ # limitations under the License. kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "A pipeline with sidecars.", + "inputs": [], "name": "Sidecar"}' generateName: sidecar- apiVersion: argoproj.io/v1alpha1 spec: diff --git a/sdk/python/tests/compiler/testdata/timeout.yaml b/sdk/python/tests/compiler/testdata/timeout.yaml index 606eb2bbc89..335668ef5eb 100644 --- a/sdk/python/tests/compiler/testdata/timeout.yaml +++ b/sdk/python/tests/compiler/testdata/timeout.yaml @@ -1,6 +1,10 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "shows how to use ContainerOp + set_retry().", "inputs": [], "name": "pipeline includes two steps which fail + randomly."}' generateName: pipeline-includes-two-steps-which-fail-randomly- spec: activeDeadlineSeconds: 50 diff --git a/sdk/python/tests/compiler/testdata/volume.yaml b/sdk/python/tests/compiler/testdata/volume.yaml index 73175df6e91..470fc321288 100644 --- a/sdk/python/tests/compiler/testdata/volume.yaml +++ b/sdk/python/tests/compiler/testdata/volume.yaml @@ -14,6 +14,9 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "A pipeline with volume.", + "inputs": [], "name": "Volume"}' generateName: volume- spec: arguments: diff --git a/sdk/python/tests/compiler/testdata/volume_snapshotop_rokurl.yaml b/sdk/python/tests/compiler/testdata/volume_snapshotop_rokurl.yaml index 20106baef00..4b6c5bb9344 100644 --- a/sdk/python/tests/compiler/testdata/volume_snapshotop_rokurl.yaml +++ b/sdk/python/tests/compiler/testdata/volume_snapshotop_rokurl.yaml @@ -1,6 +1,10 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "The fifth example of the + design doc.", "inputs": [{"default": null, "description": "", "name": "rok_url", + "type": ""}], "name": "VolumeSnapshotOp RokURL"}' generateName: volumesnapshotop-rokurl- spec: arguments: diff --git a/sdk/python/tests/compiler/testdata/volume_snapshotop_sequential.yaml b/sdk/python/tests/compiler/testdata/volume_snapshotop_sequential.yaml index d6a772ad94a..8d03485c2f0 100644 --- a/sdk/python/tests/compiler/testdata/volume_snapshotop_sequential.yaml +++ b/sdk/python/tests/compiler/testdata/volume_snapshotop_sequential.yaml @@ -1,6 +1,10 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "The fourth example of + the design doc.", "inputs": [{"default": null, "description": "", "name": "url", + "type": ""}], "name": "VolumeSnapshotOp Sequential"}' generateName: volumesnapshotop-sequential- spec: arguments: diff --git a/sdk/python/tests/compiler/testdata/volumeop_basic.yaml b/sdk/python/tests/compiler/testdata/volumeop_basic.yaml index 4172075e989..eda060bc615 100644 --- a/sdk/python/tests/compiler/testdata/volumeop_basic.yaml +++ b/sdk/python/tests/compiler/testdata/volumeop_basic.yaml @@ -1,6 +1,10 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "A Basic Example on VolumeOp + Usage.", "inputs": [{"default": null, "description": "", "name": "size", "type": + ""}], "name": "VolumeOp Basic"}' generateName: volumeop-basic- spec: arguments: diff --git a/sdk/python/tests/compiler/testdata/volumeop_dag.yaml b/sdk/python/tests/compiler/testdata/volumeop_dag.yaml index 3f8836927a8..31158a458cc 100644 --- a/sdk/python/tests/compiler/testdata/volumeop_dag.yaml +++ b/sdk/python/tests/compiler/testdata/volumeop_dag.yaml @@ -1,6 +1,9 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "The second example of + the design doc.", "inputs": [], "name": "Volume Op DAG"}' generateName: volume-op-dag- spec: arguments: diff --git a/sdk/python/tests/compiler/testdata/volumeop_parallel.yaml b/sdk/python/tests/compiler/testdata/volumeop_parallel.yaml index 83e11f021b9..064b16a1432 100644 --- a/sdk/python/tests/compiler/testdata/volumeop_parallel.yaml +++ b/sdk/python/tests/compiler/testdata/volumeop_parallel.yaml @@ -1,6 +1,9 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "The first example of the + design doc.", "inputs": [], "name": "VolumeOp Parallel"}' generateName: volumeop-parallel- spec: arguments: diff --git a/sdk/python/tests/compiler/testdata/volumeop_sequential.yaml b/sdk/python/tests/compiler/testdata/volumeop_sequential.yaml index 8fd38be9a00..827d4b2445f 100644 --- a/sdk/python/tests/compiler/testdata/volumeop_sequential.yaml +++ b/sdk/python/tests/compiler/testdata/volumeop_sequential.yaml @@ -1,6 +1,9 @@ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: + annotations: + pipelines.kubeflow.org/pipeline_spec: '{"description": "The third example of the + design doc.", "inputs": [], "name": "VolumeOp Sequential"}' generateName: volumeop-sequential- spec: arguments: diff --git a/sdk/python/tests/components/test_components.py b/sdk/python/tests/components/test_components.py index 047741e199a..2c621a000a5 100644 --- a/sdk/python/tests/components/test_components.py +++ b/sdk/python/tests/components/test_components.py @@ -17,7 +17,6 @@ import unittest from pathlib import Path -sys.path.insert(0, __file__ + '/../../../') import kfp import kfp.components as comp diff --git a/sdk/python/tests/components/test_graph_components.py b/sdk/python/tests/components/test_graph_components.py index dcee65d2096..b9d295546d2 100644 --- a/sdk/python/tests/components/test_graph_components.py +++ b/sdk/python/tests/components/test_graph_components.py @@ -17,7 +17,6 @@ import unittest from pathlib import Path -sys.path.insert(0, __file__ + '/../../../') import kfp.components as comp from kfp.components._structures import ComponentReference, ComponentSpec, ContainerSpec, GraphInputArgument, GraphSpec, InputSpec, InputValuePlaceholder, GraphImplementation, OutputPathPlaceholder, OutputSpec, TaskOutputArgument, TaskSpec diff --git a/sdk/python/tests/components/test_python_op.py b/sdk/python/tests/components/test_python_op.py index 5a90df81663..4e495484432 100644 --- a/sdk/python/tests/components/test_python_op.py +++ b/sdk/python/tests/components/test_python_op.py @@ -184,6 +184,81 @@ def add_multiply_two_numbers(a: float, b: float) -> NamedTuple('DummyName', [('s self.helper_test_2_in_2_out_component_using_local_call(func, op, output_names=['sum', 'product']) + def test_extract_component_interface(self): + from typing import NamedTuple + def my_func( + required_param, + int_param: int = 42, + float_param : float = 3.14, + str_param : str = 'string', + bool_param : bool = True, + none_param = None, + custom_type_param: 'Custom type' = None, + ) -> NamedTuple('DummyName', [ + #('required_param',), # All typing.NamedTuple fields must have types + ('int_param', int), + ('float_param', float), + ('str_param', str), + ('bool_param', bool), + #('custom_type_param', 'Custom type'), #SyntaxError: Forward reference must be an expression -- got 'Custom type' + ('custom_type_param', 'CustomType'), + ] + ): + '''Function docstring''' + pass + + component_spec = comp._python_op._extract_component_interface(my_func) + + from kfp.components._structures import InputSpec, OutputSpec + self.assertEqual( + component_spec.inputs, + [ + InputSpec(name='required_param'), + InputSpec(name='int_param', type='int', default='42', optional=True), + InputSpec(name='float_param', type='float', default='3.14', optional=True), + InputSpec(name='str_param', type='str', default='string', optional=True), + InputSpec(name='bool_param', type='bool', default='True', optional=True), + InputSpec(name='none_param', optional=True), # No default='None' + InputSpec(name='custom_type_param', type='Custom type', optional=True), + ] + ) + self.assertEqual( + component_spec.outputs, + [ + OutputSpec(name='int_param', type='int'), + OutputSpec(name='float_param', type='float'), + OutputSpec(name='str_param', type='str'), + OutputSpec(name='bool_param', type='bool'), + #OutputSpec(name='custom_type_param', type='Custom type', default='None'), + OutputSpec(name='custom_type_param', type='CustomType'), + ] + ) + + self.maxDiff = None + self.assertDictEqual( + component_spec.to_dict(), + { + 'name': 'My func', + 'description': 'Function docstring\n', + 'inputs': [ + {'name': 'required_param'}, + {'name': 'int_param', 'type': 'int', 'default': '42', 'optional': True}, + {'name': 'float_param', 'type': 'float', 'default': '3.14', 'optional': True}, + {'name': 'str_param', 'type': 'str', 'default': 'string', 'optional': True}, + {'name': 'bool_param', 'type': 'bool', 'default': 'True', 'optional': True}, + {'name': 'none_param', 'optional': True}, # No default='None' + {'name': 'custom_type_param', 'type': 'Custom type', 'optional': True}, + ], + 'outputs': [ + {'name': 'int_param', 'type': 'int'}, + {'name': 'float_param', 'type': 'float'}, + {'name': 'str_param', 'type': 'str'}, + {'name': 'bool_param', 'type': 'bool'}, + {'name': 'custom_type_param', 'type': 'CustomType'}, + ] + } + ) + @unittest.skip #TODO: #Simplified multi-output syntax is not implemented yet def test_func_to_container_op_multiple_named_typed_outputs_using_list_syntax(self): def add_multiply_two_numbers(a: float, b: float) -> [('sum', float), ('product', float)]: diff --git a/sdk/python/tests/components/test_structure_model_base.py b/sdk/python/tests/components/test_structure_model_base.py index 5077d68e83e..95bd8bad0df 100644 --- a/sdk/python/tests/components/test_structure_model_base.py +++ b/sdk/python/tests/components/test_structure_model_base.py @@ -230,5 +230,20 @@ def test_handle_from_to_dict_for_union_dict_class(self): TestModel1.from_dict({'prop_0': '', 'prop_5': [val5.to_dict(), None]}) + def test_handle_comparisons(self): + class A(ModelBase): + def __init__(self, a, b): + super().__init__(locals()) + + self.assertEqual(A(1, 2), A(1, 2)) + self.assertNotEqual(A(1, 2), A(1, 3)) + + class B(ModelBase): + def __init__(self, a, b): + super().__init__(locals()) + + self.assertNotEqual(A(1, 2), B(1, 2)) + + if __name__ == '__main__': unittest.main() diff --git a/sdk/python/tests/dsl/component_tests.py b/sdk/python/tests/dsl/component_tests.py index 05c3b6cd75f..73599c2faf8 100644 --- a/sdk/python/tests/dsl/component_tests.py +++ b/sdk/python/tests/dsl/component_tests.py @@ -63,7 +63,7 @@ def a_op(field_l: Integer()) -> {'field_m': GCSPath(), 'field_n': {'customized_t @component def b_op(field_x: {'customized_type': {'property_a': 'value_a', 'property_b': 'value_b'}}, - field_y: 'GcsUri', + field_y: 'GcsUri', # noqa: F821 TODO field_z: GCSPath()) -> {'output_model_uri': 'GcsUri'}: return ContainerOp( name = 'operator b', diff --git a/sdk/python/tests/dsl/pipeline_param_tests.py b/sdk/python/tests/dsl/pipeline_param_tests.py index deb5eef8708..5d6288d8290 100644 --- a/sdk/python/tests/dsl/pipeline_param_tests.py +++ b/sdk/python/tests/dsl/pipeline_param_tests.py @@ -30,13 +30,13 @@ def test_str_repr(self): """Test string representation.""" p = PipelineParam(name='param1', op_name='op1') - self.assertEqual('{{pipelineparam:op=op1;name=param1;value=;type=;}}', str(p)) + self.assertEqual('{{pipelineparam:op=op1;name=param1}}', str(p)) p = PipelineParam(name='param2') - self.assertEqual('{{pipelineparam:op=;name=param2;value=;type=;}}', str(p)) + self.assertEqual('{{pipelineparam:op=;name=param2}}', str(p)) p = PipelineParam(name='param3', value='value3') - self.assertEqual('{{pipelineparam:op=;name=param3;value=value3;type=;}}', str(p)) + self.assertEqual('{{pipelineparam:op=;name=param3}}', str(p)) def test_extract_pipelineparams(self): """Test _extract_pipeleineparams.""" diff --git a/test/.gitignore b/test/.gitignore new file mode 100644 index 00000000000..fb86d0d665c --- /dev/null +++ b/test/.gitignore @@ -0,0 +1,2 @@ +# temporary folder used in tests +bin diff --git a/test/README.md b/test/README.md index d5fcaf9878a..8b605edee66 100644 --- a/test/README.md +++ b/test/README.md @@ -65,6 +65,26 @@ However you can keep them by providing additional parameter. argo submit integration_test_gke.yaml -p branch="my-branch" -p cleanup="false" ``` +### Run presubmit-tests-with-pipeline-deployment.sh locally + +Run the following commands from root of kubeflow/pipelines repo. +``` +#$PULL_PULL_SHA and $WORKSPACE are env variables set by Prow +export PULL_PULL_SHA=pull-sha-placeholder +export WORKSPACE=$(pwd) # root of kubeflow/pipelines git repo +export SA_KEY_FILE=PATH/TO/YOUR/GCP/PROJECT/SERVICE/ACCOUNT/KEY +# (optional) uncomment the following to keep reusing the same cluster +# export TEST_CLUSTER=YOUR_PRECONFIGURED_CLUSTER_NAME +# (optional) uncomment the following to disable built image caching +# export DISABLE_IMAGE_CACHING=true + +./test/presubmit-tests-with-pipeline-deployment.sh \ + --workflow_file e2e_test_gke_v2.yaml \ # You can specify other workflows you want to test too. + --test_result_folder ${FOLDER_NAME_TO_HOLD_TEST_RESULT} \ + --test_result_bucket ${YOUR_GCS_TEST_RESULT_BUCKET} \ + --project ${YOUR_GCS_PROJECT} +``` + ## Troubleshooting **Q: Why is my test taking so long on GKE?** diff --git a/test/build-images.sh b/test/build-images.sh new file mode 100755 index 00000000000..472c219948e --- /dev/null +++ b/test/build-images.sh @@ -0,0 +1,59 @@ +#!/bin/bash +# +# Copyright 2018 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -ex + +IMAGE_BUILDER_ARG="" +if [ "$PROJECT" != "ml-pipeline-test" ]; then + COPIED_IMAGE_BUILDER_IMAGE=${GCR_IMAGE_BASE_DIR}/image-builder + echo "Copy image builder image to ${COPIED_IMAGE_BUILDER_IMAGE}" + yes | gcloud container images add-tag \ + gcr.io/ml-pipeline-test/image-builder:v20181128-0.1.3-rc.1-109-ga5a14dc-e3b0c4 \ + ${COPIED_IMAGE_BUILDER_IMAGE}:latest + IMAGE_BUILDER_ARG="-p image-builder-image=${COPIED_IMAGE_BUILDER_IMAGE}" +fi + +# Image caching can be turned off by setting $DISABLE_IMAGE_CACHING env flag. +# Note that GCR_IMAGE_BASE_DIR contains commit hash, so whenever there's a code +# change, we won't use caches for sure. +BUILT_IMAGES=$(gcloud container images list --repository=${GCR_IMAGE_BASE_DIR}) +if + test -z "$DISABLE_IMAGE_CACHING" && \ + echo "$BUILT_IMAGES" | grep api-server && \ + echo "$BUILT_IMAGES" | grep frontend && \ + echo "$BUILT_IMAGES" | grep scheduledworkflow && \ + echo "$BUILT_IMAGES" | grep persistenceagent; +then + echo "docker images for api-server, frontend, scheduledworkflow and \ + persistenceagent are already built in ${GCR_IMAGE_BASE_DIR}." +else + echo "submitting argo workflow to build docker images for commit ${PULL_PULL_SHA}..." + # Build Images + ARGO_WORKFLOW=`argo submit ${DIR}/build_image.yaml \ + -p image-build-context-gcs-uri="$remote_code_archive_uri" \ + ${IMAGE_BUILDER_ARG} \ + -p api-image="${GCR_IMAGE_BASE_DIR}/api-server" \ + -p frontend-image="${GCR_IMAGE_BASE_DIR}/frontend" \ + -p scheduledworkflow-image="${GCR_IMAGE_BASE_DIR}/scheduledworkflow" \ + -p persistenceagent-image="${GCR_IMAGE_BASE_DIR}/persistenceagent" \ + -n ${NAMESPACE} \ + --serviceaccount test-runner \ + -o name + ` + echo "build docker images workflow submitted successfully" + source "${DIR}/check-argo-status.sh" + echo "build docker images workflow completed" +fi diff --git a/test/build_image.yaml b/test/build_image.yaml index 44337eba1d7..79e8fa20e76 100644 --- a/test/build_image.yaml +++ b/test/build_image.yaml @@ -25,6 +25,8 @@ spec: arguments: parameters: - name: image-build-context-gcs-uri + - name: image-builder-image + value: gcr.io/ml-pipeline-test/image-builder:v20181128-0.1.3-rc.1-109-ga5a14dc-e3b0c4 - name: api-image - name: frontend-image - name: scheduledworkflow-image @@ -105,7 +107,7 @@ spec: valueFrom: path: /outputs/strict-image-name/file container: - image: gcr.io/ml-pipeline-test/image-builder:v20181128-0.1.3-rc.1-109-ga5a14dc-e3b0c4 + image: "{{workflow.parameters.image-builder-image}}" imagePullPolicy: 'Always' args: [ "--image-build-context-gcs-uri", "{{inputs.parameters.image-build-context-gcs-uri}}", diff --git a/test/check-argo-status.sh b/test/check-argo-status.sh index 64f99841825..80c4d4803cb 100755 --- a/test/check-argo-status.sh +++ b/test/check-argo-status.sh @@ -25,7 +25,8 @@ echo "check status of argo workflow $ARGO_WORKFLOW...." # probing the argo workflow status until it completed. Timeout after 30 minutes for i in $(seq 1 ${PULL_ARGO_WORKFLOW_STATUS_MAX_ATTEMPT}) do - WORKFLOW_STATUS=`kubectl get workflow $ARGO_WORKFLOW -n ${NAMESPACE} --show-labels` + WORKFLOW_STATUS=`kubectl get workflow $ARGO_WORKFLOW -n ${NAMESPACE} --show-labels 2>&1` \ + || echo kubectl get workflow failed with "$WORKFLOW_STATUS" # Tolerate temporary network failure during kubectl get workflow echo $WORKFLOW_STATUS | grep ${WORKFLOW_COMPLETE_KEYWORD} && s=0 && break || s=$? && printf "Workflow ${ARGO_WORKFLOW} is not finished.\n${WORKFLOW_STATUS}\nSleep for 20 seconds...\n" && sleep 20 done @@ -54,4 +55,4 @@ if [[ $WORKFLOW_STATUS = *"${WORKFLOW_FAILED_KEYWORD}"* ]]; then exit 1 else argo get ${ARGO_WORKFLOW} -n ${NAMESPACE} -fi \ No newline at end of file +fi diff --git a/test/component_test.yaml b/test/component_test.yaml index 6fa7bffba2c..4ef88ab2785 100644 --- a/test/component_test.yaml +++ b/test/component_test.yaml @@ -25,6 +25,8 @@ spec: arguments: parameters: - name: image-build-context-gcs-uri + - name: image-builder-image + value: gcr.io/ml-pipeline-test/image-builder:v20181128-0.1.3-rc.1-109-ga5a14dc-e3b0c4 - name: commit-sha - name: component-image-prefix - name: target-image-prefix @@ -158,7 +160,7 @@ spec: valueFrom: path: /outputs/strict-image-name/file container: - image: gcr.io/ml-pipeline-test/image-builder:v20181128-0.1.3-rc.1-109-ga5a14dc-e3b0c4 + image: "{{workflow.parameters.image-builder-image}}" imagePullPolicy: 'Always' args: [ "--image-build-context-gcs-uri", "{{inputs.parameters.image-build-context-gcs-uri}}", diff --git a/test/deploy-cluster.sh b/test/deploy-cluster.sh new file mode 100755 index 00000000000..10f6e0e5d2a --- /dev/null +++ b/test/deploy-cluster.sh @@ -0,0 +1,73 @@ +#!/bin/bash +# +# Copyright 2018 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -ex + +# Specify TEST_CLUSTER env variable to use an existing cluster. +TEST_CLUSTER_PREFIX=${WORKFLOW_FILE%.*} +TEST_CLUSTER_DEFAULT=$(echo $TEST_CLUSTER_PREFIX | cut -d _ -f 1)-${PULL_PULL_SHA:0:7}-${RANDOM} +TEST_CLUSTER=${TEST_CLUSTER:-${TEST_CLUSTER_DEFAULT}} +SHOULD_CLEANUP_CLUSTER=false + +function clean_up { + set +e # the following clean up commands shouldn't exit on error + + echo "Status of pods before clean up:" + kubectl get pods --all-namespaces + + echo "Clean up..." + if [ $SHOULD_CLEANUP_CLUSTER == true ]; then + # --async doesn't wait for this operation to complete, so we can get test + # results faster + yes | gcloud container clusters delete ${TEST_CLUSTER} --async + fi +} +trap clean_up EXIT SIGINT SIGTERM + +cd ${DIR} +# test if ${TEST_CLUSTER} exists or not +if gcloud container clusters describe ${TEST_CLUSTER} &>/dev/null; then + echo "Use existing test cluster: ${TEST_CLUSTER}" +else + echo "Creating a new test cluster: ${TEST_CLUSTER}" + SHOULD_CLEANUP_CLUSTER=true + # "storage-rw" is needed to allow VMs to push to gcr.io + # reference: https://cloud.google.com/compute/docs/access/service-accounts#accesscopesiam + SCOPE_ARG="--scopes=storage-rw" + # Machine type and cluster size is the same as kubeflow deployment to + # easily compare performance. We can reduce usage later. + NODE_POOL_CONFIG_ARG="--num-nodes=2 --machine-type=n1-standard-8 \ + --enable-autoscaling --max-nodes=8 --min-nodes=2" + gcloud container clusters create ${TEST_CLUSTER} ${SCOPE_ARG} ${NODE_POOL_CONFIG_ARG} +fi + +gcloud container clusters get-credentials ${TEST_CLUSTER} + +# when we reuse a cluster when debugging, clean up its kfp installation first +# this does nothing with a new cluster +kubectl delete namespace ${NAMESPACE} --wait || echo "No need to delete ${NAMESPACE} namespace. It doesn't exist." +kubectl create namespace ${NAMESPACE} --dry-run -o yaml | kubectl apply -f - + +if [ -z $SA_KEY_FILE ]; then + SA_KEY_FILE=${DIR}/key.json + # The service account key is for default VM service account. + # ref: https://cloud.google.com/compute/docs/access/service-accounts#compute_engine_default_service_account + # It was generated by the following command + # `gcloud iam service-accounts keys create $SA_KEY_FILE --iam-account ${VM_SERVICE_ACCOUNT}` + # Because there's a limit of 10 keys per service account, we are reusing the same key stored in the following bucket. + gsutil cp "gs://ml-pipeline-test-keys/ml-pipeline-test-sa-key.json" $SA_KEY_FILE +fi +kubectl create secret -n ${NAMESPACE} generic user-gcp-sa --from-file=user-gcp-sa.json=$SA_KEY_FILE --dry-run -o yaml | kubectl apply -f - diff --git a/test/deploy-kubeflow.sh b/test/deploy-kubeflow.sh deleted file mode 100755 index 8c2c7a3fb42..00000000000 --- a/test/deploy-kubeflow.sh +++ /dev/null @@ -1,59 +0,0 @@ -#!/bin/bash -# -# Copyright 2018 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -set -ex - -TEST_CLUSTER_PREFIX=${WORKFLOW_FILE%.*} -TEST_CLUSTER=$(echo $TEST_CLUSTER_PREFIX | cut -d _ -f 1)-${PULL_PULL_SHA:0:7}-${RANDOM} - -# Install ksonnet -KS_VERSION="0.13.0" - -curl -LO https://github.com/ksonnet/ksonnet/releases/download/v${KS_VERSION}/ks_${KS_VERSION}_linux_amd64.tar.gz -tar -xzf ks_${KS_VERSION}_linux_amd64.tar.gz -chmod +x ./ks_${KS_VERSION}_linux_amd64/ks -mv ./ks_${KS_VERSION}_linux_amd64/ks /usr/local/bin/ - -## Download latest kubeflow release source code -KUBEFLOW_SRC=${DIR}/kubeflow_latest_release -mkdir ${KUBEFLOW_SRC} -cd ${KUBEFLOW_SRC} -export KUBEFLOW_TAG=pipelines -curl https://raw.githubusercontent.com/kubeflow/kubeflow/${KUBEFLOW_TAG}/scripts/download.sh | bash - -export CLIENT_ID=${RANDOM} -export CLIENT_SECRET=${RANDOM} -KFAPP=${TEST_CLUSTER} - -function clean_up { - echo "Clean up..." - cd ${DIR}/${KFAPP} - ${KUBEFLOW_SRC}/scripts/kfctl.sh delete all - # delete the storage - gcloud deployment-manager --project=${PROJECT} deployments delete ${KFAPP}-storage --quiet -} -trap clean_up EXIT SIGINT SIGTERM - -cd ${DIR} -${KUBEFLOW_SRC}/scripts/kfctl.sh init ${KFAPP} --platform ${PLATFORM} --project ${PROJECT} --skipInitProject - -cd ${KFAPP} -${KUBEFLOW_SRC}/scripts/kfctl.sh generate platform -${KUBEFLOW_SRC}/scripts/kfctl.sh apply platform -${KUBEFLOW_SRC}/scripts/kfctl.sh generate k8s -${KUBEFLOW_SRC}/scripts/kfctl.sh apply k8s - -gcloud container clusters get-credentials ${TEST_CLUSTER} diff --git a/test/deploy-pipeline-lite.sh b/test/deploy-pipeline-lite.sh new file mode 100755 index 00000000000..ae141d680be --- /dev/null +++ b/test/deploy-pipeline-lite.sh @@ -0,0 +1,47 @@ +#!/bin/bash +# +# Copyright 2018 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -ex + +if ! which kustomize; then + # Download kustomize cli tool + TOOL_DIR=${DIR}/bin + mkdir -p ${TOOL_DIR} + wget https://github.com/kubernetes-sigs/kustomize/releases/download/v3.1.0/kustomize_3.1.0_linux_amd64 -O ${TOOL_DIR}/kustomize + chmod +x ${TOOL_DIR}/kustomize + PATH=${PATH}:${TOOL_DIR} +fi + +# delete argo first because KFP comes with argo too +kubectl delete namespace argo --wait || echo "No argo installed" + +KFP_MANIFEST_DIR=${DIR}/manifests +pushd ${KFP_MANIFEST_DIR} + +# This is the recommended approach to do this. +# reference: https://github.com/kubernetes-sigs/kustomize/blob/master/docs/eschewedFeatures.md#build-time-side-effects-from-cli-args-or-env-variables +kustomize edit set image gcr.io/ml-pipeline/api-server=${GCR_IMAGE_BASE_DIR}/api-server:latest +kustomize edit set image gcr.io/ml-pipeline/persistenceagent=${GCR_IMAGE_BASE_DIR}/persistenceagent:latest +kustomize edit set image gcr.io/ml-pipeline/scheduledworkflow=${GCR_IMAGE_BASE_DIR}/scheduledworkflow:latest +kustomize edit set image gcr.io/ml-pipeline/frontend=${GCR_IMAGE_BASE_DIR}/frontend:latest +cat kustomization.yaml + +kustomize build . | kubectl apply -f - +# show current info +echo "Status of pods after kubectl apply" +kubectl get pods -n ${NAMESPACE} + +popd diff --git a/test/deploy-pipeline.sh b/test/deploy-pipeline.sh deleted file mode 100755 index 45063c41c53..00000000000 --- a/test/deploy-pipeline.sh +++ /dev/null @@ -1,64 +0,0 @@ -#!/bin/bash -# -# Copyright 2018 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -set -ex - - -usage() -{ - echo "usage: deploy.sh - [--gcr_image_base_dir the gcr image base directory including images such as apiImage and persistenceAgentImage] - [--gcr_image_tag the tags for images such as apiImage and persistenceAgentImage] - [-h help]" -} -GCR_IMAGE_TAG=latest - -while [ "$1" != "" ]; do - case $1 in - --gcr_image_base_dir ) shift - GCR_IMAGE_BASE_DIR=$1 - ;; - --gcr_image_tag ) shift - GCR_IMAGE_TAG=$1 - ;; - -h | --help ) usage - exit - ;; - * ) usage - exit 1 - esac - shift -done - -cd ${DIR}/${KFAPP} - -## Update pipeline component image -pushd ks_app -# Delete pipeline component first before applying so we guarantee the pipeline component is new. -ks delete default -c pipeline -sleep 60s - -ks param set pipeline apiImage ${GCR_IMAGE_BASE_DIR}/api-server:${GCR_IMAGE_TAG} -ks param set pipeline persistenceAgentImage ${GCR_IMAGE_BASE_DIR}/persistenceagent:${GCR_IMAGE_TAG} -ks param set pipeline scheduledWorkflowImage ${GCR_IMAGE_BASE_DIR}/scheduledworkflow:${GCR_IMAGE_TAG} -ks param set pipeline uiImage ${GCR_IMAGE_BASE_DIR}/frontend:${GCR_IMAGE_TAG} -# Swap the metadata/artifact storage PD to avoid reusing the old data. -# We should remove this hack when we deprecate ksonnet. -# See https://github.com/kubeflow/pipelines/pull/1805#issuecomment-520204987 for context -ks param set pipeline minioPd ${KFAPP}-storage-metadata-store -ks param set pipeline mysqlPd ${KFAPP}-storage-artifact-store -ks apply default -c pipeline -popd diff --git a/test/e2e_test_gke_v2.yaml b/test/e2e_test_gke_v2.yaml index 58c6347a799..6873326974c 100644 --- a/test/e2e_test_gke_v2.yaml +++ b/test/e2e_test_gke_v2.yaml @@ -25,6 +25,8 @@ spec: arguments: parameters: - name: image-build-context-gcs-uri + - name: image-builder-image + value: gcr.io/ml-pipeline-test/image-builder:v20181128-0.1.3-rc.1-109-ga5a14dc-e3b0c4 - name: target-image-prefix - name: test-results-gcs-dir - name: initialization-test-image-suffix @@ -196,7 +198,7 @@ spec: valueFrom: path: /outputs/strict-image-name/file container: - image: gcr.io/ml-pipeline-test/image-builder:v20181128-0.1.3-rc.1-109-ga5a14dc-e3b0c4 + image: "{{workflow.parameters.image-builder-image}}" imagePullPolicy: 'Always' args: [ "--image-build-context-gcs-uri", "{{inputs.parameters.image-build-context-gcs-uri}}", diff --git a/test/install-argo.sh b/test/install-argo.sh index 6e2ef523d19..fad8af483b0 100755 --- a/test/install-argo.sh +++ b/test/install-argo.sh @@ -20,20 +20,25 @@ set -ex kubectl config set-context $(kubectl config current-context) --namespace=default echo "Add necessary cluster role bindings" ACCOUNT=$(gcloud info --format='value(config.account)') -kubectl create clusterrolebinding PROW_BINDING --clusterrole=cluster-admin --user=$ACCOUNT -kubectl create clusterrolebinding DEFAULT_BINDING --clusterrole=cluster-admin --serviceaccount=default:default +kubectl create clusterrolebinding PROW_BINDING --clusterrole=cluster-admin --user=$ACCOUNT --dry-run -o yaml | kubectl apply -f - +kubectl create clusterrolebinding DEFAULT_BINDING --clusterrole=cluster-admin --serviceaccount=default:default --dry-run -o yaml | kubectl apply -f - -echo "install argo" ARGO_VERSION=v2.3.0 -mkdir -p ~/bin/ -export PATH=~/bin/:$PATH -curl -sSL -o ~/bin/argo https://github.com/argoproj/argo/releases/download/$ARGO_VERSION/argo-linux-amd64 -chmod +x ~/bin/argo -#kubectl create ns argo -#kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo/$ARGO_VERSION/manifests/install.yaml + +# if argo is not installed +if ! which argo; then + echo "install argo" + mkdir -p ~/bin/ + export PATH=~/bin/:$PATH + curl -sSL -o ~/bin/argo https://github.com/argoproj/argo/releases/download/$ARGO_VERSION/argo-linux-amd64 + chmod +x ~/bin/argo +fi + +kubectl create ns argo --dry-run -o yaml | kubectl apply -f - +kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo/$ARGO_VERSION/manifests/install.yaml # Some workflows are deployed to the non-default namespace where the GCP credential secret is stored # In this case, the default service account in that namespace doesn't have enough permission echo "add service account for running the test workflow" -kubectl create serviceaccount test-runner -n ${NAMESPACE} -kubectl create clusterrolebinding test-admin-binding --clusterrole=cluster-admin --serviceaccount=${NAMESPACE}:test-runner +kubectl create serviceaccount test-runner -n ${NAMESPACE} --dry-run -o yaml | kubectl apply -f - +kubectl create clusterrolebinding test-admin-binding --clusterrole=cluster-admin --serviceaccount=${NAMESPACE}:test-runner --dry-run -o yaml | kubectl apply -f - diff --git a/test/manifests/kustomization.yaml b/test/manifests/kustomization.yaml new file mode 100644 index 00000000000..bc55ef3361c --- /dev/null +++ b/test/manifests/kustomization.yaml @@ -0,0 +1,7 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +# Actual image overrides will be added in test scripts. +images: [] +resources: +- ../../manifests/kustomize/env/dev diff --git a/test/minikube/OWNERS b/test/minikube/OWNERS deleted file mode 100644 index 23d368bc30f..00000000000 --- a/test/minikube/OWNERS +++ /dev/null @@ -1,6 +0,0 @@ -approvers: - - vicaire - - IronPan -reviewers: - - vicaire - - IronPan \ No newline at end of file diff --git a/test/minikube/install_and_start_minikube_without_vm.sh b/test/minikube/install_and_start_minikube_without_vm.sh deleted file mode 100755 index 88736744ad4..00000000000 --- a/test/minikube/install_and_start_minikube_without_vm.sh +++ /dev/null @@ -1,46 +0,0 @@ -#!/bin/bash -ex -# -# Copyright 2018 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -#This scripts installs Minikube, configures it to run without root access and starts it without VM -#See https://github.com/kubernetes/minikube#linux-continuous-integration-without-vm-support - -MINIKUBE_VERSION=${MINIKUBE_VERSION:-latest} -KUBECTL_VERSION=${KUBECTL_VERSION:-$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)} -KUBERNETES_VERSION=${KUBERNETES_VERSION:-v1.12.2} - -curl -Lo minikube https://storage.googleapis.com/minikube/releases/${MINIKUBE_VERSION}/minikube-linux-amd64 && chmod +x minikube && sudo cp minikube /usr/local/bin/ && rm minikube -curl -Lo kubectl https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl && chmod +x kubectl && sudo cp kubectl /usr/local/bin/ && rm kubectl - -export MINIKUBE_WANTUPDATENOTIFICATION=false -export MINIKUBE_WANTREPORTERRORPROMPT=false -export MINIKUBE_HOME=$HOME -export CHANGE_MINIKUBE_NONE_USER=true -mkdir -p $HOME/.kube -touch $HOME/.kube/config - -export KUBECONFIG=$HOME/.kube/config -sudo -E /usr/local/bin/minikube start --vm-driver=none --kubernetes-version=$KUBERNETES_VERSION - -# this for loop waits until kubectl can access the api server that Minikube has created -for i in {1..150}; do # timeout for 5 minutes - kubectl get po &> /dev/null - if [ $? -ne 1 ]; then - break - fi - sleep 2 -done - -# kubectl commands are now able to interact with Minikube cluster diff --git a/test/minikube/install_docker.sh b/test/minikube/install_docker.sh deleted file mode 100755 index 7f2cb0110ac..00000000000 --- a/test/minikube/install_docker.sh +++ /dev/null @@ -1,34 +0,0 @@ -#!/bin/bash -ex -# -# Copyright 2018 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -DOCKER_VERSION=18.06.1~ce~3-0~$(. /etc/os-release; echo "$ID") - -sudo apt-get update -y -sudo apt-get install -y \ - apt-transport-https \ - ca-certificates \ - curl \ - software-properties-common - -curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg | sudo apt-key add - - -sudo add-apt-repository \ - "deb [arch=amd64] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") \ - $(lsb_release -cs) \ - stable" - -sudo apt-get update -y -sudo apt-get install docker-ce="${DOCKER_VERSION}" -y diff --git a/test/presubmit-tests-gce-minikube.sh b/test/presubmit-tests-gce-minikube.sh deleted file mode 100755 index 1c683b0470f..00000000000 --- a/test/presubmit-tests-gce-minikube.sh +++ /dev/null @@ -1,87 +0,0 @@ -#!/bin/bash -ex -# -# Copyright 2018 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -#set -o nounset - -#This script runs the presubmit tests on a Minikube cluster. -#The script creates a new GCE VM, sets up Minikube, copies the SSH keys and repo and and then runs the tests. -#This script is usually run from the PROW cluster, but it can be run locally: -#ssh_key_file=~/.ssh/id_rsa WORKSPACE=/var/tmp PULL_PULL_SHA=master test/presubmit-tests-gce-minikube.sh --workflow_file integration_test_gke.yaml --test_result_folder api_integration_test - -WORKSPACE=$WORKSPACE -PULL_PULL_SHA=$PULL_PULL_SHA -ARTIFACT_DIR=$WORKSPACE/_artifacts -PROJECT_ID=${PROJECT_ID:-ml-pipeline-test} -ZONE=${ZONE:-us-west1-a} - -repo_test_dir=$(dirname $0) - -instance_name=${instance_name:-test-minikube-${PULL_PULL_SHA:0:6}-$(date +%s)-$(echo "$@" | md5sum | cut -c 1-6)} - -firewall_rule_name=allow-prow-ssh-$instance_name - -# activating the service account -gcloud auth activate-service-account --key-file="${GOOGLE_APPLICATION_CREDENTIALS}" - -#Function to delete VM -function delete_vm { - if [ "$keep_created_vm" != true ]; then - echo "Deleting VM $instance_name" - gcloud compute instances delete $instance_name --zone=$ZONE --quiet - fi - echo "Deleting the SSH firewall rule $firewall_rule_name" - gcloud compute firewall-rules delete $firewall_rule_name -} - -#Setting the exit handler to delete VM. The VM will be deleted when the script exists (either completes or fails) -#TODO: Find a more resilent way to clean up VMs. Right now the VM is not deleted if the machine running this script fails. (See https://github.com/kubeflow/pipelines/issues/1064) -trap delete_vm EXIT - -#Creating the VM -gcloud config set project $PROJECT_ID -gcloud config set compute/zone $ZONE - -machine_type=n1-standard-16 -boot_disk_size=200GB - -gcloud compute instances create $instance_name --zone=$ZONE --machine-type=$machine_type --boot-disk-size=$boot_disk_size --scopes=storage-rw --tags=presubmit-test-vm - -#Adding firewall entry that allows the current instance to access the newly created VM using SSH. -#This is needed for cases when the current instance is in different project (e.g. in the Prow cluster project) -self_external_ip=$(curl -sSL "http://metadata/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip" -H "Metadata-Flavor: Google") -gcloud compute firewall-rules create $firewall_rule_name --allow tcp:22 --source-ranges=${self_external_ip}/32 --target-tags=presubmit-test-vm - -#Workaround the problems with prow cluster and GCE SSH access. -#Prow tests run as root. GCE instances do not allow SSH access for root. -if [ "$(whoami)" == root ]; then - export USER=not-root -fi - -#Copy service account keys -gcloud compute scp --zone=$ZONE --verbosity=error "$GOOGLE_APPLICATION_CREDENTIALS" $instance_name:"~/service-account.json" - -#Copy repo -tar --directory=.. -cz pipelines | gcloud compute ssh --zone=$ZONE $instance_name -- tar -xz #Assumes that the current directory on target VM is ~ - -#Installing software on VM -gcloud compute ssh --zone=$ZONE $instance_name -- "~/pipelines/test/minikube/install_docker_minikube_argo.sh" - -#Running the presubmit tests -gcloud compute ssh --zone=$ZONE $instance_name -- PULL_PULL_SHA="$PULL_PULL_SHA" PULL_BASE_SHA="$PULL_BASE_SHA" WORKSPACE="~/${WORKSPACE}" GOOGLE_APPLICATION_CREDENTIALS="~/service-account.json" "~/pipelines/test/presubmit-tests.sh" --cluster-type none "$@" - -#Copy back the artifacts -mkdir -p "${ARTIFACT_DIR}" -gcloud compute scp --zone=$ZONE --verbosity=error --recurse $instance_name:"~/${ARTIFACT_DIR}/*" "${ARTIFACT_DIR}/" diff --git a/test/presubmit-tests-with-pipeline-deployment.sh b/test/presubmit-tests-with-pipeline-deployment.sh index 816763c9d23..d1a506e06e6 100755 --- a/test/presubmit-tests-with-pipeline-deployment.sh +++ b/test/presubmit-tests-with-pipeline-deployment.sh @@ -20,6 +20,7 @@ usage() { echo "usage: deploy.sh [--platform the deployment platform. Valid values are: [gcp, minikube]. Default is gcp.] + [--project the gcp project. Default is ml-pipeline-test. Only used when platform is gcp.] [--workflow_file the file name of the argo workflow to run] [--test_result_bucket the gcs bucket that argo workflow store the result to. Default is ml-pipeline-test [--test_result_folder the gcs folder that argo workflow store the result to. Always a relative directory to gs:///[PULL_SHA]] @@ -30,7 +31,6 @@ usage() PLATFORM=gcp PROJECT=ml-pipeline-test TEST_RESULT_BUCKET=ml-pipeline-test -GCR_IMAGE_BASE_DIR=gcr.io/ml-pipeline-test/${PULL_PULL_SHA} TIMEOUT_SECONDS=1800 NAMESPACE=kubeflow @@ -39,6 +39,9 @@ while [ "$1" != "" ]; do --platform ) shift PLATFORM=$1 ;; + --project ) shift + PROJECT=$1 + ;; --workflow_file ) shift WORKFLOW_FILE=$1 ;; @@ -61,40 +64,34 @@ while [ "$1" != "" ]; do done # Variables +GCR_IMAGE_BASE_DIR=gcr.io/${PROJECT}/${PULL_PULL_SHA} TEST_RESULTS_GCS_DIR=gs://${TEST_RESULT_BUCKET}/${PULL_PULL_SHA}/${TEST_RESULT_FOLDER} DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" > /dev/null && pwd)" +# Configure `time` command output format. +TIMEFORMAT="[test-timing] It took %lR." + echo "presubmit test starts" -source "${DIR}/test-prep.sh" +time source "${DIR}/test-prep.sh" +echo "test env prepared" -# Deploy Kubeflow -source "${DIR}/deploy-kubeflow.sh" +time source "${DIR}/deploy-cluster.sh" +echo "cluster deployed" # Install Argo CLI and test-runner service account -source "${DIR}/install-argo.sh" +time source "${DIR}/install-argo.sh" +echo "argo installed" -# Build Images -echo "submitting argo workflow to build docker images for commit ${PULL_PULL_SHA}..." -ARGO_WORKFLOW=`argo submit ${DIR}/build_image.yaml \ --p image-build-context-gcs-uri="$remote_code_archive_uri" \ --p api-image="${GCR_IMAGE_BASE_DIR}/api-server" \ --p frontend-image="${GCR_IMAGE_BASE_DIR}/frontend" \ --p scheduledworkflow-image="${GCR_IMAGE_BASE_DIR}/scheduledworkflow" \ --p persistenceagent-image="${GCR_IMAGE_BASE_DIR}/persistenceagent" \ --n ${NAMESPACE} \ ---serviceaccount test-runner \ --o name -` -echo "build docker images workflow submitted successfully" -source "${DIR}/check-argo-status.sh" -echo "build docker images workflow completed" +time source "${DIR}/build-images.sh" +echo "KFP images built" -# Deploy the pipeline -source ${DIR}/deploy-pipeline.sh --gcr_image_base_dir ${GCR_IMAGE_BASE_DIR} +time source "${DIR}/deploy-pipeline-lite.sh" +echo "KFP lite deployed" echo "submitting argo workflow to run tests for commit ${PULL_PULL_SHA}..." ARGO_WORKFLOW=`argo submit ${DIR}/${WORKFLOW_FILE} \ -p image-build-context-gcs-uri="$remote_code_archive_uri" \ +${IMAGE_BUILDER_ARG} \ -p target-image-prefix="${GCR_IMAGE_BASE_DIR}/" \ -p test-results-gcs-dir="${TEST_RESULTS_GCS_DIR}" \ -p cluster-type="${CLUSTER_TYPE}" \ @@ -102,7 +99,6 @@ ARGO_WORKFLOW=`argo submit ${DIR}/${WORKFLOW_FILE} \ --serviceaccount test-runner \ -o name ` - echo "test workflow submitted successfully" -source "${DIR}/check-argo-status.sh" +time source "${DIR}/check-argo-status.sh" echo "test workflow completed" diff --git a/test/sample-test/run_sample_test.py b/test/sample-test/run_sample_test.py index 9624d2dc952..0e69f3ef007 100644 --- a/test/sample-test/run_sample_test.py +++ b/test/sample-test/run_sample_test.py @@ -96,15 +96,6 @@ def main(): 'steps': '5' } - elif args.testname == 'kubeflow_training_classification': - params = { - 'output': args.output, - 'project': 'ml-pipeline-test', - 'evaluation': 'gs://ml-pipeline-dataset/sample-test/flower/eval15.csv', - 'train': 'gs://ml-pipeline-dataset/sample-test/flower/train30.csv', - 'hidden-layer-size': '10,5', - 'steps': '5' - } elif args.testname == 'xgboost_training_cm': params = { 'output': args.output, @@ -150,22 +141,7 @@ def main(): ###### Validate the results for specific test cases ###### #TODO: Add result check for tfx-cab-classification after launch. - if args.testname == 'kubeflow_training_classification': - cm_tar_path = './confusion_matrix.tar.gz' - utils.get_artifact_in_minio(workflow_json, 'confusion-matrix', cm_tar_path, - 'mlpipeline-ui-metadata') - with tarfile.open(cm_tar_path) as tar_handle: - file_handles = tar_handle.getmembers() - assert len(file_handles) == 1 - - with tar_handle.extractfile(file_handles[0]) as f: - cm_data = json.load(io.TextIOWrapper(f)) - utils.add_junit_test( - test_cases, 'confusion matrix format', - (len(cm_data['outputs'][0]['schema']) == 3), - 'the column number of the confusion matrix output is not equal to three' - ) - elif args.testname == 'xgboost_training_cm': + if args.testname == 'xgboost_training_cm': cm_tar_path = './confusion_matrix.tar.gz' utils.get_artifact_in_minio(workflow_json, 'confusion-matrix', cm_tar_path, 'mlpipeline-ui-metadata') diff --git a/test/sample-test/run_test.sh b/test/sample-test/run_test.sh index 21de19d2eb8..9e292fd8956 100755 --- a/test/sample-test/run_test.sh +++ b/test/sample-test/run_test.sh @@ -203,17 +203,6 @@ xgboost_training_cm_injection() { sed -i "s|gcr.io/ml-pipeline/ml-pipeline-local-roc:\([a-zA-Z0-9_.-]\)\+|${LOCAL_ROC_IMAGE}|g" ${TEST_NAME}.yaml } -################################################################################ -# Utility function to inject correct images to python files for -# kubeflow_training_classification test. -################################################################################ -kubeflow_training_classification_injection() { - sed -i "s|gcr.io/ml-pipeline/ml-pipeline-dataflow-tft:\([a-zA-Z0-9_.-]\)\+|${DATAFLOW_TFT_IMAGE}|g" ${TEST_NAME}.py - sed -i "s|gcr.io/ml-pipeline/ml-pipeline-kubeflow-tf-trainer:\([a-zA-Z0-9_.-]\)\+|${KUBEFLOW_DNNTRAINER_IMAGE}|g" ${TEST_NAME}.py - sed -i "s|gcr.io/ml-pipeline/ml-pipeline-dataflow-tf-predict:\([a-zA-Z0-9_.-]\)\+|${DATAFLOW_PREDICT_IMAGE}|g" ${TEST_NAME}.py - sed -i "s|gcr.io/ml-pipeline/ml-pipeline-local-confusion-matrix:\([a-zA-Z0-9_.-]\)\+|${LOCAL_CONFUSIONMATRIX_IMAGE}|g" ${TEST_NAME}.py -} - if [[ -z "$RESULTS_GCS_DIR" ]]; then usage exit 1 @@ -231,17 +220,7 @@ echo "Run the sample tests..." # Run the tests preparation ${TEST_NAME} -if [[ "${TEST_NAME}" == "kubeflow_training_classification" ]]; then - #TODO(numerology): convert the sed commands to sed -e - # 's|gcr.io/ml-pipeline/|gcr.io/ml-pipeline-test/' and tag replacement. Also - # let the postsubmit tests refer to yaml files. - if [ -n "${DATAFLOW_TFT_IMAGE}" ];then - kubeflow_training_classification_injection - fi - - dsl-compile --py "${TEST_NAME}.py" --output "${TEST_NAME}.yaml" - check_result ${TEST_NAME} -elif [[ "${TEST_NAME}" == "tfx_cab_classification" ]]; then +if [[ "${TEST_NAME}" == "tfx_cab_classification" ]]; then dsl-compile --py "${TEST_NAME}.py" --output "${TEST_NAME}.yaml" if [[ -n "${DATAFLOW_TFT_IMAGE}" ]]; then tfx_cab_classification_injection @@ -268,43 +247,6 @@ elif [[ "${TEST_NAME}" == "parallel_join" ]]; then elif [[ "${TEST_NAME}" == "recursion" ]]; then dsl-compile --py "${TEST_NAME}.py" --output "${TEST_NAME}.yaml" check_result ${TEST_NAME} -elif [[ "${TEST_NAME}" == "kubeflow_pipeline_using_TFX_OSS_components" ]]; then - # CMLE model name format: A name should start with a letter and contain only - # letters, numbers and underscores. - DEPLOYER_MODEL=`cat /proc/sys/kernel/random/uuid` - DEPLOYER_MODEL=Notebook_tfx_taxi_`echo ${DEPLOYER_MODEL//-/_}` - - export LC_ALL=C.UTF-8 - export LANG=C.UTF-8 - if [[ -n "${DATAFLOW_TFT_IMAGE}" ]]; then - papermill --prepare-only -p EXPERIMENT_NAME "${TEST_NAME}-test" -p OUTPUT_DIR \ - ${RESULTS_GCS_DIR} -p PROJECT_NAME ml-pipeline-test \ - -p BASE_IMAGE ${TARGET_IMAGE_PREFIX}pusherbase:dev -p TARGET_IMAGE \ - ${TARGET_IMAGE_PREFIX}pusher:dev -p TARGET_IMAGE_TWO \ - ${TARGET_IMAGE_PREFIX}pusher_two:dev \ - -p KFP_PACKAGE /tmp/kfp.tar.gz -p DEPLOYER_MODEL ${DEPLOYER_MODEL} \ - -p DATAFLOW_TFDV_IMAGE ${DATAFLOW_TFDV_IMAGE} -p DATAFLOW_TFT_IMAGE \ - ${DATAFLOW_TFT_IMAGE} -p DATAFLOW_TFMA_IMAGE ${DATAFLOW_TFMA_IMAGE} -p \ - DATAFLOW_TF_PREDICT_IMAGE ${DATAFLOW_PREDICT_IMAGE} \ - -p KUBEFLOW_TF_TRAINER_IMAGE ${KUBEFLOW_DNNTRAINER_IMAGE} -p \ - KUBEFLOW_DEPLOYER_IMAGE ${KUBEFLOW_DEPLOYER_IMAGE} \ - -p TRAIN_DATA gs://ml-pipeline-dataset/sample-test/taxi-cab-classification/train50.csv \ - -p EVAL_DATA gs://ml-pipeline-dataset/sample-test/taxi-cab-classification/eval20.csv \ - -p HIDDEN_LAYER_SIZE 10 -p STEPS 50 \ - "KubeFlow Pipeline Using TFX OSS Components.ipynb" "${TEST_NAME}.ipynb" - else - papermill --prepare-only -p EXPERIMENT_NAME "${TEST_NAME}-test" -p \ - OUTPUT_DIR ${RESULTS_GCS_DIR} -p PROJECT_NAME ml-pipeline-test \ - -p BASE_IMAGE ${TARGET_IMAGE_PREFIX}pusherbase:dev -p TARGET_IMAGE \ - ${TARGET_IMAGE_PREFIX}pusher:dev -p TARGET_IMAGE_TWO \ - ${TARGET_IMAGE_PREFIX}pusher_two:dev \ - -p KFP_PACKAGE /tmp/kfp.tar.gz -p DEPLOYER_MODEL ${DEPLOYER_MODEL} \ - -p TRAIN_DATA gs://ml-pipeline-dataset/sample-test/taxi-cab-classification/train50.csv \ - -p EVAL_DATA gs://ml-pipeline-dataset/sample-test/taxi-cab-classification/eval20.csv \ - -p HIDDEN_LAYER_SIZE 10 -p STEPS 50 \ - "KubeFlow Pipeline Using TFX OSS Components.ipynb" "${TEST_NAME}.ipynb" - fi - check_notebook_result ${TEST_NAME} elif [[ "${TEST_NAME}" == "lightweight_component" ]]; then export LC_ALL=C.UTF-8 export LANG=C.UTF-8 diff --git a/test/sample_test.yaml b/test/sample_test.yaml index 86b3f7d745f..bd1671787f0 100644 --- a/test/sample_test.yaml +++ b/test/sample_test.yaml @@ -25,6 +25,8 @@ spec: arguments: parameters: - name: image-build-context-gcs-uri + - name: image-builder-image + value: gcr.io/ml-pipeline-test/image-builder:v20181128-0.1.3-rc.1-109-ga5a14dc-e3b0c4 - name: target-image-prefix - name: test-results-gcs-dir - name: sample-tests-image-suffix @@ -65,10 +67,8 @@ spec: - name: test-name value: "{{item}}" withItems: - - kubeflow_training_classification - tfx_cab_classification - xgboost_training_cm - - kubeflow_pipeline_using_TFX_OSS_components - lightweight_component - dsl_static_type_checking # Build and push image @@ -90,7 +90,7 @@ spec: valueFrom: path: /outputs/strict-image-name/file container: - image: gcr.io/ml-pipeline-test/image-builder:v20181128-0.1.3-rc.1-109-ga5a14dc-e3b0c4 + image: "{{workflow.parameters.image-builder-image}}" imagePullPolicy: 'Always' args: [ "--image-build-context-gcs-uri", "{{inputs.parameters.image-build-context-gcs-uri}}", diff --git a/test/test-prep.sh b/test/test-prep.sh index f252d392ba5..5a941e488e8 100755 --- a/test/test-prep.sh +++ b/test/test-prep.sh @@ -16,8 +16,10 @@ set -x -# activating the service account -gcloud auth activate-service-account --key-file="${GOOGLE_APPLICATION_CREDENTIALS}" +if [[ ! -z "${GOOGLE_APPLICATION_CREDENTIALS}" ]]; then + # activating the service account + gcloud auth activate-service-account --key-file="${GOOGLE_APPLICATION_CREDENTIALS}" +fi gcloud config set compute/zone us-east1-b gcloud config set core/project ${PROJECT} diff --git a/test/tools/project-cleaner/main.go b/test/tools/project-cleaner/main.go new file mode 100644 index 00000000000..d59d44e4720 --- /dev/null +++ b/test/tools/project-cleaner/main.go @@ -0,0 +1,164 @@ +// Copyright 2019 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "context" + "flag" + "fmt" + "io/ioutil" + "log" + "os" + "strings" + "time" + + "google.golang.org/api/container/v1" + "gopkg.in/yaml.v2" +) + +// project_cleanup tool provides support to cleanup different resources insides a GCP project. +// Types of resources to be cleaned up in a single execution of the tool is specified as YAML file +// (run-time parameter flag) to the tool. Each GCP resource is associated with a handler function +// that performs cleanup. Adding a new resource for cleanup involves - adding a helper function to +// ProjectCleaner type and updating the switch condition in the CleanupProject method. + +var ( + resourceSpec = flag.String("resource_spec", "", "resource spec yaml file") +) + +// ResourceSpec is the top-level spec specifying GCP resources to be cleaned up in a project. +type ResourceSpec struct { + // ProjectId - GCP Project Id + ProjectId string `yaml:"project-id"` + // Resources - Spec for each resource to be cleaned up by the tool. + Resources []GCPResource `yaml:"resources"` +} + +// GCPResource specifies an individual GCP resource to be cleaned up. +type GCPResource struct { + // Resource - name of the resource is used as key to the handler function. + Resource string `yaml:"resource"` + // Zones - Multiple zones support. + Zones []string `yaml:"zones"` + // NamePrefixes - Only support for pre-fixed based filtering as go-client doesn't support regex + // based filtering. + NamePrefixes []string `yaml:"name-prefixes"` + // TimeLapseInHours - Number of hours before the resource needs to be cleaned up. + TimeLapseInHours int `yaml:"time-lapse-hours"` +} + +// ProjectCleaner - top level type providing handlers for different GCP resources. +type ProjectCleaner struct { + ProjectId string + Resources []GCPResource +} + +// GKEClusterHandler provides support to cleanup GCP resources within the project. +func (p *ProjectCleaner) GKEClusterHandler(resource GCPResource) { + // See https://cloud.google.com/docs/authentication/. + // Use GOOGLE_APPLICATION_CREDENTIALS environment variable to specify a service account key file + // to authenticate to the API. + ctx := context.Background() + svc, err := container.NewService(ctx) + if err != nil { + log.Fatalf("Could not initialize gke client: %v", err) + } + + elapsedTime := time.Now().Add(time.Hour * (-1) * time.Duration(resource.TimeLapseInHours)) + for _, zone := range resource.Zones { + log.Printf("Listing gke clusters in zone: %v", zone) + listResponse, err := svc.Projects.Zones.Clusters.List(p.ProjectId, zone).Do() + if err != nil { + log.Printf("Failed listing gke clusters in zone: %v", zone) + } + + for _, cluster := range listResponse.Clusters { + createdTime, err := time.Parse(time.RFC3339, cluster.CreateTime) + if err != nil { + log.Printf("Unable to parse created time for cluster: %s. Ignoring cluster", + cluster.Name) + continue + } + + if p.checkForPrefix(cluster.Name, resource.NamePrefixes) && createdTime.Before(elapsedTime) { + log.Printf("Found cluster: %s for deletion", cluster.Name) + if op, err := svc.Projects.Zones.Clusters.Delete(p.ProjectId, zone, cluster.Name).Do(); + err != nil { + log.Printf("Encountered error calling delete on cluster: %s Error: %v", + cluster.Name, err) + } else { + log.Printf("Kicked off cluster delete : %v", op.TargetLink) + } + } + } + } +} + +// checkForPrefix - helper function to check if testStr string has any of the prefix specified in +// prefixes +func (p *ProjectCleaner) checkForPrefix(testStr string, prefixes []string) bool { + log.Printf("Performing prefix check for String: %s for Prefixes: %v", testStr, prefixes) + for _, prefix := range prefixes { + if strings.HasPrefix(testStr, prefix) { + return true + } + } + return false +} + +// CleanupProject iterates over each resource specified in the spec and calls the corresponding +// cleanup handler method. +func (p *ProjectCleaner) CleanupProject() { + for _, resource := range p.Resources { + switch resource.Resource { + case "gke-cluster": + p.GKEClusterHandler(resource) + default: + log.Printf("Un-identified resource: %v found in spec. Ignoring", resource.Resource) + } + } +} + +func main() { + flag.Parse() + log.SetOutput(os.Stdout) + + if *resourceSpec == "" { + fmt.Fprintln(os.Stderr, "missing resource_spec parameter") + flag.Usage() + os.Exit(2) + } + + if _, err := os.Stat(*resourceSpec); err != nil { + log.Fatalf("missing resource spec file") + } + + var r ResourceSpec + yamlFile, err := ioutil.ReadFile(*resourceSpec) + if err != nil { + log.Fatalf("Error reading yaml file") + } + + err = yaml.Unmarshal(yamlFile, &r) + if err != nil { + log.Fatalf("Unmarshal: %v", err) + } + + p := ProjectCleaner{ + ProjectId: r.ProjectId, + Resources: r.Resources, + } + p.CleanupProject() +} diff --git a/test/minikube/install_argo_client.sh b/test/tools/project-cleaner/project_cleaner.sh similarity index 72% rename from test/minikube/install_argo_client.sh rename to test/tools/project-cleaner/project_cleaner.sh index d55772aa536..c6cee72ebb5 100755 --- a/test/minikube/install_argo_client.sh +++ b/test/tools/project-cleaner/project_cleaner.sh @@ -1,4 +1,4 @@ -#!/bin/bash -ex +#!/bin/bash # # Copyright 2018 Google LLC # @@ -14,7 +14,14 @@ # See the License for the specific language governing permissions and # limitations under the License. -ARGO_VERSION=${ARGO_VERSION:-v2.3.0} +set -ex -sudo curl -sSL -o /usr/local/bin/argo https://github.com/argoproj/argo/releases/download/${ARGO_VERSION}/argo-linux-amd64 -sudo chmod +x /usr/local/bin/argo +SCRIPT_DIR=$(dirname "${BASH_SOURCE}") +pushd "${SCRIPT_DIR}" + +echo "Building project cleaner tool" +go build . + +echo "Executing project cleaner" +./project-cleaner --resource_spec resource_spec.yaml +popd diff --git a/test/minikube/install_docker_minikube_argo.sh b/test/tools/project-cleaner/resource_spec.yaml old mode 100755 new mode 100644 similarity index 56% rename from test/minikube/install_docker_minikube_argo.sh rename to test/tools/project-cleaner/resource_spec.yaml index 3b88e188861..726d7c5abbf --- a/test/minikube/install_docker_minikube_argo.sh +++ b/test/tools/project-cleaner/resource_spec.yaml @@ -1,6 +1,4 @@ -#!/bin/bash -ex -# -# Copyright 2018 Google LLC +# Copyright 2019 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -14,9 +12,12 @@ # See the License for the specific language governing permissions and # limitations under the License. -repo_test_dir=$(dirname $0) - -DOCKER_VERSION=18.06.1~ce~3-0~$(. /etc/os-release; echo "$ID") "${repo_test_dir}/install_docker.sh" -MINIKUBE_VERSION=v0.30.0 KUBECTL_VERSION=v1.12.2 KUBERNETES_VERSION=v1.12.2 "${repo_test_dir}/install_and_start_minikube_without_vm.sh" -ARGO_VERSION=v2.3.0 "${repo_test_dir}/install_argo_client.sh" -sudo apt-get install socat #needed for port forwarding +project-id: ml-pipeline-test +resources: + - resource: gke-cluster + zones: + - us-east1-b + name-prefixes: + - e2e- + - sample- + time-lapse-hours: 24