Sync fork #3
Merged
Sync fork #3
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Components - Filesystem * Get subdirectory * Get file * List items * Fixed ls paramaters for Alpine
* Frontend - Add support for Minio artifact URIs * Fixed style * Removed unused variable * Old Node does not support named capture groups in regexes * Made the URL relative
* Get kubeflow namespace from kfp UI * Add a react context that makes namespace available anywhere. * Move KFP_FLAGS check to index.tsx * Update index.tsx
…t('user-gcp-sa')" (#2639)
* added new secret support
* updated the documentation and env settings
* updated after feedback
* added tests
* nameing issue fixed
* renamed test to follow unittest standard
* updated after feedback
* the new test after renaming
* added the test to main
* updates after feedback
* added licensce agreement
* removed space
* updated the volume named to be generated
* secret_name as volume name and updated test
* updated the file structure
* fixed build
* add namespace to some run APIs * update only the create run api * add resourcereference for namespace runs * add variables in const * add types to toModel func * bug fix * strip the namespace resource reference when mapping to the db model * add unit tests * use gofmt * replace belonging relationshipreference to owner * put a todo for further investigation of using namespace or uuid * apply gofmt * revert minor change * Update model_converter.go
* Added CsvExampleGen component * Switched to using some processing code from the component class Needs testing * Renamed output_examples to example_artifacts for consistency with the original component * Fixed the docstring a bit * Added StatisticsGen First draft * Added SchemaGen First draft * Fixed the input_dict construction * Use None defaults * Switched to TFX container image * Updated component definitions * Fixed StatisticsGen and SchemaGen Input artifacts must have splits. Split URIs should end with "/'. The ciomponents now work. Also printing component_class_instance for debugging. * Printing component instance in CsvExampleGen * Moved components to directories * Updated the sample TFX pipeline * Renamed ExamplesPath to Examples for data passing components * Corrected output_component_file paths * Added the Transform component The component uses almost completely generic code. * Added the Trainer component * Added the Evaluator component * Added the ExampleValidator component * Added the BigQueryExampleGen component * Added the ImportExampleGen component * Updated the sample Added ExampleValidator, Transform, Trainer, Evaluator * Upgraded to TFX 0.15.0 * Upgraded the sample to 0.15.0 * Silence Flake8 for annotations
* initial * store the ns * update namespace mapping * add comments to check reference exist func
* update server and sdk version to 0.1.37 * update api package version to latest * fix test error
* add namespace to some run APIs * update only the create run api * add resourcereference for namespace runs * add variables in const * add namespace to run_pipeline * add types to toModel func * minor fix * bug fix * strip the namespace resource reference when mapping to the db model * add unit tests * use gofmt * replace belonging relationshipreference to owner * put a todo for further investigation of using namespace or uuid * apply gofmt * revert minor change * use owner relationship * add docstring for namespace * add namespace to all create run func * added todos for wrapping the configrations for different auth methods; default namespaces
…hing with other template names (#1555) Case exhibiting the problem: ``` def add(a, b): ... @dsl.pipeline(name="add') def some_name(): add(...) ```
* FE changes for version ui * sort methods; remove console * format * remove obsolete snapshots * cosmetic change: simplify href link construction * fix the clone run behavior when version present * format * run-integration-test * frontend-integration-test: try to simulate uploading file * fix frontend-integration-test * address comments * address comments and add one more unit test
* TFjob v1 launcher * fix comments
…2688) It now only changes the case of the first letter.
#2695) * Update to tfx 0.15.0 release * update file name * modernize naming * move to core * Update sample test launcher/config/file name. * No longer need precompiled package. * add dependency * add pip * no longer call dsl-compile * fix launcher * give additional time for init * Try skip check in backend docker * fix * try fix * revert the timeout * fix per Alexey's suggestion * fix typo * print log * fix * remove log
* add namespace to some run APIs * update only the create run api * add resourcereference for namespace runs * pass user identity header from the gRPC server to KFP service * add variables in const * declare a flag and fill in the authorizations * add types to toModel func * bug fix * strip the namespace resource reference when mapping to the db model * add unit tests * add authorization * interpret json response * use gofmt * add more meaningful error message; format * refactoring codes * replace belonging relationshipreference to owner * put a todo for further investigation of using namespace or uuid * apply gofmt * revert minor change * refactor codes * minor change * use internal server error in kfam client * minor change * use timeout in kfam client * make kfam service host/port configurable * minor changes * update name * rename * update the util function to accept a list of resourcereferences * better error message * reformat * remove IsRequestAuthorized func * add kfam host and port in config.json * generalize the auth code * rename KFAMInterface to KFAMClientInterface * add kfam fake for tests * add build bazel * add unit tests for util func * remove the config * add unit test for authorization with httptest * only intialize the kfam client when kubeflow deployment * minor change * fix typo * wrap the whole auth func * update authz logic to be enabled when it is kubeflow deployment * change flag from kubeflow deployment to multiuser mode * gofmt * move fake kfam to the original kfam; create multiple fake kfam clients * combine authorize func, add unit tests for util_test * wrap errors * fix unit test * service unauthorized info to user * better user errors * revert some accidental change * Update util.go * make functions local * deduplicate return values from isauthorized
* Improve test cloudbuild yaml readability * Update api_server.yaml
* Fix combined unique keys * Move the index removal after automigration in case table pipeline_versions doesn't exist for first time deployment
* Added manifest for deploying on aws using s3 * Revert "Added manifest for deploying on aws using s3" This reverts commit 6a9c498. * Added readme and link to kubeflow-aws on how to deploy lightweight pipeline on AWS * updated readme on how to deploy on aws * Update README.md * Update README.md
This change adds the necessary config-map related to gRPC MLMD server. To make the names more clear, this change also modifies the existing 'metadata-configmap' which provides mysql configurations to 'metadata-mysql-configmap'
* fix link with permalink * pin to 0.1.37 release
* Add samples to manage Azure Databricks in Kubeflow Pipelines. azure-samples/kfp-azure-databricks folder contains a set of Ops which allow us to manage Azure Databricks resources directly from our Kubeflow Pipelines. These Ops leverage Azure Databricks Operator for Kubernetes to manage those resources. They wrap ResourceOp and abstract pipeline users from Kubernetes Operator details. Supported Ops: * CreateClusterOp, to create a cluster in Databricks. * DeleteClusterOp, to delete an existing cluster from Databricks. * CreateJobOp, to create a Spark Job in Databricks. * DeleteJobOp, to delete an existing Spark Job from Databricks. * SubmitRunOp, to submit a Job Run in Databricks. * DeleteRunOp, to delete an existing Run. azure-samples/databricks-pipelines folder contains several pipeline samples: * databricks_operator_*.py samples show how to use ResourceOp to manage Databricks resources using Azure Databricks Operator. * databricks_pkg_*.py samples show how to use Ops in kfp-azure-databricks package to do the same. * Fix small bug on script to compile, upload and run azure pipeline samples * Removes azure databricks operator pipeline samples * Added new tests for Azure Databricks Delete Ops plus minor tweaks to those Ops * Removes bash script and adds Python CLI to compile and run pipelines * Changes the way kfp host is handled in pipeline cli
* update config * update manifest to test * clean proxy deployment
* update link * update tag in link and move
addresses issue #2253. Passing `nil` as an argument to the varargs `...DeleteOption` parameter of `client.Delete()` will panic since client-go will iterate over the DeleteOption array and invoke each (since DeleteOption is defined as `type DeleteOptionFunc func(*DeleteOptions)`.
* Add mkp deployment for test * Add mkp deployment script This commit adds mkp deployment script including cluster deployment, argo installation and kfp mkp deployment. * Add sleep 60 after kfp deployment This commits adds sleep 60 after kfp deployment to make sure kfp resources are ready when running argo submit. * Typo fix This commit fixes zone variable and deletes extra comments. * Remove cluster deployment and argo installation in mkp script This commit removes cluster deployment and argo installation in mkp script. Those logic are the same with before and can use existing script. * Revert unrelated changes This commit reverts unrelated changes on test/manifests/kustomization.yaml. * Remove WI set up on mkp deployment This commit removes WI set up on mkp deployment.
* done * all done Co-authored-by: renmingu <40223865+renmingu@users.noreply.github.com>
* done * all done * bump version to 0.2.1 * correct to 0.2.0 * done Co-authored-by: renmingu <40223865+renmingu@users.noreply.github.com>
* done * all done * bump version to 0.2.1 * correct to 0.2.0 * done * done Co-authored-by: renmingu <40223865+renmingu@users.noreply.github.com>
* done * all done * bump version to 0.2.1 * correct to 0.2.0 * done * backup * auto release deployer for each release * done * fix sequence * fix deployer * fix primary image Co-authored-by: renmingu <40223865+renmingu@users.noreply.github.com>
* to 45m * 45m Co-authored-by: renmingu <40223865+renmingu@users.noreply.github.com>
* fix the metadata writer manifest * fix Co-authored-by: renmingu <40223865+renmingu@users.noreply.github.com>
Co-authored-by: renmingu <40223865+renmingu@users.noreply.github.com>
Signed-off-by: Ilias Katsakioris <elikatsis@arrikto.com>
* Pin sample doc links during backend image build * Update sample_config.json
… dbfsblock (#2817) * Adds secretscope and workspaceitem Ops to Azure Databricks for KFP * Adds dbfsblock Op to Azure Databricks for KFP * Adds code coverage to Azure Databricks for KFP tests * Changes email address to generic one * Sets single source of truth for package version * Removes all warnings from tests * Removes deprecated calls to Compiler().compile in pipelines * Removes unnecessary .gitignore line Co-authored-by: creddy123 <31089923+creddy123@users.noreply.github.com>
* Add mkp deployment for test * Add mkp deployment script This commit adds mkp deployment script including cluster deployment, argo installation and kfp mkp deployment. * Add sleep 60 after kfp deployment This commits adds sleep 60 after kfp deployment to make sure kfp resources are ready when running argo submit. * Typo fix This commit fixes zone variable and deletes extra comments. * Remove cluster deployment and argo installation in mkp script This commit removes cluster deployment and argo installation in mkp script. Those logic are the same with before and can use existing script. * Revert unrelated changes This commit reverts unrelated changes on test/manifests/kustomization.yaml. * Add kfp deployment options param in post submit script This commit adds kfp deployment options in post submit script. Current options are standalone and mkp deployment. * Fix on naming and syntax This commit fixes small naming and syntax problem pointed on PR review.
* Have custom visualization on by default * Add flag to gcp market
* Add mkp deployment for test * Add mkp deployment script This commit adds mkp deployment script including cluster deployment, argo installation and kfp mkp deployment. * Add sleep 60 after kfp deployment This commits adds sleep 60 after kfp deployment to make sure kfp resources are ready when running argo submit. * Typo fix This commit fixes zone variable and deletes extra comments. * Remove cluster deployment and argo installation in mkp script This commit removes cluster deployment and argo installation in mkp script. Those logic are the same with before and can use existing script. * Revert unrelated changes This commit reverts unrelated changes on test/manifests/kustomization.yaml. * Add kfp deployment options param in post submit script This commit adds kfp deployment options in post submit script. Current options are standalone and mkp deployment. * Remove WI set up on mkp deployment This commit removes WI set up on mkp deployment. * Fix on naming and syntax This commit fixes small naming and syntax problem pointed on PR review. * Read marketplace tools image from dockerfile This commit changes marketplace tool image name from hard-coded to be read from manifest.
* Making json output format easier to read * Update sdk/python/kfp/cli/diagnose_me_cli.py good catch :) Co-Authored-By: Jiaxiao Zheng <jxzheng@google.com> * update based on feedback * Sample env validation pipeline with diagnose_me * removing files that I accidentally added * update based on review feedback * diagnose_me based validation pipeline * updating sample based on review feedback * updating project_id comment * removed sequential execution * changing comment on base image Co-authored-by: Jiaxiao Zheng <jxzheng@google.com>
1.14.8-gke.17 is no longer supported while 1.14.10-gke.17 and 1.14.9-gke.23 have broken Workflow Identity.
* SDK - Components refactoring This change is a pure refactoring of the implementation of component task creation. For pipelines compiled using the DSL compiler (the compile() function or the command-line program) nothing should change. The main goal of the refactoring is to change the way the component instantiation can be customized. Previously, the flow was like this: `ComponentSpec` + arguments --> `TaskSpec` --resolving+transform--> `ContainerOp` This PR changes it to more direct path: `ComponentSpec` + arguments --constructor--> `ContainerOp` or `ComponentSpec` + arguments --constructor--> `TaskSpec` or `ComponentSpec` + arguments --constructor--> `SomeCustomTask` The original approach where the flow always passes through `TaskSpec` had some issues since TaskSpec only accepts string arguments (and two other reference classes). This made it harder to handle custom types of arguments like PipelineParam or Channel. Low-level refactoring changes: Resolving of command-line argument placeholders has been extracted into a function usable by different task constructors. Changed `_components._created_task_transformation_handler` to `_components._container_task_constructor`. Previously, the handler was receiving a `TaskSpec` instance. Now it receives `ComponentSpec` + arguments [+ `ComponentReference`]. Moved the `ContainerOp` construction handler setup to the `kfp.dsl.Pipeline` context class as planned. Extracted `TaskSpec` creation to `_components._create_task_spec_from_component_and_arguments`. Refactored `_dsl_bridge.create_container_op_from_task` to `_components._resolve_command_line_and_paths` which returns `_ResolvedCommandLineAndPaths`. Renamed `_dsl_bridge._create_container_op_from_resolved_task` to `_dsl_bridge._create_container_op_from_component_and_arguments`. The signature of `_components._resolve_graph_task` was changed and it now returns `_ResolvedGraphTask` instead of modified `TaskSpec`. Some of the component tests still expect ContainerOp and its attributes. These tests will be changed later. * Adapted the _python_op tests * Fixed linter failure I do not want to add any top-level kfp imports in this file to prevent circular references. * Added docstrings * FIxed the return type forward reference
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.