Add docs for GCP Dataproc deployment

[abhi8893]

Description

This PR adds docs for the deployment of Kedro projects to GCP Dataproc (Serverless).

What does this guide include? ✅

• Dataproc serverless deployment
• Base design pattern for both dev and prod workflows intended to allow developers to design their own deployment workflow
• Basic guide to GCP resource provisioning

What does this guide NOT include? ❌

• Full fledged dataproc pipeline deployment guide
• CI/CD workflow guidance
• GCP Best practices including IAM, Networking
• Spark performance tuning guide

(WIP) Checklist:

Please note that the current docs are very much WIP, and aren't verbose enough for developers unfamiliar with GCP. I will refine them soon!

• Add an overall context section
• Add descriptions for substeps
• Refine entrypoint kedro run args implementation
• Add GCP resource links
• Add FAQs

Review guidance needed

In addition to a review of the overall approach, please provide guidance on the following:

Q1: Kedro entrypoint script arguments

The recommended entrypoint script invokes kedro's built in cli main entrypoint as follows:

With kedro package wheel install:

import sys
from <PACKAGE_NAME>.__main__ import main

main(sys.argv[1:])

Without kedro package wheel install:

from kedro.framework import cli

cli.main(sys.argv[1:])

However, the implementation in this PR relies on passing the arbitrary kedro args from one py script i.e. deployment/dataproc/serverless/submit_batches.py to the main entrypoint script deployment/dataproc/entrypoint.py.
As I was unable to implement parsing arbitrary args with dashes --, I implemented it as a single --kedro-run-args named arg.

Requesting for a review to enable a better implementation here.

Q2: Incorporating spark configs while submitting jobs

Spark configs can be divided into 2 parts:

1. Spark config set at creation of SparkContext => These can't be set / overriden in a SparkSession by kedro hook (if implemented)

• Examples: spark.driver.memory, spark.executor.instances

4. Spark config set that can be set both at creation of SparkContext and overriden for any new SparkSession

• Examples: Most spark SQL configs

Since, the proposed implementation does NOT read in spark.yml config for the project when submitting the job to dataproc, this requires duplicating some of the configs in the submission script (outside kedro).

How do we enable passing of these spark configs at job/batches submission time?

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

• Read the contributing guidelines
• Signed off each commit with a Developer Certificate of Origin (DCO)
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes (NA)
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes (NA)
• Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team (NA)

[astrojuanlu]

Rendered result: https://kedro--4393.org.readthedocs.build/en/4393/deployment/gcp_dataproc.html

[astrojuanlu]

Thanks a lot for this contribution @abhi8893! 🙏🏼 We'll give it a look shortly.

[abhi8893]

Thanks @astrojuanlu ! I will also revist it again to improve the flow and address any comments you may have 🙂

[merelcht]

Thanks for this extensive contribution @abhi8893 ⭐

I've done a very quick initial review mostly just looking at wording/spelling. I'll do a more thorough review and will try to test this as well.

[merelcht]

Suggested change

`Dataproc serverless` lets you run Spark workloads without requiring you to provision and manage your own Dataproc cluster. An advantage over `Dataproc compute engine` is that `Dataproc serverless` supports custom containers allowing you package your dependencies at build time. Refer [here](https://cloud.google.com/dataproc-serverless/docs/overview#s8s-compared) for the official comparison between Dataproc serverless and compute engine.

`Dataproc Serverless` lets you run Spark workloads without requiring you to provision and manage your own Dataproc cluster. An advantage over `Dataproc compute engine` is that `Dataproc Serverless` supports custom containers allowing you package your dependencies at build time. Refer [here](https://cloud.google.com/dataproc-serverless/docs/overview#s8s-compared) for the official comparison between Dataproc Serverless and compute engine.

[merelcht]

To respond to your point about the parsing of the Kedro CLI args:

As I was unable to implement parsing arbitrary args with dashes --, I implemented it as a single --kedro-run-args named arg.

Your implementation looks fine to me. In Kedro we use Click for CLI, which can be a tricky library to work with at times. So depending on the format you receive the arguments in, it is indeed difficult to parse. Did you find any issues with this implementation, as in is there anything a user can't do now?

[abhi8893]

@merelcht Thanks for the review! I have incorporated all the changes. If you feel this is better incorporated as a kedro blog post, then also happy to do so! Is there a repo which holds the source for kedro blogs?