Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Google cloud dataproc: Add an operator to run a serverless spark in dataproc #22019

Closed
1 of 2 tasks
ajayganti3 opened this issue Mar 6, 2022 · 3 comments
Closed
1 of 2 tasks

Google cloud dataproc: Add an operator to run a serverless spark in dataproc #22019

ajayganti3 opened this issue Mar 6, 2022 · 3 comments
Assignees
@ajayganti3
Labels
kind:feature Feature Requests provider:google Google (including GCP) related issues

Comments

@ajayganti3
Copy link

ajayganti3 commented Mar 6, 2022 •
edited
Loading

Description

Recently GCP has launched serverless spark on dataproc. It will be really great if Airflow has an operator to submit a serverless spark job. Please refer to the dataproc serverless documentation and its gloud sdk documentation

Use case/motivation

Using this serverless dataproc we can run spark jobs easily without managing any infrastructure, autoscaling and work loads.

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@ajayganti3 ajayganti3 added the kind:feature Feature Requests label Mar 6, 2022
@boring-cyborg
Copy link

boring-cyborg bot commented Mar 6, 2022

Thanks for opening your first issue here! Be sure to follow the issue template!

@abhinavraj23 abhinavraj23 mentioned this issue Mar 6, 2022
Closed
@mik-laj mik-laj added the provider:google Google (including GCP) related issues label Mar 7, 2022
@aoelvp94
Copy link

DId you see this PR #19248?

@potiuk
Copy link
Member

potiuk commented Mar 15, 2022

Closing as implemented in #19248

@potiuk potiuk closed this as completed Mar 15, 2022
@abhinavraj23 abhinavraj23 mentioned this issue Jun 20, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ajayganti3 ajayganti3

Labels
kind:feature Feature Requests provider:google Google (including GCP) related issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: add an operator to run a transfer job abhinavraj23/airflow
4 participants
@potiuk @mik-laj @aoelvp94 @ajayganti3