Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AIRFLOW-1085] Allow the spark_home, driver_memory, and java_class to be passed to the SparkSubmitOperator #2207

Closed
wants to merge 1 commit into from

Conversation

camshrun
Copy link

Dear Airflow maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

Description

  • Adds the ability to pass the Spark home, driver memory, and Java main class to the SparkSubmitOperator.

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:
    Adds:
  • contrib.hooks.spark_submit_hook.TestSparkSubmitHook.test_submit_cmd
    Modifies:
  • contrib.hooks.spark_submit_hook.TestSparkSubmitHook.test_build_command
  • contrib.operators.spark_submit_operator.TestSparkSubmitOperator.test_execute

Commits

  • My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

@codecov-io
Copy link

Codecov Report

Merging #2207 into master will increase coverage by 0.13%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2207      +/-   ##
==========================================
+ Coverage    67.5%   67.64%   +0.13%     
==========================================
  Files         142      134       -8     
  Lines       10845    10737     -108     
==========================================
- Hits         7321     7263      -58     
+ Misses       3524     3474      -50
Impacted Files Coverage Δ
airflow/__init__.py 75.6% <0%> (-3.94%) ⬇️
airflow/macros/__init__.py
airflow/utils/__init__.py
airflow/www/utils.py
airflow/executors/__init__.py
airflow/api/__init__.py
airflow/hooks/__init__.py
airflow/operators/__init__.py
airflow/task_runner/__init__.py
airflow/jobs.py 73.35% <0%> (+0.41%) ⬆️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b55f41f...e5e0e84. Read the comment docs.

@camshrun camshrun closed this Apr 3, 2017
@camshrun camshrun deleted the sparkSubmitImprovements branch April 3, 2017 17:48
potiuk added a commit to potiuk/airflow that referenced this pull request Apr 4, 2022
The 0.2.2 release of `cgroupspy` makes it Python 3.10 compatible.
We can remove vendoring done as of apache#22209 apache#22208 apache#2207 apache#22206

Discussion and links:

* cloudsigma/cgroupspy#14
@potiuk potiuk mentioned this pull request Apr 4, 2022
potiuk added a commit that referenced this pull request Apr 4, 2022
The 0.2.2 release of `cgroupspy` makes it Python 3.10 compatible.
We can remove vendoring done as of #22209 #22208 #2207 #22206

Discussion and links:

* cloudsigma/cgroupspy#14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants