New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Simplify running spark jobs in docker container #7

Open

rohitbansal-tw wants to merge 4 commits into data-derp:main from rohitbansal-tw:main

Contributor

rohitbansal-tw commented Apr 26, 2024 •

edited

Loading

Setup docker containers to run spark programs easily, without being concerned of version compatibility/conflicts
For support of running spark exercises, created spark.py so that common init block across programs could reside there
Updated README to support running on either via docker, or locally on JDK 11 + Spark 3.4.x

rohitbansal-tw added 4 commits

April 26, 2024 23:55


          Simplify running spark jobs in docker container

a31d0d1


          Fix pyspark version in requirements to be consistent

0ec229a


          Add conf for running spark master locally without docker container

4f6fa2e


          Add docker container to run streaming examples

46617b9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet