New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add spark master and worker image build #275

Open

thanh-nguyen-dang wants to merge 21 commits into master from feat/spark-master-worker

Contributor

thanh-nguyen-dang commented Jul 1, 2024

Link to JIRA ticket if there is one:
https://ctds-planx.atlassian.net/browse/MIDRC-557
https://ctds-planx.atlassian.net/browse/MIDRC-730

New Features

Breaking Changes

Bug Fixes

Improvements

Dependency updates

Deployment changes

github-actions bot commented Jul 1, 2024

The style in this PR agrees with black. ✔️

This formatting comment was generated automatically by a script in uc-cdis/wool.

thanh-nguyen-dang force-pushed the feat/spark-master-worker branch 9 times, most recently from c51a77f to 679e572 Compare

July 1, 2024 19:14

github-actions bot commented Jul 1, 2024

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_etl.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here

thanh-nguyen-dang force-pushed the feat/spark-master-worker branch from 679e572 to 8ad52dc Compare

July 1, 2024 20:14

github-actions bot commented Jul 1, 2024

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_etl.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here

thanh-nguyen-dang force-pushed the feat/spark-master-worker branch 17 times, most recently from e1813a4 to 93d6fb2 Compare

July 8, 2024 18:29

github-actions bot commented Jul 10, 2024

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_etl.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here

thanh-nguyen-dang force-pushed the feat/spark-master-worker branch 2 times, most recently from 0cbeb34 to 1e9a282 Compare

July 12, 2024 15:07

github-actions bot commented Jul 12, 2024

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_etl.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here

github-actions bot commented Jul 15, 2024

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_etl.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here

thanh-nguyen-dang force-pushed the feat/spark-master-worker branch from 76660ee to f71e4d0 Compare

July 15, 2024 21:33

github-actions bot commented Jul 15, 2024

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_etl.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here

thanh-nguyen-dang added 21 commits

August 21, 2024 04:11


          add nested properties for collection indices

2c2793d


          handle case root node does not have nested edge

19ec234


          fix multiple parents case

3ab1b54


          update types for nested translator in collector etl

d3abf18


          add spark master and worker image build

0dbec2d


          add setup.py

128eba6


          fix to add pip install

237769f


          add command to run pip install

ef9dcd7


          add log4j.properties

cbfb5e9


          fix log4j

faa7d93


          add net-tools netcat

671c8c4


          correct the appender console

efb05b8


          remove unnecessary files

1a9df4f


          use built base images

3bb5093


          update branch of compose-etl used to test

baec931


          remove run_config

cdbf5bc


          use old Dockerfile


          set default value of driver.host to localhost

705d8b0


          remove adding log4j properties files since we added in base image

75c83ad


          try to use spark base image

2c7cf44


          add nested translation test for injection

thanh-nguyen-dang force-pushed the feat/spark-master-worker branch from f71e4d0 to 2836057 Compare

August 28, 2024 23:31

github-actions bot commented Aug 29, 2024

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_etl.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet