Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

debug: use compose-etl debug branch #291

Merged
merged 17 commits into from
Jan 7, 2025
Merged

debug: use compose-etl debug branch #291

Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
1c825e5
debug: use compose-etl debug branch
m0nhawk Dec 6, 2024
5596996
Update image_build_and_test.yaml
m0nhawk Dec 6, 2024
60c714f
Update image_build_and_test.yaml
m0nhawk Dec 6, 2024
ab03d47
debug: update
m0nhawk Dec 6, 2024
9e5b001
debug
m0nhawk Dec 6, 2024
e3b9fb1
debug
m0nhawk Dec 6, 2024
ca132ba
debug
m0nhawk Dec 6, 2024
3f835e1
debug
m0nhawk Dec 6, 2024
ecdfb66
debug
m0nhawk Dec 6, 2024
c1cc749
debug
m0nhawk Dec 6, 2024
094dda1
debug
m0nhawk Dec 6, 2024
dae0ec4
debug
m0nhawk Dec 6, 2024
6047a0c
debug
m0nhawk Dec 6, 2024
67e30c8
debug
m0nhawk Dec 9, 2024
c0714be
Add virtual environment variables to Dockerfile
m0nhawk Dec 9, 2024
73a28e0
Update git clone branch to master
m0nhawk Dec 9, 2024
7ad4ebb
Remove dfs.permissions property from HDFS configuration
m0nhawk Dec 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 12 additions & 10 deletions .github/workflows/image_build_and_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,21 +3,23 @@ on: push

jobs:

# # first, build the image
# build:
# name: Build and push image
# uses: uc-cdis/.github/.github/workflows/image_build_push.yaml@master
# secrets:
# ECR_AWS_ACCESS_KEY_ID: ${{ secrets.ECR_AWS_ACCESS_KEY_ID }}
# ECR_AWS_SECRET_ACCESS_KEY: ${{ secrets.ECR_AWS_SECRET_ACCESS_KEY }}
# QUAY_USERNAME: ${{ secrets.QUAY_USERNAME }}
# QUAY_ROBOT_TOKEN: ${{ secrets.QUAY_ROBOT_TOKEN }}
# first, build the image
build:
name: Build and push image
uses: uc-cdis/.github/.github/workflows/image_build_push.yaml@master
with:
BUILD_PLATFORMS: "linux/amd64"
secrets:
ECR_AWS_ACCESS_KEY_ID: ${{ secrets.ECR_AWS_ACCESS_KEY_ID }}
ECR_AWS_SECRET_ACCESS_KEY: ${{ secrets.ECR_AWS_SECRET_ACCESS_KEY }}
QUAY_USERNAME: ${{ secrets.QUAY_USERNAME }}
QUAY_ROBOT_TOKEN: ${{ secrets.QUAY_ROBOT_TOKEN }}

# then run the tests
test:
name: Run tests
runs-on: ubuntu-20.04
# needs: build
needs: build

services:
postgres:
Expand Down
17 changes: 8 additions & 9 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,6 @@
COPY --from=builder /venv /venv
COPY --from=builder /${appname} /${appname}


ENV DEBIAN_FRONTEND=noninteractive \
SQOOP_VERSION="1.4.7" \
HADOOP_VERSION="3.3.2" \
Expand All @@ -67,12 +66,14 @@
RUN mkdir -p /usr/share/man/man1
RUN mkdir -p /usr/share/man/man7


RUN dnf -y update
RUN dnf -y install \
wget tar unzip vim
RUN dnf -y install java-11-amazon-corretto

java-11-amazon-corretto \
postgresql15 \
tar \
unzip \
vim \
wget

RUN wget ${SQOOP_INSTALLATION_URL} \
&& mkdir -p $SQOOP_HOME \
Expand Down Expand Up @@ -101,7 +102,7 @@
RUN wget ${MAVEN_ES_SPARK_VERSION}-javadoc.jar -O ${ES_HADOOP_HOME_VERSION}/dist/${ES_SPARK_20_2_11}-${ES_HADOOP_VERSION}-javadoc.jar
RUN wget ${MAVEN_ES_SPARK_VERSION}-sources.jar -O ${ES_HADOOP_HOME_VERSION}/dist/${ES_SPARK_20_2_11}-${ES_HADOOP_VERSION}-sources.jar

ENV HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop \

Check warning on line 105 in Dockerfile

View workflow job for this annotation

GitHub Actions / Build and push image / Build Image and Push

Variables should be defined before their use 

UndefinedVar: Usage of undefined variable '$LD_LIBRARY_PATH'
More info: https://docs.docker.com/go/dockerfile/rule/undefined-var/
HADOOP_MAPRED_HOME=$HADOOP_HOME \
HADOOP_COMMON_HOME=$HADOOP_HOME \
HADOOP_HDFS_HOME=$HADOOP_HOME \
Expand All @@ -117,16 +118,14 @@

RUN mkdir -p $ACCUMULO_HOME $HIVE_HOME $HBASE_HOME $HCAT_HOME $ZOOKEEPER_HOME
RUN chown -R gen3:gen3 $HADOOP_HOME
RUN mkdir -p /result && chown -R gen3:gen3 /result && chmod -R 700 /result
#RUN hdfs dfs -chown gen3 /result

ENV PATH=${SQOOP_HOME}/bin:${HADOOP_HOME}/sbin:$HADOOP_HOME/bin:${JAVA_HOME}/bin:${PATH}


# Switch to non-root user 'gen3' for the serving process
USER gen3

RUN source /venv/bin/activate
ENV PATH="/venv/bin:$PATH" \
VIRTUAL_ENV="/venv"

ENV PYTHONUNBUFFERED=1 \
PYTHONIOENCODING=UTF-8
Loading
Loading