Currently developing scripts to facilitate data transfer between sensor device manufactures and IDEA-FAST's internal data platform, and a separate consumer API to provide updates on device wear and use.
Poetry is used for dependency management and pyenv to manage python installations, so please install both on your local machine. We use python 3.8 by default, so please make sure this is installed via pyenv, e.g.
pyenv install 3.8.0 && pyenv global 3.8.0
Once done, you can install dependencies for this project via:
poetry install --no-dev
To setup a virtual environment with your local pyenv version run:
poetry shell
Copy .NAME.dev.env.example file to .NAME.dev.env where NAME is the python package (consumer, data_transfer).
Then add relevant local/live values and credentials. You will need .NAME.dev.env.example if deploying
For development, install additional dependencies through:
poetry install
poetry run pre-commit install
When developing the consumer API run:
poetry run consumer
When developing the data transfer jobs run where DEVICE_TYPE is the type of device (e.g., DRM, BTF, etc.) and study_site is one of the core study sites. Note, for BTF you can add timespan parameters. View the DAGs in data_transfer for more information:
python data_transfer/main.py $DEVICE_TYPE $STUDY_SITE
python data_transfer/main.py BTF $STUDY_SITE $REFERENCE_DAY $DAYS_TIMESPAN
Nox is used for automation and standardisation of tests, type hints, automatic code formatting, and linting. Any contribution needs to pass these tests before creating a Pull Request.
To run all these libraries:
poetry run nox -r
Or individual checks by choosing one of the options from the list:
poetry run nox -rs [tests, mypy, isort, lint, black, live_tests]
Docker is used to build images both for the consumer and data_transfer services. A docker-compose file configures the required services and image needed for local development. This can be run locally when developing if interacting with a service (e.g., consumer) to mirror deployment usage. To run the compose file you first need to create two docker networks:
docker network create web
docker network create database
Semantic versioning is used when core changes are merged
to master to enable continuous deployment. To build an image locally run the following
where $VERSION is your desired version and $REPO is the name of the image:
poetry run build $REPO $VERSION
# e.g., poetry run build dtransfer 0.0.1-DRM
The compose file uses specified .env files and runs all services:
poetry run compose
Note: for local development docker is primarily used to test changes prior to deploying live or to interact with services in isolation.
Docker Hub is used to store images. To push
to an image to a $REPO run the following:
poetry run publish $VERSION $REPO
Due to our current manual setup, the pipeline needs to be manually initates within the container, e.g.
docker exec -it $CONTAINERNAME sh
Add /proc/1/fd/1 to have the output logged to the container logs (good practise). You can also keep tabs on the logs and store this in a separate file. Finally, you can pull down a (most recent) copy of this log file for investigation on your machine.
python data_transfer/main.py $DEVICE $STUDYSITE -1 >> /proc/1/fd/1
docker logs -f $CONTAINERNAME > path/to/log/file.log &
scp username@ip:/remote/file.txt /local/directory