A complete development environment for Azure DataBricks workflows.
This solution includes a development container with all the required tooling to connect, test, and deploy python based DataBrick workflows.
-
Download and unpack the Release that matches the DataBricks runtime of your cluster. (For example Masonry v0.0.*-10.4-LTS for Runtime 10.4 LTS)
NOTE: 'LITE' versions use an exposed requirements.txt file to complete environment.
-
Next follow the installing prerequisites documentation to setup the local environnement.
-
Configure the Azure DataBricks cluster for connection.
-
Gather the required information by obtaining cluster information.
-
Next follow the using the dev container documentation to start up the development environment.
See building python library for building python libraries.
See running unit tests for information on running unit tests.
See run full code base linting locally for steps on running a full code base lint solution.
See Project Roadmap for an up to date development roadmap.
See masonry workflow for a complete workflow overview with CI/CD example.
- Author - Mickey "ScruffyFurn" MacDonald
Copyright (c) 2022 Mickey MacDonald