The Data Platform will be a centralised, in-house platform to provide hosting and tools for data:
- storage
- discovery
- analysis
- dissemination
- governance
This is the Ministry of Justice repository for the core infrastructure of the Data Platform.
Please read the contributing guide before sending pull requests, or creating issues.
We have a series of repositories for our work. We have adopted the naming
convention of naming each repository starting with data-platform-*.
We also name things so that users can understand what a repository does through its name.
| Name | Description |
|---|---|
| Data Platform (this repository) | Core infrastructure, public facing documentation, feature work, enhancements, and issues |
| Data Platform products | User-created services that are hosted on the Data Platform |
| Data Platform support | This repository is used for support and provides templated forms for our users. |
| Modernisation Platform environments repository | Hosting environment for the data platform |
Unit tests are implemented per container using pytest. We use moto to mock out AWS services.
For example, to test daap-get-glue-metadata, run:
cd containers/daap-get-glue-metadata
pip install -r tests/requirements.txt
pytestTo test the full docker image without deploying it to lambda, follow the instructions in Testing AWS base images.
These are the requirements currently met by this repository in line with the FLOSS Best Practices Criteria (Passing Badge)
Basic project site content:
- The project site describes what the software does
- Provides information on how to interact with the project
- explains contribution process
FLOSS license:
- Uses MIT licence
Documentation:
- The project provides basic documentation
Public version-controlled source repository
- GIT version control
Bug-reporting process:
- The project provides a process for users to submit bug reports
- The project uses GitHub issues/ zen hub to track issues
Other security issues:
- secured private credentials via Kuberentes secrets and GitHub secrets
Secured delivery against man-in-the-middle (MITM) attacks:
- Delivered via terraform secured by HTPPS and spcific IAM roles useable only by the GitHub action
- No Cryptographic hases transported over HTTP
Static code analysis:
- Use of CodeQL
Other:
-
The project site supports HTTPS using TLS
-
The project is maintained
For more information please see Criteria
It may be also useful to look at:
We are currently in the research and design phase, and are not yet accepting requests to host new data products.
In the meantime please get in touch via our #data-platform Slack channel with
any questions.
Slack: #data-platform
Email: data-platform@digital.justice.gov.uk
As above - preferably our Slack channel: #data-platform (or email data-platform@digital.justice.gov.uk)
The published
documentation is created by editing *.html.md.erb files,
found in the docs source folder.
The syntax is Markdown, more details can be found here.
For guidance see the Tech Docs Template Write your content.
While editing the files locally, you can start a Docker container that will use Middleman to act as a server hosting the web pages. See preview docs.
Every change should be reviewed in a pull request, no matter how minor. PR request reviewer/s should be enabled within the main branch protection settings.
Merging the changes to the main branch automatically publishes the
changes via GH Action. See publishing.
You can preview how your changes will look, if you've cloned this repository to your local machine, run this command:
bash scripts/docs/docker.sh previewThis will run a preview web server on http://localhost:4567 which you can open in your browser.
Use bash scripts/docs/docker.sh check to compile
the site to HTML and check the URLs are valid.
This is only accessible on your computer, and won't be accessible to anyone else.
For more details see the tech-docs-github-pages-publisher repository.
Any changes you push/merge into the
main branch should be published to GitHub Pages site automatically.
The web page layout is configured using the config/tech-docs.yml file.
The template can be configured in config/tech-docs.yml
Further configuration options are described on the Tech Docs Template website: Global Configuration.
10th August 2023