Architecture
We have a Kubernetes based architecture.
We have deployments for the backend, the frontend and the scheduler as well as a stateful set for the postgres database.
The frontend deployment serves all static frontend files to the user using an nginx webserver.
The backend deployment exposes all API functionality using the python flask server from the backend image.
The scheduler deployment periodically checks the postgres database for new jobs and starts them using the Kubernetes API. It also exposes routes to query available servers, job logs and to delete running jobs.
The postgres statefulset uses the Postgres HELM chart and stores all data models.
For each deployment/statefulset exists a service that routes to all pods that belong the deployment/statefulset. The backend and frontend services are exposed outside of the cluster using ingresses.
When a user accesses the URL, the frontend gets send to the user from the frontend deployment. The user then makes requests to the backend deployment using the frontend on his computer.
When the user starts a job, the backend deployment then creates a new waiting job entry in the postgres database. The scheduler periodically queries the postgres database for waiting jobs. If he finds one, he uses the Kubernetes API to create a new Job using the spec from the job (and experiment). The job runs and once it is finished, it sends its results to the backend which writes them into the postgres database and then sends a signal to the user using websockets.