Revisit configuration management to support modification via UI

[nscuro]

Most of the configuration options for new services are currently done via application.properties or environment variables.

This is in contrast to DT 4.x, which used primarily the UI. The drawback to UI configuration is that a new instance cannot easily be deployed with the desired settings in place, but requires manual adjustments.

However, there is still a requirement to have configuration options tweakable via UI. We have to find a way to allow for both models, as apparently relying on only one doesn't cut it.

[nscuro]

Given there is no way to achieve a truly consistent view of configurations for all applications in the system, without:

• Performing excessive database queries (i.e. at least one per processed Kafka record)
• Using a centralized solution such as ZooKeeper, etcd, Consul, that supports watches

The (hopefully final) proposal is as follows:

• Services query the database whenever they need to read a configuration
• For some services, these queries will be very low volume (e.g. mirror-service); Caching not strictly required, but doesn't hurt either
• For services where we anticipate a high volume of queries (e.g. vulnerability-analyzer), we utilize local read-through caching with short expiration intervals (few seconds up to 1min)

This should allow services like the vulnerability-analyzer to continue processing multiple thousands of records per second, without hammering the database. In the worst case, the cached configuration will be stale for up to 1min, but given the other options we have, that is actually not that bad.

Queries to fetch configuration options are lightweight. The biggest factor will be network latency, which the file-based configuration approach naturally didn't have.

[nscuro]

On the Quarkus side of things, I'm thinking we should plug into the existing config framework: https://quarkus.io/guides/config-extending-support#custom-config-source

By the looks of it, this can also be made to support reloads at runtime, and even change notification. Looking into it more.