feat: Add logic for queue controller (Part 1)

[Sovietaced]

Pull Request Template

Description

This pull request implements logic for the queue controller but does not wire it up to actually work yet. The queue controller reconciles queue custom resources against the Armada server so that queues can be managed by GitOps instead of using armadactl manually.

I wanted to break this pull request out since I think the wiring part will be where most of the discussion and back/forth will be. The next pull request will include code for how to configure the API connection to the Armada API server and how to load that config through helm charts.

As part of this pull request I re arranged some of the common helpers so they could be used by the queue reconciling logic too.

Related to #323

Please select the type of change your PR introduces:

• Bugfix
• Feature
• Code Style Update (formatting, renaming)
• Refactor (code changes that do not fix a bug or add a feature)
• Documentation Update
• Other (please describe):

How Has This Been Tested?

This pull request includes unit tests and has been tested against our production environment.

Checklist:

• My code follows the style guidelines of this project.
• I have performed a self-review of my code.
• I have commented on my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings.
• I have added tests that prove my fix is effective or that my feature works.
• New and existing unit tests pass locally with my changes.
• Any dependent changes have been merged and published in downstream modules.

[Sovietaced]

Moved this so it could be used by other controller logic

[Sovietaced]

Commenting this out for now to avoid any NPEs in this intermediate state where the queue client can be nil.

[dejanzele]

Given that queues can be updated outside of Operator/CRD, how will we reconcile those changes?

[Sovietaced]

Given that queues can be updated outside of Operator/CRD, how will we reconcile those changes?

I am under the impression that someone would want to use the Queue CRs for a GitOps workflow. In such a scenario manually manipulating queues through armadactl would seem like an anti-pattern.

Admittedly I'm not very familiar with building controllers so I'm not sure if reconciliation will happen without triggers from CR updates or not. If not, we might want to have some background trigger to prevent drift between the CR and reality.

[dejanzele]

@Sovietaced it would be an anti-pattern to update via armadactl or via REST/GRPC API, but nothing prevents users except a convention.

Kubernetes Operators watch and reconcile changes for CRDs, but it does not know anything about queues which are a concept in Armada, so it won't do anything if the queue got updated externally.
For initial version we could even start without it, and afterwards implement a watcher process which would check CRD Queues against the ones in database, and reconcile changes at some time intervals (or a smarter approach, this is just from top of my head)

[dejanzele]

@Sovietaced, on a successful reconcile, you could return ctrl.Result{RequeueAfter: requeueAfter} and control when you want to run the reconciliation loop again.