Convert fleet-agent to controller-runtime

[manno]

We discussed wrangler vs controller-runtime multiple times. How feasible is it to switch frameworks?

Fleet-agent is relatively limited. When migrating to controller-runtime:

• how easy is it to port the different "timers", time.After,EnqueueAfter
• are we missing something when we drop apply, e.g. cleanup, updates
• agent sets up dynamic watches on deployed resources, ?
• should we extract existing "skip" logic into predicates/event filters first or are wrangler handlers too different?
• will probably need a separate go.mod

If it's not possible to move the "monitor" part of the agent, because of timers and internal/cmd/agent/trigger/watcher.go, does it help to split the agent, so we can at least port the deployer?

---

The fleet-agent controller was split into three processes:

• registration
• cluster status updater
• fleet-agent controller, watches bundledeployments

Only the last part is an actual controller. Cluster status updates work on a timer and registration is done when the agent starts up. A new function registration.Get was added, to just return an existing registration to the other processes.

The agent to a statefulset, to avoid leader election. The code of the handlers was spread between the "handler", the deploy "manager" and the helm deployer. It has been consolidated into the deployer package.

"Trigger" was renamed to driftdetect as it uses the helm history to detect drift for deployments. Which is similar to themonitor, which updates the bundle deployments status.

After splitting the agent, the fleet-agent controller was converted to controller-runtime:

• new bundledeployment controller + reconciler
• agent reconciler only sets status when exiting reconcile
• when deployed resources change, re-trigger bundledeployment via status update
• simplify bundlestatus condition handling
• switch to zap logging

[aruiz14]

After chatting about this, and how we are using leader election, I was wondering if it is really necessary.
At the moment, it's used to prevent multiple instances of the fleet-agent (although it's deployment specifies only 1 replica), to act on the same BundleDeployment resource, ending up with repeated work or conflicts in the worst case.

I believe we can use an alternative approach based on optimistic locking, hence eliminating the restriction of having one single active instance:

1. BundleDeployment status reflects that an instance is already processing it (e.g. a new in progress condition).
2. When an instance of fleet-agent observes an actionable BundleDeployment, and tries to update the status sub-resource to in progress.
   • If it was already updated by another instance, Kubernetes will reject the change due to different resource version, should abort any further operation on that resource.
   • If it succeeds, it can continue with the deployment.
3. In the event of stale "in progress" BundleDeployment(e.g. the fleet-agent instance died in the middle of a deployment and it was not able to recover its progress), the fleet-controller must apply some expiration period and decide to mark it as an error or unfreeze the object by removing the in-progress condition.

This is actually not tied to the use of controller-runtime, but I thought I should bring it up here since it may be relevant to your current work.

[manno]

I mentioned this doc about wrangler handlers: https://github.com/rancher/rancher/wiki/Tips-for-Writing-Rancher-Handlers#why-was-my-object-enqueued