[Discuss] Support managed downgrades

[joshdover]

In some scenarios, users may want to cancel an Agent upgrade rollout and revert back to a previous version. For example, if there is a bug in a newer version of Agent that is affecting their configuration.

Downgrading the Agent in place is quite complex and may be hard to support. New features added in future versions may be impossible or very costly to support in a way that can be downgraded, for example:

• Secret management and storage on the host
• Encrypted disk queues
• Schema of local state managed by inputs

An alternative approach could be to support downgrades via an unenroll and re-enroll flow, this could work like (very hand wave-y):

• Fleet Server sends down a DOWNGRADE action that includes an enrollment token for the same policy
• Agent "soft" unenrolls by emptying remaining data in the queues and then uninstalling itself
• Agent then reinstalls itself on the older version with the enrollment token sent by Fleet Server
  • How this process happens is the most ambiguous aspect. Could we put a script into a tmp directory that gets executed after the uninstall completes?

We may also want to support preserving the agent.id during this flow for minimal disruption in ingest processing and data correlation.

[lucabelluccini]

We might follow same strategies used by Over-the-air firmware updates (e.g. https://www.ti.com/lit/wp/sway021/sway021.pdf?ts=1662466900068&ref_url=https%253A%252F%252Fwww.google.com%252F or https://mongoose-os.com/docs/mongoose-os/userguide/ota.md#implementation-details)