Document the Fleet and Elastic Agent architecture

[dedemorton]

Need to add a topic with a diagram that shows:

• how all the components of the Fleet architecture work together (Fleet, Fleet Server, Elastic Agent, Integration packages, Kibana, Elasticsearch, etc)

• how data flows through the system and is processed.

• security and comms (see [DOCS]: Add security and comms info to Ingest Manager doc observability-docs#127) - the info will be in the FAQ for 7.10, but should be more visible when we go GA.

[dedemorton]

We should also explain Fleet management. I took this from something that @mostlyjason wrote, so there are more details here than we should expose externally. Putting the info here so we don't lose track.

There are two types of agent management in Fleet:

Infrastructure
The infrastructure of the agent includes the agent binary and settings installed on the host. This determines whether the agent is enrolled into Fleet, what version of the agent is running, and which agent policy. This can be managed by Fleet or an external management solution like ESS or K8s.

For additional context, the agent also has some capability restrictions like the input allowlist that allows operators to restrict what can run on their platform. It can only be set on the filesystem and cannot be managed by Fleet. These are essential to implement restrictions for the Elastic Cloud use case because we can restrict access to the filesystem to prevent changes.

Agent policy
This specifies the configuration for the Agent, which integrations are running, whether monitoring is enabled, which outputs are used, input settings, etc. This can be centrally managed by Fleet or the agent can run in standalone mode and get configuration from the filesystem.

[jen-huang]

While looking over enablement content from @mostlyjason, I suggested to add these points about Fleet which might be pertinent here too:

• Fleet and Integrations apps in Kibana require superuser role to access
• Fleet will automatically install some integrations by default when the apps are accessed for the first time: system, fleet_server, elastic_agent
• These integrations will be automatically upgraded when user visits the apps too
• Endpoint integration is also automatically upgraded if it is installed
• Fleet uses Kibana saved objects to store integration installation info, agent & integration policies
• Fleet reads from system ES indices for agent information (Fleet Server writes to these system indices)

[dedemorton]

@ollyhowell I'm working on a topic about Fleet Management to glue the sections of the reorganized Fleet/Agent Guide together. I think it makes sense for me to talk about the two types of management (infrastructure and policy) in the section that I'm working on right now, so I don't think you need to cover it in the overview. Next week, take a look at what I've written, and you can decide.

[dedemorton]

Looking at the 7.16 docs, there are some pretty glaring inaccuracies on the overview page right now (for example, integrations are no longer part of Fleet): https://www.elastic.co/guide/en/fleet/master/fleet-overview.html

I’m tempted to do a quick PR to fix the inaccuracies and repetition, but I don't want to duplicate effort.

@ollyhowell WDYT? Should I spruce up the existing overview now? I'm not sure how much you plan to touch it with the content you're writing.

EDITED: I've created an issue here that describes problems with the overview: elastic/observability-docs#1269

[dedemorton]

Closing because this issue is quite old and the details are probably stale. If this work is still required, please open a new issue.