[Request] Document Fleet's package management and setup process in detail

[kpollich]

Description

The Fleet team would like to create some documentation about Fleet's setup process in which we bootstrap various Fleet dependencies and assets like component templates, ingest pipelines, output settings, etc.

Collaboration

• The docs team will define with product team the structure and location, and the product team will provide the initial content

I'll provide an initial outline below that should hopefully give us a good start on content here, but we should determine the specifics about where/how to present this documentation in the comments.

Contact Person: @kpollich

Suggested Target Release

N/A

Stakeholders

• @joshdover
• @mostlyjason

Also cc @nchaulet for input and guidance.

Content outline provided below

Fleet setup

Fleet runs a series of setup steps when Kibana starts up to bootstrap various dependencies and apply any preconfiguration settings specified in kibana.yml. These steps are listed in the order of their execution below.

Note: we have a separate issue to improve our preconfig docs here #113

Default settings

Fleet configures a few defaults for settings if they are not provided via preconfiguration or in the case of a first-time setup where settings have not been set in the UI.

• In cloud, Fleet resolves a Fleet Server host URL automatically and sets it as a default
• Fleet sets a default output pointed at Elasticsearch

Elasticsearch assets

Fleet creates the following Elasticsearch assets:

• A default component template named .fleet-component-template-{version} that contains common index settings and mappings for data sent via Elastic Agent
• An ingest pipeline named .fleet-final-pipeline-1 that applies some final processing to data sent via Elastic Agent

Integration installation and updates

• Install any integrations specified in preconfiguration
• Determine any installed integrations that qualify for automatic updates and install their latest versions

Bundled integrations

In 8.1, Kibana will begin shipping with "bundled" integrations to support offline upgrades for select Elastic product integrations like APM and Synthetics.

When Kibana is upgraded and a new version of one of these bundled integrations is detected in the Kibana source, it will be installed from disk, and will not require an available connection to the Elastic Package Registry. This improves support for air-gapped environments and ensures stability when upgrading and booting Kibana.

As of 8.1, these bundled integrations are:

• APM
• Elastic Agent
• Endpoint
• Fleet Server
• Synthetics

Package policy upgrades

Fleet will detect any integrations with the "Keep Policies up to Date" setting enabled and apply any policy upgrades in the case where these integrations have been updated.

Default enrollment token

Fleet will create an enrollment token for any preconfigured agent policies.

[mostlyjason]

Thanks for writing this! My main question is where this will live since its not something users need for our most common use cases. Perhaps it'd fit with this architecture doc we intended to write at some point? #127

[nchaulet]

Fleet sets a default output pointed at Elasticsearch

On premise Fleet set a default output pointed at what is in xpack.fleet.agents.elasticsearch.hosts or localhost:9200

[joshdover]

Should we include details on the finer points of how preconfigured outputs, package policies, and agent policies handle changes to the preconfigured values? I don't think we need to go back and document this for the older versions since there's been quite a few small tweaks, but I think going forward it'd be nice to have this behavior written down somewhere. If not in the public docs, then at least within our internal Fleet dev docs.

Thanks for writing this! My main question is where this will live since its not something users need for our most common use cases.

I think this could make sense to include or at least link to from the Fleet settings in Kibana doc as well.

We should also include a link or some of this information about bundled packages in the air-gapped environments document.