From 31ce0de76c359e9a9c400d4870047197b45e65ce Mon Sep 17 00:00:00 2001 From: Jaime Soriano Pastor Date: Mon, 6 Dec 2021 18:59:43 +0100 Subject: [PATCH] Review documentation regarding self-managed package registry (#1287) Starting on 7.16 we are going to officialy support self-managed package registry deployments for air-gapped environments. Update documentation referring to this feature. --- .../beats-agent-comparison.asciidoc | 2 +- .../fleet/air-gapped.asciidoc | 126 ++++++++++++++++++ .../fleet/fleet-limitations.asciidoc | 2 +- docs/en/ingest-management/index.asciidoc | 2 + .../integrations/integrations.asciidoc | 7 + docs/en/ingest-management/overview.asciidoc | 3 + docs/en/integrations/air-gapped.asciidoc | 95 +------------ .../what-is-an-integration.asciidoc | 4 +- 8 files changed, 144 insertions(+), 97 deletions(-) create mode 100644 docs/en/ingest-management/fleet/air-gapped.asciidoc diff --git a/docs/en/ingest-management/beats-agent-comparison.asciidoc b/docs/en/ingest-management/beats-agent-comparison.asciidoc index eb24a0d15d..b3c6a895b2 100644 --- a/docs/en/ingest-management/beats-agent-comparison.asciidoc +++ b/docs/en/ingest-management/beats-agent-comparison.asciidoc @@ -388,7 +388,7 @@ The following table shows a comparison of capabilities supported by {beats} and |{y} |{n} |{y} -|The Integrations app requires a network connection to the {fleet-guide}/fleet-overview.html#package-registry-intro[Elastic Package Registry]. We are considering an https://github.com/elastic/integrations/issues/1178[on-prem version of EPR]. {fleet}-managed {agent}s require a connection to our artifact repository for agent binary upgrades. These are not required for standalone {agent}s or {beats}. +|The Integrations app requires a network connection to the {fleet-guide}/fleet-overview.html#package-registry-intro[{package-registry}], or a {fleet-guide}/air-gapped.html#air-gapped-diy-epr[self-managed deployment of the {package-registry}]. {fleet}-managed {agent}s require a connection to our artifact repository for agent binary upgrades. These are not required for standalone {agent}s or {beats}. |Run without root on host |{y} diff --git a/docs/en/ingest-management/fleet/air-gapped.asciidoc b/docs/en/ingest-management/fleet/air-gapped.asciidoc new file mode 100644 index 0000000000..1fb91dbbac --- /dev/null +++ b/docs/en/ingest-management/fleet/air-gapped.asciidoc @@ -0,0 +1,126 @@ +[[air-gapped]] += Air-gapped environments + +There are certain environments in which network traffic restrictions are mandatory. In these environments, the {kib} instance +isn't able to reach the public {package-registry} endpoints, like https://epr.elastic.co/[epr.elastic.co], to download +package metadata and content. + +There are two workarounds in this situation -- use a proxy server as network gateway to reach the public endpoints, +or deploy your own instance of the {package-registry}. + +[discrete] +[[air-gapped-proxy-server]] +== Use a proxy server + +If you can route traffic to the public endpoint of the {package-registry} through a network gateway, there is a property in {kib} that +can orchestrate to use a proxy server: + +[source,yaml] +---- +xpack.fleet.registryProxyUrl: your-nat-gateway.corp.net +---- + +[discrete] +[[air-gapped-diy-epr]] +== Host your own {package-registry} + +If routing traffic through a proxy server is not an option, you can host your own {package-registry}. + +The {package-registry} can be deployed and hosted on-site using one of the +available Docker images. These docker images include the {package-registry} and +a selection of packages. + +There are different distributions available: + +* {version} (recommended): +docker.elastic.co/package-registry/distribution:{version}+ - Selection of packages from the production repository released with the {version} version of the {stack}. +* production: `docker.elastic.co/package-registry/distribution:production` - Packages available in the production registry (https://epr.elastic.co). +* staging: `docker.elastic.co/package-registry/distribution:staging` - Packages available in the staging registry (https://epr-staging.elastic.co). These packages may be pending of validation. +* snapshot: `docker.elastic.co/package-registry/distribution:snapshot` - Packages under development. + +ifeval::["{release-state}"=="unreleased"] +[WARNING] +==== +Version {version} of the {package-registry} distribution has not yet been released. +==== +endif::[] + +To update the distribution image, re-pull the image and then restart the docker container. + +Every distribution contains packages that can be used by different versions of +the {stack}. The {package-registry} API exposes a {kib} version constraint that +allows for filtering packages that are compatible with a particular version. + +NOTE: These steps use the standard Docker CLI, but you can create a Kubernetes manifest +based on this information. +These images can also be used with other container runtimes compatible with Docker images. + +1. Pull the Docker image from the public Docker registry: ++ +["source", "sh", subs="attributes"] +---- +docker pull docker.elastic.co/package-registry/distribution:{version} +---- ++ +2. Save the Docker image locally: ++ +["source", "sh", subs="attributes"] +---- +docker save -o package-registry-{version}.tar docker.elastic.co/package-registry/distribution:{version} +---- ++ +TIP: Check the image size to ensure that you have enough disk space. + +3. Transfer the image to the air-gapped environment and load it: ++ +["source", "sh", subs="attributes"] +---- +docker load -i package-registry-{version}.tar +---- + +4. Run the {package-registry}: ++ +["source", "sh", subs="attributes"] +---- +docker run -it -p 8080:8080 docker.elastic.co/package-registry/distribution:{version} +---- + +5. (Optional) You can monitor the health of your {package-registry} with +requests to the root path: ++ +["source", "sh", subs="attributes"] +---- +docker run -it -p 8080:8080 \ + --health-cmd "curl -f -L http://127.0.0.1:8080/" \ + docker.elastic.co/package-registry/distribution:{version} +---- + +[discrete] +[[air-gapped-diy-epr-kibana]] +=== Connect {kib} to your hosted {package-registry} + +Use the `xpack.fleet.registryUrl` property in the {kib} config to set the URL of your hosted package registry. For example: + +[source,yaml] +---- +xpack.fleet.registryUrl: "http://package-registry.corp.net:8080" +---- + +[discrete] +[[air-gapped-tls]] +=== TLS configuration of the {package-registry} + +You can configure the {package-registry} to listen on a secure HTTPS port using TLS. + +For example, given a key and a certificate pair available in `/etc/ssl`, you +can start the {package-registry} listening on the 443 port using the following command: + +["source", "sh", subs="attributes"] +---- +docker run -it -p 443:443 \ + -v /etc/ssl/package-registry.key:/etc/ssl/package-registry.key:ro \ + -v /etc/ssl/package-registry.crt:/etc/ssl/package-registry.crt:ro \ + -e EPR_ADDRESS=0.0.0.0:443 \ + -e EPR_TLS_KEY=/etc/ssl/package-registry.key \ + -e EPR_TLS_CERT=/etc/ssl/package-registry.crt \ + docker.elastic.co/package-registry/distribution:{version} +---- diff --git a/docs/en/ingest-management/fleet/fleet-limitations.asciidoc b/docs/en/ingest-management/fleet/fleet-limitations.asciidoc index cbe3c5bfb3..4f3bf95db6 100644 --- a/docs/en/ingest-management/fleet/fleet-limitations.asciidoc +++ b/docs/en/ingest-management/fleet/fleet-limitations.asciidoc @@ -13,7 +13,7 @@ This feature has additional limitations at the current time: * Support for a limited number of integrations (more coming soon) * No output to {ls}, Kafka, or other remote clusters * No proxy support for {fleet-server} -* Requires internet access for {kib} to download integration packages from the Elastic Package Registry +* Requires internet access or a <> for {kib} to download integration packages. * {agent} requires internet access to perform binary upgrades from Fleet * Limited support in the {fleet} app for advanced {beats} settings like multiline, processors, and so on diff --git a/docs/en/ingest-management/index.asciidoc b/docs/en/ingest-management/index.asciidoc index 59f91c070a..effcc23164 100644 --- a/docs/en/ingest-management/index.asciidoc +++ b/docs/en/ingest-management/index.asciidoc @@ -50,6 +50,8 @@ include::integrations/upgrade-integration.asciidoc[leveloffset=+2] include::integrations/uninstall-integration.asciidoc[leveloffset=+2] +include::fleet/air-gapped.asciidoc[leveloffset=+2] + include::elastic-agent/install-elastic-agent.asciidoc[leveloffset=+1] include::elastic-agent/install-fleet-managed-elastic-agent.asciidoc[leveloffset=+2] diff --git a/docs/en/ingest-management/integrations/integrations.asciidoc b/docs/en/ingest-management/integrations/integrations.asciidoc index 81f6be73b3..1e9a79de05 100644 --- a/docs/en/ingest-management/integrations/integrations.asciidoc +++ b/docs/en/ingest-management/integrations/integrations.asciidoc @@ -45,3 +45,10 @@ too. |Remove an integration and its assets from {kib}. |=== + +[NOTE] +==== +The *Integrations* app in {kib} needs access to the public {package-registry} to +discover integrations. If your deployment has network restrictions, you can +{fleet-guide}/air-gapped.html#air-gapped-diy-epr[deploy your own self-managed {package-registry}]. +==== diff --git a/docs/en/ingest-management/overview.asciidoc b/docs/en/ingest-management/overview.asciidoc index 5ab15a21c3..9027643afd 100644 --- a/docs/en/ingest-management/overview.asciidoc +++ b/docs/en/ingest-management/overview.asciidoc @@ -58,6 +58,9 @@ To do so, add the following setting to your `kibana.yml` file: xpack.fleet.registryProxyUrl: ---- +You can find more information about running the {package-registry} in air-gapped +environments in the section about {fleet-guide}/air-gapped.html[Air-gapped environments]. + [discrete] [[configuring-integrations]] == {agent} {integrations} diff --git a/docs/en/integrations/air-gapped.asciidoc b/docs/en/integrations/air-gapped.asciidoc index fc462a2133..588cc90659 100644 --- a/docs/en/integrations/air-gapped.asciidoc +++ b/docs/en/integrations/air-gapped.asciidoc @@ -1,96 +1,5 @@ [[air-gapped]] = Air-gapped environments -There are certain environments in which network traffic restrictions are mandatory. In these environments, the Kibana instance -isn't able to reach the public Elastic Package Registry (EPR) endpoints, like https://epr.elastic.co/[epr.elastic.co], to download -package metadata and content. - -There are two workarounds in this situation -- use a proxy server as network gateway to reach the public endpoints, -or deploy your own instance of the Elastic Package Registry. - -[discrete] -[[air-gapped-proxy-server]] -== Use a proxy server - -If you can route traffic to the public endpoint of EPR through a network gateway, there is a property in Kibana that -can orchestrate to use a proxy server: - -[source,yaml] ----- -xpack.fleet.registryProxyUrl: your-nat-gateway.corp.net ----- - -For more information, see the {fleet-guide}/fleet-overview.html#package-registry-intro[Fleet and Elastic Agent Guide]. - -[discrete] -[[air-gapped-diy-epr]] -== Host your own Elastic Package Registry - -beta::[] - -If routing traffic through a proxy server is not an option, you can host your own Elastic Package Registry. -The Package Storage instance must be deployed and hosted on-site as a Docker container. -Package Storage is a special distribution of the Package Registry which already includes packages. -There are different distributions available: - -* production (recommended): `docker.elastic.co/package-registry/distribution:production` - stable, tested package revisions -* staging: `docker.elastic.co/package-registry/distribution:staging` - package revisions ready for testing before release -* snapshot: `docker.elastic.co/package-registry/distribution:snapshot` - package revisions updated on daily basis - -If you want to update the Package Storage image, you need to re-pull the image and restart docker container. - -Every distribution contains packages that can be used by different versions of the {stack}. As we adopted a continuous delivery pipeline for packages, -we haven't introduced the box release approach so far (7.13.0, 7.14.0, etc.). The Package Registry API exposes a {kib} version constraint -that allows for filtering packages that are compatible with particular stack version. - -NOTE: These steps use the standard Docker CLI, but it shouldn't be hard to transform them into Kubernetes descriptor file. -Here is the k8s descriptor used by the e2e-testing project: https://github.com/elastic/e2e-testing/blob/k8s-deployment/cli/config/kubernetes/base/package-registry/[yaml files]. - -1. Pull the Docker image from the public Docker registry: -+ -[source,bash] ----- -docker pull docker.elastic.co/package-registry/distribution:production ----- - -2. Save the Docker image locally: -+ -[source,bash] ----- -docker save -o epr.tar docker.elastic.co/package-registry/distribution:production ----- -+ -TIP: please mind the image size, so you won't hit any capacity limit. - -3. Transfer the image to the air-gapped environment and load: -+ -[source,bash] ----- -docker load -i epr.tar ----- - -4. Run the Package Registry: -+ -[source,bash] ----- -docker run -it docker.elastic.co/package-registry/distribution:production ----- - -5. (Optional) Define the internal healthcheck for the service as: -+ -[source,bash] ----- -curl -f http://127.0.0.1:8080 ----- - -[discrete] -[[air-gapped-diy-epr-kibana]] -=== Connect Kibana to the hosted Package Registry - -There is a dedicated property in the {kib} config to change the URL of the Package Registry's endpoint to a custom one. -The example below connects to an internally hosted instance: - -[source,yaml] ----- -xpack.fleet.registryUrl: "http://package-registry.corp.net:8080" ----- +You can find the documentation about {fleet} and the {package-registry} in +air-gapped environments in {fleet-guide}/air-gapped.html. diff --git a/docs/en/integrations/what-is-an-integration.asciidoc b/docs/en/integrations/what-is-an-integration.asciidoc index fdbb6bbe38..ac19b7ff6c 100644 --- a/docs/en/integrations/what-is-an-integration.asciidoc +++ b/docs/en/integrations/what-is-an-integration.asciidoc @@ -31,9 +31,9 @@ All packages must adhere to the <> -- a form . Publish the integration to the package registry + -Once an integration (package) has been created, it needs to be built. Built integrations are stored in the {package-storage-repo-link}[Package Storage repository] and served up via the {package-registry-repo-link}[Elastic Package Registry (EPR)]. +Once an integration (package) has been created, it needs to be built. Built integrations are stored in the {package-storage-repo-link}[Package Storage repository] and served up via the {package-registry-repo-link}[{package-registry}]. The {fleet} UI in {kib} connects to the {package-registry} and allows users to discover, install, and configure Elastic Packages. -The EPR can also be run locally for testing purposes. +The {package-registry} can also be {fleet-guide}/air-gapped.html#air-gapped-diy-epr[deployed on-premise in air-gapped environments]. . Install the integration +