Skip to content

Commit

Permalink
Integrations developer guide (elastic#769)
Browse files Browse the repository at this point in the history
Co-authored-by: sorantis <dimitri.mazmanov@elastic.co>
Co-authored-by: mtojek <marcin.tojek@elastic.co>
Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>
  • Loading branch information
4 people committed Sep 7, 2021
1 parent c75d263 commit a732a67
Show file tree
Hide file tree
Showing 15 changed files with 2,181 additions and 0 deletions.
96 changes: 96 additions & 0 deletions docs/en/integrations/air-gapped.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
[[air-gapped]]
= Air-gapped environments

There are certain environments in which network traffic restrictions are mandatory. In these environments, the Kibana instance
isn't able to reach the public Elastic Package Registry (EPR) endpoints, like https://epr.elastic.co/[epr.elastic.co], to download
package metadata and content.

There are two workarounds in this situation -- use a proxy server as network gateway to reach the public endpoints,
or deploy your own instance of the Elastic Package Registry.

[discrete]
[[air-gapped-proxy-server]]
== Use a proxy server

If you can route traffic to the public endpoint of EPR through a network gateway, there is a property in Kibana that
can orchestrate to use a proxy server:

[source,yaml]
----
xpack.ingestManager.registryProxyUrl: your-nat-gateway.corp.net
----

For more information, see the {fleet-guide}/fleet-overview.html#package-registry-intro[Fleet User Guide].

[discrete]
[[air-gapped-diy-epr]]
== Host your own Elastic Package Registry

IMPORTANT: This procedure is not officially supported.

If routing traffic through a proxy server is not an option, you can host your own Elastic Package Registry.
The Package Storage instance must be deployed and hosted on-site as a Docker container.
Package Storage is a special distribution of the Package Registry which already includes packages.
There are different distributions available:

* production (recommended): `docker.elastic.co/package-registry/distribution:production` - stable, tested package revisions
* staging: `docker.elastic.co/package-registry/distribution:staging` - package revisions ready for testing before release
* snapshot: `docker.elastic.co/package-registry/distribution:snapshot` - package revisions updated on daily basis

If you want to update the Package Storage image, you need to re-pull the image and restart docker container.

Every distribution contains packages that can be used by different versions of the {stack}. As we adopted a continuous delivery pipeline for packages,
we haven't introduced the box release approach so far (7.13.0, 7.14.0, etc.). The Package Registry API exposes a {kib} version constraint
that allows for filtering packages that are compatible with particular stack version.

NOTE: These steps use the standard Docker CLI, but it shouldn't be hard to transform them into Kubernetes descriptor file.
Here is the k8s descriptor used by the e2e-testing project: https://github.com/elastic/e2e-testing/blob/k8s-deployment/cli/config/kubernetes/base/package-registry/[yaml files].

1. Pull the Docker image from the public Docker registry:
+
[source,bash]
----
docker pull docker.elastic.co/package-registry/distribution:production
----

2. Save the Docker image locally:
+
[source,bash]
----
docker save -o epr.tar docker.elastic.co/package-registry/distribution:production
----
+
TIP: please mind the image size, so you won't hit any capacity limit.

3. Transfer the image to the air-gapped environment and load:
+
[source,bash]
----
docker load -i epr.tar
----

4. Run the Package Registry:
+
[source,bash]
----
docker run -it docker.elastic.co/package-registry/distribution:production
----

5. (Optional) Define the internal healthcheck for the service as:
+
[source,bash]
----
curl -f http://127.0.0.1:8080
----

[discrete]
[[air-gapped-diy-epr-kibana]]
=== Connect Kibana to the hosted Package Registry

There is a dedicated property in the {kib} config to change the URL of the Package Registry's endpoint to a custom one.
The example below connects to an internally hosted instance:

[source,yaml]
----
xpack.fleet.registryUrl: "http://package-registry.corp.net:8080"
----
67 changes: 67 additions & 0 deletions docs/en/integrations/asset-testing.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
[[asset-testing]]
= Asset testing

Elastic Packages define assets to be loaded into {es} and {kib}. Asset loading tests exercise install a package to ensure that its assets are loaded into {es} and {kib} as expected.

[discrete]
[[asset-testing-concepts]]
== Conceptual process

Conceptually, running an asset load test involves the following steps:

. Build the package.
. Deploy {es}, {kib}, and the {package-registry} (all of which are part of the {stack}). This step takes time, so you should typically do it once as a pre-requisite to running asset loading tests on multiple packages.
. Install the package.
. Use various {kib} and {es} APIs to confirm that the package assets were loaded into {kib} and {es} as expected.
. Remove the package.

[discrete]
[[define-asset-test]]
== Define an asset loading test

As a package developer, there is no work required to define an asset loading test for your package. All the necessary information is contained in the package files.

[discrete]
[[running-asset-test]]
== Run an asset loading test

First, you must build your package. This step corresponds to step 1 in the <<asset-testing-concepts,Conceptual process>> section.

Navigate to the root folder of the package, or any sub-folder under it, and run the following command.

[source,terminal]
----
elastic-package build
----

Next, deploy {es}, {kib}, and the {package-registry}. This step corresponds to step 2 in the <<asset-testing-concepts,Conceptual process>> section.

[source,terminal]
----
elastic-package stack up -d
----

To view a list of the available options for this command, run `elastic-package stack up -h` or `elastic-package help stack up`.

Next, set the environment variables that are required for additional `elastic-package` commands.

[source,terminal]
----
$(elastic-package stack shellinit)
----

Next, invoke the asset loading test runner. This step corresponds to steps 3 to 5 in the <<asset-testing-concepts,Conceptual process>> section.

Navigate to the root folder of the package, or any sub-folder under it, and run the following command.

[source,terminal]
----
elastic-package test asset
----

Finally, when all the asset loading tests have completed, bring down the {stack}. This step corresponds to step 4 in the <<asset-testing-concepts,Conceptual process>> section.

[source,terminal]
----
elastic-package stack down
----
Loading

0 comments on commit a732a67

Please sign in to comment.