Autodiscovery guide updates #1357

kmshultz · 2017-06-16T23:21:50Z

TODO:

Verify and finish list of caveats under #how-it-works/#different-execution.
Clean up #service-identifiers section at the end
Add more detail to Kubernetes instructions, i.e. actual commands

kmshultz · 2017-06-16T23:29:06Z

I've introduced a few new terms here:

service identifier — the document needs a convenient way to refer to image name/label. I considered 'container identifier' but that may not be so backend-agnostic (one day when we have non-Docker backends)
template source — what files, KV stores, and K8s annotations are. It would seem we already had a term like this—Configuration Backend—except it only applies to KV stores. Calling them all configuration backends could create confusion that e.g. 'kubernetes' is a valid value for SD_CONFIG_BACKEND.

Voice any opposition to these terms, dear readers, or forever hold your peace.

cc @technovangelist if others concur on these terms, we should use them in the Autodiscovery video, too.

xvello

Some additions, but a solid improvement on the existing page!

xvello · 2017-06-21T09:15:18Z

content/guides/autodiscovery.md

+There are a few caveats:
+
+- The Agent caches templates it gets from KV stores. Changes to KV store templates require a restart of the Agent.
+- something about precedence of templates when multiple sources provided?


KV-store templates take precendence over conf.d/auto_conf files indeed

@xvello got it. And KV takes precedence over k8s annotations, which take precedence over files?

k8s annotations take precedence over templates (KV then files)

xvello · 2017-06-21T09:16:07Z

content/guides/autodiscovery.md

+
+There are a few caveats:
+
+- The Agent caches templates it gets from KV stores. Changes to KV store templates require a restart of the Agent.


Reading the source, it looks like we are watching for changes in the KV-store and reloading templates when one is detected. @hkaj is that working on all three supported backends?

xvello · 2017-06-21T09:23:53Z

content/guides/autodiscovery.md

@@ -48,155 +86,169 @@ By default, the Datadog Agent includes Autodiscovery support for:
 - Redis
 - Riak

-These are provided by the configuration templates in the Datadog Agent `conf.d/auto_conf` directory.
+Storing templates as local files is easy to understand and doesn't require an external service. The downside is that you must redeploy the Agent container each time you change, add, or remove templates. You may also have to maintain your own docker-dd-agent container if you want to add your own templates.


Should we mention the /conf.d volume they can expose, instead of rebuilding the image? See https://github.com/DataDog/docker-dd-agent/blob/master/entrypoint.sh#L172

Ok, got it. But I think no matter what, users must restart Agent containers to get new check configs, correct? i.e. there is no SIGHUP or similar way to tell Agent to reload the files?

In the case of host deployment, restarting the collector (supervisorctl restart datadog-agent:collector) is enough. But copying conf files from /conf.d is handled by the container's entrypoint, so a container restart is indeed needed

xvello · 2017-06-21T09:29:58Z

content/guides/autodiscovery.md


-Let's take the example of the port variable: a RabbitMQ container with the management module enabled has 6 exposed ports by default. The list of ports as seen by the agent is: `[4369, 5671, 5672, 15671, 15672, 25672]`. **Notice the order. The Agent always sorts values in ascending order.**
+Autodiscovery supports Consul, etcd, and Zookeeper as template sources. To use a KV store, configure its parameters in `datadog.conf` or in environment variables passed to docker-dd-agent when starting the container.


we could link environment variables to https://github.com/DataDog/docker-dd-agent/#environment-variables

xvello · 2017-06-21T09:33:29Z

content/guides/autodiscovery.md


-### Configuring etcd or Consul in `datadog.conf`
+If you are using Consul and the Consul cluster requires token authentication, set `consul_token`.


FWI, 5.15 will support user/pass auth for etcd. See DataDog/dd-agent#3357

xvello · 2017-06-21T09:38:15Z

content/guides/autodiscovery.md

+
+You can also add a network name suffix to the `%%host%%` variable—`%%host_bridge%%`, `%%host_swarm%%`, etc—for containers attached to multiple networks. When `%%host%%` does not have a suffix, Autodiscovery picks the container's bridge network IP address.
+
+### Service Identifiers


This section should be expanded with examples and pushed further up the page.
See the ticket I sent you about user confusion on the matching

Will do, I hadn't addressed this section just yet. I did link to it from earlier in the page; you think it needs to be more prominent, though?

Well, I had two support cases that could've been solved by using container labels instead of toying around with the image names, so I'd vote for pushing it up. But that's probably sampling biais

technovangelist · 2017-06-21T17:05:47Z

re service identifiers. since services are a thing in k8s and a different thing in docker swarm, is it worth/possible to choose a different word, esp since this service is different from that service? in fact 'service' is a very overloaded term in this article

technovangelist · 2017-06-21T17:08:53Z

content/guides/autodiscovery.md


-Datadog automatically keeps track of what is running where, thanks to its Autodiscovery feature. Autodiscovery allows you to define configuration templates that will be applied automatically to monitor your containers.
+The Datadog Agent can automatically track which services are running where, thanks to its Autodiscovery feature. Autodiscovery lets you define configuration templates for Agent checks and specify which container types each check should apply to. The Agent enables, disables, and recompiles static check configurations from the templates as containers come and go. When your NGINX container moves from 10.0.0.6 to 10.0.0.17, Autodiscovery helps the Agent update its NGINX check configuration with the new IP address so it can keep collecting NGINX metrics without any action on your part.


...which container types...??? are they different types or just different containers?

The agent....recompiles.... ?? Recompiles? That sounds super impressive but is that accurate? maybe I am assuming too much from that word.

Hmm, yes, 'containers' is probably better, since individual containers of the same type may be targeted via labels.

Agreed, recompiles is a little puffy. 're-generates'?

technovangelist · 2017-06-21T17:13:01Z

content/guides/autodiscovery.md


-## How it works
+<div class="alert alert-info">
+Autodiscovery was previously called Service Discovery. It is still known as Service Discovery in the Agent's code and in configuration options.


this sounds a bit awkward. It was previously called...and it still is...

"Autodiscovery was previously called Service Discovery, and it's still called that in the Agent's code and in configuration options."

technovangelist · 2017-06-21T17:19:43Z

content/guides/autodiscovery.md


-The Autodiscovery feature watches for Docker events like when a container is created, destroyed, started or stopped. When one of these happens, the Agent identifies which service is impacted, loads the configuration template for this image, and automatically sets up its checks.
+In a traditional non-container environment, Datadog Agent configuration is, like the environment in which it runs, static. The Agent reads check configurations from disk when it starts, and as long as it's running, it continuously applies every configured check. The configuration files are static, and any network-related options configured within them serve to identify specific instances of a monitored service. When an Agent check cannot connect to such a service, it's probably not because the service was re-homed somewhere else; either the service is down, or the check configuration was unable to reach the service to begin with. In any case, the Agent continues to attempt connecting to the service until an administrator steps in to troubleshoot.


Thats a lot, of commas, in the, first, sentence.

for the second sentence, the way I read that is that if the config changes, the changes are applied right away without restarting the agent, but then you say the files are static. a bit confusing

When an Agent check cannot connect to such a service.... you aren't talking about a service and havent defined one..

either the service is down, or the check configuration was unable to reach the service to begin with. - the configuration cannot reach it, or the check cannot??

until an administrator steps in to troubleshoot... just the act of troubleshooting stops the attempt to connect? wont it still try. If the admin successfully solves the problem rather than just troubleshooting, i hope it will still attempt to connect.

Thats a lot, of commas, in the, first, sentence.

In a traditional non-container environment, Datadog Agent configuration is—like the environment in which it runs—static.

for the second sentence, the way I read that is that if the config changes, the changes are applied right away without restarting the agent

Which words tripped you up, 'continuously applies'? If so, is 'continuously runs' better?

When an Agent check cannot connect to such a service.... you aren't talking about a service and havent defined one..

The previous sentence says "any network-related options configured within them serve to identify specific instances of a monitored service". Would a parenthetical example have made it clear? "...instances of a monitored service (e.g. a redis instance)".

technovangelist · 2017-06-21T17:20:39Z

content/guides/autodiscovery.md


-Configuration templates can be defined by simple template files or as single key-value stores using etcd or Consul.
+With Autodiscovery enabled, the Agent runs checks differently.


is this sentence necessary?

It's kind of blog posty, but it makes the transition to the next two sections less jarring. It matches the section names, i.e. 'Different '

technovangelist · 2017-06-21T17:22:37Z

content/guides/autodiscovery.md


-To use Autodiscovery, you'll first need to run the Datadog Agent as a service.
+First, Autodiscovery uses **templates** for check configuration, wherein two template variables—`%%host%%` and `%%port%%`—appear in place of any normally-hardcoded network option values. Because orchestration platforms like Docker Swarm deploy (and redeploy) containers on arbitrary hosts, static configuration files are not suitable for checks that collect data from network endpoints. For example: a template for the Agent's [Go Expvar check](https://github.com/DataDog/integrations-core/blob/master/go_expvar/conf.yaml.example) would contain the option `expvar_url: http://%%host%%:%%port%%`. For containers that have more than one IP or exposed port, Autodiscovery can pick the right one(s) using [template variable indexes](#template-variable-indexes).


normally-hardcoded? is it better to just say hardcoded?

%%port%%`—appear .... missing a space?

I intentionally left the space out. We haven't been consistent in the docs on whether we have a space on either side of an em-dash, but lately I've been leaving them out.

technovangelist · 2017-06-21T17:36:27Z

content/guides/autodiscovery.md


-Let's take the example of the port variable: a RabbitMQ container with the management module enabled has 6 exposed ports by default. The list of ports as seen by the agent is: `[4369, 5671, 5672, 15671, 15672, 25672]`. **Notice the order. The Agent always sorts values in ascending order.**
+Autodiscovery supports Consul, etcd, and Zookeeper as template sources. To use a KV store, configure its parameters in `datadog.conf` or in environment variables passed to docker-dd-agent when starting the container.


Autodiscovery supports.... key value based template sources

That's implied by the heading.

technovangelist · 2017-06-21T17:39:53Z

content/guides/autodiscovery.md


-To pass the settings listed above as environment variables when starting the Datadog Agent in Docker Swarm, you would run the command:
+Each template is defined as a three-tuple: check name, `init_config`, and `instances`. The `docker_images` option from the previous section, which was used to provide service identifiers to Autodiscovery, is not required here; for KV store template sources, service identifiers appear as first-level keys under `check_config`. (Also note, the file-based template in the previous section didn't need a check name; the Agent infers it from the filename.)


would the term be three tuple, or just tuple?

3-tuple is most accurate.

technovangelist · 2017-06-21T17:41:29Z

content/guides/autodiscovery.md


-### Template structure in key-value stores
+Notice that each of the three values is a list. Autodiscovery assembles list items into check configurations based on shared list indexes. In this case, it composes the first (and only) check from `check_names[0]`, `init_configs[0]` and `instances[0]`.


is it worth saying its a YAML formatted list (or is it aJSON formatted list)?

technovangelist · 2017-06-21T17:41:55Z

content/guides/autodiscovery.md


-Note that in the structure above, you may have multiple checks for a single container. For example you may run a Java service that provides an HTTP API, using the HTTP check and the JMX integration at the same time. To declare that in templates, simply add elements to the `check_names`, `init_configs`, and `instances lists`. These elements will be matched together based on their index in their respective lists.
+Again, the list orders matter. The HTTP check will only work if all its elements have the same index (1) across the lists (they do).


list orders or list order?

The HTTP check will only work if all its elements have the same index (1) across the lists (they do).
I don't understand this

technovangelist · 2017-06-21T17:44:25Z

content/guides/autodiscovery.md

+metadata:
+  name: apache
+  annotations:
+    service-discovery.datadoghq.com/apache.check_names: '["apache","http_check"]'


awesome, didn't know you could add multiple checks, but makes sense.

IIRC that's 5.14 only (must check the milestone). We could document that to avoid support cases with old versions

technovangelist · 2017-06-21T17:46:50Z

this feels like a long document. @kmshultz you talked about splitting it up into autodisco on k8s, autodisco on docker, etc. any further ideas on this? If i just care about docker swarm, i probably don't want to be confused about k8s.

kmshultz · 2017-06-22T19:46:19Z

Re: splitting this into smaller documents, I agree we should do that. But as the docs site is currently organized, there's no good way to do that, i.e. since the only semblance of organization we have now is /guides, /references, /integrations, and a few others. We could put AD usage instructions into the K8s integration page, the ECS page, etc (though there is no Docker Swarm integration page), but I'd rather have those pages link over to this guide using the relevant anchor, i.e. /guides/autodiscovery#kubernetes. Integrations pages should only 1) describe how to configure the integration, and 2) list their published metrics, events, and service checks.

When we redesign the docs site soon, we can organize it some other way, i.e. /agent/install#kubernetes, /agent/usage#kubernetes. But even then, I think some of the platform-agnostic info in this doc will remain here (and then, we'll no longer call it a "guide").

Thoughts @technovangelist @jyee @irabinovitch?

kmshultz · 2017-06-22T19:56:20Z

@technovangelist

re service identifiers. since services are a thing in k8s and a different thing in docker swarm, is it worth/possible to choose a different word, esp since this service is different from that service? in fact 'service' is a very overloaded term in this article

Perhaps 'container identifier' is indeed best. Care to offer a suggestion?

kmshultz · 2017-07-05T23:19:13Z

@xvello I've added some more content here. Mind giving it another look, please? 🙇

irabinovitch · 2017-07-06T06:28:21Z

content/guides/autodiscovery.md

 kind: guide
 listorder: 10
 ---

-Docker is being [adopted rapidly](https://www.datadoghq.com/docker-adoption/) and platforms like Docker Swarm, Kubernetes and Amazon's ECS make running services easier and more resilient by managing orchestration and replication across hosts. But all of that makes monitoring more difficult. How can you monitor a service which is dynamically shifting from one host to another?
+Docker is being [adopted rapidly](https://www.datadoghq.com/docker-adoption/). Orchestration platforms like Docker Swarm, Kubernetes, and Amazon ECS make running Docker-ized services easier and more resilient by managing orchestration and replication across hosts. But all of that makes monitoring more difficult. How can you reliably monitor a service which is unpredictably shifting from one host to another?


s/Docker/Containers/ ?

is unpredictably the right word? We didn't like dynamically?

would "dynamically being shifted" make more sense?

As docker is the only supported runtime for now, I'd go with writing docker explicitly, but we'll transition to a generic "containers" once rkt is supported

unpredictably, dynamically... I think it's a matter of taste. To me, "unpredictably" better evokes the problem. Do you think it's inaccurate, or you just don't like the tone/sound of it?

As docker is the only supported runtime for now, I'd go with writing docker explicitly, but we'll transition to a generic "containers" once rkt is supported

Indeed, and if we avoided saying Docker here, we'd want to remove it from the page's title, too.

irabinovitch · 2017-07-06T06:29:54Z

content/guides/autodiscovery.md

-## How it works
+<div class="alert alert-info">
+Autodiscovery was previously called Service Discovery. It's still called Service Discovery in the Agent's code and in Autodiscovery configuration options.
+</div>


Do we need to mention this?

I think so, because current client know and might search for the old term.
That could even be at the top of the page

@xvello at first I did have it at the top, just below the page title. Do you think that's better?

yes, as in "just so you know... now let's get it over with"

xvello · 2017-07-06T08:01:49Z

content/guides/autodiscovery.md

 kind: guide
 listorder: 10
 ---

-Docker is being [adopted rapidly](https://www.datadoghq.com/docker-adoption/) and platforms like Docker Swarm, Kubernetes and Amazon's ECS make running services easier and more resilient by managing orchestration and replication across hosts. But all of that makes monitoring more difficult. How can you monitor a service which is dynamically shifting from one host to another?
+Docker is being [adopted rapidly](https://www.datadoghq.com/docker-adoption/). Orchestration platforms like Docker Swarm, Kubernetes, and Amazon ECS make running Docker-ized services easier and more resilient by managing orchestration and replication across hosts. But all of that makes monitoring more difficult. How can you reliably monitor a service which is unpredictably shifting from one host to another?


As docker is the only supported runtime for now, I'd go with writing docker explicitly, but we'll transition to a generic "containers" once rkt is supported

xvello · 2017-07-06T08:10:01Z

content/guides/autodiscovery.md

+
+If you use Kubernetes, see the [Kubernetes integration page](http://docs.datadoghq.com/integrations/kubernetes/#installation) for instructions on running docker-dd-agent. If you use Amazon ECS, see [its integration page](http://docs.datadoghq.com/integrations/ecs/#installation).
+
+If you use Docker Swarm, run the following command on one of your manager nodes:

    docker service create \


We don't have agent install doc for swarm yet, we could add it here

But the agent setup page need a revamp as the k8s and docker ones are getting more and more complex (different ways to deploy all mushed in the same page with no hierarchy). I'm torn between revamping these pages or just create doc.dd.com pages and link to them.

xvello · 2017-07-06T08:11:56Z

content/guides/autodiscovery.md


-The configuration templates in `conf.d/auto_conf` directory are nearly identical to the example YAML configuration files provided in [the Datadog `conf.d` directory](https://github.com/DataDog/dd-agent/tree/master/conf.d), but with one important field added. The `docker_images` field is required and identifies the container image(s) to which the configuration template should be applied.
+1. Add them to each host that runs docker-dd-agent and [mount the directory that contains them](https://github.com/DataDog/docker-dd-agent#configuration-files) into the docker-dd-agent container when starting it
+1. Package them into your own release of docker-dd-agent


I'd go with:

Build a custom docker image based on docker-dd-agent with your custom templates added in the /etc/dd-agent/conf.d/auto_conf

xvello · 2017-07-06T08:13:54Z

content/guides/autodiscovery.md


-## Configuration templates with key-value stores
+If this is too limiting—if you need to apply different check configurations to different containers running the same image—use [labels](#container-labels) to identify the containers. Label each container differently, then add each label to any template file's `docker_images` list (yes, `docker_images` is where to put _any_ kind of container identifier, not just images).


xvello · 2017-07-06T08:17:59Z

content/guides/autodiscovery.md

+metadata:
+  name: apache
+  annotations:
+    service-discovery.datadoghq.com/apache.check_names: '["apache","http_check"]'


IIRC that's 5.14 only (must check the milestone). We could document that to avoid support cases with old versions

xvello · 2017-07-06T08:19:57Z

content/guides/autodiscovery.md

+
+If you provide a template for the same check type via multiple template sources, the Agent will prefer, in increasing order of preference:
+
+* Files


IIRC files are last in the precedence order. Annotations are first for sure, so order is Annotations -> K/V -> files

I think it's better to list them in order of lookup instead of increasing order of preference (as that's the opposite, right?)

Ok, maybe this is confusing because I put it in ascending order of precedence. I thought another engineer told me KV takes highest precedence but perhaps I misunderstood.

kmshultz · 2017-07-06T16:38:28Z

I'm torn between revamping these pages or just create doc.dd.com pages and link to them.

@xvello Re: the Agent install instructions for each platform, I agree we need a separate space for this. With the Docs redesign underway very soon, I'm planning to have a dedicated section for the Agent (i.e. docs.datadoghq.com/agent) where we comprehensively cover its architecture (different daemons), install instructions for different platforms (similar to dogweb), all configuration options and environment variables, etc.

I don't like having the install docs embedded in Integrations pages. And I'm not a big fan of having them in dogweb, either, though I realize it's a nice first-time-user flow to login and be presented with a one-liner to copy and paste.

Kent Shultz added 6 commits June 15, 2017 18:50

Initial commit Autodiscovery re-factor

a7eaed4

Significant refactoring

95b2a66

More updates

e48201e

More edits

14fd43f

Minor edits

6d5c67c

More edits

4bb034c

kmshultz requested review from irabinovitch, jyee, technovangelist and hkaj June 16, 2017 23:21

Kent Shultz added 12 commits June 16, 2017 16:59

Styling updates to note

fd03d54

Minor edit

637a7ad

Typo

91588f2

More updates

361198d

Cooler headings

5a0a2b4

More edits

ee7032e

Some clarifications

66a31d9

Minor edits

2e0cc47

Minor edit

dda55e3

Few updates

edd1205

Condense Autodiscovery/Service Discovery caveat

507f905

Minor edit

971322a

xvello suggested changes Jun 21, 2017

View reviewed changes

technovangelist reviewed Jun 21, 2017

View reviewed changes

Address some comments

2bcbe01

Kent Shultz added 9 commits June 30, 2017 11:40

Minor wording edits

f4c9fe3

More updates

e08d6ac

More updates

de05f2d

More updates

8231463

More updates

882cb4f

More updates

9d6110e

More updates

5752199

Address #1260; other edits

d730922

More updates; add troubleshooting section

9cb6fe2

Kent Shultz added 2 commits July 5, 2017 16:38

Minor updates

4044abc

Few more edits

b3de5b8

kmshultz mentioned this pull request Jul 5, 2017

Adding some highlight and notes for the service discovery #1260

Closed

kmshultz changed the title ~~[WIP] Autodiscovery guide updates~~ Autodiscovery guide updates Jul 5, 2017

irabinovitch reviewed Jul 6, 2017

View reviewed changes

irabinovitch requested review from irabinovitch and shashankpradhan July 6, 2017 06:31

xvello reviewed Jul 6, 2017

View reviewed changes

Kent Shultz added 8 commits July 6, 2017 09:53

Reverse order of template source precedence

bb771d3

Put SD->AD caveat at very top of the page

cfabca8

Minor edit

9bd9a64

Change wording for Build Your Own Container

5a5c3e9

Few updates

1fb3776

Address xvello comments

8063e45

Minor update

ddba431

More polishing

325db99

kmshultz merged commit 16cd602 into master Jul 7, 2017

kmshultz deleted the kent/autodiscovery-guide-updates branch July 20, 2017 16:01


		There are a few caveats:

		- The Agent caches templates it gets from KV stores. Changes to KV store templates require a restart of the Agent.


		Let's take the example of the port variable: a RabbitMQ container with the management module enabled has 6 exposed ports by default. The list of ports as seen by the agent is: `[4369, 5671, 5672, 15671, 15672, 25672]`. Notice the order. The Agent always sorts values in ascending order.
		Autodiscovery supports Consul, etcd, and Zookeeper as template sources. To use a KV store, configure its parameters in `datadog.conf` or in environment variables passed to docker-dd-agent when starting the container.


		### Configuring etcd or Consul in `datadog.conf`
		If you are using Consul and the Consul cluster requires token authentication, set `consul_token`.


		You can also add a network name suffix to the `%%host%%` variable—`%%host_bridge%%`, `%%host_swarm%%`, etc—for containers attached to multiple networks. When `%%host%%` does not have a suffix, Autodiscovery picks the container's bridge network IP address.

		### Service Identifiers


		Datadog automatically keeps track of what is running where, thanks to its Autodiscovery feature. Autodiscovery allows you to define configuration templates that will be applied automatically to monitor your containers.
		The Datadog Agent can automatically track which services are running where, thanks to its Autodiscovery feature. Autodiscovery lets you define configuration templates for Agent checks and specify which container types each check should apply to. The Agent enables, disables, and recompiles static check configurations from the templates as containers come and go. When your NGINX container moves from 10.0.0.6 to 10.0.0.17, Autodiscovery helps the Agent update its NGINX check configuration with the new IP address so it can keep collecting NGINX metrics without any action on your part.


		The Autodiscovery feature watches for Docker events like when a container is created, destroyed, started or stopped. When one of these happens, the Agent identifies which service is impacted, loads the configuration template for this image, and automatically sets up its checks.
		In a traditional non-container environment, Datadog Agent configuration is, like the environment in which it runs, static. The Agent reads check configurations from disk when it starts, and as long as it's running, it continuously applies every configured check. The configuration files are static, and any network-related options configured within them serve to identify specific instances of a monitored service. When an Agent check cannot connect to such a service, it's probably not because the service was re-homed somewhere else; either the service is down, or the check configuration was unable to reach the service to begin with. In any case, the Agent continues to attempt connecting to the service until an administrator steps in to troubleshoot.


		Configuration templates can be defined by simple template files or as single key-value stores using etcd or Consul.
		With Autodiscovery enabled, the Agent runs checks differently.


		To use Autodiscovery, you'll first need to run the Datadog Agent as a service.
		First, Autodiscovery uses templates for check configuration, wherein two template variables—`%%host%%` and `%%port%%`—appear in place of any normally-hardcoded network option values. Because orchestration platforms like Docker Swarm deploy (and redeploy) containers on arbitrary hosts, static configuration files are not suitable for checks that collect data from network endpoints. For example: a template for the Agent's [Go Expvar check](https://github.com/DataDog/integrations-core/blob/master/go_expvar/conf.yaml.example) would contain the option `expvar_url: http://%%host%%:%%port%%`. For containers that have more than one IP or exposed port, Autodiscovery can pick the right one(s) using [template variable indexes](#template-variable-indexes).


		To pass the settings listed above as environment variables when starting the Datadog Agent in Docker Swarm, you would run the command:
		Each template is defined as a three-tuple: check name, `init_config`, and `instances`. The `docker_images` option from the previous section, which was used to provide service identifiers to Autodiscovery, is not required here; for KV store template sources, service identifiers appear as first-level keys under `check_config`. (Also note, the file-based template in the previous section didn't need a check name; the Agent infers it from the filename.)


		### Template structure in key-value stores
		Notice that each of the three values is a list. Autodiscovery assembles list items into check configurations based on shared list indexes. In this case, it composes the first (and only) check from `check_names[0]`, `init_configs[0]` and `instances[0]`.


		Note that in the structure above, you may have multiple checks for a single container. For example you may run a Java service that provides an HTTP API, using the HTTP check and the JMX integration at the same time. To declare that in templates, simply add elements to the `check_names`, `init_configs`, and `instances lists`. These elements will be matched together based on their index in their respective lists.
		Again, the list orders matter. The HTTP check will only work if all its elements have the same index (1) across the lists (they do).


		## Configuration templates with key-value stores
		If this is too limiting—if you need to apply different check configurations to different containers running the same image—use [labels](#container-labels) to identify the containers. Label each container differently, then add each label to any template file's `docker_images` list (yes, `docker_images` is where to put _any_ kind of container identifier, not just images).


		If you provide a template for the same check type via multiple template sources, the Agent will prefer, in increasing order of preference:

		* Files

Autodiscovery guide updates #1357

Autodiscovery guide updates #1357

Conversation

kmshultz commented Jun 16, 2017 • edited Loading

kmshultz commented Jun 16, 2017

xvello left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

technovangelist commented Jun 21, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

technovangelist commented Jun 21, 2017

kmshultz commented Jun 22, 2017

kmshultz commented Jun 22, 2017

kmshultz commented Jul 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xvello Jul 6, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kmshultz commented Jul 6, 2017 • edited Loading

kmshultz commented Jun 16, 2017 •

edited

Loading

technovangelist commented Jun 21, 2017 •

edited

Loading

xvello Jul 6, 2017 •

edited

Loading

kmshultz commented Jul 6, 2017 •

edited

Loading