Creating index template for Kibana to configure index replicas #1323

ewolinetz · 2018-08-24T18:20:27Z

Addresses #1315

richm · 2018-08-24T18:50:37Z

/retest

ewolinetz · 2018-08-24T20:43:34Z

/retest

lukas-vlcek · 2018-08-27T14:12:15Z

elasticsearch/index_templates/common.settings.kibana.template.json

+  "order": 0,
+  "settings": {
+    "index.number_of_replicas": $REPLICA_SHARDS,
+    "index.number_of_shards": PRIMARY_SHARDS


Missing $ ? ($PRIMARY_SHARDS)

lukas-vlcek · 2018-08-27T14:13:44Z

Left one comment.
Apart from this LGTM

ewolinetz · 2018-08-27T16:18:08Z

Squashed commits

ewolinetz · 2018-08-28T17:08:29Z

/hold
looking on my local deployment I don't see that Kibana has the correct replica count even though our template is there with replicas 0

$ oc exec -c elasticsearch logging-es-data-master-n95jnq3h-1-tvqg4 -- es_util --query=_template/common.settings.kibana.template.json

{"common.settings.kibana.template.json":{"order":0,"template":".kibana*","settings":{"index":{"number_of_shards":"1","number_of_replicas":"0"}},"mappings":{},"aliases":{}}}


health status index                                                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .kibana                                                         o0U2BGXbTt6U9LGLGaFJKw   1   1          1            0      3.2kb          3.2kb

ewolinetz · 2018-08-28T20:25:11Z

@lukas-vlcek any thoughts? it appears that .kibana is still being created with 1 replica even though my index template says otherwise..

lukas-vlcek · 2018-08-28T20:49:27Z

@ewolinetz let's check how exactly Kibana creates its indices. Can we locate particular line in the source code?
Also if we can get the Elasticsearch log which is related to .kibana index creation then this can explain how the index was created (which API was used and if any index templates were applied, all this info is in the ES log).

ewolinetz · 2018-08-28T22:59:46Z

@lukas-vlcek

From the ES logs:

...
[2018-08-28T21:58:44,153][WARN ][c.f.s.c.PrivilegesEvaluator] .kibana does not exist in cluster metadata
[2018-08-28T21:58:44,308][INFO ][o.e.c.m.MetaDataCreateIndexService] [logging-es-data-master-2kezn13a] [.kibana] creating index, cause [api], templates [], shards [1]/[1], mappings [_de
fault_, index-pattern, server, visualization, search, timelion-sheet, config, dashboard, url]
Elasticsearch Version: 5.6.10
Search Guard Version: <unknown>
Reload config on all nodes
Auto-expand replicas disabled
/usr/share/elasticsearch/init
[2018-08-28 21:58:46,797][INFO ][container.run            ] Updating replica count to 0
/etc/elasticsearch /usr/share/elasticsearch/init
Search Guard Admin v5
Will connect to localhost:9300 ... done
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initia
lization logging.
[2018-08-28T21:58:49,674][WARN ][c.f.s.a.BackendRegistry  ] Authentication finally failed for null
Elasticsearch Version: 5.6.10
Search Guard Version: <unknown>
[2018-08-28T21:58:54,593][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [logging-es-data-master-2kezn13a] updating number_of_replicas to [0] for indices [.searchguard]
Reload config on all nodes
Update number of replicas to 0 with result: true
/usr/share/elasticsearch/init
[2018-08-28 21:58:55,136][INFO ][container.run            ] Setting filters to allocate shard to this node only
{"acknowledged":true}[2018-08-28 21:58:55,428][INFO ][container.run            ] Adding index templates
[2018-08-28 21:58:55,679][INFO ][container.run            ] Create index template 'com.redhat.viaq-openshift-operations.template.json'
{"acknowledged":true}[2018-08-28 21:58:56,560][INFO ][container.run            ] Create index template 'com.redhat.viaq-openshift-orphaned.template.json'
{"acknowledged":true}[2018-08-28 21:58:57,508][INFO ][container.run            ] Create index template 'com.redhat.viaq-openshift-project.template.json'
{"acknowledged":true}[2018-08-28 21:58:58,129][INFO ][container.run            ] Create index template 'common.settings.kibana.template.json'
{"acknowledged":true}[2018-08-28 21:58:58,830][INFO ][container.run            ] Create index template 'common.settings.operations.orphaned.json'
{"acknowledged":true}[2018-08-28 21:58:59,392][INFO ][container.run            ] Create index template 'common.settings.operations.template.json'
{"acknowledged":true}[2018-08-28 21:59:00,078][INFO ][container.run            ] Create index template 'common.settings.project.template.json'
{"acknowledged":true}[2018-08-28 21:59:00,858][INFO ][container.run            ] Create index template 'org.ovirt.viaq-collectd.template.json'
{"acknowledged":true}[2018-08-28 21:59:01,367][INFO ][container.run            ] Finished adding index templates
...

From /usr/share/kibana/src/core_plugins/elasticsearch/lib/create_kibana_index.js:

return callWithInternalUser('indices.create', {
    index: index,
    body: {
      settings: {
        number_of_shards: 1,
        'index.mapper.dynamic': false
      },
      mappings
    }
  }).catch(handleError('Unable to create Kibana index "<%= kibana.index %>"')).then(function () {
    return callWithInternalUser('cluster.health', {
      waitForStatus: 'yellow',
      index: index
    }).catch(handleError('Waiting for Kibana index "<%= kibana.index %>" to come online failed.'));
  });

anpingli · 2018-08-29T02:14:41Z

@ewolinetz What will happen if we delete the .kibana index manually. Is the new .kibana index replicas=0?

lukas-vlcek · 2018-08-29T09:39:53Z

@ewolinetz the .kibana index is created with shards [1]/[1] (1 primary, 1 replica shards) way before we are putting our index templates into the cluster. I do not know where the number of replicas is configured in this case, however, if we just want to change the number of replicas for the testing case then we can simply change it on the fly using the index settings update API.

ewolinetz · 2018-08-29T14:09:27Z

@lukas-vlcek unfortunately when i've tried that before the request is redirected to a kibana user index instead of changing the .kibana index.

@anpingli I'll try that and see. Hopefully it doesn't redirect it to a kibana user index as well...

lukas-vlcek · 2018-08-29T14:50:34Z

@ewolinetz even when using admin certs?

jcantrill · 2018-08-29T14:55:21Z

The multitenant code will attempt to modify anything that has Kibana in the request and a bearer token is provided. This is the source of a known issue where we are unable to delete index patterns. Maybe we should consider allowing admin users who provide a special header to take a request as is with no modifications. I would expect the admin certs to already behave this way but maybe there is something we are modifying

lukas-vlcek · 2018-08-29T15:07:47Z

FYI, the number_of_replicas: 1 value comes from the defaults. See https://github.com/elastic/elasticsearch/blob/master/docs/reference/indices/create-index.asciidoc#index-settings. Seems other people were running into YELLOW cluster issue in case of a single node cluster too.

May be we can accept the YELLOW cluster state when running particular test?

ewolinetz · 2018-08-29T16:05:54Z

I can delete the indices, and then I see them recreated correctly with 0 replicas.

[2018-08-29T15:48:02,816][INFO ][o.e.c.m.MetaDataDeleteIndexService] [logging-es-data-master-aw89bch5] [.kibana/XDC_dKYWSTyWCDJ0_9wISw] deleting index
[2018-08-29T15:48:09,983][WARN ][c.f.s.c.PrivilegesEvaluator] .kibana does not exist in cluster metadata
[2018-08-29T15:48:09,990][INFO ][o.e.c.m.MetaDataCreateIndexService] [logging-es-data-master-aw89bch5] [.kibana] creating index, cause [api], templates [common.settings.kibana.template.json], shards [1]/[0], mappings [_default_, index-pattern, server, visualization, search, timelion-sheet, config, dashboard, url]
[2018-08-29T15:48:10,015][INFO ][o.e.c.r.a.AllocationService] [logging-es-data-master-aw89bch5] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana][0]] ...]).

ewolinetz · 2018-08-29T16:06:55Z

I think we are running into an issue where we are not ready (the index templates are still being seeded) and then ES is somehow allowing traffic through.

ewolinetz · 2018-08-29T23:42:07Z

Confirmed that #1326 fixes the issue of the index being created before the index template is seeded

ewolinetz · 2018-08-30T19:07:27Z

/hold

ewolinetz · 2018-08-30T19:08:28Z

/hold cancel

jcantrill

/lgtm

ewolinetz requested a review from lukas-vlcek August 24, 2018 18:20

openshift-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Aug 24, 2018

openshift-ci-robot requested review from jcantrill and richm August 24, 2018 18:20

lukas-vlcek reviewed Aug 27, 2018

View reviewed changes

ewolinetz force-pushed the kibana_index_template branch 2 times, most recently from 43d84bb to 01f9327 Compare August 27, 2018 16:18

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 28, 2018

ewolinetz force-pushed the kibana_index_template branch from 4cce24f to d1068b1 Compare August 29, 2018 16:05

ewolinetz mentioned this pull request Aug 29, 2018

Improve readiness probe #1326

Merged

jcantrill added the release/3.11 label Aug 30, 2018

ewolinetz force-pushed the kibana_index_template branch from d1068b1 to 7e3b004 Compare August 30, 2018 19:07

Creating index template for Kibana to configure index replicas

724762e

ewolinetz force-pushed the kibana_index_template branch from 7e3b004 to 724762e Compare August 30, 2018 19:08

openshift-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Aug 30, 2018

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 30, 2018

jcantrill approved these changes Aug 30, 2018

View reviewed changes

openshift-ci-robot assigned jcantrill Aug 30, 2018

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Aug 30, 2018

openshift-merge-robot merged commit fb43723 into openshift:master Aug 30, 2018

ewolinetz deleted the kibana_index_template branch July 10, 2019 17:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Creating index template for Kibana to configure index replicas #1323

Creating index template for Kibana to configure index replicas #1323

ewolinetz commented Aug 24, 2018

richm commented Aug 24, 2018

ewolinetz commented Aug 24, 2018

lukas-vlcek Aug 27, 2018

lukas-vlcek commented Aug 27, 2018

ewolinetz commented Aug 27, 2018

ewolinetz commented Aug 28, 2018

ewolinetz commented Aug 28, 2018

lukas-vlcek commented Aug 28, 2018 •

edited

Loading

ewolinetz commented Aug 28, 2018

anpingli commented Aug 29, 2018

lukas-vlcek commented Aug 29, 2018 •

edited

Loading

ewolinetz commented Aug 29, 2018

lukas-vlcek commented Aug 29, 2018

jcantrill commented Aug 29, 2018

lukas-vlcek commented Aug 29, 2018 •

edited

Loading

ewolinetz commented Aug 29, 2018

ewolinetz commented Aug 29, 2018

ewolinetz commented Aug 29, 2018

ewolinetz commented Aug 30, 2018

ewolinetz commented Aug 30, 2018

jcantrill left a comment

Creating index template for Kibana to configure index replicas #1323

Creating index template for Kibana to configure index replicas #1323

Conversation

ewolinetz commented Aug 24, 2018

richm commented Aug 24, 2018

ewolinetz commented Aug 24, 2018

lukas-vlcek Aug 27, 2018

Choose a reason for hiding this comment

lukas-vlcek commented Aug 27, 2018

ewolinetz commented Aug 27, 2018

ewolinetz commented Aug 28, 2018

ewolinetz commented Aug 28, 2018

lukas-vlcek commented Aug 28, 2018 • edited Loading

ewolinetz commented Aug 28, 2018

anpingli commented Aug 29, 2018

lukas-vlcek commented Aug 29, 2018 • edited Loading

ewolinetz commented Aug 29, 2018

lukas-vlcek commented Aug 29, 2018

jcantrill commented Aug 29, 2018

lukas-vlcek commented Aug 29, 2018 • edited Loading

ewolinetz commented Aug 29, 2018

ewolinetz commented Aug 29, 2018

ewolinetz commented Aug 29, 2018

ewolinetz commented Aug 30, 2018

ewolinetz commented Aug 30, 2018

jcantrill left a comment

Choose a reason for hiding this comment

lukas-vlcek commented Aug 28, 2018 •

edited

Loading

lukas-vlcek commented Aug 29, 2018 •

edited

Loading

lukas-vlcek commented Aug 29, 2018 •

edited

Loading