Asset not stored in ES

[godber]

I've confirmed that using the latest master (aa06747) of teraslice and the version of elasticsearch_api that it installs by default (when running npm install) that assets do not get loaded.

Steps to reproduce:

• Remove all old tera indices

curl -Ss -XDELETE localhost:9200/tera*

• Start teraslice

⌂355% 1d [godber:~/Workspace/terascope/opensource/teraslice] master+* 40s ± node ./service.js -c examples/config/processor-master.yaml  | bunyan
[2017-10-10T20:41:01.111Z]  INFO: teraslice/53684 on stretch.local: Service starting
[2017-10-10T20:41:01.117Z]  INFO: teraslice/53684 on stretch.local: node stretch.local is attempting to connect to cluster_master: http://127.0.0.1:5678 (module=node_master)
[2017-10-10T20:41:01.230Z]  INFO: teraslice/53684 on stretch.local: Starting 1 cluster_master
[2017-10-10T20:41:01.239Z]  INFO: teraslice/53684 on stretch.local: Starting 1 assets_service
[2017-10-10T20:41:02.137Z]  INFO: teraslice/53687 on stretch.local: Creating connection for elasticsearch
[2017-10-10T20:41:02.142Z]  INFO: teraslice/53686 on stretch.local: listening on port 5678 (module=cluster_master)
[2017-10-10T20:41:02.278Z]  INFO: teraslice/53686 on stretch.local: Initializing (module=cluster_service)
[2017-10-10T20:41:02.298Z]  INFO: teraslice/53686 on stretch.local: Creating connection for elasticsearch
[2017-10-10T20:41:02.401Z]  INFO: teraslice/53687 on stretch.local: Using elasticsearch hosts: 127.0.0.1:9200
[2017-10-10T20:41:02.554Z]  INFO: teraslice/53686 on stretch.local: Using elasticsearch hosts: 127.0.0.1:9200
[2017-10-10T20:41:02.584Z]  INFO: teraslice/53687 on stretch.local: assets_service is listening on port 45679 (module=assets_service)
[2017-10-10T20:41:02.715Z] ERROR: teraslice/53686 on stretch.local: [index_already_exists_exception] index [teracluster__state-2017.10/Znc04AhIQC-gzFiFk1Sh7A] already exists, with { index_uuid="Znc04AhIQC-gzFiFk1Sh7A" & index="teracluster__state-2017.10" } (module=elasticsearch_backend)
[2017-10-10T20:41:02.782Z]  INFO: teraslice/53686 on stretch.local: initializing (module=state_storage)
[2017-10-10T20:41:02.886Z]  INFO: teraslice/53686 on stretch.local: Initializing (module=job_storage)
[2017-10-10T20:41:02.900Z]  INFO: teraslice/53686 on stretch.local: initializing (module=state_storage)
[2017-10-10T20:41:02.901Z]  INFO: teraslice/53686 on stretch.local: Initializing (module=ex_storage)
[2017-10-10T20:41:02.902Z]  INFO: teraslice/53686 on stretch.local: Initializing (module=jobs_service)
[2017-10-10T20:41:02.923Z]  INFO: teraslice/53686 on stretch.local: Jobs queue initialization complete (module=jobs_service)
[2017-10-10T20:41:03.798Z]  INFO: teraslice/53684 on stretch.local: node has successfully connected to: http://127.0.0.1:5678 (module=node_master)
[2017-10-10T20:41:03.807Z]  INFO: teraslice/53686 on stretch.local: node stretch.local has connected (module=cluster_service)

• Post an asset to the asset endpoint:

curl -XPOST -H "Content-Type: application/octet-stream" localhost:5678/assets --data-binary @dropdoc-1.0.0.zip
{"_id":"a9e68d8e86060b7ebcdc5c1683aa47be84c59e06"}

• The following message is printed to the teraslice master:

[2017-10-10T20:41:16.176Z]  INFO: teraslice/53687 on stretch.local: loading an asset (module=assets_service)

• Note that documents is 0 in the assets index.

curl -Ss localhost:9200/_cat/indices/tera*?v
health status index                      uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   teracluster__state-2017.10 Znc04AhIQC-gzFiFk1Sh7A   5   1          0            0       810b           810b
yellow open   teracluster__jobs          yXgS_D6oSOSmogXyJSvJmQ   5   1          0            0       810b           810b
yellow open   teracluster__ex            Olz7ENUhScGDDJQTB7gbeQ   5   1          0            0       810b           810b
yellow open   teracluster__assets        El57gd0PS6Su9o3WBwERGA   5   1          0            0       810b           810b

Not sure if either of these have an impact on this
#583 and #584

[godber]

The asset also doesn't show up here:

curl -sS localhost:5678/txt/assets
name  version  id  _created  description
----  -------  --  --------  -----------

[godber]

We tracked this down and the specific scenario is when the asset exists in the masters asset directory but not in ES.