Support Elasticsearch 6.x

[genthalili]

Any update to support ES 6.x?

[gnanakeethan]

7.x is also out now. Any updates?

[karussell]

We welcome any contribution. Would you have interests in creating a pull request 😄 ?

[gnanakeethan]

@karussell I am new to this space, as well as Java. I will see if I can make the patches up to a good level.

[simonpoole]

I suppose that if somebody is going to invest the time, it might be a good idea to investigate if their are any advantages to switching to https://opendistro.github.io/for-elasticsearch/ (I suspect that mainly the additional monitoring etc could be interesting).

[simonpoole]

Pro memoria: switching to 7.x and later will require migrating to the REST client instead of the deprecated transport client. Probably not a big deal as such, but still implies some work.

[missinglink]

FYI I just upgraded Pelias to ES6 (also like 99% support for ES7 too) I documented all the steps in this ticket:
pelias/pelias#719

[tuempeltaucher]

I changed the code to support ES 7.6.2, no need to switch to the REST client.

I won't create a pull request because unit tests does not work and the embedded elasticsearch crashes.

tuempeltaucher@cd0d967

[otbutz]

Updating ES would also require that we settle for a solution of #285

[adonig]

Compatibility with ES 5.6.0 should be sufficient to make it run with ES 6 and 7. Sadly photon insists on ES 5.5.0 and I don't even know how to install this version of ES on OS X. Probably using Docker during development is the only way on OS X?

[lonvia]

After some version research, the way forward on this seems to be the following:

• update to ES 5.6.0
• come up with a solution for embedding
• switch to HTTP transport for client/server communication
• update to ES 7

[lonvia]

I think I've come up with a solution for embedding. We can simple offer an alternative Lucene backend for that. I've implemented enough of a proof-of-concept implementation to be convinced it is possible. The ES backend would remain but with standalone mode only.

Now there is another unexpected complication with switching to the HTTP client. By loosing embedded mode, we also loose the framework for our integration tests. I've looked around what is recommended for integration tests these days and it's all a bit depressing. Here are the options I found:

• Spinning up a docker instance. (Starting a full OS for every test? Not an option.)
• https://github.com/alexcojocaru/elasticsearch-maven-plugin did look a bit more promising but as far as I understand it simply downloads and installs the full 250MB zip file from Elastic. Not lightweight either.
• It might be possible to abuse ES' internal integration tester. My personal favourite but does not look straightforward.
• Require the developer to provide an ES instance themselves and use that.

Any other options?

[otbutz]

Do we really want to support two different backends? That's additional backend specific quirks and tickets. IMHO it's easier to streamline the setup process.

Require the developer to provide an ES instance themselves and use that.

I don't really see a problem with that.

[nyilmaz]

For the testing side, https://www.testcontainers.org/modules/elasticsearch/ can be used, as spring-data-elasticsearch uses

[lonvia]

Yet another suggestion for tests/embedded mode: https://github.com/codelibs/opensearch-runner

[lonvia]

#799 adds support for OpenSearch 2.10. opensearch-runner neatly solved the embedding and testing problem.

The current OpenSearch Photon version is still experimental but the experience with porting was good enough, so that's probably where development is going to go in the end. I've also experimented with solr but with the embedding as show-stopper out of the way the OpenSearch route is simply be faster.

Closing here. There will be no support for ElasticSearch. Please use the OpenSearch port.