Indexing: enable Dataverse to use Solr in a distributed environment

[kcondon]

Dataverse is currently coded to connect to solr on localhost. In order to run in a distributed environment, the app needs to be able to connect to a remote solr instance. Ideally this would be done with solr cloud and a zookeeper ensemble. A list of zookeepers would be stored somewhere which the app would use to initialize a CloudSolrServer object.

[pdurbin]

In 57dc52c I enabled a new SolrHostColonPort setting and documented it at https://github.com/IQSS/dataverse/blob/master/doc/Sphinx/source/Installers/dataverse-installer-main.rst#solrhostcolonport

Here's an example of how to use it:

curl -X PUT http://localhost:8080/api/s/settings/:SolrHostColonPort/localhost:8983

@kcondon is this good enough for Dataverse 4.0 or is the use of ZooKeeper a hard requirement? Can we defer ZooKeeper to a future release? I'll put this in QA so you can at least test this new setting.

[kcondon]

OK, this config param works. Will need to test on multi server env. As for whether it is ok for prod, maybe a discussion for others as well?

[kcondon]

Tested on multiple systems and they are both able to update index and see each other's updates. Do not know yet how it performs under load.

Benson still hopes we can use Zookeeper and thinks the config isn't that much more involved.

Closing as basic functionality has been delivered.

[pdurbin]

Benson still hopes we can use Zookeeper and thinks the config isn't that much more involved.

Right, he showed me we'd start Solr like this:

java -DzkRun -Dboostrap_confdir=solr/collection1/conf -Dbootstrap_conf=true -jar start.jar

At least that's how developers would start Solr with a single collection/core/thing. In production ZooKeeper would manage multiple.