Upgrade to Solr 7.3.0

[djbrooke]

Solr 7.3.0 was released recently. Since we've not yet cut the release (4.9) that includes our last Solr upgrade (#4158), let's upgrade to the newest version.

[matthew-a-dunlap]

Update was confirmed working normally as well as in vagrant and docker-aio.

[kcondon]

Solr start message warns about dramatically increasing file descriptors and processes. This did not happen in 7.2.1. This may require a tweak to solr init script:

solr-7.3.0]$ bin/solr start
*** [WARN] *** Your open file limit is currently 1024.
It should be set to 65000 to avoid operational disruption.
If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
*** [WARN] *** Your Max Processes Limit is currently 1024.
It should be set to 65000 to avoid operational disruption.
If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
Warning: Available entropy is low. As a result, use of the UUIDField, SSL, or any other features that require
RNG might not work properly. To check for the amount of available entropy, use 'cat /proc/sys/kernel/random/entropy_avail'.

Waiting up to 180 seconds to see Solr running on port 8983 []
Started Solr server on port 8983 (pid=3731). Happy searching!

I edit /etc/init.d/solr and added:

# increase file descriptors per solr 7.3.0 doc
  ulimit -n 65000

I ended up editing /etc/security/limits.conf
solr soft nproc 63000

Because my system seems to have a max process limit of 63705, I stayed at 63000.

At first I tried putting both settings in either init.d/solr or limits.conf but they only worked when I split them as above. Not sure why but likely something to do with running init as root but sudoing as solr to start job.

[pdurbin]

@kcondon as we discussed at standup I just merged the latest from develop (including the merged Solr 7.3.0 upgrade) into my metrics branch and deployed it (4527-metrics-api-900973b) to the server we talked about.

[kcondon]

index all took 14 hours for prod copy:
4472 dataverses and 81744 datasets indexed. index all took 50754458 milliseconds. Solr index was not cleared before indexing.

Solr index was cleared before indexing, despite message.