New datasets not visible in gui despite being seemingly created and indexed

[jamessi1989]

We have upgraded from 6.2 to 6.5 and somewhere along the way lost the ability to see new datasets in the GUI.

What steps does it take to reproduce the issue?
deploy 6.2 via ansible and upgrade ? I think this will be hard to reproduce easily.

• When does this issue occur?
  After creating a dataset without errors it does not appear in the gui. Manual indexed does not make the dataset visible.

• Which page(s) does it occurs on?
  In the root dataverse view datasets are not visible unless solr is restarted. ( datasets can be accessed directly via url)

• What happens?
  We have tried reinstalling solr. Different versions. Same issue. We see it in two enviroments.

• To whom does it occur (all users, curators, superusers)?
  All users

• What did you expect to happen?
  dataset appears in view

Which version of Dataverse are you using?
6.5

Any related open or closed issues to this bug report?
More information and logs in https://dataverse.zulipchat.com/#narrow/channel/378866-troubleshooting/topic/Custom.20.E2.80.9CPerma.E2.80.9D.20PID.20Setup.20.26.20Indexing.20Problem

Screenshots:

No matter the issue, screenshots are always welcome.

The two top datasets for example did not appear after creating in this view only after restarting solr. They

Are you thinking about creating a pull request for this issue?
I dont think we have the capacity but we would like to find a workaround or solution or know if anyone else receives the same problem.

[pdurbin]

I confirmed with @jamessi1989 that he's not simply seeing the expected 5-10 second delay we introduced in this PR:

• Solr: Try Soft Commit on Indexing #10547

[qqmyers]

FWIW: If indexing fails, there should be entries in the logs/process-failures directory that might indicate if there are schema issues (missing fields, etc.)

[jamessi1989]

There are no indexing failures during a datset creation. It is simply not visible by dataverse in the gui unless I restart solr. I have attached logs of the what is written to the solr log after I create the dataset and then the restart of solr.

solrRestartLog.log
solrAfterCreate.log

[jamessi1989]

I've been troubleshooting the issue. The key changes I made were setting autoCommit with openSearcher set to true and adding a 2-second autoSoftCommit . These adjustments resolved the issue for me.

It seems that the old Solr configuration doesn’t work out of the box (at least the configuration consistent with Ansible deployments) with Dataverse's current setup, so this may need to be noted in upgrade advisories. I’ll continue monitoring.

[pdurbin]

The fix was to make sure to upgrade Solr and use the config (with autocommit changes) mentioned in the Dataverse 6.3 release notes. See also https://dataverse.zulipchat.com/#narrow/channel/378866-troubleshooting/topic/.E2.9C.94.20indexing.20problem/near/493958217