Versioning Screen consistently freezing

[abelpink]

If I create a new version of a record and then navigate to the Versioning Tab in the edit record section, my browser consistently freezes and becomes unresponsive. I get a browser pop-up asking whether I want to exit or wait.

Additionally, the thumbnail image for the versioned record won't appear even though I have ingested a thumbnail bitstream.

[aseyedia]

@abelpink Thanks for letting me know, can you give me a link as an example? An Item with versions in it?

[abelpink]

This is the main perpetrator: https://pedsdspace01.research.chop.edu/entities/documentation/c00edc4a-9997-4287-a4e6-86a07b614f7b

[aseyedia]

Bizarre. The CPU usage just for this page shoots up to 100% and the memory usage just keeps climbing until it is killed.

[aseyedia]

@abelpink Are there other pages where this happens

[aseyedia]

 versionitem_id | version_number |      version_date       |                version_summary          >
----------------+----------------+-------------------------+----------------------------------------->
              1 |              1 | 2024-06-12 12:49:33     |                                         >
              2 |              2 |                         |                                         >
              3 |              3 | 2024-07-29 15:47:01.509 | Version 5.3 of the PEDSnet CDM ETL Conve>
              4 |              4 | 2024-07-29 15:59:31.433 | Version 5.4 of the PEDSnet CDM ETL Conve
              

Version 2 is completely missing for some reason. Strange.

[aseyedia]

It's failing a healthcheck:

[dspace@pedsdspace01 dspace]$ ./bin/dspace healthcheck

#### General Information [took: 0s] [# lines: 5]
Generated: Wed Jul 31 16:39:30 EDT 2024
From - Till: 2024-07-24 - 2024-07-31
Url: https://pedsdspace01.research.chop.edu

Assetstore size     : 730 MB
Log dir size        : 799 MB

###############################


#### Checksum [took: 9s] [# lines: 1]
No md5 checks made!

###############################


#### Item summary [took: 0s] [# lines: 51]
Exception occurred when processing report - org.hibernate.exception.SQLGrammarException: JDBC exception executing SQL [select c1_1.uuid,c1_2.uuid,c1_1.admin,c1_1.collection_id,c1_1.logo_bitstream_id,c1_1.submitter,c1_1.template_item_id,sum(b2_1.size_bytes) from public.item i1_0 join public.collection2item c1_0 on i1_0.uuid=c1_0.item_id join (public.collection c1_1 join public.dspaceobject c1_2 on c1_1.uuid=c1_2.uuid) on c1_1.uuid=c1_0.collection_id join public.item2bundle b1_0 on i1_0.uuid=b1_0.item_id join public.bundle b1_1 on b1_1.uuid=b1_0.bundle_id join public.bundle2bitstream b2_0 on b1_1.uuid=b2_0.bundle_id join public.bitstream b2_1 on b2_1.uuid=b2_0.bitstream_id group by c1_1.uuid,case when c1_1.uuid is not null then 3 end,c1_1.admin,c1_1.collection_id,c1_1.logo_bitstream_id,c1_1.submitter,c1_1.template_item_id] [ERROR: column "c1_2.uuid" must appear in the GROUP BY clause or be used in an aggregate function
  Position: 18] [n/a]
	at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:91)
	at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:58)
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:108)
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:94)
	at org.hibernate.sql.results.jdbc.internal.DeferredResultSetAccess.executeQuery(DeferredResultSetAccess.java:264)
	at org.hibernate.sql.results.jdbc.internal.DeferredResultSetAccess.getResultSet(DeferredResultSetAccess.java:167)
	at org.hibernate.sql.results.jdbc.internal.JdbcValuesResultSetImpl.advanceNext(JdbcValuesResultSetImpl.java:218)
	at org.hibernate.sql.results.jdbc.internal.JdbcValuesResultSetImpl.processNext(JdbcValuesResultSetImpl.java:98)
	at org.hibernate.sql.results.jdbc.internal.AbstractJdbcValues.next(AbstractJdbcValues.java:19)
	at org.hibernate.sql.results.internal.RowProcessingStateStandardImpl.next(RowProcessingStateStandardImpl.java:66)
	at org.hibernate.sql.results.spi.ListResultsConsumer.consume(ListResultsConsumer.java:202)
	at org.hibernate.sql.results.spi.ListResultsConsumer.consume(ListResultsConsumer.java:33)
	at org.hibernate.sql.exec.internal.JdbcSelectExecutorStandardImpl.doExecuteQuery(JdbcSelectExecutorStandardImpl.java:209)
	at org.hibernate.sql.exec.internal.JdbcSelectExecutorStandardImpl.executeQuery(JdbcSelectExecutorStandardImpl.java:83)
	at org.hibernate.sql.exec.spi.JdbcSelectExecutor.list(JdbcSelectExecutor.java:76)
	at org.hibernate.sql.exec.spi.JdbcSelectExecutor.list(JdbcSelectExecutor.java:65)
	at org.hibernate.query.sqm.internal.ConcreteSqmSelectQueryPlan.lambda$new$2(ConcreteSqmSelectQueryPlan.java:137)
	at org.hibernate.query.sqm.internal.ConcreteSqmSelectQueryPlan.withCacheableSqmInterpretation(ConcreteSqmSelectQueryPlan.java:381)
	at org.hibernate.query.sqm.internal.ConcreteSqmSelectQueryPlan.performList(ConcreteSqmSelectQueryPlan.java:303)
	at org.hibernate.query.sqm.internal.QuerySqmImpl.doList(QuerySqmImpl.java:509)
	at org.hibernate.query.spi.AbstractSelectionQuery.list(AbstractSelectionQuery.java:427)
	at org.hibernate.query.Query.getResultList(Query.java:120)
	at org.dspace.content.dao.impl.CollectionDAOImpl.getCollectionsWithBitstreamSizesTotal(CollectionDAOImpl.java:179)
	at org.dspace.content.CollectionServiceImpl.getCollectionsWithBitstreamSizesTotal(CollectionServiceImpl.java:925)
	at org.dspace.health.ItemCheck.getCollectionSizesInfo(ItemCheck.java:144)
	at org.dspace.health.ItemCheck.run(ItemCheck.java:75)
	at org.dspace.health.Check.report(Check.java:31)
	at org.dspace.health.Report.run(Report.java:69)
	at org.dspace.health.Report.main(Report.java:196)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:283)
	at org.dspace.app.launcher.ScriptLauncher.handleScript(ScriptLauncher.java:134)
	at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:99)
Caused by: org.postgresql.util.PSQLException: ERROR: column "c1_2.uuid" must appear in the GROUP BY clause or be used in an aggregate function
  Position: 18
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2725)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2412)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:371)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:502)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:419)
	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:194)
	at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:137)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:123)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:123)
	at org.hibernate.sql.results.jdbc.internal.DeferredResultSetAccess.executeQuery(DeferredResultSetAccess.java:246)
	... 31 more

###############################


#### User summary [took: 0s] [# lines: 19]
Exception occurred when processing report - org.hibernate.HibernateException: Calling method 'createQuery' is not valid without an active transaction (Current status: MARKED_ROLLBACK)
	at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:339)
	at jdk.proxy2/jdk.proxy2.$Proxy159.createQuery(Unknown Source)
	at org.dspace.core.AbstractHibernateDAO.createQuery(AbstractHibernateDAO.java:147)
	at org.dspace.content.dao.impl.MetadataFieldDAOImpl.findByElement(MetadataFieldDAOImpl.java:130)
	at org.dspace.content.MetadataFieldServiceImpl.findByElement(MetadataFieldServiceImpl.java:107)
	at org.dspace.eperson.EPersonServiceImpl.findAll(EPersonServiceImpl.java:316)
	at org.dspace.eperson.EPersonServiceImpl.findAll(EPersonServiceImpl.java:292)
	at org.dspace.health.UserCheck.run(UserCheck.java:42)
	at org.dspace.health.Check.report(Check.java:31)
	at org.dspace.health.Report.run(Report.java:69)
	at org.dspace.health.Report.main(Report.java:196)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:283)
	at org.dspace.app.launcher.ScriptLauncher.handleScript(ScriptLauncher.java:134)
	at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:99)

###############################


#### Log Analyser Check [took: 3s] [# lines: 3]
====
Exception occurred!
org.dspace.discovery.SearchServiceException: Calling method 'createQuery' is not valid without an active transaction (Current status: MARKED_ROLLBACK)

Something happened to this DSpace instance 😢

[aseyedia]

Posted in the DSpace Slack, you can view here.

[aseyedia]

I restored to the 7/24 version and I'm still getting the crashing error, potentially due to the fact that this version also has an empty second row:

 versionitem_id | version_number |    version_date     | version_summary | versionhistory_id |              eperson_id              |               item_i>
----------------+----------------+---------------------+-----------------+-------------------+--------------------------------------+--------------------->
              1 |              1 | 2024-06-12 12:49:33 |                 |                 1 | c428d716-f7e4-4cc3-9c5b-bb529b256e36 | 1d9ad00b-f0ed-44a9-9>
              2 |              2 |                     |                 |                 1 |                                      |
(2 rows)

if just deleting this row works, then I'll see if deleting the up-to-date version of the database works.

[aseyedia]

Deleting the second row like that didn't work. I do have one earlier backup but that's from 7/15. It would be pretty rough to lose half a month of content but I just want to see if there is any issue. What if we just deleted this item from the up-to-date version of the database and then tried to reproduce the issue? That would at least let us know that if the problem is deeper than Item-specific.

[aseyedia]

#16 (comment)

I was able to reproduce this issue in the PCORnet instance of DSpace, which, to our knowledge, does not have this "Version History page crashing" issue, so it stands to reason that this error message is completely separate and is likely a bug within the DSpace backend code (which I could fix for them on my spare them but right now I'm more focused on getting our stuff to work). which also has the crashing "Version History" bug, even though it never had Entities set up, which means "Version History" is entirely separate from Entities.

Here's what we have learned so far:

1. ./bin/dspace healthcheck crashes on both the PEDSpace and the QFSpace instances, which suggests that this is potentially a DSpace issue.
2. The "Version HIstory" crashing issue is present in the QFSpace instance as well, which, as I mentioned earlier, does not have Entities setup, again suggesting that this might be a DSpace issue.

[aseyedia]

I posted about the issue DSpace/dspace-angular#3252 and they fixed it in a day DSpace/dspace-angular#3253. I just manually copied and pasted their solution and it works.