AssertionError at OCachePointer.incrementReadersReferrer

[jayaramcs]

OrientDB Version: 2.2.33

Java Version: 1.8.162

OS: windows 10

Expected behavior

No assertion error

Actual behavior

The following exception is thrown.

SEVERE [null]: Exception `0059A5E4` in storage `plocal:C:/Basis/Projects/CyberTriage/CyberTriageApp/build/testuserdir/databases/2.0/CyberTriage`: 2.2.33 (build 77584cd6827f647cf4aa231cf27bd6f10bc04e2c, branch 2.2.x)
java.lang.AssertionError
	at com.orientechnologies.orient.core.storage.cache.OCachePointer.incrementReadersReferrer(OCachePointer.java:92)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.load(OWOWCache.java:790)
	at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.updateCache(O2QCache.java:1155)
	at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.doLoad(O2QCache.java:361)
	at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.load(O2QCache.java:294)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.loadPage(ODurableComponent.java:148)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecordBuffer(OPaginatedCluster.java:713)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:689)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:668)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doReadRecord(OAbstractPaginatedStorage.java:4203)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:3819)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:1409)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx$SimpleRecordReader.readRecord(ODatabaseDocumentTx.java:3412)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:2023)
	at com.orientechnologies.orient.core.tx.OTransactionNoTx.loadRecord(OTransactionNoTx.java:98)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:1772)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:103)
	at com.orientechnologies.orient.core.iterator.OIdentifiableIterator.readCurrentRecord(OIdentifiableIterator.java:300)
	at com.orientechnologies.orient.core.iterator.ORecordIteratorClusters.hasNext(ORecordIteratorClusters.java:167)
	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1636)
	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1585)
	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:522)
	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:485)
	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:70)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3417)
[catch] at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3335)
	at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:78)
	at com.orientechnologies.orient.core.sql.query.OSQLAsynchQuery.run(OSQLAsynchQuery.java:74)
	at com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.run(OSQLSynchQuery.java:102)
	at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:33)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.query(ODatabaseDocumentTx.java:760)

Steps to reproduce

Our system inserts a large number of records and when a query is issued simultaneously, it results in the above error.
We upgraded from 2.2.19 to 2.2.33. We have not observed this issue in the past versions.

[andrii0lomakin]

@jayaramcs Probably this exception is not the root cause. Could you send me the full log from your server since server start?

[jayaramcs]

@Laa I reran the scenario and there is not much else in the logs.

I then increased the disk cache size OGlobalConfiguration.DISK_CACHE_SIZE.setValue(16384) and the issue went away. But the problem is that our application cannot be shipped with such a high value for cache.

I am attaching the full relevant logs ..

orientlogs.log

[andrii0lomakin]

@jayaramcs Could assign 0 to the setting "query.scanPrefetchPages" and repeat your load test?

[jayaramcs]

@Laa That did not help. we run into new errors. (full log attached)

java.lang.IllegalArgumentException: Amount of pages to load from cache should be not less than 1 but passed value is 0
	at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.doLoad(O2QCache.java:322)

orientlogs.log

[andrii0lomakin]

@jayaramcs I see, I suppose that is documentation problem, could you use 1 then, it should have the same effect and you will not see this error.

[jayaramcs]

@Laa This seems to have fixed the issue per my preliminary tests, Thank you!
I'd think this would reduce my query performance?
Is this the permanent fix for this issue or do you plan on something more permanent?

[andrii0lomakin]

@jayaramcs probably we will have a permanent one. I need to think about the strategy of development a bit. But in such case, I would need your help.

[jayaramcs]

@Laa Yeah no problem. I will tag my current branch and save it.

[andrii0lomakin]

Fixed. Will be released in next version.