com.orientechnologies.orient.core.exception.OStorageException: Error during transaction commit

[KishoDai]

OrientDB Version: Enterprise 2.2.26

Java Version: 1.8

OS: CentOS

Expected behavior

Actual behavior

2017-09-19 12:34:16.208 [pool-3-thread-7][c.m.o.s.r.sql.OrientSql]ERROR create edge MemberHasIp from #67:107166 to #158:432 @ Error during transaction commit
DB name="sns"
DB name="sns"
2017-09-19 12:34:16.211 [pool-3-thread-7][c.m.o.s.r.sql.OrientSql]ERROR
com.orientechnologies.orient.core.exception.OStorageException: Error during transaction commit
DB name="sns"
DB name="sns"
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.throwSerializedException(OChannelBinaryAsynchClient.java:442)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:393)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:275)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:167)
at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:2225)
at com.orientechnologies.orient.client.remote.OStorageRemote$27.execute(OStorageRemote.java:1200)
at com.orientechnologies.orient.client.remote.OStorageRemote$2.execute(OStorageRemote.java:198)
at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:243)
at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperationRetry(OStorageRemote.java:195)
at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperation(OStorageRemote.java:206)
at com.orientechnologies.orient.client.remote.OStorageRemote.command(OStorageRemote.java:1176)
at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
at com.orientechnologies.orient.core.command.OCommandRequest$execute$1.call(Unknown Source)
at cn.memedai.orientdb.sns.realtime.sql.OrientSql.execute(OrientSql.groovy:38)
at sun.reflect.GeneratedMethodAccessor70.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:169)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:166)
at cn.memedai.orientdb.sns.realtime.sql.OrientSql.createEdge(OrientSql.groovy:62)
at cn.memedai.orientdb.sns.realtime.sql.OrientSql$createEdge.call(Unknown Source)
at cn.memedai.orientdb.sns.realtime.service.impl.toorientdb.CreditTradeAuditCtaOrderDeviceinfoToOrientDBServiceImpl.process(CreditTradeAuditCtaOrderDeviceinfoToOrientDBServiceImpl.groovy:135)
at cn.memedai.orientdb.sns.realtime.service.RealTimeService$process$3.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at cn.memedai.orientdb.sns.realtime.service.impl.tomysql.CreditTradeAuditCtaOrderDeviceinfoToMysqlServiceImpl$process.call(Unknown Source)
at cn.memedai.orientdb.sns.realtime.RealTimeDispatch$_pollAndProcess_closure3$_closure7.doCall(RealTimeDispatch.groovy:146)
at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at groovy.lang.Closure.call(Closure.java:414)
at groovy.lang.Closure.call(Closure.java:430)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2040)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2025)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2066)
at org.codehaus.groovy.runtime.dgm$162.invoke(Unknown Source)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at cn.memedai.orientdb.sns.realtime.RealTimeDispatch$_pollAndProcess_closure3.doCall(RealTimeDispatch.groovy:143)
at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at groovy.lang.Closure.call(Closure.java:414)
at groovy.lang.Closure.call(Closure.java:430)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2040)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2025)
at org.codehaus.groovy.runtime.dgm$158.invoke(Unknown Source)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at cn.memedai.orientdb.sns.realtime.RealTimeDispatch.pollAndProcess(RealTimeDispatch.groovy:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:174)
at cn.memedai.orientdb.sns.realtime.RealTimeDispatch.startThread(RealTimeDispatch.groovy:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:384)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:69)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:166)
at cn.memedai.orientdb.sns.realtime.RealTimeDispatch$_start_closure1$_closure4$_closure5.doCall(RealTimeDispatch.groovy:70)
at cn.memedai.orientdb.sns.realtime.RealTimeDispatch$_start_closure1$_closure4$_closure5.doCall(RealTimeDispatch.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at groovy.lang.Closure.call(Closure.java:414)
at groovy.lang.Closure.call(Closure.java:408)
at groovy.lang.Closure.run(Closure.java:495)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: null
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerAbstract$SBTreeBonsaiContainer.access$010(OSBTreeCollectionManagerAbstract.java:267)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerAbstract.releaseSBTree(OSBTreeCollectionManagerAbstract.java:201)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag.releaseTree(OSBTreeRidBag.java:983)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag.updateSize(OSBTreeRidBag.java:1031)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag.size(OSBTreeRidBag.java:748)
at com.orientechnologies.orient.core.db.record.ridbag.ORidBag.toStream(ORidBag.java:222)
at com.orientechnologies.orient.core.serialization.serializer.record.binary.ORecordSerializerBinaryV0.serializeValue(ORecordSerializerBinaryV0.java:763)
at com.orientechnologies.orient.core.serialization.serializer.record.binary.ORecordSerializerBinaryV0.serialize(ORecordSerializerBinaryV0.java:385)
at com.orientechnologies.orient.core.serialization.serializer.record.binary.ORecordSerializerBinary.toStream(ORecordSerializerBinary.java:99)
at com.orientechnologies.orient.core.record.impl.ODocument.toStream(ODocument.java:2381)
at com.orientechnologies.orient.core.record.impl.ODocument.toStream(ODocument.java:664)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commitEntry(OAbstractPaginatedStorage.java:4380)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1697)
at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.commit(ODistributedStorage.java:1389)
at com.orientechnologies.orient.server.distributed.impl.ODistributedTransactionManager$1.call(ODistributedTransactionManager.java:117)
at com.orientechnologies.orient.core.db.OScenarioThreadLocal.executeAsDistributed(OScenarioThreadLocal.java:70)
at com.orientechnologies.orient.server.distributed.impl.ODistributedTransactionManager.commit(ODistributedTransactionManager.java:114)
at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.commit(ODistributedStorage.java:1432)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:541)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:99)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2895)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2864)
at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.commit(OrientTransactionalGraph.java:182)
at com.orientechnologies.orient.graph.sql.OGraphCommandExecutorSQLFactory.runInConfiguredTxMode(OGraphCommandExecutorSQLFactory.java:265)
at com.orientechnologies.orient.graph.sql.OCommandExecutorSQLCreateEdge.execute(OCommandExecutorSQLCreateEdge.java:143)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:74)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3213)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3151)
at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.command(ODistributedStorage.java:261)
at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.command(ONetworkProtocolBinary.java:1529)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:646)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:384)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:207)
at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)

Steps to reproduce

[andrii0lomakin]

@KishoDai I suppose you created a database under standalone server instance and then use it under distributed database it will not work, unfortunately. You need to export your database into JSON, start distributed server and import database to this server.

[KishoDai]

@Laa
Wow, Why so ?

[KishoDai]

Hi @Laa
You mean If I run in standalone mode , then I can't run dserver.sh later?

[KishoDai]

Hi @Laa
I encounted all kinds of exceptions recently after run under master-slave distributed mode .

[andrii0lomakin]

@KishoDai , those modes use different settings so you need to run in one of those modes. I will switch you to @tglman he can help you with distributed.

[KishoDai]

Hi @tglman
Need your help.

[tglman]

hi @KishoDai,

The issue here is due to the fact that you imported the database in standalone and run after in distributed, this can be done, but in distributed environment some feature ore turned off because they are not supported (tree ridbags), so you should disable this in standalone when you do the import.
for disable the ridbag tree you should put http://orientdb.com/docs/last/Configuration.html#ridbagembeddedtosbtreebonsaithreshold to -1 you can do this in the java code as well doing OGlobalConfiguration.RID_BAG_EMBEDDED_TO_SBTREEBONSAI_THRESHOLD.setValue(-1);

this will fix also the other few issue that you had.

Regards

[KishoDai]

Hi @tglman
Thank you very much. I will have a try.

[KishoDai]

Hi @tglman
http://orientdb.com/docs/last/RidBag.html
In this page, it said "NOTE: While running in distributed mode SBTrees are not supported. If using a distributed database then you must set ridBag.embeddedToSbtreeBonsaiThreshold=Integer.MAX_VALUE to avoid replication errors.
"
That means I shoud set the variable to Integer.MAX_VALUES in distributed mode?

[tglman]

Hi @KishoDai,

Yes I followed my memory and I was wrong, the correct way is MAX_VALUE like the docs says.

Regards

[KishoDai]

Hi @tglman

orientdb {db=sns}> config get ridBag.embeddedToSbtreeBonsaiThreshold

Local configuration: ridBag.embeddedToSbtreeBonsaiThreshold = 2147483647

It shows as above, I think it's correct?

[tglman]

hi @KishoDai

Yes it's correct.

Regards

[tglman]

Hi @KishoDai,

Did you have the chance to try the case ? is it fixed ?

Regards

[KishoDai]

Hi @tglman
I tried to export to json under standalone, the logs as below:

Done. Exported 197400555 of total 197400556 records. 1 records were detected as broken

Exporting index info...

• Index OUser.name...OK
• Index Apply.applyNo...OK
• Index Device.deviceId...OK
• Index AddressBookFlow.unique...OK
• Index Location.unique...OK
• Index PhoneCleanMark.mark...OK
• Index OFunction.name...OK
• Index Phone.phone...OK
• Index Store.storeId...OK
• Index dictionary...OK
• Index Member.memberId...OK
• Index Apply.createdDatetime...OK
• Index Ip.ip...OK
• Index PhoneSource.source...OK
• Index Order.orderNo...OK
• Index Order.createdDatetime...OK
• Index Address.unique...OK
• Index ORole.name...OK
  OK (18 indexes)
  Exporting manual indexes content...
• Exporting index dictionary ...OK (entries=0)
  OK (1 manual indexes)

Database export completed in 41706041ms

But when I import under the distributed mode, it show the errors as below:

Importing database database /mnt/sns_20190920.json.gz...
Started import of database 'plocal:/mnt/orientdb-enterprise-2.2.26/databases/sns' from /tmp/snsimport1580046398883498988sns_20190920.json.gz...
Non merge mode (-merge=false): removing all default non security classesError on database import happened just before line 0, column 1
com.orientechnologies.orient.core.exception.ORecordNotFoundException: The record has no id. It is probably new or still transient
DB name="sns"
at com.orientechnologies.orient.core.record.ORecordAbstract.reload(ORecordAbstract.java:259)
at com.orientechnologies.orient.core.record.ORecordAbstract.reload(ORecordAbstract.java:245)
at com.orientechnologies.orient.core.type.ODocumentWrapper.reload(ODocumentWrapper.java:86)
at com.orientechnologies.orient.core.type.ODocumentWrapperNoClass.reload(ODocumentWrapperNoClass.java:66)
at com.orientechnologies.orient.core.index.OIndexManagerAbstract.reload(OIndexManagerAbstract.java:97)
at com.orientechnologies.orient.core.index.OIndexManagerRemote.dropIndex(OIndexManagerRemote.java:95)
at com.orientechnologies.orient.core.index.OIndexManagerProxy.dropIndex(OIndexManagerProxy.java:97)
at com.orientechnologies.orient.core.db.tool.ODatabaseImport.removeDefaultNonSecurityClasses(ODatabaseImport.java:411)
at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:161)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.importDatabase(ONetworkProtocolBinary.java:2973)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:702)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:384)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:207)
at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)

Error: com.orientechnologies.common.io.OIOException

Error: java.io.EOFException

Importing database schema...OK (56 classes)Error on database import happened just before line 0, column 38244
com.orientechnologies.orient.core.exception.ORecordNotFoundException: The record has no id. It is probably new or still transient
DB name="sns"
at com.orientechnologies.orient.core.record.ORecordAbstract.reload(ORecordAbstract.java:259)
at com.orientechnologies.orient.core.record.ORecordAbstract.reload(ORecordAbstract.java:245)
at com.orientechnologies.orient.core.type.ODocumentWrapper.reload(ODocumentWrapper.java:86)
at com.orientechnologies.orient.core.type.ODocumentWrapperNoClass.reload(ODocumentWrapperNoClass.java:66)
at com.orientechnologies.orient.core.index.OIndexManagerAbstract.reload(OIndexManagerAbstract.java:97)
at com.orientechnologies.orient.core.index.OIndexManagerRemote.dropIndex(OIndexManagerRemote.java:95)
at com.orientechnologies.orient.core.index.OIndexManagerProxy.dropIndex(OIndexManagerProxy.java:97)
at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importRecords(ODatabaseImport.java:1017)
at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:183)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.importDatabase(ONetworkProtocolBinary.java:2973)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:702)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:384)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:207)
at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)

Error: com.orientechnologies.common.io.OIOException

Error: java.io.EOFException

[tglman]

Hi @KishoDai,

Since 2.2.33 we introduced auto migration of ridbag tree this issue is fixed, for the other errors there are others issue for tracking them, closing this.

Regards