Hello community, we are using OrientDB in our product, and since the update from version 2.1.15 to version 2.2.34, we have the problem described below, which occurs sporadically. Is there anything we can do to mitigate it or to identify the actual error source? We are grateful for every kind of help.
- Simon *OrientDB release?* 2.2.34 *What steps will reproduce the problem?* Unknown - our attempts to reproduce the problem have been very unsatisfying yet. Nonetheless, the problem occurs again and again (without a recognizable pattern), in different runtime environments of different customers, and in different situations. It seems to happen in random situations when the database is accessed. It happens that one query is successful, and the next one fails (with the error below), and then every subsequent query also fails. Symptoms: An InvalidClassException occurs sometimes when querying the database: Caused by: com.orientechnologies.common.io.OIOException: com.o; serializable and externalizable flags conflict at com.orientechnologies.orient.client.remote.OStorageRemote.handleIOException(OStorageRemote.java:321) at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:292) at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperationRetry(OStorageRemote.java:204) at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperation(OStorageRemote.java:215) at com.orientechnologies.orient.client.remote.OStorageRemote.command(OStorageRemote.java:1187) at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69) at com.tinkerpop.blueprints.impls.orient.OrientGraphCommand.execute(OrientGraphCommand.java:49) at com.mycompany.BaseDAOImpl.query(OrientConnection.java:188) ... 13 common frames omitted Caused by: java.io.InvalidClassException: com.o; serializable and externalizable flags conflict at java.base/java.io.ObjectStreamClass.readNonProxy(Unknown Source) at java.base/java.io.ObjectInputStream.readClassDescriptor(Unknown Source) at java.base/java.io.ObjectInputStream.readNonProxyDesc(Unknown Source) at java.base/java.io.ObjectInputStream.readClassDesc(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream.readObject(Unknown Source) at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.throwSerializedException(OChannelBinaryAsynchClient.java:436) at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:400) at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:283) at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:167) at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:2365) at com.orientechnologies.orient.client.remote.OStorageRemote$27.execute(OStorageRemote.java:1211) at com.orientechnologies.orient.client.remote.OStorageRemote$2.execute(OStorageRemote.java:207) at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:252) ... 19 common frames omitted The *BaseDAOImpl.query* looks like: public Iterable<Vertex> query(String query, Object... params) throws PersistenceException { OrientBaseGraph graph = OrientGraph.getActiveGraph(); try { OCommandSQL command = new OCommandSQL(query); graph.clearCache(); return graph.command(command).execute(params); } catch (Exception e) { throw new PersistenceException("Query triggered an exception", e); } } It is called in the following context: OrientGraphFactory factory = new OrientGraphFactory("remote:localhost/mydb", user, password).setupPool(10, 50); setRequireTransaction(true); setAutoStartTx(false); OrientGraph db = factory.getTx(); ... baseDAO.query("select from MyClass"); ... db.shutdown(); *If you're using custom settings please provide them below (to dump all the settings run the application using the JVM argument -Denvironment.dumpCfgAtStartup=true):* OrientDB 2.2.34 (build f340442755a31eabc91b87cb3ef99eda5cee6ebd, branch 2.2.x) configuration dump: - ENVIRONMENT + environment.dumpCfgAtStartup = true + environment.concurrent = true + environment.lockManager.concurrency.level = 64 + environment.allowJVMShutdown = true - SCRIPT + script.pool.maxSize = 20 - MEMORY + memory.useUnsafe = true + memory.chunk.size = 2147483647 + memory.directMemory.safeMode = true + memory.directMemory.trackMode = false + memory.directMemory.onlyAlignedMemoryAccess = true - JVM + jvm.gc.delayForOptimize = 600 - STORAGE + storage.openFiles.limit = 512 + storage.componentsLock.cache = 10000 + storage.diskCache.pinnedPages = 20 + storage.diskCache.bufferSize = 4054 + storage.diskCache.writeCachePart = 15 + storage.diskCache.writeCachePageTTL = 86400 + storage.diskCache.writeCachePageFlushInterval = 25 + storage.diskCache.writeCacheFlushInactivityInterval = 60000 + storage.diskCache.writeCacheFlushLockTimeout = -1 + storage.diskCache.diskFreeSpaceCheckInterval = 5 + storage.diskCache.diskFreeSpaceCheckIntervalInPages = 2048 + storage.diskCache.keepState = true + storage.diskCache.checksumMode = StoreAndSwitchReadOnlyMode + storage.configuration.syncOnUpdate = true + storage.compressionMethod = nothing + storage.encryptionMethod = nothing + storage.encryptionKey = <hidden> + storage.makeFullCheckpointAfterCreate = false + storage.makeFullCheckpointAfterOpen = false + storage.makeFullCheckpointAfterClusterCreate = true + storage.trackChangedRecordsInWAL = false + storage.useWAL = true + storage.wal.syncOnPageFlush = true + storage.wal.cacheSize = 3000 + storage.wal.fileAutoCloseInterval = 10 + storage.wal.maxSegmentSize = 128 + storage.wal.maxSize = 4096 + storage.wal.commitTimeout = 1000 + storage.wal.shutdownTimeout = 10000 + storage.wal.fuzzyCheckpointInterval = 300 + storage.wal.reportAfterOperationsDuringRestore = 10000 + storage.wal.restore.batchSize = 50000 + storage.wal.readCacheSize = 1000 + storage.wal.fuzzyCheckpointShutdownWait = 600 + storage.wal.fullCheckpointShutdownTimeout = 600 + storage.wal.path = null + storage.diskCache.pageSize = 64 + storage.diskCache.diskFreeSpaceLimit = 256 + storage.lowestFreeListBound = 16 + storage.lockTimeout = 0 + storage.record.lockTimeout = 2000 + storage.useTombstones = false - RECORD + record.downsizing.enabled = true - OBJECT + object.saveOnlyDirty = false - DB + db.pool.min = 50 + db.pool.max = 100 + db.pool.idleTimeout = 0 + db.pool.idleCheckDelay = 0 + db.mvcc.throwfast = false + db.validation = true - NONTX + nonTX.recordUpdate.synch = false + nonTX.clusters.sync.immediately = manindex - TX + tx.trackAtomicOperations = false + tx.pageCacheSize = 12 - INDEX + index.embeddedToSbtreeBonsaiThreshold = 40 + index.sbtreeBonsaiToEmbeddedThreshold = -1 - HASHTABLE + hashTable.slitBucketsBuffer.length = 1500 - INDEX + index.auto.synchronousAutoRebuild = true + index.auto.lazyUpdates = 10000 + index.flushAfterCreate = true + index.manual.lazyUpdates = 1 + index.durableInNonTxMode = true + index.ignoreNullValuesDefault = false + index.txMode = FULL + index.cursor.prefetchSize = 10000 - SBTREE + sbtree.maxDepth = 64 + sbtree.maxKeySize = 10240 + sbtree.maxEmbeddedValueSize = 40960 - SBTREEBONSAI + sbtreebonsai.bucketSize = 2 + sbtreebonsai.linkBagCache.size = 100000 + sbtreebonsai.linkBagCache.evictionSize = 1000 + sbtreebonsai.freeSpaceReuseTrigger = 0.5 - RIDBAG + ridBag.embeddedDefaultSize = 4 + ridBag.embeddedToSbtreeBonsaiThreshold = 40 + ridBag.sbtreeBonsaiToEmbeddedToThreshold = -1 - COLLECTIONS + collections.preferSBTreeSet = false - FILE + file.trackFileClose = false + file.lock = true + file.deleteDelay = 10 + file.deleteRetry = 50 - SECURITY + security.userPasswordSaltIterations = 65536 + security.userPasswordSaltCacheSize = 500 + security.userPasswordDefaultAlgorithm = PBKDF2WithHmacSHA256 - NETWORK + network.maxConcurrentSessions = 1000 + network.socketBufferSize = 0 + network.lockTimeout = 15000 + network.socketTimeout = 15000 + network.requestTimeout = 3600000 + network.retry.strategy = auto + network.retry = 5 + network.retryDelay = 500 + network.binary.loadBalancing.enabled = false + network.binary.loadBalancing.timeout = 2000 + network.binary.maxLength = 16384 + network.binary.readResponse.maxTimes = 20 + network.binary.debug = false + network.http.installDefaultCommands = true + network.http.serverInfo = OrientDB Server v.2.2.34 (build f340442755a31eabc91b87cb3ef99eda5cee6ebd, branch 2.2.x) + network.http.maxLength = 1000000 + network.http.streaming = false + network.http.charset = utf-8 + network.http.jsonResponseError = true + network.http.jsonp = false + network.http.sessionExpireTimeout = 300 + network.http.useToken = false + network.token.secretKey = + network.token.encryptionAlgorithm = HmacSHA256 + network.token.expireTimeout = 60 - PROFILER + profiler.enabled = false + profiler.config = null + profiler.autoDump.interval = 0 + profiler.autoDump.type = full + profiler.maxValues = 200 + profiler.memoryCheckInterval = 120000 - SEQUENCE + sequence.maxRetry = 100 + sequence.retryDelay = 200 - STORAGEPROFILER + storageProfiler.intervalBetweenSnapshots = 100 + storageProfiler.cleanUpInterval = 5000 - LOG + log.console.level = info + log.file.level = fine - CLASS + class.minimumClusters = 0 - LOG + log.console.ansi = auto - CACHE + cache.local.impl = com.orientechnologies.orient.core.cache.ORecordCacheWeakRefs - COMMAND + command.timeout = 0 + command.cache.enabled = false + command.cache.evictStrategy = PER_CLUSTER + command.cache.minExecutionTime = 10 + command.cache.maxResultsetSize = 500 - QUERY + query.useSoftReferencesInResultSet = false + query.parallelAuto = false + query.parallelMinimumRecords = 300000 + query.parallelResultQueueSize = 20000 + query.scanPrefetchPages = 20 + query.scanBatchSize = 1000 + query.scanThresholdTip = 50000 + query.limitThresholdTip = 10000 + query.live.support = true + query.timeout.defaultStrategy = EXCEPTION - LUCENE + lucene.query.pageSize = 10000 - STATEMENT + statement.cacheSize = 100 - SQL + sql.graphConsistencyMode = tx - CLIENT + client.channel.maxPool = 200 + client.connectionPool.waitTimeout = 5000 + client.channel.dbReleaseWaitTimeout = 10000 + client.ssl.enabled = false + client.ssl.keyStore = null + client.ssl.keyStorePass = null + client.ssl.trustStore = null + client.ssl.trustStorePass = null - SERVER + server.openAllDatabasesAtStartup = false + server.channel.cleanDelay = 5000 + server.cache.staticFile = false + server.log.dumpClientExceptionLevel = FINE + server.log.dumpClientExceptionFullStackTrace = false - DISTRIBUTED + distributed.dumpStatsEvery = 0 + distributed.crudTaskTimeout = 10000 + distributed.maxStartupDelay = 10000 + distributed.commandTaskTimeout = 120000 + distributed.commandQuickTaskTimeout = 5000 + distributed.commandLongTaskTimeout = 86400000 + distributed.deployDbTaskTimeout = 1200000 + distributed.deployChunkTaskTimeout = 60000 + distributed.deployDbTaskCompression = 7 + distributed.asynchQueueSize = 0 + distributed.asynchResponsesTimeout = 15000 + distributed.purgeResponsesTimerDelay = 15000 + distributed.conflictResolverRepairerChain = quorum,content,majority,version + distributed.conflictResolverRepairerCheckEvery = 5000 + distributed.conflictResolverRepairerBatch = 50 + distributed.txAliveTimeout = 30000 + distributed.requestChannels = 1 + distributed.responseChannels = 1 + distributed.queueTimeout = 15000 + distributed.heartbeatTimeout = 10000 + distributed.checkHealthCanOfflineServer = false + distributed.checkHealthEvery = 10000 + distributed.autoRemoveOfflineServers = 0 + distributed.publishNodeStatusEvery = 10000 + distributed.localQueueSize = 10000 + distributed.dbWorkerThreads = 0 + distributed.queueMaxSize = 10000 + distributed.backupDirectory = ../backup/databases + distributed.backupTryIncrementalFirst = true + distributed.checkIntegrityLastTxs = 16 + distributed.concurrentTxMaxAutoRetry = 15 + distributed.atomicLockTimeout = 1000 + distributed.concurrentTxAutoRetryDelay = 10 - DB + db.document.serializer = ORecordSerializerBinary - CLIENT + client.krb5.config = null + client.krb5.ccname = null + client.krb5.ktname = null + client.credentialinterceptor = null + client.ci.keyalgorithm = AES + client.ci.ciphertransform = AES/CBC/PKCS5Padding + client.ci.keystore.file = null + client.ci.keystore.password = null - SECURITY + security.createDefaultUsers = true - SERVER + server.security.file = null - ORIENT + orient.initInServletContextListener = true - JNA + jna.disable.system.library = true - DB + db.makeFullCheckpointOnIndexChange = true + db.makeFullCheckpointOnSchemaChange = true - CLIENT + client.session.tokenBased = true - OAUTH2 + oauth2.secretkey = - STORAGE + storage.cluster.usecrc32 = false - LAZYSET + lazyset.workOnStream = true - DB + db.mvcc = true + db.use.distributedVersion = false - MVRBTREE + mvrbtree.timeout = 0 + mvrbtree.nodePageSize = 256 + mvrbtree.loadFactor = 0.7 + mvrbtree.optimizeThreshold = 100000 + mvrbtree.entryPoints = 64 + mvrbtree.optimizeEntryPointsFactor = 1.0 + mvrbtree.entryKeysInMemory = false + mvrbtree.entryValuesInMemory = false + mvrbtree.ridBinaryThreshold = -1 + mvrbtree.ridNodePageSize = 64 + mvrbtree.ridNodeSaveMemory = false - TX + tx.commit.synch = false + tx.autoRetry = 1 + tx.log.fileType = classic + tx.log.synch = false + tx.useLog = true - INDEX + index.auto.rebuildAfterNotSoftClose = true - CLIENT + client.channel.minPool = 50 - STORAGE + storage.keepOpen = true - CACHE + cache.local.enabled = true - DISTRIBUTED + distributed.shutdownTimeout = 2000 -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to orient-database+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.