Resending with screenshots.
> I am not able reproduce this issue in small reproducer project but this is > consistently happening in my project. So I debugged this issue and attached > the screenshot in this mail. > > > > *NOTE: *This issue occurs if the statistics are enabled on cache > configuration level [cacheCfg.setStatisticsEnabled(true)]. > > > > As shown in screenshots there 22 cache metrics in cacheMetrics hashmap. > All these cache metrics get serialized successfully. But at the time of > deserialization on client node only first metrics get de-serialized > successfully but all other metrics till iteration count 13 are > de-serialized as null value and on iteration 14 “ref” byte value in > “OptimizedObjectInputStream.readObject0()” method is read as 81 and code > throws exception. > > > > I think this where it is going wrong. The object copy at the time of > serialization and de-serialization should be same but that’s not happening > in Ignite 2.7 version. So I debugged this on Ignite 2.6 version. On 2.6 all > 22 cacheMetrics are being de-serialized successfully. > > > > *AffinityJob result being serialized on server:* > > > > [image: server.png] > > > > > > *AffinityJob result being de-serialized on client:* > > > > [image: client.jpg] > > Thanks, > Prasad > > > On Wed, Jan 9, 2019 at 6:41 PM Ilya Kasnacheev <ilya.kasnach...@gmail.com> > wrote: > >> Hello! >> >> Do you have a reproducer project to reliably confirm this issue? >> >> Regards, >> -- >> Ilya Kasnacheev >> >> >> ср, 9 янв. 2019 г. в 12:39, Akash Shinde <akashshi...@gmail.com>: >> >>> Added dev@ignite.apache.org. >>> >>> Should I log Jira for this issue? >>> >>> Thanks, >>> Akash >>> >>> >>> >>> On Tue, Jan 8, 2019 at 6:16 PM Akash Shinde <akashshi...@gmail.com> >>> wrote: >>> >>> > Hi, >>> > >>> > No both nodes, client and server are running on Ignite 2.7 version. I >>> am >>> > starting both server and client from Intellij IDE. >>> > >>> > Version printed in Server node log: >>> > Ignite ver. >>> 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db >>> > >>> > Version in client node log: >>> > Ignite ver. >>> 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db >>> > >>> > Thanks, >>> > Akash >>> > >>> > On Tue, Jan 8, 2019 at 5:18 PM Mikael <mikael-arons...@telia.com> >>> wrote: >>> > >>> >> Hi! >>> >> >>> >> Any chance you might have one node running 2.6 or something like that >>> ? >>> >> >>> >> It looks like it get a different object that does not match the one >>> >> expected in 2.7 >>> >> >>> >> Mikael >>> >> Den 2019-01-08 kl. 12:21, skrev Akash Shinde: >>> >> >>> >> Before submitting the affinity task ignite first gets the affinity >>> cached >>> >> function (AffinityInfo) by submitting the cluster wide task >>> "AffinityJob". >>> >> But while in the process of retrieving the output of this AffinityJob, >>> >> ignite deserializes this output. I am getting exception while >>> deserailizing >>> >> this output. >>> >> In TcpDiscoveryNode.readExternal() method while deserailizing the >>> >> CacheMetrics object from input stream on 14th iteration I am getting >>> >> following exception. Complete stack trace is given in this mail chain. >>> >> >>> >> Caused by: java.io.IOException: Unexpected error occurred during >>> >> unmarshalling of an instance of the class: >>> >> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. >>> >> >>> >> This is working fine on Ignite 2.6 version but giving problem on 2.7. >>> >> >>> >> Is this a bug or am I doing something wrong? >>> >> >>> >> Can someone please help? >>> >> >>> >> On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde <akashshi...@gmail.com> >>> >> wrote: >>> >> >>> >>> Hi, >>> >>> >>> >>> When execute affinity.partition(key), I am getting following >>> exception >>> >>> on Ignite 2.7. >>> >>> >>> >>> Stacktrace: >>> >>> >>> >>> 2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR >>> >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: >>> >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, >>> >>> sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, >>> >>> jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, >>> >>> isCancelled=false, retry=null] >>> >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object >>> >>> with optimized marshaller >>> >>> at >>> >>> >>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146) >>> >>> at >>> >>> >>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831) >>> >>> at >>> >>> >>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081) >>> >>> at >>> >>> >>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316) >>> >>> at >>> >>> >>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) >>> >>> at >>> >>> >>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) >>> >>> at >>> >>> >>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) >>> >>> at >>> >>> >>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) >>> >>> at >>> >>> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> >>> at >>> >>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> >>> at java.lang.Thread.run(Thread.java:748) >>> >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to >>> >>> unmarshal object with optimized marshaller >>> >>> at >>> >>> >>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765) >>> >>> at >>> >>> >>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964) >>> >>> at >>> >>> >>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) >>> >>> at >>> >>> >>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313) >>> >>> at >>> >>> >>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102) >>> >>> at >>> >>> >>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) >>> >>> at >>> >>> >>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140) >>> >>> ... 10 common frames omitted >>> >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to >>> >>> deserialize object with given class loader: >>> >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to >>> >>> deserialize object >>> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]] >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237) >>> >>> at >>> >>> >>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) >>> >>> at >>> >>> >>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762) >>> >>> ... 16 common frames omitted >>> >>> Caused by: java.io.IOException: Failed to deserialize object >>> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3] >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228) >>> >>> ... 18 common frames omitted >>> >>> Caused by: java.io.IOException: Failed to deserialize object >>> >>> >>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment] >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> >>> at >>> >>> >>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >>> >>> ... 21 common frames omitted >>> >>> Caused by: java.io.IOException: Failed to deserialize field >>> >>> [name=assignment] >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >>> >>> ... 27 common frames omitted >>> >>> Caused by: java.io.IOException: Failed to deserialize object >>> >>> >>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode] >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519) >>> >>> ... 30 common frames omitted >>> >>> Caused by: java.io.IOException: Unexpected error occurred during >>> >>> unmarshalling of an instance of the class: >>> >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. >>> Check >>> >>> that all nodes are running the same version of Ignite and that all >>> nodes >>> >>> have GridOptimizedMarshaller configured with identical optimized >>> classes >>> >>> lists, if any (see setClassNames and setClassNamesPath methods). If >>> your >>> >>> serialized classes implement java.io.Externalizable interface, >>> verify that >>> >>> serialization logic is correct. >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> >>> at >>> >>> >>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >>> >>> ... 41 common frames omitted >>> >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread( >>> >>> qzkpr01.p13.eng.in03.com:50024)] INFO >>> org.apache.zookeeper.ClientCnxn >>> >>> - Opening socket connection to server >>> >>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to >>> >>> authenticate using SASL (unknown error) >>> >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread( >>> >>> qzkpr01.p13.eng.in03.com:50024)] INFO >>> org.apache.zookeeper.ClientCnxn >>> >>> - Socket connection established to >>> >>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session >>> >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread( >>> >>> qzkpr01.p13.eng.in03.com:50024)] WARN >>> org.apache.zookeeper.ClientCnxn >>> >>> - Unable to reconnect to ZooKeeper service, session >>> 0x3673a59f5dd01c7 has >>> >>> expired >>> >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread( >>> >>> qzkpr01.p13.eng.in03.com:50024)] INFO >>> org.apache.zookeeper.ClientCnxn >>> >>> - Unable to reconnect to ZooKeeper service, session >>> 0x3673a59f5dd01c7 has >>> >>> expired, closing socket connection >>> >>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO >>> >>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired) >>> >>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO >>> >>> org.apache.zookeeper.ZooKeeper - Initiating client connection, >>> >>> connectString=qzkpr01.p13.eng.in03.com:50024, >>> >>> qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka >>> >>> sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288 >>> >>> 2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO >>> >>> org.apache.zookeeper.ClientCnxn - EventThread shut down for session: >>> >>> 0x3673a59f5dd01c7 >>> >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread( >>> >>> qzkpr03.p13.eng.in03.com:50024)] INFO >>> org.apache.zookeeper.ClientCnxn >>> >>> - Opening socket connection to server >>> >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to >>> >>> authenticate using SASL (unknown error) >>> >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread( >>> >>> qzkpr03.p13.eng.in03.com:50024)] INFO >>> org.apache.zookeeper.ClientCnxn >>> >>> - Socket connection established to >>> >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session >>> >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread( >>> >>> qzkpr03.p13.eng.in03.com:50024)] INFO >>> org.apache.zookeeper.ClientCnxn >>> >>> - Session establishment complete on server >>> >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid = >>> >>> 0x3673a59f5dd01ce, negotiated timeout = 10000 >>> >>> 2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO >>> >>> org.I0Itec.zkclient.ZkClient - zookeeper state changed >>> (SyncConnected) >>> >>> 2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR >>> >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: >>> >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, >>> >>> sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, >>> >>> jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, >>> >>> isCancelled=false, retry=null] >>> >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object >>> >>> with optimized marshaller >>> >>> at >>> >>> >>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146) >>> >>> at >>> >>> >>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831) >>> >>> at >>> >>> >>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081) >>> >>> at >>> >>> >>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316) >>> >>> at >>> >>> >>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) >>> >>> at >>> >>> >>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) >>> >>> at >>> >>> >>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) >>> >>> at >>> >>> >>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) >>> >>> at >>> >>> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> >>> at >>> >>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> >>> at java.lang.Thread.run(Thread.java:748) >>> >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to >>> >>> unmarshal object with optimized marshaller >>> >>> at >>> >>> >>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765) >>> >>> at >>> >>> >>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964) >>> >>> at >>> >>> >>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) >>> >>> at >>> >>> >>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313) >>> >>> at >>> >>> >>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102) >>> >>> at >>> >>> >>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) >>> >>> at >>> >>> >>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140) >>> >>> ... 10 common frames omitted >>> >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to >>> >>> deserialize object with given class loader: >>> >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to >>> >>> deserialize object >>> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]] >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237) >>> >>> at >>> >>> >>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) >>> >>> at >>> >>> >>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762) >>> >>> ... 16 common frames omitted >>> >>> Caused by: java.io.IOException: Failed to deserialize object >>> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3] >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228) >>> >>> ... 18 common frames omitted >>> >>> Caused by: java.io.IOException: Failed to deserialize object >>> >>> >>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment] >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> >>> at >>> >>> >>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >>> >>> ... 21 common frames omitted >>> >>> Caused by: java.io.IOException: Failed to deserialize field >>> >>> [name=assignment] >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >>> >>> ... 27 common frames omitted >>> >>> Caused by: java.io.IOException: Failed to deserialize object >>> >>> >>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode] >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519) >>> >>> ... 30 common frames omitted >>> >>> Caused by: java.io.IOException: Unexpected error occurred during >>> >>> unmarshalling of an instance of the class: >>> >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. >>> Check >>> >>> that all nodes are running the same version of Ignite and that all >>> nodes >>> >>> have GridOptimizedMarshaller configured with identical optimized >>> classes >>> >>> lists, if any (see setClassNames and setClassNamesPath methods). If >>> your >>> >>> serialized classes implement java.io.Externalizable interface, >>> verify that >>> >>> serialization logic is correct. >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> >>> at >>> >>> >>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >>> >>> ... 41 common frames omitted >>> >>> 2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR >>> >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response: >>> >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046, >>> >>> sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, >>> >>> jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null, >>> >>> isCancelled=false, retry=null] >>> >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object >>> >>> with optimized marshaller >>> >>> at >>> >>> >>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146) >>> >>> at >>> >>> >>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831) >>> >>> at >>> >>> >>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081) >>> >>> at >>> >>> >>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316) >>> >>> at >>> >>> >>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) >>> >>> at >>> >>> >>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) >>> >>> at >>> >>> >>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) >>> >>> at >>> >>> >>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) >>> >>> at >>> >>> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> >>> at >>> >>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> >>> at java.lang.Thread.run(Thread.java:748) >>> >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to >>> >>> unmarshal object with optimized marshaller >>> >>> at >>> >>> >>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765) >>> >>> at >>> >>> >>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964) >>> >>> at >>> >>> >>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) >>> >>> at >>> >>> >>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313) >>> >>> at >>> >>> >>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102) >>> >>> at >>> >>> >>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) >>> >>> at >>> >>> >>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140) >>> >>> ... 10 common frames omitted >>> >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to >>> >>> deserialize object with given class loader: >>> >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to >>> >>> deserialize object >>> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]] >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237) >>> >>> at >>> >>> >>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) >>> >>> at >>> >>> >>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762) >>> >>> ... 16 common frames omitted >>> >>> Caused by: java.io.IOException: Failed to deserialize object >>> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3] >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228) >>> >>> ... 18 common frames omitted >>> >>> Caused by: java.io.IOException: Failed to deserialize object >>> >>> >>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment] >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> >>> at >>> >>> >>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >>> >>> ... 21 common frames omitted >>> >>> Caused by: java.io.IOException: Failed to deserialize field >>> >>> [name=assignment] >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >>> >>> ... 27 common frames omitted >>> >>> Caused by: java.io.IOException: Failed to deserialize object >>> >>> >>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode] >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519) >>> >>> ... 30 common frames omitted >>> >>> Caused by: java.io.IOException: Unexpected error occurred during >>> >>> unmarshalling of an instance of the class: >>> >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. >>> Check >>> >>> that all nodes are running the same version of Ignite and that all >>> nodes >>> >>> have GridOptimizedMarshaller configured with identical optimized >>> classes >>> >>> lists, if any (see setClassNames and setClassNamesPath methods). If >>> your >>> >>> serialized classes implement java.io.Externalizable interface, >>> verify that >>> >>> serialization logic is correct. >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) >>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425) >>> >>> at >>> >>> >>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949) >>> >>> at >>> >>> >>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) >>> >>> ... 41 common frames omitted >>> >>> >>> >>> Thanks, >>> >>> Akash >>> >>> >>> >> >>> >>
