[
https://issues.apache.org/jira/browse/IGNITE-20688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850685#comment-17850685
]
Mikhail Petrov commented on IGNITE-20688:
-----------------------------------------
Root cause of the issue:
Ignite does not consider references (handles) during the Binary Object
detachment process.
As a result, after detaching a binary object, it may contain references to the
part of the binary array that was cut off during the detaching procedure.
This issue primarily affects Java thin client and cache operations that use
collections (maps/arrays/lists, etc.) as the value.
Java thin client serializes collection as a single object and reuses references
across all collections elements.
When a cache request with a serialized collection is received by the Ignite
server node, it is unmarshalled to a collection of binary objects, each
containing its own chunk of of the serialized value bytes after "detach"
procedure. This is where the problem with orphaned references can arises.
The following approaches for solving this issue was considered:
1. The most preferred way is to completely avoid unmarshalling value bytes
serialized by the thin client during cache#put operations (marshalling cache
value bytes stored in cache during thin client cache#get operations) on server
nodes (see https://issues.apache.org/jira/browse/IGNITE-12625).
Currently this approach is error prone and may cause more problems than it
solves. Ignite intercepts values during cache operations processing - it can be
JCache interceptors or different listeners/filters/policies. In this cases we
are forced to unmarshal raw value bytes to Binary Object and we must avoid of
detach operation as it can lead to the mentioned orphaned references problem.
Keep in mind that if the cache value is a collection of objects, then after
unmarshalling we will end up with a collection of binary objects, each
containing a reference to a byte array that contains ALL the elements of the
serialized collections and corresponding offset. If we will try to store these
Binary Objects in cache or send them over the network, we may end up with
sending/storing byte array containing ALL serialized collection elements for
each collection's element. And this situation is difficult to detect because of
mentioned interceptors.
Here you should also consider that collections serialized on thin client may
differ from the ones serialized on Ignite node. Because thin client serializes
collection as single element and Ignite node serialize each element
independently. As a result it is possible that in some cases keys that are
stored using thin client will not be visible from the nodes and vice versa or
we can face problems with CAS operations.
2. Resolving orphaned references during Binary Object detach procedure - we
just replace orphaned reference with object and reassemble the binary object.
This approach eliminates differences in collection serialization between Ignite
node and Thin client. So mentioned problems with CAS or missing keys are no
longer available. The goal is not to avoid the binary object detach process,
but to make it work correctly. Also it is compatible with thin client of old
versions.
But detection of orphaned references is costly - we need to iterate over object
binary representation. Not to mention reassembling the object if we need to
replace the reference to the real object.
The second approach was chosen. As a result we will get performance drop for
cache operations that uses collections as value and are performed from java
thin client. The degree of degradation depends on the structure of the
collection's values and the length of the collection.
You can consider it to be around 4-5% in case cross object references are not
found. Still it can be completely eliminated by wrapping collection to object.
> Java Thin Client - Error while deserializing Collection
> -------------------------------------------------------
>
> Key: IGNITE-20688
> URL: https://issues.apache.org/jira/browse/IGNITE-20688
> Project: Ignite
> Issue Type: Bug
> Components: binary, thin client
> Affects Versions: 2.9, 2.10, 2.12, 2.13, 2.14, 2.15
> Reporter: Rahul Mohan
> Assignee: Mikhail Petrov
> Priority: Critical
> Labels: ise
> Attachments: image001.png
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> I have encountered an issue in deserializing cache values which are of
> Collection type.
> The issue occurs if a field in different objects within the collection
> points to the same reference.
> *Versions:*
> org.apache.ignite:ignite-core:2.9.0 to org.apache.ignite:ignite-core:2.15.0
>
> {code:java}
> Person.java
> public class Person implements Serializable {
> private String id;
> private String firstName;
> private String lastName;
> private double salary;
> private String country;
> private String deleted;
> private Set<String> accounts;
> }
> Client
> ClientCacheConfiguration cacheCfg = new
> ClientCacheConfiguration().setName(cacheName).
> setCacheMode(CacheMode.REPLICATED).
>
> setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
>
> cache = client.getOrCreateCache(cacheCfg);
>
> Set<String> set = new HashSet<>();
> set.add("1");
>
> List<Person> persons = new ArrayList<>();
> persons.add(new Person("105286a4","Jack","Smith",10000f,
> "USA","false", set));
> persons.add(new Person("98545b0fd3af","John", "Doe", 500000f,
> "Australia","false", null));
> persons.add(new Person("98545b0fd3afd","Hari","M",400000f,
> "India", null, null));
> persons.add(new
> Person("985488b0fd3ae","Bugs","Bunny",300000f,"Wabbit Land ", null, set));
> cache.put("group1", value) // Write collection to cache
>
> List<Person> persons = (List<Person>) cache.get("group1"); // Get
> from cache, Exception here {code}
>
> *Exception:*
> {code:java}
> class org.apache.ignite.binary.BinaryObjectException: Failed to deserialize
> object [typeName=com.ignite.example.model.Person]
> at
> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:927)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
> at
> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:316)
> at
> org.apache.ignite.internal.client.thin.ClientBinaryMarshaller.deserialize(ClientBinaryMarshaller.java:74)
> at
> org.apache.ignite.internal.client.thin.ClientUtils.unwrapBinary(ClientUtils.java:557)
> at
> org.apache.ignite.internal.client.thin.ClientUtils.unwrapCollection(ClientUtils.java:578)
> at
> org.apache.ignite.internal.client.thin.ClientUtils.unwrapBinary(ClientUtils.java:562)
> at
> org.apache.ignite.internal.client.thin.ClientUtils.readObject(ClientUtils.java:546)
> at
> org.apache.ignite.internal.client.thin.TcpClientCache.readObject(TcpClientCache.java:556)
> at
> org.apache.ignite.internal.client.thin.TcpClientCache.readObject(TcpClientCache.java:561)
> at
> org.apache.ignite.internal.client.thin.TcpClientCache$$Lambda$395/1950117092.apply(Unknown
> Source)
> at
> org.apache.ignite.internal.client.thin.TcpClientChannel.receive(TcpClientChannel.java:284)
> at
> org.apache.ignite.internal.client.thin.TcpClientChannel.service(TcpClientChannel.java:219)
> at
> org.apache.ignite.internal.client.thin.ReliableChannel.service(ReliableChannel.java:198)
> at
> org.apache.ignite.internal.client.thin.ReliableChannel.affinityService(ReliableChannel.java:261)
> at
> org.apache.ignite.internal.client.thin.TcpClientCache.cacheSingleKeyOperation(TcpClientCache.java:508)
> at
> org.apache.ignite.internal.client.thin.TcpClientCache.get(TcpClientCache.java:111)
> at
> com.ignite.example.service.ApacheIgniteService.printAllKeys(ApacheIgniteService.java:117)
> at
> com.ignite.example.service.ApacheIgniteService.init(ApacheIgniteService.java:103)
> at
> com.ignite.example.IgniteCacheExampleApplication.run(IgniteCacheExampleApplication.java:22)
> at
> org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:768)
> at
> org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:752)
> at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:314)
> at
> com.ignite.example.IgniteCacheExampleApplication.main(IgniteCacheExampleApplication.java:17)
> Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to
> read field [name=accounts]
> at
> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
> at
> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:888)
> ... 24 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -430
> at
> org.apache.ignite.internal.binary.streams.BinaryHeapInputStream.readByteAndShift(BinaryHeapInputStream.java:120)
> at
> org.apache.ignite.internal.binary.streams.BinaryAbstractInputStream.readByte(BinaryAbstractInputStream.java:37)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.<init>(BinaryReaderExImpl.java:219)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.<init>(BinaryReaderExImpl.java:186)
> at
> org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1801)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1748)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1978)
> at
> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
> at
> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
> {code}
> *Analysis*
> The exception occurs due to a OutOfBounds read happening in
> obj = BinaryUtils.doReadObject(in, ctx, ldr, this);
> This is because *handlePos* points to a location in the ByteBuffer which is
> outside that of the Buffer in the current object resulting in a negative
> offset (java.lang.ArrayIndexOutOfBoundsException: {*}-430{*})
> {{!image001.png!}}
> The serialization works and the Byte array response returned from the server
> is the same as that was written. It's the way HANDLE types within a
> Collection are read that's probably causing this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
