Vladimir, Could you please elaborate as to why adding ByteBuffer to the Binary protocol is “not a good idea”? What is inherently wrong about it?
As for your suggestion, while I certainly see your point, it comes with the inconvenience of implementing Binarylizable for every entity type in an application, which makes me wonder if the self imposed rules such as “type system should be kept as small as possible” could be relaxed in the name of greater good and user friendliness. Ignite API is not easy to grok as it is already. Thanks Andrey _____________________________ From: Vladimir Ozerov <voze...@gridgain.com<mailto:voze...@gridgain.com>> Sent: Monday, November 6, 2017 10:22 PM Subject: Re: Deserialization of BinaryObject's byte arrays To: <dev@ignite.apache.org<mailto:dev@ignite.apache.org>> Cc: Valentin Kulichenko <valentin.kuliche...@gmail.com<mailto:valentin.kuliche...@gmail.com>> Hi Andrey, While we deifnitely need to support byte buffers, adding new type to protocol is not very good idea. Binary protocol is the heart of our ssytem, which is used not oly for plain Java, but for other platforms and SQL. And ByteBuffer is essentially the same byte array, but with different API. What you describe is a special case of byte array read/write. For this reason our type system should be kept as small as possible and it doesn't require new type to be introduced. Instead, we'd better to handle everything on API level only. For example, we can add new methods to reader/writer/builder, which will handle this. E.g.: BinaryWriter.writeByteArray(<some producer here>); BinaryReader.readByteArray(<some consumer here>); Vladimir. On Mon, Nov 6, 2017 at 9:25 PM, Andrey Kornev <andrewkor...@hotmail.com<mailto:andrewkor...@hotmail.com>> wrote: > Will do! Thanks! > ________________________________ > From: Valentin Kulichenko > <valentin.kuliche...@gmail.com<mailto:valentin.kuliche...@gmail.com>> > Sent: Monday, November 6, 2017 9:17 AM > To: dev@ignite.apache.org<mailto:dev@ignite.apache.org> > Subject: Re: Deserialization of BinaryObject's byte arrays > > This use case was discussed couple of times already, so it seems to be > pretty important for users. And I like the approach suggested by Andrey. > > Andrey, can you create a Jira ticket for this change? Would you like to > contribute it? > > -Val > > On Fri, Nov 3, 2017 at 4:18 PM, Dmitriy Setrakyan > <dsetrak...@apache.org<mailto:dsetrak...@apache.org>> > wrote: > > > This makes sense to me. Sounds like a useful feature. > > > > Would be nice to hear what others in the community think? > > > > D. > > > > On Fri, Nov 3, 2017 at 1:07 PM, Andrey Kornev > > <andrewkor...@hotmail.com<mailto:andrewkor...@hotmail.com>> > > wrote: > > > > > Hello, > > > > > > We store lots of byte arrays (serialized graph-like data structures) as > > > fields of BinaryObject. Later on, while reading such field, > > > BinaryInputStream implementation creates an on-heap array and copies > the > > > bytes from the BinaryObject's internal backing array to the new array. > > > > > > While in general case it works just fine, in our case, a lot of CPU is > > > spent on copying of the bytes and significant amount garbage is > generated > > > as well. > > > > > > I'd like Ignite Community to consider the following proposal: introduce > > > support for serialization/deserialization of the ByteBuffer class. A > > > BinaryInputStream implementation would special case ByteBuffer fields > > same > > > way it currently special cases various Java types (collections, dates, > > > timestamps, enums, etc.) > > > > > > The application developer would simply call > > BinaryObjectBuilder.setField(...) > > > passing in an instance of ByteBuffer. During serialization, the > > > ByteBuffer's bytes would simply be copied to the backing array (similar > > to > > > how regular byte arrays are serialized today) and in the binary > object's > > > metadata the field marked as, say, GridBinaryMarshaller.BYTE_BUFFER. > > > > > > During deserialization of the field, instead of allocating a new byte > > > array and transferring the bytes from the BinaryObject's backing array, > > > BinaryInputStream would simply wrap the required sub-array into a > > read-only > > > instance of ByteBuffer using "public static ByteBuffer wrap(byte[] > array, > > > int offset, int length)" and return the instance to the application. > > > > > > This approach doesn't require any array allocation/data copying and is > > > faster and significantly reduces pressure on the garbage collector. The > > > benefits are especially great when application stores significant > amount > > of > > > its data as byte arrays. > > > > > > Clearly, this approach equally applies to arrays of other primitive > > types. > > > > > > A minor complication does arise when dealing with off-heap > BinaryObjects, > > > but nothing that can't solved with a little bit of Java reflection > > wizardry > > > (or by other means). > > > > > > Thanks, > > > Andrey > > > > > >
