Hi Alexander,
Would new implementation of GridLongList be able to read value serialized
by old implementation?
Is it possible old implementation would not be able to read value from new?
Sincerely,
Dmitriy Pavlov
пн, 7 мая 2018 г. в 11:30, Александр Меньшиков <sharple...@gmail.com>:
> Hi.
> I have finished this task. Just replaced
>
> `writer.writeLongArray("arr", arr)`
> with
> `writer.writeLongArray("arr", arr, idx)`
>
> Please review and merge if okay.
>
> Jira: https://issues.apache.org/jira/browse/IGNITE-8054
> PR: https://github.com/apache/ignite/pull/3748
> CI:
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_RunAll&branch_IgniteTests24Java8=pull%2F3748%2Fhead&tab=buildTypeStatusDiv
>
> 2018-03-27 11:50 GMT+03:00 Александр Меньшиков <sharple...@gmail.com>:
>
>> The ticket is created:
>> https://issues.apache.org/jira/browse/IGNITE-8054
>>
>> 2018-03-27 10:04 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>:
>>
>>> Sorry, Dmitiry, I meant Ignite, not GridGain (memories of pre-apache
>>> coding). I am assuming that Alex Menshikov was referring to GridLongList
>>> class in Ignite.
>>>
>>> D.
>>>
>>> On Mon, Mar 26, 2018 at 11:52 PM, Dmitry Pavlov <dpavlov....@gmail.com>
>>> wrote:
>>>
>>> > Hi Dmitriy, did you mean GridList?
>>> >
>>> > I don't understand what does it mean GridGain compress.
>>> >
>>> > вт, 27 мар. 2018 г. в 3:06, Dmitriy Setrakyan <dsetrak...@apache.org>:
>>> >
>>> > > Thanks, Alex.
>>> > >
>>> > > GridGain automatically compresses all the internal types. Somehow it
>>> > looks
>>> > > like the GridLongList may have been mixed. Can you please file a
>>> ticket
>>> > for
>>> > > 2.5 release?
>>> > >
>>> > > D.
>>> > >
>>> > > On Mon, Mar 26, 2018 at 4:55 AM, Александр Меньшиков <
>>> > sharple...@gmail.com
>>> > > >
>>> > > wrote:
>>> > >
>>> > > > I investigated network loading and found that a big part of
>>> internal
>>> > data
>>> > > > inside messages is `GridLongList`.
>>> > > > It is a part of `GridDhtTxFinishRequest`,
>>> > > > `GridDhtAtomicDeferredUpdateResponse`,
>>> `GridDhtAtomicUpdateRequest`,
>>> > > > `GridNearAtomicFullUpdateRequest` and `NearCacheUpdates`.
>>> > > >
>>> > > > So I think it has the sense to optimize `GridLongList`
>>> serialization.
>>> > > >
>>> > > >
>>> > > > Here we serialize all elements and don't take into account `idx`
>>> value:
>>> > > >
>>> > > > ```
>>> > > >
>>> > > > @Override public boolean writeTo(ByteBuffer buf, MessageWriter
>>> writer)
>>> > {
>>> > > >
>>> > > > writer.setBuffer(buf);
>>> > > >
>>> > > >
>>> > > >
>>> > > > if (!writer.isHeaderWritten()) {
>>> > > >
>>> > > > if (!writer.writeHeader(directType(), fieldsCount()))
>>> > > >
>>> > > > return false;
>>> > > >
>>> > > >
>>> > > >
>>> > > > writer.onHeaderWritten();
>>> > > >
>>> > > > }
>>> > > >
>>> > > >
>>> > > >
>>> > > > switch (writer.state()) {
>>> > > >
>>> > > > case 0:
>>> > > >
>>> > > > if (!writer.writeLongArray("arr", arr))
>>> > > >
>>> > > > return false;
>>> > > >
>>> > > >
>>> > > >
>>> > > > writer.incrementState();
>>> > > >
>>> > > >
>>> > > >
>>> > > > case 1:
>>> > > >
>>> > > > if (!writer.writeInt("idx", idx))
>>> > > >
>>> > > > return false;
>>> > > >
>>> > > >
>>> > > >
>>> > > > writer.incrementState();
>>> > > >
>>> > > >
>>> > > >
>>> > > > }
>>> > > >
>>> > > >
>>> > > >
>>> > > > return true;
>>> > > >
>>> > > > }
>>> > > >
>>> > > > ```
>>> > > >
>>> > > >
>>> > > >
>>> > > > Which is not happening in another serialization method in the same
>>> > class:
>>> > > >
>>> > > >
>>> > > >
>>> > > > ```
>>> > > >
>>> > > > public static void writeTo(DataOutput out, @Nullable GridLongList
>>> list)
>>> > > > throws IOException {
>>> > > >
>>> > > > out.writeInt(list != null ? list.idx : -1);
>>> > > >
>>> > > >
>>> > > >
>>> > > > if (list != null) {
>>> > > >
>>> > > > for (int i = 0; i < list.idx; i++)
>>> > > >
>>> > > > out.writeLong(list.arr[i]);
>>> > > >
>>> > > > }
>>> > > >
>>> > > > }
>>> > > >
>>> > > > ```
>>> > > >
>>> > > >
>>> > > > So, we can simply reduce messages size by sending only a valuable
>>> part
>>> > of
>>> > > > the array.
>>> > > > If you don't mind I will create an issue in Jira for this.
>>> > > >
>>> > > >
>>> > > > By the way, `long` is a huge type. As I see in most cases
>>> > `GridLongList`
>>> > > > uses for counters.
>>> > > > And I have checked the possibility of compress `long` into smaller
>>> > types
>>> > > as
>>> > > > `int`, `short` or `byte` in test
>>> > > > `GridCacheInterceptorAtomicRebalanceTest` (took it by random).
>>> > > > And found out that all `long` in`GridLongList` can be cast to
>>> `int` and
>>> > > 70%
>>> > > > of them to shorts.
>>> > > > Such conversion is quite fast about 1.1 (ns) per element (I have
>>> > checked
>>> > > it
>>> > > > by JMH test).
>>> > > >
>>> > > >
>>> > > >
>>> > > > Of course, there are a lot of ways to compress data,
>>> > > > but I know proprietary GridGain plug-in has different
>>> `MessageWriter`
>>> > > > implementation.
>>> > > > So maybe it is unnecessary and some compression already exists in
>>> this
>>> > > > proprietary plug-in.
>>> > > > Does someone know something about it?
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>
