Hi Dmitry,
It's 100% compatible. I just change one line for serializing only valuable
part of the array
-- just use another method of a writer.
I think it's enough for now. We can return to a discussion about compact
GridLongList
in future when the 3.0 release will start coming.
2018-05-08 19:33 GMT+03:00 Dmitry Pavlov <dpavlov....@gmail.com>:
> Hi Alexander,
>
> Would new implementation of GridLongList be able to read value serialized
> by old implementation?
>
> Is it possible old implementation would not be able to read value from new?
>
> Sincerely,
> Dmitriy Pavlov
>
> пн, 7 мая 2018 г. в 11:30, Александр Меньшиков <sharple...@gmail.com>:
>
> > Hi.
> > I have finished this task. Just replaced
> >
> > `writer.writeLongArray("arr", arr)`
> > with
> > `writer.writeLongArray("arr", arr, idx)`
> >
> > Please review and merge if okay.
> >
> > Jira: https://issues.apache.org/jira/browse/IGNITE-8054
> > PR: https://github.com/apache/ignite/pull/3748
> > CI:
> > https://ci.ignite.apache.org/viewType.html?buildTypeId=
> IgniteTests24Java8_RunAll&branch_IgniteTests24Java8=
> pull%2F3748%2Fhead&tab=buildTypeStatusDiv
> >
> > 2018-03-27 11:50 GMT+03:00 Александр Меньшиков <sharple...@gmail.com>:
> >
> >> The ticket is created:
> >> https://issues.apache.org/jira/browse/IGNITE-8054
> >>
> >> 2018-03-27 10:04 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>:
> >>
> >>> Sorry, Dmitiry, I meant Ignite, not GridGain (memories of pre-apache
> >>> coding). I am assuming that Alex Menshikov was referring to
> GridLongList
> >>> class in Ignite.
> >>>
> >>> D.
> >>>
> >>> On Mon, Mar 26, 2018 at 11:52 PM, Dmitry Pavlov <dpavlov....@gmail.com
> >
> >>> wrote:
> >>>
> >>> > Hi Dmitriy, did you mean GridList?
> >>> >
> >>> > I don't understand what does it mean GridGain compress.
> >>> >
> >>> > вт, 27 мар. 2018 г. в 3:06, Dmitriy Setrakyan <dsetrak...@apache.org
> >:
> >>> >
> >>> > > Thanks, Alex.
> >>> > >
> >>> > > GridGain automatically compresses all the internal types. Somehow
> it
> >>> > looks
> >>> > > like the GridLongList may have been mixed. Can you please file a
> >>> ticket
> >>> > for
> >>> > > 2.5 release?
> >>> > >
> >>> > > D.
> >>> > >
> >>> > > On Mon, Mar 26, 2018 at 4:55 AM, Александр Меньшиков <
> >>> > sharple...@gmail.com
> >>> > > >
> >>> > > wrote:
> >>> > >
> >>> > > > I investigated network loading and found that a big part of
> >>> internal
> >>> > data
> >>> > > > inside messages is `GridLongList`.
> >>> > > > It is a part of `GridDhtTxFinishRequest`,
> >>> > > > `GridDhtAtomicDeferredUpdateResponse`,
> >>> `GridDhtAtomicUpdateRequest`,
> >>> > > > `GridNearAtomicFullUpdateRequest` and `NearCacheUpdates`.
> >>> > > >
> >>> > > > So I think it has the sense to optimize `GridLongList`
> >>> serialization.
> >>> > > >
> >>> > > >
> >>> > > > Here we serialize all elements and don't take into account `idx`
> >>> value:
> >>> > > >
> >>> > > > ```
> >>> > > >
> >>> > > > @Override public boolean writeTo(ByteBuffer buf, MessageWriter
> >>> writer)
> >>> > {
> >>> > > >
> >>> > > > writer.setBuffer(buf);
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > if (!writer.isHeaderWritten()) {
> >>> > > >
> >>> > > > if (!writer.writeHeader(directType(),
> fieldsCount()))
> >>> > > >
> >>> > > > return false;
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > writer.onHeaderWritten();
> >>> > > >
> >>> > > > }
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > switch (writer.state()) {
> >>> > > >
> >>> > > > case 0:
> >>> > > >
> >>> > > > if (!writer.writeLongArray("arr", arr))
> >>> > > >
> >>> > > > return false;
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > writer.incrementState();
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > case 1:
> >>> > > >
> >>> > > > if (!writer.writeInt("idx", idx))
> >>> > > >
> >>> > > > return false;
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > writer.incrementState();
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > }
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > return true;
> >>> > > >
> >>> > > > }
> >>> > > >
> >>> > > > ```
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > Which is not happening in another serialization method in the
> same
> >>> > class:
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > ```
> >>> > > >
> >>> > > > public static void writeTo(DataOutput out, @Nullable GridLongList
> >>> list)
> >>> > > > throws IOException {
> >>> > > >
> >>> > > > out.writeInt(list != null ? list.idx : -1);
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > if (list != null) {
> >>> > > >
> >>> > > > for (int i = 0; i < list.idx; i++)
> >>> > > >
> >>> > > > out.writeLong(list.arr[i]);
> >>> > > >
> >>> > > > }
> >>> > > >
> >>> > > > }
> >>> > > >
> >>> > > > ```
> >>> > > >
> >>> > > >
> >>> > > > So, we can simply reduce messages size by sending only a valuable
> >>> part
> >>> > of
> >>> > > > the array.
> >>> > > > If you don't mind I will create an issue in Jira for this.
> >>> > > >
> >>> > > >
> >>> > > > By the way, `long` is a huge type. As I see in most cases
> >>> > `GridLongList`
> >>> > > > uses for counters.
> >>> > > > And I have checked the possibility of compress `long` into
> smaller
> >>> > types
> >>> > > as
> >>> > > > `int`, `short` or `byte` in test
> >>> > > > `GridCacheInterceptorAtomicRebalanceTest` (took it by random).
> >>> > > > And found out that all `long` in`GridLongList` can be cast to
> >>> `int` and
> >>> > > 70%
> >>> > > > of them to shorts.
> >>> > > > Such conversion is quite fast about 1.1 (ns) per element (I have
> >>> > checked
> >>> > > it
> >>> > > > by JMH test).
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > Of course, there are a lot of ways to compress data,
> >>> > > > but I know proprietary GridGain plug-in has different
> >>> `MessageWriter`
> >>> > > > implementation.
> >>> > > > So maybe it is unnecessary and some compression already exists in
> >>> this
> >>> > > > proprietary plug-in.
> >>> > > > Does someone know something about it?
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
>
