Re: Obsolete classnames.properties
Hello! Makes sense calling it during every build, if it can be made reliably. Regards, -- Ilya Kasnacheev вт, 18 мая 2021 г. в 01:51, Данилов Семён : > Hello, Igniters! > > I've recently stumbled across an issue where some tests can't be run > locally due to the absence of some entries in "classnames.properties" file. > For example, TcpRestUnmarshalVulnerabilityTest can't be run locally > because "classnames.properties" > lacks org.apache.ignite.configuration.EncryptionConfiguration > and org.apache.ignite.configuration.PageReplacementMode. On teamcity, > however, such tests pass without any problem, because > "classnames.properties" is also generated > by org.apache.ignite.tools.classgen.ClassesGenerator during process-classes > phase in maven build. > > I propose removing this file, as it is generated during the build which > means it's a pointless job keeping it up-to-date in the repository. Also > it's impossible to tell right now if it is or it is not up-to-date (without > executing ClassesGenerator). > > Kind regards, Semyon. >
Re: [DISCUSSION] Array to BinaryObject serialization
Hello,
> However, for internal platform and services implementations we should fix the
> root cause:
> avoid extra deserialization->serialization pass completely.
> This will also improve performance.
Pavel, thanks for the feedback.
If I understand correctly, your suggestion is to know data size at the start of
reading service parameters.
Is it correct?
Right now, when the service method invoked we pass an array of parameters
through platform reader/writer machinery.
On java side parameters read one by one and we don’t know the size of the data
on the read start.
AFAICU, this will require x2 memory on the .Net or thin client-side.
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/services/PlatformServices.java#L289
> if we are to break compatibility, I would like to see it done for some really
> common pain point.
Ilya, can you, please, provide a list of common issues with Ignite that can be
resolved
only with compatibility breakage?
> 4 мая 2021 г., в 12:58, Ilya Kasnacheev
> написал(а):
>
> Hello!
>
> If we really decide to break some compatibility then Array to BinaryObject
> serialization will be very, very low on my personal list.
>
> I just don't see how this issue is relevant. I have been reading and
> answering user list for a few years now, and I don't remember a single
> question about storing ConcreteType[] as value and complaining about type
> information loss.
>
> If you have a good scenario how do we keep persistent store binary
> compatibility here, without adding a lot of new code and still checking for
> both old and new approaches - you can go forward for sure.
>
> However, it does seem questionable where we have a new wrapper class
> specifically for top level arrays. You can have this wrapper in your own
> client code and it should work OK.
>
> Bottom line, if we are to break compatibility, I would like to see it done
> for some really common pain point.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 30 апр. 2021 г. в 17:34, Nikolay Izhikov :
>
>> Hello, Ilya.
>>
>> Thanks for the feedback!
>>
>>> For me it sounds like something we would like to do in 3.0
>>
>> Ignite 3 is a very long way to go, so I prefer to target this fix in
>> Ignite 2.x.
>>
>>> I think making it default "true" is a breaking change and is not
>> possible in a minor release
>>
>> Yes, you are correct it is a breaking change.
>> It seems for me, we all agreed that breaking changes are possible in minor
>> releases.
>>
>> Anyway, if we will decide do not to enable this feature by default it’s OK
>> for me.
>> We still can implement it and improve the binary SerDe mechanism.
>>
>>
>>> 30 апр. 2021 г., в 17:23, Ilya Kasnacheev
>> написал(а):
>>>
>>> Hello!
>>>
>>> For me it sounds like something we would like to do in 3.0 (if indeed it
>>> will have arrays as possible value (or key) type), but doing it in 2.x
>>> raises concerns whether it has enough time left to stabilize.
>>>
>>> Also, I think making it default "true" is a breaking change and is not
>>> possible in a minor release, case in point,
>>> IGNITE_BINARY_SORT_OBJECT_FIELDS is still waiting for 3.0 and it is less
>>> destructive.
>>>
>>> Of course I would also like to hear what other community members think.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> пт, 30 апр. 2021 г. в 17:16, Nikolay Izhikov :
>>>
Igniters,
Want to clarify my proposal about new array store format.
I think we should store array in special binary wrapper that will keep
original component type
```
public class BinaryArrayWrapper implements BinaryObjectEx,
>> Externalizable {
/** Type ID. */
private int compTypeId;
/** Raw data. */
private String compClsName;
/** Value. */
private Object[] arr;
// Further implementation.
}
```
> 30 апр. 2021 г., в 16:31, Nikolay Izhikov
написал(а):
>
> Hello, Igniters.
>
> Currently, binary marshaller works as follows(Say, we have a class
`User` then):
>
> IgniteBinary#toBinary(User)` -> BinaryObject
> IgniteBinary#toBinary(User[])` -> Object[]
> IgniteBinary#toBinary(Object[])` -> Object[]
>
> This means, that we lose array component type information during binary
serialization.
> AFAIK, it’s a design choice made during binary infrastructure
development.
>
> This lead to the following disadvantages:
>
> 1. `IgniteBinary` can’t be used as a universal SerDe mechanism.
> 2. Ignite internals(service grid, .Net calls) contains many tweaks and
hacks to deal with custom user array and still has many issues [1]
>
> I propose to make breaking changes and fix the custom user array SeDe
>> as
follows:
>
> 1. Implement binary serialization that correctly Ser and Deser
array using some kin
Re: [VOTE][EXTENSION] Release Apache Ignite spring-data-all-ext extensions 1.0.0 RC3
Hello!
Do I understand correctly that voting did not succeed due to insufficient
participation? Can you post the vote results?
I have checked examples and it indeed works.
I have a list of things to do before the RC4:
Please copy over examples directory and pom.xml to spring module 2.2.
There's no reason why spring data 2.0 should have examples but spring data
2.2 should not.
Release a single zip file called spring-data-all-ext-1.0.0-src.zip with the
zipped source. No reason to post the same source 4 times if we are
releasing them all together.
Put all files in archive in spring-data-all-ext-1.0.0-src/ directory. Top
level in zip archive should have just 1 directory.
README.md should not be a Readme of Apache Ignite. Better keep it short and
write something about what's Apache Ignite Extensions and where to get them.
Remove scripts/ directory and .gitignore.
Make mvn -Pcheckstyle,check-licenses validate pass - currently it fails.
I promise to help with release voting agitation if these points are met.
They will also increase the quality of release greatly.
I can file a ticket with these steps if you like, or you may track it
yourself.
Regards,
--
Ilya Kasnacheev
пн, 17 мая 2021 г. в 18:07, Mikhail Petrov :
> +1
>
> I created separate Spring Applications for each version of Spring Data
> Integration using the RC repository and following the documentation. And
> tested common use cases.
>
> On 14.05.2021 15:39, Nikita Amelchev wrote:
> >> How do I actually run these examples?
> > Use the 'examples' maven profile, please.
> >
> >> I don't understand why we are releasing four source ZIPs if their
> contents are identical? Can we release a single one, name it
> spring-data-all-ext-1.0.0-src.zip?
> > Modules are independent. For example, we can release spring-data-ext
> > v1.1 without releasing others. I think each module should have its
> > independent sources.
> >
> >> Can we find a way to drop all non-relevant modules?
> > I think we can use TC to prepare the build as well as the Ignite
> > release process do it. I suggest using it in future releases when such
> > a build appears.
> >
> > пт, 14 мая 2021 г. в 15:02, Ilya Kasnacheev :
> >> Hello!
> >>
> >> How do I actually run these examples? These do not seem to have a
> pom.xml
> >> of their own now.
> >>
> >> I don't understand why we are releasing four source ZIPs if their
> contents
> >> are identical? Can we release a single one, name it
> >> spring-data-all-ext-1.0.0-src.zip? Can we find a way to drop all
> >> non-relevant modules?
> >>
> >> All of the files need to be in one directory, such
> >> as spring-data-all-ext-1.0.0-src/modules, etc, but they are located in
> >> archive root, which differs from how we do it with Apache Ignite src
> zip.
> >>
> >> Source release should differ from git repo dump: it does not need
> >> .gitignore or scripts/ directory since these are of no use for source
> >> distribution user.
> >>
> >> Regards,
> >> --
> >> Ilya Kasnacheev
> >>
> >>
> >> пт, 14 мая 2021 г. в 14:32, Nikita Amelchev :
> >>
> >>> Hello, Ilya.
> >>>
> Is there any way to find and run these examples with spring-data ext
> >>> 1.0.0?
> >>>
> >>> Spring data modules were migrated to the extensions repo with their
> >>> examples. Examples for 2.0 still available in the package:
> >>>
> >>>
> >>>
> modules/spring-data-2.0-ext/examples/main/java/org/apache/ignite/springdata20/examples/
> >>>
> >>> Other versions were not covered with examples.
> >>>
> I have also found that in spring-data 2.2 source archive at least,
> >>> there's full git repo of ignite-ext including .gitignore, committer
> >>> scripts, sources for all modules such as flink, but no spring data
> examples.
> >>>
> >>> The source package is a full copy of the extensions repo as the build
> >>> process requires the parent pom. Build instruction is mentioned in the
> >>> DEVNOTES file.
> >>>
> >>> пт, 14 мая 2021 г. в 13:53, Ilya Kasnacheev >:
> Hello!
>
> We had Spring Data examples in 2.9 in the ZIP file:
> examples/config/spring/example-spring-data.xml:
> Ignition.start("examples/config/spring/example-spring-data.xml");
>
> >>>
> examples/src/main/java/org/apache/ignite/examples/springdata/SpringApplicationConfiguration.java
> :return
> Ignition.start("examples/config/spring/example-spring-data.xml");
>
> examples/pom.xml:
> >>> ignite-spring-data_2.0
> Is there any way to find and run these examples with spring-data ext
> >>> 1.0.0?
> I have also found that in spring-data 2.2 source archive at least,
> >>> there's
> full git repo of ignite-ext including .gitignore, committer scripts,
> sources for all modules such as flink, but no spring data examples.
>
> Please clarify.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 12 мая 2021 г. в 18:28, Nikita Amelchev :
>
> > Dear Ignite Community,
> >
> > I have uploaded a release cand
Re: [DISCUSSION] Array to BinaryObject serialization
Hello!
Obvious issues are Lazy SQL, Event Driven Services, Sort Binary Object
Fields.
Regards,
--
Ilya Kasnacheev
ср, 19 мая 2021 г. в 12:56, Nikolay Izhikov :
> Hello,
>
> > However, for internal platform and services implementations we should
> fix the root cause:
> > avoid extra deserialization->serialization pass completely.
> > This will also improve performance.
>
> Pavel, thanks for the feedback.
> If I understand correctly, your suggestion is to know data size at the
> start of reading service parameters.
> Is it correct?
>
> Right now, when the service method invoked we pass an array of parameters
> through platform reader/writer machinery.
> On java side parameters read one by one and we don’t know the size of the
> data on the read start.
>
> AFAICU, this will require x2 memory on the .Net or thin client-side.
>
>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/services/PlatformServices.java#L289
>
>
> > if we are to break compatibility, I would like to see it done for some
> really common pain point.
>
> Ilya, can you, please, provide a list of common issues with Ignite that
> can be resolved
> only with compatibility breakage?
>
> > 4 мая 2021 г., в 12:58, Ilya Kasnacheev
> написал(а):
> >
> > Hello!
> >
> > If we really decide to break some compatibility then Array to
> BinaryObject
> > serialization will be very, very low on my personal list.
> >
> > I just don't see how this issue is relevant. I have been reading and
> > answering user list for a few years now, and I don't remember a single
> > question about storing ConcreteType[] as value and complaining about type
> > information loss.
> >
> > If you have a good scenario how do we keep persistent store binary
> > compatibility here, without adding a lot of new code and still checking
> for
> > both old and new approaches - you can go forward for sure.
> >
> > However, it does seem questionable where we have a new wrapper class
> > specifically for top level arrays. You can have this wrapper in your own
> > client code and it should work OK.
> >
> > Bottom line, if we are to break compatibility, I would like to see it
> done
> > for some really common pain point.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пт, 30 апр. 2021 г. в 17:34, Nikolay Izhikov :
> >
> >> Hello, Ilya.
> >>
> >> Thanks for the feedback!
> >>
> >>> For me it sounds like something we would like to do in 3.0
> >>
> >> Ignite 3 is a very long way to go, so I prefer to target this fix in
> >> Ignite 2.x.
> >>
> >>> I think making it default "true" is a breaking change and is not
> >> possible in a minor release
> >>
> >> Yes, you are correct it is a breaking change.
> >> It seems for me, we all agreed that breaking changes are possible in
> minor
> >> releases.
> >>
> >> Anyway, if we will decide do not to enable this feature by default it’s
> OK
> >> for me.
> >> We still can implement it and improve the binary SerDe mechanism.
> >>
> >>
> >>> 30 апр. 2021 г., в 17:23, Ilya Kasnacheev
> >> написал(а):
> >>>
> >>> Hello!
> >>>
> >>> For me it sounds like something we would like to do in 3.0 (if indeed
> it
> >>> will have arrays as possible value (or key) type), but doing it in 2.x
> >>> raises concerns whether it has enough time left to stabilize.
> >>>
> >>> Also, I think making it default "true" is a breaking change and is not
> >>> possible in a minor release, case in point,
> >>> IGNITE_BINARY_SORT_OBJECT_FIELDS is still waiting for 3.0 and it is
> less
> >>> destructive.
> >>>
> >>> Of course I would also like to hear what other community members think.
> >>>
> >>> Regards,
> >>> --
> >>> Ilya Kasnacheev
> >>>
> >>>
> >>> пт, 30 апр. 2021 г. в 17:16, Nikolay Izhikov :
> >>>
> Igniters,
>
> Want to clarify my proposal about new array store format.
> I think we should store array in special binary wrapper that will keep
> original component type
>
> ```
> public class BinaryArrayWrapper implements BinaryObjectEx,
> >> Externalizable {
> /** Type ID. */
> private int compTypeId;
>
> /** Raw data. */
> private String compClsName;
>
> /** Value. */
> private Object[] arr;
>
> // Further implementation.
> }
> ```
>
>
> > 30 апр. 2021 г., в 16:31, Nikolay Izhikov
> написал(а):
> >
> > Hello, Igniters.
> >
> > Currently, binary marshaller works as follows(Say, we have a class
> `User` then):
> >
> > IgniteBinary#toBinary(User)` -> BinaryObject
> > IgniteBinary#toBinary(User[])` -> Object[]
> > IgniteBinary#toBinary(Object[])` -> Object[]
> >
> > This means, that we lose array component type information during
> binary
> serialization.
> > AFAIK, it’s a design choice made during binary infrastructure
> development.
> >
> > This lead to the following disadvantages:
> >
> > 1. `Ig
Re: [DISCUSSION] Array to BinaryObject serialization
Thanks, Ilya.
Can you put more context on this?
I don’t familiar with these issues.
> 19 мая 2021 г., в 13:02, Ilya Kasnacheev
> написал(а):
>
> Hello!
>
> Obvious issues are Lazy SQL, Event Driven Services, Sort Binary Object
> Fields.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 19 мая 2021 г. в 12:56, Nikolay Izhikov :
>
>> Hello,
>>
>>> However, for internal platform and services implementations we should
>> fix the root cause:
>>> avoid extra deserialization->serialization pass completely.
>>> This will also improve performance.
>>
>> Pavel, thanks for the feedback.
>> If I understand correctly, your suggestion is to know data size at the
>> start of reading service parameters.
>> Is it correct?
>>
>> Right now, when the service method invoked we pass an array of parameters
>> through platform reader/writer machinery.
>> On java side parameters read one by one and we don’t know the size of the
>> data on the read start.
>>
>> AFAICU, this will require x2 memory on the .Net or thin client-side.
>>
>>
>> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/services/PlatformServices.java#L289
>>
>>
>>> if we are to break compatibility, I would like to see it done for some
>> really common pain point.
>>
>> Ilya, can you, please, provide a list of common issues with Ignite that
>> can be resolved
>> only with compatibility breakage?
>>
>>> 4 мая 2021 г., в 12:58, Ilya Kasnacheev
>> написал(а):
>>>
>>> Hello!
>>>
>>> If we really decide to break some compatibility then Array to
>> BinaryObject
>>> serialization will be very, very low on my personal list.
>>>
>>> I just don't see how this issue is relevant. I have been reading and
>>> answering user list for a few years now, and I don't remember a single
>>> question about storing ConcreteType[] as value and complaining about type
>>> information loss.
>>>
>>> If you have a good scenario how do we keep persistent store binary
>>> compatibility here, without adding a lot of new code and still checking
>> for
>>> both old and new approaches - you can go forward for sure.
>>>
>>> However, it does seem questionable where we have a new wrapper class
>>> specifically for top level arrays. You can have this wrapper in your own
>>> client code and it should work OK.
>>>
>>> Bottom line, if we are to break compatibility, I would like to see it
>> done
>>> for some really common pain point.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> пт, 30 апр. 2021 г. в 17:34, Nikolay Izhikov :
>>>
Hello, Ilya.
Thanks for the feedback!
> For me it sounds like something we would like to do in 3.0
Ignite 3 is a very long way to go, so I prefer to target this fix in
Ignite 2.x.
> I think making it default "true" is a breaking change and is not
possible in a minor release
Yes, you are correct it is a breaking change.
It seems for me, we all agreed that breaking changes are possible in
>> minor
releases.
Anyway, if we will decide do not to enable this feature by default it’s
>> OK
for me.
We still can implement it and improve the binary SerDe mechanism.
> 30 апр. 2021 г., в 17:23, Ilya Kasnacheev
написал(а):
>
> Hello!
>
> For me it sounds like something we would like to do in 3.0 (if indeed
>> it
> will have arrays as possible value (or key) type), but doing it in 2.x
> raises concerns whether it has enough time left to stabilize.
>
> Also, I think making it default "true" is a breaking change and is not
> possible in a minor release, case in point,
> IGNITE_BINARY_SORT_OBJECT_FIELDS is still waiting for 3.0 and it is
>> less
> destructive.
>
> Of course I would also like to hear what other community members think.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 30 апр. 2021 г. в 17:16, Nikolay Izhikov :
>
>> Igniters,
>>
>> Want to clarify my proposal about new array store format.
>> I think we should store array in special binary wrapper that will keep
>> original component type
>>
>> ```
>> public class BinaryArrayWrapper implements BinaryObjectEx,
Externalizable {
>> /** Type ID. */
>> private int compTypeId;
>>
>> /** Raw data. */
>> private String compClsName;
>>
>> /** Value. */
>> private Object[] arr;
>>
>> // Further implementation.
>> }
>> ```
>>
>>
>>> 30 апр. 2021 г., в 16:31, Nikolay Izhikov
>> написал(а):
>>>
>>> Hello, Igniters.
>>>
>>> Currently, binary marshaller works as follows(Say, we have a class
>> `User` then):
>>>
>>> IgniteBinary#toBinary(User)` -> BinaryObject
>>> IgniteBinary#toBinary(User[])` -> Object[]
>>> IgniteBinary#toBinary(Object[])` -> Object[]
>>>
>>> This means, that we
Re: [DISCUSSION] Array to BinaryObject serialization
Igniters.
Just to clarify the issue:
```
public class BinaryObjectTest extends GridCommonAbstractTest {
/** */
@Test
public void testArray() throws Exception {
Ignite ign = startGrid();
IgniteCache cache = ign.createCache("my-cache");
cache.put(1, new TestClass1[] {new TestClass1(), new TestClass1()});
TestClass1[] obj = cache.get(1); //This will fail with
ClassCastException.
assertEquals(TestClass1[].class, obj.getClass());
}
}
```
> 19 мая 2021 г., в 13:04, Nikolay Izhikov написал(а):
>
> Thanks, Ilya.
>
> Can you put more context on this?
> I don’t familiar with these issues.
>
>> 19 мая 2021 г., в 13:02, Ilya Kasnacheev
>> написал(а):
>>
>> Hello!
>>
>> Obvious issues are Lazy SQL, Event Driven Services, Sort Binary Object
>> Fields.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> ср, 19 мая 2021 г. в 12:56, Nikolay Izhikov :
>>
>>> Hello,
>>>
However, for internal platform and services implementations we should
>>> fix the root cause:
avoid extra deserialization->serialization pass completely.
This will also improve performance.
>>>
>>> Pavel, thanks for the feedback.
>>> If I understand correctly, your suggestion is to know data size at the
>>> start of reading service parameters.
>>> Is it correct?
>>>
>>> Right now, when the service method invoked we pass an array of parameters
>>> through platform reader/writer machinery.
>>> On java side parameters read one by one and we don’t know the size of the
>>> data on the read start.
>>>
>>> AFAICU, this will require x2 memory on the .Net or thin client-side.
>>>
>>>
>>> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/services/PlatformServices.java#L289
>>>
>>>
if we are to break compatibility, I would like to see it done for some
>>> really common pain point.
>>>
>>> Ilya, can you, please, provide a list of common issues with Ignite that
>>> can be resolved
>>> only with compatibility breakage?
>>>
4 мая 2021 г., в 12:58, Ilya Kasnacheev
>>> написал(а):
Hello!
If we really decide to break some compatibility then Array to
>>> BinaryObject
serialization will be very, very low on my personal list.
I just don't see how this issue is relevant. I have been reading and
answering user list for a few years now, and I don't remember a single
question about storing ConcreteType[] as value and complaining about type
information loss.
If you have a good scenario how do we keep persistent store binary
compatibility here, without adding a lot of new code and still checking
>>> for
both old and new approaches - you can go forward for sure.
However, it does seem questionable where we have a new wrapper class
specifically for top level arrays. You can have this wrapper in your own
client code and it should work OK.
Bottom line, if we are to break compatibility, I would like to see it
>>> done
for some really common pain point.
Regards,
--
Ilya Kasnacheev
пт, 30 апр. 2021 г. в 17:34, Nikolay Izhikov :
> Hello, Ilya.
>
> Thanks for the feedback!
>
>> For me it sounds like something we would like to do in 3.0
>
> Ignite 3 is a very long way to go, so I prefer to target this fix in
> Ignite 2.x.
>
>> I think making it default "true" is a breaking change and is not
> possible in a minor release
>
> Yes, you are correct it is a breaking change.
> It seems for me, we all agreed that breaking changes are possible in
>>> minor
> releases.
>
> Anyway, if we will decide do not to enable this feature by default it’s
>>> OK
> for me.
> We still can implement it and improve the binary SerDe mechanism.
>
>
>> 30 апр. 2021 г., в 17:23, Ilya Kasnacheev
> написал(а):
>>
>> Hello!
>>
>> For me it sounds like something we would like to do in 3.0 (if indeed
>>> it
>> will have arrays as possible value (or key) type), but doing it in 2.x
>> raises concerns whether it has enough time left to stabilize.
>>
>> Also, I think making it default "true" is a breaking change and is not
>> possible in a minor release, case in point,
>> IGNITE_BINARY_SORT_OBJECT_FIELDS is still waiting for 3.0 and it is
>>> less
>> destructive.
>>
>> Of course I would also like to hear what other community members think.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пт, 30 апр. 2021 г. в 17:16, Nikolay Izhikov :
>>
>>> Igniters,
>>>
>>> Want to clarify my proposal about new array store format.
>>> I think we should store array in special binary wrapper that will keep
>>> original component type
>>>
>>> ```
>>> public class BinaryArrayWrapper implements BinaryObj
Re: [DISCUSSION] Array to BinaryObject serialization
Hello!
Yes, it does not look pretty, I agree. But I'm saying that I've not
encountered this issue on the user list before, and it can probably be
easily countered by storing some one-field ArrayWrapper object in the cache
as value.
Maybe we should just automate that, e.g., whenever user stores Type[],
always store one-field ArrayWrapper object, and automatically unwrap it on
get()
This way we may not even need any changes to storage format, if binary
marshaller does not mind.
Regards,
Regards,
--
Ilya Kasnacheev
ср, 19 мая 2021 г. в 13:33, Nikolay Izhikov :
> Igniters.
>
> Just to clarify the issue:
>
> ```
> public class BinaryObjectTest extends GridCommonAbstractTest {
> /** */
> @Test
> public void testArray() throws Exception {
> Ignite ign = startGrid();
>
> IgniteCache cache =
> ign.createCache("my-cache");
>
> cache.put(1, new TestClass1[] {new TestClass1(), new
> TestClass1()});
> TestClass1[] obj = cache.get(1); //This will fail with
> ClassCastException.
>
> assertEquals(TestClass1[].class, obj.getClass());
> }
> }
> ```
>
> > 19 мая 2021 г., в 13:04, Nikolay Izhikov
> написал(а):
> >
> > Thanks, Ilya.
> >
> > Can you put more context on this?
> > I don’t familiar with these issues.
> >
> >> 19 мая 2021 г., в 13:02, Ilya Kasnacheev
> написал(а):
> >>
> >> Hello!
> >>
> >> Obvious issues are Lazy SQL, Event Driven Services, Sort Binary Object
> >> Fields.
> >>
> >> Regards,
> >> --
> >> Ilya Kasnacheev
> >>
> >>
> >> ср, 19 мая 2021 г. в 12:56, Nikolay Izhikov :
> >>
> >>> Hello,
> >>>
> However, for internal platform and services implementations we should
> >>> fix the root cause:
> avoid extra deserialization->serialization pass completely.
> This will also improve performance.
> >>>
> >>> Pavel, thanks for the feedback.
> >>> If I understand correctly, your suggestion is to know data size at the
> >>> start of reading service parameters.
> >>> Is it correct?
> >>>
> >>> Right now, when the service method invoked we pass an array of
> parameters
> >>> through platform reader/writer machinery.
> >>> On java side parameters read one by one and we don’t know the size of
> the
> >>> data on the read start.
> >>>
> >>> AFAICU, this will require x2 memory on the .Net or thin client-side.
> >>>
> >>>
> >>>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/services/PlatformServices.java#L289
> >>>
> >>>
> if we are to break compatibility, I would like to see it done for some
> >>> really common pain point.
> >>>
> >>> Ilya, can you, please, provide a list of common issues with Ignite that
> >>> can be resolved
> >>> only with compatibility breakage?
> >>>
> 4 мая 2021 г., в 12:58, Ilya Kasnacheev
> >>> написал(а):
>
> Hello!
>
> If we really decide to break some compatibility then Array to
> >>> BinaryObject
> serialization will be very, very low on my personal list.
>
> I just don't see how this issue is relevant. I have been reading and
> answering user list for a few years now, and I don't remember a single
> question about storing ConcreteType[] as value and complaining about
> type
> information loss.
>
> If you have a good scenario how do we keep persistent store binary
> compatibility here, without adding a lot of new code and still
> checking
> >>> for
> both old and new approaches - you can go forward for sure.
>
> However, it does seem questionable where we have a new wrapper class
> specifically for top level arrays. You can have this wrapper in your
> own
> client code and it should work OK.
>
> Bottom line, if we are to break compatibility, I would like to see it
> >>> done
> for some really common pain point.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 30 апр. 2021 г. в 17:34, Nikolay Izhikov :
>
> > Hello, Ilya.
> >
> > Thanks for the feedback!
> >
> >> For me it sounds like something we would like to do in 3.0
> >
> > Ignite 3 is a very long way to go, so I prefer to target this fix in
> > Ignite 2.x.
> >
> >> I think making it default "true" is a breaking change and is not
> > possible in a minor release
> >
> > Yes, you are correct it is a breaking change.
> > It seems for me, we all agreed that breaking changes are possible in
> >>> minor
> > releases.
> >
> > Anyway, if we will decide do not to enable this feature by default
> it’s
> >>> OK
> > for me.
> > We still can implement it and improve the binary SerDe mechanism.
> >
> >
> >> 30 апр. 2021 г., в 17:23, Ilya Kasnacheev <
> ilya.kasnach...@gmail.com>
> > написал(а):
> >>
> >> Hello!
> >>
> >> For me it sounds like something we would like to do in 3.0 (if
> indeed
> >>> it
> >> will have arrays as possible value (or
Re: [MTCGA]: new failures in builds [5994077, 5994075] needs to be handled
Hello! I suggest that you may revert this change right away, and reopen the ticket. Then patch author (Semyon) can take a look. Regards, -- Ilya Kasnacheev пн, 17 мая 2021 г. в 14:16, Maxim Muzafarov : > Folks, > > All tests passed successfully in the branch with reverted commit [1]: > > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_JdbcDriver&tab=buildTypeStatusDiv&branch_IgniteTests24Java8=pull%2F9100%2Fhead > > Will anyone take a look? > I propose to revert the [1] commit and rework the patch. > > [1] > https://github.com/apache/ignite/commit/e32691b9d9ef103aa884cf0d4f48abe9d98c8e50 > > On Fri, 14 May 2021 at 17:47, Maxim Muzafarov wrote: > > > > Folks, > > > > The JDBC Driver fails with the same error in the PR branch also [2]. > > The most suspicious commit is related to the IGNITE-14575 issue. I've > > left comment [2]. > > Can you re-check the patch? > > > > [1] > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_JdbcDriver?branch=pull%2F9012%2Fhead&mode=builds > > [2] > https://issues.apache.org/jira/browse/IGNITE-14575?focusedCommentId=17344632&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17344632 > > > > On Wed, 5 May 2021 at 13:13, Ilya Kasnacheev > wrote: > > > > > > Hello! > > > > > > I have re-run these suites and they did not fail. > > > > > > I think it is some transient issue with TC. > > > > > > Regards, > > > -- > > > Ilya Kasnacheev > > > > > > > > > ср, 5 мая 2021 г. в 04:55, : > > > > > > > Hi Igniters, > > > > > > > > I've detected some new issue on TeamCity to be handled. You are > more than > > > > welcomed to help. > > > > > > > > If your changes can lead to this failure(s): We're grateful that > you were > > > > a volunteer to make the contribution to this project, but things > change and > > > > you may no longer be able to finalize your contribution. > > > > Could you respond to this email and indicate if you wish to > continue and > > > > fix test failures or step down and some committer may revert you > commit. > > > > > > > > *New Critical Failure in master RDD > > > > > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Rdd?branch=%3Cdefault%3E > > > > Changes may lead to failure were done by > > > > - ilya kasnacheev > > > > https://ci.ignite.apache.org/viewModification.html?modId=924767 > > > > - semyon danilov > > > > https://ci.ignite.apache.org/viewModification.html?modId=924731 > > > > > > > > *New Critical Failure in master JDBC Driver > > > > > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_JdbcDriver?branch=%3Cdefault%3E > > > > Changes may lead to failure were done by > > > > - ilya kasnacheev > > > > https://ci.ignite.apache.org/viewModification.html?modId=924767 > > > > - semyon danilov > > > > https://ci.ignite.apache.org/viewModification.html?modId=924731 > > > > > > > > - Here's a reminder of what contributors were agreed to do > > > > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute > > > > - Should you have any questions please contact > > > > dev@ignite.apache.org > > > > > > > > Best Regards, > > > > Apache Ignite TeamCity Bot > > > > https://github.com/apache/ignite-teamcity-bot > > > > Notification generated at 04:55:24 05-05-2021 > > > > >
Re: [DISCUSSION] Array to BinaryObject serialization
Ilya,
> Maybe we should just automate that, e.g., whenever user stores Type[], always
> store one-field ArrayWrapper object, and automatically unwrap it on get()
Yes, I’m talking about this opportunity from the beginning of the thread.
> 1. Implement binary serialization that correctly Ser and Deser array using
> some kind of the wrapper (BinaryArrayWrapper).
> 19 мая 2021 г., в 14:05, Ilya Kasnacheev
> написал(а):
>
> Hello!
>
> Yes, it does not look pretty, I agree. But I'm saying that I've not
> encountered this issue on the user list before, and it can probably be
> easily countered by storing some one-field ArrayWrapper object in the cache
> as value.
>
> Maybe we should just automate that, e.g., whenever user stores Type[],
> always store one-field ArrayWrapper object, and automatically unwrap it on
> get()
> This way we may not even need any changes to storage format, if binary
> marshaller does not mind.
>
> Regards,
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 19 мая 2021 г. в 13:33, Nikolay Izhikov :
>
>> Igniters.
>>
>> Just to clarify the issue:
>>
>> ```
>> public class BinaryObjectTest extends GridCommonAbstractTest {
>>/** */
>>@Test
>>public void testArray() throws Exception {
>>Ignite ign = startGrid();
>>
>>IgniteCache cache =
>> ign.createCache("my-cache");
>>
>>cache.put(1, new TestClass1[] {new TestClass1(), new
>> TestClass1()});
>>TestClass1[] obj = cache.get(1); //This will fail with
>> ClassCastException.
>>
>>assertEquals(TestClass1[].class, obj.getClass());
>>}
>> }
>> ```
>>
>>> 19 мая 2021 г., в 13:04, Nikolay Izhikov
>> написал(а):
>>>
>>> Thanks, Ilya.
>>>
>>> Can you put more context on this?
>>> I don’t familiar with these issues.
>>>
19 мая 2021 г., в 13:02, Ilya Kasnacheev
>> написал(а):
Hello!
Obvious issues are Lazy SQL, Event Driven Services, Sort Binary Object
Fields.
Regards,
--
Ilya Kasnacheev
ср, 19 мая 2021 г. в 12:56, Nikolay Izhikov :
> Hello,
>
>> However, for internal platform and services implementations we should
> fix the root cause:
>> avoid extra deserialization->serialization pass completely.
>> This will also improve performance.
>
> Pavel, thanks for the feedback.
> If I understand correctly, your suggestion is to know data size at the
> start of reading service parameters.
> Is it correct?
>
> Right now, when the service method invoked we pass an array of
>> parameters
> through platform reader/writer machinery.
> On java side parameters read one by one and we don’t know the size of
>> the
> data on the read start.
>
> AFAICU, this will require x2 memory on the .Net or thin client-side.
>
>
>
>> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/services/PlatformServices.java#L289
>
>
>> if we are to break compatibility, I would like to see it done for some
> really common pain point.
>
> Ilya, can you, please, provide a list of common issues with Ignite that
> can be resolved
> only with compatibility breakage?
>
>> 4 мая 2021 г., в 12:58, Ilya Kasnacheev
> написал(а):
>>
>> Hello!
>>
>> If we really decide to break some compatibility then Array to
> BinaryObject
>> serialization will be very, very low on my personal list.
>>
>> I just don't see how this issue is relevant. I have been reading and
>> answering user list for a few years now, and I don't remember a single
>> question about storing ConcreteType[] as value and complaining about
>> type
>> information loss.
>>
>> If you have a good scenario how do we keep persistent store binary
>> compatibility here, without adding a lot of new code and still
>> checking
> for
>> both old and new approaches - you can go forward for sure.
>>
>> However, it does seem questionable where we have a new wrapper class
>> specifically for top level arrays. You can have this wrapper in your
>> own
>> client code and it should work OK.
>>
>> Bottom line, if we are to break compatibility, I would like to see it
> done
>> for some really common pain point.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пт, 30 апр. 2021 г. в 17:34, Nikolay Izhikov :
>>
>>> Hello, Ilya.
>>>
>>> Thanks for the feedback!
>>>
For me it sounds like something we would like to do in 3.0
>>>
>>> Ignite 3 is a very long way to go, so I prefer to target this fix in
>>> Ignite 2.x.
>>>
I think making it default "true" is a breaking change and is not
>>> possible in a minor release
>>>
>>> Yes, you are correct it is a breaking change.
>>> It seems for me, we all agreed that b
Re: [DISCUSSION] Array to BinaryObject serialization
Hello!
Why do you need to take compatibility into account here at all?
You can start writing entries to cache in new format while reading both old
and new format.
I consider returning Object[] instead of ConcreteType[] a bug so it's
totally OK to start returning the "better" ConcreteType[] instead.
I think you can just fix the bug while preserving accessibility of old
(pre-bugfix) data as it was pre-2.11.
Regards,
--
Ilya Kasnacheev
ср, 19 мая 2021 г. в 14:08, Nikolay Izhikov :
> Ilya,
>
> > Maybe we should just automate that, e.g., whenever user stores Type[],
> always store one-field ArrayWrapper object, and automatically unwrap it on
> get()
>
> Yes, I’m talking about this opportunity from the beginning of the thread.
>
> > 1. Implement binary serialization that correctly Ser and Deser array
> using some kind of the wrapper (BinaryArrayWrapper).
>
>
> > 19 мая 2021 г., в 14:05, Ilya Kasnacheev
> написал(а):
> >
> > Hello!
> >
> > Yes, it does not look pretty, I agree. But I'm saying that I've not
> > encountered this issue on the user list before, and it can probably be
> > easily countered by storing some one-field ArrayWrapper object in the
> cache
> > as value.
> >
> > Maybe we should just automate that, e.g., whenever user stores Type[],
> > always store one-field ArrayWrapper object, and automatically unwrap it
> on
> > get()
> > This way we may not even need any changes to storage format, if binary
> > marshaller does not mind.
> >
> > Regards,
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > ср, 19 мая 2021 г. в 13:33, Nikolay Izhikov :
> >
> >> Igniters.
> >>
> >> Just to clarify the issue:
> >>
> >> ```
> >> public class BinaryObjectTest extends GridCommonAbstractTest {
> >>/** */
> >>@Test
> >>public void testArray() throws Exception {
> >>Ignite ign = startGrid();
> >>
> >>IgniteCache cache =
> >> ign.createCache("my-cache");
> >>
> >>cache.put(1, new TestClass1[] {new TestClass1(), new
> >> TestClass1()});
> >>TestClass1[] obj = cache.get(1); //This will fail with
> >> ClassCastException.
> >>
> >>assertEquals(TestClass1[].class, obj.getClass());
> >>}
> >> }
> >> ```
> >>
> >>> 19 мая 2021 г., в 13:04, Nikolay Izhikov
> >> написал(а):
> >>>
> >>> Thanks, Ilya.
> >>>
> >>> Can you put more context on this?
> >>> I don’t familiar with these issues.
> >>>
> 19 мая 2021 г., в 13:02, Ilya Kasnacheev
> >> написал(а):
>
> Hello!
>
> Obvious issues are Lazy SQL, Event Driven Services, Sort Binary Object
> Fields.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 19 мая 2021 г. в 12:56, Nikolay Izhikov :
>
> > Hello,
> >
> >> However, for internal platform and services implementations we
> should
> > fix the root cause:
> >> avoid extra deserialization->serialization pass completely.
> >> This will also improve performance.
> >
> > Pavel, thanks for the feedback.
> > If I understand correctly, your suggestion is to know data size at
> the
> > start of reading service parameters.
> > Is it correct?
> >
> > Right now, when the service method invoked we pass an array of
> >> parameters
> > through platform reader/writer machinery.
> > On java side parameters read one by one and we don’t know the size of
> >> the
> > data on the read start.
> >
> > AFAICU, this will require x2 memory on the .Net or thin client-side.
> >
> >
> >
> >>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/services/PlatformServices.java#L289
> >
> >
> >> if we are to break compatibility, I would like to see it done for
> some
> > really common pain point.
> >
> > Ilya, can you, please, provide a list of common issues with Ignite
> that
> > can be resolved
> > only with compatibility breakage?
> >
> >> 4 мая 2021 г., в 12:58, Ilya Kasnacheev
> > написал(а):
> >>
> >> Hello!
> >>
> >> If we really decide to break some compatibility then Array to
> > BinaryObject
> >> serialization will be very, very low on my personal list.
> >>
> >> I just don't see how this issue is relevant. I have been reading and
> >> answering user list for a few years now, and I don't remember a
> single
> >> question about storing ConcreteType[] as value and complaining about
> >> type
> >> information loss.
> >>
> >> If you have a good scenario how do we keep persistent store binary
> >> compatibility here, without adding a lot of new code and still
> >> checking
> > for
> >> both old and new approaches - you can go forward for sure.
> >>
> >> However, it does seem questionable where we have a new wrapper class
> >> specifically for top level arrays. You can have this wrapper in your
> >> own
> >> client code and it should work OK.
> >>
Re: [DISCUSSION] Array to BinaryObject serialization
Ilya.
Actually, current behaviour described even in documentation [1]
> Note that not all objects are converted to the binary object format. The
> following classes are never converted (e.g., the toBinary(Object) method
> returns the original object, and instances of these classes are stored
> without changes):
...
> ... arrays of objects (but the objects inside them are reconverted if they
> are binary)
[1]
https://ignite.apache.org/docs/latest/key-value-api/binary-objects#enabling-binary-mode-for-caches
> 19 мая 2021 г., в 14:12, Ilya Kasnacheev
> написал(а):
>
> Hello!
>
> Why do you need to take compatibility into account here at all?
> You can start writing entries to cache in new format while reading both old
> and new format.
>
> I consider returning Object[] instead of ConcreteType[] a bug so it's
> totally OK to start returning the "better" ConcreteType[] instead.
>
> I think you can just fix the bug while preserving accessibility of old
> (pre-bugfix) data as it was pre-2.11.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 19 мая 2021 г. в 14:08, Nikolay Izhikov :
>
>> Ilya,
>>
>>> Maybe we should just automate that, e.g., whenever user stores Type[],
>> always store one-field ArrayWrapper object, and automatically unwrap it on
>> get()
>>
>> Yes, I’m talking about this opportunity from the beginning of the thread.
>>
>>> 1. Implement binary serialization that correctly Ser and Deser array
>> using some kind of the wrapper (BinaryArrayWrapper).
>>
>>
>>> 19 мая 2021 г., в 14:05, Ilya Kasnacheev
>> написал(а):
>>>
>>> Hello!
>>>
>>> Yes, it does not look pretty, I agree. But I'm saying that I've not
>>> encountered this issue on the user list before, and it can probably be
>>> easily countered by storing some one-field ArrayWrapper object in the
>> cache
>>> as value.
>>>
>>> Maybe we should just automate that, e.g., whenever user stores Type[],
>>> always store one-field ArrayWrapper object, and automatically unwrap it
>> on
>>> get()
>>> This way we may not even need any changes to storage format, if binary
>>> marshaller does not mind.
>>>
>>> Regards,
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> ср, 19 мая 2021 г. в 13:33, Nikolay Izhikov :
>>>
Igniters.
Just to clarify the issue:
```
public class BinaryObjectTest extends GridCommonAbstractTest {
/** */
@Test
public void testArray() throws Exception {
Ignite ign = startGrid();
IgniteCache cache =
ign.createCache("my-cache");
cache.put(1, new TestClass1[] {new TestClass1(), new
TestClass1()});
TestClass1[] obj = cache.get(1); //This will fail with
ClassCastException.
assertEquals(TestClass1[].class, obj.getClass());
}
}
```
> 19 мая 2021 г., в 13:04, Nikolay Izhikov
написал(а):
>
> Thanks, Ilya.
>
> Can you put more context on this?
> I don’t familiar with these issues.
>
>> 19 мая 2021 г., в 13:02, Ilya Kasnacheev
написал(а):
>>
>> Hello!
>>
>> Obvious issues are Lazy SQL, Event Driven Services, Sort Binary Object
>> Fields.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> ср, 19 мая 2021 г. в 12:56, Nikolay Izhikov :
>>
>>> Hello,
>>>
However, for internal platform and services implementations we
>> should
>>> fix the root cause:
avoid extra deserialization->serialization pass completely.
This will also improve performance.
>>>
>>> Pavel, thanks for the feedback.
>>> If I understand correctly, your suggestion is to know data size at
>> the
>>> start of reading service parameters.
>>> Is it correct?
>>>
>>> Right now, when the service method invoked we pass an array of
parameters
>>> through platform reader/writer machinery.
>>> On java side parameters read one by one and we don’t know the size of
the
>>> data on the read start.
>>>
>>> AFAICU, this will require x2 memory on the .Net or thin client-side.
>>>
>>>
>>>
>> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/services/PlatformServices.java#L289
>>>
>>>
if we are to break compatibility, I would like to see it done for
>> some
>>> really common pain point.
>>>
>>> Ilya, can you, please, provide a list of common issues with Ignite
>> that
>>> can be resolved
>>> only with compatibility breakage?
>>>
4 мая 2021 г., в 12:58, Ilya Kasnacheev
>>> написал(а):
Hello!
If we really decide to break some compatibility then Array to
>>> BinaryObject
serialization will be very, very low on my personal list.
I just don't see how this issue is relevant. I have been reading and
>>>
AggregateUnionTransposeRule fails when some inputs have unique grouping key
Hi, The AggregateUnionTransposeRule attempts to push the Aggregate below the Union. Before: Aggregate[group=$0, agg=SUM($1] Union[all] Input1 Input2 After: Aggregate[group=$0, agg=SUM($1] Union[all] Aggregate[group=$0, agg=SUM($1] Input1 Aggregate[group=$0, agg=SUM($1] Input2 When pushing the Aggregate, it checks whether the input is definitively unique on the grouping key. If yes, the Aggregate is not installed on top of the input, assuming that the result would be the same as without the aggregate. This generates a type mismatch exception when aggregation is pushed only to some of the inputs: Aggregate[group=$0, agg=SUM($1] Union[all] Aggregate[group=$0, agg=SUM($1] Input1 Input2 It seems that the uniqueness check should not be in that rule at all, and the aggregate should be pushed unconditionally. Motivation: we already have AggregateRemoveRule that removes unnecessary aggregates. No need to duplicate the same non-trivial logic twice. Does the proposal make sense to you? Regards, Vladimir.
[CANCEL] [VOTE] Release Apache Ignite spring-data-all-ext extensions 1.0.0 RC3
Folks, Canceling due to insufficient participation. The release source package and maven building will be revisited. Spring examples will be copied to the Spring 2.2 module. I will prepare RC4 after fixing issues.
[CANCEL] [VOTE] Release Apache Ignite performance-statistics-ext extension 1.0.0 RC3
Folks, Canceling due to insufficient participation. The release source package and maven building will be revisited. I will prepare RC4 after fixing issues.
Re: AggregateUnionTransposeRule fails when some inputs have unique grouping key
Hi Vladimir, Looks like this message is for d...@calcite.apache.org, not dev@ignite.apache.org, or am I mistaken? On Wed, May 19, 2021 at 2:25 PM Vladimir Ozerov wrote: > Hi, > > The AggregateUnionTransposeRule attempts to push the Aggregate below the > Union. > > Before: > Aggregate[group=$0, agg=SUM($1] > Union[all] > Input1 > Input2 > > After: > Aggregate[group=$0, agg=SUM($1] > Union[all] > Aggregate[group=$0, agg=SUM($1] > Input1 > Aggregate[group=$0, agg=SUM($1] > Input2 > > When pushing the Aggregate, it checks whether the input is definitively > unique on the grouping key. If yes, the Aggregate is not installed on top > of the input, assuming that the result would be the same as without the > aggregate. This generates a type mismatch exception when aggregation is > pushed only to some of the inputs: > Aggregate[group=$0, agg=SUM($1] > Union[all] > Aggregate[group=$0, agg=SUM($1] > Input1 > Input2 > > It seems that the uniqueness check should not be in that rule at all, and > the aggregate should be pushed unconditionally. Motivation: we already have > AggregateRemoveRule that removes unnecessary aggregates. No need to > duplicate the same non-trivial logic twice. > > Does the proposal make sense to you? > > Regards, > Vladimir. >
Re: [VOTE][EXTENSION] Release Apache Ignite spring-data-all-ext extensions 1.0.0 RC3
Hello, Ilya.
Thank you for feedback.
Yes, the vote was cancelled. I have published results.
I will create issues and prepare RC4 after fixing them.
ср, 19 мая 2021 г., 12:58 Ilya Kasnacheev :
> Hello!
>
> Do I understand correctly that voting did not succeed due to insufficient
> participation? Can you post the vote results?
>
> I have checked examples and it indeed works.
>
> I have a list of things to do before the RC4:
> Please copy over examples directory and pom.xml to spring module 2.2.
> There's no reason why spring data 2.0 should have examples but spring data
> 2.2 should not.
> Release a single zip file called spring-data-all-ext-1.0.0-src.zip with the
> zipped source. No reason to post the same source 4 times if we are
> releasing them all together.
> Put all files in archive in spring-data-all-ext-1.0.0-src/ directory. Top
> level in zip archive should have just 1 directory.
> README.md should not be a Readme of Apache Ignite. Better keep it short and
> write something about what's Apache Ignite Extensions and where to get
> them.
> Remove scripts/ directory and .gitignore.
> Make mvn -Pcheckstyle,check-licenses validate pass - currently it fails.
>
> I promise to help with release voting agitation if these points are met.
> They will also increase the quality of release greatly.
>
> I can file a ticket with these steps if you like, or you may track it
> yourself.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 17 мая 2021 г. в 18:07, Mikhail Petrov :
>
> > +1
> >
> > I created separate Spring Applications for each version of Spring Data
> > Integration using the RC repository and following the documentation. And
> > tested common use cases.
> >
> > On 14.05.2021 15:39, Nikita Amelchev wrote:
> > >> How do I actually run these examples?
> > > Use the 'examples' maven profile, please.
> > >
> > >> I don't understand why we are releasing four source ZIPs if their
> > contents are identical? Can we release a single one, name it
> > spring-data-all-ext-1.0.0-src.zip?
> > > Modules are independent. For example, we can release spring-data-ext
> > > v1.1 without releasing others. I think each module should have its
> > > independent sources.
> > >
> > >> Can we find a way to drop all non-relevant modules?
> > > I think we can use TC to prepare the build as well as the Ignite
> > > release process do it. I suggest using it in future releases when such
> > > a build appears.
> > >
> > > пт, 14 мая 2021 г. в 15:02, Ilya Kasnacheev >:
> > >> Hello!
> > >>
> > >> How do I actually run these examples? These do not seem to have a
> > pom.xml
> > >> of their own now.
> > >>
> > >> I don't understand why we are releasing four source ZIPs if their
> > contents
> > >> are identical? Can we release a single one, name it
> > >> spring-data-all-ext-1.0.0-src.zip? Can we find a way to drop all
> > >> non-relevant modules?
> > >>
> > >> All of the files need to be in one directory, such
> > >> as spring-data-all-ext-1.0.0-src/modules, etc, but they are located in
> > >> archive root, which differs from how we do it with Apache Ignite src
> > zip.
> > >>
> > >> Source release should differ from git repo dump: it does not need
> > >> .gitignore or scripts/ directory since these are of no use for source
> > >> distribution user.
> > >>
> > >> Regards,
> > >> --
> > >> Ilya Kasnacheev
> > >>
> > >>
> > >> пт, 14 мая 2021 г. в 14:32, Nikita Amelchev :
> > >>
> > >>> Hello, Ilya.
> > >>>
> > Is there any way to find and run these examples with spring-data ext
> > >>> 1.0.0?
> > >>>
> > >>> Spring data modules were migrated to the extensions repo with their
> > >>> examples. Examples for 2.0 still available in the package:
> > >>>
> > >>>
> > >>>
> >
> modules/spring-data-2.0-ext/examples/main/java/org/apache/ignite/springdata20/examples/
> > >>>
> > >>> Other versions were not covered with examples.
> > >>>
> > I have also found that in spring-data 2.2 source archive at least,
> > >>> there's full git repo of ignite-ext including .gitignore, committer
> > >>> scripts, sources for all modules such as flink, but no spring data
> > examples.
> > >>>
> > >>> The source package is a full copy of the extensions repo as the build
> > >>> process requires the parent pom. Build instruction is mentioned in
> the
> > >>> DEVNOTES file.
> > >>>
> > >>> пт, 14 мая 2021 г. в 13:53, Ilya Kasnacheev <
> ilya.kasnach...@gmail.com
> > >:
> > Hello!
> >
> > We had Spring Data examples in 2.9 in the ZIP file:
> > examples/config/spring/example-spring-data.xml:
> >
> Ignition.start("examples/config/spring/example-spring-data.xml");
> >
> > >>>
> >
> examples/src/main/java/org/apache/ignite/examples/springdata/SpringApplicationConfiguration.java
> > :return
> > Ignition.start("examples/config/spring/example-spring-data.xml");
> >
> > examples/pom.xml:
> > >>> ignite-spring-data_2.0
> > Is there any way to find and run these examples with spring-data ext
> > >>> 1.0.0?
Re: [DISCUSSION] Array to BinaryObject serialization
Hello!
We don't have to do it in toBinary() - we may preserve its behavior if you
like, and only do that transition for Cache API, where additional support
may be added to
e.g.
org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl
Regards,
--
Ilya Kasnacheev
ср, 19 мая 2021 г. в 14:23, Nikolay Izhikov :
> Ilya.
>
> Actually, current behaviour described even in documentation [1]
>
>
> > Note that not all objects are converted to the binary object format. The
> following classes are never converted (e.g., the toBinary(Object) method
> returns the original object, and instances of these classes are stored
> without changes):
> ...
> > ... arrays of objects (but the objects inside them are reconverted if
> they are binary)
>
> [1]
> https://ignite.apache.org/docs/latest/key-value-api/binary-objects#enabling-binary-mode-for-caches
>
> > 19 мая 2021 г., в 14:12, Ilya Kasnacheev
> написал(а):
> >
> > Hello!
> >
> > Why do you need to take compatibility into account here at all?
> > You can start writing entries to cache in new format while reading both
> old
> > and new format.
> >
> > I consider returning Object[] instead of ConcreteType[] a bug so it's
> > totally OK to start returning the "better" ConcreteType[] instead.
> >
> > I think you can just fix the bug while preserving accessibility of old
> > (pre-bugfix) data as it was pre-2.11.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > ср, 19 мая 2021 г. в 14:08, Nikolay Izhikov :
> >
> >> Ilya,
> >>
> >>> Maybe we should just automate that, e.g., whenever user stores Type[],
> >> always store one-field ArrayWrapper object, and automatically unwrap it
> on
> >> get()
> >>
> >> Yes, I’m talking about this opportunity from the beginning of the
> thread.
> >>
> >>> 1. Implement binary serialization that correctly Ser and Deser array
> >> using some kind of the wrapper (BinaryArrayWrapper).
> >>
> >>
> >>> 19 мая 2021 г., в 14:05, Ilya Kasnacheev
> >> написал(а):
> >>>
> >>> Hello!
> >>>
> >>> Yes, it does not look pretty, I agree. But I'm saying that I've not
> >>> encountered this issue on the user list before, and it can probably be
> >>> easily countered by storing some one-field ArrayWrapper object in the
> >> cache
> >>> as value.
> >>>
> >>> Maybe we should just automate that, e.g., whenever user stores Type[],
> >>> always store one-field ArrayWrapper object, and automatically unwrap it
> >> on
> >>> get()
> >>> This way we may not even need any changes to storage format, if binary
> >>> marshaller does not mind.
> >>>
> >>> Regards,
> >>>
> >>> Regards,
> >>> --
> >>> Ilya Kasnacheev
> >>>
> >>>
> >>> ср, 19 мая 2021 г. в 13:33, Nikolay Izhikov :
> >>>
> Igniters.
>
> Just to clarify the issue:
>
> ```
> public class BinaryObjectTest extends GridCommonAbstractTest {
> /** */
> @Test
> public void testArray() throws Exception {
> Ignite ign = startGrid();
>
> IgniteCache cache =
> ign.createCache("my-cache");
>
> cache.put(1, new TestClass1[] {new TestClass1(), new
> TestClass1()});
> TestClass1[] obj = cache.get(1); //This will fail with
> ClassCastException.
>
> assertEquals(TestClass1[].class, obj.getClass());
> }
> }
> ```
>
> > 19 мая 2021 г., в 13:04, Nikolay Izhikov
> написал(а):
> >
> > Thanks, Ilya.
> >
> > Can you put more context on this?
> > I don’t familiar with these issues.
> >
> >> 19 мая 2021 г., в 13:02, Ilya Kasnacheev >
> написал(а):
> >>
> >> Hello!
> >>
> >> Obvious issues are Lazy SQL, Event Driven Services, Sort Binary
> Object
> >> Fields.
> >>
> >> Regards,
> >> --
> >> Ilya Kasnacheev
> >>
> >>
> >> ср, 19 мая 2021 г. в 12:56, Nikolay Izhikov :
> >>
> >>> Hello,
> >>>
> However, for internal platform and services implementations we
> >> should
> >>> fix the root cause:
> avoid extra deserialization->serialization pass completely.
> This will also improve performance.
> >>>
> >>> Pavel, thanks for the feedback.
> >>> If I understand correctly, your suggestion is to know data size at
> >> the
> >>> start of reading service parameters.
> >>> Is it correct?
> >>>
> >>> Right now, when the service method invoked we pass an array of
> parameters
> >>> through platform reader/writer machinery.
> >>> On java side parameters read one by one and we don’t know the size
> of
> the
> >>> data on the read start.
> >>>
> >>> AFAICU, this will require x2 memory on the .Net or thin
> client-side.
> >>>
> >>>
> >>>
>
> >>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/services/PlatformServices.java#L289
> >>>
> >>>
> if we are to break compatibility, I
Re: AggregateUnionTransposeRule fails when some inputs have unique grouping key
Hi Pavel, Yes, I missed the list, sorry. ср, 19 мая 2021 г. в 14:40, Pavel Tupitsyn : > Hi Vladimir, > > Looks like this message is for d...@calcite.apache.org, not > dev@ignite.apache.org, > or am I mistaken? > > On Wed, May 19, 2021 at 2:25 PM Vladimir Ozerov > wrote: > > > Hi, > > > > The AggregateUnionTransposeRule attempts to push the Aggregate below the > > Union. > > > > Before: > > Aggregate[group=$0, agg=SUM($1] > > Union[all] > > Input1 > > Input2 > > > > After: > > Aggregate[group=$0, agg=SUM($1] > > Union[all] > > Aggregate[group=$0, agg=SUM($1] > > Input1 > > Aggregate[group=$0, agg=SUM($1] > > Input2 > > > > When pushing the Aggregate, it checks whether the input is definitively > > unique on the grouping key. If yes, the Aggregate is not installed on top > > of the input, assuming that the result would be the same as without the > > aggregate. This generates a type mismatch exception when aggregation is > > pushed only to some of the inputs: > > Aggregate[group=$0, agg=SUM($1] > > Union[all] > > Aggregate[group=$0, agg=SUM($1] > > Input1 > > Input2 > > > > It seems that the uniqueness check should not be in that rule at all, and > > the aggregate should be pushed unconditionally. Motivation: we already > have > > AggregateRemoveRule that removes unnecessary aggregates. No need to > > duplicate the same non-trivial logic twice. > > > > Does the proposal make sense to you? > > > > Regards, > > Vladimir. > > >
Re: [MTCGA]: new failures in builds [5994077, 5994075] needs to be handled
Hi! The fix was merged yesterday and the JDBC driver suite https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_JdbcDriver?branch=%3Cdefault%3E&buildTypeTab=overview&mode=builds#all-projects is stable again. Semyon. 19.05.2021, 14:07, "Ilya Kasnacheev" : > Hello! > > I suggest that you may revert this change right away, and reopen the ticket. > > Then patch author (Semyon) can take a look. > > Regards, > -- > Ilya Kasnacheev > > пн, 17 мая 2021 г. в 14:16, Maxim Muzafarov : > >> Folks, >> >> All tests passed successfully in the branch with reverted commit [1]: >> >> >> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_JdbcDriver&tab=buildTypeStatusDiv&branch_IgniteTests24Java8=pull%2F9100%2Fhead >> >> Will anyone take a look? >> I propose to revert the [1] commit and rework the patch. >> >> [1] >> >> https://github.com/apache/ignite/commit/e32691b9d9ef103aa884cf0d4f48abe9d98c8e50 >> >> On Fri, 14 May 2021 at 17:47, Maxim Muzafarov wrote: >> > >> > Folks, >> > >> > The JDBC Driver fails with the same error in the PR branch also [2]. >> > The most suspicious commit is related to the IGNITE-14575 issue. I've >> > left comment [2]. >> > Can you re-check the patch? >> > >> > [1] >> >> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_JdbcDriver?branch=pull%2F9012%2Fhead&mode=builds >> > [2] >> >> https://issues.apache.org/jira/browse/IGNITE-14575?focusedCommentId=17344632&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17344632 >> > >> > On Wed, 5 May 2021 at 13:13, Ilya Kasnacheev >> wrote: >> > > >> > > Hello! >> > > >> > > I have re-run these suites and they did not fail. >> > > >> > > I think it is some transient issue with TC. >> > > >> > > Regards, >> > > -- >> > > Ilya Kasnacheev >> > > >> > > >> > > ср, 5 мая 2021 г. в 04:55, : >> > > >> > > > Hi Igniters, >> > > > >> > > > I've detected some new issue on TeamCity to be handled. You are >> more than >> > > > welcomed to help. >> > > > >> > > > If your changes can lead to this failure(s): We're grateful that >> you were >> > > > a volunteer to make the contribution to this project, but things >> change and >> > > > you may no longer be able to finalize your contribution. >> > > > Could you respond to this email and indicate if you wish to >> continue and >> > > > fix test failures or step down and some committer may revert you >> commit. >> > > > >> > > > * New Critical Failure in master RDD >> > > > >> >> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Rdd?branch=%3Cdefault%3E >> > > > Changes may lead to failure were done by >> > > > - ilya kasnacheev >> > > > https://ci.ignite.apache.org/viewModification.html?modId=924767 >> > > > - semyon danilov >> > > > https://ci.ignite.apache.org/viewModification.html?modId=924731 >> > > > >> > > > * New Critical Failure in master JDBC Driver >> > > > >> >> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_JdbcDriver?branch=%3Cdefault%3E >> > > > Changes may lead to failure were done by >> > > > - ilya kasnacheev >> > > > https://ci.ignite.apache.org/viewModification.html?modId=924767 >> > > > - semyon danilov >> > > > https://ci.ignite.apache.org/viewModification.html?modId=924731 >> > > > >> > > > - Here's a reminder of what contributors were agreed to do >> > > > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute >> > > > - Should you have any questions please contact >> > > > dev@ignite.apache.org >> > > > >> > > > Best Regards, >> > > > Apache Ignite TeamCity Bot >> > > > https://github.com/apache/ignite-teamcity-bot >> > > > Notification generated at 04:55:24 05-05-2021 >> > > >
