Pavel, I totally support that. Also, if we are aiming for stronger platform-independance, in our schemas we may want to support bit-notation (int32, uint64)? For example "long" can mean a different type on different platforms and it's easy to confuse them (happens often when using ODBC for example).
Best Regards, Igor On Tue, Nov 24, 2020 at 1:34 PM Pavel Tupitsyn <ptupit...@apache.org> wrote: > Igniters, > > I think we should support unsigned data types: > uByte, uShort, uInt, uLong > > Java does not have them, but many other languages do, > and with the growing number of thin clients this is important. > > For example, in current Ignite.NET implementation we store unsigned values > as signed internally, > but this is a huge pain when it comes to metadata, binary objects, etc. > (it is easy to deserialize int as uint when you have a class, but not with > BinaryObject.GetField) > > Any objections? > > On Tue, Nov 24, 2020 at 12:28 PM Andrey Mashenkov < > andrey.mashen...@gmail.com> wrote: > > > Denis, > > > > Good point. Both serializers use reflection API. > > However, we will allow users to configure static schema along with > 'strict' > > schema mode, we still need to validate user classes on client nodes > against > > the latest schema in the grid and reflection API is the only way to do > it. > > One can find a few articles on the internet on how to enable reflection > in > > GraalVM. > > > > I'll create a task for supporting GraalVM, and maybe someone who is > > familiar with GraalVM will suggest a solution or a proper workaround. Or > > I'll do it a bit later. > > If no workaround is found, we could allow users to write it's own > > serializer, but I don't think it is a good idea to expose any internal > > classes to the public. > > > > On Tue, Nov 24, 2020 at 2:55 AM Denis Magda <dma...@apache.org> wrote: > > > > > Andrey, thanks for the update, > > > > > > Does any of the serializers take into consideration the > > > native-image-generation feature of GraalVM? > > > https://www.graalvm.org/reference-manual/native-image/ > > > > > > With the current binary marshaller, we can't even generate a native > image > > > for the code using our thin client APIs. > > > > > > - > > > Denis > > > > > > > > > On Mon, Nov 23, 2020 at 4:39 AM Andrey Mashenkov < > > > andrey.mashen...@gmail.com> > > > wrote: > > > > > > > Hi Igniters, > > > > > > > > I'd like to continue discussion of IEP-54 (Schema-first approach). > > > > > > > > Hope everyone who is interested had a chance to get familiar with the > > > > proposal [1]. > > > > Please, do not hesitate to ask questions and share your ideas. > > > > > > > > I've prepared a prototype of serializer [2] for the data layout > > described > > > > in the proposal. > > > > In prototy, I compared 2 approaches to (de)serialize objects, the > first > > > one > > > > uses java reflection/unsafe API and similar to one we already use in > > > Ignite > > > > and the second one generates serializer for particular user class and > > > uses > > > > Janino library for compilation. > > > > Second one shows better results in benchmarks. > > > > I think we can go with it as default serializer and have > > reflection-based > > > > implementation as a fallback if someone will have issues with the > first > > > > one. > > > > WDYT? > > > > > > > > There are a number of tasks under the umbrella ticket [3] waiting for > > the > > > > assignee. > > > > > > > > BTW, I'm going to create more tickets for schema manager modes > > > > implementation, but would like to clarify some details. > > > > > > > > I thought schemaManager on each node should held: > > > > 1. Local mapping of "schema version" <--> validated local key/value > > > > classes pair. > > > > 2. Cluster-wide schema changes history. > > > > On the client side. Before any key-value API operation we should > > > validate a > > > > schema for a given key-value pair. > > > > If there is no local-mapping exists for a given key-value pair or if > a > > > > cluster wide schema has a more recent version then the key-value pair > > > > should be validated against the latest version and local mapping > should > > > be > > > > updated/actualized. > > > > If an object doesn't fit to the latest schema then it depends on > schema > > > > mode: either fail the operation ('strict' mode) or a new mapping > should > > > be > > > > created and a new schema version should be propagated to the cluster. > > > > > > > > On the server side we usually have no key-value classes and we > operate > > > with > > > > tuples. > > > > As schema change history is available and a tuple has schema version, > > > then > > > > it is possible to upgrade any received tuple to the last version > > without > > > > desialization. > > > > Thus we could allow nodes to send key-value pairs of previous > versions > > > (if > > > > they didn't receive a schema update yet) without reverting schema > > changes > > > > made by a node with newer classes. > > > > > > > > Alex, Val, Ivan did you mean the same? > > > > > > > > > > > > [1] > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach > > > > [2] > https://github.com/apache/ignite/tree/ignite-13618/modules/commons > > > > [3] https://issues.apache.org/jira/browse/IGNITE-13616 > > > > > > > > On Thu, Sep 17, 2020 at 9:21 AM Ivan Pavlukhin <vololo...@gmail.com> > > > > wrote: > > > > > > > > > Folks, > > > > > > > > > > Please do not ignore history. We had a thread [1] with many bright > > > > > ideas. We can resume it. > > > > > > > > > > [1] > > > > > > > > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/Applicability-of-term-cache-to-Apache-Ignite-td36541.html > > > > > > > > > > 2020-09-10 0:08 GMT+03:00, Denis Magda <dma...@apache.org>: > > > > > > Val, makes sense, thanks for explaining. > > > > > > > > > > > > Agree that we need to have a separate discussion thread for the > > > "table" > > > > > and > > > > > > "cache" terms substitution. I'll appreciate it if you start the > > > thread > > > > > > sharing pointers to any relevant IEPs and reasoning behind the > > > > suggested > > > > > > change. > > > > > > > > > > > > - > > > > > > Denis > > > > > > > > > > > > > > > > > > On Tue, Sep 8, 2020 at 6:01 PM Valentin Kulichenko < > > > > > > valentin.kuliche...@gmail.com> wrote: > > > > > > > > > > > >> Hi Denis, > > > > > >> > > > > > >> I guess the wording in the IEP is a little bit confusing. All it > > > means > > > > > is > > > > > >> that you should not create nested POJOs, but rather inline > fields > > > > into a > > > > > >> single POJO that is mapped to a particular schema. In other > words, > > > > > nested > > > > > >> POJOs are not supported. > > > > > >> > > > > > >> Alex, is this correct? Please let me know if I'm missing > > something. > > > > > >> > > > > > >> As for the "cache" term, I agree that it is outdated, but I'm > not > > > sure > > > > > >> what we can replace it with. "Table" is tightly associated with > > SQL, > > > > but > > > > > >> SQL is optional in our case. Do you want to create a separate > > > > discussion > > > > > >> about this? > > > > > >> > > > > > >> -Val > > > > > >> > > > > > >> On Tue, Sep 8, 2020 at 4:37 PM Denis Magda <dma...@apache.org> > > > wrote: > > > > > >> > > > > > >>> Val, > > > > > >>> > > > > > >>> I've checked the IEP again and have a few questions. > > > > > >>> > > > > > >>> Arbitrary nested objects and collections are not allowed as > > column > > > > > >>> values. > > > > > >>> > Nested POJOs should either be inlined into schema, or stored > as > > > > BLOBs > > > > > >>> > > > > > >>> > > > > > >>> Could you provide a DDL code snippet showing how the inlining > of > > > > POJOs > > > > > >>> is > > > > > >>> supposed to work? > > > > > >>> > > > > > >>> Also, we keep using the terms "cache" and "table" throughout > the > > > IEP. > > > > > Is > > > > > >>> it > > > > > >>> the right time to discuss an alternate name that would replace > > > those > > > > > >>> too? > > > > > >>> Personally, the "table" should stay and the "cache" should go > > > > > >>> considering > > > > > >>> that SQL is one of the primary APIs in Ignite and that DDL is > > > > supported > > > > > >>> out-of-the-box. > > > > > >>> > > > > > >>> > > > > > >>> - > > > > > >>> Denis > > > > > >>> > > > > > >>> > > > > > >>> On Mon, Sep 7, 2020 at 12:26 PM Valentin Kulichenko < > > > > > >>> valentin.kuliche...@gmail.com> wrote: > > > > > >>> > > > > > >>> > Ivan, > > > > > >>> > > > > > > >>> > I see your point. I agree that with the automatic updates we > > step > > > > > into > > > > > >>> the > > > > > >>> > schema-last territory. > > > > > >>> > > > > > > >>> > Actually, if we support automatic evolution, we can as well > > > support > > > > > >>> > creating a cache without schema and inferring it from the > first > > > > > >>> > insert. > > > > > >>> In > > > > > >>> > other words, we can have both "schema-first" and > "schema-last" > > > > modes. > > > > > >>> > > > > > > >>> > Alexey, what do you think? > > > > > >>> > > > > > > >>> > -Val > > > > > >>> > > > > > > >>> > On Mon, Sep 7, 2020 at 5:59 AM Alexey Goncharuk < > > > > > >>> > alexey.goncha...@gmail.com> > > > > > >>> > wrote: > > > > > >>> > > > > > > >>> > > Ivan, > > > > > >>> > > > > > > > >>> > > Thank you, I got your concern now. As it is mostly > regarding > > > the > > > > > >>> > > terminology, I am absolutely fine with changing the name to > > > > > whatever > > > > > >>> fits > > > > > >>> > > the approach best. Dynamic or evolving schema sounds > great. I > > > > will > > > > > >>> make > > > > > >>> > > corresponding changes to the IEP once we settle on the > name. > > > > > >>> > > > > > > > >>> > > пн, 7 сент. 2020 г. в 11:33, Ivan Pavlukhin < > > > vololo...@gmail.com > > > > >: > > > > > >>> > > > > > > > >>> > > > Hi Val, > > > > > >>> > > > > > > > > >>> > > > Thank you for your answer! > > > > > >>> > > > > > > > > >>> > > > My understanding is a little bit different. Yes, schema > > > > evolution > > > > > >>> > > > definitely should be possible. But I see a main > difference > > in > > > > > "how > > > > > >>> > > > schema is updated". I treat a common SQL approach > > > schema-first. > > > > > >>> Schema > > > > > >>> > > > and data manipulation operations are clearly separated > and > > it > > > > > >>> enables > > > > > >>> > > > interesting capabilities, e.g. preventing untended schema > > > > changes > > > > > >>> > > > by > > > > > >>> > > > mistaken data operations, restricting user permissions to > > > > change > > > > > >>> > > > schema. > > > > > >>> > > > > > > > > >>> > > > > Schema-first means that schema exists in advance and > all > > > the > > > > > >>> stored > > > > > >>> > > data > > > > > >>> > > > is compliant with it - that's exactly what is proposed. > > > > > >>> > > > > > > > > >>> > > > A schema-last approach mentioned in [1] also assumes that > > > > schema > > > > > >>> > > > exists, but it is inferred from data. Is not it more > > similar > > > to > > > > > >>> > > > the > > > > > >>> > > > proposing approach? > > > > > >>> > > > > > > > > >>> > > > And I would like to say, that my main concern so far is > > > mostly > > > > > >>> > > > about > > > > > >>> > > > terminology. And I suppose if it confuses me then others > > > might > > > > be > > > > > >>> > > > confused as well. My feeling is closer to "dynamic or > > liquid > > > or > > > > > >>> > > > may > > > > > >>> be > > > > > >>> > > > evolving schema". > > > > > >>> > > > > > > > > >>> > > > [1] > > > > > >>> > > > > > > > > >>> > > > > > > > > > > https://people.cs.umass.edu/~yanlei/courses/CS691LL-f06/papers/SH05.pdf > > > > > >>> > > > > > > > > >>> > > > 2020-09-07 0:47 GMT+03:00, Valentin Kulichenko < > > > > > >>> > > > valentin.kuliche...@gmail.com>: > > > > > >>> > > > > Hi Ivan, > > > > > >>> > > > > > > > > > >>> > > > > I don't see an issue with that. Schema-first means that > > > > schema > > > > > >>> exists > > > > > >>> > > in > > > > > >>> > > > > advance and all the stored data is compliant with it - > > > that's > > > > > >>> exactly > > > > > >>> > > > what > > > > > >>> > > > > is proposed. There are no restrictions prohibiting > > changes > > > to > > > > > >>> > > > > the > > > > > >>> > > schema. > > > > > >>> > > > > > > > > > >>> > > > > -Val > > > > > >>> > > > > > > > > > >>> > > > > On Sat, Sep 5, 2020 at 9:52 PM Ivan Pavlukhin < > > > > > >>> vololo...@gmail.com> > > > > > >>> > > > wrote: > > > > > >>> > > > > > > > > > >>> > > > >> Alexey, > > > > > >>> > > > >> > > > > > >>> > > > >> I am a little bit confused with terminology. My > > > > understanding > > > > > >>> > conforms > > > > > >>> > > > >> to a survey [1] (see part X Semi Structured Data). Can > > we > > > > > >>> > > > >> really > > > > > >>> > treat > > > > > >>> > > > >> a "dynamic schema" approach as a kind of > "schema-first"? > > > > > >>> > > > >> > > > > > >>> > > > >> [1] > > > > > >>> > > > >> > > > > > >>> > > > > > > > >>> > > > > > > > > > https://people.cs.umass.edu/~yanlei/courses/CS691LL-f06/papers/SH05.pdf > > > > > >>> > > > >> > > > > > >>> > > > >> 2020-09-02 1:53 GMT+03:00, Denis Magda < > > dma...@apache.org > > > >: > > > > > >>> > > > >> >> > > > > > >>> > > > >> >> However, could you please elaborate on the relation > > > > between > > > > > >>> > Ignite > > > > > >>> > > > and > > > > > >>> > > > >> >> ORM? > > > > > >>> > > > >> >> Is there a use case for Hibernate running on top of > > > > Ignite > > > > > >>> > > > >> >> (I > > > > > >>> > > haven't > > > > > >>> > > > >> >> seen > > > > > >>> > > > >> >> one so far)? If so, what is missing exactly on the > > > Ignite > > > > > >>> side to > > > > > >>> > > > >> support > > > > > >>> > > > >> >> this? In my understanding, all you need is SQL API > > > which > > > > we > > > > > >>> > already > > > > > >>> > > > >> have. > > > > > >>> > > > >> >> Am I missing something? > > > > > >>> > > > >> > > > > > > >>> > > > >> > > > > > > >>> > > > >> > Good point, yes, if all the ORM integrations use > > Ignite > > > > SQL > > > > > >>> APIs > > > > > >>> > > > >> > internally, then they can easily translate an Entity > > > > object > > > > > >>> into > > > > > >>> > an > > > > > >>> > > > >> > INSERT/UPDATE statement that lists all the object's > > > > fields. > > > > > >>> > Luckily, > > > > > >>> > > > >> > our > > > > > >>> > > > >> > Spring Data integration is already based on the > Ignite > > > SQL > > > > > >>> > > > >> > APIs > > > > > >>> > and > > > > > >>> > > > >> > needs > > > > > >>> > > > >> > to be improved once the schema-first approach is > > > > supported. > > > > > >>> That > > > > > >>> > > would > > > > > >>> > > > >> > solve a ton of usability issues. > > > > > >>> > > > >> > > > > > > >>> > > > >> > I would revise the Hibernate integration as well > > during > > > > the > > > > > >>> Ignite > > > > > >>> > > 3.0 > > > > > >>> > > > >> dev > > > > > >>> > > > >> > phase. Can't say if it's used a lot but Spring Data > is > > > > > >>> > > > >> > getting > > > > > >>> > > > traction > > > > > >>> > > > >> for > > > > > >>> > > > >> > sure. > > > > > >>> > > > >> > > > > > > >>> > > > >> > @Michael Pollind, I'll loop you in as long as you've > > > > started > > > > > >>> > working > > > > > >>> > > > on > > > > > >>> > > > >> the > > > > > >>> > > > >> > Ignite support for Micornaut Data > > > > > >>> > > > >> > < > > > > > >>> > > > > https://micronaut-projects.github.io/micronaut-data/latest/guide/> > > > > > >>> > > > and > > > > > >>> > > > >> > came across some challenges. Just watch this > > discussion. > > > > > >>> > > > >> > That's > > > > > >>> > what > > > > > >>> > > > is > > > > > >>> > > > >> > coming in Ignite 3.0. > > > > > >>> > > > >> > > > > > > >>> > > > >> > > > > > > >>> > > > >> > - > > > > > >>> > > > >> > Denis > > > > > >>> > > > >> > > > > > > >>> > > > >> > > > > > > >>> > > > >> > On Mon, Aug 31, 2020 at 5:11 PM Valentin Kulichenko > < > > > > > >>> > > > >> > valentin.kuliche...@gmail.com> wrote: > > > > > >>> > > > >> > > > > > > >>> > > > >> >> Hi Denis, > > > > > >>> > > > >> >> > > > > > >>> > > > >> >> Generally speaking, I believe that the schema-first > > > > > approach > > > > > >>> > > natively > > > > > >>> > > > >> >> addresses the issue if duplicate fields in key and > > > value > > > > > >>> objects, > > > > > >>> > > > >> because > > > > > >>> > > > >> >> schema will be created for a cache, not for an > > object, > > > as > > > > > it > > > > > >>> > > happens > > > > > >>> > > > >> now. > > > > > >>> > > > >> >> Basically, the schema will define whether there is > a > > > > > primary > > > > > >>> key > > > > > >>> > or > > > > > >>> > > > >> >> not, > > > > > >>> > > > >> >> and which fields are included in case there is one. > > Any > > > > API > > > > > >>> that > > > > > >>> > we > > > > > >>> > > > >> would > > > > > >>> > > > >> >> have must be compliant with this, so it becomes > > fairly > > > > easy > > > > > >>> > > > >> >> to > > > > > >>> > work > > > > > >>> > > > >> >> with > > > > > >>> > > > >> >> data as with a set of records, rather than > key-value > > > > pairs. > > > > > >>> > > > >> >> > > > > > >>> > > > >> >> However, could you please elaborate on the relation > > > > between > > > > > >>> > Ignite > > > > > >>> > > > and > > > > > >>> > > > >> >> ORM? > > > > > >>> > > > >> >> Is there a use case for Hibernate running on top of > > > > Ignite > > > > > >>> > > > >> >> (I > > > > > >>> > > haven't > > > > > >>> > > > >> >> seen > > > > > >>> > > > >> >> one so far)? If so, what is missing exactly on the > > > Ignite > > > > > >>> side to > > > > > >>> > > > >> support > > > > > >>> > > > >> >> this? In my understanding, all you need is SQL API > > > which > > > > we > > > > > >>> > already > > > > > >>> > > > >> have. > > > > > >>> > > > >> >> Am I missing something? > > > > > >>> > > > >> >> > > > > > >>> > > > >> >> -Val > > > > > >>> > > > >> >> > > > > > >>> > > > >> >> On Mon, Aug 31, 2020 at 2:08 PM Denis Magda < > > > > > >>> dma...@apache.org> > > > > > >>> > > > wrote: > > > > > >>> > > > >> >> > > > > > >>> > > > >> >> > Val, > > > > > >>> > > > >> >> > > > > > > >>> > > > >> >> > I would propose adding another point to the > > > motivations > > > > > >>> > > > >> >> > list > > > > > >>> > > which > > > > > >>> > > > >> >> > is > > > > > >>> > > > >> >> > related to the ORM frameworks such as Spring > Data, > > > > > >>> Hibernate, > > > > > >>> > > > >> Micronaut > > > > > >>> > > > >> >> and > > > > > >>> > > > >> >> > many others. > > > > > >>> > > > >> >> > > > > > > >>> > > > >> >> > Presently, the storage engine requires to > > distinguish > > > > key > > > > > >>> > objects > > > > > >>> > > > >> >> > from > > > > > >>> > > > >> >> the > > > > > >>> > > > >> >> > value ones that complicate the usage of Ignite > with > > > > those > > > > > >>> ORM > > > > > >>> > > > >> >> > frameworks > > > > > >>> > > > >> >> > (especially if a key object comprises several > > > fields). > > > > > >>> > > > >> >> > More > > > > > >>> on > > > > > >>> > > this > > > > > >>> > > > >> can > > > > > >>> > > > >> >> be > > > > > >>> > > > >> >> > found here: > > > > > >>> > > > >> >> > > > > > > >>> > > > >> >> > > > > > > >>> > > > >> >> > > > > > >>> > > > >> > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > > > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Key-and-Value-fields-with-same-name-and-SQL-DML-td47557.html > > > > > >>> > > > >> >> > > > > > > >>> > > > >> >> > It will be nice if the new schema-first approach > > > allows > > > > > us > > > > > >>> to > > > > > >>> > > work > > > > > >>> > > > >> with > > > > > >>> > > > >> >> > a > > > > > >>> > > > >> >> > single entity object when it comes to the ORMs. > > With > > > no > > > > > >>> need to > > > > > >>> > > > >> >> > split > > > > > >>> > > > >> >> > the > > > > > >>> > > > >> >> > entity into a key and value. Just want to be sure > > > that > > > > > the > > > > > >>> > Ignite > > > > > >>> > > > >> >> > 3.0 > > > > > >>> > > > >> >> > has > > > > > >>> > > > >> >> > all the essential public APIs that would support > > the > > > > > >>> > > single-entity > > > > > >>> > > > >> >> > based > > > > > >>> > > > >> >> > approach. > > > > > >>> > > > >> >> > > > > > > >>> > > > >> >> > What do you think? > > > > > >>> > > > >> >> > > > > > > >>> > > > >> >> > - > > > > > >>> > > > >> >> > Denis > > > > > >>> > > > >> >> > > > > > > >>> > > > >> >> > > > > > > >>> > > > >> >> > On Fri, Aug 28, 2020 at 3:50 PM Valentin > > Kulichenko < > > > > > >>> > > > >> >> > valentin.kuliche...@gmail.com> wrote: > > > > > >>> > > > >> >> > > > > > > >>> > > > >> >> > > Igniters, > > > > > >>> > > > >> >> > > > > > > > >>> > > > >> >> > > One of the big changes proposed for Ignite 3.0 > is > > > the > > > > > >>> > so-called > > > > > >>> > > > >> >> > > "schema-first approach". To add more clarity, > > I've > > > > > >>> > > > >> >> > > started > > > > > >>> > > > writing > > > > > >>> > > > >> >> > > the > > > > > >>> > > > >> >> > IEP > > > > > >>> > > > >> >> > > for this change: > > > > > >>> > > > >> >> > > > > > > > >>> > > > >> >> > > > > > > > >>> > > > >> >> > > > > > > >>> > > > >> >> > > > > > >>> > > > >> > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach > > > > > >>> > > > >> >> > > > > > > > >>> > > > >> >> > > Please take a look and let me know if there are > > any > > > > > >>> immediate > > > > > >>> > > > >> >> > > thoughts, > > > > > >>> > > > >> >> > > suggestions, or objections. > > > > > >>> > > > >> >> > > > > > > > >>> > > > >> >> > > -Val > > > > > >>> > > > >> >> > > > > > > > >>> > > > >> >> > > > > > > >>> > > > >> >> > > > > > >>> > > > >> > > > > > > >>> > > > >> > > > > > >>> > > > >> > > > > > >>> > > > >> -- > > > > > >>> > > > >> > > > > > >>> > > > >> Best regards, > > > > > >>> > > > >> Ivan Pavlukhin > > > > > >>> > > > >> > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > -- > > > > > >>> > > > > > > > > >>> > > > Best regards, > > > > > >>> > > > Ivan Pavlukhin > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Best regards, > > > > > Ivan Pavlukhin > > > > > > > > > > > > > > > > > -- > > > > Best regards, > > > > Andrey V. Mashenkov > > > > > > > > > > > > > -- > > Best regards, > > Andrey V. Mashenkov > > >
