Re: [DISCUSSION] Code style. Variable abbrevations
Enforced validation doesn't guarantee code consistency. No validation in the world prevent me from typing manually "mess" instead of "msg"/"message" or "svc" instead of "srvc"/"service" (btw "svc" is more common imo). And the fact that someone name a variable "count" instead of "cnt" doesn't make the whole project immature. -- Regards, Konstantin Orlov > On 17 Jun 2021, at 08:34, Ivan Pavlukhin wrote: > > Hi Nikolay, > > Thanks, it is rather interesting. > > Do we all agree that using different conventions for different code > packages does not break "consistency"? Or did I get something wrong? > > 2021-06-17 7:12 GMT+03:00, Николай Ижиков : >> Hello, Ivan. >> >> We can create checkstyle rule to enforce usage of abbreviations. >> Internal/public code differs by package. >> >> PoC of rule [1] >> >> [1] https://github.com/apache/ignite/pull/9153 >> >>> 17 июня 2021 г., в 07:01, Ivan Pavlukhin >>> написал(а): >>> >>> Nikita, >>> >>> Do you have a plan in your mind how to make Ignite codebase consistent? >>> >>> AFAIR, we had it intentionally inconsistent for a long time at least >>> for one sake: for internal code we used special conventions (e.g. >>> abbreviations, shorten getters) and common Java conventions for public >>> API and examples (e.g. no abbreviations and usual getters). >>> >>> 2021-06-16 23:37 GMT+03:00, Nikita Ivanov : Consistency is what makes it easier to contribute to the project and attract new members. Consistency implies strong, well defined and universally enforced rules. Just because we have some individuals who are lazy or inexperienced - it does not mean that the entire project should relax the basic-level engineering discipline. On a personal node - nothing screams "immaturity" louder that a code that uses inconsistent naming, commenting, code style & organization, etc. -- Nikita Ivanov > On Wed, Jun 16, 2021 at 5:56 AM Andrey Gura wrote: > > Hi, > > I understand that this rule seems too strict or may be weird. But > removing the rule could lead to review comments like "use future > instead of fut". So my proposal is to change rule from "required" to > "recommended". > > On Wed, Jun 16, 2021 at 2:49 PM Valentin Kulichenko > wrote: >> >> Konstantin, thanks for chiming in. >> >> That's exactly my concern. Overformalization is generally not a good >> thing. >> Someone used "mess" to abbreviate "message"? That is surely not a good >> coding style, but that's what code reviews are for. I believe that our >> committers are more than capable of identifying such issues. >> >> At the same time, with the current rules, we are *forced* to use >> abbreviations like "locAddrGrpMgr", whether we like it or not. All it >> does >> is makes it harder to contribute to Ignite, without providing any >> clear >> value. >> >> -Val >> >> On Thu, Jun 10, 2021 at 9:46 AM Konstantin Orlov >> wrote: >> >>> +1 for making this optional >>> >>> Common abbreviation rules is good for sure, but sometimes I getting >>> sick >>> of all those locAddrGrpMgr. >>> >>> >>> -- >>> Regards, >>> Konstantin Orlov >>> >>> >>> >>> On 7 Jun 2021, at 14:31, Nikolay Izhikov wrote: Hello, Anton, Alexei. Thanks for the feedback. Personally, I’m pretty happy current abbreviation rules too. Let see what we can do to make our codebase even more consistent. > 7 июня 2021 г., в 13:23, Alexei Scherbakov < >>> alexey.scherbak...@gmail.com> написал(а): > > -1 > Common abbrevs add quality to the code. > > пн, 7 июн. 2021 г. в 12:38, Anton Vinogradov : > >> -1 here. >> We can fix the code and set up the rule. >> >> This will help to prevent having a weird abbreviation like "mess" >> (from >> "message") or "ign" (from "Ignite"). >> Also, the abbreviations list (hardcoded at IDEA plugin) allows to >> have >>> same >> names across the whole code, this should simplify the reading. >> >> On Sat, Jun 5, 2021 at 10:49 PM Valentin Kulichenko < >> valentin.kuliche...@gmail.com> wrote: >> >>> I also support removing this requirement. It’s not the first >>> time >>> someone >>> brings this up, and so far we haven’t been able to fix it. Not >>> worth >>> it >> in >>> my view. >>> >>> -Val >>> >>> On Sat, Jun 5, 2021 at 11:54 AM Nikolay Izhikov >>> >>> wrote: >>> Hello, guys. Thanks for the feedback. Dmitry, >>
Re: [DISCUSSION] Code style. Variable abbrevations
I'm sorry, but the rule is not strict and, moreover, not clear enough for constans. See here [1] ``` Type and method names are usually not abbreviated (except for the well-accepted abbreviations such as EOF, Impl, Fifo, etc.). Abbreviation applies to local variables, method parameters, class fields and in **some cases public static fileds** (constants) of the class. ``` But there is not any list that contains these cases. I suppose, that constant naming should not be abbreviated. As I see, the most cases of violations, described in initial post, are about constant's namings. I suppose that we should define strict and clear rules about constants naming. [1] -- https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules чт, 17 июн. 2021 г. в 10:10, Konstantin Orlov : > Enforced validation doesn't guarantee code consistency. No validation in > the world prevent me from typing manually "mess" instead of "msg"/"message" > or "svc" instead of "srvc"/"service" (btw "svc" is more common imo). > > And the fact that someone name a variable "count" instead of "cnt" doesn't > make the whole project immature. > > -- > Regards, > Konstantin Orlov > > > > > > On 17 Jun 2021, at 08:34, Ivan Pavlukhin wrote: > > > > Hi Nikolay, > > > > Thanks, it is rather interesting. > > > > Do we all agree that using different conventions for different code > > packages does not break "consistency"? Or did I get something wrong? > > > > 2021-06-17 7:12 GMT+03:00, Николай Ижиков : > >> Hello, Ivan. > >> > >> We can create checkstyle rule to enforce usage of abbreviations. > >> Internal/public code differs by package. > >> > >> PoC of rule [1] > >> > >> [1] https://github.com/apache/ignite/pull/9153 > >> > >>> 17 июня 2021 г., в 07:01, Ivan Pavlukhin > >>> написал(а): > >>> > >>> Nikita, > >>> > >>> Do you have a plan in your mind how to make Ignite codebase consistent? > >>> > >>> AFAIR, we had it intentionally inconsistent for a long time at least > >>> for one sake: for internal code we used special conventions (e.g. > >>> abbreviations, shorten getters) and common Java conventions for public > >>> API and examples (e.g. no abbreviations and usual getters). > >>> > >>> 2021-06-16 23:37 GMT+03:00, Nikita Ivanov : > Consistency is what makes it easier to contribute to the project and > attract new members. Consistency implies strong, well defined and > universally enforced rules. Just because we have some individuals who > are lazy or inexperienced - it does not mean that the entire project > should relax the basic-level engineering discipline. > > On a personal node - nothing screams "immaturity" louder that a code > that uses inconsistent naming, commenting, code style & organization, > etc. > -- > Nikita Ivanov > > > On Wed, Jun 16, 2021 at 5:56 AM Andrey Gura > wrote: > > > > Hi, > > > > I understand that this rule seems too strict or may be weird. But > > removing the rule could lead to review comments like "use future > > instead of fut". So my proposal is to change rule from "required" to > > "recommended". > > > > On Wed, Jun 16, 2021 at 2:49 PM Valentin Kulichenko > > wrote: > >> > >> Konstantin, thanks for chiming in. > >> > >> That's exactly my concern. Overformalization is generally not a good > >> thing. > >> Someone used "mess" to abbreviate "message"? That is surely not a > good > >> coding style, but that's what code reviews are for. I believe that > our > >> committers are more than capable of identifying such issues. > >> > >> At the same time, with the current rules, we are *forced* to use > >> abbreviations like "locAddrGrpMgr", whether we like it or not. All > it > >> does > >> is makes it harder to contribute to Ignite, without providing any > >> clear > >> value. > >> > >> -Val > >> > >> On Thu, Jun 10, 2021 at 9:46 AM Konstantin Orlov < > kor...@gridgain.com> > >> wrote: > >> > >>> +1 for making this optional > >>> > >>> Common abbreviation rules is good for sure, but sometimes I getting > >>> sick > >>> of all those locAddrGrpMgr. > >>> > >>> > >>> -- > >>> Regards, > >>> Konstantin Orlov > >>> > >>> > >>> > >>> > On 7 Jun 2021, at 14:31, Nikolay Izhikov > wrote: > > Hello, Anton, Alexei. > > Thanks for the feedback. > > Personally, I’m pretty happy current abbreviation rules too. > Let see what we can do to make our codebase even more consistent. > > > 7 июня 2021 г., в 13:23, Alexei Scherbakov < > >>> alexey.scherbak...@gmail.com> написал(а): > > > > -1 > > Common abbrevs add quality to the code. > > > > пн, 7 июн. 2021 г. в 12:38, Anton Vinogradov : > > > >> -1 here. > >> We can fix the
Re: [DISCUSSION] Code style. Variable abbrevations
> No validation in the world prevent me from typing manually "mess" instead of > "msg"/«message" Code review will do. > btw "svc" is more common imo Agree. I think you can start a discussion and change some abbrevs if you wish. > 17 июня 2021 г., в 10:23, Ivan Daschinsky написал(а): > > I'm sorry, but the rule is not strict and, moreover, not clear enough for > constans. See here [1] > ``` > Type and method names are usually not abbreviated (except for the > well-accepted abbreviations such as EOF, Impl, Fifo, etc.). > > Abbreviation applies to local variables, method parameters, class fields > and in **some cases public static fileds** (constants) of the class. > ``` > But there is not any list that contains these cases. I suppose, that > constant naming should not be abbreviated. > As I see, the most cases of violations, described in initial post, are > about constant's namings. > > I suppose that we should define strict and clear rules about constants > naming. > > [1] -- https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules > > > чт, 17 июн. 2021 г. в 10:10, Konstantin Orlov : > >> Enforced validation doesn't guarantee code consistency. No validation in >> the world prevent me from typing manually "mess" instead of "msg"/"message" >> or "svc" instead of "srvc"/"service" (btw "svc" is more common imo). >> >> And the fact that someone name a variable "count" instead of "cnt" doesn't >> make the whole project immature. >> >> -- >> Regards, >> Konstantin Orlov >> >> >> >> >>> On 17 Jun 2021, at 08:34, Ivan Pavlukhin wrote: >>> >>> Hi Nikolay, >>> >>> Thanks, it is rather interesting. >>> >>> Do we all agree that using different conventions for different code >>> packages does not break "consistency"? Or did I get something wrong? >>> >>> 2021-06-17 7:12 GMT+03:00, Николай Ижиков : Hello, Ivan. We can create checkstyle rule to enforce usage of abbreviations. Internal/public code differs by package. PoC of rule [1] [1] https://github.com/apache/ignite/pull/9153 > 17 июня 2021 г., в 07:01, Ivan Pavlukhin > написал(а): > > Nikita, > > Do you have a plan in your mind how to make Ignite codebase consistent? > > AFAIR, we had it intentionally inconsistent for a long time at least > for one sake: for internal code we used special conventions (e.g. > abbreviations, shorten getters) and common Java conventions for public > API and examples (e.g. no abbreviations and usual getters). > > 2021-06-16 23:37 GMT+03:00, Nikita Ivanov : >> Consistency is what makes it easier to contribute to the project and >> attract new members. Consistency implies strong, well defined and >> universally enforced rules. Just because we have some individuals who >> are lazy or inexperienced - it does not mean that the entire project >> should relax the basic-level engineering discipline. >> >> On a personal node - nothing screams "immaturity" louder that a code >> that uses inconsistent naming, commenting, code style & organization, >> etc. >> -- >> Nikita Ivanov >> >>> On Wed, Jun 16, 2021 at 5:56 AM Andrey Gura >> wrote: >>> >>> Hi, >>> >>> I understand that this rule seems too strict or may be weird. But >>> removing the rule could lead to review comments like "use future >>> instead of fut". So my proposal is to change rule from "required" to >>> "recommended". >>> >>> On Wed, Jun 16, 2021 at 2:49 PM Valentin Kulichenko >>> wrote: Konstantin, thanks for chiming in. That's exactly my concern. Overformalization is generally not a good thing. Someone used "mess" to abbreviate "message"? That is surely not a >> good coding style, but that's what code reviews are for. I believe that >> our committers are more than capable of identifying such issues. At the same time, with the current rules, we are *forced* to use abbreviations like "locAddrGrpMgr", whether we like it or not. All >> it does is makes it harder to contribute to Ignite, without providing any clear value. -Val On Thu, Jun 10, 2021 at 9:46 AM Konstantin Orlov < >> kor...@gridgain.com> wrote: > +1 for making this optional > > Common abbreviation rules is good for sure, but sometimes I getting > sick > of all those locAddrGrpMgr. > > > -- > Regards, > Konstantin Orlov > > > > >> On 7 Jun 2021, at 14:31, Nikolay Izhikov >> wrote: >> >> Hello, Anton, Alexei. >> >> Thanks for the feedback. >> >> Personally, I’m pretty happy current abbreviation rules too. >> L
Re: [DISCUSSION] Javadoc style requirements and checks in Ignite-3.
Hi, Unfortunately, such things as the "many private APIs" rule can't be formalized for any validation tool. So we have to choose between rules like "everything should be documented" and "private API documentation is not mandatory". The community prefers the second approach. I don't want to argue about it (although I prefer to document the private API ). My goal is getting a consistent and customizable way to validate javadoc where it is mandatory (public API). It also could be applied to private API, but it should be another rule set which doesn't require javadoc but validates existing javadoc. On Wed, Jun 16, 2021 at 11:32 PM Nikita Ivanov wrote: > > Hi, > In my strong opinion Javadoc is a code. Any errors in Javadoc are > equal to the bug in the code and must be treated as such. Proper and > extensive Javadoc for all public and many private APIs is absolutely > essential for this project, its growth and maturity. > > I'm surprised this community still needs to debate fundamental > engineering issues like that... > -- > Nikita Ivanov > > On Wed, Jun 16, 2021 at 4:47 AM Andrey Gura wrote: > > > > Hi, > > > > I think that scope should be limited by public API for beginning. > > Also I don't sure that we should support specific tags like @apiNote, > > @implSpec, @implNote. > > > > Let's move using the following plan: > > > > 1. Create an empty (effectively) checkstyle config for javadoc > > checking and commit it to the repository. After this step it will be > > possible to configure TC in order to use this configuration. > > 2. Configure TC. > > 3. Commit a non-empty checkstyle config, but all modules should be > > excluded from checking on this step. > > 4. For each module create a JIRA issue. Module maintainers fix > > corresponding tickets and remove module exclusion from checking. > > 5. Add information about javadoc validation targets to DEVNOTES.md > > file (could be done on step 3) > > > > Any objections? > > > > On Tue, Apr 20, 2021 at 11:40 AM Andrey Mashenkov > > wrote: > > > > > > I've fixed the PR. > > > > > > Now, > > > 1. Javadoc style is only checked if it is present for the element. All > > > declared javadoc tags MUST have a description. > > > So, one should either write correct javadoc or don't write at all. > > > 2. Additional checks performed for non-internal and non-test classes. All > > > public and protected elements must have non-emtpy javadoc. > > > > > > So, checkstyle detects 500+ missed descriptions (missed javadocs, tags > > > descriptions, tags themselves) in public scope > > > and 180+ bad-styles (missed brased, spaces, empty-lines) javadocs in whole > > > codebase. > > > > > > I'd suggest to force non-empty javadocs for all non-test classes including > > > internal (but their members) as well. > > > > > > > > > > > > On Mon, Apr 19, 2021 at 6:37 PM Andrey Mashenkov > > > > > > wrote: > > > > > > > Alexey, > > > > > > > > These are bad examples and unfortunately, most of the Ignite code > > > > doesn't > > > > look self-descriptive. > > > > (Do you feel the difference between @SerializableTransient and > > > > @TransientSerializable?) > > > > > > > > To understand the semantic of e.g. 'metricsHistSize' you have to analyze > > > > its usages. > > > > Getter and setter for this property look better, but it still not clear > > > > what happens with metric values above the limit? > > > > > > > > I have another example, one of the core components > > > > GridDhtPartitionDemander > > > > has totally useless javadoc. > > > > It is obviously has nothing with thread pool + "local cache" phrase > > > > looks > > > > very confusing. > > > > > > > > /** > > > > * Thread pool for requesting partitions from other nodes and > > > > populating local cache. > > > > */ > > > > public class GridDhtPartitionDemander > > > > > > > > To understand the purpose of this internal component I have to analyze > > > > its > > > > code and usages. > > > > However, if one will face unexpected behavior, he/she may be confused if > > > > it is a lack of javadoc or wrong behavior. > > > > > > > > There is no way to restrict or avoid such issues with any checks. > > > > Agree, meaningful javadocs as self-descriptive code is more about > > > > culture > > > > and discipline. > > > > > > > > The absence of javadoc on internal API, unreasonably raises the entry > > > > threshold for new developers and makes the development of > > > > intra-component > > > > interaction harder. > > > > I admit a high-level description for public classes may be enough to > > > > resolve this without pushing developers to write empty/useless docs for > > > > every method/field. > > > > > > > > > > > > On Mon, Apr 19, 2021 at 5:21 PM Alexey Kukushkin < > > > > kukushkinale...@gmail.com> wrote: > > > > > > > >> I personally do not understand why we need mandatory javadoc even in > > > >> public > > > >> modules. I think javadoc is needed only when the code is not > > > >> self-descriptive. What value does a javadoc li
Re: [VOTE] Release pyignite 0.5.0-rc1
+1 From me
Checked on Ubuntu 20.04 and windows 10
1. Installation from wheels for pythons 3.6 3.7 3.8 3.9
2. Native module work
3. Examples
Checked on Ubuntu 20.04 building from source package and correct work of
result package.
Checked all sha256 checksums and gpg signatures.
Let's extend voting period till June 18, 15:00 UTC
ср, 16 июн. 2021 г. в 17:34, Ivan Daschinsky :
> The vote will end at June, 17 15:00 UTC.
>
> ср, 16 июн. 2021 г. в 17:33, Ivan Daschinsky :
> >
> > Dear Igniters!
> >
> > Release candidate binaries for subj are uploaded and ready for vote
> > You can find them here:
> > https://dist.apache.org/repos/dist/dev/ignite/pyignite/0.5.0-rc1
> >
> > If you follow the link above, you will find source package (*.tar.gz and
> *.zip)
> > and binary packages (wheels) for windows (amd64) and linux (x86_64)
> > for pythons 36, 37, 38, 39. Also, there are sha512 and gpg signatures.
> > Code signing keys can be found here --
> https://downloads.apache.org/ignite/KEYS
> > Here you can find instructions how to verify packages
> > https://www.apache.org/info/verification.html
> >
> > You can install binary package for specific version of python using pip
> > For example do this on linux for python 3.8
> > >> pip install pyignite-0.5.0-cp38-cp38-manylinux1_x86_64.whl
> >
> > You can build and install package from source using this command:
> > >> pip install pyignite-0.5.0.tar.gz
> > You can build wheel on your platform using this command:
> > >> pip wheel --no-deps pyignite-0.5.0.tar.gz
> >
> > For building C module, you should have python headers and C compiler
> installed.
> > (i.e. for ubuntu sudo apt install build-essential python3-dev)
> > In Mac OS X xcode-tools and python from homebrew are the best option.
> >
> > In order to check whether C module works, use following:
> > >> from pyignite import _cutils
> > >> print(_cutils.hashcode('test'))
> > >> 3556498
> >
> > You can find documentation here:
> > https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.0.rc1
> >
> > You can find examples here (to check them, you should start ignite
> locally):
> >
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.0.rc1/examples.html
> > Also, examples can be found in source archive in examples subfolder.
> > docker-compose.yml is supplied in order to start ignite quickly. (Use
> > `docker-compose up -d` to start 3 nodes cluster and `docker-compose
> > down` to shut down it)
> >
> > Release notes:
> >
> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=blob;f=RELEASE_NOTES.txt;h=9d2ae81af2de22ce9e8c9d3b7ece14dd9e75ca0e;hb=61c83cb0ab6752f019518b4a2cb0724bd027755f
> >
> > Git release tag was created:
> >
> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=tag;h=refs/tags/0.5.0.rc1
> >
> > The vote is formal, see voting guidelines
> > https://www.apache.org/foundation/voting.html
> >
> > +1 - to accept pyignite-0.5.0-rc1
> > 0 - don't care either way
> > -1 - DO NOT accept pyignite-0.5.0-rc1
>
--
Sincerely yours, Ivan Daschinskiy
Re: Exceeding the DataStorageConfiguration#getMaxWalArchiveSize due to historical rebalance
Created the first task by this discussion IGNITE-14923. 13.05.2021, 18:37, "Stanislav Lukyanov" : > What I mean by degradation when archive size < min is that, for example, > historical rebalance is available for a smaller timespan than expected by the > system design. > It may not be an issue of course, especially for a new cluster. If > "degradation" is the wrong word we can call it "non-steady state" :) > In any case, I think we're on the same page. > >> On 11 May 2021, at 13:18, Andrey Gura wrote: >> >> Stan >> >>> If archive size is less than min or more than max then the system >>> functionality can degrade (e.g. historical rebalance may not work as >>> expected). >> >> Why does the condition "archive size is less than min" lead to system >> degradation? Actually, the described case is a normal situation for >> brand new clusters. >> >> I'm okay with the proposed minWalArchiveSize property. Looks like >> relatively understandable property. >> >> On Sun, May 9, 2021 at 7:12 PM Stanislav Lukyanov >> wrote: >>> Discuss this with Kirill verbally. >>> >>> Kirill showed me that having the min threshold doesn't quite work. >>> It doesn't work because we no longer know how much WAL we should remove if >>> we reach getMaxWalArchiveSize. >>> >>> For example, say we have minWalArchiveTimespan=2 hours and >>> maxWalArchiveSize=2GB. >>> Say, under normal load on stable topology 2 hours of WAL use 1 GB of space. >>> Now, say we're doing historical rebalance and reserve the WAL archive. >>> The WAL archive starts growing and soon it occupies 2 GB. >>> Now what? >>> We're supposed to give up WAL reservations and start agressively removing >>> WAL archive. >>> But it is not clear when can we stop removing WAL archive - since last 2 >>> hours of WAL are larger than our maxWalArchiveSize >>> there is no meaningful point the system can use as a "minimum" WAL size. >>> >>> I understand the description above is a bit messy but I believe that >>> whoever is interested in this will understand it >>> after drawing this on paper. >>> >>> I'm giving up on my latest suggestion about time-based minimum. Let's keep >>> it simple. >>> >>> I suggest the minWalArchiveSize and maxWalArchvieSize properties as the >>> solution, >>> with the behavior as initially described by Kirill. >>> >>> Stan >>> On 7 May 2021, at 15:09, ткаленко кирилл wrote: Stas hello! I didn't quite get your last idea. What will we do if we reach getMaxWalArchiveSize? Shall we not delete the segment until minWalArchiveTimespan? 06.05.2021, 20:00, "Stanislav Lukyanov" : > An interesting suggestion I heard today. > > The minWalArchiveSize property might actually be minWalArchiveTimespan - > i.e. be a number of seconds instead of a number of bytes! > > I think this makes perfect sense from the user point of view. > "I want to have WAL archive for at least N hours but I have a limit of M > gigabytes to store it". > > Do we have checkpoint timestamp stored anywhere? (cp start markers?) > Perhaps we can actually implement this? > > Thanks, > Stan > >> On 6 May 2021, at 14:13, Stanislav Lukyanov >> wrote: >> >> +1 to cancel WAL reservation on reaching getMaxWalArchiveSize >> +1 to add a public property to replace >> IGNITE_THRESHOLD_WAL_ARCHIVE_SIZE_PERCENTAGE >> >> I don't like the name getWalArchiveSize - I think it's a bit confusing >> (is it the current size? the minimal size? the target size?) >> I suggest to name the property geMintWalArchiveSize. I think that this >> is exactly what it is - the minimal size of the archive that we want to >> have. >> The archive size at all times should be between min and max. >> If archive size is less than min or more than max then the system >> functionality can degrade (e.g. historical rebalance may not work as >> expected). >> I think these rules are intuitively understood from the "min" and "max" >> names. >> >> Ilya's suggestion about throttling is great although I'd do this in a >> different ticket. >> >> Thanks, >> Stan >> >>> On 5 May 2021, at 19:25, Maxim Muzafarov wrote: >>> >>> Hello, Kirill >>> >>> +1 for this change, however, there are too many configuration settings >>> that exist for the user to configure Ignite cluster. It is better to >>> keep the options that we already have and fix the behaviour of the >>> rebalance process as you suggested. >>> >>> On Tue, 4 May 2021 at 19:01, ткаленко кирилл >>> wrote: Hi Ilya! Then we can greatly reduce the user load on the cluster until the rebalance is over. Which can be critical for the user. 04.05.2021, 18:43, "Ilya Kasnacheev" : > Hello! > > Maybe we ca
Re: [VOTE] Release pyignite 0.5.0-rc1
+1 from me
Best Regards,
Igor
On Thu, Jun 17, 2021 at 12:10 PM Ivan Daschinsky
wrote:
> +1 From me
> Checked on Ubuntu 20.04 and windows 10
> 1. Installation from wheels for pythons 3.6 3.7 3.8 3.9
> 2. Native module work
> 3. Examples
>
> Checked on Ubuntu 20.04 building from source package and correct work of
> result package.
>
> Checked all sha256 checksums and gpg signatures.
>
> Let's extend voting period till June 18, 15:00 UTC
>
>
>
> ср, 16 июн. 2021 г. в 17:34, Ivan Daschinsky :
>
> > The vote will end at June, 17 15:00 UTC.
> >
> > ср, 16 июн. 2021 г. в 17:33, Ivan Daschinsky :
> > >
> > > Dear Igniters!
> > >
> > > Release candidate binaries for subj are uploaded and ready for vote
> > > You can find them here:
> > > https://dist.apache.org/repos/dist/dev/ignite/pyignite/0.5.0-rc1
> > >
> > > If you follow the link above, you will find source package (*.tar.gz
> and
> > *.zip)
> > > and binary packages (wheels) for windows (amd64) and linux (x86_64)
> > > for pythons 36, 37, 38, 39. Also, there are sha512 and gpg signatures.
> > > Code signing keys can be found here --
> > https://downloads.apache.org/ignite/KEYS
> > > Here you can find instructions how to verify packages
> > > https://www.apache.org/info/verification.html
> > >
> > > You can install binary package for specific version of python using pip
> > > For example do this on linux for python 3.8
> > > >> pip install pyignite-0.5.0-cp38-cp38-manylinux1_x86_64.whl
> > >
> > > You can build and install package from source using this command:
> > > >> pip install pyignite-0.5.0.tar.gz
> > > You can build wheel on your platform using this command:
> > > >> pip wheel --no-deps pyignite-0.5.0.tar.gz
> > >
> > > For building C module, you should have python headers and C compiler
> > installed.
> > > (i.e. for ubuntu sudo apt install build-essential python3-dev)
> > > In Mac OS X xcode-tools and python from homebrew are the best option.
> > >
> > > In order to check whether C module works, use following:
> > > >> from pyignite import _cutils
> > > >> print(_cutils.hashcode('test'))
> > > >> 3556498
> > >
> > > You can find documentation here:
> > >
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.0.rc1
> > >
> > > You can find examples here (to check them, you should start ignite
> > locally):
> > >
> >
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.0.rc1/examples.html
> > > Also, examples can be found in source archive in examples subfolder.
> > > docker-compose.yml is supplied in order to start ignite quickly. (Use
> > > `docker-compose up -d` to start 3 nodes cluster and `docker-compose
> > > down` to shut down it)
> > >
> > > Release notes:
> > >
> >
> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=blob;f=RELEASE_NOTES.txt;h=9d2ae81af2de22ce9e8c9d3b7ece14dd9e75ca0e;hb=61c83cb0ab6752f019518b4a2cb0724bd027755f
> > >
> > > Git release tag was created:
> > >
> >
> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=tag;h=refs/tags/0.5.0.rc1
> > >
> > > The vote is formal, see voting guidelines
> > > https://www.apache.org/foundation/voting.html
> > >
> > > +1 - to accept pyignite-0.5.0-rc1
> > > 0 - don't care either way
> > > -1 - DO NOT accept pyignite-0.5.0-rc1
> >
>
>
> --
> Sincerely yours, Ivan Daschinskiy
>
Re: [VOTE] Release pyignite 0.5.0-rc1
+1
Checked pip install from tar.gz on Python 3.8 on Ubuntu 20.04, ran some of
the examples.
On Thu, Jun 17, 2021 at 2:32 PM Igor Sapego wrote:
> +1 from me
>
> Best Regards,
> Igor
>
>
> On Thu, Jun 17, 2021 at 12:10 PM Ivan Daschinsky
> wrote:
>
> > +1 From me
> > Checked on Ubuntu 20.04 and windows 10
> > 1. Installation from wheels for pythons 3.6 3.7 3.8 3.9
> > 2. Native module work
> > 3. Examples
> >
> > Checked on Ubuntu 20.04 building from source package and correct work of
> > result package.
> >
> > Checked all sha256 checksums and gpg signatures.
> >
> > Let's extend voting period till June 18, 15:00 UTC
> >
> >
> >
> > ср, 16 июн. 2021 г. в 17:34, Ivan Daschinsky :
> >
> > > The vote will end at June, 17 15:00 UTC.
> > >
> > > ср, 16 июн. 2021 г. в 17:33, Ivan Daschinsky :
> > > >
> > > > Dear Igniters!
> > > >
> > > > Release candidate binaries for subj are uploaded and ready for vote
> > > > You can find them here:
> > > > https://dist.apache.org/repos/dist/dev/ignite/pyignite/0.5.0-rc1
> > > >
> > > > If you follow the link above, you will find source package (*.tar.gz
> > and
> > > *.zip)
> > > > and binary packages (wheels) for windows (amd64) and linux (x86_64)
> > > > for pythons 36, 37, 38, 39. Also, there are sha512 and gpg
> signatures.
> > > > Code signing keys can be found here --
> > > https://downloads.apache.org/ignite/KEYS
> > > > Here you can find instructions how to verify packages
> > > > https://www.apache.org/info/verification.html
> > > >
> > > > You can install binary package for specific version of python using
> pip
> > > > For example do this on linux for python 3.8
> > > > >> pip install pyignite-0.5.0-cp38-cp38-manylinux1_x86_64.whl
> > > >
> > > > You can build and install package from source using this command:
> > > > >> pip install pyignite-0.5.0.tar.gz
> > > > You can build wheel on your platform using this command:
> > > > >> pip wheel --no-deps pyignite-0.5.0.tar.gz
> > > >
> > > > For building C module, you should have python headers and C compiler
> > > installed.
> > > > (i.e. for ubuntu sudo apt install build-essential python3-dev)
> > > > In Mac OS X xcode-tools and python from homebrew are the best option.
> > > >
> > > > In order to check whether C module works, use following:
> > > > >> from pyignite import _cutils
> > > > >> print(_cutils.hashcode('test'))
> > > > >> 3556498
> > > >
> > > > You can find documentation here:
> > > >
> > https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.0.rc1
> > > >
> > > > You can find examples here (to check them, you should start ignite
> > > locally):
> > > >
> > >
> >
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.0.rc1/examples.html
> > > > Also, examples can be found in source archive in examples subfolder.
> > > > docker-compose.yml is supplied in order to start ignite quickly. (Use
> > > > `docker-compose up -d` to start 3 nodes cluster and `docker-compose
> > > > down` to shut down it)
> > > >
> > > > Release notes:
> > > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=blob;f=RELEASE_NOTES.txt;h=9d2ae81af2de22ce9e8c9d3b7ece14dd9e75ca0e;hb=61c83cb0ab6752f019518b4a2cb0724bd027755f
> > > >
> > > > Git release tag was created:
> > > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=tag;h=refs/tags/0.5.0.rc1
> > > >
> > > > The vote is formal, see voting guidelines
> > > > https://www.apache.org/foundation/voting.html
> > > >
> > > > +1 - to accept pyignite-0.5.0-rc1
> > > > 0 - don't care either way
> > > > -1 - DO NOT accept pyignite-0.5.0-rc1
> > >
> >
> >
> > --
> > Sincerely yours, Ivan Daschinskiy
> >
>
IEP-75 Thin Client MsgPack Serialization for 3.0
Igniters, I have drafted an IEP on thin client serialization format [1], please review and let me know what you think. [1] https://cwiki.apache.org/confluence/display/IGNITE/IEP-75+Thin+Client+Serialization
Re: IEP-75 Thin Client MsgPack Serialization for 3.0
Hi, Pavel. Have you considered format with schema? Or schemaless of a candidate format was a prerequisite? As for me, msgpack is great, but I suppose that we should benchmark formats thoroughly. And not only for Java. чт, 17 июн. 2021 г. в 15:29, Pavel Tupitsyn : > Igniters, > > I have drafted an IEP on thin client serialization format [1], > please review and let me know what you think. > > [1] > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-75+Thin+Client+Serialization > -- Sincerely yours, Ivan Daschinskiy
Re: IEP-75 Thin Client MsgPack Serialization for 3.0
Could you please share your code for benchmarks? чт, 17 июн. 2021 г. в 15:56, Ivan Daschinsky : > Hi, Pavel. Have you considered format with schema? Or schemaless of a > candidate format was a prerequisite? > As for me, msgpack is great, but I suppose that we should benchmark > formats thoroughly. And not only for Java. > > чт, 17 июн. 2021 г. в 15:29, Pavel Tupitsyn : > >> Igniters, >> >> I have drafted an IEP on thin client serialization format [1], >> please review and let me know what you think. >> >> [1] >> >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-75+Thin+Client+Serialization >> > > > -- > Sincerely yours, Ivan Daschinskiy > -- Sincerely yours, Ivan Daschinskiy
Re: IEP-75 Thin Client MsgPack Serialization for 3.0
Ivan, > Have you considered format with schema? 1. We should be able to serialize arbitrary user data on the client side. I think we don't want to require extra steps from the user. 2. MsgPack can be also used in a schemaful way, when user objects are written as arrays, not as maps - so that field names are not included. > we should benchmark formats thoroughly Strictly speaking, the IEP is about the format (the spec), not about the implementation. The format itself is simple and efficient, there is nothing to make it slower than anything else. C# impl proves this by beating every competitor [1]. > Could you please share your code for benchmarks? The code is linked in the IEP [2] [1] https://aloiskraus.wordpress.com/2019/09/29/net-serialization-benchmark-2019-roundup/ [2] https://github.com/apache/ignite/pull/9178 On Thu, Jun 17, 2021 at 4:02 PM Ivan Daschinsky wrote: > Could you please share your code for benchmarks? > > чт, 17 июн. 2021 г. в 15:56, Ivan Daschinsky : > > > Hi, Pavel. Have you considered format with schema? Or schemaless of a > > candidate format was a prerequisite? > > As for me, msgpack is great, but I suppose that we should benchmark > > formats thoroughly. And not only for Java. > > > > чт, 17 июн. 2021 г. в 15:29, Pavel Tupitsyn : > > > >> Igniters, > >> > >> I have drafted an IEP on thin client serialization format [1], > >> please review and let me know what you think. > >> > >> [1] > >> > >> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-75+Thin+Client+Serialization > >> > > > > > > -- > > Sincerely yours, Ivan Daschinskiy > > > > > -- > Sincerely yours, Ivan Daschinskiy >
Re: IEP-75 Thin Client MsgPack Serialization for 3.0
Also, it's well known use case of msgpack in the world of memory grids -- tarantool.io uses msgpack for clients binary protocol [1] So writing connectors to tarantool is quite easy task. [1] -- https://www.tarantool.io/en/doc/latest/dev_guide/internals/box_protocol/ чт, 17 июн. 2021 г. в 16:15, Pavel Tupitsyn : > Ivan, > > > Have you considered format with schema? > > 1. We should be able to serialize arbitrary user data on the client side. > I think we don't want to require extra steps from the user. > > 2. MsgPack can be also used in a schemaful way, when user objects are > written as arrays, not as maps - so that field names are not included. > > > > we should benchmark formats thoroughly > > Strictly speaking, the IEP is about the format (the spec), not about the > implementation. > The format itself is simple and efficient, there is nothing to make it > slower than anything else. > C# impl proves this by beating every competitor [1]. > > > > Could you please share your code for benchmarks? > > The code is linked in the IEP [2] > > > [1] > > https://aloiskraus.wordpress.com/2019/09/29/net-serialization-benchmark-2019-roundup/ > [2] https://github.com/apache/ignite/pull/9178 > > On Thu, Jun 17, 2021 at 4:02 PM Ivan Daschinsky > wrote: > > > Could you please share your code for benchmarks? > > > > чт, 17 июн. 2021 г. в 15:56, Ivan Daschinsky : > > > > > Hi, Pavel. Have you considered format with schema? Or schemaless of a > > > candidate format was a prerequisite? > > > As for me, msgpack is great, but I suppose that we should benchmark > > > formats thoroughly. And not only for Java. > > > > > > чт, 17 июн. 2021 г. в 15:29, Pavel Tupitsyn : > > > > > >> Igniters, > > >> > > >> I have drafted an IEP on thin client serialization format [1], > > >> please review and let me know what you think. > > >> > > >> [1] > > >> > > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-75+Thin+Client+Serialization > > >> > > > > > > > > > -- > > > Sincerely yours, Ivan Daschinskiy > > > > > > > > > -- > > Sincerely yours, Ivan Daschinskiy > > > -- Sincerely yours, Ivan Daschinskiy
Re: [DISCUSSION] Add cache statistics switch to control script
Hi, Andrey Thanks for your comments, I have updated the ticket. Initially, I assumed the 'VisorCacheToggleStatisticsTask' should be used. But, as I see, VisorCacheToggleStatisticsTask doesn't perform any checks of cache existence [1], and this work is done by the IgniteVisorCmd [2]. Moreover, it seems that will be more convenient to pass regular expression for cache name (in a similar way as in '--cache list' command) instead of cache names list and add ability to show status of statistics switch (enabled / disabled), e.g.: --cache statistics enable|disable|status regexPattern or --cache metrics enable|disable|status regexPattern So, it seems that we should make another class for this purpose. > E.g. what if I pass the cache group name instead of the name of a particular cache? As I understand, we can enable / disable cache statistics via JMX and VisorCMD per cache, not per group. I think we should keep the same approach. > What if I will not pass any cache names? In case of an empty 'regexPattern', the control script should finish execution with an error. If no matching caches are found, the control script should print a message about matching caches' absence. > Also we don't use the word "statistics" anymore (while it is still called statistics in configuration). > What about the word "metrics"? I'm ok with the word "metrics" instead of the word "statistics" if no one minds. 1. https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/visor/cache/VisorCacheToggleStatisticsTask.java#L58 2. https://github.com/apache/ignite/blob/master/modules/visor-console/src/main/scala/org/apache/ignite/visor/commands/cache/VisorCacheCommand.scala#L304 ср, 16 июн. 2021 г. в 16:01, Andrey Gura : > Hi, Ilya > > Could you please provide more details about how it should work (in > issue description)? E.g. what if I pass the cache group name instead > of the name of a particular cache? What if I will not pass any cache > names? > > Also we don't use the word "statistics" anymore (while it is still > called statistics in configuration). What about the word "metrics"? > > On Wed, Jun 16, 2021 at 2:31 PM Shishkov Ilya > wrote: > > > > Dear Ignite Community! > > I propose to discuss the addition of a new feature for control script: > > cache statistics switch (enabling / disabling). Currently, cache > statistics > > can be toggled only via IgniteVisorCmd or JMX. > > I've created the corresponding issue [1] and suggest adding an extra > option > > to a '--cache' command of the control script, eg.: > > > > --cache statistics enable|disable [cacheName1,...,cacheNameN] > > > > Please, tell me if you have any suggestions or remarks against this > feature. > > > > 1. https://issues.apache.org/jira/browse/IGNITE-14913 >
Re: IEP-75 Thin Client MsgPack Serialization for 3.0
Hi Pavel, What you suggest looks promising: arbitrary object graph and platform independence aspects in particular. In IEP-54 we support only flat objects and only some standard types and assume inner objects of custom types will be serialized to byte[] somehow and their schema will not be managed by Ignite. And, we want to offer some default serializers to make it easier for end-user. AFAIU, MsgPack is suitable for the purpose as we don't want to invent yet another effective binary format. With an additional code harness, we can write object schema (field names) within the object itself. Is it right? I am just confused with "a schemaful way" and "field names are not included" in the same sentence. On Thu, Jun 17, 2021 at 4:46 PM Ivan Daschinsky wrote: > Also, it's well known use case of msgpack in the world of memory grids -- > tarantool.io uses msgpack for clients binary protocol [1] > So writing connectors to tarantool is quite easy task. > > [1] -- > https://www.tarantool.io/en/doc/latest/dev_guide/internals/box_protocol/ > > чт, 17 июн. 2021 г. в 16:15, Pavel Tupitsyn : > > > Ivan, > > > > > Have you considered format with schema? > > > > 1. We should be able to serialize arbitrary user data on the client side. > > I think we don't want to require extra steps from the user. > > > > 2. MsgPack can be also used in a schemaful way, when user objects are > > written as arrays, not as maps - so that field names are not included. > > > > > > > we should benchmark formats thoroughly > > > > Strictly speaking, the IEP is about the format (the spec), not about the > > implementation. > > The format itself is simple and efficient, there is nothing to make it > > slower than anything else. > > C# impl proves this by beating every competitor [1]. > > > > > > > Could you please share your code for benchmarks? > > > > The code is linked in the IEP [2] > > > > > > [1] > > > > > https://aloiskraus.wordpress.com/2019/09/29/net-serialization-benchmark-2019-roundup/ > > [2] https://github.com/apache/ignite/pull/9178 > > > > On Thu, Jun 17, 2021 at 4:02 PM Ivan Daschinsky > > wrote: > > > > > Could you please share your code for benchmarks? > > > > > > чт, 17 июн. 2021 г. в 15:56, Ivan Daschinsky : > > > > > > > Hi, Pavel. Have you considered format with schema? Or schemaless of a > > > > candidate format was a prerequisite? > > > > As for me, msgpack is great, but I suppose that we should benchmark > > > > formats thoroughly. And not only for Java. > > > > > > > > чт, 17 июн. 2021 г. в 15:29, Pavel Tupitsyn : > > > > > > > >> Igniters, > > > >> > > > >> I have drafted an IEP on thin client serialization format [1], > > > >> please review and let me know what you think. > > > >> > > > >> [1] > > > >> > > > >> > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-75+Thin+Client+Serialization > > > >> > > > > > > > > > > > > -- > > > > Sincerely yours, Ivan Daschinskiy > > > > > > > > > > > > > -- > > > Sincerely yours, Ivan Daschinskiy > > > > > > > > -- > Sincerely yours, Ivan Daschinskiy > -- Best regards, Andrey V. Mashenkov
Re: IEP-75 Thin Client MsgPack Serialization for 3.0
Andrey, i'm sorry but what do you mean as additional code harness? Usually, POJO is serialized simply as map. чт, 17 июн. 2021 г., 19:55 Andrey Mashenkov : > Hi Pavel, > > What you suggest looks promising: arbitrary object graph and platform > independence aspects in particular. > > In IEP-54 we support only flat objects and only some standard types and > assume inner objects of custom types will be serialized to byte[] somehow > and their schema will not be managed by Ignite. > And, we want to offer some default serializers to make it easier for > end-user. > > AFAIU, MsgPack is suitable for the purpose as we don't want to invent yet > another effective binary format. > With an additional code harness, we can write object schema (field names) > within the object itself. > Is it right? > > I am just confused with "a schemaful way" and "field names are not > included" in the same sentence. > > > On Thu, Jun 17, 2021 at 4:46 PM Ivan Daschinsky > wrote: > > > Also, it's well known use case of msgpack in the world of memory grids -- > > tarantool.io uses msgpack for clients binary protocol [1] > > So writing connectors to tarantool is quite easy task. > > > > [1] -- > > https://www.tarantool.io/en/doc/latest/dev_guide/internals/box_protocol/ > > > > чт, 17 июн. 2021 г. в 16:15, Pavel Tupitsyn : > > > > > Ivan, > > > > > > > Have you considered format with schema? > > > > > > 1. We should be able to serialize arbitrary user data on the client > side. > > > I think we don't want to require extra steps from the user. > > > > > > 2. MsgPack can be also used in a schemaful way, when user objects are > > > written as arrays, not as maps - so that field names are not included. > > > > > > > > > > we should benchmark formats thoroughly > > > > > > Strictly speaking, the IEP is about the format (the spec), not about > the > > > implementation. > > > The format itself is simple and efficient, there is nothing to make it > > > slower than anything else. > > > C# impl proves this by beating every competitor [1]. > > > > > > > > > > Could you please share your code for benchmarks? > > > > > > The code is linked in the IEP [2] > > > > > > > > > [1] > > > > > > > > > https://aloiskraus.wordpress.com/2019/09/29/net-serialization-benchmark-2019-roundup/ > > > [2] https://github.com/apache/ignite/pull/9178 > > > > > > On Thu, Jun 17, 2021 at 4:02 PM Ivan Daschinsky > > > wrote: > > > > > > > Could you please share your code for benchmarks? > > > > > > > > чт, 17 июн. 2021 г. в 15:56, Ivan Daschinsky : > > > > > > > > > Hi, Pavel. Have you considered format with schema? Or schemaless > of a > > > > > candidate format was a prerequisite? > > > > > As for me, msgpack is great, but I suppose that we should benchmark > > > > > formats thoroughly. And not only for Java. > > > > > > > > > > чт, 17 июн. 2021 г. в 15:29, Pavel Tupitsyn >: > > > > > > > > > >> Igniters, > > > > >> > > > > >> I have drafted an IEP on thin client serialization format [1], > > > > >> please review and let me know what you think. > > > > >> > > > > >> [1] > > > > >> > > > > >> > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-75+Thin+Client+Serialization > > > > >> > > > > > > > > > > > > > > > -- > > > > > Sincerely yours, Ivan Daschinskiy > > > > > > > > > > > > > > > > > -- > > > > Sincerely yours, Ivan Daschinskiy > > > > > > > > > > > > > -- > > Sincerely yours, Ivan Daschinskiy > > > > > -- > Best regards, > Andrey V. Mashenkov >
Re: IEP-75 Thin Client MsgPack Serialization for 3.0
Ivan, Ok, I've just thought if "fields are not included" then we need to bother about them by ourselves. чт, 17 июн. 2021 г., 20:10 Ivan Daschinsky : > Andrey, i'm sorry but what do you mean as additional code harness? Usually, > POJO is serialized simply as map. > > чт, 17 июн. 2021 г., 19:55 Andrey Mashenkov : > > > Hi Pavel, > > > > What you suggest looks promising: arbitrary object graph and platform > > independence aspects in particular. > > > > In IEP-54 we support only flat objects and only some standard types and > > assume inner objects of custom types will be serialized to byte[] somehow > > and their schema will not be managed by Ignite. > > And, we want to offer some default serializers to make it easier for > > end-user. > > > > AFAIU, MsgPack is suitable for the purpose as we don't want to invent yet > > another effective binary format. > > With an additional code harness, we can write object schema (field names) > > within the object itself. > > Is it right? > > > > I am just confused with "a schemaful way" and "field names are not > > included" in the same sentence. > > > > > > On Thu, Jun 17, 2021 at 4:46 PM Ivan Daschinsky > > wrote: > > > > > Also, it's well known use case of msgpack in the world of memory grids > -- > > > tarantool.io uses msgpack for clients binary protocol [1] > > > So writing connectors to tarantool is quite easy task. > > > > > > [1] -- > > > > https://www.tarantool.io/en/doc/latest/dev_guide/internals/box_protocol/ > > > > > > чт, 17 июн. 2021 г. в 16:15, Pavel Tupitsyn : > > > > > > > Ivan, > > > > > > > > > Have you considered format with schema? > > > > > > > > 1. We should be able to serialize arbitrary user data on the client > > side. > > > > I think we don't want to require extra steps from the user. > > > > > > > > 2. MsgPack can be also used in a schemaful way, when user objects are > > > > written as arrays, not as maps - so that field names are not > included. > > > > > > > > > > > > > we should benchmark formats thoroughly > > > > > > > > Strictly speaking, the IEP is about the format (the spec), not about > > the > > > > implementation. > > > > The format itself is simple and efficient, there is nothing to make > it > > > > slower than anything else. > > > > C# impl proves this by beating every competitor [1]. > > > > > > > > > > > > > Could you please share your code for benchmarks? > > > > > > > > The code is linked in the IEP [2] > > > > > > > > > > > > [1] > > > > > > > > > > > > > > https://aloiskraus.wordpress.com/2019/09/29/net-serialization-benchmark-2019-roundup/ > > > > [2] https://github.com/apache/ignite/pull/9178 > > > > > > > > On Thu, Jun 17, 2021 at 4:02 PM Ivan Daschinsky > > > > > wrote: > > > > > > > > > Could you please share your code for benchmarks? > > > > > > > > > > чт, 17 июн. 2021 г. в 15:56, Ivan Daschinsky >: > > > > > > > > > > > Hi, Pavel. Have you considered format with schema? Or schemaless > > of a > > > > > > candidate format was a prerequisite? > > > > > > As for me, msgpack is great, but I suppose that we should > benchmark > > > > > > formats thoroughly. And not only for Java. > > > > > > > > > > > > чт, 17 июн. 2021 г. в 15:29, Pavel Tupitsyn < > ptupit...@apache.org > > >: > > > > > > > > > > > >> Igniters, > > > > > >> > > > > > >> I have drafted an IEP on thin client serialization format [1], > > > > > >> please review and let me know what you think. > > > > > >> > > > > > >> [1] > > > > > >> > > > > > >> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-75+Thin+Client+Serialization > > > > > >> > > > > > > > > > > > > > > > > > > -- > > > > > > Sincerely yours, Ivan Daschinskiy > > > > > > > > > > > > > > > > > > > > > -- > > > > > Sincerely yours, Ivan Daschinskiy > > > > > > > > > > > > > > > > > > -- > > > Sincerely yours, Ivan Daschinskiy > > > > > > > > > -- > > Best regards, > > Andrey V. Mashenkov > > >
Re: IEP-75 Thin Client MsgPack Serialization for 3.0
Andrey, >> arbitrary object graph Also, that is not true, msgpack format doesn't handle circular graphs. Think about msgpack as binary json. You couldn't understand full structure of message if you didn't deserialize it fully before, maps and arrays are serialized just as contiguos chunks of values/kv-pairs. Msgpack is a really dumb and simple format. Also, as for me, I cannot understand why current ignite serialization (BinaryObjectBuilder or Binarilizable) is slower than raw message pack serializer. I suppose that this is an issue and we should investigate it. Pavel, why do you use PooledMessageBufferOutput in benchmarks? I'm sorry, but is it fair to use it? >> The code is linked in the IEP [2] Double checked -- there is not any links to PR either in IEP or in jira issue
Re: IEP-75 Thin Client MsgPack Serialization for 3.0
>> Double checked -- there is not any links to PR either in IEP or in jira issue Sorry, there is a link in IEP, but not in jira ticket. чт, 17 июн. 2021 г. в 21:39, Ivan Daschinsky : > Andrey, > >> arbitrary object graph > Also, that is not true, msgpack format doesn't handle circular graphs. > Think about msgpack as binary json. You couldn't understand full structure > of message if you didn't deserialize it fully before, maps and arrays are > serialized just as contiguos chunks > of values/kv-pairs. Msgpack is a really dumb and simple format. > > Also, as for me, I cannot understand why current ignite serialization > (BinaryObjectBuilder or Binarilizable) is slower than raw message pack > serializer. > I suppose that this is an issue and we should investigate it. > > Pavel, why do you use PooledMessageBufferOutput in benchmarks? I'm > sorry, but is it fair to use it? > > >> The code is linked in the IEP [2] > Double checked -- there is not any links to PR either in IEP or in jira > issue > -- Sincerely yours, Ivan Daschinskiy
Re: IEP-75 Thin Client MsgPack Serialization for 3.0
Ivan, thankd for clarification. Binarilizable interface forces user to write serialization code. We can support this or similar interface. But I'd like Ignite has some default serializer in addition. It can be also useful e.g. in compute for param and result serialization. BinaryObjectBuider requires an Ignite node for object construction, but we are looking for a detached builder and won't care about schemas. AFAIR, BinaryObject creates an objectReader on every single field read operation. So, BO solution produces a lot of garbage and BO has noticable overhead which affects the object footprint. чт, 17 июн. 2021 г., 21:41 Ivan Daschinsky : > >> Double checked -- there is not any links to PR either in IEP or in jira > issue > Sorry, there is a link in IEP, but not in jira ticket. > > чт, 17 июн. 2021 г. в 21:39, Ivan Daschinsky : > > > Andrey, > > >> arbitrary object graph > > Also, that is not true, msgpack format doesn't handle circular graphs. > > Think about msgpack as binary json. You couldn't understand full > structure > > of message if you didn't deserialize it fully before, maps and arrays are > > serialized just as contiguos chunks > > of values/kv-pairs. Msgpack is a really dumb and simple format. > > > > Also, as for me, I cannot understand why current ignite serialization > > (BinaryObjectBuilder or Binarilizable) is slower than raw message pack > > serializer. > > I suppose that this is an issue and we should investigate it. > > > > Pavel, why do you use PooledMessageBufferOutput in benchmarks? I'm > > sorry, but is it fair to use it? > > > > >> The code is linked in the IEP [2] > > Double checked -- there is not any links to PR either in IEP or in jira > > issue > > > > > -- > Sincerely yours, Ivan Daschinskiy >
Re: IEP-75 Thin Client MsgPack Serialization for 3.0
Andrey, here we discuss serialization format, as far as I understand. Current implementation of ignite binary object serialization can be rewritten. If we do not care about fast (O(1)) field lookup, about schema validation and so on, msgpack is a really good option. It is also good for client binary protocol, i.e. tarantool uses it. >> Binarilizable interface forces user to write serialization code I am talking about speed comparison. You can see from Pavel's data, jackson-msgpack shows a pathetic performance comparing with a ignite's default binary marshaller. If you want really fast serialization -- the only option is to write code by yourself or use code generation. Default packer from msgpack-core java package is similar to BinaryWriter. So I am wondering why packer from msgpack-core show better performance than BinaryWriter. And I suppose that benchmark is not quite fair. чт, 17 июн. 2021 г. в 22:19, Andrey Mashenkov : > Ivan, thankd for clarification. > > Binarilizable interface forces user to write serialization code. We can > support this or similar interface. > But I'd like Ignite has some default serializer in addition. It can be also > useful e.g. in compute for param and result serialization. > > BinaryObjectBuider requires an Ignite node for object construction, but we > are looking for a detached builder and won't care about schemas. > > AFAIR, BinaryObject creates an objectReader on every single field read > operation. > So, BO solution produces a lot of garbage and BO has noticable overhead > which affects the object footprint. > > чт, 17 июн. 2021 г., 21:41 Ivan Daschinsky : > > > >> Double checked -- there is not any links to PR either in IEP or in > jira > > issue > > Sorry, there is a link in IEP, but not in jira ticket. > > > > чт, 17 июн. 2021 г. в 21:39, Ivan Daschinsky : > > > > > Andrey, > > > >> arbitrary object graph > > > Also, that is not true, msgpack format doesn't handle circular graphs. > > > Think about msgpack as binary json. You couldn't understand full > > structure > > > of message if you didn't deserialize it fully before, maps and arrays > are > > > serialized just as contiguos chunks > > > of values/kv-pairs. Msgpack is a really dumb and simple format. > > > > > > Also, as for me, I cannot understand why current ignite serialization > > > (BinaryObjectBuilder or Binarilizable) is slower than raw message pack > > > serializer. > > > I suppose that this is an issue and we should investigate it. > > > > > > Pavel, why do you use PooledMessageBufferOutput in benchmarks? I'm > > > sorry, but is it fair to use it? > > > > > > >> The code is linked in the IEP [2] > > > Double checked -- there is not any links to PR either in IEP or in jira > > > issue > > > > > > > > > -- > > Sincerely yours, Ivan Daschinskiy > > > -- Sincerely yours, Ivan Daschinskiy
Re: [VOTE] Release pyignite 0.5.0-rc1
+ 1 for me. Checked for Mac OS -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
