Re: How to free up space on disc after removing entries from IgniteCache with enabled PDS?
Anton, Switching a partition to and from the SHRINKING state will require intricate synchronizations in order to properly determine the start position for historical rebalance without PME. I would still go with an offline-node approach, but instead of cleaning the persistence, we can do effective defragmentation when the node is offline because we are sure that there is no concurrent load. After the defragmentation completes, we bring the node back to the cluster and historical rebalance will kick in automatically. It will still require manual node restarts, but since the data is not removed, there are no additional risks. Also, this will be an excellent solution for those who can afford downtime and execute the defragment command on all nodes in the cluster simultaneously - this will be the fastest way possible. --AG пн, 30 сент. 2019 г. в 09:29, Anton Vinogradov : > Alexei, > >> stopping fragmented node and removing partition data, then starting it > again > > That's exactly what we're doing to solve the fragmentation issue. > The problem here is that we have to perform N/B restart-rebalance > operations (N - cluster size, B - backups count) and it takes a lot of time > with risks to lose the data. > > On Fri, Sep 27, 2019 at 5:49 PM Alexei Scherbakov < > alexey.scherbak...@gmail.com> wrote: > > > Probably this should be allowed to do using public API, actually this is > > same as manual rebalancing. > > > > пт, 27 сент. 2019 г. в 17:40, Alexei Scherbakov < > > alexey.scherbak...@gmail.com>: > > > > > The poor man's solution for the problem would be stopping fragmented > node > > > and removing partition data, then starting it again allowing full state > > > transfer already without deletes. > > > Rinse and repeat for all owners. > > > > > > Anton Vinogradov, would this work for you as workaround ? > > > > > > чт, 19 сент. 2019 г. в 13:03, Anton Vinogradov : > > > > > >> Alexey, > > >> > > >> Let's combine your and Ivan's proposals. > > >> > > >> >> vacuum command, which acquires exclusive table lock, so no > concurrent > > >> activities on the table are possible. > > >> and > > >> >> Could the problem be solved by stopping a node which needs to be > > >> defragmented, clearing persistence files and restarting the node? > > >> >> After rebalancing the node will receive all data back without > > >> fragmentation. > > >> > > >> How about to have special partition state SHRINKING? > > >> This state should mean that partition unavailable for reads and > updates > > >> but > > >> should keep it's update-counters and should not be marked as lost, > > renting > > >> or evicted. > > >> At this state we able to iterate over the partition and apply it's > > entries > > >> to another file in a compact way. > > >> Indices should be updated during the copy-on-shrink procedure or at > the > > >> shrink completion. > > >> Once shrank file is ready we should replace the original partition > file > > >> with it and mark it as MOVING which will start the historical > rebalance. > > >> Shrinking should be performed during the low activity periods, but > even > > in > > >> case we found that activity was high and historical rebalance is not > > >> suitable we may just remove the file and use regular rebalance to > > restore > > >> the partition (this will also lead to shrink). > > >> > > >> BTW, seems, we able to implement partition shrink in a cheap way. > > >> We may just use rebalancing code to apply fat partition's entries to > the > > >> new file. > > >> So, 3 stages here: local rebalance, indices update and global > historical > > >> rebalance. > > >> > > >> On Thu, Sep 19, 2019 at 11:43 AM Alexey Goncharuk < > > >> alexey.goncha...@gmail.com> wrote: > > >> > > >> > Anton, > > >> > > > >> > > > >> > > >> The solution which Anton suggested does not look easy because > it > > >> will > > >> > > most likely significantly hurt performance > > >> > > Mostly agree here, but what drop do we expect? What price do we > > ready > > >> to > > >> > > pay? > > >> > > Not sure, but seems some vendors ready to pay, for example, 5% > drop > > >> for > > >> > > this. > > >> > > > >> > 5% may be a big drop for some use-cases, so I think we should look > at > > >> how > > >> > to improve performance, not how to make it worse. > > >> > > > >> > > > >> > > > > >> > > >> it is hard to maintain a data structure to choose "page from > > >> free-list > > >> > > with enough space closest to the beginning of the file". > > >> > > We can just split each free-list bucket to the couple and use > first > > >> for > > >> > > pages in the first half of the file and the second for the last. > > >> > > Only two buckets required here since, during the file shrink, > first > > >> > > bucket's window will be shrank too. > > >> > > Seems, this give us the same price on put, just use the first > bucket > > >> in > > >> > > case it's not empty. > > >> > > Remove price (with merge) will be increased, of course. > > >> > > > > >> > > The compromise solut
Re: How to free up space on disc after removing entries from IgniteCache with enabled PDS?
Alexey, Sounds good to me. On Thu, Oct 3, 2019 at 10:51 AM Alexey Goncharuk wrote: > Anton, > > Switching a partition to and from the SHRINKING state will require > intricate synchronizations in order to properly determine the start > position for historical rebalance without PME. > > I would still go with an offline-node approach, but instead of cleaning the > persistence, we can do effective defragmentation when the node is offline > because we are sure that there is no concurrent load. After the > defragmentation completes, we bring the node back to the cluster and > historical rebalance will kick in automatically. It will still require > manual node restarts, but since the data is not removed, there are no > additional risks. Also, this will be an excellent solution for those who > can afford downtime and execute the defragment command on all nodes in the > cluster simultaneously - this will be the fastest way possible. > > --AG > > пн, 30 сент. 2019 г. в 09:29, Anton Vinogradov : > > > Alexei, > > >> stopping fragmented node and removing partition data, then starting it > > again > > > > That's exactly what we're doing to solve the fragmentation issue. > > The problem here is that we have to perform N/B restart-rebalance > > operations (N - cluster size, B - backups count) and it takes a lot of > time > > with risks to lose the data. > > > > On Fri, Sep 27, 2019 at 5:49 PM Alexei Scherbakov < > > alexey.scherbak...@gmail.com> wrote: > > > > > Probably this should be allowed to do using public API, actually this > is > > > same as manual rebalancing. > > > > > > пт, 27 сент. 2019 г. в 17:40, Alexei Scherbakov < > > > alexey.scherbak...@gmail.com>: > > > > > > > The poor man's solution for the problem would be stopping fragmented > > node > > > > and removing partition data, then starting it again allowing full > state > > > > transfer already without deletes. > > > > Rinse and repeat for all owners. > > > > > > > > Anton Vinogradov, would this work for you as workaround ? > > > > > > > > чт, 19 сент. 2019 г. в 13:03, Anton Vinogradov : > > > > > > > >> Alexey, > > > >> > > > >> Let's combine your and Ivan's proposals. > > > >> > > > >> >> vacuum command, which acquires exclusive table lock, so no > > concurrent > > > >> activities on the table are possible. > > > >> and > > > >> >> Could the problem be solved by stopping a node which needs to be > > > >> defragmented, clearing persistence files and restarting the node? > > > >> >> After rebalancing the node will receive all data back without > > > >> fragmentation. > > > >> > > > >> How about to have special partition state SHRINKING? > > > >> This state should mean that partition unavailable for reads and > > updates > > > >> but > > > >> should keep it's update-counters and should not be marked as lost, > > > renting > > > >> or evicted. > > > >> At this state we able to iterate over the partition and apply it's > > > entries > > > >> to another file in a compact way. > > > >> Indices should be updated during the copy-on-shrink procedure or at > > the > > > >> shrink completion. > > > >> Once shrank file is ready we should replace the original partition > > file > > > >> with it and mark it as MOVING which will start the historical > > rebalance. > > > >> Shrinking should be performed during the low activity periods, but > > even > > > in > > > >> case we found that activity was high and historical rebalance is not > > > >> suitable we may just remove the file and use regular rebalance to > > > restore > > > >> the partition (this will also lead to shrink). > > > >> > > > >> BTW, seems, we able to implement partition shrink in a cheap way. > > > >> We may just use rebalancing code to apply fat partition's entries to > > the > > > >> new file. > > > >> So, 3 stages here: local rebalance, indices update and global > > historical > > > >> rebalance. > > > >> > > > >> On Thu, Sep 19, 2019 at 11:43 AM Alexey Goncharuk < > > > >> alexey.goncha...@gmail.com> wrote: > > > >> > > > >> > Anton, > > > >> > > > > >> > > > > >> > > >> The solution which Anton suggested does not look easy > because > > it > > > >> will > > > >> > > most likely significantly hurt performance > > > >> > > Mostly agree here, but what drop do we expect? What price do we > > > ready > > > >> to > > > >> > > pay? > > > >> > > Not sure, but seems some vendors ready to pay, for example, 5% > > drop > > > >> for > > > >> > > this. > > > >> > > > > >> > 5% may be a big drop for some use-cases, so I think we should look > > at > > > >> how > > > >> > to improve performance, not how to make it worse. > > > >> > > > > >> > > > > >> > > > > > >> > > >> it is hard to maintain a data structure to choose "page from > > > >> free-list > > > >> > > with enough space closest to the beginning of the file". > > > >> > > We can just split each free-list bucket to the couple and use > > first > > > >> for > > > >> > > pages in the first half of the file and the second for the last. > > > >> > > Only
Re: How to free up space on disc after removing entries from IgniteCache with enabled PDS?
Anton, do you think we should file a single ticket for this or should we go with an IEP? As of now, the change does not look big enough for an IEP for me. чт, 3 окт. 2019 г. в 11:18, Anton Vinogradov : > Alexey, > > Sounds good to me. > > On Thu, Oct 3, 2019 at 10:51 AM Alexey Goncharuk < > alexey.goncha...@gmail.com> > wrote: > > > Anton, > > > > Switching a partition to and from the SHRINKING state will require > > intricate synchronizations in order to properly determine the start > > position for historical rebalance without PME. > > > > I would still go with an offline-node approach, but instead of cleaning > the > > persistence, we can do effective defragmentation when the node is offline > > because we are sure that there is no concurrent load. After the > > defragmentation completes, we bring the node back to the cluster and > > historical rebalance will kick in automatically. It will still require > > manual node restarts, but since the data is not removed, there are no > > additional risks. Also, this will be an excellent solution for those who > > can afford downtime and execute the defragment command on all nodes in > the > > cluster simultaneously - this will be the fastest way possible. > > > > --AG > > > > пн, 30 сент. 2019 г. в 09:29, Anton Vinogradov : > > > > > Alexei, > > > >> stopping fragmented node and removing partition data, then starting > it > > > again > > > > > > That's exactly what we're doing to solve the fragmentation issue. > > > The problem here is that we have to perform N/B restart-rebalance > > > operations (N - cluster size, B - backups count) and it takes a lot of > > time > > > with risks to lose the data. > > > > > > On Fri, Sep 27, 2019 at 5:49 PM Alexei Scherbakov < > > > alexey.scherbak...@gmail.com> wrote: > > > > > > > Probably this should be allowed to do using public API, actually this > > is > > > > same as manual rebalancing. > > > > > > > > пт, 27 сент. 2019 г. в 17:40, Alexei Scherbakov < > > > > alexey.scherbak...@gmail.com>: > > > > > > > > > The poor man's solution for the problem would be stopping > fragmented > > > node > > > > > and removing partition data, then starting it again allowing full > > state > > > > > transfer already without deletes. > > > > > Rinse and repeat for all owners. > > > > > > > > > > Anton Vinogradov, would this work for you as workaround ? > > > > > > > > > > чт, 19 сент. 2019 г. в 13:03, Anton Vinogradov : > > > > > > > > > >> Alexey, > > > > >> > > > > >> Let's combine your and Ivan's proposals. > > > > >> > > > > >> >> vacuum command, which acquires exclusive table lock, so no > > > concurrent > > > > >> activities on the table are possible. > > > > >> and > > > > >> >> Could the problem be solved by stopping a node which needs to > be > > > > >> defragmented, clearing persistence files and restarting the node? > > > > >> >> After rebalancing the node will receive all data back without > > > > >> fragmentation. > > > > >> > > > > >> How about to have special partition state SHRINKING? > > > > >> This state should mean that partition unavailable for reads and > > > updates > > > > >> but > > > > >> should keep it's update-counters and should not be marked as lost, > > > > renting > > > > >> or evicted. > > > > >> At this state we able to iterate over the partition and apply it's > > > > entries > > > > >> to another file in a compact way. > > > > >> Indices should be updated during the copy-on-shrink procedure or > at > > > the > > > > >> shrink completion. > > > > >> Once shrank file is ready we should replace the original partition > > > file > > > > >> with it and mark it as MOVING which will start the historical > > > rebalance. > > > > >> Shrinking should be performed during the low activity periods, but > > > even > > > > in > > > > >> case we found that activity was high and historical rebalance is > not > > > > >> suitable we may just remove the file and use regular rebalance to > > > > restore > > > > >> the partition (this will also lead to shrink). > > > > >> > > > > >> BTW, seems, we able to implement partition shrink in a cheap way. > > > > >> We may just use rebalancing code to apply fat partition's entries > to > > > the > > > > >> new file. > > > > >> So, 3 stages here: local rebalance, indices update and global > > > historical > > > > >> rebalance. > > > > >> > > > > >> On Thu, Sep 19, 2019 at 11:43 AM Alexey Goncharuk < > > > > >> alexey.goncha...@gmail.com> wrote: > > > > >> > > > > >> > Anton, > > > > >> > > > > > >> > > > > > >> > > >> The solution which Anton suggested does not look easy > > because > > > it > > > > >> will > > > > >> > > most likely significantly hurt performance > > > > >> > > Mostly agree here, but what drop do we expect? What price do > we > > > > ready > > > > >> to > > > > >> > > pay? > > > > >> > > Not sure, but seems some vendors ready to pay, for example, 5% > > > drop > > > > >> for > > > > >> > > this. > > > > >> > > > > > >> > 5% may be a big drop for some use-cases, s
Re: Replacing default work dir from tmp to current dir
Yes, my expectation was "~" to be the user home as well, too bad we missed this during the review. The current process directory is broken because a simple "cd" will make the persistent data disappear, which may be even worse than the temp directory. чт, 3 окт. 2019 г. в 01:45, Denis Magda : > I was always expecting this to be a user *home* directory that can be > resolved in any operating system and will work for any language supported > by Ignite. So, I'm with Pavel here. > > Alex G, what's your thinking? Sounds like we need to change this one more > time. > > - > Denis > > > On Wed, Oct 2, 2019 at 12:17 PM Pavel Tupitsyn > wrote: > > > Everyone above agreed to `~/ignite/work`, then somehow we jumped to > > `user.dir/ignite/work`. > > To me `user.dir` looked like synonym for ~, but turns out this is not > true. > > I think others may be confused in the same way. > > > > Denis Magda, Alexey Goncharuk, and others - please confirm that you > > understand that `user.dir` means current directory, not user home > > directory. > > > > In my opinion, this is very broken. Current work dir can be literally > > anything, e.g.: > > `cd / && ~/my-ignite-app/run.sh` will cause an attempt to create ignite > dir > > in system root, and so on. > > > > > > > > On Wed, Oct 2, 2019 at 9:46 PM Ilya Kasnacheev < > ilya.kasnach...@gmail.com> > > wrote: > > > > > Hello! > > > > > > I think this is a sensible default and it was certainly not chosen by > > > mistake. It was intentional expectation that your project is started > from > > > project root and data is located under it. > > > > > > If this breaks .Net, I am deeply sorry. > > > However, I think we should change .net to provide non-default workdir > > > location when none is specified. > > > > > > Can you please clarify scenarios that are broken now? > > > > > > Regards, > > > > > > ср, 2 окт. 2019 г., 20:28 Pavel Tupitsyn : > > > > > > > Igniters, > > > > > > > > Looks like we made a mistake while implementing IGNITE-12057: > > > > `user.dir` is NOT user home directory, it is where JVM has been > started > > > > from, which is rather arbitrary. > > > > (Among other things this breaks Ignite.NET usage from tools like > > LINQPad, > > > > because `user.dir` ends up pointing to Program Files, which is not > > > > writable without elevation) > > > > > > > > We should use `user.home` system property instead, see > > > > > > > > > > https://docs.oracle.com/javase/tutorial/essential/environment/sysprop.html > > > > > > > > Thoughts, objections? > > > > > > > > On Mon, Sep 2, 2019 at 1:57 PM Ilya Kasnacheev < > > > ilya.kasnach...@gmail.com> > > > > wrote: > > > > > > > > > Hello again! > > > > > > > > > > Please note that I have updated release notes for IGNITE-12057 as > > well > > > as > > > > > added them for my ticket. Release Engineers, please make sure you > > > include > > > > > the latest one. > > > > > > > > > > Regards, > > > > > -- > > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > пн, 2 сент. 2019 г. в 13:33, Ilya Kasnacheev < > > > ilya.kasnach...@gmail.com > > > > >: > > > > > > > > > > > Hello! > > > > > > > > > > > > I have pushed an amended fix to both master and ignite-2.7.6. > > > > > > > > > > > > Regards, > > > > > > -- > > > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > > > > пт, 30 авг. 2019 г. в 21:48, Denis Magda : > > > > > > > > > > > >> Ilya, > > > > > >> > > > > > >> I forgot to push "Send for review" button. You can see my minor > > > > comment > > > > > >> now. > > > > > >> > > > > > >> - > > > > > >> Denis > > > > > >> > > > > > >> > > > > > >> On Fri, Aug 30, 2019 at 5:47 AM Ilya Kasnacheev < > > > > > >> ilya.kasnach...@gmail.com> > > > > > >> wrote: > > > > > >> > > > > > >> > Hello! > > > > > >> > > > > > > >> > Waiting for a minor comment from Denis, as soon as I see/fix > it > > > I'm > > > > > >> going > > > > > >> > to commit. > > > > > >> > > > > > > >> > Regards, > > > > > >> > Ilya. > > > > > >> > -- > > > > > >> > Ilya Kasnacheev > > > > > >> > > > > > > >> > > > > > > >> > пт, 30 авг. 2019 г. в 11:30, Alexey Goncharuk < > > > > > >> alexey.goncha...@gmail.com > > > > > >> > >: > > > > > >> > > > > > > >> > > Hello Ilya, > > > > > >> > > > > > > > >> > > Just curious, when are you planning to commit your changes > to > > > the > > > > > >> 2.7.6 > > > > > >> > > branch? > > > > > >> > > > > > > > >> > > ср, 28 авг. 2019 г. в 04:57, Denis Magda >: > > > > > >> > > > > > > > >> > > > Ok, seems like we came to a consensus. Let’s ensure that > the > > > > path > > > > > >> for > > > > > >> > our > > > > > >> > > > work dir is user.dir/ignite/work and restart the vote. > > > > > >> > > > > > > > > >> > > > Denis > > > > > >> > > > > > > > > >> > > > On Tuesday, August 27, 2019, Ilya Kasnacheev < > > > > > >> > ilya.kasnach...@gmail.com> > > > > > >> > > > wrote: > > > > > >> > > > > > > > > >> > > > > Hello! > > > > > >> > > > > > > > > > >> > > > > I have took the liberty to implement the change to > > existing > > > > cod
Re: How to free up space on disc after removing entries from IgniteCache with enabled PDS?
Alexey, As for me, it does not matter will it be IEP, umbrella or a single issue. The most important thing is Assignee :) On Thu, Oct 3, 2019 at 11:59 AM Alexey Goncharuk wrote: > Anton, do you think we should file a single ticket for this or should we go > with an IEP? As of now, the change does not look big enough for an IEP for > me. > > чт, 3 окт. 2019 г. в 11:18, Anton Vinogradov : > > > Alexey, > > > > Sounds good to me. > > > > On Thu, Oct 3, 2019 at 10:51 AM Alexey Goncharuk < > > alexey.goncha...@gmail.com> > > wrote: > > > > > Anton, > > > > > > Switching a partition to and from the SHRINKING state will require > > > intricate synchronizations in order to properly determine the start > > > position for historical rebalance without PME. > > > > > > I would still go with an offline-node approach, but instead of cleaning > > the > > > persistence, we can do effective defragmentation when the node is > offline > > > because we are sure that there is no concurrent load. After the > > > defragmentation completes, we bring the node back to the cluster and > > > historical rebalance will kick in automatically. It will still require > > > manual node restarts, but since the data is not removed, there are no > > > additional risks. Also, this will be an excellent solution for those > who > > > can afford downtime and execute the defragment command on all nodes in > > the > > > cluster simultaneously - this will be the fastest way possible. > > > > > > --AG > > > > > > пн, 30 сент. 2019 г. в 09:29, Anton Vinogradov : > > > > > > > Alexei, > > > > >> stopping fragmented node and removing partition data, then > starting > > it > > > > again > > > > > > > > That's exactly what we're doing to solve the fragmentation issue. > > > > The problem here is that we have to perform N/B restart-rebalance > > > > operations (N - cluster size, B - backups count) and it takes a lot > of > > > time > > > > with risks to lose the data. > > > > > > > > On Fri, Sep 27, 2019 at 5:49 PM Alexei Scherbakov < > > > > alexey.scherbak...@gmail.com> wrote: > > > > > > > > > Probably this should be allowed to do using public API, actually > this > > > is > > > > > same as manual rebalancing. > > > > > > > > > > пт, 27 сент. 2019 г. в 17:40, Alexei Scherbakov < > > > > > alexey.scherbak...@gmail.com>: > > > > > > > > > > > The poor man's solution for the problem would be stopping > > fragmented > > > > node > > > > > > and removing partition data, then starting it again allowing full > > > state > > > > > > transfer already without deletes. > > > > > > Rinse and repeat for all owners. > > > > > > > > > > > > Anton Vinogradov, would this work for you as workaround ? > > > > > > > > > > > > чт, 19 сент. 2019 г. в 13:03, Anton Vinogradov : > > > > > > > > > > > >> Alexey, > > > > > >> > > > > > >> Let's combine your and Ivan's proposals. > > > > > >> > > > > > >> >> vacuum command, which acquires exclusive table lock, so no > > > > concurrent > > > > > >> activities on the table are possible. > > > > > >> and > > > > > >> >> Could the problem be solved by stopping a node which needs to > > be > > > > > >> defragmented, clearing persistence files and restarting the > node? > > > > > >> >> After rebalancing the node will receive all data back without > > > > > >> fragmentation. > > > > > >> > > > > > >> How about to have special partition state SHRINKING? > > > > > >> This state should mean that partition unavailable for reads and > > > > updates > > > > > >> but > > > > > >> should keep it's update-counters and should not be marked as > lost, > > > > > renting > > > > > >> or evicted. > > > > > >> At this state we able to iterate over the partition and apply > it's > > > > > entries > > > > > >> to another file in a compact way. > > > > > >> Indices should be updated during the copy-on-shrink procedure or > > at > > > > the > > > > > >> shrink completion. > > > > > >> Once shrank file is ready we should replace the original > partition > > > > file > > > > > >> with it and mark it as MOVING which will start the historical > > > > rebalance. > > > > > >> Shrinking should be performed during the low activity periods, > but > > > > even > > > > > in > > > > > >> case we found that activity was high and historical rebalance is > > not > > > > > >> suitable we may just remove the file and use regular rebalance > to > > > > > restore > > > > > >> the partition (this will also lead to shrink). > > > > > >> > > > > > >> BTW, seems, we able to implement partition shrink in a cheap > way. > > > > > >> We may just use rebalancing code to apply fat partition's > entries > > to > > > > the > > > > > >> new file. > > > > > >> So, 3 stages here: local rebalance, indices update and global > > > > historical > > > > > >> rebalance. > > > > > >> > > > > > >> On Thu, Sep 19, 2019 at 11:43 AM Alexey Goncharuk < > > > > > >> alexey.goncha...@gmail.com> wrote: > > > > > >> > > > > > >> > Anton, > > > > > >> > > > > > > >> > > > > > > >> > > >> The solution w
Re: Replacing default work dir from tmp to current dir
As for .NET. Should not we define IGNITE_HOME for it? чт, 3 окт. 2019 г. в 12:13, Ivan Pavlukhin : > > Folks, > > I am with Ilya here. I remind that we are talking not about general > case for Ignite usage. Generally IGNITE_HOME should be defined. > Otherwise we fallback to a default, and user.dir usually points to a > directory where java launcher command was called (work dir). > > user.home seems to cause more surprises to me: > * user.home can be undefined for JVM; > * user.home can be not writable as well (e.g. some special service user). > > And as far as know other vendors usually place files required for an > application in current work dir. > > чт, 3 окт. 2019 г. в 01:45, Denis Magda : > > > > I was always expecting this to be a user *home* directory that can be > > resolved in any operating system and will work for any language supported > > by Ignite. So, I'm with Pavel here. > > > > Alex G, what's your thinking? Sounds like we need to change this one more > > time. > > > > - > > Denis > > > > > > On Wed, Oct 2, 2019 at 12:17 PM Pavel Tupitsyn wrote: > > > > > Everyone above agreed to `~/ignite/work`, then somehow we jumped to > > > `user.dir/ignite/work`. > > > To me `user.dir` looked like synonym for ~, but turns out this is not > > > true. > > > I think others may be confused in the same way. > > > > > > Denis Magda, Alexey Goncharuk, and others - please confirm that you > > > understand that `user.dir` means current directory, not user home > > > directory. > > > > > > In my opinion, this is very broken. Current work dir can be literally > > > anything, e.g.: > > > `cd / && ~/my-ignite-app/run.sh` will cause an attempt to create ignite > > > dir > > > in system root, and so on. > > > > > > > > > > > > On Wed, Oct 2, 2019 at 9:46 PM Ilya Kasnacheev > > > wrote: > > > > > > > Hello! > > > > > > > > I think this is a sensible default and it was certainly not chosen by > > > > mistake. It was intentional expectation that your project is started > > > > from > > > > project root and data is located under it. > > > > > > > > If this breaks .Net, I am deeply sorry. > > > > However, I think we should change .net to provide non-default workdir > > > > location when none is specified. > > > > > > > > Can you please clarify scenarios that are broken now? > > > > > > > > Regards, > > > > > > > > ср, 2 окт. 2019 г., 20:28 Pavel Tupitsyn : > > > > > > > > > Igniters, > > > > > > > > > > Looks like we made a mistake while implementing IGNITE-12057: > > > > > `user.dir` is NOT user home directory, it is where JVM has been > > > > > started > > > > > from, which is rather arbitrary. > > > > > (Among other things this breaks Ignite.NET usage from tools like > > > LINQPad, > > > > > because `user.dir` ends up pointing to Program Files, which is not > > > > > writable without elevation) > > > > > > > > > > We should use `user.home` system property instead, see > > > > > > > > > > > > https://docs.oracle.com/javase/tutorial/essential/environment/sysprop.html > > > > > > > > > > Thoughts, objections? > > > > > > > > > > On Mon, Sep 2, 2019 at 1:57 PM Ilya Kasnacheev < > > > > ilya.kasnach...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hello again! > > > > > > > > > > > > Please note that I have updated release notes for IGNITE-12057 as > > > well > > > > as > > > > > > added them for my ticket. Release Engineers, please make sure you > > > > include > > > > > > the latest one. > > > > > > > > > > > > Regards, > > > > > > -- > > > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > > > > пн, 2 сент. 2019 г. в 13:33, Ilya Kasnacheev < > > > > ilya.kasnach...@gmail.com > > > > > >: > > > > > > > > > > > > > Hello! > > > > > > > > > > > > > > I have pushed an amended fix to both master and ignite-2.7.6. > > > > > > > > > > > > > > Regards, > > > > > > > -- > > > > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > > > > > > > пт, 30 авг. 2019 г. в 21:48, Denis Magda : > > > > > > > > > > > > > >> Ilya, > > > > > > >> > > > > > > >> I forgot to push "Send for review" button. You can see my minor > > > > > comment > > > > > > >> now. > > > > > > >> > > > > > > >> - > > > > > > >> Denis > > > > > > >> > > > > > > >> > > > > > > >> On Fri, Aug 30, 2019 at 5:47 AM Ilya Kasnacheev < > > > > > > >> ilya.kasnach...@gmail.com> > > > > > > >> wrote: > > > > > > >> > > > > > > >> > Hello! > > > > > > >> > > > > > > > >> > Waiting for a minor comment from Denis, as soon as I see/fix it > > > > I'm > > > > > > >> going > > > > > > >> > to commit. > > > > > > >> > > > > > > > >> > Regards, > > > > > > >> > Ilya. > > > > > > >> > -- > > > > > > >> > Ilya Kasnacheev > > > > > > >> > > > > > > > >> > > > > > > > >> > пт, 30 авг. 2019 г. в 11:30, Alexey Goncharuk < > > > > > > >> alexey.goncha...@gmail.com > > > > > > >> > >: > > > > > > >> > > > > > > > >> > > Hello Ilya, > > > > > > >> > > > > > > > > >> > > Just curious, when are you planning to commit your changes to > > > > the > > > > > > >> 2.7.6 > > > > > > >> > >
Re: Replacing default work dir from tmp to current dir
Folks, I am with Ilya here. I remind that we are talking not about general case for Ignite usage. Generally IGNITE_HOME should be defined. Otherwise we fallback to a default, and user.dir usually points to a directory where java launcher command was called (work dir). user.home seems to cause more surprises to me: * user.home can be undefined for JVM; * user.home can be not writable as well (e.g. some special service user). And as far as know other vendors usually place files required for an application in current work dir. чт, 3 окт. 2019 г. в 01:45, Denis Magda : > > I was always expecting this to be a user *home* directory that can be > resolved in any operating system and will work for any language supported > by Ignite. So, I'm with Pavel here. > > Alex G, what's your thinking? Sounds like we need to change this one more > time. > > - > Denis > > > On Wed, Oct 2, 2019 at 12:17 PM Pavel Tupitsyn wrote: > > > Everyone above agreed to `~/ignite/work`, then somehow we jumped to > > `user.dir/ignite/work`. > > To me `user.dir` looked like synonym for ~, but turns out this is not true. > > I think others may be confused in the same way. > > > > Denis Magda, Alexey Goncharuk, and others - please confirm that you > > understand that `user.dir` means current directory, not user home > > directory. > > > > In my opinion, this is very broken. Current work dir can be literally > > anything, e.g.: > > `cd / && ~/my-ignite-app/run.sh` will cause an attempt to create ignite dir > > in system root, and so on. > > > > > > > > On Wed, Oct 2, 2019 at 9:46 PM Ilya Kasnacheev > > wrote: > > > > > Hello! > > > > > > I think this is a sensible default and it was certainly not chosen by > > > mistake. It was intentional expectation that your project is started from > > > project root and data is located under it. > > > > > > If this breaks .Net, I am deeply sorry. > > > However, I think we should change .net to provide non-default workdir > > > location when none is specified. > > > > > > Can you please clarify scenarios that are broken now? > > > > > > Regards, > > > > > > ср, 2 окт. 2019 г., 20:28 Pavel Tupitsyn : > > > > > > > Igniters, > > > > > > > > Looks like we made a mistake while implementing IGNITE-12057: > > > > `user.dir` is NOT user home directory, it is where JVM has been started > > > > from, which is rather arbitrary. > > > > (Among other things this breaks Ignite.NET usage from tools like > > LINQPad, > > > > because `user.dir` ends up pointing to Program Files, which is not > > > > writable without elevation) > > > > > > > > We should use `user.home` system property instead, see > > > > > > > > > https://docs.oracle.com/javase/tutorial/essential/environment/sysprop.html > > > > > > > > Thoughts, objections? > > > > > > > > On Mon, Sep 2, 2019 at 1:57 PM Ilya Kasnacheev < > > > ilya.kasnach...@gmail.com> > > > > wrote: > > > > > > > > > Hello again! > > > > > > > > > > Please note that I have updated release notes for IGNITE-12057 as > > well > > > as > > > > > added them for my ticket. Release Engineers, please make sure you > > > include > > > > > the latest one. > > > > > > > > > > Regards, > > > > > -- > > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > пн, 2 сент. 2019 г. в 13:33, Ilya Kasnacheev < > > > ilya.kasnach...@gmail.com > > > > >: > > > > > > > > > > > Hello! > > > > > > > > > > > > I have pushed an amended fix to both master and ignite-2.7.6. > > > > > > > > > > > > Regards, > > > > > > -- > > > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > > > > пт, 30 авг. 2019 г. в 21:48, Denis Magda : > > > > > > > > > > > >> Ilya, > > > > > >> > > > > > >> I forgot to push "Send for review" button. You can see my minor > > > > comment > > > > > >> now. > > > > > >> > > > > > >> - > > > > > >> Denis > > > > > >> > > > > > >> > > > > > >> On Fri, Aug 30, 2019 at 5:47 AM Ilya Kasnacheev < > > > > > >> ilya.kasnach...@gmail.com> > > > > > >> wrote: > > > > > >> > > > > > >> > Hello! > > > > > >> > > > > > > >> > Waiting for a minor comment from Denis, as soon as I see/fix it > > > I'm > > > > > >> going > > > > > >> > to commit. > > > > > >> > > > > > > >> > Regards, > > > > > >> > Ilya. > > > > > >> > -- > > > > > >> > Ilya Kasnacheev > > > > > >> > > > > > > >> > > > > > > >> > пт, 30 авг. 2019 г. в 11:30, Alexey Goncharuk < > > > > > >> alexey.goncha...@gmail.com > > > > > >> > >: > > > > > >> > > > > > > >> > > Hello Ilya, > > > > > >> > > > > > > > >> > > Just curious, when are you planning to commit your changes to > > > the > > > > > >> 2.7.6 > > > > > >> > > branch? > > > > > >> > > > > > > > >> > > ср, 28 авг. 2019 г. в 04:57, Denis Magda : > > > > > >> > > > > > > > >> > > > Ok, seems like we came to a consensus. Let’s ensure that the > > > > path > > > > > >> for > > > > > >> > our > > > > > >> > > > work dir is user.dir/ignite/work and restart the vote. > > > > > >> > > > > > > > > >> > > > Denis > > > > > >> > > > > > > > > >> > > > On Tuesday, August 27, 2019, Ilya Kasnacheev <
[jira] [Created] (IGNITE-12255) Cache affinity fetching and calculation on client nodes may be broken in some cases
Pavel Kovalenko created IGNITE-12255: Summary: Cache affinity fetching and calculation on client nodes may be broken in some cases Key: IGNITE-12255 URL: https://issues.apache.org/jira/browse/IGNITE-12255 Project: Ignite Issue Type: Bug Components: cache Affects Versions: 2.7, 2.5 Reporter: Pavel Kovalenko Assignee: Pavel Kovalenko Fix For: 2.8 We have a cluster with server and client nodes. We dynamically start several caches on a cluster. Periodically we create and destroy some temporary cache in a cluster to move up cluster topology version. At the same time, a random client node chooses a random existing cache and performs operations on that cache. It leads to an exception on client node that affinity is not initialized for a cache during cache operation like: Affinity for topology version is not initialized [topVer = 8:10, head = 8:2] This exception means that the last affinity for a cache is calculated on version [8,2]. This is a cache start version. It happens because during creating/destroying some temporary cache we don’t re-calculate affinity for all existing but not already accessed caches on client nodes. Re-calculate in this case is cheap - we just copy affinity assignment and increment topology version. As a solution, we need to fetch affinity on client node join for all caches. Also, we need to re-calculate affinity for all affinity holders (not only for started caches or only configured caches) for all topology events that happened in a cluster on a client node. This solution showed the existing race between client node join and concurrent cache destroy. The race is the following: Client node (with some configured caches) joins to a cluster sending SingleMessage to coordinator during client PME. This SingleMessage contains affinity fetch requests for all cluster caches. When SingleMessage is in-flight server nodes finish client PME and also process and finish cache destroy PME. When a cache is destroyed affinity for that cache is cleared. When SingleMessage delivered to coordinator it doesn’t have affinity for a requested cache because the cache is already destroyed. It leads to assertion error on the coordinator and unpredictable behavior on the client node. The race may be fixed with the following change: If the coordinator doesn’t have an affinity for requested cache from the client node, it doesn’t break PME with assertion error, just doesn’t send affinity for that cache to a client node. When the client node receives FullMessage and sees that affinity for some requested cache doesn’t exist, it just closes cache proxy for user interactions which throws CacheStopped exception for every attempt to use that cache. This is safe behavior because cache destroy event should be happened on the client node soon and destroy that cache completely. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Replacing default work dir from tmp to current dir
Ivan, which vendors place files into current work dir, can you please give an example? > Generally IGNITE_HOME should be defined This is an inconvenience for the users, bad usability. As a user, why would I want to define a system-wide property just to use some library? > As for .NET. Should not we define IGNITE_HOME for it? No, for the reasons stated above. I'd like everyone to pay more attention to Maven/NuGet distribution scenario. Forget about zip archive for a while. As a user, I add a dependency to Ignite package and call Ignition.start(). That's all, it should work right away, no env vars, no additional configuration. And current work dir should not matter, because different tools, IDEs and workflows dictate different work directories. > user.home can be not writable as well In this case users won't even be able to use Maven or NuGet, let's not consider those rare scenarios. To summarize: home directory is the way to go as a default location. On Thu, Oct 3, 2019 at 12:14 PM Ivan Pavlukhin wrote: > As for .NET. Should not we define IGNITE_HOME for it? > > чт, 3 окт. 2019 г. в 12:13, Ivan Pavlukhin : > > > > Folks, > > > > I am with Ilya here. I remind that we are talking not about general > > case for Ignite usage. Generally IGNITE_HOME should be defined. > > Otherwise we fallback to a default, and user.dir usually points to a > > directory where java launcher command was called (work dir). > > > > user.home seems to cause more surprises to me: > > * user.home can be undefined for JVM; > > * user.home can be not writable as well (e.g. some special service user). > > > > And as far as know other vendors usually place files required for an > > application in current work dir. > > > > чт, 3 окт. 2019 г. в 01:45, Denis Magda : > > > > > > I was always expecting this to be a user *home* directory that can be > > > resolved in any operating system and will work for any language > supported > > > by Ignite. So, I'm with Pavel here. > > > > > > Alex G, what's your thinking? Sounds like we need to change this one > more > > > time. > > > > > > - > > > Denis > > > > > > > > > On Wed, Oct 2, 2019 at 12:17 PM Pavel Tupitsyn > wrote: > > > > > > > Everyone above agreed to `~/ignite/work`, then somehow we jumped to > > > > `user.dir/ignite/work`. > > > > To me `user.dir` looked like synonym for ~, but turns out this is > not true. > > > > I think others may be confused in the same way. > > > > > > > > Denis Magda, Alexey Goncharuk, and others - please confirm that you > > > > understand that `user.dir` means current directory, not user home > > > > directory. > > > > > > > > In my opinion, this is very broken. Current work dir can be literally > > > > anything, e.g.: > > > > `cd / && ~/my-ignite-app/run.sh` will cause an attempt to create > ignite dir > > > > in system root, and so on. > > > > > > > > > > > > > > > > On Wed, Oct 2, 2019 at 9:46 PM Ilya Kasnacheev < > ilya.kasnach...@gmail.com> > > > > wrote: > > > > > > > > > Hello! > > > > > > > > > > I think this is a sensible default and it was certainly not chosen > by > > > > > mistake. It was intentional expectation that your project is > started from > > > > > project root and data is located under it. > > > > > > > > > > If this breaks .Net, I am deeply sorry. > > > > > However, I think we should change .net to provide non-default > workdir > > > > > location when none is specified. > > > > > > > > > > Can you please clarify scenarios that are broken now? > > > > > > > > > > Regards, > > > > > > > > > > ср, 2 окт. 2019 г., 20:28 Pavel Tupitsyn : > > > > > > > > > > > Igniters, > > > > > > > > > > > > Looks like we made a mistake while implementing IGNITE-12057: > > > > > > `user.dir` is NOT user home directory, it is where JVM has been > started > > > > > > from, which is rather arbitrary. > > > > > > (Among other things this breaks Ignite.NET usage from tools like > > > > LINQPad, > > > > > > because `user.dir` ends up pointing to Program Files, which is > not > > > > > > writable without elevation) > > > > > > > > > > > > We should use `user.home` system property instead, see > > > > > > > > > > > > > > > > https://docs.oracle.com/javase/tutorial/essential/environment/sysprop.html > > > > > > > > > > > > Thoughts, objections? > > > > > > > > > > > > On Mon, Sep 2, 2019 at 1:57 PM Ilya Kasnacheev < > > > > > ilya.kasnach...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Hello again! > > > > > > > > > > > > > > Please note that I have updated release notes for IGNITE-12057 > as > > > > well > > > > > as > > > > > > > added them for my ticket. Release Engineers, please make sure > you > > > > > include > > > > > > > the latest one. > > > > > > > > > > > > > > Regards, > > > > > > > -- > > > > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > > > > > > > пн, 2 сент. 2019 г. в 13:33, Ilya Kasnacheev < > > > > > ilya.kasnach...@gmail.com > > > > > > >: > > > > > > > > > > > > > > > Hello! > > > > > > > > > > > > > > > > I have pushed an am
Re: Replacing default work dir from tmp to current dir
Pavel. > As a user, why would I want to define a system-wide property just to use > some library? Why do you think Ignite is a library? May be the root of usability issues in using same distribution for a library and server side dbms? В Чт, 03/10/2019 в 13:40 +0300, Pavel Tupitsyn пишет: > Ivan, which vendors place files into current work dir, can you please give > an example? > > > Generally IGNITE_HOME should be defined > > This is an inconvenience for the users, bad usability. > As a user, why would I want to define a system-wide property just to use > some library? > > > As for .NET. Should not we define IGNITE_HOME for it? > > No, for the reasons stated above. > > I'd like everyone to pay more attention to Maven/NuGet distribution > scenario. Forget about zip archive for a while. > As a user, I add a dependency to Ignite package and call Ignition.start(). > That's all, it should work right away, no env vars, no additional > configuration. > And current work dir should not matter, because different tools, IDEs and > workflows dictate different work directories. > > > user.home can be not writable as well > > In this case users won't even be able to use Maven or NuGet, let's not > consider those rare scenarios. > > > To summarize: home directory is the way to go as a default location. > > On Thu, Oct 3, 2019 at 12:14 PM Ivan Pavlukhin wrote: > > > As for .NET. Should not we define IGNITE_HOME for it? > > > > чт, 3 окт. 2019 г. в 12:13, Ivan Pavlukhin : > > > > > > Folks, > > > > > > I am with Ilya here. I remind that we are talking not about general > > > case for Ignite usage. Generally IGNITE_HOME should be defined. > > > Otherwise we fallback to a default, and user.dir usually points to a > > > directory where java launcher command was called (work dir). > > > > > > user.home seems to cause more surprises to me: > > > * user.home can be undefined for JVM; > > > * user.home can be not writable as well (e.g. some special service user). > > > > > > And as far as know other vendors usually place files required for an > > > application in current work dir. > > > > > > чт, 3 окт. 2019 г. в 01:45, Denis Magda : > > > > > > > > I was always expecting this to be a user *home* directory that can be > > > > resolved in any operating system and will work for any language > > > > supported > > > > by Ignite. So, I'm with Pavel here. > > > > > > > > Alex G, what's your thinking? Sounds like we need to change this one > > > > more > > > > time. > > > > > > > > - > > > > Denis > > > > > > > > > > > > On Wed, Oct 2, 2019 at 12:17 PM Pavel Tupitsyn > > > > wrote: > > > > > > > > > Everyone above agreed to `~/ignite/work`, then somehow we jumped to > > > > > `user.dir/ignite/work`. > > > > > To me `user.dir` looked like synonym for ~, but turns out this is > > > > not true. > > > > > I think others may be confused in the same way. > > > > > > > > > > Denis Magda, Alexey Goncharuk, and others - please confirm that you > > > > > understand that `user.dir` means current directory, not user home > > > > > directory. > > > > > > > > > > In my opinion, this is very broken. Current work dir can be literally > > > > > anything, e.g.: > > > > > `cd / && ~/my-ignite-app/run.sh` will cause an attempt to create > > > > ignite dir > > > > > in system root, and so on. > > > > > > > > > > > > > > > > > > > > On Wed, Oct 2, 2019 at 9:46 PM Ilya Kasnacheev < > > > > ilya.kasnach...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hello! > > > > > > > > > > > > I think this is a sensible default and it was certainly not chosen > > > > by > > > > > > mistake. It was intentional expectation that your project is > > > > started from > > > > > > project root and data is located under it. > > > > > > > > > > > > If this breaks .Net, I am deeply sorry. > > > > > > However, I think we should change .net to provide non-default > > > > workdir > > > > > > location when none is specified. > > > > > > > > > > > > Can you please clarify scenarios that are broken now? > > > > > > > > > > > > Regards, > > > > > > > > > > > > ср, 2 окт. 2019 г., 20:28 Pavel Tupitsyn : > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > Looks like we made a mistake while implementing IGNITE-12057: > > > > > > > `user.dir` is NOT user home directory, it is where JVM has been > > > > started > > > > > > > from, which is rather arbitrary. > > > > > > > (Among other things this breaks Ignite.NET usage from tools like > > > > > > > > > > LINQPad, > > > > > > > because `user.dir` ends up pointing to Program Files, which is > > > > not > > > > > > > writable without elevation) > > > > > > > > > > > > > > We should use `user.home` system property instead, see > > > > > > > > > > > https://docs.oracle.com/javase/tutorial/essential/environment/sysprop.html > > > > > > > > > > > > > > Thoughts, objections? > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 1:57 PM Ilya Kasnacheev < > > > > > > > > > > > >
Re: Replacing default work dir from tmp to current dir
Nikolay, Ignite is widely used in embedded scenarios; the ability to process data in-process locally is very powerful and I see no reason why we should remove it. I absolutely agree with Pavel T. on the subject. As an example, both SQLite, and rocksdb are distributed as a library, so I see no issues in Ignite server side being a library. As long as Ignite is available as a maven library, we should provide a consistent node behavior; "current directory" as persistence directory is not consistent. чт, 3 окт. 2019 г. в 13:52, Nikolay Izhikov : > Pavel. > > > As a user, why would I want to define a system-wide property just to use > > some library? > > Why do you think Ignite is a library? > May be the root of usability issues in using same distribution for a > library and server side dbms? > > > В Чт, 03/10/2019 в 13:40 +0300, Pavel Tupitsyn пишет: > > Ivan, which vendors place files into current work dir, can you please > give > > an example? > > > > > Generally IGNITE_HOME should be defined > > > > This is an inconvenience for the users, bad usability. > > As a user, why would I want to define a system-wide property just to use > > some library? > > > > > As for .NET. Should not we define IGNITE_HOME for it? > > > > No, for the reasons stated above. > > > > I'd like everyone to pay more attention to Maven/NuGet distribution > > scenario. Forget about zip archive for a while. > > As a user, I add a dependency to Ignite package and call > Ignition.start(). > > That's all, it should work right away, no env vars, no additional > > configuration. > > And current work dir should not matter, because different tools, IDEs and > > workflows dictate different work directories. > > > > > user.home can be not writable as well > > > > In this case users won't even be able to use Maven or NuGet, let's not > > consider those rare scenarios. > > > > > > To summarize: home directory is the way to go as a default location. > > > > On Thu, Oct 3, 2019 at 12:14 PM Ivan Pavlukhin > wrote: > > > > > As for .NET. Should not we define IGNITE_HOME for it? > > > > > > чт, 3 окт. 2019 г. в 12:13, Ivan Pavlukhin : > > > > > > > > Folks, > > > > > > > > I am with Ilya here. I remind that we are talking not about general > > > > case for Ignite usage. Generally IGNITE_HOME should be defined. > > > > Otherwise we fallback to a default, and user.dir usually points to a > > > > directory where java launcher command was called (work dir). > > > > > > > > user.home seems to cause more surprises to me: > > > > * user.home can be undefined for JVM; > > > > * user.home can be not writable as well (e.g. some special service > user). > > > > > > > > And as far as know other vendors usually place files required for an > > > > application in current work dir. > > > > > > > > чт, 3 окт. 2019 г. в 01:45, Denis Magda : > > > > > > > > > > I was always expecting this to be a user *home* directory that can > be > > > > > resolved in any operating system and will work for any language > > > > > > supported > > > > > by Ignite. So, I'm with Pavel here. > > > > > > > > > > Alex G, what's your thinking? Sounds like we need to change this > one > > > > > > more > > > > > time. > > > > > > > > > > - > > > > > Denis > > > > > > > > > > > > > > > On Wed, Oct 2, 2019 at 12:17 PM Pavel Tupitsyn < > ptupit...@apache.org> > > > > > > wrote: > > > > > > > > > > > Everyone above agreed to `~/ignite/work`, then somehow we jumped > to > > > > > > `user.dir/ignite/work`. > > > > > > To me `user.dir` looked like synonym for ~, but turns out this is > > > > > > not true. > > > > > > I think others may be confused in the same way. > > > > > > > > > > > > Denis Magda, Alexey Goncharuk, and others - please confirm that > you > > > > > > understand that `user.dir` means current directory, not user home > > > > > > directory. > > > > > > > > > > > > In my opinion, this is very broken. Current work dir can be > literally > > > > > > anything, e.g.: > > > > > > `cd / && ~/my-ignite-app/run.sh` will cause an attempt to create > > > > > > ignite dir > > > > > > in system root, and so on. > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 2, 2019 at 9:46 PM Ilya Kasnacheev < > > > > > > ilya.kasnach...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Hello! > > > > > > > > > > > > > > I think this is a sensible default and it was certainly not > chosen > > > > > > by > > > > > > > mistake. It was intentional expectation that your project is > > > > > > started from > > > > > > > project root and data is located under it. > > > > > > > > > > > > > > If this breaks .Net, I am deeply sorry. > > > > > > > However, I think we should change .net to provide non-default > > > > > > workdir > > > > > > > location when none is specified. > > > > > > > > > > > > > > Can you please clarify scenarios that are broken now? > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > ср, 2 окт. 2019 г., 20:28 Pavel Tupitsyn >: > > > > > > > > > > > > > > > Igniters, > > >
Re: Replacing default work dir from tmp to current dir
Alexey. > Ignite is widely used in embedded scenarios; the ability to process data > in-process locally is very powerful and I see no reason why we should > remove it I don't propose to remove something. I wrote about "same distribution". Seems, we should have different defaults and even distributions for different usage scenarios. > As an example, > both SQLite, and rocksdb are distributed as a library, so I see no issues > in Ignite server side being a library. As far as I know, both rocksdb and SQLite is local only libraries and don't have any distrubted features. >"current directory" as persistence directory is not consistent. I agree with you. How it happens, that after wide discussion we implemented, reviewed and merged wrong defaults? As I know, we have explicit release only to change this default. This release is broken, isn't it? В Чт, 03/10/2019 в 14:03 +0300, Alexey Goncharuk пишет: > Nikolay, > > Ignite is widely used in embedded scenarios; the ability to process data > in-process locally is very powerful and I see no reason why we should > remove it. I absolutely agree with Pavel T. on the subject. As an example, > both SQLite, and rocksdb are distributed as a library, so I see no issues > in Ignite server side being a library. > > As long as Ignite is available as a maven library, we should provide a > consistent node behavior; "current directory" as persistence directory is > not consistent. > > чт, 3 окт. 2019 г. в 13:52, Nikolay Izhikov : > > > Pavel. > > > > > As a user, why would I want to define a system-wide property just to use > > > some library? > > > > Why do you think Ignite is a library? > > May be the root of usability issues in using same distribution for a > > library and server side dbms? > > > > > > В Чт, 03/10/2019 в 13:40 +0300, Pavel Tupitsyn пишет: > > > Ivan, which vendors place files into current work dir, can you please > > > > give > > > an example? > > > > > > > Generally IGNITE_HOME should be defined > > > > > > This is an inconvenience for the users, bad usability. > > > As a user, why would I want to define a system-wide property just to use > > > some library? > > > > > > > As for .NET. Should not we define IGNITE_HOME for it? > > > > > > No, for the reasons stated above. > > > > > > I'd like everyone to pay more attention to Maven/NuGet distribution > > > scenario. Forget about zip archive for a while. > > > As a user, I add a dependency to Ignite package and call > > > > Ignition.start(). > > > That's all, it should work right away, no env vars, no additional > > > configuration. > > > And current work dir should not matter, because different tools, IDEs and > > > workflows dictate different work directories. > > > > > > > user.home can be not writable as well > > > > > > In this case users won't even be able to use Maven or NuGet, let's not > > > consider those rare scenarios. > > > > > > > > > To summarize: home directory is the way to go as a default location. > > > > > > On Thu, Oct 3, 2019 at 12:14 PM Ivan Pavlukhin > > > > wrote: > > > > > > > As for .NET. Should not we define IGNITE_HOME for it? > > > > > > > > чт, 3 окт. 2019 г. в 12:13, Ivan Pavlukhin : > > > > > > > > > > Folks, > > > > > > > > > > I am with Ilya here. I remind that we are talking not about general > > > > > case for Ignite usage. Generally IGNITE_HOME should be defined. > > > > > Otherwise we fallback to a default, and user.dir usually points to a > > > > > directory where java launcher command was called (work dir). > > > > > > > > > > user.home seems to cause more surprises to me: > > > > > * user.home can be undefined for JVM; > > > > > * user.home can be not writable as well (e.g. some special service > > > > user). > > > > > > > > > > And as far as know other vendors usually place files required for an > > > > > application in current work dir. > > > > > > > > > > чт, 3 окт. 2019 г. в 01:45, Denis Magda : > > > > > > > > > > > > I was always expecting this to be a user *home* directory that can > > > > be > > > > > > resolved in any operating system and will work for any language > > > > > > > > supported > > > > > > by Ignite. So, I'm with Pavel here. > > > > > > > > > > > > Alex G, what's your thinking? Sounds like we need to change this > > > > one > > > > > > > > more > > > > > > time. > > > > > > > > > > > > - > > > > > > Denis > > > > > > > > > > > > > > > > > > On Wed, Oct 2, 2019 at 12:17 PM Pavel Tupitsyn < > > > > ptupit...@apache.org> > > > > > > > > wrote: > > > > > > > > > > > > > Everyone above agreed to `~/ignite/work`, then somehow we jumped > > > > to > > > > > > > `user.dir/ignite/work`. > > > > > > > To me `user.dir` looked like synonym for ~, but turns out this is > > > > > > > > not true. > > > > > > > I think others may be confused in the same way. > > > > > > > > > > > > > > Denis Magda, Alexey Goncharuk, and others - please confirm that > > > > you > > > > > > > understand that `user.dir` means
[jira] [Created] (IGNITE-12256) Fix double invocation of javaMajorVersion in scripts
Ilya Kasnacheev created IGNITE-12256:
Summary: Fix double invocation of javaMajorVersion in scripts
Key: IGNITE-12256
URL: https://issues.apache.org/jira/browse/IGNITE-12256
Project: Ignite
Issue Type: Improvement
Components: general
Affects Versions: 2.7.6
Reporter: Ilya Kasnacheev
Assignee: Ilya Kasnacheev
Most of our shell script look as folows:
{code}
#
# Discover path to Java executable and check it's version.
#
checkJava
#
# Discover IGNITE_HOME environment variable.
#
setIgniteHome
#
# Final JVM_OPTS for Java 9+ compatibility
#
javaMajorVersion "${JAVA_HOME}/bin/java"
{code}
It makes no sense to me since we already call javaMajorVersion in checkJava.
Let's try to get rid of it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12257) [ML] Add Feature Filter for ML Partitioned Dataset
Alexey Zinoviev created IGNITE-12257:
Summary: [ML] Add Feature Filter for ML Partitioned Dataset
Key: IGNITE-12257
URL: https://issues.apache.org/jira/browse/IGNITE-12257
Project: Ignite
Issue Type: Improvement
Affects Versions: 2.9
Reporter: Alexey Zinoviev
Assignee: Alexey Zinoviev
Fix For: 2.9
The behavior of this method ignores possible feature choosing on the previous
levels and we have no ability to make feature engineering during the
preprocessing like simple sql: filter, exclude, produce new features and so on
public SimpleDatasetData build(
LearningEnvironment env,
Iterator> upstreamData, long upstreamDataSize, C ctx) {
// Prepares the matrix of features in flat column-major format.
int cols = -1;
double[] features = null;
int ptr = 0;
while (upstreamData.hasNext()) {
UpstreamEntry entry = upstreamData.next();
Vector row = preprocessor.apply(entry.getKey(), entry.getValue()).features();
if (cols < 0) {
cols = row.size();
features = new double[Math.toIntExact(upstreamDataSize * cols)];
}
else
assert row.size() == cols : "Feature extractor must return exactly " + cols +
" features";
for (int i = 0; i < cols; i++)
features[Math.toIntExact(i * upstreamDataSize + ptr)] = row.get(i);
ptr++;
}
return new SimpleDatasetData(features, Math.toIntExact(upstreamDataSize));
}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [jira] [Created] (IGNITE-12208) If JAVA_HOME is not configured, the ignitevisorcmd.sh script cannot run
Hello! I have found that this call is not needed at all, so I've started https://issues.apache.org/jira/browse/IGNITE-12256 Regards, -- Ilya Kasnacheev вт, 24 сент. 2019 г. в 06:44, liyuj <18624049...@163.com>: > Hi, > > I have pushed PR: 6899, and I hope the community will merge as soon as > possible. > > 在 2019/9/23 下午5:37, Nikolay Izhikov 写道: > > Hello, liyuj > > > > Thanks for your interest in Igniter. > > > > Feel free to assign ticket to yourself. > > Looking forward for your contribution! > > > > В Сб, 21/09/2019 в 19:31 +0800, liyuj пишет: > >> Hi community, > >> > >> The issue is relatively simple, and I can contribute. > >> Can someone assign this to me? My JIRA ID is liyuj. > >> > >> 在 2019/9/21 下午4:47, YuJue Li (Jira) 写道: > >>> YuJue Li created IGNITE-12208: > >>> - > >>> > >>>Summary: If JAVA_HOME is not configured, the > ignitevisorcmd.sh script cannot run > >>>Key: IGNITE-12208 > >>>URL: > https://issues.apache.org/jira/browse/IGNITE-12208 > >>>Project: Ignite > >>> Issue Type: Improvement > >>> Components: visor > >>> Affects Versions: 2.7.6 > >>>Environment: Debian LInux OS > >>> > >>> sudo apt install openjdk-8-jdk > >>> Reporter: YuJue Li > >>>Fix For: 2.8 > >>> > >>> > >>> If JAVA_HOME is not configured, the ignitevisorcmd.sh script cannot > run. > >>> > >>> there is no error message output, and the document does not seem to > indicate that JAVA_HOME must be configured. > >>> > >>> Versions 2.7.0 do not have this problem, but versions 2.7.5 and 2.7.6 > do. > >>> > >>> > >>> > >>> -- > >>> This message was sent by Atlassian Jira > >>> (v8.3.4#803005) > >> > >
Re: help on IGNITE-11894
Hello! I think that you should not attempt to fix Cache Store if you don't have first-hand experience of using it. Please redirect your efforts to some area where, as the bare minimum, you are able to write such examples yourself. Regards, -- Ilya Kasnacheev ср, 11 сент. 2019 г. в 19:13, Amit Chavan : > Thanks, Nikolay. > > Some questions I have - > 1. Looking at the Ignite core module I see following classes that are jdbc > stores CacheJdbcBlobStore and CacheJdbcPojoStore. Are these the only ones > or am I missing more? > 2. I see the fetchSize is part of JdbcDialect. Which implementation of the > JdbcDilect is used to access CacheStores? > 3. Are there any examples of UT or docs which shows how the cachestores are > being used so it helps me understand the full end to end workings on the > CacheStores. > > Thanks, > Amit > > On Mon, Sep 9, 2019 at 2:50 PM Nikolay Izhikov > wrote: > > > Hell, Amit. > > > > You can write you question here, on the dev-list or in the ticket. > > I think, Ignite community will help you with this and ongoing > contribution. > > > > Welcome! > > > > В Пн, 09/09/2019 в 11:14 -0700, Amit Chavan пишет: > > > Hello, > > > > > > I need some help on the ticket > > > https://issues.apache.org/jira/browse/IGNITE-11894 since this is my > time > > > working on ignite code base. I am not able to move the ticket in > progress > > > and would like to chat with a developer on what is expected for the > > ticket. > > > Is there a slack channel or message client the team uses to > communicate? > > > > > > Thanks, > > > Amit > > >
Re: Replacing default work dir from tmp to current dir
Pavel, > Ivan, which vendors place files into current work dir, can you please give an > example? Cockroachdb stores files relative to current work dir (yes, "cd" sensitive). As I understood aforementioned SQLite do the same [1]. > In this case users won't even be able to use Maven or NuGet, let's not > consider those rare scenarios. Cannot tell about NuGet. Maven is typically used during development, usually there is no Maven in production deployments. Folks, I believe we should definitely wait a reply from Ilya, as the fix was not done blindly. [1] https://www.sqlite.org/c3ref/data_directory.html чт, 3 окт. 2019 г. в 14:19, Nikolay Izhikov : > > Alexey. > > > Ignite is widely used in embedded scenarios; the ability to process data > > in-process locally is very powerful and I see no reason why we should > > remove it > > I don't propose to remove something. > I wrote about "same distribution". > > Seems, we should have different defaults and even distributions for different > usage scenarios. > > > As an example, > > both SQLite, and rocksdb are distributed as a library, so I see no issues > > in Ignite server side being a library. > > As far as I know, both rocksdb and SQLite is local only libraries and don't > have any distrubted features. > > > >"current directory" as persistence directory is not consistent. > > I agree with you. > How it happens, that after wide discussion we implemented, reviewed and > merged wrong defaults? > > As I know, we have explicit release only to change this default. > > This release is broken, isn't it? > > В Чт, 03/10/2019 в 14:03 +0300, Alexey Goncharuk пишет: > > Nikolay, > > > > Ignite is widely used in embedded scenarios; the ability to process data > > in-process locally is very powerful and I see no reason why we should > > remove it. I absolutely agree with Pavel T. on the subject. As an example, > > both SQLite, and rocksdb are distributed as a library, so I see no issues > > in Ignite server side being a library. > > > > As long as Ignite is available as a maven library, we should provide a > > consistent node behavior; "current directory" as persistence directory is > > not consistent. > > > > чт, 3 окт. 2019 г. в 13:52, Nikolay Izhikov : > > > > > Pavel. > > > > > > > As a user, why would I want to define a system-wide property just to use > > > > some library? > > > > > > Why do you think Ignite is a library? > > > May be the root of usability issues in using same distribution for a > > > library and server side dbms? > > > > > > > > > В Чт, 03/10/2019 в 13:40 +0300, Pavel Tupitsyn пишет: > > > > Ivan, which vendors place files into current work dir, can you please > > > > > > give > > > > an example? > > > > > > > > > Generally IGNITE_HOME should be defined > > > > > > > > This is an inconvenience for the users, bad usability. > > > > As a user, why would I want to define a system-wide property just to use > > > > some library? > > > > > > > > > As for .NET. Should not we define IGNITE_HOME for it? > > > > > > > > No, for the reasons stated above. > > > > > > > > I'd like everyone to pay more attention to Maven/NuGet distribution > > > > scenario. Forget about zip archive for a while. > > > > As a user, I add a dependency to Ignite package and call > > > > > > Ignition.start(). > > > > That's all, it should work right away, no env vars, no additional > > > > configuration. > > > > And current work dir should not matter, because different tools, IDEs > > > > and > > > > workflows dictate different work directories. > > > > > > > > > user.home can be not writable as well > > > > > > > > In this case users won't even be able to use Maven or NuGet, let's not > > > > consider those rare scenarios. > > > > > > > > > > > > To summarize: home directory is the way to go as a default location. > > > > > > > > On Thu, Oct 3, 2019 at 12:14 PM Ivan Pavlukhin > > > > > > wrote: > > > > > > > > > As for .NET. Should not we define IGNITE_HOME for it? > > > > > > > > > > чт, 3 окт. 2019 г. в 12:13, Ivan Pavlukhin : > > > > > > > > > > > > Folks, > > > > > > > > > > > > I am with Ilya here. I remind that we are talking not about general > > > > > > case for Ignite usage. Generally IGNITE_HOME should be defined. > > > > > > Otherwise we fallback to a default, and user.dir usually points to a > > > > > > directory where java launcher command was called (work dir). > > > > > > > > > > > > user.home seems to cause more surprises to me: > > > > > > * user.home can be undefined for JVM; > > > > > > * user.home can be not writable as well (e.g. some special service > > > > > > user). > > > > > > > > > > > > And as far as know other vendors usually place files required for an > > > > > > application in current work dir. > > > > > > > > > > > > чт, 3 окт. 2019 г. в 01:45, Denis Magda : > > > > > > > > > > > > > > I was always expecting this to be a user *home* directory that can > > > > > > be > > > > > > > resolved in any operating system and will work for any language >
Re: Text queries/indexes (GridLuceneIndex, @QueryTextFiled)
Ivan, Regarding you question about Lucene search response. *IndexSearcher.search()* always returns result sorted at least by *score *(*relevance*) or by defined *Sort *which includes ordering fields and rules. This means than even for now *GridLunceneIndex* result will be incorrect in case of distributed queries as they are merged in arbitrary way. Under the hood *ScoreDoc* object is used to fetch desired document/record and this class contains *docId* and *score*. So small wrapper with Comparable interface may solve merging of ordered results. BR, Yuriy Shuliha пт, 27 вер. 2019 о 18:48 Павлухин Иван пише: > Yuriy, > > Thank you for providing details! Quite interesting. > > Yes, we already have support of distributed limit and merging sorted > subresults for SQL queries. E.g. ReduceIndexSorted and > MergeStreamIterator are used for merging sorted streams. > > Could you please also clarify about score/relevance? Is it provided by > Lucene engine for each query result? I am thinking how to do sorted > merge properly in this case. > > ср, 25 сент. 2019 г. в 18:56, Yuriy Shuliga : > > > > Ivan, > > > > Thank you for interesting question! > > > > Text searches (or full text searches) are mostly human-oriented. And the > > point of user's interest is topmost part of response. > > Then user can read it, evaluate and use the given records for further > > purposes. > > > > Particularly in our case, we use Ignite for operations with financial > data, > > and there lots of text stuff like assets names, fin. instruments, > companies > > etc. > > In order to operate with this quickly and reliably, users used to work > with > > text search, type-ahead completions, suggestions. > > > > For this purposes we are indexing particular string data in separate > caches. > > > > Sorting capabilities and response size limitations are very important > > there. As our API have to provide most relevant information in view of > > limited size. > > > > Now let me comment some Ignite/Lucene perspective. > > Actually Ignite queries and Lucene returns *TopDocs.scoresDocs *already > > sorted by *score *(relevance). So most relevant documents are on the top. > > And currently distributed queries responses from different nodes are > merged > > into final query cursor queue in arbitrary way. > > So in fact we already have the score order ruined here. Also Ignite > > requests all possible documents from Lucene that is redundant and not > good > > for performance. > > > > I'm implementing *limit* parameter to be part of *TextQuery *and have to > > notice that we still have to add sorting for text queries processing in > > order to have applicable results. > > > > *Limit* parameter itself should improve the part of issues from above, > but > > definitely, sorting by document score at least should be implemented > along > > with limit. > > > > This is a pretty short commentary if you still have any questions, please > > ask, do not hesitate) > > > > BR, > > Yuriy Shuliha > > > > чт, 19 вер. 2019 о 11:38 Павлухин Иван пише: > > > > > Yuriy, > > > > > > Greatly appreciate your interest. > > > > > > Could you please elaborate a little bit about sorting? What tasks does > > > it help to solve and how? It would be great to provide an example. > > > > > > ср, 18 сент. 2019 г. в 09:39, Alexei Scherbakov < > > > alexey.scherbak...@gmail.com>: > > > > > > > > Denis, > > > > > > > > I like the idea of throwing an exception for enabled text queries on > > > > persistent caches. > > > > > > > > Also I'm fine with proposed limit for unsorted searches. > > > > > > > > Yury, please proceed with ticket creation. > > > > > > > > вт, 17 сент. 2019 г., 22:06 Denis Magda : > > > > > > > > > Igniters, > > > > > > > > > > I see nothing wrong with Yury's proposal in regards full-text > search > > > API > > > > > evolution as long as Yury is ready to push it forward. > > > > > > > > > > As for the in-memory mode only, it makes total sense for in-memory > data > > > > > grid deployments when Ignite caches data of an underlying DB like > > > Postgres. > > > > > As part of the changes, I would simply throw an exception (by > default) > > > if > > > > > the one attempts to use text indices with the native persistence > > > enabled. > > > > > If the person is ready to live with that limitation that an > explicit > > > > > configuration change is needed to come around the exception. > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > > - > > > > > Denis > > > > > > > > > > > > > > > On Tue, Sep 17, 2019 at 7:44 AM Yuriy Shuliga > > > wrote: > > > > > > > > > > > Hello to all again, > > > > > > > > > > > > Thank you for important comments and notes given below! > > > > > > > > > > > > Let me answer and continue the discussion. > > > > > > > > > > > > (I) Overall needs in Lucene indexing > > > > > > > > > > > > Alexei has referenced to > > > > > > https://issues.apache.org/jira/browse/IGNITE-5371 where > > > > > > absence of index persistence was declared as
Re: Text queries/indexes (GridLuceneIndex, @QueryTextFiled)
Denis, Thank you for your attention to this. as for now, the https://issues.apache.org/jira/browse/IGNITE-12189 ticket is still pending review. Do we have a chance to move it forward somehow? BR, Yuriy Shuliha пн, 30 вер. 2019 о 23:35 Denis Magda пише: > Yuriy, > > I've seen you opening a pull-request with the first changes: > https://issues.apache.org/jira/browse/IGNITE-12189 > > Alex Scherbakov and Ivan are you the right guys to do the review? > > - > Denis > > > On Fri, Sep 27, 2019 at 8:48 AM Павлухин Иван wrote: > > > Yuriy, > > > > Thank you for providing details! Quite interesting. > > > > Yes, we already have support of distributed limit and merging sorted > > subresults for SQL queries. E.g. ReduceIndexSorted and > > MergeStreamIterator are used for merging sorted streams. > > > > Could you please also clarify about score/relevance? Is it provided by > > Lucene engine for each query result? I am thinking how to do sorted > > merge properly in this case. > > > > ср, 25 сент. 2019 г. в 18:56, Yuriy Shuliga : > > > > > > Ivan, > > > > > > Thank you for interesting question! > > > > > > Text searches (or full text searches) are mostly human-oriented. And > the > > > point of user's interest is topmost part of response. > > > Then user can read it, evaluate and use the given records for further > > > purposes. > > > > > > Particularly in our case, we use Ignite for operations with financial > > data, > > > and there lots of text stuff like assets names, fin. instruments, > > companies > > > etc. > > > In order to operate with this quickly and reliably, users used to work > > with > > > text search, type-ahead completions, suggestions. > > > > > > For this purposes we are indexing particular string data in separate > > caches. > > > > > > Sorting capabilities and response size limitations are very important > > > there. As our API have to provide most relevant information in view of > > > limited size. > > > > > > Now let me comment some Ignite/Lucene perspective. > > > Actually Ignite queries and Lucene returns *TopDocs.scoresDocs *already > > > sorted by *score *(relevance). So most relevant documents are on the > top. > > > And currently distributed queries responses from different nodes are > > merged > > > into final query cursor queue in arbitrary way. > > > So in fact we already have the score order ruined here. Also Ignite > > > requests all possible documents from Lucene that is redundant and not > > good > > > for performance. > > > > > > I'm implementing *limit* parameter to be part of *TextQuery *and have > to > > > notice that we still have to add sorting for text queries processing in > > > order to have applicable results. > > > > > > *Limit* parameter itself should improve the part of issues from above, > > but > > > definitely, sorting by document score at least should be implemented > > along > > > with limit. > > > > > > This is a pretty short commentary if you still have any questions, > please > > > ask, do not hesitate) > > > > > > BR, > > > Yuriy Shuliha > > > > > > чт, 19 вер. 2019 о 11:38 Павлухин Иван пише: > > > > > > > Yuriy, > > > > > > > > Greatly appreciate your interest. > > > > > > > > Could you please elaborate a little bit about sorting? What tasks > does > > > > it help to solve and how? It would be great to provide an example. > > > > > > > > ср, 18 сент. 2019 г. в 09:39, Alexei Scherbakov < > > > > alexey.scherbak...@gmail.com>: > > > > > > > > > > Denis, > > > > > > > > > > I like the idea of throwing an exception for enabled text queries > on > > > > > persistent caches. > > > > > > > > > > Also I'm fine with proposed limit for unsorted searches. > > > > > > > > > > Yury, please proceed with ticket creation. > > > > > > > > > > вт, 17 сент. 2019 г., 22:06 Denis Magda : > > > > > > > > > > > Igniters, > > > > > > > > > > > > I see nothing wrong with Yury's proposal in regards full-text > > search > > > > API > > > > > > evolution as long as Yury is ready to push it forward. > > > > > > > > > > > > As for the in-memory mode only, it makes total sense for > in-memory > > data > > > > > > grid deployments when Ignite caches data of an underlying DB like > > > > Postgres. > > > > > > As part of the changes, I would simply throw an exception (by > > default) > > > > if > > > > > > the one attempts to use text indices with the native persistence > > > > enabled. > > > > > > If the person is ready to live with that limitation that an > > explicit > > > > > > configuration change is needed to come around the exception. > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > > > > > - > > > > > > Denis > > > > > > > > > > > > > > > > > > On Tue, Sep 17, 2019 at 7:44 AM Yuriy Shuliga > > > > > wrote: > > > > > > > > > > > > > Hello to all again, > > > > > > > > > > > > > > Thank you for important comments and notes given below! > > > > > > > > > > > > > > Let me answer and continue the discussion. > > > > > > > > > > > > > > (I) Overall needs in Lucene indexin
Re: [DISCUSSION][IEP-35] Replace RunningQueryManager with GridSystemViewManager
Hello, Ivan. Thanks for feedback. Initial IEP [1] naming was changed during code review. I updated the IEP [1] with the current naming. Can you take a look and check is all clear now? [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392 В Ср, 02/10/2019 в 17:21 +0300, Ivan Pavlukhin пишет: > Hi Nikolay, > > Actually I do not fully understand what is SystemView API. I have not > found it in IEP [1] (I searched for words "system" and "view"). > > RunningQueryManager is a component responsible for tracking running > queries internally. This info is exposed to users as SQL view via > SqlSystemViewRunningQueries. In the same package you can find a plenty > of other SQL views. > > [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392 > > вт, 1 окт. 2019 г. в 06:42, Nikolay Izhikov : > > > > Hello, Igniters. > > > > Since the last release `RunningQueryManager` [1] was added. > > It used to track a running query. > > > > In IEP-35 [2] SystemView API was added. > > SystemView API supposed to be used to track all kinds of internal Ignite > > objects. > > > > I think this RunningQueryManager should be replaced [3] with the more > > unified SystemView API. > > > > Any objections? > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10754 > > [2] > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392 > > [3] https://issues.apache.org/jira/browse/IGNITE-12223 > > [4] https://issues.apache.org/jira/browse/IGNITE-12224 > > > signature.asc Description: This is a digitally signed message part
Re: help on IGNITE-11894
Ilya, > I think that you should not attempt to fix Cache Store if you don't have > first-hand experience of using it. Not sure that I understand "first-hand experience" properly. But it seems to me that almost every Ignite developer started contribution without first-hand experience in one or another area. Amit, I believe you can find documentation about CacheStore without a problems by searching in Google or on Ignite docs site [1]. Unit tests can also be found by usages of org.apache.ignite.configuration.CacheConfiguration#setCacheStoreFactory. Of course, if something is very hard (there are a lot of hard stuff in Ignite) it might be better to pick a simpler task as a starter. [1] https://apacheignite.readme.io/docs чт, 3 окт. 2019 г. в 15:03, Ilya Kasnacheev : > > Hello! > > I think that you should not attempt to fix Cache Store if you don't have > first-hand experience of using it. Please redirect your efforts to some > area where, as the bare minimum, you are able to write such examples > yourself. > > Regards, > -- > Ilya Kasnacheev > > > ср, 11 сент. 2019 г. в 19:13, Amit Chavan : > > > Thanks, Nikolay. > > > > Some questions I have - > > 1. Looking at the Ignite core module I see following classes that are jdbc > > stores CacheJdbcBlobStore and CacheJdbcPojoStore. Are these the only ones > > or am I missing more? > > 2. I see the fetchSize is part of JdbcDialect. Which implementation of the > > JdbcDilect is used to access CacheStores? > > 3. Are there any examples of UT or docs which shows how the cachestores are > > being used so it helps me understand the full end to end workings on the > > CacheStores. > > > > Thanks, > > Amit > > > > On Mon, Sep 9, 2019 at 2:50 PM Nikolay Izhikov > > wrote: > > > > > Hell, Amit. > > > > > > You can write you question here, on the dev-list or in the ticket. > > > I think, Ignite community will help you with this and ongoing > > contribution. > > > > > > Welcome! > > > > > > В Пн, 09/09/2019 в 11:14 -0700, Amit Chavan пишет: > > > > Hello, > > > > > > > > I need some help on the ticket > > > > https://issues.apache.org/jira/browse/IGNITE-11894 since this is my > > time > > > > working on ignite code base. I am not able to move the ticket in > > progress > > > > and would like to chat with a developer on what is expected for the > > > ticket. > > > > Is there a slack channel or message client the team uses to > > communicate? > > > > > > > > Thanks, > > > > Amit > > > > > -- Best regards, Ivan Pavlukhin
Re: Replacing default work dir from tmp to current dir
> > Seems, we should have different defaults and even distributions for > different usage scenarios. > I still do not understand why defaults should be different for embedded and "traditional RDBMS-like" installations. Having different defaults will likely confuse users, not make usability easier. Personally, I would forbid to start Ignite if IGNITE_HOME is not set, but this suggestion was not accepted by the community. As far as I know, both rocksdb and SQLite is local only libraries and don't > have any distrubted features. See no difference here. Imagine a user starts only one Ignite node for development or just to play (which, I believe, happes quite a lot) - same as with local databases. BTW, it is impossible to start SQLite without database path, so a user either provides a full path, or a relative path from the current directory - which is an explicit action from a user. > I agree with you. > How it happens, that after wide discussion we implemented, reviewed and > merged wrong defaults? > > As I know, we have explicit release only to change this default. > > This release is broken, isn't it? > I think this is just a miscommunication. Ilya made a fix which was exactly what he meant it to be. As for the release - it may have worse usability, but not more 'broken' as the previous one with the temp directory. At least the data will not be physically removed after the machine restart.
Re: Text queries/indexes (GridLuceneIndex, @QueryTextFiled)
Yuri, I've done with review. No crime found, but trivial compatibility bug. On Thu, Oct 3, 2019 at 3:54 PM Yuriy Shuliga wrote: > Denis, > > Thank you for your attention to this. > as for now, the https://issues.apache.org/jira/browse/IGNITE-12189 ticket > is still pending review. > Do we have a chance to move it forward somehow? > > BR, > Yuriy Shuliha > > пн, 30 вер. 2019 о 23:35 Denis Magda пише: > > > Yuriy, > > > > I've seen you opening a pull-request with the first changes: > > https://issues.apache.org/jira/browse/IGNITE-12189 > > > > Alex Scherbakov and Ivan are you the right guys to do the review? > > > > - > > Denis > > > > > > On Fri, Sep 27, 2019 at 8:48 AM Павлухин Иван > wrote: > > > > > Yuriy, > > > > > > Thank you for providing details! Quite interesting. > > > > > > Yes, we already have support of distributed limit and merging sorted > > > subresults for SQL queries. E.g. ReduceIndexSorted and > > > MergeStreamIterator are used for merging sorted streams. > > > > > > Could you please also clarify about score/relevance? Is it provided by > > > Lucene engine for each query result? I am thinking how to do sorted > > > merge properly in this case. > > > > > > ср, 25 сент. 2019 г. в 18:56, Yuriy Shuliga : > > > > > > > > Ivan, > > > > > > > > Thank you for interesting question! > > > > > > > > Text searches (or full text searches) are mostly human-oriented. And > > the > > > > point of user's interest is topmost part of response. > > > > Then user can read it, evaluate and use the given records for further > > > > purposes. > > > > > > > > Particularly in our case, we use Ignite for operations with financial > > > data, > > > > and there lots of text stuff like assets names, fin. instruments, > > > companies > > > > etc. > > > > In order to operate with this quickly and reliably, users used to > work > > > with > > > > text search, type-ahead completions, suggestions. > > > > > > > > For this purposes we are indexing particular string data in separate > > > caches. > > > > > > > > Sorting capabilities and response size limitations are very important > > > > there. As our API have to provide most relevant information in view > of > > > > limited size. > > > > > > > > Now let me comment some Ignite/Lucene perspective. > > > > Actually Ignite queries and Lucene returns *TopDocs.scoresDocs > *already > > > > sorted by *score *(relevance). So most relevant documents are on the > > top. > > > > And currently distributed queries responses from different nodes are > > > merged > > > > into final query cursor queue in arbitrary way. > > > > So in fact we already have the score order ruined here. Also Ignite > > > > requests all possible documents from Lucene that is redundant and not > > > good > > > > for performance. > > > > > > > > I'm implementing *limit* parameter to be part of *TextQuery *and have > > to > > > > notice that we still have to add sorting for text queries processing > in > > > > order to have applicable results. > > > > > > > > *Limit* parameter itself should improve the part of issues from > above, > > > but > > > > definitely, sorting by document score at least should be implemented > > > along > > > > with limit. > > > > > > > > This is a pretty short commentary if you still have any questions, > > please > > > > ask, do not hesitate) > > > > > > > > BR, > > > > Yuriy Shuliha > > > > > > > > чт, 19 вер. 2019 о 11:38 Павлухин Иван пише: > > > > > > > > > Yuriy, > > > > > > > > > > Greatly appreciate your interest. > > > > > > > > > > Could you please elaborate a little bit about sorting? What tasks > > does > > > > > it help to solve and how? It would be great to provide an example. > > > > > > > > > > ср, 18 сент. 2019 г. в 09:39, Alexei Scherbakov < > > > > > alexey.scherbak...@gmail.com>: > > > > > > > > > > > > Denis, > > > > > > > > > > > > I like the idea of throwing an exception for enabled text queries > > on > > > > > > persistent caches. > > > > > > > > > > > > Also I'm fine with proposed limit for unsorted searches. > > > > > > > > > > > > Yury, please proceed with ticket creation. > > > > > > > > > > > > вт, 17 сент. 2019 г., 22:06 Denis Magda : > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > I see nothing wrong with Yury's proposal in regards full-text > > > search > > > > > API > > > > > > > evolution as long as Yury is ready to push it forward. > > > > > > > > > > > > > > As for the in-memory mode only, it makes total sense for > > in-memory > > > data > > > > > > > grid deployments when Ignite caches data of an underlying DB > like > > > > > Postgres. > > > > > > > As part of the changes, I would simply throw an exception (by > > > default) > > > > > if > > > > > > > the one attempts to use text indices with the native > persistence > > > > > enabled. > > > > > > > If the person is ready to live with that limitation that an > > > explicit > > > > > > > configuration change is needed to come around the exception. > > > > > > > > > > > > > > Th
Re: Replacing default work dir from tmp to current dir
Hello! I want to point out that I didn't change this location (current dir). It was already implemented when I raised this issue, the only change I did was to swap current dir/work to current dir/ignite/work to avoid confusion whose work dir that is. I also communicated this to you all in ML when I discovered that current dir is used. I think that current dir is actually *well defined* when starting a project. A project is expected to be started from the same dir, and all "Run..." dialogs usually allow specifying that one. For embedded scenarios, you definitely not want work dir from two different Ignite-using tools to interfere. For embedded scenarios, you should only expect that current dir is writable. Even after these considerations, it's too late to change that because people don't expect this dir to move with every release of Ignite, and we already did it once. Regards, -- Ilya Kasnacheev чт, 3 окт. 2019 г. в 17:34, Alexey Goncharuk : > > > > Seems, we should have different defaults and even distributions for > > different usage scenarios. > > > I still do not understand why defaults should be different for embedded and > "traditional RDBMS-like" installations. Having different defaults will > likely confuse users, not make usability easier. Personally, I would forbid > to start Ignite if IGNITE_HOME is not set, but this suggestion was not > accepted by the community. > > As far as I know, both rocksdb and SQLite is local only libraries and don't > > have any distrubted features. > > See no difference here. Imagine a user starts only one Ignite node for > development or just to play (which, I believe, happes quite a lot) - same > as with local databases. BTW, it is impossible to start SQLite without > database path, so a user either provides a full path, or a relative path > from the current directory - which is an explicit action from a user. > > > > I agree with you. > > How it happens, that after wide discussion we implemented, reviewed and > > merged wrong defaults? > > > > As I know, we have explicit release only to change this default. > > > > This release is broken, isn't it? > > > I think this is just a miscommunication. Ilya made a fix which was exactly > what he meant it to be. As for the release - it may have worse usability, > but not more 'broken' as the previous one with the temp directory. At least > the data will not be physically removed after the machine restart. >
Re: Replacing default work dir from tmp to current dir
> Cannot tell about NuGet. Maven is typically used during development, usually there is no Maven in production deployments. NuGet and Maven are very similar. Yes, both of them are build-time tools, production is unrelated. For production-ready deployments we can expect users to tweak Ignite to their needs, set custom storage dirs, adjust heap sizes and so on. I'm talking about new users, about "getting started" scenarios - it is super important to make Ignite easy to get started with, provide reasonable defaults for all the configuration properties. For Ignite.NET, LINQPad is one of those "get started in 2 clicks" scenarios. And this scenario got broken as explained above. 2.7.5 and earlier used temp dir, which worked. 2.7.6 fails: "Work directory does not exist and cannot be created: C:\Program Files\LINQPad5\ignite\work" For Java there is JPad, which will fail in the same way - when you run code from there, `user.dir` points to Program Files. I expect that there are more use cases like this, and `user.home` is a reasonable solution. On Thu, Oct 3, 2019 at 5:56 PM Ilya Kasnacheev wrote: > Hello! > > I want to point out that I didn't change this location (current dir). It > was already implemented when I raised this issue, the only change I did was > to swap current dir/work to current dir/ignite/work to avoid confusion > whose work dir that is. > > I also communicated this to you all in ML when I discovered that current > dir is used. > > I think that current dir is actually *well defined* when starting a > project. A project is expected to be started from the same dir, and all > "Run..." dialogs usually allow specifying that one. > > For embedded scenarios, you definitely not want work dir from two different > Ignite-using tools to interfere. For embedded scenarios, you should only > expect that current dir is writable. > > Even after these considerations, it's too late to change that because > people don't expect this dir to move with every release of Ignite, and we > already did it once. > > Regards, > -- > Ilya Kasnacheev > > > чт, 3 окт. 2019 г. в 17:34, Alexey Goncharuk : > > > > > > > Seems, we should have different defaults and even distributions for > > > different usage scenarios. > > > > > I still do not understand why defaults should be different for embedded > and > > "traditional RDBMS-like" installations. Having different defaults will > > likely confuse users, not make usability easier. Personally, I would > forbid > > to start Ignite if IGNITE_HOME is not set, but this suggestion was not > > accepted by the community. > > > > As far as I know, both rocksdb and SQLite is local only libraries and > don't > > > have any distrubted features. > > > > See no difference here. Imagine a user starts only one Ignite node for > > development or just to play (which, I believe, happes quite a lot) - same > > as with local databases. BTW, it is impossible to start SQLite without > > database path, so a user either provides a full path, or a relative path > > from the current directory - which is an explicit action from a user. > > > > > > > I agree with you. > > > How it happens, that after wide discussion we implemented, reviewed and > > > merged wrong defaults? > > > > > > As I know, we have explicit release only to change this default. > > > > > > This release is broken, isn't it? > > > > > I think this is just a miscommunication. Ilya made a fix which was > exactly > > what he meant it to be. As for the release - it may have worse usability, > > but not more 'broken' as the previous one with the temp directory. At > least > > the data will not be physically removed after the machine restart. > > >
Re: How to free up space on disc after removing entries from IgniteCache with enabled PDS?
Hi I'm not sure that node offline is a best way to do that. Cons: - different caches may have different defragmentation but we force to stop whole node - offline node is a maintenance operation will require to add +1 backup to reduce the risk of data loss - baseline auto adjustment? - impact to index rebuild? - cache configuration changes (or destroy) during node offline What about other ways without node stop? E.g. make cache group on a node offline? Add *defrag *command to control.sh to force start rebalance internally in the node with expected impact to performance. On Thu, Oct 3, 2019 at 12:08 PM Anton Vinogradov wrote: > Alexey, > As for me, it does not matter will it be IEP, umbrella or a single issue. > The most important thing is Assignee :) > > On Thu, Oct 3, 2019 at 11:59 AM Alexey Goncharuk < > alexey.goncha...@gmail.com> > wrote: > > > Anton, do you think we should file a single ticket for this or should we > go > > with an IEP? As of now, the change does not look big enough for an IEP > for > > me. > > > > чт, 3 окт. 2019 г. в 11:18, Anton Vinogradov : > > > > > Alexey, > > > > > > Sounds good to me. > > > > > > On Thu, Oct 3, 2019 at 10:51 AM Alexey Goncharuk < > > > alexey.goncha...@gmail.com> > > > wrote: > > > > > > > Anton, > > > > > > > > Switching a partition to and from the SHRINKING state will require > > > > intricate synchronizations in order to properly determine the start > > > > position for historical rebalance without PME. > > > > > > > > I would still go with an offline-node approach, but instead of > cleaning > > > the > > > > persistence, we can do effective defragmentation when the node is > > offline > > > > because we are sure that there is no concurrent load. After the > > > > defragmentation completes, we bring the node back to the cluster and > > > > historical rebalance will kick in automatically. It will still > require > > > > manual node restarts, but since the data is not removed, there are no > > > > additional risks. Also, this will be an excellent solution for those > > who > > > > can afford downtime and execute the defragment command on all nodes > in > > > the > > > > cluster simultaneously - this will be the fastest way possible. > > > > > > > > --AG > > > > > > > > пн, 30 сент. 2019 г. в 09:29, Anton Vinogradov : > > > > > > > > > Alexei, > > > > > >> stopping fragmented node and removing partition data, then > > starting > > > it > > > > > again > > > > > > > > > > That's exactly what we're doing to solve the fragmentation issue. > > > > > The problem here is that we have to perform N/B restart-rebalance > > > > > operations (N - cluster size, B - backups count) and it takes a lot > > of > > > > time > > > > > with risks to lose the data. > > > > > > > > > > On Fri, Sep 27, 2019 at 5:49 PM Alexei Scherbakov < > > > > > alexey.scherbak...@gmail.com> wrote: > > > > > > > > > > > Probably this should be allowed to do using public API, actually > > this > > > > is > > > > > > same as manual rebalancing. > > > > > > > > > > > > пт, 27 сент. 2019 г. в 17:40, Alexei Scherbakov < > > > > > > alexey.scherbak...@gmail.com>: > > > > > > > > > > > > > The poor man's solution for the problem would be stopping > > > fragmented > > > > > node > > > > > > > and removing partition data, then starting it again allowing > full > > > > state > > > > > > > transfer already without deletes. > > > > > > > Rinse and repeat for all owners. > > > > > > > > > > > > > > Anton Vinogradov, would this work for you as workaround ? > > > > > > > > > > > > > > чт, 19 сент. 2019 г. в 13:03, Anton Vinogradov >: > > > > > > > > > > > > > >> Alexey, > > > > > > >> > > > > > > >> Let's combine your and Ivan's proposals. > > > > > > >> > > > > > > >> >> vacuum command, which acquires exclusive table lock, so no > > > > > concurrent > > > > > > >> activities on the table are possible. > > > > > > >> and > > > > > > >> >> Could the problem be solved by stopping a node which needs > to > > > be > > > > > > >> defragmented, clearing persistence files and restarting the > > node? > > > > > > >> >> After rebalancing the node will receive all data back > without > > > > > > >> fragmentation. > > > > > > >> > > > > > > >> How about to have special partition state SHRINKING? > > > > > > >> This state should mean that partition unavailable for reads > and > > > > > updates > > > > > > >> but > > > > > > >> should keep it's update-counters and should not be marked as > > lost, > > > > > > renting > > > > > > >> or evicted. > > > > > > >> At this state we able to iterate over the partition and apply > > it's > > > > > > entries > > > > > > >> to another file in a compact way. > > > > > > >> Indices should be updated during the copy-on-shrink procedure > or > > > at > > > > > the > > > > > > >> shrink completion. > > > > > > >> Once shrank file is ready we should replace the original > > partition > > > > > file > > > > > > >> with it and mark it as MOVING which will start the historical > > >
Re: Replacing default work dir from tmp to current dir
Hello! We can try and fallback to home dir with warning, when file cannot be created in current dir. WDYT? Regards, -- Ilya Kasnacheev чт, 3 окт. 2019 г. в 20:05, Pavel Tupitsyn : > > Cannot tell about NuGet. Maven is typically used during development, > usually there is no Maven in production deployments. > NuGet and Maven are very similar. Yes, both of them are build-time tools, > production is unrelated. > For production-ready deployments we can expect users to tweak Ignite to > their needs, set custom storage dirs, adjust heap sizes and so on. > > I'm talking about new users, about "getting started" scenarios - > it is super important to make Ignite easy to get started with, provide > reasonable defaults for all the configuration properties. > > For Ignite.NET, LINQPad is one of those "get started in 2 clicks" > scenarios. And this scenario got broken as explained above. > 2.7.5 and earlier used temp dir, which worked. 2.7.6 fails: "Work directory > does not exist and cannot be created: C:\Program > Files\LINQPad5\ignite\work" > > For Java there is JPad, which will fail in the same way - when you run code > from there, `user.dir` points to Program Files. > > I expect that there are more use cases like this, and `user.home` is a > reasonable solution. > > On Thu, Oct 3, 2019 at 5:56 PM Ilya Kasnacheev > wrote: > > > Hello! > > > > I want to point out that I didn't change this location (current dir). It > > was already implemented when I raised this issue, the only change I did > was > > to swap current dir/work to current dir/ignite/work to avoid confusion > > whose work dir that is. > > > > I also communicated this to you all in ML when I discovered that current > > dir is used. > > > > I think that current dir is actually *well defined* when starting a > > project. A project is expected to be started from the same dir, and all > > "Run..." dialogs usually allow specifying that one. > > > > For embedded scenarios, you definitely not want work dir from two > different > > Ignite-using tools to interfere. For embedded scenarios, you should only > > expect that current dir is writable. > > > > Even after these considerations, it's too late to change that because > > people don't expect this dir to move with every release of Ignite, and we > > already did it once. > > > > Regards, > > -- > > Ilya Kasnacheev > > > > > > чт, 3 окт. 2019 г. в 17:34, Alexey Goncharuk >: > > > > > > > > > > Seems, we should have different defaults and even distributions for > > > > different usage scenarios. > > > > > > > I still do not understand why defaults should be different for embedded > > and > > > "traditional RDBMS-like" installations. Having different defaults will > > > likely confuse users, not make usability easier. Personally, I would > > forbid > > > to start Ignite if IGNITE_HOME is not set, but this suggestion was not > > > accepted by the community. > > > > > > As far as I know, both rocksdb and SQLite is local only libraries and > > don't > > > > have any distrubted features. > > > > > > See no difference here. Imagine a user starts only one Ignite node for > > > development or just to play (which, I believe, happes quite a lot) - > same > > > as with local databases. BTW, it is impossible to start SQLite without > > > database path, so a user either provides a full path, or a relative > path > > > from the current directory - which is an explicit action from a user. > > > > > > > > > > I agree with you. > > > > How it happens, that after wide discussion we implemented, reviewed > and > > > > merged wrong defaults? > > > > > > > > As I know, we have explicit release only to change this default. > > > > > > > > This release is broken, isn't it? > > > > > > > I think this is just a miscommunication. Ilya made a fix which was > > exactly > > > what he meant it to be. As for the release - it may have worse > usability, > > > but not more 'broken' as the previous one with the temp directory. At > > least > > > the data will not be physically removed after the machine restart. > > > > > >
Re: Issue with adding nested index dynamically
Hi, thanks for starting this thread!
SQL folks, could you please confirm that's neither known nor expected
limitation? If that's a real issue then we will just need to go through a
standard contribution process.
-
Denis
On Thu, Oct 3, 2019 at 12:19 PM Hemambara wrote:
> We have to add indexes on cache dynamically on java pojo with nested
> objects.
> In the below example we do not have @QuerySqlField. In this case if I try
> to
> add index on "username" dynamically using "CREATE INDEX" it worked. But if
> I
> want to add index on "Address.zipcode" - this is not working as we are NOT
> creating aliases for these fields which are getting added dynamically. I
> see
> there is a bug in this implementation. I fixed the bug (please see below)
> and rebuilt the jar in my local and it worked fine. Proposing this bug to
> this open source community so that it can be fixed in ignite and we can
> start using the jar from central repo. We have just started using ignite
> and
> planning to take this to production. So it would be helpful if we can
> implement this bug fix.
>
> User{
> String userName;
> Address address;
>
> }
>
> Address{
> String streetName;
> String zipcode;
> }
>
> Bug Fix: QueryUtils.class
> --
> Method name :
> --
> QueryBinaryProperty buildBinaryProperty(GridKernalContext ctx, String
> pathStr,
> Class resType, Map aliases, @Nullable Boolean
> isKeyField, boolean notNull, Object dlftVal,
> int precision, int scale)
>
>
>
> Change from :
> -
> String alias = (String)aliases.get(fullName.toString());
>
> to :
> -
> String alias = aliases.get(fullName.toString()) == null ?
> fullName.toString() : aliases.get(fullName.toString());
>
> With this way we will always make sure we have default alias (if not
> provided), otherwise aliases on dynamic query fields and dynamic indexes
> are
> not working properly
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Replacing default work dir from tmp to current dir
Ilya, fallback is a good idea. Still I'd prefer to have user.home as a default, and fallback to user.dir when home does not work for some reason. On Thu, Oct 3, 2019 at 11:07 PM Ilya Kasnacheev wrote: > Hello! > > We can try and fallback to home dir with warning, when file cannot be > created in current dir. > > WDYT? > > Regards, > -- > Ilya Kasnacheev > > > чт, 3 окт. 2019 г. в 20:05, Pavel Tupitsyn : > > > > Cannot tell about NuGet. Maven is typically used during development, > > usually there is no Maven in production deployments. > > NuGet and Maven are very similar. Yes, both of them are build-time tools, > > production is unrelated. > > For production-ready deployments we can expect users to tweak Ignite to > > their needs, set custom storage dirs, adjust heap sizes and so on. > > > > I'm talking about new users, about "getting started" scenarios - > > it is super important to make Ignite easy to get started with, provide > > reasonable defaults for all the configuration properties. > > > > For Ignite.NET, LINQPad is one of those "get started in 2 clicks" > > scenarios. And this scenario got broken as explained above. > > 2.7.5 and earlier used temp dir, which worked. 2.7.6 fails: "Work > directory > > does not exist and cannot be created: C:\Program > > Files\LINQPad5\ignite\work" > > > > For Java there is JPad, which will fail in the same way - when you run > code > > from there, `user.dir` points to Program Files. > > > > I expect that there are more use cases like this, and `user.home` is a > > reasonable solution. > > > > On Thu, Oct 3, 2019 at 5:56 PM Ilya Kasnacheev < > ilya.kasnach...@gmail.com> > > wrote: > > > > > Hello! > > > > > > I want to point out that I didn't change this location (current dir). > It > > > was already implemented when I raised this issue, the only change I did > > was > > > to swap current dir/work to current dir/ignite/work to avoid confusion > > > whose work dir that is. > > > > > > I also communicated this to you all in ML when I discovered that > current > > > dir is used. > > > > > > I think that current dir is actually *well defined* when starting a > > > project. A project is expected to be started from the same dir, and all > > > "Run..." dialogs usually allow specifying that one. > > > > > > For embedded scenarios, you definitely not want work dir from two > > different > > > Ignite-using tools to interfere. For embedded scenarios, you should > only > > > expect that current dir is writable. > > > > > > Even after these considerations, it's too late to change that because > > > people don't expect this dir to move with every release of Ignite, and > we > > > already did it once. > > > > > > Regards, > > > -- > > > Ilya Kasnacheev > > > > > > > > > чт, 3 окт. 2019 г. в 17:34, Alexey Goncharuk < > alexey.goncha...@gmail.com > > >: > > > > > > > > > > > > > Seems, we should have different defaults and even distributions for > > > > > different usage scenarios. > > > > > > > > > I still do not understand why defaults should be different for > embedded > > > and > > > > "traditional RDBMS-like" installations. Having different defaults > will > > > > likely confuse users, not make usability easier. Personally, I would > > > forbid > > > > to start Ignite if IGNITE_HOME is not set, but this suggestion was > not > > > > accepted by the community. > > > > > > > > As far as I know, both rocksdb and SQLite is local only libraries and > > > don't > > > > > have any distrubted features. > > > > > > > > See no difference here. Imagine a user starts only one Ignite node > for > > > > development or just to play (which, I believe, happes quite a lot) - > > same > > > > as with local databases. BTW, it is impossible to start SQLite > without > > > > database path, so a user either provides a full path, or a relative > > path > > > > from the current directory - which is an explicit action from a user. > > > > > > > > > > > > > I agree with you. > > > > > How it happens, that after wide discussion we implemented, reviewed > > and > > > > > merged wrong defaults? > > > > > > > > > > As I know, we have explicit release only to change this default. > > > > > > > > > > This release is broken, isn't it? > > > > > > > > > I think this is just a miscommunication. Ilya made a fix which was > > > exactly > > > > what he meant it to be. As for the release - it may have worse > > usability, > > > > but not more 'broken' as the previous one with the temp directory. At > > > least > > > > the data will not be physically removed after the machine restart. > > > > > > > > > >
GridGain Web Console is available free of charge for Apache Ignite
Igniters, There is good news. GridGain made its distribution of Web Console completely free. It goes with advanced monitoring and management dashboard and other handy screens. More details are here: https://www.gridgain.com/resources/blog/gridgain-road-simplicity-new-docs-and-free-tools-apache-ignite - Denis
[MTCGA]: new failures in builds [4657112] needs to be handled
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. If your changes can lead to this failure(s): We're grateful that you were a volunteer to make the contribution to this project, but things change and you may no longer be able to finalize your contribution. Could you respond to this email and indicate if you wish to continue and fix test failures or step down and some committer may revert you commit. *Recently contributed test failed in master JavaEmbeddedIgniteRDDWithLocalStoreSelfTest.testStoreDataToIgniteWithOptionSkipStore https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=5554421907791235109&branch=%3Cdefault%3E&tab=testDetails Changes may lead to failure were done by - kcmvp https://ci.ignite.apache.org/viewModification.html?modId=891036 - alexey zinoviev https://ci.ignite.apache.org/viewModification.html?modId=891032 - Here's a reminder of what contributors were agreed to do https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute - Should you have any questions please contact dev@ignite.apache.org Best Regards, Apache Ignite TeamCity Bot https://github.com/apache/ignite-teamcity-bot Notification generated at 04:53:56 04-10-2019
Re: [MTCGA]: new failures in builds [4657112] needs to be handled
Ok, will try to fix it today пт, 4 окт. 2019 г., 4:54 : > Hi Igniters, > > I've detected some new issue on TeamCity to be handled. You are more than > welcomed to help. > > If your changes can lead to this failure(s): We're grateful that you were > a volunteer to make the contribution to this project, but things change and > you may no longer be able to finalize your contribution. > Could you respond to this email and indicate if you wish to continue and > fix test failures or step down and some committer may revert you commit. > > *Recently contributed test failed in master > JavaEmbeddedIgniteRDDWithLocalStoreSelfTest.testStoreDataToIgniteWithOptionSkipStore > > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=5554421907791235109&branch=%3Cdefault%3E&tab=testDetails > Changes may lead to failure were done by > - kcmvp > https://ci.ignite.apache.org/viewModification.html?modId=891036 > - alexey zinoviev > https://ci.ignite.apache.org/viewModification.html?modId=891032 > > - Here's a reminder of what contributors were agreed to do > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute > - Should you have any questions please contact > dev@ignite.apache.org > > Best Regards, > Apache Ignite TeamCity Bot > https://github.com/apache/ignite-teamcity-bot > Notification generated at 04:53:56 04-10-2019 >
