Can you please share the code? On Fri, Feb 4, 2022 at 8:36 AM Sumit Deshinge <sumit.deshi...@gmail.com> wrote:
> That's correct, but this is when we use Pessimistic + Repeatable_Read > transaction at Server side where we are using an iterator to read, remove > and insert into new cache in the transaction. And this issue is not > observed every time, but intermittently. > > And note that, if I don't use transactions or use Optimistic + > Serializable transactions, then everything works fine. > > On Thu, Feb 3, 2022 at 11:24 PM Pavel Tupitsyn <ptupit...@apache.org> > wrote: > >> So we have the following situation: >> * Put 5000 unique keys with putAll >> * Use cache iterator, observe less than 5000 keys >> >> Is that correct? >> >> On Thu, Feb 3, 2022 at 7:53 PM Sumit Deshinge <sumit.deshi...@gmail.com> >> wrote: >> >>> Yes, that I am sure, because the keys are generated using ignite uuid, >>> which internally is based on hostname, and all the clients are hosted on >>> machines with unique hostnames. >>> >>> On Wed, Feb 2, 2022 at 3:23 PM Pavel Tupitsyn <ptupit...@apache.org> >>> wrote: >>> >>>> Are you sure that all entry keys are unique? >>>> E.g. if you do 5000 puts but some keys are the same, the result will be >>>> less than 5000 entries. >>>> >>>> On Wed, Feb 2, 2022 at 12:27 PM Sumit Deshinge < >>>> sumit.deshi...@gmail.com> wrote: >>>> >>>>> No, cache does not have entries. Somehow the number of entries >>>>> returned are less than the number of entries put by all thin clients. >>>>> >>>>> On Wed, Feb 2, 2022 at 1:33 PM Pavel Tupitsyn <ptupit...@apache.org> >>>>> wrote: >>>>> >>>>>> Do you mean that cache has some entries, but the iterator does not >>>>>> return them? >>>>>> >>>>>> On Wed, Feb 2, 2022 at 10:38 AM Sumit Deshinge < >>>>>> sumit.deshi...@broadcom.com> wrote: >>>>>> >>>>>>> Hi Pavel, >>>>>>> >>>>>>> I am trying to remove the data from one cache (on which I am >>>>>>> iterating) to another cache in transaction. >>>>>>> When the iterator says no further elements, I again try getting a >>>>>>> new iterator after few seconds, to check if there is any new data >>>>>>> available. >>>>>>> >>>>>>> In this process, I am missing one or two entries. But if I remove >>>>>>> the transaction or add optmistic+serializable instead >>>>>>> of pessimistic+repeatable_read transaction type, then this loss of data >>>>>>> is >>>>>>> not observed with the same steps mentioned. >>>>>>> >>>>>>> >>>>>>> On Wed, Feb 2, 2022 at 1:00 PM Pavel Tupitsyn <ptupit...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>>> > While iterating over the cache, data is removed from the cache >>>>>>>> >>>>>>>> Sumit, as I understand, you read data while you also remove it, so >>>>>>>> it is not clear what the expectation is. >>>>>>>> >>>>>>>> On Wed, Feb 2, 2022 at 10:28 AM Sumit Deshinge < >>>>>>>> sumit.deshi...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Thank you Surinder and Pavel. I will give this approach a try. >>>>>>>>> But even in case of iterator, when I try refreshing the iterator >>>>>>>>> once it reached to last record, i.e. new iterator, it does not give >>>>>>>>> all the >>>>>>>>> entries as described in the first email steps. >>>>>>>>> >>>>>>>>> On Fri, Jan 28, 2022 at 4:08 PM Pavel Tupitsyn < >>>>>>>>> ptupit...@apache.org> wrote: >>>>>>>>> >>>>>>>>>> Cache iterator does not guarantee that you'll see all entries if >>>>>>>>>> there are concurrent updates, I think you are facing a race >>>>>>>>>> condition. >>>>>>>>>> Please try ContinuousQuery as Surinder suggests, it will catch >>>>>>>>>> all data changes. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Jan 28, 2022 at 1:32 PM Surinder Mehra < >>>>>>>>>> redni...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Just curious, why can't we use continuous query here with >>>>>>>>>>> "appropriate" event type to write to another cache. So your >>>>>>>>>>> listener will >>>>>>>>>>> do below things >>>>>>>>>>> 1. Write entry to another cache >>>>>>>>>>> 2. remove entry from source cache >>>>>>>>>>> >>>>>>>>>>> Just an idea, please correct if I am wrong >>>>>>>>>>> >>>>>>>>>>> On Fri, Jan 28, 2022 at 3:49 PM Sumit Deshinge < >>>>>>>>>>> sumit.deshi...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> We are running apache ignite 2.11 with cache configuration >>>>>>>>>>>> FULL_SYNC and REPLICATED mode. >>>>>>>>>>>> >>>>>>>>>>>> Our use case is : >>>>>>>>>>>> >>>>>>>>>>>> 1. *Multiple thin clients are adding data* into a cache using >>>>>>>>>>>> putAll operation. >>>>>>>>>>>> 2. *Simultaneously the server is reading the data* using >>>>>>>>>>>> server cache iterator. >>>>>>>>>>>> 3. *While iterating over the cache, data is removed from the >>>>>>>>>>>> cache and added into new cache using a transaction*, i.e. >>>>>>>>>>>> transaction with remove and put operations. We have transaction >>>>>>>>>>>> *concurrency >>>>>>>>>>>> - pessimistic and isolation levels - repeatable_read*. >>>>>>>>>>>> >>>>>>>>>>>> But we are seeing few missing entries at server side, i.e. >>>>>>>>>>>> server is not able to read all the data put by client. E.g. in one >>>>>>>>>>>> of the >>>>>>>>>>>> run, all thin clients put 5000 entries, server is able to read >>>>>>>>>>>> only 4999 >>>>>>>>>>>> entries. Here we saw 1 entry was not read by server. >>>>>>>>>>>> >>>>>>>>>>>> *Another observation is that, if we remove the transaction in >>>>>>>>>>>> the second step above, or use optimistic transaction with >>>>>>>>>>>> serializable >>>>>>>>>>>> isolation level, then this issue is not observed*. >>>>>>>>>>>> >>>>>>>>>>>> What could be the possible problem in this use case >>>>>>>>>>>> with pessimistic concurrency and repeatable_read isolation level? >>>>>>>>>>>> This is >>>>>>>>>>>> particularly important as this configuration is resulting in data >>>>>>>>>>>> loss. >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Regards, >>>>>>>>>>>> Sumit Deshinge >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Regards, >>>>>>>>> Sumit Deshinge >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Sumit Deshinge >>>>>>> >>>>>>> R&D Engineer | Symantec Enterprise Division >>>>>>> >>>>>>> Broadcom Software >>>>>>> >>>>>>> Email: Sumit Deshinge <sumit.deshi...@broadcom.com> >>>>>>> >>>>>>> >>>>>>> This electronic communication and the information and any files >>>>>>> transmitted with it, or attached to it, are confidential and are >>>>>>> intended >>>>>>> solely for the use of the individual or entity to whom it is addressed >>>>>>> and >>>>>>> may contain information that is confidential, legally privileged, >>>>>>> protected >>>>>>> by privacy laws, or otherwise restricted from disclosure to anyone >>>>>>> else. If >>>>>>> you are not the intended recipient or the person responsible for >>>>>>> delivering >>>>>>> the e-mail to the intended recipient, you are hereby notified that any >>>>>>> use, >>>>>>> copying, distributing, dissemination, forwarding, printing, or copying >>>>>>> of >>>>>>> this e-mail is strictly prohibited. If you received this e-mail in >>>>>>> error, >>>>>>> please return the e-mail to the sender, delete it from your computer, >>>>>>> and >>>>>>> destroy any printed copy of it. >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> Sumit Deshinge >>>>> >>>>> >>> >>> -- >>> Regards, >>> Sumit Deshinge >>> >>> > > -- > Regards, > Sumit Deshinge > >
