Hello Piotr, That's a known problem and I thought a JIRA ticket already exists. However, failed to locate it. The ticket for the improvement should be created as a result of this conversation.
Speaking of an initial query type, I would differentiate from ScanQueries and SqlQueries. For the former, it sounds reasonable to apply the partitionCounter logic. As for the latter, Vladimir Ozerov will it be addressed as part of MVCC/Transactional SQL activities? Btw, Piotr what's your initial query type? -- Denis On Thu, Dec 13, 2018 at 3:28 AM Piotr Romański <piotr.roman...@gmail.com> wrote: > Hi, as suggested by Ilya here: > > http://apache-ignite-users.70518.x6.nabble.com/Continuous-queries-and-duplicates-td25314.html > I'm resending it to the developers list. > > From that thread we know that there might be duplicates between initial > query results and listener entries received as part of continuous query. > That means that users need to manually dedupe data. > > In my opinion the manual deduplication in some use cases may lead to > possible memory problems on the client side. In order to remove duplicated > notifications which we are receiving in the local listener, we need to keep > all initial query results in memory (or at least their unique ids). > Unfortunately, there is no way (is there?) to find a point in time when we > can be sure that no dups will arrive anymore. That would mean that we need > to keep that data indefinitely and use it every time a new notification > arrives. In case of multiple continuous queries run from a single JVM, this > might eventually become a memory or performance problem. I can see the > following possible improvements to Ignite: > > 1. The deduplication between initial query and incoming notification could > be done fully in Ignite. As far as I know there is already the > updateCounter and partition id for all the objects so it could be used > internally. > > 2. Add a guarantee that notifications arriving in the local listener after > query() method returns are not duplicates. This kind of functionality would > require a specific synchronization inside Ignite. It would also mean that > the query() method cannot return before all potential duplicates are > processed by a local listener what looks wrong. > > 3. Notify users that starting from a given notification they can be sure > they will not receive any duplicates anymore. This could be an additional > boolean flag in the CacheQueryEntryEvent. > > 4. CacheQueryEntryEvent already exposes the partitionUpdateCounter. > Unfortunately we don't have this information for initial query results. If > we had, a client could manually deduplicate notifications and get rid of > initial query results for a given partition after newer notifications > arrive. Also it would be very convenient to expose partition id as well but > now we can figure it out using the affinity service. The assumption here is > that notifications are ordered by partitionUpdateCounter (is it true?). > > Please correct me if I'm missing anything. > > What do you think? > > Piotr >
