Sumit, Can you please write down the all steps to reproduce the issue, or provide a reproducer for it? It is also nice to have the Ignite version you're using, and cache configuration from which you're trying to delete keys.
The log you've mentioned looks fine. On Mon, 9 May 2022 at 10:50, Sumit Deshinge <sumit.deshi...@gmail.com> wrote: > > Hi Maxim, > > Thank you for your reply. > 1. Read-Repair - I am using cache write sync mode FULL_SYNC and cache > rebalance mode as SYNC along with REPLICATED cache mode. So read repair may > not work in this case, right ? > 2. Also I verified partition checksum and the result was no conflicts found. > Even cache invalid state exception is also not observed. > > After ignite server restart, some of the cache entries are not deleted using > removeAll api, so I tried with remove api and the returned value was false. > > Note that while the ignite server was restarted, ignite clients are > continuously and parallelly re-trying to write the data into the caches, i.e. > when ignite server is down, cluster is inactive. > Can this create a potential consistency issue? > > Few logs after restart : > > WARNING: Nodes started on local machine require more than 80% of physical RAM > what can lead to significant slowdown due to swapping (please decrease JVM > heap size, data region size or checkpoint buffer size) [required=10852MB, > available=12287MB] > Starting binary memory restore for: [.........] > WARNING: Ignite node stopped in the middle of checkpoint. Will restore memory > state and finish checkpoint on node start. > INFO:Started page memory [memoryAllocated=2.0 GiB, pages=508192, > tableSize=39.5 MiB, replacementSize=62.1 KiB, checkpointBuffer=512.0 MiB] > INFO:Found last checkpoint marker [cpId=12f873e6-f191-4a40-8c2f-926c3bd3d036, > pos=WALPointer [idx=4, fileOff=68632206, len=955879]] > INFO:Finished applying memory changes [changesApplied=19636, time=391 ms] > INFO:Checkpoint finished [cpId=12f873e6-f191-4a40-8c2f-926c3bd3d036, > pages=7837, markPos=WALPointer [idx=4, fileOff=68632206, len=955879], > pagesWrite=766ms, fsync=24847ms, total=25613ms] > INFO:Binary memory state restored at node startup [restoredPtr=WALPointer > [idx=4, fileOff=69588085, len=0]] > INFO:Resuming logging to WAL segment > INFO:Started cache in recovery mode [name=mycache, id=-478150787, > dataRegionName=default, mode=REPLICATED, atomicity=ATOMIC, > backups=2147483647, mvcc=false] > INFO:Resolved page store work directory > INFO:Finished recovery for cache [cache=abc, grp=abc, > startVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]] > > > On Fri, May 6, 2022 at 7:36 PM Maxim Muzafarov <mmu...@apache.org> wrote: >> >> Hello, >> >> The crash-recovery machinery for the restarted node should works fine. >> >> Do you have any error during the cache key removal procedure? >> Have you checked the data using the Repair Read tool [1] (the >> CHECK_ONLY strategy [2]) or the idle verify procedure [3]? >> >> >> [1] https://ignite.apache.org/docs/latest/read-repair >> [2] https://blogs.apache.org/ignite/entry/apache-ignite-2-13-0 >> [3] >> https://ignite.apache.org/docs/latest/tools/control-script#verifying-partition-checksums >> >> On Thu, 5 May 2022 at 12:24, Sumit Deshinge <sumit.deshi...@gmail.com> wrote: >> > >> > Has anyone faced such an issue earlier? Any pointers are much appreciated. >> > >> > On Thu, Apr 28, 2022 at 10:20 PM Sumit Deshinge <sumit.deshi...@gmail.com> >> > wrote: >> >> >> >> Note that this is not happening with all the data set but only for cache >> >> entries which were written into ignite before server restart and tried to >> >> be deleted after restart. >> >> >> >> On Thu, Apr 28, 2022 at 10:18 PM Sumit Deshinge >> >> <sumit.deshi...@gmail.com> wrote: >> >>> >> >>> Hi, >> >>> >> >>> I am using ignite with native persistence enabled along with cache write >> >>> sync mode as FULL_SYNC and cache rebalance mode as sync. >> >>> >> >>> In normal workflow where there is no restart, I can see all data is >> >>> being written, read and deleted from cache successfully. >> >>> >> >>> But in case of ignite server restart, where some of the data is written >> >>> before restart, for the same set of data after restart, ignite is able >> >>> to read that data but not able to remove it from the cache. >> >>> I verified the hashcode and equals for the key I am using for remove and >> >>> the one in ignite is same. >> >>> But still remove operation does not actually remove data from cache. >> >>> Checked what's the result of contains operation and it gives false. >> >>> >> >>> This is observed only in case of restart. I can see there are warning as >> >>> below after restart but I expect ignite just gives these warnings before >> >>> it recovers : >> >>> - Ignite node stopped in the middle of checkpoint. Will restore memory >> >>> state and finish checkpoint on node start. >> >>> - Started cache in recovery mode .... >> >>> - Finished recovery for cache .... >> >>> >> >>> Any idea or configuration that needs to be done during such case ? >> >>> >> >>> -- >> >>> Regards, >> >>> Sumit Deshinge >> >>> >> >> >> >> >> >> -- >> >> Regards, >> >> Sumit Deshinge >> >> >> > >> > >> > -- >> > Regards, >> > Sumit Deshinge >> > > > > > -- > Regards, > Sumit Deshinge >
