Hello! Unfortunately you seem to be right and this is an issue.
I have filed a ticket https://issues.apache.org/jira/browse/IGNITE-12425 to track it. Regards, -- Ilya Kasnacheev пн, 2 дек. 2019 г. в 16:33, peter108418 <[email protected]>: > Hi > > We have recently started to encounter what appears to be deadlocks on one > of > our new clusters. We believe it may be due to "data patterns" being > slightly > different and more dense than our other existing (working) production > clusters. We have some workarounds, but we think this might be an issue > with > Ignite. Hopefully someone is able to narrow down the cause further? :) > > Firstly, I'll describe the issues that we are seeing, and how to reproduce. > Then I'll try explain what we are trying to accomplish, maybe there is a > better solution to our problem? > > *The problem:* > We have encountered an odd deadlock issue when, on the same cache where > read-through is enabled, concurrently calls are made to "getAll" and > "invokeAll". We are sorting the keys similarly across both calls. > Replacing one "side" with either multiple "get"s, or multiple "invoke"s, > seems to fix the problem, but performance is worse. > > I have created a test case that can reproduce it. The test creates > - 1 thread doing a getAll({1, 2}), > - 2 threads doing an invokeAll({2, 3}) and an invokeAll({1, 3}) > > These 3 threads are executed, and may or may not end up in a deadlock, > usually the test case captures the deadlock state before 50 repetitions. > Please see attached sample maven project to reproduce: > https://drive.google.com/open?id=1GJ78dsulJ0XG-erNkN_vm3ordKr0nqS6 > Run with "mvn clean test" > > I have also posted the test code, (partial) log output and (partial) > stacktrace below. > > > *What we are trying to do:* > I believe our use-case to be fairly "normal"? We use Ignite as a > cache-layer, with a read-through adapter to a backing data-store. As data > continuously enters the backing data-store, we have a service that keeps > the > Ignite cache up-to-date. > > We have a large amount of historical data, going years back. The backing > data-store is the "master", we are not using Ignite Persistence. We use > Ignite as a cache layer as typically we recalculate on the same data > multiple times. We key our data by time chunks, where the "value" is a > container/collection of records within the time-range defined by the key. > We decided to go with an IgniteCache with read-through enabled to automate > cache-loading. To reduce the number of queries against the data-store, we > usually call "getAll" on the cache, as the resulting set of keys provided > to > the CacheStore.loadAll can often be merged into a smaller number of queries > (example: joining time-ranges "08:00:00 - 08:15:00" and "08:15:00 - > 08:30:00" to larger single time-range "08:00:00 - 08:30:00"). > > As we continuously load new data into the backing data-store, entries in > Ignite become inconsistent with the data-store, especially those around > "now" but out-of-order records also occur. > To handle this, we have a separate Ignite Service that fetches new records > from the data-store and updates the Ignite Cache using invokeAll and an > entry-processor. > Our reasoning here is to only forward the "new" records (in the scale of > 10s > of records) and merged them into the container (containing 1000s of > records) > "locally", instead of "getting" the container, merging and then "putting" > which would transfer a large amount of data back and forth. > > > > > > Relevant log fragment: > > > Dump of relevant threads: > > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
