[
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216890#comment-16216890
]
Andrey Kuznetsov commented on IGNITE-5553:
------------------------------------------
Indeed, we can put current set size into set header cache entry. This will fix
size(), but we have broken iterator() implementation as well.
Currently, set implementation maintains plain Java sets on every node, see
CacheDataStructuresManager.setDataMap. These sets duplicate backing-cache
entries, both primary and backup. size() and iterator() calls issue distributed
queries to collect/filter data from all setDataMap's. And setDataMaps remain
empty after cluster is recovered from checkpoint.
Now I see the following options to fix the issue.
#1 - Naive. Iterate over all datastructure-backing caches entries during
recover from checkpoint procedure, filter set-related entries and refill
setDataMap's.
Pros: easy to implement
Cons: inpredictable time/memory overhead.
#2 - More realistic. Avoid node-local copies of cache data. Maintain linked
list in datastructure-backing cache: key is set item, value is next set item.
List head is stored in set header cache entry (this set item is youngest one).
Iterators build on top of this structure are fail-fast.
Pros: less memory overhead, no need to maintain node-local mirrors of cache data
Cons: iterators are not fail-safe.
#3 - Option #2 modified. We can store reference counter and 'removed' flag
along with next item reference. This allows to make iterators fail safe.
Pros: iterators are fail-safe
Cons: slightly more complicated implementation, may affect performance, also I
see no way to handle active iterators on remote nodes failures.
[~sergey-chugunov], any thoughts? Or maybe we should discuss this on devlist?
> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -----------------------------------------------------------------------------
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
> Issue Type: Bug
> Components: data structures, persistence
> Affects Versions: 2.1
> Reporter: Dmitriy Pavlov
> Assignee: Andrey Kuznetsov
> Priority: Critical
> Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link
> to test
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests&testNameId=-7043871603266099589&tab=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does:
> size of *IgniteSet* must be stored is cache instead of volatile in-memory
> fields of random classes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
