[
https://issues.apache.org/jira/browse/CASSANDRA-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15687809#comment-15687809
]
anmols commented on CASSANDRA-12796:
------------------------------------
It looks like the proposed solution is only partially adequate for 3.0.x; while
it enables memtable _flushes_ to proceed while a partition is being indexed,
the read ordering {{OpGroup}} introduced in CASSANDRA-11905 continues to block
memtable memory from being _reclaimed_. In our environment, this ultimately
blocks new memtables from being allocated while indexing is underway, which in
turn blocks new mutations from finishing while they for new memtables to become
available. That ultimately also leads to heap exhaustion as blocked mutations
accumulate.
So it looks like the granularity of the read {{OpOrder}} lock also needs to be
reduced. Following the logic of CASSANDRA-11905, before 3.0.x that was
implicitly accomplished by using a pager to read the partition, which handled
the read {{OpOrder}} correctly behind the scenes. Is doing something like that
an option now?
> Heap exhaustion when rebuilding secondary index over a table with wide
> partitions
> ---------------------------------------------------------------------------------
>
> Key: CASSANDRA-12796
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12796
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Milan Majercik
> Priority: Critical
>
> We have a table with rather wide partition and a secondary index defined over
> it. As soon as we try to rebuild the index we observed exhaustion of Java
> heap and eventual OOM error. After a lengthy investigation we have managed to
> find a culprit which appears to be a wrong granule of barrier issuances in
> method {{org.apache.cassandra.db.Keyspace.indexRow}}:
> {code}
> try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start()){html}
> {
> Set<SecondaryIndex> indexes =
> cfs.indexManager.getIndexesByNames(idxNames);
> Iterator<ColumnFamily> pager = QueryPagers.pageRowLocally(cfs,
> key.getKey(), DEFAULT_PAGE_SIZE);
> while (pager.hasNext())
> {
> ColumnFamily cf = pager.next();
> ColumnFamily cf2 = cf.cloneMeShallow();
> for (Cell cell : cf)
> {
> if (cfs.indexManager.indexes(cell.name(), indexes))
> cf2.addColumn(cell);
> }
> cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
> }
> }
> {code}
> Please note the operation group granule is a partition of the source table
> which poses a problem for wide partition tables as flush runnable
> ({{org.apache.cassandra.db.ColumnFamilyStore.Flush.run()}}) won't proceed
> with flushing secondary index memtable before completing operations prior
> recent issue of the barrier. In our situation the flush runnable waits until
> whole wide partition gets indexed into the secondary index memtable before
> flushing it. This causes an exhaustion of the heap and eventual OOM error.
> After we changed granule of barrier issue in method
> {{org.apache.cassandra.db.Keyspace.indexRow}} to query page as opposed to
> table partition secondary index (see
> [https://github.com/mmajercik/cassandra/commit/7e10e5aa97f1de483c2a5faf867315ecbf65f3d6?diff=unified]),
> rebuild started to work without heap exhaustion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
