[
https://issues.apache.org/jira/browse/CASSANDRA-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Milan Majercik updated CASSANDRA-12796:
---------------------------------------
Description:
We have a table with rather wide partition with a secondary index defined over
it. As soon as we try to rebuild the index we observed exhaustion of Java heap
and eventual OOM error. After a lengthy investigation we have managed to find a
culprit which appears to be a wrong granule of barrier issuances in method
{{org.apache.cassandra.db.Keyspace.indexRow}}:
{code}
try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start()){html}
{
Set<SecondaryIndex> indexes =
cfs.indexManager.getIndexesByNames(idxNames);
Iterator<ColumnFamily> pager = QueryPagers.pageRowLocally(cfs,
key.getKey(), DEFAULT_PAGE_SIZE);
while (pager.hasNext())
{
ColumnFamily cf = pager.next();
ColumnFamily cf2 = cf.cloneMeShallow();
for (Cell cell : cf)
{
if (cfs.indexManager.indexes(cell.name(), indexes))
cf2.addColumn(cell);
}
cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
}
}
{code}
Please note the operation group granule is a partition of the source table
which poses a problem for wide partition tables as flush runnable
({{org.apache.cassandra.db.ColumnFamilyStore.Flush.run()}}) won't proceed with
flushing secondary index memtable before completing operations prior recent
issue of the barrier. In our situation the flush runnable waits until whole
wide partition gets indexed into the secondary index memtable before flushing
it causing an exhaustion of the heap and eventual OOM error.
was:
We have a table with rather wide partition with a secondary index defined over
it. As soon as we try to rebuild the index we observed exhaustion of Java heap
and eventual OOM error. After a lengthy investigation we have managed to find a
culprit which appears to be a wrong granule of barrier issuances in method
{{org.apache.cassandra.db.Keyspace.indexRow}}:
{code}
{html}<b>try (OpOrder.Group opGroup =
cfs.keyspace.writeOrder.start())</b>{html}
{
Set<SecondaryIndex> indexes =
cfs.indexManager.getIndexesByNames(idxNames);
Iterator<ColumnFamily> pager = QueryPagers.pageRowLocally(cfs,
key.getKey(), DEFAULT_PAGE_SIZE);
while (pager.hasNext())
{
ColumnFamily cf = pager.next();
ColumnFamily cf2 = cf.cloneMeShallow();
for (Cell cell : cf)
{
if (cfs.indexManager.indexes(cell.name(), indexes))
cf2.addColumn(cell);
}
cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
}
}
{code}
> Heap exhaustion when rebuilding secondary index over a table with wide
> partitions
> ---------------------------------------------------------------------------------
>
> Key: CASSANDRA-12796
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12796
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Milan Majercik
> Priority: Critical
>
> We have a table with rather wide partition with a secondary index defined
> over it. As soon as we try to rebuild the index we observed exhaustion of
> Java heap and eventual OOM error. After a lengthy investigation we have
> managed to find a culprit which appears to be a wrong granule of barrier
> issuances in method {{org.apache.cassandra.db.Keyspace.indexRow}}:
> {code}
> try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start()){html}
> {
> Set<SecondaryIndex> indexes =
> cfs.indexManager.getIndexesByNames(idxNames);
> Iterator<ColumnFamily> pager = QueryPagers.pageRowLocally(cfs,
> key.getKey(), DEFAULT_PAGE_SIZE);
> while (pager.hasNext())
> {
> ColumnFamily cf = pager.next();
> ColumnFamily cf2 = cf.cloneMeShallow();
> for (Cell cell : cf)
> {
> if (cfs.indexManager.indexes(cell.name(), indexes))
> cf2.addColumn(cell);
> }
> cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
> }
> }
> {code}
> Please note the operation group granule is a partition of the source table
> which poses a problem for wide partition tables as flush runnable
> ({{org.apache.cassandra.db.ColumnFamilyStore.Flush.run()}}) won't proceed
> with flushing secondary index memtable before completing operations prior
> recent issue of the barrier. In our situation the flush runnable waits until
> whole wide partition gets indexed into the secondary index memtable before
> flushing it causing an exhaustion of the heap and eventual OOM error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
