[
https://issues.apache.org/jira/browse/CASSANDRA-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12973361#action_12973361
]
Carlos Alvarez commented on CASSANDRA-1876:
-------------------------------------------
In the case depicted by Germán, we could have a couple of thousands of small
sstables but global, 'umbrella', index. This 'umbrella' index would be
compacted, so a lookup for any key would have to be performed one time for any
group of sstables under an umbrella index.
We think that this way we would have a compacted row (because there is only one
pointer in the umbrella index: a record in this index would have a sstable
identifier and a offset inside the sstable) but without the cost of writing
billions of unchanged rows to a new file (the cost of reading the entire
sstables still persists).
Suposse there are 8 sstables to compact in one bucket. The process would look
like:
- Open the iterator to read all the columns in all the sstables
- Reduce the columns as it is done in the current version
- Just write the sstable id and offset of this column to a new index (global to
all the eight sstables being 'semicompacted') and update the new, global, bloom
filter.
The new 'compacted' sstable is just one (umbrella) index, one bloom filter and
eight sstables. Reading a column would imply:
- Checking the bloom filter (assume a true positive)
- Find the value in the index (there is a increased cost because the index
pointers would include the sstable id)
- Find the value in the corresponding sstable.
> Allow minor Parallel Compaction
> -------------------------------
>
> Key: CASSANDRA-1876
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1876
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Germán Kondolf
> Priority: Minor
> Attachments: 1876-reformatted.txt, compactionPatch-V2.txt
>
>
> Hi,
> According to the dev's list discussion (1) I've patched the CompactionManager
> to allow parallel compaction.
> Mainly it splits the sstables to compact in the desired buckets, configured
> by a new parameter: compaction_parallelism with the current default of "1".
> Then, it just submits the units of work to a new executor and waits for the
> finalization.
> The patch was created in the trunk, so I don't know the exact affected
> version, I assume that is 0.8.
> I'll try to apply this patch to 0.6.X also for my current production
> installation, and then reattach it.
> (1) http://markmail.org/thread/cldnqfh3s3nufnke
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
