Potentially the pending compactions are a symptom and not the root
cause/problem.
When updating a 3rd column family with a larger sstable_size_in_mb it
looks like the schema may not be in a good state
[default@xxxx] UPDATE COLUMN FAMILY screenshots WITH
compaction_strategy=LeveledCompactionStrategy AND
compaction_strategy_options={sstable_size_in_mb: 200};
290cf619-57b0-3ad1-9ae3-e313290de9c9
Waiting for schema agreement...
Warning: unreachable nodes 10.8.30.102The schema has not settled in 10
seconds; further migrations are ill-advised until it does.
Versions are UNREACHABLE:[10.8.30.102],
290cf619-57b0-3ad1-9ae3-e313290de9c9:[10.8.30.15, 10.8.30.14, 10.8.30.13,
10.8.30.103, 10.8.30.104, 10.8.30.105, 10.8.30.106],
f1de54f5-8830-31a6-9cdd-aaa6220cccd1:[10.8.30.101]
However, tpstats looks good. And the schema changes eventually do get
applied on *all* the nodes (even the ones that seem to have different
schema versions). There are no communications issues between the nodes and
they are all in the same rack
root@xxxx:~# nodetool tpstats
Pool Name Active Pending Completed Blocked
All time blocked
ReadStage 0 0 1254592 0
0
RequestResponseStage 0 0 9480827 0
0
MutationStage 0 0 8662263 0
0
ReadRepairStage 0 0 339158 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 0 1469197 0
0
AntiEntropyStage 0 0 0 0
0
MigrationStage 0 0 1808 0
0
MemtablePostFlusher 0 0 248 0
0
StreamStage 0 0 0 0
0
FlushWriter 0 0 248 0
4
MiscStage 0 0 0 0
0
commitlog_archiver 0 0 0 0
0
InternalResponseStage 0 0 5286 0
0
HintedHandoff 0 0 21 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
BINARY 0
READ 0
MUTATION 0
REQUEST_RESPONSE 0
So I'm guessing maybe the different schema versions may be potentially
stopping compactions? Will compactions still happen if there are different
versions of the schema?
On 9/18/12 11:38 AM, "Michael Kjellman" <[email protected]> wrote:
>Thanks, I just modified the schema on the worse offending column family
>(as determined by the .json) from 10MB to 200MB.
>
>Should I kick off a compaction on this cf now/repair?/scrub?
>
>Thanks
>
>-michael
>
>From: Віталій Тимчишин <[email protected]<mailto:[email protected]>>
>Reply-To: "[email protected]<mailto:[email protected]>"
><[email protected]<mailto:[email protected]>>
>To: "[email protected]<mailto:[email protected]>"
><[email protected]<mailto:[email protected]>>
>Subject: Re: persistent compaction issue (1.1.4 and 1.1.5)
>
>I've started to use LeveledCompaction some time ago and from my
>experience this indicates some SST on lower levels than they should be.
>The compaction is going, moving them up level by level, but total count
>does not change as new data goes in.
>The numbers are pretty high as for me. Such numbers mean a lot of files
>(over 100K in single directory) and a lot of thinking for compaction
>executor to decide what to compact next. I can see numbers like 5K-10K
>and still thing this is high number. If I were you, I'd increase
>sstable_size_in_mb 10-20 times it is now.
>
>2012/9/17 Michael Kjellman
><[email protected]<mailto:[email protected]>>
>Hi All,
>
>I have an issue where each one of my nodes (currently all running at
>1.1.5) is reporting around 30,000 pending compactions. I understand that
>a pending compaction doesn't necessarily mean it is a scheduled task
>however I'm confused why this behavior is occurring. It is the same on
>all nodes, occasionally goes down 5k pending compaction tasks, and then
>returns to 25,000-35,000 compaction tasks pending.
>
>I have tried a repair operation/scrub operation on two of the nodes and
>while compactions initially happen the number of pending compactions does
>not decrease.
>
>Any ideas? Thanks for your time.
>
>Best,
>michael
>
>
>'Like' us on Facebook for exclusive content and other resources on all
>Barracuda Networks solutions.
>
>Visit http://barracudanetworks.com/facebook
>
>
>
>
>
>
>
>--
>Best regards,
> Vitalii Tymchyshyn
>
>'Like' us on Facebook for exclusive content and other resources on all
>Barracuda Networks solutions.
>
>Visit http://barracudanetworks.com/facebook
>
>
>
>
'Like' us on Facebook for exclusive content and other resources on all
Barracuda Networks solutions.
Visit http://barracudanetworks.com/facebook
