Nate,
Definitely thank you for this advice. After leaving the new Cassandra node
running on the 2GB instance for the past couple of days, I think I've had
ample reason to report complete success in getting it stabilized on that
instance! Here are the changes I've been able to make:
I think manipulating the key cache and other stuff like concurrent writes
and some of the other stuff I worked on based on that thread from the
cassandra list definitely was key in getting Cassandra to work on the new
instance.
Check out the before and after (before working/ after working):
Before in cassandra-env.sh:
MAX_HEAP_SIZE="800M"
HEAP_NEWSIZE="200M"
After:
MAX_HEAP_SIZE="512M"
HEAP_NEWSIZE="100M"
And before in the cassandra.yaml file:
concurrent_writes: 32
compaction_throughput_mb_per_sec: 16
key_cache_size_in_mb:
key_cache_save_period: 14400
# native_transport_max_threads: 128
And after:
concurrent_writes: 2
compaction_throughput_mb_per_sec: 8
key_cache_size_in_mb: 4
key_cache_save_period: 0
native_transport_max_threads: 4
That really made the difference. I'm a puppet user, so these changes are in
puppet. So any new 2GB instances I should bring up on Digital Ocean should
absolutely work the way the first 2GB node does, there. But I was able to
make enough sense of your chef recipe to adapt what you were showing me.
Thanks again!
Tim
On Fri, Feb 20, 2015 at 10:31 PM, Tim Dunphy <bluethu...@gmail.com> wrote:
> The most important things to note:
>> - don't include JNA (it needs to lock pages larger than what will be
>> available)
>> - turn down threadpools for transports
>> - turn compaction throughput way down
>> - make concurrent reads and writes very small
>> I have used the above run a healthy 5 node clusters locally in it's own
>> private network with a 6th monitoring server for light to moderate local
>> testing in 16g of laptop ram. YMMV but it is possible.
>
>
> Thanks!! That was very helpful. I just tried applying your suggestions to
> my cassandra.yaml file. I used the info from your chef recipe. Well like
> I've been saying typically it takes about 5 hours or so for this situation
> to shake itself out. I'll provide an update to the list once I have a
> better idea of how this is working.
>
> Thanks again!
> Tim
>
> On Fri, Feb 20, 2015 at 9:37 PM, Nate McCall <n...@thelastpickle.com>
> wrote:
>
>> I frequently test with multi-node vagrant-based clusters locally. The
>> following chef attributes should give you an idea of what to turn down in
>> cassandra.yaml and cassandra-env.sh to build a decent testing cluster:
>>
>> :cassandra => {'cluster_name' => 'VerifyCluster',
>> 'package_name' => 'dsc20',
>> 'version' => '2.0.11',
>> 'release' => '1',
>> 'setup_jna' => false,
>> 'max_heap_size' => '512M',
>> 'heap_new_size' => '100M',
>> 'initial_token' => server['initial_token'],
>> 'seeds' => "192.168.33.10",
>> 'listen_address' => server['ip'],
>> 'broadcast_address' => server['ip'],
>> 'rpc_address' => server['ip'],
>> 'conconcurrent_reads' => "2",
>> 'concurrent_writes' => "2",
>> 'memtable_flush_queue_size' => "2",
>> 'compaction_throughput_mb_per_sec' => "8",
>> 'key_cache_size_in_mb' => "4",
>> 'key_cache_save_period' => "0",
>> 'native_transport_min_threads' => "2",
>> 'native_transport_max_threads' => "4",
>> 'notify_restart' => true,
>> 'reporter' => {
>> 'riemann' => {
>> 'enable' => true,
>> 'host' => '192.168.33.51'
>> },
>> 'graphite' => {
>> 'enable' => true,
>> 'host' => '192.168.33.51'
>> }
>> }
>> },
>>
>> The most important things to note:
>> - don't include JNA (it needs to lock pages larger than what will be
>> available)
>> - turn down threadpools for transports
>> - turn compaction throughput way down
>> - make concurrent reads and writes very small
>>
>> I have used the above run a healthy 5 node clusters locally in it's own
>> private network with a 6th monitoring server for light to moderate local
>> testing in 16g of laptop ram. YMMV but it is possible.
>>
>
>
>
> --
> GPG me!!
>
> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>
>
--
GPG me!!
gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
