Glad to hear that.
About the Proxy
Putting an HA proxy in front of Cassandra is an anti-pattern as you create
a single point of failure. You can multiply them, but why would you do that
anyway when most (all ?) of the modern Cassandra clients handle this for
you through TokenAware / RoundRobin / LatencyAware strategies ?
About the tokens:
You can use vnodes (set "num_tokens" to 64 or 128 let's say, comment
"#initial_token") and let Cassandra handle node/tokens distribution
combined to a Murmur3Partitioner and you're all set. Not sure what you're
doing with IPs, still don't get it but it is either useless or even bad
from what I understand.
Finally (as you find out) you can't auto_bootstrap a seed node indeed.
Hope this help.
C*heers,
Alain
2015-09-07 18:11 GMT+02:00 ICHIBA Sara <ichi.s...@gmail.com>:
> I think I know where my problem is coming from. I took a look at the log
> of cassandra on each node and I saw something related to bootstrap. it says
> that the node is a seed so there will be no bootstraping. Actually I made a
> mistake. in the cassandra.yaml file each node have two ips as seeds. the ip
> of the machine itself and the ip of the real seed server. Once i removed
> the local IP the problem seems to be fixed.
>
> 2015-09-07 18:01 GMT+02:00 ICHIBA Sara <ichi.s...@gmail.com>:
>
>> Thank you all for your answers.
>>
>> @Alain:
>> Can you detail actions performed,
>> >>like how you load data
>> >>>i have a haproxy in front of my cassandra database, so i'm sure that
>> my application queries each time a different coordinator
>>
>> >>what scaleup / scaledown are and precise if you let it decommission
>> fully (streams finished, node removed from nodetool status)
>> >>> i'm using openstack platform to autoscale cassandra cluster.
>> Actually, in openstack, the combination of ceilometer + heat allow to users
>> to automate the deployment of their applications and supervise their
>> resources. they can order the scale up (adding of new nodes automatically)
>> when resources(cpu, ram,...) are needed or scaledown (remove unecessary VMs
>> automatically).
>> so with heat i can spawn automatically a cluster of 2 cassandra VMs
>> (create the cluster and configure each cassandra server with a template).
>> My cluster can go from 2 nodes to 6 based on the workloads. when their is a
>> scaledown action, heat automatically execute a script on my node and
>> decommission it before removing it.
>>
>> >>Also, I am not sure of the meaning of this --> " i'm affecting to each
>> of my node a different token based on there ip address (the token is
>> A+B+C+D and the ip is A.B.C.D)".
>>
>> look at this:
>> [root@demo-server-seed-wgevseugyjd7 ~]# nodetool status bng;
>> Datacenter: DC1
>> ===============
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host
>> ID Rack
>> UN 40.0.0.149 789.03 KB 189 100.0%
>> bd0b2616-18d9-4bc2-a80b-eebd67474712 RAC1
>> UN 40.0.0.168 300.38 KB 208 100.0%
>> ebd9732b-ebfc-4a6c-b354-d7df860b57b0 RAC1
>>
>> the node with address 40.0.0.149 have the token 189=40+0+0+149
>> and the node with address 40.0.0.168 have the token 208=40+0+0+168
>>
>> this way i'm sure that each node in my cluster will have a different
>> token. I don't know what will happen if all the node have the same token??
>>
>> >>Aren't you using RandomPartitioner or Murmur3Partitioner
>>
>> i'm using the default one which is
>> partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>>
>>
>> in order to configure cassandra on each node i'm using this script
>>
>> inputs:
>> - name: IP
>> - name: SEED
>> config: |
>> #!/bin/bash -v
>> cat << EOF >> /etc/resolv.conf
>> nameserver 8.8.8.8
>> nameserver 192.168.5.1
>> EOF
>>
>> DEFAULT=${DEFAULT:-/etc/cassandra/default.conf}
>> CONFIG=/etc/cassandra/conf
>> IFS="." read a b c d <<< $IP
>> s="$[a[0]+b[0]+c[0]+d[0]]"
>> sed -i -e "s/^cluster_name.*/cluster_name: 'Cassandra cluster for
>> freeradius'/" $CONFIG/cassandra.yaml
>> sed -i -e "s/^num_tokens.*/num_tokens: $s/" $CONFIG/cassandra.yaml
>> sed -i -e "s/^listen_address.*/listen_address: $IP/"
>> $CONFIG/cassandra.yaml
>> sed -i -e "s/^rpc_address.*/rpc_address: 0.0.0.0/"
>> $CONFIG/cassandra.yaml
>> sed -i -e "s/^broadcast_address.*/broadcast_address: $IP/"
>> $CONFIG/cassandra.yaml
>> sed -i -e "s/broadcast_rpc_address.*/broadcast_rpc_address: $IP/"
>> $CONFIG/cassandra.yaml
>> sed -i -e
>> "s/^commitlog_segment_size_in_mb.*/commitlog_segment_size_in_mb: 32/"
>> $CONFIG/cassandra.yaml
>> sed -i -e "s/# JVM_OPTS=\"\$JVM_OPTS
>> -Djava.rmi.server.hostname=<public name>\"/JVM_OPTS=\"\$JVM_OPTS
>> -Djava.rmi.server.hostname=$IP\"/" $CONFIG/cassandra-env.sh
>> sed -i -e "s/- seeds.*/- seeds: \"$SEED\"/" $CONFIG/cassandra.yaml
>>
>> sed -i -e "s/^endpoint_snitch.*/endpoint_snitch:
>> GossipingPropertyFileSnitch/" $CONFIG/cassandra.yaml
>> echo MAX_HEAP_SIZE="4G" >> $CONFIG/cassandra-env.sh
>> echo HEAP_NEWSIZE="800M" >> $CONFIG/cassandra-env.sh
>> service cassandra stop
>> rm -rf /var/lib/cassandra/data/system/*
>> service cassandra start
>>
>>
>>
>> 2015-09-07 16:30 GMT+02:00 Ryan Svihla <r...@foundev.pro>:
>>
>>> If that's what tracing is telling you then it's fine and just a product
>>> of data distribution (note your token count isn't identical anyway).
>>>
>>> If you're doing cl one queries directly against particular nodes and
>>> getting different results it sounds like dropped mutations, streaming
>>> errors and or timeouts. Does running repair or reading at CL level all give
>>> you an accurate total record count?
>>>
>>> nodetool tpstats should help post bootstrap identify dropped mutations
>>> but you also want to monitor logs for any errors (in general this is always
>>> good advice for any system).. There could be a myriad or problems with
>>> bootstrapping new nodes, usually this is related to under provisioning.
>>>
>>> On Mon, Sep 7, 2015 at 8:19 AM Alain RODRIGUEZ <arodr...@gmail.com>
>>> wrote:
>>>
>>>> Hi Sara,
>>>>
>>>> Can you detail actions performed, like how you load data, what scaleup
>>>> / scaledown are and precise if you let it decommission fully (streams
>>>> finished, node removed from nodetool status) etc ?
>>>>
>>>> This would help us to help you :).
>>>>
>>>> Also, what happens if you query using "CONSISTENCY LOCAL_QUORUM;" (or
>>>> ALL) before your select ? If not using cqlsh, set the Consistency Level of
>>>> your client to LOCAL_QUORUM or ALL and try to select again.
>>>>
>>>> Also, I am not sure of the meaning of this --> " i'm affecting to each
>>>> of my node a different token based on there ip address (the token is
>>>> A+B+C+D and the ip is A.B.C.D)". Aren't you using RandomPartitioner or
>>>> Murmur3Partitioner ?
>>>>
>>>> C*heers,
>>>>
>>>> Alain
>>>>
>>>>
>>>>
>>>> 2015-09-07 12:01 GMT+02:00 Edouard COLE <edouard.c...@rgsystem.com>:
>>>>
>>>>> Please, don't mail me directly
>>>>>
>>>>> I read your answer, but I cannot help anymore
>>>>>
>>>>> And answering with "Sorry, I can't help" is pointless :)
>>>>>
>>>>> Wait for the community to answer
>>>>>
>>>>> De : ICHIBA Sara [mailto:ichi.s...@gmail.com]
>>>>> Envoyé : Monday, September 07, 2015 11:34 AM
>>>>> À : user@cassandra.apache.org
>>>>> Objet : Re: cassandra scalability
>>>>>
>>>>> when there's a scaledown action, i make sure to decommission the node
>>>>> before. but still, I don't understand why I'm having this behaviour. is it
>>>>> normal. what do you do normally to remove a node? is it related to tokens?
>>>>> i'm affecting to each of my node a different token based on there ip
>>>>> address (the token is A+B+C+D and the ip is A.B.C.D)
>>>>>
>>>>> 2015-09-07 11:28 GMT+02:00 ICHIBA Sara <ichi.s...@gmail.com>:
>>>>> at the biginning it looks like this:
>>>>>
>>>>> [root@demo-server-seed-k6g62qr57nok ~]# nodetool status
>>>>> Datacenter: DC1
>>>>> ===============
>>>>> Status=Up/Down
>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>> -- Address Load Tokens Owns Host
>>>>> ID Rack
>>>>> UN 40.0.0.208 128.73 KB 248 ?
>>>>> 6e7788f9-56bf-4314-a23a-3bf1642d0606 RAC1
>>>>> UN 40.0.0.209 114.59 KB 249 ?
>>>>> 84f6f0be-6633-4c36-b341-b968ff91a58f RAC1
>>>>> UN 40.0.0.205 129.53 KB 245 ?
>>>>> aa233dc2-a8ae-4c00-af74-0a119825237f RAC1
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [root@demo-server-seed-k6g62qr57nok ~]# nodetool status
>>>>> service_dictionary
>>>>> Datacenter: DC1
>>>>> ===============
>>>>> Status=Up/Down
>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>> -- Address Load Tokens Owns (effective) Host
>>>>> ID Rack
>>>>> UN 40.0.0.208 128.73 KB 248 68.8%
>>>>> 6e7788f9-56bf-4314-a23a-3bf1642d0606 RAC1
>>>>> UN 40.0.0.209 114.59 KB 249 67.8%
>>>>> 84f6f0be-6633-4c36-b341-b968ff91a58f RAC1
>>>>> UN 40.0.0.205 129.53 KB 245 63.5%
>>>>> aa233dc2-a8ae-4c00-af74-0a119825237f RAC1
>>>>>
>>>>> the result of the query select * from service_dictionary.table1; gave
>>>>> me
>>>>> 70 rows from 40.0.0.205
>>>>> 64 from 40.0.0.209
>>>>> 54 from 40.0.0.208
>>>>>
>>>>> 2015-09-07 11:13 GMT+02:00 Edouard COLE <edouard.c...@rgsystem.com>:
>>>>> Could you provide the result of :
>>>>> - nodetool status
>>>>> - nodetool status YOURKEYSPACE
>>>>>
>>>>>
>>>>>
>>>> --
>>> Regards,
>>>
>>> Ryan Svihla
>>
>>
>>
>
