*I took that opportunity to upgrade from 3.1.1 to 3.0.9* If my guess is right and you meant that you upgraded from 2.1.1 to 3.0.9 directly, then this might cause some issues (not necessarily the issue at hand though). The proper upgrade process should be to 2.1.9 and from there upgrade to 3.0.x.
https://docs.datastax.com/en/upgrade/doc/upgrade/cassandra/upgrdCassandra.html Hope this helps. Shalom Sagges DBA T: +972-74-700-4035 <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson> <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections <https://engage.liveperson.com/idc-mobile-first-consumer/?utm_medium=email&utm_source=mkto&utm_campaign=idcsig> On Tue, Nov 22, 2016 at 2:44 AM, Bertrand Brelier < bertrand.brel...@gmail.com> wrote: > Hello Shalom, Vladimir, > > Thanks for your help. > > I had initially 3 nodes, had a hardware failure and reinstalled Cassandra > on the node (I took that opportunity to upgrade from 3.1.1 to 3.0.9). I ran > nodetool upgradesstables and nodetool repair on each node once I updated > Cassandra. > > The 3 nodes are in the same private network, I am using the private IPs > for the seeds and the listen_address and the public IPs for rpc_address > > I am using ssl to encrypt the communication between the nodes, so I am > using the port 7001 : > > telnet PRIVATEIP 7001 > Trying PRIVATEIP... > Connected to PRIVATEIP. > > Each node can connect with any other node. > > I selected some old data from the new node : > > CONSISTENCY; > Current consistency level is ONE. > select count(*) from ....; > > count > ------- > 0 > > CONSISTENCY ALL; > Consistency level set to ALL. > count(*) from ....; > > count > ------- > 64 > > When I switched to ALL I could get the data while the initial level ONE > did not have any data. I did not expect to get any data with ALL, am I > missing something ? > > I do not know if this is related, but while I was inquiring the database, > I had the following messages in the debug.log : > > DEBUG [ReadRepairStage:15292] 2016-11-21 18:15:59,719 > ReadCallback.java:234 - Digest mismatch: > org.apache.cassandra.service.DigestMismatchException: Mismatch for key > DecoratedKey(2288259866140251828, 00040000002a0000040000421500) ( > d41d8cd98f00b204e9800998ecf8427e vs ce211ac5533e1a146d9fee734fd8de26) > at > org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at org.apache.cassandra.service.ReadCallback$ > AsyncRepairRunner.run(ReadCallback.java:225) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_111] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_111] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111] > > > Thanks for your help, > > Cheers, > > Bertrand > > > On 16-11-21 01:28 AM, Shalom Sagges wrote: > > I believe the logs should show you what the issue is. > Also, can the node "talk" with the others? (i.e. telnet to the other nodes > on port 7000). > > > Shalom Sagges > DBA > T: +972-74-700-4035 > <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson> > <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections > > <https://engage.liveperson.com/idc-mobile-first-consumer/?utm_medium=email&utm_source=mkto&utm_campaign=idcsig> > > > On Sun, Nov 20, 2016 at 8:50 PM, Bertrand Brelier < > bertrand.brel...@gmail.com> wrote: > >> Hello Jonathan, >> >> No, the new node is not a seed in my cluster. >> >> When I ran nodetool bootstrap resume >> Node is already bootstrapped. >> >> Cheers, >> >> Bertrand >> >> On Sun, Nov 20, 2016 at 1:43 PM, Jonathan Haddad <j...@jonhaddad.com> >> wrote: >> >>> Did you add the new node as a seed? If you did, it wouldn't bootstrap, >>> and you should run repair. >>> On Sun, Nov 20, 2016 at 10:36 AM Bertrand Brelier < >>> bertrand.brel...@gmail.com> wrote: >>> >>>> Hello everybody, >>>> >>>> I am using a 3-node Cassandra cluster with Cassandra 3.0.10. >>>> >>>> I recently added a new node (to make it a 3-node cluster). >>>> >>>> I am using a replication factor of 3 , so I expected to have a copy of >>>> the same data on each node : >>>> >>>> CREATE KEYSPACE mydata WITH replication = {'class': 'SimpleStrategy', >>>> 'replication_factor': '3'} AND durable_writes = true; >>>> >>>> But the new node has less data that the other 2 : >>>> >>>> Datacenter: datacenter1 >>>> ======================= >>>> Status=Up/Down >>>> |/ State=Normal/Leaving/Joining/Moving >>>> -- Address Load Tokens Owns (effective) Host >>>> ID Rack >>>> UN XXX.XXX.XXX.XXX 53.28 GB 256 100.0% xxxxxx rack1 >>>> UN XXX.XXX.XXX.XXX 64.7 GB 256 100.0% xxxxxx rack1 >>>> UN XXX.XXX.XXX.XXX 1.28 GB 256 100.0% xxxxxx rack1 >>>> >>>> >>>> On the new node : >>>> >>>> /XXXXXX/data-6d674a40efab11e5b67e6d75503d5d02/: >>>> total 1.2G >>>> >>>> on one of the old nodes : >>>> >>>> /XXXXXX/data-6d674a40efab11e5b67e6d75503d5d02/: >>>> total 52G >>>> >>>> >>>> I am monitoring the amount of data on each node, and they grow at the >>>> same rate. So I suspect that my new data are replicated on the 3 nodes >>>> but the old data stored on the first 2 nodes are not replicated on the >>>> new node. >>>> >>>> I ran nodetool repair (on each node, one at a time), but the new node >>>> still does not have a copy of the old data. >>>> >>>> Could you please help me understand why the old data is not replicated >>>> to the new node ? Please let me know if you need further information. >>>> >>>> Thank you, >>>> >>>> Cheers, >>>> >>>> Bertrand >>>> >>>> >> > > This message may contain confidential and/or privileged information. > If you are not the addressee or authorized to receive this on behalf of > the addressee you must not use, copy, disclose or take action based on this > message or any information herein. > If you have received this message in error, please advise the sender > immediately by reply email and delete this message. Thank you. > > > -- This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this on behalf of the addressee you must not use, copy, disclose or take action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply email and delete this message. Thank you.
