If you are getting the "Cluster schema does not agree" then you have a sick
cluster and replication will not be working properly.
Open the cassandra-cli client and type "describe cluster;" you should see a
single schema version such as...
[default@unknown] describe cluster;
Cluster Information:
Snitch: org.apache.cassandra.locator.SimpleSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
e69ffb30-5525-11e0-0000-242d50cf1fbd: [127.0.0.1, 127.0.0.2]
But I think you will see multiple versions.
The easiest thing to do is drop the entire cluster and start again.
Hope that helps.
Aaron
On 24 Mar 2011, at 11:45, ko...@vivinavi.com wrote:
> Hi Aaron
>
> Thank you so much for your reply and advice.
> I watched your presentation. it"s helpful for me.
>
> Anyway I did as following with 2 node servers.(53;1st node,54;2nd node)
> I started the following write/read.php program by through Thrift at 53 server.
> <?php
> include_once...
> $servers[0]['host'] = '**.**.254.53';
> $servers[0]['port'] = '9160';
> $KeyspaceName = 'Twissandra';
> $ColumnFamilyName = 'user';
> $Key = 'LA';
> $conn = new Connection($KeyspaceName, $servers);
> $users = new ColumnFamily($conn, $ColumnFamilyName);
> $users->insert($Key, array('Movie' => 'Hollywood', 'Seaside' =>
> 'SantaMonica' , 'Museum' => 'Getty' , 'Japanese' => 'Little Tokyo' ));
> $ret=$users->get($Key);
> print_r($ret);
> ?>
> I could get $ret at 53 server.
> Then I started the following read.php program at 54 server.
> <?php
> include_once...
> $servers[0]['host'] = '**.**.254.54';
> $servers[0]['port'] = '9160';
> $KeyspaceName = 'Twissandra';
> $ColumnFamilyName = 'user';
> $Key = 'LA';
> $conn = new Connection($KeyspaceName, $servers);
> $users = new ColumnFamily($conn, $ColumnFamilyName);
> $ret=$users->get($Key);
> print_r($ret);
> ?>
> I could get $ret at 54 server.
> So I confirmed these data are replicated from 53 to 54.
> However I did same thing from 54 to 53 with only changed data(key='hahaha').
> (These are same keyspace,CF)
> I could see data on 54 server.(no problem)
> But on 53 server I had an following error message.
> Fatal error: Uncaught exception 'cassandra_TimedOutException' in
> /usr/share/php5/Thrift/thrift/packages/cassandra/Cassandra.php:253
> Stack trace:
> #0 /usr/share/php5/Thrift/thrift/packages/cassandra/Cassandra.php(253):
> thrift_protocol_read_binary(Object(TBinaryProtocolAccelerated),
> 'cassandra_Cassa...', false)
> #1 /usr/share/php5/Thrift/thrift/packages/cassandra/Cassandra.php(226):
> CassandraClient->recv_get_slice()
> #2 /usr/share/php5/Thrift/columnfamily.php(254):
> CassandraClient->get_slice('hahaha', Object(cassandra_ColumnParent),
> Object(cassandra_SlicePredicate), 1)
> #3 /var/www/thrift/PhpcassaRead.php(44): ColumnFamily->get('hahaha')
> #4 {main}
> thrown in /usr/share/php5/Thrift/thrift/packages/cassandra/Cassandra.php on
> line 253
> I checked on Cassandra-cli at 53 server.
> [default@Twissandra] get user['hahaha'];
> null
>
> So I think replication doesn't work from 54 to 53.
> I know when I add node and then set data at 1st node, this data will be
> provided to all other nodes.
> But I'm not sure when I set data at 2nd node, this data should be replicated
> from 2nd to 1st node.
> I though this will be replicated because of ring.
>
> And when I tried to create keyspace or CF several times on 54(2nd server), I
> had a following error message.
> "Cluster schema does not yet agree"
> i don't know how to schema agree.
>
> Am I wrong?
> Thank you for your advice in advance.
>
> Mamoru Mac Kondo
>
> (2011/03/23 11:32), Aaron Morton wrote:
>>
>> What process are you using to confirm the data was replicated to another
>> server ? And what makes you say the data is not replicated ? I think your
>> understanding of the replication may be a little off, you rarely read from
>> one node. Have a look at
>> http://thelastpickle.com/2011/02/07/Introduction-to-Cassandra/
>>
>> What happens when you try to create a keyspace or CF and it fails?
>>
>> It's easier to help if you provide the commands you use and any error
>> messages. Also remember to check the logs on the servers for errors.
>>
>> Aaron
>>
>> On 24/03/2011, at 5:58 AM, "ko...@vivinavi.com" <ko...@vivinavi.com> wrote:
>>
>>> Hi maki-san
>>>
>>> I am so sorry this was my mistake.
>>> I expected when I set data at one node, the data should be copied to the
>>> other node
>>> at the same keyspace and same column family by replication.
>>> This replication was working.
>>> I just made a mistake to get data by wrong Key.(First character of key was
>>> lower-case character
>>> but I thought first character was capitalized. I tried to get by key=User
>>> but key=user)
>>> I am so sorry for this confusion.
>>> So now I confirmed the data I set at **.**.254.53 was replicated to
>>> **.**.254.54.
>>> (At first I started 53 with autobootstrap=false then started 54 with
>>> autobootstrap=true and then
>>> finally restarted 53 with autobootstrap=true. So 53 server is like a master)
>>>
>>> However when I set data at 54 server, this data never been replicated.
>>> So this is Not bidirectional replication ?
>>> Cassandra at multi-node is ring (This means no master, no slave?).
>>> (Off course seeds in cassandra.yaml of both server are the same)
>>> So I expected I can replicate bidirectinally.
>>> Am I wrong?
>>> And when I run at multi-node, I can't create keyspace, column family on
>>> Cassandra-cli at other 54 server
>>> (like a slave server).
>>> Is this normal?
>>> How can I create same keyspace and column family at multi-node?
>>> or I have to restart at single-node, then can I create same keyspace and
>>> column family?
>>>
>>> Thank you for your help in advance.
>>>
>>> Mamoru Mac Kondo
>>>
>>> (2011/03/22 18:43), Maki Watanabe wrote:
>>>> How do you define your Keyspace?
>>>> As you may know, in Cassandra, replication (factor) is defined as the
>>>> attribute of Keyspace.
>>>> And what do you mean:
>>>>> However replication never happened.
>>>>> I can't get data I set at other node.
>>>> What did you do on cassandra, and what did you get in response?
>>>>
>>>> maki
>>>>
>>>>
>>>> 2011/3/23 ko...@vivinavi.com<ko...@vivinavi.com>:
>>>>> Hi Sasha
>>>>> Thank you so much for your advice.
>>>>> I changed JMX_PORT from 10036 to 8080 in cassandra-env.sh.
>>>>> Now nodetool ring is working as following.
>>>>>
>>>>> # nodetool --host **.**.254.54 ring
>>>>> Address Status State Load Owns Token
>>>>>
>>>>> 31247585259092561925693111230676487333
>>>>> **.**.254.53 Up Normal 51.3 KB 84.50%
>>>>> 4871825541058236750403047111542070004
>>>>> **.**.254.54 Up Normal 66.71 KB 15.50%
>>>>> 31247585259092561925693111230676487333
>>>>>
>>>>> Then it seems I could set data to other node by Cassandra-cli --host other
>>>>> node IP --port 9160.(Currently only 2 nodes)
>>>>> However replication never happened.
>>>>> I can't get data I set at other node.
>>>>> I don't know what's wrong.
>>>>> (I thought replication starts when cassandra -p restart)
>>>>> Please advice me how to do to start replication.
>>>>> Thank you for your advice in advance.
>>>>>
>>>>>
>>>>> (2011/03/18 23:38), Sasha Dolgy wrote:
>>>>>
>>>>> You need to specify the -jmxport with nodetool
>>>>>
>>>>> On Mar 19, 2011 2:48 AM, "ko...@vivinavi.com"<ko...@vivinavi.com> wrote:
>>>>>> Hi everyone
>>>>>>
>>>>>> I am still new to Cassandra, Thrift.
>>>>>> But anyway Cassandra 0.7.4, Thrift 0.5.0 are working on java 1.6.0.18 of
>>>>>> Debian 5.0.7.at single node.
>>>>>> Then I had to try and check multi node on 2 servers.
>>>>>> (JVM_PORT=10036 on /etc/cassandra-env.sh)
>>>>>> I modified /etc/cassandra/cassandra.yaml as following.
>>>>>> auto_bootstrap:false ->true
>>>>>> seeds: -127.0.0.1 -> add Global IP addres of 2 servers(incl.own server)
>>>>>> listen_address:localhost -> Own Global IP address(or own host name on
>>>>>> /etc/hosts)
>>>>>> rpc_address:localhost ->0.0.0.0
>>>>>> I run master server and then slave server.
>>>>>> netstat -nl is as following. on both servers.
>>>>>> Proto Recv-Q Send-Q Local Address Foreign Address State
>>>>>> tcp 0 0 0.0.0.0:9160 0.0.0.0:* LISTEN
>>>>>> tcp 0 0 0.0.0.0:10036 0.0.0.0:* LISTEN
>>>>>> tcp 0 0 **.**.**.**:7000 0.0.0.0:* LISTEN
>>>>>>
>>>>>> However it seems Cassandra doesn't work.
>>>>>> Because I can't get any data from Cluster (always null, data is broken?)
>>>>>> So I checked the nodetool (nodetool --host IP ring).
>>>>>> The nodetool had errors as following.
>>>>>> Error connection to remote JMX agent!
>>>>>> java.io.IOException: Failed to retrieve RMIServer stub:
>>>>>> javax.naming.ServiceUnavailableException [Root exception is
>>>>>> java.rmi.ConnectException: Connection refused to host: **.**.**.**;
>>>>>> nested exception is:
>>>>>> java.net.ConnectException: Connection refused]
>>>>>> at
>>>>>> javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:342)
>>>>>> at
>>>>>>
>>>>>> javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
>>>>>> at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:137)
>>>>>> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:107)
>>>>>> at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:511)
>>>>>> Caused by: javax.naming.ServiceUnavailableException [Root exception is
>>>>>> java.rmi.ConnectException: Connection refused to host: **.**.**.**;
>>>>>> nested exception is:
>>>>>> java.net.ConnectException: Connection refused]
>>>>>> at
>>>>>> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
>>>>>> at
>>>>>>
>>>>>> com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203)
>>>>>> at javax.naming.InitialContext.lookup(InitialContext.java:409)
>>>>>> at
>>>>>>
>>>>>> javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1902)
>>>>>> at
>>>>>>
>>>>>> javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1871)
>>>>>> at
>>>>>> javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:276)
>>>>>> ... 4 more
>>>>>> Caused by: java.rmi.ConnectException: Connection refused to host:
>>>>>> **.**.**.**; nested exception is:
>>>>>> java.net.ConnectException: Connection refused
>>>>>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
>>>>>> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
>>>>>> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
>>>>>> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:340)
>>>>>> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>>>>>> at
>>>>>> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:114)
>>>>>> ... 9 more
>>>>>> Caused by: java.net.ConnectException: Connection refused
>>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>>> at
>>>>>>
>>>>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:310)
>>>>>> at
>>>>>>
>>>>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:176)
>>>>>> at
>>>>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:163)
>>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
>>>>>> at java.net.Socket.connect(Socket.java:546)
>>>>>> at java.net.Socket.connect(Socket.java:495)
>>>>>> at java.net.Socket.<init>(Socket.java:392)
>>>>>> at java.net.Socket.<init>(Socket.java:206)
>>>>>> at
>>>>>>
>>>>>> sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
>>>>>> at
>>>>>>
>>>>>> sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:146)
>>>>>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
>>>>>> ... 14 more
>>>>>>
>>>>>> I don't know how to do.
>>>>>> Please advise what am I wrong.
>>>>>> And please let me know how to delete or re-build cluster.
>>>>>> Because I can't even set data or make new column family currently on
>>>>>> Cassandra-Cli.
>>>>>> Thank you for your help in advance.
>>>>>>
>>>>>> --
>>>>>>
>>>>> **
>>>
>>>
>
