Cassandra 1.0 - is disk seek required to access SSTable metadata

[Maciej Miklas]

Hi all,

older Cassandra versions had to read columns from each SSTable with
positive bloom filter in order to find recent value.
This was optimized with: Improve read performance in update-intensive
workload 
Now each SSTable has metadata - SSTableMetadata.

Bloom filter is stored in RAM, but what about metadata?
Is disk seek required to access it?

Thanks,
Maciej

Re: Unreachable node, not in nodetool ring

[Alain RODRIGUEZ]

Hi,

I finally successfully removed the "ghost" node using
unsafeAssassinateEndpoint() as described there :
http://tumblr.doki-pen.org/post/22654515359/assinating-cassandra-nodes,
I hope this can help more people.

Nodetool gossipinfo gives me now the following info for the ghost node :

/10.56.62.211
  RELEASE_VERSION:1.1.2
  RPC_ADDRESS:0.0.0.0
  REMOVAL_COORDINATOR:REMOVER,85070591730234615865843651857942052864
  SCHEMA:59adb24e-f3cd-3e02-97f0-5b395827453f
  STATUS:LEFT,42529904547457370790386101505459979624,1344611213445
  LOAD:11594.0
  DC:eu-west
  RACK:1b

Instead of :

/10.56.62.211
  RELEASE_VERSION:1.1.2
  LOAD:11594.0
  RACK:1b
  SCHEMA:59adb24e-f3cd-3e02-97f0-5b395827453f
  DC:eu-west
  REMOVAL_COORDINATOR:REMOVER,85070591730234615865843651857942052864
  STATUS:removed,170141183460469231731687303715884105727,1342453967415
  RPC_ADDRESS:0.0.0.0

Cassandra-cli describe cluster now don't show me any unreachable node.

The only issue that remains is that my nodes aren't well load balanced
yet... After repairing, cleaning up, restarting all nodes I still have
the following ring :

Address DC  RackStatus State   Load
OwnsToken

85070591730234615865843651857942052864
10.59.21.241eu-west 1b  Up Normal  103.19 GB
50.00%  0
10.58.83.109eu-west 1b  Up Normal  62.62 GB
50.00%  85070591730234615865843651857942052864

Any idea on why I can't get the load well balanced in this cluster ?

Alain

Re: Use of SSD for commitlog

[Hiller, Dean]

Probably not since it is sequential writes….(ie. Seek performance is the big 
hit and if it is sequential it should not be seeking and is about just as fast 
as an SSD in theory).  In practice, I have not measure the performance of one 
vs. the other though…that I always the best way to go.(you could write a micro 
benchmark test with warmup writes and then stream writes to and see how it does 
without cassandra).

Dean

From: Darvin Denmian mailto:darvin.denm...@gmail.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Tuesday, August 7, 2012 8:34 PM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Use of SSD for commitlog

Hi,

Can somebody tell me if is there some benefit in use SSD Disks
for commitlog?

THanks!

lost keyspace

[Arend-Jan Wijtzes]

Hi,

Today we rebooted a node in our cluster for maintenance and after that
one of the keyspaces went missing. This is what we did leading up to
this:

nodetool drain -h localhost
reboot

restart cassandra
There was an error during startup:
ERROR [MutationStage:109] 2012-08-08 14:05:56,443 RowMutationVerbHandler.java 
(line 61) Error in row mutation 
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1014

(then we found keyspace twitter is missing)

The sstables are still there so I tried to re-create the keyspace:
cassandra-cli -h localhost
[default@unknown] create keyspace twitter with placement_strategy = 
'org.apache.cassandra.locator.SimpleStrategy' and strategy_options = 
{replication_factor:3} ; 
b890b65a-40a6-3fc5-838d-34d58368cb64
Waiting for schema agreement...
... schemas agree across the cluster
NotFoundException()

default@unknown] use twitter;
Keyspace 'twitter' not found.

So we can't create the keyspace. The cassandra version is 1.1.0.
What is the proper way to deal with this?

Thanks for your help.

Arend-Jan
-- 
Arend-Jan Wijtzes -- Wiseguys -- www.wise-guys.nl

Re: Use of SSD for commitlog

[Darvin Denmian]

Thanks for your reply Dean,

considering your reply maybe I use a 15k RPM SCSI Disk, I think it'll
perform better than a SSD disk.

On Wed, Aug 8, 2012 at 10:01 AM, Hiller, Dean  wrote:

> Probably not since it is sequential writes….(ie. Seek performance is the
> big hit and if it is sequential it should not be seeking and is about just
> as fast as an SSD in theory).  In practice, I have not measure the
> performance of one vs. the other though…that I always the best way to
> go.(you could write a micro benchmark test with warmup writes and then
> stream writes to and see how it does without cassandra).
>
> Dean
>
> From: Darvin Denmian  darvin.denm...@gmail.com>>
> Reply-To: "user@cassandra.apache.org" <
> user@cassandra.apache.org>
> Date: Tuesday, August 7, 2012 8:34 PM
> To: "user@cassandra.apache.org" <
> user@cassandra.apache.org>
> Subject: Use of SSD for commitlog
>
> Hi,
>
> Can somebody tell me if is there some benefit in use SSD Disks
> for commitlog?
>
> THanks!
>

Re: Use of SSD for commitlog

[Hiller, Dean]

A 7.5 is probably fine and can still beat it as it is going to be the speed of 
writing not seeking(and I am not sure if they spec hard drives with a write 
time when not seeking….not sure).  Remember that drives are rated on how fast 
they spin…this disk should not be spinning a lot(in theory)…it is always 
writing.

Even on a read, it would do one seek and then do a very long sequential 
read(and I think that only happens on startup anyways).

Later,
Dean

From: Darvin Denmian mailto:darvin.denm...@gmail.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Wednesday, August 8, 2012 7:16 AM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Re: Use of SSD for commitlog

Thanks for your reply Dean,

considering your reply maybe I use a 15k RPM SCSI Disk, I think it'll 
perform better than a SSD disk.

On Wed, Aug 8, 2012 at 10:01 AM, Hiller, Dean 
mailto:dean.hil...@nrel.gov>> wrote:
Probably not since it is sequential writes….(ie. Seek performance is the big 
hit and if it is sequential it should not be seeking and is about just as fast 
as an SSD in theory).  In practice, I have not measure the performance of one 
vs. the other though…that I always the best way to go.(you could write a micro 
benchmark test with warmup writes and then stream writes to and see how it does 
without cassandra).

Dean

From: Darvin Denmian 
mailto:darvin.denm...@gmail.com>>>
Reply-To: 
"user@cassandra.apache.org>"
 
mailto:user@cassandra.apache.org>>>
Date: Tuesday, August 7, 2012 8:34 PM
To: 
"user@cassandra.apache.org>"
 
mailto:user@cassandra.apache.org>>>
Subject: Use of SSD for commitlog

Hi,

Can somebody tell me if is there some benefit in use SSD Disks
for commitlog?

THanks!

Re: Use of SSD for commitlog

[Amit Kumar]

There is a really good presentation about SSD and Cassandra on youtube by Rick 
Branson. I highly recommend watching it.

http://www.youtube.com/watch?v=zQdDi9pdf3I


Amit
On Aug 8, 2012, at 6:23 AM, Hiller, Dean wrote:

> A 7.5 is probably fine and can still beat it as it is going to be the speed 
> of writing not seeking(and I am not sure if they spec hard drives with a 
> write time when not seeking….not sure).  Remember that drives are rated on 
> how fast they spin…this disk should not be spinning a lot(in theory)…it is 
> always writing.
> 
> Even on a read, it would do one seek and then do a very long sequential 
> read(and I think that only happens on startup anyways).
> 
> Later,
> Dean
> 
> From: Darvin Denmian 
> mailto:darvin.denm...@gmail.com>>
> Reply-To: "user@cassandra.apache.org" 
> mailto:user@cassandra.apache.org>>
> Date: Wednesday, August 8, 2012 7:16 AM
> To: "user@cassandra.apache.org" 
> mailto:user@cassandra.apache.org>>
> Subject: Re: Use of SSD for commitlog
> 
> Thanks for your reply Dean,
> 
> considering your reply maybe I use a 15k RPM SCSI Disk, I think it'll 
> perform better than a SSD disk.
> 
> On Wed, Aug 8, 2012 at 10:01 AM, Hiller, Dean 
> mailto:dean.hil...@nrel.gov>> wrote:
> Probably not since it is sequential writes….(ie. Seek performance is the big 
> hit and if it is sequential it should not be seeking and is about just as 
> fast as an SSD in theory).  In practice, I have not measure the performance 
> of one vs. the other though…that I always the best way to go.(you could write 
> a micro benchmark test with warmup writes and then stream writes to and see 
> how it does without cassandra).
> 
> Dean
> 
> From: Darvin Denmian 
> mailto:darvin.denm...@gmail.com>>>
> Reply-To: 
> "user@cassandra.apache.org>"
>  
> mailto:user@cassandra.apache.org>>>
> Date: Tuesday, August 7, 2012 8:34 PM
> To: 
> "user@cassandra.apache.org>"
>  
> mailto:user@cassandra.apache.org>>>
> Subject: Use of SSD for commitlog
> 
> Hi,
> 
> Can somebody tell me if is there some benefit in use SSD Disks
> for commitlog?
> 
> THanks!
> 

Re: lost keyspace

[Arend-Jan Wijtzes]

Forgot to mention that the keyspace 'twitter' was created, then droppend
and re-created a couple of days ago.

How about if I create a new keyspace with the same definition and then
copy the existing tables into the proper place and call nodetool
refresh on each node. Would that work or are there references in the
tables to the keyspace name?


On Wed, Aug 08, 2012 at 03:03:52PM +0200, Arend-Jan Wijtzes wrote:
> Hi,
> 
> Today we rebooted a node in our cluster for maintenance and after that
> one of the keyspaces went missing. This is what we did leading up to
> this:
> 
> nodetool drain -h localhost
> reboot
> 
> restart cassandra
> There was an error during startup:
> ERROR [MutationStage:109] 2012-08-08 14:05:56,443 RowMutationVerbHandler.java 
> (line 61) Error in row mutation 
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1014
> 
> (then we found keyspace twitter is missing)
> 
> The sstables are still there so I tried to re-create the keyspace:
> cassandra-cli -h localhost
> [default@unknown] create keyspace twitter with placement_strategy = 
> 'org.apache.cassandra.locator.SimpleStrategy' and strategy_options = 
> {replication_factor:3} ; 
> b890b65a-40a6-3fc5-838d-34d58368cb64
> Waiting for schema agreement...
> ... schemas agree across the cluster
> NotFoundException()
> 
> default@unknown] use twitter;
> Keyspace 'twitter' not found.
> 
> So we can't create the keyspace. The cassandra version is 1.1.0.
> What is the proper way to deal with this?
> 
> Thanks for your help.
> 
> Arend-Jan
> -- 
> Arend-Jan Wijtzes -- Wiseguys -- www.wise-guys.nl

-- 
Arend-Jan Wijtzes -- Wiseguys -- www.wise-guys.nl

Re: lost keyspace

[Mateusz Korniak]

On Wednesday 08 of August 2012, Arend-Jan Wijtzes wrote:
> Forgot to mention that the keyspace 'twitter' was created, then droppend
> and re-created a couple of days ago.
> 
> How about if I create a new keyspace with the same definition and then
> copy the existing tables into the proper place and call nodetool
> refresh on each node. Would that work or are there references in the
> tables to the keyspace name?

Worth to try, should work.  Perhaps use  _copy_ of SStables.
I did that during schema version desync solving few times on 1.0.x.
Make sure you have same schema version on all nodes first.

Regards,
-- 
Mateusz Korniak

Re: Use of SSD for commitlog

[Darvin Denmian]

Great !

On Wed, Aug 8, 2012 at 11:41 AM, Amit Kumar wrote:

>
> There is a really good presentation about SSD and Cassandra on youtube by
> Rick Branson. I highly recommend watching it.
>
> http://www.youtube.com/watch?v=zQdDi9pdf3I
>
>
> Amit
> On Aug 8, 2012, at 6:23 AM, Hiller, Dean wrote:
>
> > A 7.5 is probably fine and can still beat it as it is going to be the
> speed of writing not seeking(and I am not sure if they spec hard drives
> with a write time when not seeking….not sure).  Remember that drives are
> rated on how fast they spin…this disk should not be spinning a lot(in
> theory)…it is always writing.
> >
> > Even on a read, it would do one seek and then do a very long sequential
> read(and I think that only happens on startup anyways).
> >
> > Later,
> > Dean
> >
> > From: Darvin Denmian  darvin.denm...@gmail.com>>
> > Reply-To: "user@cassandra.apache.org"
> mailto:user@cassandra.apache.org>>
> > Date: Wednesday, August 8, 2012 7:16 AM
> > To: "user@cassandra.apache.org" <
> user@cassandra.apache.org>
> > Subject: Re: Use of SSD for commitlog
> >
> > Thanks for your reply Dean,
> >
> > considering your reply maybe I use a 15k RPM SCSI Disk, I think
> it'll perform better than a SSD disk.
> >
> > On Wed, Aug 8, 2012 at 10:01 AM, Hiller, Dean  > wrote:
> > Probably not since it is sequential writes….(ie. Seek performance is the
> big hit and if it is sequential it should not be seeking and is about just
> as fast as an SSD in theory).  In practice, I have not measure the
> performance of one vs. the other though…that I always the best way to
> go.(you could write a micro benchmark test with warmup writes and then
> stream writes to and see how it does without cassandra).
> >
> > Dean
> >
> > From: Darvin Denmian  darvin.denm...@gmail.com>>>
> > Reply-To: "user@cassandra.apache.org >>" <
> user@cassandra.apache.org user@cassandra.apache.org>>
> > Date: Tuesday, August 7, 2012 8:34 PM
> > To: "user@cassandra.apache.org user@cassandra.apache.org>" <
> user@cassandra.apache.org user@cassandra.apache.org>>
> > Subject: Use of SSD for commitlog
> >
> > Hi,
> >
> > Can somebody tell me if is there some benefit in use SSD Disks
> > for commitlog?
> >
> > THanks!
> >
>
>

Re: change cluster name

[Tim Dunphy]

Hello,

Thanks for your replies.

DNS seems to be properly configured:

[root@beta:~] #host beta.jokefire.com
beta.jokefire.com has address 198.101.216.47

And I forgot the step to use the nodetool flush command. Taking it from the
top again:


[default@unknown] use system;
Authenticated to keyspace: system
[default@system] set LocationInfo[utf8('L')][utf8('Test
Cluster')]=utf8('Jokefire Cluster');
Value inserted.
Elapsed time: 31 msec(s).
[default@system] exit;

Then I ran nodetool flush:

[root@beta:/etc/alternatives/cassandrahome] #./bin/nodetool -host
beta.jokefire.com flush

Then I update the cluster name in the yaml file:

# The name of the cluster. This is mainly used to prevent machines in
# one logical cluster from joining another.
cluster_name: 'Jokefire Cluster'

And if I try to start cassandra I see the following error:

ERROR 11:32:45,278 Fatal exception during initialization
org.apache.cassandra.config.ConfigurationException: Saved cluster name Test
Cluster != configured name Jokefire Cluster
at
org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:299)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:169)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:356)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:107)

Not sure what else I might be doing wrong here, but it would be great to
hear what someone else thinks of this.

Thanks!
Tim




On Wed, Aug 8, 2012 at 12:59 AM, Abhijit Chanda
wrote:

> have you flushed the node using NODETOOL after renaming the cluster?
>



-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B

Re: lost keyspace

[Arend-Jan Wijtzes]

On Wed, Aug 08, 2012 at 05:08:56PM +0200, Mateusz Korniak wrote:
> On Wednesday 08 of August 2012, Arend-Jan Wijtzes wrote:
> > Forgot to mention that the keyspace 'twitter' was created, then droppend
> > and re-created a couple of days ago.
> > 
> > How about if I create a new keyspace with the same definition and then
> > copy the existing tables into the proper place and call nodetool
> > refresh on each node. Would that work or are there references in the
> > tables to the keyspace name?
> 
> Worth to try, should work.  Perhaps use  _copy_ of SStables.
> I did that during schema version desync solving few times on 1.0.x.
> Make sure you have same schema version on all nodes first.

OK that worked. One extra step was required and that was to rename the 
keyspace part in the filenames of the sstables.

Still would like to know why the original keyspace got lost and if there
is a 'proper' way to restore it.

-- 
Arend-Jan Wijtzes -- Wiseguys -- www.wise-guys.nl

Re: change cluster name

[Abhijit Chanda]

If you have used the above commands then i guess problem lies in this step.
[default@system] set LocationInfo[utf8('L')][utf8('*Test
Cluster*')]=utf8('Jokefire
Cluster');

It should be like this:  set LocationInfo[utf8('L')][utf8('Cluster
Name')]=utf8('Jokefire Cluster');

Re: change cluster name

[Tim Dunphy]

whoops! sorry about that! trying again..

[default@unknown] connect beta.jokefire.com/9160;
Connected to: "Test Cluster" on beta.jokefire.com/9160

[default@unknown] use system;
Authenticated to keyspace: system
[default@system] set LocationInfo[utf8('L')][utf8('Cluster
Name')]=utf8('Jokefire Cluster'
);
Value inserted.
Elapsed time: 38 msec(s).

[default@system] exit;
[root@beta:~] #nodetool -h beta.jokefire.com flush


[root@beta:~] #grep cluster_name
/etc/alternatives/cassandrahome/conf/cassandra.yaml
cluster_name: 'Jokefire Cluster'

And I still get the same result:

ERROR 12:02:24,023 Fatal exception during initialization
org.apache.cassandra.config.ConfigurationException: Saved cluster name Test
Cluster != configured name Jokefire Cluster
at
org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:299)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:169)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:356)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:107)


And just making sure this is clear, DNS appears to be fine:

[root@beta:~] #grep beta /etc/hosts
198.101.216.47  beta.jokefire.com beta

[root@beta:~] #host beta.jokefire.com
beta.jokefire.com has address 198.101.216.47

thanks I appreciate the advice you've been giving me... hope I can get this
to work!

On Wed, Aug 8, 2012 at 11:49 AM, Abhijit Chanda
wrote:

> If you have used the above commands then i guess problem lies in this step.
>
> [default@system] set LocationInfo[utf8('L')][utf8('*Test 
> Cluster*')]=utf8('Jokefire
> Cluster');
>
> It should be like this:  set LocationInfo[utf8('L')][utf8('Cluster
> Name')]=utf8('Jokefire Cluster');
>
>
>


-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B

Re: Use of SSD for commitlog

[Terje Marthinussen]

Probably you can get an intel 320 160GB or a Samsung 830 for the same price as 
the 146GB 15k rpm drive.

Overprovision the SSD 20% and off you go.

It will beat the HDD both sequentially and randomly.

Terje

On Aug 8, 2012, at 11:41 PM, Amit Kumar  wrote:

> 
> There is a really good presentation about SSD and Cassandra on youtube by 
> Rick Branson. I highly recommend watching it.
> 
> http://www.youtube.com/watch?v=zQdDi9pdf3I
> 
> 
> Amit
> On Aug 8, 2012, at 6:23 AM, Hiller, Dean wrote:
> 
>> A 7.5 is probably fine and can still beat it as it is going to be the speed 
>> of writing not seeking(and I am not sure if they spec hard drives with a 
>> write time when not seeking….not sure).  Remember that drives are rated on 
>> how fast they spin…this disk should not be spinning a lot(in theory)…it is 
>> always writing.
>> 
>> Even on a read, it would do one seek and then do a very long sequential 
>> read(and I think that only happens on startup anyways).
>> 
>> Later,
>> Dean
>> 
>> From: Darvin Denmian 
>> mailto:darvin.denm...@gmail.com>>
>> Reply-To: "user@cassandra.apache.org" 
>> mailto:user@cassandra.apache.org>>
>> Date: Wednesday, August 8, 2012 7:16 AM
>> To: "user@cassandra.apache.org" 
>> mailto:user@cassandra.apache.org>>
>> Subject: Re: Use of SSD for commitlog
>> 
>> Thanks for your reply Dean,
>> 
>> considering your reply maybe I use a 15k RPM SCSI Disk, I think it'll 
>> perform better than a SSD disk.
>> 
>> On Wed, Aug 8, 2012 at 10:01 AM, Hiller, Dean 
>> mailto:dean.hil...@nrel.gov>> wrote:
>> Probably not since it is sequential writes….(ie. Seek performance is the big 
>> hit and if it is sequential it should not be seeking and is about just as 
>> fast as an SSD in theory).  In practice, I have not measure the performance 
>> of one vs. the other though…that I always the best way to go.(you could 
>> write a micro benchmark test with warmup writes and then stream writes to 
>> and see how it does without cassandra).
>> 
>> Dean
>> 
>> From: Darvin Denmian 
>> mailto:darvin.denm...@gmail.com>>>
>> Reply-To: 
>> "user@cassandra.apache.org>"
>>  
>> mailto:user@cassandra.apache.org>>>
>> Date: Tuesday, August 7, 2012 8:34 PM
>> To: 
>> "user@cassandra.apache.org>"
>>  
>> mailto:user@cassandra.apache.org>>>
>> Subject: Use of SSD for commitlog
>> 
>> Hi,
>> 
>> Can somebody tell me if is there some benefit in use SSD Disks
>> for commitlog?
>> 
>> THanks!
>> 
> 

Re: change cluster name

[Abhijit Chanda]

Hey Tim,

Can you  repeat these steps as given below? I guess now  you are missing
step no 6.

Start the Cassandra-cli connected  to your node.
Run the following:

   1. use system;
   2. set LocationInfo[utf8('L')][utf8('ClusterName')]=utf8('Brisk
   Cluster');
   3. exit;
   4. Run nodetool flush on this node.
   5. Update the cassandra.yaml file for the cluster_name
   6. Restart the node.

Check whether everything is fine or not. If not then delete the files in
your system keyspace folder except files like Schema*.*. Every time you
start cassandra node, it will compare the value between the one in system
keyspace and the one in you configuration file, if they're not equal,
you'll get an error. And please don't forget to keep the backup before
deletion if anything goes wrong.

Hope this will help you

Abhijit

Re: change cluster name

[Tim Dunphy]

Hi Abhijit,

Thanks and actually I did restart cassandra at the end. I suspect I will
have to delete the files in my system keyspace folder. I'm not at liberty
to do that at the moment, but I will give that shot as soon as I'm able.

Thank you again
Tim

On Wed, Aug 8, 2012 at 12:24 PM, Abhijit Chanda
wrote:

> Hey Tim,
>
> Can you  repeat these steps as given below? I guess now  you are missing
> step no 6.
>
> Start the Cassandra-cli connected  to your node.
> Run the following:
>
>1. use system;
>2. set LocationInfo[utf8('L')][utf8('ClusterName')]=utf8('Brisk
>Cluster');
>3. exit;
>4. Run nodetool flush on this node.
>5. Update the cassandra.yaml file for the cluster_name
>6. Restart the node.
>
> Check whether everything is fine or not. If not then delete the files in
> your system keyspace folder except files like Schema*.*. Every time you
> start cassandra node, it will compare the value between the one in system
> keyspace and the one in you configuration file, if they're not equal,
> you'll get an error. And please don't forget to keep the backup before
> deletion if anything goes wrong.
>
> Hope this will help you
>
> Abhijit
>



-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B

Re: How to purge the Hinted data from Cassandra 1.0.11

[Roshan]

I managed to delete the hints from JConsole by using HintedHadOffManager
MBean.

Thanks.



--
View this message in context: 
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/How-to-purge-the-Hinted-data-from-Cassandra-1-0-11-tp7581614p7581641.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
Nabble.com.

Syncing nodes + Cassandra Data Availability

[Ben Kaehne]

Good morning,

Our application runs on a 3 node cassandra cluster with RF of 3.

We use quorum operations against this cluster in hopes of garunteeing
consistency.

One scenario in which an issue can occur here is:
Out of our 3 nodes, only 2 are up.
We perform a write to say, a new key.
The down node is started again, at the same time, a different node is
brought offline.
At this point. The data we have written above is on one node, but not the
other online node. Meaning quorum reads will fail.

Surely other people have encountered such issue before.

We disabled hinted handoffs originally as to not have to worry about race
conditions of disk space on servers filling up due to piling up handoffs.
Although perhaps this may somewhat aid the situation (although from what I
read, it does not completely remedy the circumstance).

If so, how are you dealing with it?
>From what I understand a read repair (in which we have set to 1.0) will
only be performed on a successful read occurs, in which will not happen
here.

nodetool repair seems rather slow, is manual and does not suit our
situation where data has to be available apon demand.

Regards,

-- 
-Ben

Re: change cluster name

[rajesh.ba...@orkash.com]

Hi Tim,
I had a similar problem and I sorted it out in the following manner.
The cluster name is stored in two places. In he cassandra.yaml and the 
system keyspace. Everytime you start a node it checks the cluster name 
in both these places. If changes in the configuration files is not 
working then i would suggest you delete the files in your system 
keyspace folder except files like Schema*.*.

Please backup the system keyspace before deleting.

Regards
Rajesh Kumar

On Wednesday 08 August 2012 09:04 PM, Tim Dunphy wrote:

Hello,

Thanks for your replies.

DNS seems to be properly configured:

[root@beta:~] #host beta.jokefire.com 
beta.jokefire.com  has address 198.101.216.47

And I forgot the step to use the nodetool flush command. Taking it 
from the top again:



[default@unknown] use system;
Authenticated to keyspace: system
[default@system] set LocationInfo[utf8('L')][utf8('Test 
Cluster')]=utf8('Jokefire Cluster');

Value inserted.
Elapsed time: 31 msec(s).
[default@system] exit;

Then I ran nodetool flush:

[root@beta:/etc/alternatives/cassandrahome] #./bin/nodetool -host 
beta.jokefire.com  flush


Then I update the cluster name in the yaml file:

# The name of the cluster. This is mainly used to prevent machines in
# one logical cluster from joining another.
cluster_name: 'Jokefire Cluster'

And if I try to start cassandra I see the following error:

ERROR 11:32:45,278 Fatal exception during initialization
org.apache.cassandra.config.ConfigurationException: Saved cluster name 
Test Cluster != configured name Jokefire Cluster
at 
org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:299)
at 
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:169)
at 
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:356)
at 
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:107)


Not sure what else I might be doing wrong here, but it would be great 
to hear what someone else thinks of this.


Thanks!
Tim




On Wed, Aug 8, 2012 at 12:59 AM, Abhijit Chanda 
mailto:abhijit.chan...@gmail.com>> wrote:


have you flushed the node using NODETOOL after renaming the cluster?




--
GPG me!!

gpg --keyserver pool.sks-keyservers.net 
 --recv-keys F186197B