Re: Seed nodes in cassandra.yaml can not be hostnames

[Peter Schuller]

> I just discovered that using host names for seed nodes in cassandra.yaml do
> not work. This is done on purpose?

I believe so yes, to avoid relying on DNS to map correctly given that
everything else is based on IP address. (IIRC, someone chime in if
there is a different reason.)

-- 
/ Peter Schuller (@scode on twitter)

Re: messages stopped for 3 minutes?

[Yang]

h. never mind, possibly the first 24 seconds delay was caused by GC,
the GC logging was not printed in system.log,
I found one line on stdout that possibly corresponds to that.


I found I left out the enable parallel remark  param, let me add that and retry.
Thanks

Yang

On Mon, Sep 26, 2011 at 12:25 AM, Yang  wrote:
> ok, I got more detailed logging that seems to give some clue out of
> this myth 
>
> I enabled TRACE level logging for Gossiper, also added a debug to
> MessageDeliveryTask so that every arrival msg shows up in logging.
>
> I have 2 nodes, all the requests come from the coordinator node,
> basically I see 2 possible issues.
>
> 1)
> in the front of bad_coord.txt, we see that the coordinator was
> receiving messages correctly until
>
> DEBUG [RequestResponseStage:22] 2011-09-26 05:43:35,521
> MessageDeliveryTask.java (line 60) delivering msg
>
>
> then the line right after it is
> DEBUG [RequestResponseStage:24] 2011-09-26 05:43:59,516
> MessageDeliveryTask.java (line 60) delivering msg
>
> so there is a 24 seconds delay , in this period, no messages were
> delivered on the coordinator. this is enough to trigger a Timeout and
> Unavailable. indeed we see time out  and UnavailableException later in
> the log.
>
>
> during this period, on the replica box, in bad_replica.txt we see
> correct message delivery up to
> DEBUG [ReadStage:107] 2011-09-26 05:43:35,214 MessageDeliveryTask.java
> (line 60) delivering msg
>
> then on the next line, it starts doing something different:
> TRACE [GossipTasks:1] 2011-09-26 05:43:35,570 Gossiper.java (line 125)
> My heartbeat is now 5721
> TRACE [GossipTasks:1] 2011-09-26 05:43:35,570 Gossiper.java (line 357)
> Gossip Digests are : /10.71.13.5:1317009996:5765
> ec2-50-19-172-99.compute-1.amazonaws.com/10.71.21.46:1317010027:5721
>
>
> I did not read the code in detail to understand what this means, but
> it seems to correspond with the "silent period" on the coord in terms
> of time. it keeps on doing this till
>  INFO [GossipTasks:1] 2011-09-26 05:43:53,638 Gossiper.java (line 721)
> InetAddress /10.71.13.5 is now dead.
>
> and
> DEBUG [MutationStage:55] 2011-09-26 05:43:59,208
> MessageDeliveryTask.java (line 60) delivering msg
>  INFO [GossipStage:1] 2011-09-26 05:43:59,211 Gossiper.java (line 707)
> InetAddress /10.71.13.5 is now UP
>
>
> = so, here the problem is that the messages stopped showing up
> from 05:43:35 to 05:43:59 on the coord, though the replica is trying
> to send them . not sure if this is due to network.
>
>
> 2) the other problem actually causes a longer delay on application side
> in the end of bad_coord.txt we see that the TimeoutException and
> AvailableException were printed out actually much later, after
> 06:06:33
> TRACE [GossipStage:1] 2011-09-26 06:06:33,107 Gossiper.java (line 802)
> /10.71.21.46local generation 1317010027, remote generation 1317010027
> TRACE [GossipStage:1] 2011-09-26 06:06:33,107 Gossiper.java (line 847)
> Updating heartbeat state version to 7119 from 7118 for /10.71.21.46
> ...
> ERROR [New I/O server worker #1-15] 2011-09-26 05:43:59,500
> MY_APPLICATION_CODE.java (line 234) :
> Caused by: TimedOutException()
>
> ** the main issue here is that the timestamp of the logging line
> itself says 05:43:59, which is correctly when the TimeOutException was
> generated**, but the line was actually not printed out till 06:06:33,
> 23minutes later. this causes a 23minutes pause visible to the clients.
>
> this 23minutes delay is much more damaging than the 24 seconds message
> drop period in 1). I wonder what could have caused this 23minutes
> delay. could be lock contention on grabbing the log4j appender ?? but
> it shouldn't be, since I only added the extra logging AFTER I saw the
> problem appear before.
>
>
> Thanks for your patience for reading thus far :)
>
>
> Yang
>
>
>
>
>
> On Sun, Sep 25, 2011 at 10:32 AM, Yang  wrote:
>> thanks Jonathan,
>>
>>
>> I really don't know, I just did further tests to catch the jstack on
>> the receiving side over the last night. going through these stacks
>> now.  if I can't find anything suspicious, I'll add these debugging to
>> the sending side too.
>>
>> another useful piece of info: when I did a single-node setup, I also
>> found a lot of TimedOutException, similar to what I see with the
>> 2-node setup. I think I didn't see the UnavailableException, probably
>> because it's just a single node, and the node always believes itself
>> to be available.
>>
>> I think GC issue is not the culprit here, since I don't see any length
>> GC logging around when the delay is happening. no compaction/flushing
>> either
>>
>>
>>
>> On Sun, Sep 25, 2011 at 6:33 AM, Jonathan Ellis  wrote:
>>> What makes you think the problem is on the receiving node, rather than
>>> the sending node?
>>>
>>> On Sun, Sep 25, 2011 at 1:19 AM, Yang  wrote:
 I constantly see TimedOutException , then followed by
 UnavailableException in my logs,
 so I added some extra debugging to Gos

[ANN] Mojo's Cassandra Maven Plugin 0.8.6-1 released

[Stephen Connolly]

The Mojo team is pleased to announce the release of Mojo's Cassandra
Maven Plugin version 0.8.6-1.

Mojo's Cassandra Plugin is used when you want to install and control a
test instance of Apache Cassandra from within your Apache Maven build.

The Cassandra Plugin has the following goals.

  * cassandra:start Starts up a test instance of Cassandra in the background.
  * cassandra:stop Stops the test instance of Cassandra that was
started using cassandra:start.
  * cassandra:start-cluster Starts up a test cluster of Cassandra in
the background bound to the local loopback IP addresses 127.0.0.1,
127.0.0.2, etc.
  * cassandra:stop Stops the test cluster of Cassandra that was
started using cassandra:start.
  * cassandra:run Starts up a test instance of Cassandra in the foreground.
  * cassandra:load Runs a cassandra-cli script against the test
instance of Cassandra.
  * cassandra:repair Runs nodetool repair against the test instance of
Cassandra.
  * cassandra:flush Runs nodetool flush against the test instance of Cassandra.
  * cassandra:compact Runs nodetool compact against the test instance
of Cassandra.
  * cassandra:cleanup Runs nodetool cleanup against the test instance
of Cassandra.
  * cassandra:delete Deletes the the test instance of Cassandra.
  * cassandra:cql-exec Execute a CQL statement (directly or from a
file) against the test instance of Cassandra.

http://mojo.codehaus.org/cassandra-maven-plugin/

To use this version, simply specify the version in your project's
plugin configuration:


 org.codehaus.mojo
 cassandra-maven-plugin
 0.8.6-1


Release Notes - Mojo's Cassandra Maven Plugin - Version 0.8.6-1

** Bug
* [MCASSANDRA-12] - cassandra:start fails on Cywin

** Improvement
* [MCASSANDRA-13] - Upgrade to Cassandra 0.8.6

Enjoy,

The Mojo team.

Apache, Apache Maven, Apache Cassandra, Maven and Cassandra are
trademarks of The Apache Software Foundation.

how does compaction_throughput_kb_per_sec affect disk io?

[Yan Chunlu]

I am using the default 16MB when running repair. but the disk io is still
quite high:

Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sdb 136.00 0.00  506.00   26.00 63430.00  5880.00   260.56
101.73  224.386.60 4462.62   1.88 100.00

Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sdb  58.50 21978.00  131.00  122.50 16226.00 52596.00   542.97
122.98  870.28   10.02 1790.24   3.94 100.00




the  rkB/s  and wKB/s are almost 60MB, did I misunderstand the meaning of
compaction throttle?


cassandra version is 0.8.6

Thrift CPU Usage

[Baskar Duraikannu]

Hello - 

I have been running read tests on Cassandra using "stress" tool.  I have been 
noticing that thrift seems to be taking lot of CPU over 70% when I look at the 
"CPU samples" report. Is this normal?  

CPU usage seems to go down by 5 to 10% when I change the RPC from "sync" to 
"async".  Is this normal? 

I am running Cassandra 0.8.4 on Cent OS 5.6 ( Kernel 2.6.18.238) and Oracle 
JVM. 

- 
Thanks
Baskar Duraikannu 

Quick advice needed - CF backup and restore

[Oleg Proudnikov]

Hi,

What is the easiest way to save/backup a single column family across the cluster
and later reload it? 

Thank you very much,
Oleg

Re: Seed nodes in cassandra.yaml can not be hostnames

[Jonathan Ellis]

The seed names should match what the seeds advertise as
listen_address.  I can't think of a reason host names shouldn't work,
but as Peter said, using host names is a bad idea anyway.

2011/9/25 Radim Kolar :
> I just discovered that using host names for seed nodes in cassandra.yaml do
> not work. This is done on purpose?
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: how does compaction_throughput_kb_per_sec affect disk io?

[Jonathan Ellis]

compaction throughput doesn't affect flushing or reads

On Mon, Sep 26, 2011 at 7:40 AM, Yan Chunlu  wrote:
> I am using the default 16MB when running repair. but the disk io is still
> quite high:
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sdb             136.00     0.00  506.00   26.00 63430.00  5880.00   260.56
> 101.73  224.38    6.60 4462.62   1.88 100.00
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sdb              58.50 21978.00  131.00  122.50 16226.00 52596.00   542.97
> 122.98  870.28   10.02 1790.24   3.94 100.00
>
>
>
> the  rkB/s  and wKB/s are almost 60MB, did I misunderstand the meaning of
> compaction throttle?
>
> cassandra version is 0.8.6



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Quick advice needed - CF backup and restore

[Jonathan Ellis]

dsh -g mycluster -c "nodetool -h localhost snapshot"

see http://www.netfort.gr.jp/~dancer/software/dsh.html.en

On Mon, Sep 26, 2011 at 8:36 AM, Oleg Proudnikov  wrote:
> Hi,
>
> What is the easiest way to save/backup a single column family across the 
> cluster
> and later reload it?
>
> Thank you very much,
> Oleg
>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

SCF column comparator

[Sam Hodgson]

Hi all,

Im trying to create a Threads SCF that will store message thread id's in date 
order and i want to store the threadID => subject as the supercolumns.  Please 
correct me if im incorrect but my understanding of a super column family is as 
follows:

Category: //row key
   Timestamp: //Column
  ThreadID => Thread Subject //Supercolumn

My question is how do I ensure the timestamp column is ordered chronologically 
as the documentation for the cassandra-cli specify's that the comparator type 
is applied to the supercolumns.  Also im new to Cassandra so any advice on 
alternate schemas is welcome, not sure if this would be better to split up into 
multiple CF's or somehow use an index.

Thanks in advance

Sam
  

Re: how does compaction_throughput_kb_per_sec affect disk io?

[mcasandra]

I would think that compaction_throughput_kb_per_sec does have indirect impact
on disk IO. High number means or setting it to 0 means there is no
throttling on how much IO is being performed. Wouldn't it impact normal
reads from disk during the time when disk IO or util is high which
compaction is taking place?

--
View this message in context: 
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-does-compaction-throughput-kb-per-sec-affect-disk-io-tp6831711p6832308.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
Nabble.com.

Re: Seed nodes in cassandra.yaml can not be hostnames

[Radim Kolar]

Dne 26.9.2011 16:37, Jonathan Ellis napsal(a):

The seed names should match what the seeds advertise as
listen_address.  I can't think of a reason host names shouldn't work,

I used DNS alias, that was probably reason why it didn't worked.

reverse range query performance

[Ramesh Natarajan]

Hi,

 I am trying to use the range query to retrieve a bunch of columns in
reverse order. The API documentation has a parameter bool reversed which
should return the results when queried using keys in a reverse order.

Lets say my row has about 1500 columns with column names 1 to 1500, and I
query asking for columns  1500 (start ) - 1400 (end ) with reverse set to
true.

Does cassandra read the entire row  1 - 1500 columns and then return the
result 1400 - 1500 or it is optimized to look directly into the 1400 - 1500
columns?

thanks
Ramesh


SliceRange

A SliceRange is a structure that stores basic range, ordering and limit
information for a query that will return multiple columns. It could be
thought of as Cassandra's version of LIMIT and ORDER BY.

*Attribute*

*Type*

*Default*

*Required*

*Description*

start

binary

n/a

Y

The column name to start the slice with. This attribute is not required,
though there is no default value, and can be safely set to '', i.e., an
empty byte array, to start with the first column name. Otherwise, it must be
a valid value under the rules of the Comparator defined for the given
ColumnFamily.

finish

binary

n/a

Y

The column name to stop the slice at. This attribute is not required, though
there is no default value, and can be safely set to an empty byte array to
not stop until count results are seen. Otherwise, it must also be a valid
value to the ColumnFamily Comparator.

reversed

bool

false

Y

Whether the results should be ordered in reversed order. Similar to
ORDER BY blah DESC in SQL.

count

integer

100

Y

How many columns to return. Similar to LIMIT 100 in SQL. May be arbitrarily
large, but Thrift will materialize the whole result into memory before
returning it to the client, so be aware that you may be better served by
iterating through slices by passing the last value of one call in as the
start of the next instead of increasing count arbitrarily large.

Update of column sometimes takes 10 seconds

[Rick Whitesel (rwhitese)]

Hi All:

 

We have a simple junit test that inserts a column, immediately updates
that column and then validates that the data updated. Cassandra is run
embedded in the unit test. Sometimes the test will pass, i.e. the
updated data is correct, and sometimes the test will fail. The
configuration is set to:

 

periodic and
1

 

We are running version 0.6.9. We plan to update to the latest version
but cannot until after the release we are wrapping up. We are using the
client batch mutate to create and update the data. From what I
understand, the commit log write will return immediately and the data
will be store in memory. If that is the case, then why would our test
sometimes fail?

 

-Rick Whitesel

 

 

Simplify, Standardize and Conserve

 

 

 

 

Rick Whitesel
Technical Lead
Customer Contact Business Unit

rwhit...@cisco.com  
Phone :978-936-0479


500 Beaver Brook Road
Boxboro, MA 01719
Mailing address:
1414 Massachusetts Avenue
Boxboro, MA 01719
United States
www.cisco.com  

 

 

<>

Re: Update of column sometimes takes 10 seconds

[Jim Ancona]

Do you actually see the update occur if you wait for 10 seconds (as your
subject implies), or do you just see intermittent failures when running the
unit test? If it's the latter, are you sure that the update has a greater
timestamp than the insert? I've seen similar unit tests fail because because
the timestamp values were the same.

Jim

On Mon, Sep 26, 2011 at 2:55 PM, Rick Whitesel (rwhitese) <
rwhit...@cisco.com> wrote:

> Hi All:
>
> ** **
>
> We have a simple junit test that inserts a column, immediately updates that
> column and then validates that the data updated. Cassandra is run embedded
> in the unit test. Sometimes the test will pass, i.e. the updated data is
> correct, and sometimes the test will fail. The configuration is set to:***
> *
>
> ** **
>
> periodic and
> 1
>
> ** **
>
> We are running version 0.6.9. We plan to update to the latest version but
> cannot until after the release we are wrapping up. We are using the client
> batch mutate to create and update the data. From what I understand, the
> commit log write will return immediately and the data will be store in
> memory. If that is the case, then why would our test sometimes fail?
>
> ** **
>
> -Rick Whitesel
>
> ** **
>
>  
>
> *Simplify, Standardize and Conserve*
>
> ** **
>
> ** **
>
> [image: cid:01608604-FE45-42D2-AACA-4C66FCE5AB8C@cisco.com]
>
> ** **
>
> *Rick Whitesel*
> *Technical Lead*
> *Customer Contact Business Unit**
> *
> rwhit...@cisco.com 
> Phone :*978-936-0479*
>
>
> 500 Beaver Brook Road
> Boxboro, MA 01719
> Mailing address:
> 1414 Massachusetts Avenue
> Boxboro, MA 01719
> United States
> www.cisco.com
>
> ** **
>
> ** **
>
<>

RE: Update of column sometimes takes 10 seconds

[Rick Whitesel (rwhitese)]

Waiting 10 seconds between the update and reading the updated data seems
to always work. Not waiting the 10 seconds will cause the test to
randomly pass or fail. 

 

-Rick

 

From: Jim Ancona [mailto:j...@anconafamily.com] 
Sent: Monday, September 26, 2011 3:04 PM
To: user@cassandra.apache.org
Subject: Re: Update of column sometimes takes 10 seconds

 

Do you actually see the update occur if you wait for 10 seconds (as your
subject implies), or do you just see intermittent failures when running
the unit test? If it's the latter, are you sure that the update has a
greater timestamp than the insert? I've seen similar unit tests fail
because because the timestamp values were the same. 

 

Jim

 

On Mon, Sep 26, 2011 at 2:55 PM, Rick Whitesel (rwhitese) <
rwhit...@cisco.com> wrote:

Hi All:

 

We have a simple junit test that inserts a column, immediately updates
that column and then validates that the data updated. Cassandra is run
embedded in the unit test. Sometimes the test will pass, i.e. the
updated data is correct, and sometimes the test will fail. The
configuration is set to:

 

periodic and
1

 

We are running version 0.6.9. We plan to update to the latest version
but cannot until after the release we are wrapping up. We are using the
client batch mutate to create and update the data. From what I
understand, the commit log write will return immediately and the data
will be store in memory. If that is the case, then why would our test
sometimes fail?

 

-Rick Whitesel

 

 

Simplify, Standardize and Conserve

 

 

 

 

Rick Whitesel
Technical Lead
Customer Contact Business Unit

rwhit...@cisco.com  
Phone :978-936-0479


500 Beaver Brook Road
Boxboro, MA 01719
Mailing address:
1414 Massachusetts Avenue
Boxboro, MA 01719
United States
www.cisco.com  

 

 

 

<>

GC for ParNew on 0.8.6

[Philippe]

Ever since upgrading to 0.8.6, my nodes' system.log is littered with
GCInspector logs such as these
INFO [ScheduledTasks:1] 2011-09-26 21:23:40,468 GCInspector.java (line 122)
GC for ParNew: 209 ms for 1 collections, 4747932608 used; max is 16838033408
 INFO [ScheduledTasks:1] 2011-09-26 21:23:43,788 GCInspector.java (line 122)
GC for ParNew: 218 ms for 1 collections, 4963170032 used; max is 16838033408
 INFO [ScheduledTasks:1] 2011-09-26 21:23:55,893 GCInspector.java (line 122)
GC for ParNew: 471 ms for 2 collections, 5564471992 used; max is 16838033408
 INFO [ScheduledTasks:1] 2011-09-26 21:23:57,095 GCInspector.java (line 122)
GC for ParNew: 569 ms for 2 collections, 5635053360 used; max is 16838033408
 INFO [ScheduledTasks:1] 2011-09-26 21:23:58,096 GCInspector.java (line 122)
GC for ParNew: 234 ms for 1 collections, 6101724672 used; max is 16838033408
 INFO [ScheduledTasks:1] 2011-09-26 21:23:59,129 GCInspector.java (line 122)
GC for ParNew: 573 ms for 2 collections, 5933842376 used; max is 16838033408
 INFO [ScheduledTasks:1] 2011-09-26 21:24:00,130 GCInspector.java (line 122)
GC for ParNew: 295 ms for 1 collections, 6289652528 used; max is 16838033408
 INFO [ScheduledTasks:1] 2011-09-26 21:24:01,331 GCInspector.java (line 122)
GC for ParNew: 490 ms for 2 collections, 6183013200 used; max is 16838033408
 INFO [ScheduledTasks:1] 2011-09-26 21:24:02,333 GCInspector.java (line 122)
GC for ParNew: 207 ms for 1 collections, 6350919632 used; max is 16838033408
 INFO [ScheduledTasks:1] 2011-09-26 21:24:09,464 GCInspector.java (line 122)
GC for ParNew: 440 ms for 2 collections, 6824630136 used; max is 16838033408


I have changed absolutely nothing and the work load is perhaps even lower
than before upgrading because I have paused most updates to the cluster. Did
a log level change or does this have deeper meaning ?

Thanks

Re: Update of column sometimes takes 10 seconds

[Jonathan Ellis]

Sounds a lot like this to me:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Updates-lost-td6739612.html

On Mon, Sep 26, 2011 at 1:55 PM, Rick Whitesel (rwhitese) <
rwhit...@cisco.com> wrote:

> Hi All:
>
> ** **
>
> We have a simple junit test that inserts a column, immediately updates that
> column and then validates that the data updated. Cassandra is run embedded
> in the unit test. Sometimes the test will pass, i.e. the updated data is
> correct, and sometimes the test will fail. The configuration is set to:***
> *
>
> ** **
>
> periodic and
> 1
>
> ** **
>
> We are running version 0.6.9. We plan to update to the latest version but
> cannot until after the release we are wrapping up. We are using the client
> batch mutate to create and update the data. From what I understand, the
> commit log write will return immediately and the data will be store in
> memory. If that is the case, then why would our test sometimes fail?
>
> ** **
>
> -Rick Whitesel
>
> ** **
>
>  
>
> *Simplify, Standardize and Conserve*
>
> ** **
>
> ** **
>
> [image: cid:01608604-FE45-42D2-AACA-4C66FCE5AB8C@cisco.com]
>
> ** **
>
> *Rick Whitesel*
> *Technical Lead*
> *Customer Contact Business Unit**
> *
> rwhit...@cisco.com 
> Phone :*978-936-0479*
>
>
> 500 Beaver Brook Road
> Boxboro, MA 01719
> Mailing address:
> 1414 Massachusetts Avenue
> Boxboro, MA 01719
> United States
> www.cisco.com
>
> ** **
>
> ** **
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
<>

RE: Update of column sometimes takes 10 seconds

[Rick Whitesel (rwhitese)]

Thank you to all for the quick response. The test that fails is doing a
insert, another insert (to update data) and then a get to validate. If I
make multiple copies of the same test and execute them in succession,
different copies will fail on successive runs. Each test only has a
single get, so one would assume the get in testN must be causing testN+1
to fail.  It is going to take me a little bit to dig down deep enough
into our code to see if that is the issue.

 

-Rick

 

From: Jonathan Ellis [mailto:jbel...@gmail.com] 
Sent: Monday, September 26, 2011 3:34 PM
To: user@cassandra.apache.org
Subject: Re: Update of column sometimes takes 10 seconds

 

Sounds a lot like this to me: 
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Updates
-lost-td6739612.html

On Mon, Sep 26, 2011 at 1:55 PM, Rick Whitesel (rwhitese) <
rwhit...@cisco.com> wrote:

Hi All:

 

We have a simple junit test that inserts a column, immediately updates
that column and then validates that the data updated. Cassandra is run
embedded in the unit test. Sometimes the test will pass, i.e. the
updated data is correct, and sometimes the test will fail. The
configuration is set to:

 

periodic and
1

 

We are running version 0.6.9. We plan to update to the latest version
but cannot until after the release we are wrapping up. We are using the
client batch mutate to create and update the data. From what I
understand, the commit log write will return immediately and the data
will be store in memory. If that is the case, then why would our test
sometimes fail?

 

-Rick Whitesel

 

 

Simplify, Standardize and Conserve

 

 

 

 

Rick Whitesel
Technical Lead
Customer Contact Business Unit

rwhit...@cisco.com  
Phone :978-936-0479


500 Beaver Brook Road
Boxboro, MA 01719
Mailing address:
1414 Massachusetts Avenue
Boxboro, MA 01719
United States
www.cisco.com  

 

 




-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

<>

Re: Token != DecoratedKey assertion

[aaron morton]

Looks like a mismatch between the key the index says should be at a certain 
position in the date file and the key that is actually there. 

I've not checked but scrub *may* fix this this. Try it and see. 

(repair is for repairing consistency between nodes, scrub fixes local issues 
with data. )

Cheers
 
-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 26/09/2011, at 12:53 PM, Philippe wrote:

> Juste did
> Could there be data corruption or will repairs do this?
> 
> Thanks
> 
> Le 25 sept. 2011 15:30, "Jonathan Ellis"  a écrit :
> > Assertion errors are bugs, so that should worry you.
> > 
> > However, I'd upgrade before filing a ticket. There were a lot of
> > fixes in 0.8.5.
> > 
> > On Sun, Sep 25, 2011 at 2:27 AM, Philippe  wrote:
> >> Hello,
> >> I've seen a couple of these in my logs, running 0.8.4.
> >> This is a RF=3, 3-node cluster. 2 nodes including this one are on 0.8.4 and
> >> one is on 0.8.5
> >>
> >> The node is still functionning hours later. Should I be worried ?
> >>
> >> Thanks
> >>
> >> ERROR [ReadStage:94911] 2011-09-24 22:40:30,043 
> >> AbstractCassandraDaemon.java
> >> (line 134) Fatal exception in thread Thread[ReadStage:94911,5,main]
> >> java.lang.AssertionError:
> >> DecoratedKey(Token(bytes[224ceb80b5fb11e0848783ceb9bf0002ff33]),
> >> 224ceb80b5fb11e0848783ceb9bf0002ff33) !=
> >> DecoratedKey(Token(bytes[038453154cb0005f14]), 038453154cb0005f14)
> >> in /var/lib/cassandra/data/X/PUBLIC_MONTHLY_20-g-10634-Data.db
> >> at
> >> org.apache.cassandra.db.columniterator.SSTableSliceIterator.(SSTableSliceIterator.java:59)
> >> at
> >> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
> >> at
> >> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
> >> at
> >> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1315)
> >> at
> >> org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1182)
> >> at
> >> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1222)
> >> at
> >> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1169)
> >> at org.apache.cassandra.db.Table.getRow(Table.java:385)
> >> at
> >> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> >> at
> >> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:642)
> >> at
> >> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1107)
> >> at
> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> at
> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> at java.lang.Thread.run(Thread.java:662)
> >> ERROR [ReadStage:94936] 2011-09-24 22:40:30,042 
> >> AbstractCassandraDaemon.java
> >> (line 134) Fatal exception in thread Thread[ReadStage:94936,5,main]
> >> java.lang.AssertionError: DecoratedKey(Token(bytes[]), ) !=
> >> DecoratedKey(Token(bytes[038453154c90005f14]), 038453154c90005f14)
> >> in /var/lib/cassandra/data/X/PUBLIC_MONTHLY_20-g-10634-Data.db
> >> at
> >> org.apache.cassandra.db.columniterator.SSTableSliceIterator.(SSTableSliceIterator.java:59)
> >> at
> >> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
> >> at
> >> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
> >> at
> >> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1315)
> >> at
> >> org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1182)
> >> at
> >> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1222)
> >> at
> >> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1169)
> >> at org.apache.cassandra.db.Table.getRow(Table.java:385)
> >> at
> >> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> >> at
> >> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:642)
> >> at
> >> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1107)
> >> at
> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> at
> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> at java.lang.Thread.run(Thread.java:662)
> >> ERROR [ReadStage:94713] 2011-09-24 22:40:30,041 
> >> AbstractCassandraDaemon.java
> >> (line 134) Fatal exception in thread Thread[ReadStag

Re: Thrift CPU Usage

[aaron morton]

How are you deciding what is thrift ? Thrift is used to handle connections and 
serialize  / de-serialize off the wire.

Cheers
   


-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 27/09/2011, at 2:32 AM, Baskar Duraikannu wrote:

> Hello - 
> 
> I have been running read tests on Cassandra using "stress" tool.  I have been 
> noticing that thrift seems to be taking lot of CPU over 70% when I look at 
> the "CPU samples" report. Is this normal?  
> 
> CPU usage seems to go down by 5 to 10% when I change the RPC from "sync" to 
> "async".  Is this normal? 
> 
> I am running Cassandra 0.8.4 on Cent OS 5.6 ( Kernel 2.6.18.238) and Oracle 
> JVM. 
> 
> - 
> Thanks
> Baskar Duraikannu

Re: Surgecon Meetup?

[Dan Kuebrich]

I'll be at Surge on Thursday, would love to meet up.  Anyone else planning
to be there?

On Sun, Sep 25, 2011 at 7:27 PM, Chris Burroughs
wrote:

> Surge [1] is scalability focused conference in late September hosted in
> Baltimore.  It's a pretty cool conference with a good mix of
> operationally minded people interested in scalability, distributed
> systems, systems level performance and good stuff like that.  You should
> go! [2]
>
> Anyway, I'll be there if there, and are if any other Cassandra users are
> coming I'm happy to help herd us towards meeting up, lunch, hacking,
> etc.  I *think* there might be some time for structured BoF type
> sessions as well.
>
>
> [1] http://omniti.com/surge/2011
>
> [2] Actually tickets recenlty sold out, you should go in 2012!
>

Re: Surgecon Meetup?

[Wilson Mar]

I'll be there from Tuesday night thru the weekend.

- wilson...@gmail.com, 310.320-7878


On Mon, Sep 26, 2011 at 4:55 PM, Dan Kuebrich  wrote:
> I'll be at Surge on Thursday, would love to meet up.  Anyone else planning
> to be there?
>
> On Sun, Sep 25, 2011 at 7:27 PM, Chris Burroughs 
> wrote:
>>
>> Surge [1] is scalability focused conference in late September hosted in
>> Baltimore.  It's a pretty cool conference with a good mix of
>> operationally minded people interested in scalability, distributed
>> systems, systems level performance and good stuff like that.  You should
>> go! [2]
>>
>> Anyway, I'll be there if there, and are if any other Cassandra users are
>> coming I'm happy to help herd us towards meeting up, lunch, hacking,
>> etc.  I *think* there might be some time for structured BoF type
>> sessions as well.
>>
>>
>> [1] http://omniti.com/surge/2011
>>
>> [2] Actually tickets recenlty sold out, you should go in 2012!
>
>

Re: SCF column comparator

[aaron morton]

it's the other way around…

row-key:
super-column:
(sub)column:

When using Create Column Family in the CLI:

key_validation_class applies to the row key
comparator applies to the super column (when using a Super Column 
Family)
subcomparator applied to the sub column (when using a Super Column 
Family)

In your current model consider:

- using an Integer or TimeUUID comparator
- setting the super column name to milliseconds / seconds past epoch or using a 
Time UUID. 

I would suggest trying to model your data using a standard CF to start with, it 
will be easier on your brain and there are some limitations to a super CF. 

Hope that helps. 

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 27/09/2011, at 3:46 AM, Sam Hodgson wrote:

> Hi all,
> 
> Im trying to create a Threads SCF that will store message thread id's in date 
> order and i want to store the threadID => subject as the supercolumns.  
> Please correct me if im incorrect but my understanding of a super column 
> family is as follows:
> 
> Category: //row key
>Timestamp: //Column
>   ThreadID => Thread Subject //Supercolumn
> 
> My question is how do I ensure the timestamp column is ordered 
> chronologically as the documentation for the cassandra-cli specify's that the 
> comparator type is applied to the supercolumns.  Also im new to Cassandra so 
> any advice on alternate schemas is welcome, not sure if this would be better 
> to split up into multiple CF's or somehow use an index.
> 
> Thanks in advance
> 
> Sam

Re: Thrift CPU Usage

[Baskar Duraikannu]

Aaron
From the CPU samples report. Here is the parts of the CPU samples report 
(-Xrunhprof:cpu=samples, depth=4).  

TRACE 300668:
java.net.SocketInputStream.socketRead0(SocketInputStream.java:Unknown 
line)
java.net.SocketInputStream.read(SocketInputStream.java:129)

org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
TRACE 300310:
java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java:Unknown line)
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
java.net.ServerSocket.implAccept(ServerSocket.java:462)
java.net.ServerSocket.accept(ServerSocket.java:430)
TRACE 300639:

sun.nio.ch.ServerSocketChannelImpl.accept0(ServerSocketChannelImpl.java:Unknown 
line)

sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)

org.apache.cassandra.net.MessagingService$SocketThread.run(MessagingService.java:617)
TRACE 300670:

java.net.SocketOutputStream.socketWrite0(SocketOutputStream.java:Unknown line)
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
java.net.SocketOutputStream.write(SocketOutputStream.java:136)

org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
TRACE 301122:
java.lang.Object.notify(Object.java:Unknown line)

org.apache.cassandra.utils.SimpleCondition.signal(SimpleCondition.java:62)

org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:169)

org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java


rank   self  accum   count trace method
   1 74.00% 74.00%  160934 300668 java.net.SocketInputStream.socketRead0
   2 14.85% 88.85%   32302 300310 java.net.PlainSocketImpl.socketAccept
   3  3.67% 92.52%7990 300639 sun.nio.ch.ServerSocketChannelImpl.accept0
   4  1.90% 94.43%4142 300670 java.net.SocketOutputStream.socketWrite0
   5  0.79% 95.22%1716 301122 java.lang.Object.notify

--
Thanks
Baskar Duraikannu

On Sep 26, 2011, at 4:55 PM, aaron morton wrote:

> How are you deciding what is thrift ? Thrift is used to handle connections 
> and serialize  / de-serialize off the wire.
> 
> Cheers
> 
> 
> 
> -
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 27/09/2011, at 2:32 AM, Baskar Duraikannu wrote:
> 
>> Hello - 
>> 
>> I have been running read tests on Cassandra using "stress" tool.  I have 
>> been noticing that thrift seems to be taking lot of CPU over 70% when I look 
>> at the "CPU samples" report. Is this normal?  
>> 
>> CPU usage seems to go down by 5 to 10% when I change the RPC from "sync" to 
>> "async".  Is this normal? 
>> 
>> I am running Cassandra 0.8.4 on Cent OS 5.6 ( Kernel 2.6.18.238) and Oracle 
>> JVM. 
>> 
>> - 
>> Thanks
>> Baskar Duraikannu
> 

Re: Thrift CPU Usage

[Jeremiah Jordan]

Yes.  All the stress tool does is flood data through the API, no real 
processing or anything happens.  So thrift reading/writing data should 
be the majority of the CPU time...


On 09/26/2011 08:32 AM, Baskar Duraikannu wrote:

Hello -

I have been running read tests on Cassandra using "stress" tool.  I have been noticing 
that thrift seems to be taking lot of CPU over 70% when I look at the "CPU samples" 
report. Is this normal?

CPU usage seems to go down by 5 to 10% when I change the RPC from "sync" to 
"async".  Is this normal?

I am running Cassandra 0.8.4 on Cent OS 5.6 ( Kernel 2.6.18.238) and Oracle JVM.

-
Thanks
Baskar Duraikannu

Re: reverse range query performance

[aaron morton]

Does not matter to much but are you looking to get all the columns for some 
know keys (get_slice, multiget_slice) ? Or are you getting the columns for keys 
within a range (get_range_slices)? 

If you provide do a reversed query the server will skip to the "end" of the 
column range.  Here is some info I wrote about how the the different slice 
predicates work http://thelastpickle.com/2011/07/04/Cassandra-Query-Plans/


Hope that helps. 

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 27/09/2011, at 5:51 AM, Ramesh Natarajan wrote:

> Hi,
> 
>  I am trying to use the range query to retrieve a bunch of columns in reverse 
> order. The API documentation has a parameter bool reversed which should 
> return the results when queried using keys in a reverse order.  
> 
> Lets say my row has about 1500 columns with column names 1 to 1500, and I 
> query asking for columns  1500 (start ) - 1400 (end ) with reverse set to 
> true.
> 
> Does cassandra read the entire row  1 - 1500 columns and then return the 
> result 1400 - 1500 or it is optimized to look directly into the 1400 - 1500 
> columns?
> 
> thanks
> Ramesh
> 
> 
> SliceRange
> A SliceRange is a structure that stores basic range, ordering and limit 
> information for a query that will return multiple columns. It could be 
> thought of as Cassandra's version of LIMIT and ORDER BY.
> 
> Attribute
> Type
> Default
> Required
> Description
> start
> binary
> n/a
> Y
> The column name to start the slice with. This attribute is not required, 
> though there is no default value, and can be safely set to '', i.e., an empty 
> byte array, to start with the first column name. Otherwise, it must be a 
> valid value under the rules of the Comparator defined for the given 
> ColumnFamily.
> finish
> binary
> n/a
> Y
> The column name to stop the slice at. This attribute is not required, though 
> there is no default value, and can be safely set to an empty byte array to 
> not stop until count results are seen. Otherwise, it must also be a valid 
> value to the ColumnFamily Comparator.
> reversed
> bool
> false
> Y
> Whether the results should be ordered in reversed order. Similar to ORDER BY 
> blah DESC in SQL.
> count
> integer
> 100
> Y
> How many columns to return. Similar to LIMIT 100 in SQL. May be arbitrarily 
> large, but Thrift will materialize the whole result into memory before 
> returning it to the client, so be aware that you may be better served by 
> iterating through slices by passing the last value of one call in as the 
> start of the next instead of increasing count arbitrarily large.

Re: Update of column sometimes takes 10 seconds

[aaron morton]

Try turning up the logging to DEBUG and watch the requests come through. 

Check that the two inserts do indeed have different time stamps. 

In cases of lost updates, timestamps are most often the cause of the kerfuffle.

btw, in this situation the commit log is a red hearing riding a scapegoat. Data 
is dumped into the commit log for durability at the start of the write request, 
but it is never read from there. Data is read from memtables and sstables. 
 
Cheers

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 27/09/2011, at 9:27 AM, Rick Whitesel (rwhitese) wrote:

> Thank you to all for the quick response. The test that fails is doing a 
> insert, another insert (to update data) and then a get to validate. If I make 
> multiple copies of the same test and execute them in succession, different 
> copies will fail on successive runs. Each test only has a single get, so one 
> would assume the get in testN must be causing testN+1 to fail.  It is going 
> to take me a little bit to dig down deep enough into our code to see if that 
> is the issue.
>  
> -Rick
>  
> From: Jonathan Ellis [mailto:jbel...@gmail.com] 
> Sent: Monday, September 26, 2011 3:34 PM
> To: user@cassandra.apache.org
> Subject: Re: Update of column sometimes takes 10 seconds
>  
> Sounds a lot like this to me: 
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Updates-lost-td6739612.html
> 
> On Mon, Sep 26, 2011 at 1:55 PM, Rick Whitesel (rwhitese) 
>  wrote:
> Hi All:
>  
> We have a simple junit test that inserts a column, immediately updates that 
> column and then validates that the data updated. Cassandra is run embedded in 
> the unit test. Sometimes the test will pass, i.e. the updated data is 
> correct, and sometimes the test will fail. The configuration is set to:
>  
> periodic and 
> 1
>  
> We are running version 0.6.9. We plan to update to the latest version but 
> cannot until after the release we are wrapping up. We are using the client 
> batch mutate to create and update the data. From what I understand, the 
> commit log write will return immediately and the data will be store in 
> memory. If that is the case, then why would our test sometimes fail?
>  
> -Rick Whitesel
>  
>  
> Simplify, Standardize and Conserve
>  
>  
> 
>  
> Rick Whitesel
> Technical Lead
> Customer Contact Business Unit
> 
> rwhit...@cisco.com
> Phone :978-936-0479
> 
> 500 Beaver Brook Road
> Boxboro, MA 01719
> Mailing address:
> 1414 Massachusetts Avenue
> Boxboro, MA 01719
> United States
> www.cisco.com
>  
>  
> 
> 
> 
> -- 
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com

Re: how does compaction_throughput_kb_per_sec affect disk io?

[Yan Chunlu]

okay, thanks!

On Mon, Sep 26, 2011 at 10:38 PM, Jonathan Ellis  wrote:

> compaction throughput doesn't affect flushing or reads
>
> On Mon, Sep 26, 2011 at 7:40 AM, Yan Chunlu  wrote:
> > I am using the default 16MB when running repair. but the disk io is still
> > quite high:
> > Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s
> avgrq-sz
> > avgqu-sz   await r_await w_await  svctm  %util
> > sdb 136.00 0.00  506.00   26.00 63430.00  5880.00
> 260.56
> > 101.73  224.386.60 4462.62   1.88 100.00
> > Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s
> avgrq-sz
> > avgqu-sz   await r_await w_await  svctm  %util
> > sdb  58.50 21978.00  131.00  122.50 16226.00 52596.00
> 542.97
> > 122.98  870.28   10.02 1790.24   3.94 100.00
> >
> >
> >
> > the  rkB/s  and wKB/s are almost 60MB, did I misunderstand the meaning of
> > compaction throttle?
> >
> > cassandra version is 0.8.6
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>