Re: read operation is slow
Would it perhaps be worth denormalising your data so that you can
retrieve all rows as a single row using a key encoded with the query
predicate?
Until we get a stored proc feature (dunno if planned) it's hard to
avoid round trips without denormalizing/replication of data to fit
your query paths
Simon Reavely
On Jun 11, 2010, at 9:49 PM, "caribbean410"
wrote:
Thanks for the suggestion. For the test case, it is 1 key and 1
column. I once changed 10 to 1, as I remember there is no much
difference.
I have 200k keys and each key is randomly generated. I will try the
optimized query next week. But maybe you still have to face the case
that each time a client just wants to query one key from db.
From: Dop Sun [mailto:su...@dopsun.com]
Sent: Friday, June 11, 2010 6:05 PM
To: user@cassandra.apache.org
Subject: RE: read operation is slow
And also, you are only select 1 key and 10 columns?
criteria.keyList(Lists.newArrayList(userName)).columnRange
(nameFirst, nameFirst, 10);
Then, if you have 200k keys, you have 200k Thrift calls. If this is
the case, you may need to optimize the way you do the query (to
combine multiple keys into a single query), and to reduce the number
of calls.
From: Dop Sun [mailto:su...@dopsun.com]
Sent: Saturday, June 12, 2010 8:57 AM
To: user@cassandra.apache.org
Subject: RE: read operation is slow
You mean after you “I remove some unnecessary column family and chan
ge the size of rowcache and keycache, now the latency changes from 0
.25ms to 0.09ms. In essence 0.09ms*200k=18s.”, it still takes 400 se
conds to returning?
From: Caribbean410 [mailto:caribbean...@gmail.com]
Sent: Saturday, June 12, 2010 8:48 AM
To: user@cassandra.apache.org
Subject: Re: read operation is slow
Hi, do you mean this one should not introduce much extra delay? To
read a record, I need select here, not sure where the extra delay
comes from.
On Fri, Jun 11, 2010 at 5:29 PM, Dop Sun wrote:
Jassandra is used here:
Map> map = criteria.select();
The select here basically is a call to Thrift API: get_range_slices
From: Caribbean410 [mailto:caribbean...@gmail.com]
Sent: Saturday, June 12, 2010 8:00 AM
To: user@cassandra.apache.org
Subject: Re: read operation is slow
I remove some unnecessary column family and change the size of
rowcache and keycache, now the latency changes from 0.25ms to
0.09ms. In essence 0.09ms*200k=18s. I don't know why it takes more
than 400s total. Here is the client code and cfstats. There are not
many operations here, why is the extra time so large?
long start = System.currentTimeMillis();
for (int j = 0; j < 1; j++) {
for (int i = 0; i < numOfRecords; i++) {
int n = random.nextInt(numOfRecords);
ICriteria criteria = cf.createCriteria();
userName = keySet[n];
criteria.keyList(Lists.newArrayList
(userName)).columnRange(nameFirst, nameFirst, 10);
Map> map =
criteria.select();
List list = map.get(userName);
// ByteArray bloc = list.get(0).getValue();
// byte[] byteArrayloc = bloc.toByteArray();
// loc = new String(byteArrayloc);
// readBytes = readBytes + loc.length();
readBytes = readBytes + blobSize;
}
}
long finish=System.currentTimeMillis();
float totalTime=(finish-start)/1000;
Keyspace: Keyspace1
Read Count: 60
Read Latency: 0.090530067 ms.
Write Count: 20
Write Latency: 0.01504989 ms.
Pending Tasks: 0
Column Family: Standard2
SSTable count: 3
Space used (live): 265990358
Space used (total): 265990358
Memtable Columns Count: 2615
Memtable Data Size: 2667300
Memtable Switch Count: 3
Read Count: 60
Read Latency: 0.091 ms.
Write Count: 20
Write Latency: 0.015 ms.
Pending Tasks: 0
Key cache capacity: 1000
Key cache size: 187465
Key cache hit rate: 0.0
Row cache capacity: 1000
Row cache size: 189990
Row cache hit rate: 0.68335
Compacted row minimum size: 0
Compacted row maximum size: 0
Compacted row mean size: 0
Keyspace: system
Read Count: 1
Read Latency: 10.954 ms.
Write Count: 4
Write Latency: 0.28075 ms.
Pending Tasks: 0
Column Family: HintsColumnFamily
SSTable count: 0
Space used (live): 0
Space used (total): 0
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 0
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks:
Looking for recommendation on DELL server types/spec for running Cassandra nodes
Hi, I'm looking for a recommendation on DELL server types/spec for running Cassandra nodes What model DELL servers are folks using with success? What models or type of hardware have people regretted using? -- Simon Reavely simon.reav...@gmail.com
Monitoring cassandra using Munin...how to get a hourly graph?
Hi, For those of you using Munin to monitor Cassandra's JMX stats I wondered if anyone had figured out how to get hourly graphs. By default we are getting daily, weekly, monthly, yearly but for our performance testing we really need hourly and looking on the message boards we can't figure out how to do it. Cheers, Simon -- Simon Reavely simon.reav...@gmail.com
Re: Schema question
Thanks for the writeup...good stuff!
Any lessons learnt you'd like to share or challenges that persist?
Simon Reavely
On Sep 20, 2010, at 6:37 AM, Juho Mäkinen wrote:
> We have built a facebook style "messenger" into our web site which
> uses cassandra as storage backend with two column families:
> TalkMessages and TalkLastMessages. I've uploaded a screenshot showing
> the feature in action to
> http://img138.imageshack.us/img138/3807/talkexample.jpg
>
> TalkMessages contains each message between two participants. The key
> is a string built from the two users uids "$smaller_uid:$bigger_uid".
> Each column inside this CF contains a single message. The column name
> is the message timestamp in microseconds since epoch stored as
> LongType. The column value is a JSON encoded string containing
> following fields: sender_uid, target_uid, msg.
>
> This results in following structure inside the column family.
>
> "2249:9111" => [
> 12345678 : { sender_uid : 2249, target_uid : 9111, msg : "Hello, how
> are you?" },
> 12345679 : { sender_uid : 9111, target_uid : 2249, msg : "I'm fine, thanks" }
> ]
>
> TalkLastMessages is used to quickly fetch users talk partners, the
> last message which was sent between the peers and other similar data.
> This allows us to quickly fetch all needed data which is needed to
> display a "main view" for all online friends with just one query to
> cassandra. This column family uses the user uid as is key. Each column
> represents a talk partner whom the user has been talking to and it
> uses the talk partner uid as the column name. Column value is a json
> packed structure which contains following fields:
> - last message timestamp: microseconds since epoch when a message was
> last sent between these two users.
> - unread timestamp : microseconds since epoch when the first unread
> message was sent between these two users.
> - unread : counter how many unread messages there are.
> - last message : last message between these two users.
>
> This results in following structure inside the column family for these
> two example users: 2249 and 9111.
>
> "2249" => [
> 9111 : { last_message_timestamp : 12345679, unread_timestamp :
> 12345679, unread : 1, last_message: "I'm fine, thanks" }
>
> ],
> "9111" => [
> 2249 : { last_message_timestamp : 12345679, unread_timestamp :
> 12345679, unread : 0, last_message: "I'm fine, thanks" }
> ]
>
> Displaying chat (this happends on every page load, needs to be fast)
> 1) Fetch all columns from TalkLastMessages for the user
>
> Display messages history between two participants:
> 1) Fetch last n columns from TalkMessages for the relevant
> "$smaller_uid:$bigger_uid" row.
>
> Mark all sent messages from another participant as read (when you read
> the messages)
> 1) Get column $sender_uid from row $reader_uid from TalkLastMessages
> 2) Update the JSON payload and insert the column back
>
> Sending message involves the following operations:
> 1) Insert new column to TalkMessages
> 2) Fetch relevant column from TalkLastMessages from $target_uid row
> with $sender_uid column
> 3) Update the column json payload and insert it back to TalkLastMessages
> 4) Fetch relevant column from TalkLastMessages from $sender_uid row
> with $target_uid column
> 5) Update the column json payload and insert it back to TalkLastMessages
>
> There are also other operations and the actual payload is a bit more complex.
>
> I'm happy to answer questions if somebody is interested :)
>
> - Juho Mäkinen
>
>
>
> On Mon, Sep 20, 2010 at 12:57 PM, Morten Wegelbye Nissen
> wrote:
>> Hello List,
>>
>> No matter where you read, you almost every-where read the the noSQL
>> datascema is completely different from the relational way - and after a
>> little insight in cassandra everyone can 2nd that.
>>
>> But I miss to see some real-life examples on how a real system can be
>> modelled. Lets take the example for a system where users can send messages
>> to each other. ( Completely imaginary, noone would use cassandra for a
>> mailsystem :) )
>>
>> If one should create such a system, what CF's would be used? And how would
>> you per example find all not read messages?
>>
>> ./Morten
>>
Re: Schema question
Two questions:
1. So this compaction challenge is a CPU issue or a disk IO issue in your case?
2. In other places people have recommended adjustments from the defaults to
control compaction overhead...did you adjust or experiment with how to control
compaction?
Simon Reavely
On Sep 21, 2010, at 8:48 AM, Juho Mäkinen wrote:
> Not really. The schema has worked without any problems and we haven't
> had any problems with it. We're running a five node cassandra cluster
> behind the system (it has also other uses than just this particular
> application like it stores all our blog contents and bunch of other
> data). There are about 120 000 new messages each day and the chat
> window is displayed about 1200 times per second.
>
> It's worth to note that cassandra cluster can't be just one or two
> machines to get best availability because once a node starts to do
> compacting it will severely hurt the node performance. Thus you need
> to have enough nodes that the probability that all nodes are doing
> compaction is low enough. I've implemented a php wrapper around the
> thrift api which retries operation to another server until the result
> can be obtained. The library is available at github:
> http://github.com/dynamoid/cassandra-utilities
>
> We have monitored and stored performance data for each cassandra
> request we have done. As the results are interesting I'll be posting
> them into another post into this mailing list within a moment.
>
> - Juho Mäkinen
>
>
>
>
> On Tue, Sep 21, 2010 at 1:20 PM, Simon Reavely
> wrote:
>> Thanks for the writeup...good stuff!
>> Any lessons learnt you'd like to share or challenges that persist?
>>
>>
>> Simon Reavely
>>
>>
>> On Sep 20, 2010, at 6:37 AM, Juho Mäkinen wrote:
>>
>>> We have built a facebook style "messenger" into our web site which
>>> uses cassandra as storage backend with two column families:
>>> TalkMessages and TalkLastMessages. I've uploaded a screenshot showing
>>> the feature in action to
>>> http://img138.imageshack.us/img138/3807/talkexample.jpg
>>>
>>> TalkMessages contains each message between two participants. The key
>>> is a string built from the two users uids "$smaller_uid:$bigger_uid".
>>> Each column inside this CF contains a single message. The column name
>>> is the message timestamp in microseconds since epoch stored as
>>> LongType. The column value is a JSON encoded string containing
>>> following fields: sender_uid, target_uid, msg.
>>>
>>> This results in following structure inside the column family.
>>>
>>> "2249:9111" => [
>>> 12345678 : { sender_uid : 2249, target_uid : 9111, msg : "Hello, how
>>> are you?" },
>>> 12345679 : { sender_uid : 9111, target_uid : 2249, msg : "I'm fine,
>>> thanks" }
>>> ]
>>>
>>> TalkLastMessages is used to quickly fetch users talk partners, the
>>> last message which was sent between the peers and other similar data.
>>> This allows us to quickly fetch all needed data which is needed to
>>> display a "main view" for all online friends with just one query to
>>> cassandra. This column family uses the user uid as is key. Each column
>>> represents a talk partner whom the user has been talking to and it
>>> uses the talk partner uid as the column name. Column value is a json
>>> packed structure which contains following fields:
>>> - last message timestamp: microseconds since epoch when a message was
>>> last sent between these two users.
>>> - unread timestamp : microseconds since epoch when the first unread
>>> message was sent between these two users.
>>> - unread : counter how many unread messages there are.
>>> - last message : last message between these two users.
>>>
>>> This results in following structure inside the column family for these
>>> two example users: 2249 and 9111.
>>>
>>> "2249" => [
>>> 9111 : { last_message_timestamp : 12345679, unread_timestamp :
>>> 12345679, unread : 1, last_message: "I'm fine, thanks" }
>>>
>>> ],
>>> "9111" => [
>>> 2249 : { last_message_timestamp : 12345679, unread_timestamp :
>>> 12345679, unread : 0, last_message: "I'm fine, thanks" }
>>> ]
>>>
>>> Displaying chat (this happends on every page load, needs to be fast)
>>> 1) Fetch all columns from TalkLastMessages for
Re: atomic test-or-set
Ryan, Independent of this ambiguous requirement what were you thinking about. What I am trying to ask is can you be more specific/concrete about when you can Simon Reavely On Oct 5, 2010, at 11:30 AM, Ryan King wrote: > On Tue, Oct 5, 2010 at 8:23 AM, Ian Rogers > wrote: >> >> Does Cassandra have an atomic test-or-set operation? >> >> That is, I want to check to see if a key has a value and, if not, set it to >> something. But it must be an atomic operation - I can't do a separate fetch >> and then set from the application as that creates a possible race condition >> with another application. >> >> Equivalent to the SQL "update foo set grum = 'bang' where isnull(grum) ..." > > You can (sorta) do this per-row, but you can't scan a whole CF and do this. > >> Any thoughts how to do this? > > I think you can probably do what you want with cassandra, but your > example is too vague to be sure. > > -ryan
Facebook messaging and choice of HBase over Cassandra - what can we learn?
(Posting this to both user + dev lists) I was reviewing the blog post on the facebook engineering blog from nov 15th http://www.facebook.com/note.php?note_id=454991608919# <http://www.facebook.com/note.php?note_id=454991608919#> The Underlying Technology of Messages by Kannan Muthukkaruppan <http://www.facebook.com/Kannan> As a cassandra user I think the key sentence for this community is: "We found Cassandra's eventual consistency model to be a difficult pattern to reconcile for our new Messages infrastructure." I think it would be useful to find out more about this statement from Kannan and the facebook team. Does anyone have any contacts in the Facebook team? My goal here is to understand usage patterns and whether or not the Cassandra community can learn from this decision; maybe even understand whether the Cassandra roadmap should be influenced by this decision to address a target user base. Of course we might also conclude that its just "not a Cassandra use-case"! Cheers, Simon -- Simon Reavely simon.reav...@gmail.com
