Re: Storing pre-sorted data

[Matthias Pfau]

Hi David,
yes, what we are working on could be referenced as "encrypted database 
service".


Thanks for your insights. We will continue to work on this topic!

Kind regards
Matthias

On 10/21/2011 02:31 AM, David Jeske wrote:

If I understand you correctly, you are saying that you will never have
the encryption key, but that some third-party will. Given this
description, the design space you are in has nothing to do with
Cassandra-per-se. Cassandra, like any sorted-order storage, will keep
data in the order of a key that it can read. A database can't keep data
sorted in an order that it unknown to it.

I get the idea you are trying to provide "encrypted database services"
to third-parties, and that you are trying to give them sorted-order
retrieval. This is a "hard problem". The only two options I see were
detailed in my previous explanation.

1) require the client/third-party expose some non-encrypted data, which
can be used for sorting. Leave it up to them how they can generate data
useful for sorting which does not compromise security. (previously
described as option a)

2) Use some bleeding-edge research order-preserving encryption
algorithm. (Also, don't compress the sort-key.) If the encrypted form
sorts in the same order as the unencrypted form, then any database can
store the encrypted key as if it was normal data and keep the data in
proper sorted-order. (some extra work would be required for composite
keys) (previously described as option c)

I hope that helps.. Good luck!

Re: weird problem with performance

[Jérémy SEVELLEC]

@Araron you're right and i was wrong!

2011/10/20 Yang 

> found it , https://issues.apache.org/jira/browse/CASSANDRA-3387
>
> On Thu, Oct 20, 2011 at 1:37 PM, aaron morton 
> wrote:
> > It's unlikely that HH is the issue. (Disclaimer, am not familiar with HH
> in
> > 1.0, i know it's changes a bit)
> > Take a look at the TP Stats, what's happening ?
> > Cheers
> > -
> > Aaron Morton
> > Freelance Developer
> > @aaronmorton
> > http://www.thelastpickle.com
> > On 20/10/2011, at 10:10 AM, Jérémy SEVELLEC wrote:
> >
> > Ok.
> > I think a degration could be normal because your cluster is in a degraded
> > state when a node is down.
> > With a replication_factor of 3 and with a 3 nodes cluster, each data you
> > write is replicated on each node. As One node is down, when writing, it's
> > impossible to send a replica on the down node and hints are send
> > :
> http://www.datastax.com/docs/1.0/dml/about_writes#hinted-handoff-writes
> > And it could be more expensive to achieve QUORUM when you read in that
> > context.
> > It may be one explanation. You can turn cassandra log into debug level to
> > see what happen when when there is a down node.
> > 2011/10/19 Yang 
> >>
> >> 3
> >>
> >> sorry forgot this important info
> >>
> >> On Oct 19, 2011 11:31 AM, "Jérémy SEVELLEC" 
> wrote:
> >>>
> >>> Hi, what is your replication_factor?
> >>>
> >>> 2011/10/19 Yang 
> 
>  I'm using a cassandra version compiled from 1.0.0  github HEAD.
> 
>  I have 3 nodes, A B and C,  on node A I run a client, which talks only
>  to B as the coordinator.
> 
>  the performance is pretty good, a QUORUM read+write  takes < 10ms.
> 
>  but then I shutdown C, quickly the performance starts to degrade, and
>  QUORUM read+write time steadily increase to about 300ms.
> 
>  if I shutdown A and keep C, I observe the same effect.
> 
> 
>  I understand that "2 out of 3" is going to give you faster response
>  than "2 out of 2", but  the difference should not be that dramatic as
>  10ms vs 300ms.
> 
>  any possible reasons for this?(or how to debug this?)
> 
>  Thanks
>  Yang
> >>>
> >>>
> >>>
> >>> --
> >>> Jérémy
> >
> >
> >
> > --
> > Jérémy
> >
> >
>



-- 
Jérémy

Re: Cassandra 1.0: Exception in compaction

[Sylvain Lebresne]

I believe this is the same as
https://issues.apache.org/jira/browse/CASSANDRA-3306.

The initial reporter only got this exception with leveled compaction,
is it what you are
using too (to help narrow it down)? Also, are you using windows by any chance?

--
Sylvain


On Thu, Oct 20, 2011 at 9:04 PM, Ramesh Natarajan  wrote:
> We are running a 8 node cassandra 1.0 cluster. We are seeing this
> exception quite often. Any idea how to debug this issue?
>
>
> java.lang.IllegalArgumentException: Illegal Capacity: -2
>        at java.util.ArrayList.(ArrayList.java:110)
>        at 
> org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>        at 
> org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>        at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>        at 
> org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>        at 
> org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>        at 
> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>        at 
> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>        at 
> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
>
>
>
> few lines before this error
>
>  INFO [FlushWriter:222] 2011-10-20 10:52:25,885 Memtable.java (line
> 237) Writing Memtable-participants@1907757288(6777526/153164388
> serialized/live bytes, 199339 ops)
>  INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line
> 488) Discarding obsolete commit
> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319115938691.log)
>  INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line
> 488) Discarding obsolete commit
> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319122061956.log)
>  INFO [FlushWriter:222] 2011-10-20 10:52:26,865 Memtable.java (line
> 273) Completed flushing
> /var/lib/cassandra/data/MSA/participants-h-87-Data.db (14695382 bytes)
>  INFO [FlushWriter:222] 2011-10-20 10:52:26,866 Memtable.java (line
> 237) Writing Memtable-modseq@1745889706(13206/311769 serialized/live
> bytes, 426 ops)
>  INFO [FlushWriter:222] 2011-10-20 10:52:26,896 Memtable.java (line
> 273) Completed flushing
> /var/lib/cassandra/data/MSA/modseq-h-262-Data.db (38646 bytes)
>  INFO [FlushWriter:222] 2011-10-20 10:52:26,897 Memtable.java (line
> 237) Writing Memtable-msgid@2099219781(3571249/77183008
> serialized/live bytes, 109823 ops)
>  INFO [FlushWriter:222] 2011-10-20 10:52:27,497 Memtable.java (line
> 273) Completed flushing /var/lib/cassandra/data/MSA/msgid-h-47-Data.db
> (8125165 bytes)
>  INFO [FlushWriter:222] 2011-10-20 10:52:27,498 Memtable.java (line
> 237) Writing Memtable-uid@578022704(43734344/317200563 serialized/live
> bytes, 611301 ops)
>  INFO [FlushWriter:222] 2011-10-20 10:52:29,802 Memtable.java (line
> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-291-Data.db
> (48225128 bytes)
>  INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line
> 488) Discarding obsolete commit
> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125356477.log)
>  INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line
> 488) Discarding obsolete commit
> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125683351.log)
>  INFO [MutationStage:88] 2011-10-20 10:52:29,905
> ColumnFamilyStore.java (line 664) Enqueuing flush of
> Memtable-modseq@339630706(155217/3664394 serialized/live bytes, 5007
> ops)
>  INFO [FlushWriter:222] 2011-10-20 10:52:29,905 Memtable.java (line
> 237) Writing Memtable-modseq@339630706(155217/3664394 serialized/live
> bytes, 5007 ops)
>  INFO [FlushWriter:222] 2011-10-20 10:52:30,216 Memtable.java (line
> 273) Completed flushing
> /var/lib/cassandra/data/MSA/modseq-h-263-Data.db (450477 bytes)
> ERROR [CompactionExecutor:538] 2011-10-20 10:52:36,132
> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
> Thread[CompactionExecutor:538,1,main]
>
>
>
> Another one
>
>
>
>  INFO [FlushWriter:222] 2011-10-20 10:52:39,623 Memtable.java (line
> 237) Writing Memtable-uid@2018688194(79740/578345 serialized/live
> bytes,  ops)
>  INFO [FlushWriter:222] 2011-10-20 10:52:39,777 Memtable.java (line
> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-295-Data.db
> (142584 bytes)
>  INFO [CompactionExecutor:544] 2011-10-20 10:52:39,778
> CompactionTask.java (line 119) Compacting
> [SSTableReader(path='/var/lib/cassandra/data/MSA/uid-

log line question

[Rene Kochen]

Given the following log line:

DEBUG [ReadStage:20] 11:39:07,028 collecting 0 of 2147483647: 
SuperColumn(2150726f70657274696573 [64617461:false:4@1319189945952058,])

What does "false:4" in the column mean?

Thanx!

Rene

Re: log line question

[Sylvain Lebresne]

On Fri, Oct 21, 2011 at 12:48 PM, Rene Kochen
 wrote:
> Given the following log line:
>
>
>
> DEBUG [ReadStage:20] 11:39:07,028 collecting 0 of 2147483647:
> SuperColumn(2150726f70657274696573 [64617461:false:4@1319189945952058,])
>
>
>
> What does “false:4” in the column mean?

false means the column is live (not a tombstone) and 4 is the length
of the value (the value is not displayed in that log output).

--
Sylvain

>
>
>
> Thanx!
>
>
>
> Rene

Re: Cassandra 1.0.0 - Node Load Bug

[Jeremiah Jordan]

I thought this patch made it into the 1.0 release?  I remember it being 
referenced in one of the re-rolls.


On Oct 20, 2011, at 9:56 PM, "Jonathan Ellis"  wrote:

> That looks to me like it's reporting uncompressed size as the load.
> Should be fixed in the 1.0 branch for 1.0.1.
> (https://issues.apache.org/jira/browse/CASSANDRA-3338)
> 
> On Thu, Oct 20, 2011 at 11:53 AM, Dan Hendry  
> wrote:
>> I have been playing around with Cassandra 1.0.0 in our test environment it
>> seems pretty sweet so far. I have however come across what appears to be a
>> bug tracking node load. I have enabled compression and levelled compaction
>> on all CFs (scrub  + snapshot deletion) and the nodes have been operating
>> normally for a day or two. I started getting concerned when the load as
>> reported by nodetool ring kept increasing (it seems monotonically) despite
>> seeing a compression ratio of ~2.5x (as a side note, I find it strange
>> Cassandra does not provide the compression ratio via jmx or in the logs). I
>> initially thought there might be a bug in cleaning up obsolete SSTables but
>> I then noticed the following discrepancy:
>> 
>> 
>> 
>> Nodetool ring reports:
>> 
>> 10.112.27.65datacenter1 rack1   Up Normal  8.64
>> GB 50.00%  170141183460469231731687303715884105727
>> 
>> 
>> 
>> Yet du . –h reports: only 2.4G in the data directory.
>> 
>> 
>> 
>> After restarting the node, nodetool ring reports a more accurate:
>> 
>> 10.112.27.65datacenter1 rack1   Up Normal  2.35 GB
>> 50.00%  170141183460469231731687303715884105727
>> 
>> 
>> 
>> Again, both compression and levelled compaction have been enabled on all
>> CFs. Is this a known issue or has anybody else observed a similar pattern?
>> 
>> 
>> 
>> Dan Hendry
>> 
>> (403) 660-2297
>> 
>> 
> 
> 
> 
> -- 
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com

RE: log line question

[Rene Kochen]

Thank you.

Is it also possible to see whether a row is deleted using the Thrift remove 
method. I.e. how can I see in the log whether or not there is a row tombstone 
when retrieving the row?

(I am investigating an issue where sometimes a deleted row resurrects).

Rene 

-Original Message-
From: Sylvain Lebresne [mailto:sylv...@datastax.com] 
Sent: vrijdag 21 oktober 2011 13:16
To: user@cassandra.apache.org
Subject: Re: log line question

On Fri, Oct 21, 2011 at 12:48 PM, Rene Kochen
 wrote:
> Given the following log line:
>
>
>
> DEBUG [ReadStage:20] 11:39:07,028 collecting 0 of 2147483647:
> SuperColumn(2150726f70657274696573 [64617461:false:4@1319189945952058,])
>
>
>
> What does "false:4" in the column mean?

false means the column is live (not a tombstone) and 4 is the length
of the value (the value is not displayed in that log output).

--
Sylvain

>
>
>
> Thanx!
>
>
>
> Rene

Re: Massive writes when only reading from Cassandra

[Jeremiah Jordan]

I could be totally wrong here, but If you are doing a QUORUM read and there is 
a bad value encountered from the QUORUM won't a repair happen?  I thought 
read_repair_chance 0 just means it won't query extra nodes to check for bad 
values.

-Jeremiah

On Oct 17, 2011, at 4:22 PM, Jeremy Hanna wrote:

> Even after disabling hinted handoff and setting read_repair_chance to 0 on 
> all our column families, we were still experiencing massive writes.  
> Apparently the read_repair_chance is completely ignored at any CL higher than 
> CL.ONE.  So we were doing CL.QUORUM on reads and writes and seeing massive 
> writes still.  It was because of the background read repairs being done.  We 
> did extensive logging and checking and that's all it could be as no mutations 
> were coming in via thrift to those column families.
> 
> In any case, just wanted to give some follow-up here as it's been an 
> inexplicable rock in our backpack and hopefully clears up where that setting 
> is actually used.  I'll update the storage configuration wiki to include that 
> caveat as well.
> 
> On Sep 10, 2011, at 5:14 PM, Jeremy Hanna wrote:
> 
>> Thanks for the insights.  I may first try disabling hinted handoff for one 
>> run of our data pipeline and see if it exhibits the same behavior.  Will 
>> post back if I see anything enlightening there.
>> 
>> On Sep 10, 2011, at 5:04 PM, Chris Goffinet wrote:
>> 
>>> You could tail the commit log with `strings` to see what keys are being 
>>> inserted.
>>> 
>>> On Sat, Sep 10, 2011 at 2:24 PM, Jonathan Ellis  wrote:
>>> Two possibilities:
>>> 
>>> 1) Hinted handoff (this will show up in the logs on the sending
>>> machine, on the receiving one it will just look like any other write)
>>> 
>>> 2) You have something doing writes that you're not aware of, I guess
>>> you could track that down using wireshark to see where the write
>>> messages are coming from
>>> 
>>> On Sat, Sep 10, 2011 at 3:56 PM, Jeremy Hanna
>>>  wrote:
 Oh and we're running 0.8.4 and the RF is 3.
 
 On Sep 10, 2011, at 3:49 PM, Jeremy Hanna wrote:
 
> In addition, the mutation stage and the read stage are backed up like:
> 
> Pool NameActive   Pending   Blocked
> ReadStage32   773 0
> RequestResponseStage  0 0 0
> ReadRepairStage   0 0 0
> MutationStage   158525918 0
> ReplicateOnWriteStage 0 0 0
> GossipStage   0 0 0
> AntiEntropyStage  0 0 0
> MigrationStage0 0 0
> StreamStage   0 0 0
> MemtablePostFlusher   1 5 0
> FILEUTILS-DELETE-POOL 0 0 0
> FlushWriter   2 5 0
> MiscStage 0 0 0
> FlushSorter   0 0 0
> InternalResponseStage 0 0 0
> HintedHandoff 0 0 0
> CompactionManager   n/a29
> MessagingServicen/a  0,34
> 
> On Sep 10, 2011, at 3:38 PM, Jeremy Hanna wrote:
> 
>> We are experiencing massive writes to column families when only doing 
>> reads from Cassandra.  A set of 5 hadoop jobs are reading from Cassandra 
>> and then writing out to hdfs.  That is the only thing operating on the 
>> cluster.  We are reading at CL.QUORUM with hadoop and have written with 
>> CL.QUORUM.  Read repair chance is set to 0.0 on all column families.  
>> However, in the logs, I'm seeing flush after flush of memtables and 
>> compactions taking place.  Is there something else that would be writing 
>> based on the above description?
>> 
>> Jeremy
> 
 
 
>>> 
>>> 
>>> 
>>> --
>>> Jonathan Ellis
>>> Project Chair, Apache Cassandra
>>> co-founder of DataStax, the source for professional Cassandra support
>>> http://www.datastax.com
>>> 
>> 
> 

Re: nodetool ring Load column

[Jeremiah Jordan]

Are you using compressed sstables? or the leveled sstables?  Make sure you 
include how you are configured in any JIRA you make, someone else was seeing a 
similar issue with compression turned on.

-Jeremiah

On Oct 14, 2011, at 1:13 PM, Ramesh Natarajan wrote:

> What does the Load column in nodetool ring mean?  From the output
> below it shows 101.62 GB. However if I do a disk usage it is about 6
> GB.
> 
> thanks
> Ramesh
> 
> 
> [root@CAP2-CNode1 cassandra]#
> ~root/apache-cassandra-1.0.0-rc2/bin/nodetool -h localhost ring
> Address DC  RackStatus State   Load
> OwnsToken
> 
>148873535527910577765226390751398592512
> 10.19.102.11datacenter1 rack1   Up Normal  101.62 GB
> 12.50%  0
> 10.19.102.12datacenter1 rack1   Up Normal  84.42 GB
> 12.50%  21267647932558653966460912964485513216
> 10.19.102.13datacenter1 rack1   Up Normal  95.47 GB
> 12.50%  42535295865117307932921825928971026432
> 10.19.102.14datacenter1 rack1   Up Normal  91.25 GB
> 12.50%  63802943797675961899382738893456539648
> 10.19.103.11datacenter1 rack1   Up Normal  93.98 GB
> 12.50%  85070591730234615865843651857942052864
> 10.19.103.12datacenter1 rack1   Up Normal  100.33 GB
> 12.50%  106338239662793269832304564822427566080
> 10.19.103.13datacenter1 rack1   Up Normal  74.1 GB
> 12.50%  127605887595351923798765477786913079296
> 10.19.103.14datacenter1 rack1   Up Normal  93.96 GB
> 12.50%  148873535527910577765226390751398592512
> 
> 
> 
> [root@CAP2-CNode1 cassandra]# du -hs /var/lib/cassandra/data/
> 6.0G/var/lib/cassandra/data/

Re: log line question

[Sylvain Lebresne]

I don't think there is a debug message showing this, but you can easily add one,
just add a log that prints the int and the int deserialized in
ColumnFamilySerializer.java
deserializeFromSSTableNoColumns() method. The int returned is the
local deletion time.
If it's not negative, it basically mean that the row it's
deserializing is a row tombstone.

--
Sylvain

On Fri, Oct 21, 2011 at 1:33 PM, Rene Kochen
 wrote:
> Thank you.
>
> Is it also possible to see whether a row is deleted using the Thrift remove 
> method. I.e. how can I see in the log whether or not there is a row tombstone 
> when retrieving the row?
>
> (I am investigating an issue where sometimes a deleted row resurrects).
>
> Rene
>
> -Original Message-
> From: Sylvain Lebresne [mailto:sylv...@datastax.com]
> Sent: vrijdag 21 oktober 2011 13:16
> To: user@cassandra.apache.org
> Subject: Re: log line question
>
> On Fri, Oct 21, 2011 at 12:48 PM, Rene Kochen
>  wrote:
>> Given the following log line:
>>
>>
>>
>> DEBUG [ReadStage:20] 11:39:07,028 collecting 0 of 2147483647:
>> SuperColumn(2150726f70657274696573 [64617461:false:4@1319189945952058,])
>>
>>
>>
>> What does "false:4" in the column mean?
>
> false means the column is live (not a tombstone) and 4 is the length
> of the value (the value is not displayed in that log output).
>
> --
> Sylvain
>
>>
>>
>>
>> Thanx!
>>
>>
>>
>> Rene
>

Re: Using elasticsearch on cassandra nodes

[Brian O'Neill]

Rock on.  Thanks for the point Aaron.
We're giving this a try right now to index our column families.

cheers,
-brian


On Thu, Oct 20, 2011 at 4:26 PM, aaron morton wrote:

> Solr can use a dynamic schema…
>
>
> https://github.com/apache/lucene-solr/blob/trunk/solr/example/solr/conf/schema.xml#L538
>
> But you may still want to define a schema so you can adjust the index and
> query time processing/typing of the field values.
>
> Cheers
>
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 20/10/2011, at 2:20 AM, Brian O'Neill wrote:
>
> Anthony,
>
> We're in exactly the same boat.  We are waiting on DataStax Enterprise to
> see if it can ease the pain of SOLR schemas.
>
> In the meantime, I just submitted a native REST layer for Cassandra.
> https://issues.apache.org/jira/browse/CASSANDRA-3380
> (Hopefully, it will get integrated soon. Vote it up ;)
>
> With a  simple REST layer, I'm making the case that we can use Cassandra
> just like CouchDB. (so we don't have to deploy both)
> Extending that assertion, I think I could enhance the REST layer to provide
> a stream of changes just like CouchDB does.  Elastic Search could tap into
> that stream as a river.  Just like this…
> http://www.elasticsearch.org/guide/reference/river/couchdb.html
>
> That combination would be pretty powerful.  If we can't get that setup, we
> may fallback to an AOPish strategy as well.
>
> Definitely let me know where you end up.   I'll share our findings as well.
>
> cheers,
> -brian
>
> 
> Brian O'Neill
> Lead Architect, Software Development
> Health Market Science | 2700 Horizon Drive | King of Prussia, PA 19406
> p: 215.588.6024
> blog: http://weblogs.java.net/blog/boneill42/
> blog: http://brianoneill.blogspot.com/
>
>
>
> From: Anthony Ikeda 
> Reply-To: 
> Date: Tue, 18 Oct 2011 14:18:17 -0700
> To: 
> Subject: Re: Using elasticsearch on cassandra nodes
>
> At the moment we are only prototyping so we haven't bridged the two at all.
> We had planned on creating a write-through operation that allowed us to
> filter the calls (AOP perhaps?) to manage the indexing as we stored it in
> Cassandra.
>
> We are still trying to work out if we go the elastic search route or not as
> DataStax will be releasing DataStax Enterprise 2.0 early next year with Solr
> built in and as you said the index schemas seem to be difficult to deal with
> - I really don't want to have to configure Solr, the no schema approach
> sounds much faster to get up and running.
>
> Anthony
>
>
> On Tue, Oct 18, 2011 at 6:14 AM, Brian O'Neill wrote:
>
>> Anthony,
>>
>> We've been looking at elastic search as well.  Presently we have SOLR in
>> place, but it is cumbersome dealing with SOLR schemas when indexing
>> information out of Cassandra (since you can't anticipate all the columns
>> ahead of time).
>>
>> What are you using as your bridge between Cassandra and ES?  Are you
>> developing a Cassandra river?
>>
>> -brian
>>
>>
>>
>>
>> On Mon, Oct 17, 2011 at 5:29 PM, Anthony Ikeda <
>> anthony.ikeda@gmail.com> wrote:
>>
>>> I've already posted to the elasticsearch groups and thought it prudent to
>>> also ask here.
>>>
>>> We are looking at using elastic search to index our data that
>>> we currently store to Cassandra. I was wondering if there are any concerns
>>> running elastic search on the same nodes that we use for Cassandra? We have
>>> a ring of 6 nodes (2 DCs each with 3 nodes) I was thinking of installing
>>> elastic search on 2 nodes in each datacentre - maybe all three. The only
>>> reason I'd use the same infrastructure would be because we have the
>>> distributed visibility already in place.
>>>
>>> Has anyone else taken this approach? Pros? Cons?
>>>
>>> Anthony
>>>
>>>
>>
>>
>> --
>> Brian ONeill
>> Lead Architect, Health Market Science (http://healthmarketscience.com)
>> mobile:215.588.6024
>> blog: http://weblogs.java.net/blog/boneill42/
>> blog: http://brianoneill.blogspot.com/
>>
>>
>
>


-- 
Brian ONeill
Lead Architect, Health Market Science (http://healthmarketscience.com)
mobile:215.588.6024
blog: http://weblogs.java.net/blog/boneill42/
blog: http://brianoneill.blogspot.com/

Authentication setup

[Alexander Konotop]

Hello :-)
Does anyone have a working config with normal secure authentication?
I've just installed Cassandra 1.0.0 and see that SimpleAuthenticate is
meant to be non-secure and was moved to examples. I need a production
config - so I've tried to write this to config:

authenticator: org.apache.cassandra.auth.AuthenticatedUser
authority: org.apache.cassandra.auth.AuthenticatedUser

But during cassandra startup log says:

org.apache.cassandra.config.ConfigurationException: No default
constructor for authenticator class
'org.apache.cassandra.auth.AuthenticatedUser'.


As I understand either AuthenticatedUser is a wrong class or I simply
don't know how to set it up - does it need additional configs similar to
access.properties or passwd.properties? Maybe there's a way to store
users in cassandra DB itself, like, fore example, MySQL does?

I've searched and tried lot of things the whole day but the only info
that I found were two phrases - first told that SimpleAuth is just a
toy and second told to look into source to look for more auth methods.
But, for example, this:

package org.apache.cassandra.auth;

import java.util.Collections;
import java.util.Set;

/**
 * An authenticated user and her groups.
 */
public class AuthenticatedUser
{
public final String username;
public final Set groups;

public AuthenticatedUser(String username)
{
this.username = username;
this.groups = Collections.emptySet();
}

public AuthenticatedUser(String username, Set groups)
{
this.username = username;
this.groups = Collections.unmodifiableSet(groups);
}

@Override
public String toString()
{
return String.format("#", username, groups);
}
}

tells me just about nothing :-(

Best regards
Alexander

Re: Cassandra 1.0.0 - Node Load Bug

[Jonathan Ellis]

You're right, that is in 1.0.0.  Don't know what the OP is seeing, then.

On Fri, Oct 21, 2011 at 6:32 AM, Jeremiah Jordan
 wrote:
> I thought this patch made it into the 1.0 release?  I remember it being 
> referenced in one of the re-rolls.
>
>
> On Oct 20, 2011, at 9:56 PM, "Jonathan Ellis"  wrote:
>
>> That looks to me like it's reporting uncompressed size as the load.
>> Should be fixed in the 1.0 branch for 1.0.1.
>> (https://issues.apache.org/jira/browse/CASSANDRA-3338)
>>
>> On Thu, Oct 20, 2011 at 11:53 AM, Dan Hendry  
>> wrote:
>>> I have been playing around with Cassandra 1.0.0 in our test environment it
>>> seems pretty sweet so far. I have however come across what appears to be a
>>> bug tracking node load. I have enabled compression and levelled compaction
>>> on all CFs (scrub  + snapshot deletion) and the nodes have been operating
>>> normally for a day or two. I started getting concerned when the load as
>>> reported by nodetool ring kept increasing (it seems monotonically) despite
>>> seeing a compression ratio of ~2.5x (as a side note, I find it strange
>>> Cassandra does not provide the compression ratio via jmx or in the logs). I
>>> initially thought there might be a bug in cleaning up obsolete SSTables but
>>> I then noticed the following discrepancy:
>>>
>>>
>>>
>>> Nodetool ring reports:
>>>
>>>                 10.112.27.65    datacenter1 rack1       Up     Normal  8.64
>>> GB         50.00%  170141183460469231731687303715884105727
>>>
>>>
>>>
>>> Yet du . –h reports: only 2.4G in the data directory.
>>>
>>>
>>>
>>> After restarting the node, nodetool ring reports a more accurate:
>>>
>>> 10.112.27.65    datacenter1 rack1       Up     Normal  2.35 GB
>>> 50.00%  170141183460469231731687303715884105727
>>>
>>>
>>>
>>> Again, both compression and levelled compaction have been enabled on all
>>> CFs. Is this a known issue or has anybody else observed a similar pattern?
>>>
>>>
>>>
>>> Dan Hendry
>>>
>>> (403) 660-2297
>>>
>>>
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Massive writes when only reading from Cassandra

[Jonathan Ellis]

Correct.

On Fri, Oct 21, 2011 at 6:47 AM, Jeremiah Jordan
 wrote:
> I could be totally wrong here, but If you are doing a QUORUM read and there 
> is a bad value encountered from the QUORUM won't a repair happen?  I thought 
> read_repair_chance 0 just means it won't query extra nodes to check for bad 
> values.
>
> -Jeremiah
>
> On Oct 17, 2011, at 4:22 PM, Jeremy Hanna wrote:
>
>> Even after disabling hinted handoff and setting read_repair_chance to 0 on 
>> all our column families, we were still experiencing massive writes.  
>> Apparently the read_repair_chance is completely ignored at any CL higher 
>> than CL.ONE.  So we were doing CL.QUORUM on reads and writes and seeing 
>> massive writes still.  It was because of the background read repairs being 
>> done.  We did extensive logging and checking and that's all it could be as 
>> no mutations were coming in via thrift to those column families.
>>
>> In any case, just wanted to give some follow-up here as it's been an 
>> inexplicable rock in our backpack and hopefully clears up where that setting 
>> is actually used.  I'll update the storage configuration wiki to include 
>> that caveat as well.
>>
>> On Sep 10, 2011, at 5:14 PM, Jeremy Hanna wrote:
>>
>>> Thanks for the insights.  I may first try disabling hinted handoff for one 
>>> run of our data pipeline and see if it exhibits the same behavior.  Will 
>>> post back if I see anything enlightening there.
>>>
>>> On Sep 10, 2011, at 5:04 PM, Chris Goffinet wrote:
>>>
 You could tail the commit log with `strings` to see what keys are being 
 inserted.

 On Sat, Sep 10, 2011 at 2:24 PM, Jonathan Ellis  wrote:
 Two possibilities:

 1) Hinted handoff (this will show up in the logs on the sending
 machine, on the receiving one it will just look like any other write)

 2) You have something doing writes that you're not aware of, I guess
 you could track that down using wireshark to see where the write
 messages are coming from

 On Sat, Sep 10, 2011 at 3:56 PM, Jeremy Hanna
  wrote:
> Oh and we're running 0.8.4 and the RF is 3.
>
> On Sep 10, 2011, at 3:49 PM, Jeremy Hanna wrote:
>
>> In addition, the mutation stage and the read stage are backed up like:
>>
>> Pool Name                    Active   Pending   Blocked
>> ReadStage                        32       773         0
>> RequestResponseStage              0         0         0
>> ReadRepairStage                   0         0         0
>> MutationStage                   158    525918         0
>> ReplicateOnWriteStage             0         0         0
>> GossipStage                       0         0         0
>> AntiEntropyStage                  0         0         0
>> MigrationStage                    0         0         0
>> StreamStage                       0         0         0
>> MemtablePostFlusher               1         5         0
>> FILEUTILS-DELETE-POOL             0         0         0
>> FlushWriter                       2         5         0
>> MiscStage                         0         0         0
>> FlushSorter                       0         0         0
>> InternalResponseStage             0         0         0
>> HintedHandoff                     0         0         0
>> CompactionManager               n/a        29
>> MessagingService                n/a      0,34
>>
>> On Sep 10, 2011, at 3:38 PM, Jeremy Hanna wrote:
>>
>>> We are experiencing massive writes to column families when only doing 
>>> reads from Cassandra.  A set of 5 hadoop jobs are reading from 
>>> Cassandra and then writing out to hdfs.  That is the only thing 
>>> operating on the cluster.  We are reading at CL.QUORUM with hadoop and 
>>> have written with CL.QUORUM.  Read repair chance is set to 0.0 on all 
>>> column families.  However, in the logs, I'm seeing flush after flush of 
>>> memtables and compactions taking place.  Is there something else that 
>>> would be writing based on the above description?
>>>
>>> Jeremy
>>
>
>



 --
 Jonathan Ellis
 Project Chair, Apache Cassandra
 co-founder of DataStax, the source for professional Cassandra support
 http://www.datastax.com

>>>
>>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: ConfigurationException: Nonsensical empty parameter list for CompositeType

[Vitaly Vengrov]

Thanks a lot !

Vitaly

On Fri, Oct 21, 2011 at 1:48 AM, aaron morton wrote:

> See http://www.mail-archive.com/user@cassandra.apache.org/msg18132.html
>
> https://issues.apache.org/jira/browse/CASSANDRA-3391
>
> Cheers
>
>   -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 21/10/2011, at 5:31 AM, Vitaly Vengrov wrote:
>
> Hi all.
>
> I am using cassandra 1.0.0.
> I created a keyspace with all the column family definitions at runtime and
> it works fine until I stop and then restart the cassandra server.
> During it's startup I see this error in the cassandra log :
>
> ERROR 16:22:16,977 Exception encountered during startup
> java.lang.RuntimeException: Could not inflate CFMetaData for {"keyspace":
> "V360HC", "name": "Val1H", "column_type": "Standard", "comparator_type":
> "org.apache.cassandra.db.marshal.ReversedType(org.apache.cassandra.db.marshal.LongType)",
> "subcomparator_type": null, "comment": "", "row_cache_size": 0.0,
> "key_cache_size": 20.0, "read_repair_chance": 1.0, "replicate_on_write":
> true, "gc_grace_seconds": 864000, "default_validation_class":
> "org.apache.cassandra.db.marshal.BytesType", "key_validation_class":
> "org.apache.cassandra.db.marshal.CompositeType", "min_compaction_threshold":
> 4, "max_compaction_threshold": 32, "row_cache_save_period_in_seconds": 0,
> "key_cache_save_period_in_seconds": 14400, "row_cache_keys_to_save":
> 2147483647, "merge_shards_chance": 0.1, "id": 1087, "column_metadata": [],
> "row_cache_provider": "org.apache.cassandra.cache.SerializingCacheProvider",
> "key_alias": null, "compaction_strategy":
> "org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy",
> "compaction_strategy_options": {}, "compression_options": {}}
>  at org.apache.cassandra.config.CFMetaData.fromAvro(CFMetaData.java:350)
> at org.apache.cassandra.config.KSMetaData.fromAvro(KSMetaData.java:180)
>  at org.apache.cassandra.db.DefsTable.loadFromStorage(DefsTable.java:99)
> at
> org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:508)
>  at
> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:161)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:337)
>  at
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:106)
> Caused by: org.apache.cassandra.config.ConfigurationException: Invalid
> definition for comparator org.apache.cassandra.db.marshal.CompositeType.
>  at
> org.apache.cassandra.db.marshal.TypeParser.getRawAbstractType(TypeParser.java:319)
> at
> org.apache.cassandra.db.marshal.TypeParser.getAbstractType(TypeParser.java:247)
>  at org.apache.cassandra.db.marshal.TypeParser.parse(TypeParser.java:83)
> at org.apache.cassandra.db.marshal.TypeParser.parse(TypeParser.java:92)
>  at org.apache.cassandra.config.CFMetaData.fromAvro(CFMetaData.java:346)
> ... 6 more
> Caused by: org.apache.cassandra.config.ConfigurationException: Nonsensical
> empty parameter list for CompositeType
> at
> org.apache.cassandra.db.marshal.CompositeType.getInstance(CompositeType.java:67)
>  at
> org.apache.cassandra.db.marshal.CompositeType.getInstance(CompositeType.java:61)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.cassandra.db.marshal.TypeParser.getRawAbstractType(TypeParser.java:307)
>  ... 10 more
>
> Please advise
>
> Thanks
>
> Vitaly
>
>
>

Re: Cassandra 1.0: Exception in compaction

[Ramesh Natarajan]

i am using size based compaction ( the default one ). Also this is on linux.

thanks
Ramesh

On Fri, Oct 21, 2011 at 4:25 AM, Sylvain Lebresne  wrote:
> I believe this is the same as
> https://issues.apache.org/jira/browse/CASSANDRA-3306.
>
> The initial reporter only got this exception with leveled compaction,
> is it what you are
> using too (to help narrow it down)? Also, are you using windows by any chance?
>
> --
> Sylvain
>
>
> On Thu, Oct 20, 2011 at 9:04 PM, Ramesh Natarajan  wrote:
>> We are running a 8 node cassandra 1.0 cluster. We are seeing this
>> exception quite often. Any idea how to debug this issue?
>>
>>
>> java.lang.IllegalArgumentException: Illegal Capacity: -2
>>        at java.util.ArrayList.(ArrayList.java:110)
>>        at 
>> org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>>        at 
>> org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>        at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>        at 
>> org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>        at 
>> org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>        at 
>> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>        at 
>> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>        at 
>> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>        at java.lang.Thread.run(Thread.java:662)
>>
>>
>>
>> few lines before this error
>>
>>  INFO [FlushWriter:222] 2011-10-20 10:52:25,885 Memtable.java (line
>> 237) Writing Memtable-participants@1907757288(6777526/153164388
>> serialized/live bytes, 199339 ops)
>>  INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line
>> 488) Discarding obsolete commit
>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319115938691.log)
>>  INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line
>> 488) Discarding obsolete commit
>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319122061956.log)
>>  INFO [FlushWriter:222] 2011-10-20 10:52:26,865 Memtable.java (line
>> 273) Completed flushing
>> /var/lib/cassandra/data/MSA/participants-h-87-Data.db (14695382 bytes)
>>  INFO [FlushWriter:222] 2011-10-20 10:52:26,866 Memtable.java (line
>> 237) Writing Memtable-modseq@1745889706(13206/311769 serialized/live
>> bytes, 426 ops)
>>  INFO [FlushWriter:222] 2011-10-20 10:52:26,896 Memtable.java (line
>> 273) Completed flushing
>> /var/lib/cassandra/data/MSA/modseq-h-262-Data.db (38646 bytes)
>>  INFO [FlushWriter:222] 2011-10-20 10:52:26,897 Memtable.java (line
>> 237) Writing Memtable-msgid@2099219781(3571249/77183008
>> serialized/live bytes, 109823 ops)
>>  INFO [FlushWriter:222] 2011-10-20 10:52:27,497 Memtable.java (line
>> 273) Completed flushing /var/lib/cassandra/data/MSA/msgid-h-47-Data.db
>> (8125165 bytes)
>>  INFO [FlushWriter:222] 2011-10-20 10:52:27,498 Memtable.java (line
>> 237) Writing Memtable-uid@578022704(43734344/317200563 serialized/live
>> bytes, 611301 ops)
>>  INFO [FlushWriter:222] 2011-10-20 10:52:29,802 Memtable.java (line
>> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-291-Data.db
>> (48225128 bytes)
>>  INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line
>> 488) Discarding obsolete commit
>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125356477.log)
>>  INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line
>> 488) Discarding obsolete commit
>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125683351.log)
>>  INFO [MutationStage:88] 2011-10-20 10:52:29,905
>> ColumnFamilyStore.java (line 664) Enqueuing flush of
>> Memtable-modseq@339630706(155217/3664394 serialized/live bytes, 5007
>> ops)
>>  INFO [FlushWriter:222] 2011-10-20 10:52:29,905 Memtable.java (line
>> 237) Writing Memtable-modseq@339630706(155217/3664394 serialized/live
>> bytes, 5007 ops)
>>  INFO [FlushWriter:222] 2011-10-20 10:52:30,216 Memtable.java (line
>> 273) Completed flushing
>> /var/lib/cassandra/data/MSA/modseq-h-263-Data.db (450477 bytes)
>> ERROR [CompactionExecutor:538] 2011-10-20 10:52:36,132
>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>> Thread[CompactionExecutor:538,1,main]
>>
>>
>>
>> Another one
>>
>>
>>
>>  INFO [FlushWriter:222] 2011-10-20 10:52:39,623 Memtable.java (line
>> 237) Writing Memtable-uid@2018688194(79740/578345 serialized/live
>> bytes,  ops)
>>  INFO [FlushWriter:222] 2011-10-20 10:52:39,777 M

Re: nodetool ring Load column

[Ramesh Natarajan]

I don't use compressed sstable.  I also use the default compaction
strategy. i will look at JIRA and see if there are any similarities.

thanks
Ramesh

On Fri, Oct 21, 2011 at 6:51 AM, Jeremiah Jordan
 wrote:
> Are you using compressed sstables? or the leveled sstables?  Make sure you 
> include how you are configured in any JIRA you make, someone else was seeing 
> a similar issue with compression turned on.
>
> -Jeremiah
>
> On Oct 14, 2011, at 1:13 PM, Ramesh Natarajan wrote:
>
>> What does the Load column in nodetool ring mean?  From the output
>> below it shows 101.62 GB. However if I do a disk usage it is about 6
>> GB.
>>
>> thanks
>> Ramesh
>>
>>
>> [root@CAP2-CNode1 cassandra]#
>> ~root/apache-cassandra-1.0.0-rc2/bin/nodetool -h localhost ring
>> Address         DC          Rack        Status State   Load
>> Owns    Token
>>
>>        148873535527910577765226390751398592512
>> 10.19.102.11    datacenter1 rack1       Up     Normal  101.62 GB
>> 12.50%  0
>> 10.19.102.12    datacenter1 rack1       Up     Normal  84.42 GB
>> 12.50%  21267647932558653966460912964485513216
>> 10.19.102.13    datacenter1 rack1       Up     Normal  95.47 GB
>> 12.50%  42535295865117307932921825928971026432
>> 10.19.102.14    datacenter1 rack1       Up     Normal  91.25 GB
>> 12.50%  63802943797675961899382738893456539648
>> 10.19.103.11    datacenter1 rack1       Up     Normal  93.98 GB
>> 12.50%  85070591730234615865843651857942052864
>> 10.19.103.12    datacenter1 rack1       Up     Normal  100.33 GB
>> 12.50%  106338239662793269832304564822427566080
>> 10.19.103.13    datacenter1 rack1       Up     Normal  74.1 GB
>> 12.50%  127605887595351923798765477786913079296
>> 10.19.103.14    datacenter1 rack1       Up     Normal  93.96 GB
>> 12.50%  148873535527910577765226390751398592512
>>
>>
>>
>> [root@CAP2-CNode1 cassandra]# du -hs /var/lib/cassandra/data/
>> 6.0G    /var/lib/cassandra/data/
>
>

Re: weird problem with performance

[Yang]

actually this is only an issue in HH, since HH writes all the stored
messages into the same row, so locking is a problem

2011/10/21 Jérémy SEVELLEC :
> @Araron you're right and i was wrong!
>
> 2011/10/20 Yang 
>>
>> found it , https://issues.apache.org/jira/browse/CASSANDRA-3387
>>
>> On Thu, Oct 20, 2011 at 1:37 PM, aaron morton 
>> wrote:
>> > It's unlikely that HH is the issue. (Disclaimer, am not familiar with HH
>> > in
>> > 1.0, i know it's changes a bit)
>> > Take a look at the TP Stats, what's happening ?
>> > Cheers
>> > -
>> > Aaron Morton
>> > Freelance Developer
>> > @aaronmorton
>> > http://www.thelastpickle.com
>> > On 20/10/2011, at 10:10 AM, Jérémy SEVELLEC wrote:
>> >
>> > Ok.
>> > I think a degration could be normal because your cluster is in a
>> > degraded
>> > state when a node is down.
>> > With a replication_factor of 3 and with a 3 nodes cluster, each data you
>> > write is replicated on each node. As One node is down, when writing,
>> > it's
>> > impossible to send a replica on the down node and hints are send
>> >
>> > : http://www.datastax.com/docs/1.0/dml/about_writes#hinted-handoff-writes
>> > And it could be more expensive to achieve QUORUM when you read in that
>> > context.
>> > It may be one explanation. You can turn cassandra log into debug level
>> > to
>> > see what happen when when there is a down node.
>> > 2011/10/19 Yang 
>> >>
>> >> 3
>> >>
>> >> sorry forgot this important info
>> >>
>> >> On Oct 19, 2011 11:31 AM, "Jérémy SEVELLEC" 
>> >> wrote:
>> >>>
>> >>> Hi, what is your replication_factor?
>> >>>
>> >>> 2011/10/19 Yang 
>> 
>>  I'm using a cassandra version compiled from 1.0.0  github HEAD.
>> 
>>  I have 3 nodes, A B and C,  on node A I run a client, which talks
>>  only
>>  to B as the coordinator.
>> 
>>  the performance is pretty good, a QUORUM read+write  takes < 10ms.
>> 
>>  but then I shutdown C, quickly the performance starts to degrade, and
>>  QUORUM read+write time steadily increase to about 300ms.
>> 
>>  if I shutdown A and keep C, I observe the same effect.
>> 
>> 
>>  I understand that "2 out of 3" is going to give you faster response
>>  than "2 out of 2", but  the difference should not be that dramatic as
>>  10ms vs 300ms.
>> 
>>  any possible reasons for this?(or how to debug this?)
>> 
>>  Thanks
>>  Yang
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Jérémy
>> >
>> >
>> >
>> > --
>> > Jérémy
>> >
>> >
>
>
>
> --
> Jérémy
>

Re: Storing pre-sorted data

[Stephen Connolly]

Well you could use a DOUBLE value to encode relative positions...

first item in list gets key 1.0
insert before first item -> key[first]/2.0;
append after last item -> key[last]*2.0;
insert after non-final item -> (key[n]+key[n+1])/2.0

Using double precision should give you quite a space to fit items...

you should be able to cleanly do 2^10 appends, or 2^10 insert first
before hitting the significands... and even then you have 2^51 of
those...

If you start to hit issues like heavy segments, you can re-normalize the row.

-Stephen

On 17 October 2011 10:43, Matthias Pfau  wrote:
> Thanks for that hint! However, it seems like soundex is a very language
> specific algorithm (US English). We have to get into this topic further...
>
> Kind regards
> Matthias
>
> On 10/13/2011 10:43 PM, Stephen Connolly wrote:
>>
>> Then just use a soundex function on the first word in the text... that
>> will shrink it sufficiently and give nice buckets in near sequential
>> order (http://en.wikipedia.org/wiki/Soundex)
>>
>> On 13 October 2011 21:21, Matthias Pfau  wrote:
>>>
>>> Hi Stephen,
>>> we are hashing the first 8 byte (8 US-ASCII characters) of text that has
>>> been written by humans. Wouldn't it be easy for the attacker to do a
>>> dictionary attack on this text, especially if he knows the language of
>>> the
>>> text?
>>>
>>> Kind regards
>>> Matthias
>>>
>>> On 10/13/2011 08:20 PM, Stephen Connolly wrote:

 in theory, however they have less than 32 bits of entropy from which
 they can do that, leaving them with at least 32 more bits of
 combinations to try... that's 2 billion or so... must be a big
 dictionary

 - Stephen

 ---
 Sent from my Android phone, so random spelling mistakes, random nonsense
 words and other nonsense are a direct result of using swype to type on
 the screen

 On 13 Oct 2011 17:57, "Matthias Pfau"mailto:p...@l3s.de>>
 wrote:

    Hi Stephen,
    this sounds very reasonable. But wouldn't this enable an attacker to
    execute dictionary attacks in order to "decrypt" the first 8 bytes
    of the plain text?

    Kind regards
    Matthias

    On 10/13/2011 05:03 PM, Stephen Connolly wrote:

        It wouldn't be unencrypted... which is the point

        you use a one way linear hash function to take the first, say 8
        bytes,
        of unencrypted data and turn it into 4 bytes of a sort prefix.

        You've used lost half the data in the process, so effectively
        each bit
        is an OR of two bits and you can only infer from 0 values... so
 data
        is still encrypted, but you have an approximate sorting.

        For example, if your data is US-ASCII text with no numbers, you
        could
        use Soundex to get the pre-key, so that worst case you have a
 bucket
        of values in the range.

        Using this technique, a random get will have to get the values
        at the
        desired prefix +/- a small amount rather than the whole row...
        on the
        client side you can then decrypt the data and sort that small
 bucket
        to get the correct index position.

        You could do a 1 byte prefix, but that only gives you at best 256
        buckets and assumes that the first 2 bytes are uniformly
        distributed... you've said your data is not uniformly
        distributed, so
        a linear hash function sounds like your best bet.

        your hash function should have the property that hash(A)>=
        hash(B) if
        and only if A>= B

        On 13 October 2011 08:47, Matthias Pfau>>>        >    wrote:

            Hi Stephen,
            this is a great idea but unfortunately doesn't work for us
            either as we can
            not store the data in an unencrypted form.

            Kind regards
            Matthias

            On 10/12/2011 07:42 PM, Stephen Connolly wrote:


                could you prefix the data with 3-4 bytes of a linear
                hash of the
                unencypted data? it wouldn't be a perfect sort, but
                you'd have less of a
                range to query to get the sorted values?

                - Stephen

                ---
                Sent from my Android phone, so random spelling mistakes,
                random nonsense
                words and other nonsense are a direct result of using
                swype to type on
                the screen

                On 12 Oct 2011 17:57, "Matthias Pfau">>>                >>
                wrote:

                    Unfortunately, th

Re: Cassandra 1.0: Exception in compaction

[Sylvain Lebresne]

Would you have the full log for one of those node leading to the exception
that you could share? Not sure that'll help but who knows.

--
Sylvain

On Fri, Oct 21, 2011 at 4:34 PM, Ramesh Natarajan  wrote:
> i am using size based compaction ( the default one ). Also this is on linux.
>
> thanks
> Ramesh
>
> On Fri, Oct 21, 2011 at 4:25 AM, Sylvain Lebresne  
> wrote:
>> I believe this is the same as
>> https://issues.apache.org/jira/browse/CASSANDRA-3306.
>>
>> The initial reporter only got this exception with leveled compaction,
>> is it what you are
>> using too (to help narrow it down)? Also, are you using windows by any 
>> chance?
>>
>> --
>> Sylvain
>>
>>
>> On Thu, Oct 20, 2011 at 9:04 PM, Ramesh Natarajan  wrote:
>>> We are running a 8 node cassandra 1.0 cluster. We are seeing this
>>> exception quite often. Any idea how to debug this issue?
>>>
>>>
>>> java.lang.IllegalArgumentException: Illegal Capacity: -2
>>>        at java.util.ArrayList.(ArrayList.java:110)
>>>        at 
>>> org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>>>        at 
>>> org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>        at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>        at 
>>> org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>        at 
>>> org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>        at 
>>> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>        at 
>>> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>        at 
>>> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>        at 
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>        at 
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>        at java.lang.Thread.run(Thread.java:662)
>>>
>>>
>>>
>>> few lines before this error
>>>
>>>  INFO [FlushWriter:222] 2011-10-20 10:52:25,885 Memtable.java (line
>>> 237) Writing Memtable-participants@1907757288(6777526/153164388
>>> serialized/live bytes, 199339 ops)
>>>  INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line
>>> 488) Discarding obsolete commit
>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319115938691.log)
>>>  INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line
>>> 488) Discarding obsolete commit
>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319122061956.log)
>>>  INFO [FlushWriter:222] 2011-10-20 10:52:26,865 Memtable.java (line
>>> 273) Completed flushing
>>> /var/lib/cassandra/data/MSA/participants-h-87-Data.db (14695382 bytes)
>>>  INFO [FlushWriter:222] 2011-10-20 10:52:26,866 Memtable.java (line
>>> 237) Writing Memtable-modseq@1745889706(13206/311769 serialized/live
>>> bytes, 426 ops)
>>>  INFO [FlushWriter:222] 2011-10-20 10:52:26,896 Memtable.java (line
>>> 273) Completed flushing
>>> /var/lib/cassandra/data/MSA/modseq-h-262-Data.db (38646 bytes)
>>>  INFO [FlushWriter:222] 2011-10-20 10:52:26,897 Memtable.java (line
>>> 237) Writing Memtable-msgid@2099219781(3571249/77183008
>>> serialized/live bytes, 109823 ops)
>>>  INFO [FlushWriter:222] 2011-10-20 10:52:27,497 Memtable.java (line
>>> 273) Completed flushing /var/lib/cassandra/data/MSA/msgid-h-47-Data.db
>>> (8125165 bytes)
>>>  INFO [FlushWriter:222] 2011-10-20 10:52:27,498 Memtable.java (line
>>> 237) Writing Memtable-uid@578022704(43734344/317200563 serialized/live
>>> bytes, 611301 ops)
>>>  INFO [FlushWriter:222] 2011-10-20 10:52:29,802 Memtable.java (line
>>> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-291-Data.db
>>> (48225128 bytes)
>>>  INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line
>>> 488) Discarding obsolete commit
>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125356477.log)
>>>  INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line
>>> 488) Discarding obsolete commit
>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125683351.log)
>>>  INFO [MutationStage:88] 2011-10-20 10:52:29,905
>>> ColumnFamilyStore.java (line 664) Enqueuing flush of
>>> Memtable-modseq@339630706(155217/3664394 serialized/live bytes, 5007
>>> ops)
>>>  INFO [FlushWriter:222] 2011-10-20 10:52:29,905 Memtable.java (line
>>> 237) Writing Memtable-modseq@339630706(155217/3664394 serialized/live
>>> bytes, 5007 ops)
>>>  INFO [FlushWriter:222] 2011-10-20 10:52:30,216 Memtable.java (line
>>> 273) Completed flushing
>>> /var/lib/cassandra/data/MSA/modseq-h-263-Data.db (450477 bytes)
>>> ERROR [CompactionExecutor:538] 2011-10-20 10:52:36,132
>>> AbstractCassandraDaemon.java (line 13

Specific Question, General Problem

[Ian Danforth]

All,

 I have a specific question which I think highlights a general problem.

===Specific Question===

I'm seeing read times of 2-300ms for getting a single row. This seems slow,
but is it unusual?

Stack

5 node cluster
Version .86
EC2 m1large machines
ebs drives for all data (I know, I know)

Datamodel

Millions of rows that are at most 1440 columns wide. Each column stores a
single int.

===General Problem===

I don't know what 'normal' is in Cassandra. The docs use terms like 'large'
or 'wide' rows, but I don't have any absolute numbers around that adjective.
I don't know if storing millions of rows in 5 nodes is unusual (maybe people
scale out before they get to this size). Etc.

There are plenty of people here who have an idea of what's normal for their
cluster, but only a very few who know what is normal for Cassandra in
general.

I would love, *love*, to have a document that highlighted this.

Heck I'd love to help build a 'performance calculator' in which you could
put in the number of nodes, and it would tell you how much data it would be
reasonable to store. (Yes I know there are a ton of variables involved.)

Thanks for any light that can be shed on my specific question or the general
problem.

Ian