Re: Cassandra 1.0 - is disk seek required to access SSTable metadata

[Sylvain Lebresne]

> Bloom filter is stored in RAM, but what about metadata?
> Is disk seek required to access it?

No, it's loaded in RAM when the sstable is loaded.

--
Sylvain

Re: are asynchronous schema updates possible ?

[Sylvain Lebresne]

To add to Aaron response, you can update a CF concurrently in 1.1
already. However, you cannot create multiple CF concurrently just yet,
but that will be fixed in 1.2.

--
Sylvain

On Sun, Aug 26, 2012 at 11:04 PM, aaron morton  wrote:
> Concurrent schema changes are coming in 1.2.
>
> I could not find a single issue that covered it, that may be my bad search
> fu. The issues for 1.2 are here
> https://issues.apache.org/jira/browse/CASSANDRA/fixforversion/12319262
>
> Cheers
>
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 24/08/2012, at 7:06 PM, Илья Шипицин  wrote:
>
> Hello!
>
> we are looking into concurent schema updates (when multiple instances of
> application create CFs at once.
>
> at the http://wiki.apache.org/cassandra/MultiTenant there's open ticket
> 1391, it is said it is still open.
> however, in jura is said "1.1.0 is fixed"
>
> can schema be updated asynchrously on 1.1.x ? or not ?
> if multiple server create the same CF ?
>
> Cheers,
> Ilya Shipitsin
>
>

java.lang.NoClassDefFoundError when trying to do anything on one CF on one node

[Thomas van Neerijnen]

Hi

I have a single node in a 6 node Cassandra 1.0.11 cluster that seems to
have a single column family in a weird state.

Repairs, upgradesstables, anything that touches this CF crashes.
I've drained the node, removed every file for this CF from said node,
removed the commit log, started it up and as soon as data is written to
this CF on this node I'm in the same situation again. Anyone have any
suggestions for how to fix this?
I'm tempted to remove the node and re-add it but I was hoping for something
a little less disruptive.

$ nodetool -h localhost upgradesstables Player PlayerCounters
Error occured while upgrading the sstables for keyspace Player
java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError:
Could not initialize class org.apache.cassandra.utils.NodeId$LocalIds
at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:219)
at
org.apache.cassandra.db.compaction.CompactionManager.performSSTableRewrite(CompactionManager.java:235)
at
org.apache.cassandra.db.ColumnFamilyStore.sstablesRewrite(ColumnFamilyStore.java:999)
at
org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:1652)
at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at
com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
at
com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
at
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class
org.apache.cassandra.utils.NodeId$LocalIds
at org.apache.cassandra.utils.NodeId.localIds(NodeId.java:49)
at
org.apache.cassandra.utils.NodeId.getOldLocalNodeIds(NodeId.java:79)
at
org.apache.cassandra.db.CounterColumn.computeOldShardMerger(CounterColumn.java:251)
at
org.apache.cassandra.db.CounterColumn.mergeAndRemoveOldShards(CounterColumn.java:297)
at
org.apache.cassandra.db.CounterColumn.mergeAndRemoveOldShards(CounterColumn.java:271)
at
org.apache.cassandra.db.compaction.PrecompactedRow.removeDeletedAndOldShards(PrecompactedRow.java:81)
at
org.apache.cassandra.db.compaction.PrecompactedRow.(PrecompactedRow.java:97)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:137)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:97)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:82)
at
org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:207)
at
com.google.common.collect.A

Re: Cassandra API Library.

[Filipe Gonçalves]

@Brian: you can add the Cassandra::Simple Perl client
http://fmgoncalves.github.com/p5-cassandra-simple/

2012/8/27 Paolo Bernardi 

> On 08/23/2012 01:40 PM, Thomas Spengler wrote:
>
>> 4) pelops (Thrift,Java)
>>
>>
>>  I've been using Pelops for quite some time with pretty good results; it
> felt much cleaner than Hector.
>
> Paolo
>
> --
> @bernarpa
> http://paolobernardi.**wordpress.com 
>
>


-- 
Filipe Gonçalves

Re: Cassandra and Apache Drill

[Hiller, Dean]

Many queries on small portion of the data….sounds like playORM ;).

As long as you partition your data with playOrm, you can do really fast queries 
into that data by partition using Scalabla SQL (SQL with the addition of a 
partition clause in front as to what partitions you are querying).  Joins are 
currently being added as well and should be just as fast as an RDBMS join as 
well.

Later,
Dean

From: John Onusko mailto:jonu...@actiance.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Friday, August 31, 2012 2:29 PM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Cassandra and Apache Drill

ts, but the job oriented processing is too slow when you need to execute many 
queries on a small portion of the data. It seems like Apache Drill might be the 
right answer to this problem. I see HBase mentioned as a possible integration 
point with Drill, but no mention of Cassandra. Has anyone taken a look at Dr

Re: performance is drastically degraded after 0.7.8 --> 1.0.11 upgrade

[Илья Шипицин]

it was good idea to have a look at StorageProxy :-)


1.0.10 Performance Tests
StorageProxy

RangeOperations: 546
ReadOperations: 694563
TotalHints: 0
TotalRangeLatencyMicros: 4469484
TotalReadLatencyMicros:245669679
TotalWriteLatencyMicros: 57819722
WriteOperations:208741


0.7.10 Performance Tests
StorageProxy

RangeOperations: 520
ReadOperations: 671476
TotalRangeLatencyMicros: 2208902
TotalReadLatencyMicros: 162186009
TotalWriteLatencyMicros: 33911222
WriteOperations: 204806


2012/9/3 aaron morton 

> The whole test run is taking longer ? So it could be slower queries or
> slower test setup / tear down?
>
> If you are creating and truncate the KS for each of the 500 tests is that
> taking longer ? (Schema code has changed a lot 0.7 > 1.0)
> Can you log the execution time for tests and find ones that are taking
> longer ?
>
> There are full request metrics available on the StorageProxy JMX object.
>
> Cheers
>
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 31/08/2012, at 4:45 PM, Илья Шипицин  wrote:
>
> we are using functional tests ( ~500 tests in time).
> it is hard to tell which query is slower, it is "slower in general".
>
> same hardware. 1 node, 32Gb RAM, 8Gb heap. default cassandra settings.
> as we are talking about functional tests, so we recreate KS just before
> tests are run.
>
> I do not know how to record queries (there are a lot of them), if you are
> interested, I can set up a special stand for you.
>
> 2012/8/31 aaron morton 
>
>> we are running somewhat queue-like with aggressive write-read patterns.
>>
>> We'll need some more details...
>>
>> How much data ?
>> How many machines ?
>> What is the machine spec ?
>> How many clients ?
>> Is there an example of a slow request ?
>> How are you measuring that it's slow ?
>> Is there anything unusual in the log ?
>>
>> Cheers
>>
>>  -
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 31/08/2012, at 3:30 AM, Edward Capriolo  wrote:
>>
>> If you move from 7.X to 0.8X or 1.0X you have to rebuild sstables as
>> soon as possible. If you have large bloomfilters you can hit a bug
>> where the bloom filters will not work properly.
>>
>>
>> On Thu, Aug 30, 2012 at 9:44 AM, Илья Шипицин 
>> wrote:
>>
>> we are running somewhat queue-like with aggressive write-read patterns.
>> I was looking for scripting queries from live Cassandra installation, but
>> I
>> didn't find any.
>>
>> is there something like thrift-proxy or other query logging/scripting
>> engine
>> ?
>>
>> 2012/8/30 aaron morton 
>>
>>
>> in terms of our high-rate write load cassandra1.0.11 is about 3 (three!!)
>> times slower than cassandra-0.7.8
>>
>> We've not had any reports of a performance drop off. All tests so far have
>> show improvements in both read and write performance.
>>
>> I agree, such digests save some network IO, but they seem to be very bad
>> in terms of CPU and disk IO.
>>
>> The sha1 is created so we can diagnose corruptions in the -Data component
>> of the SSTables. They are not used to save network IO.
>> It is calculated while streaming the Memtable to disk so has no impact on
>> disk IO. While not the fasted algorithm I would assume it's CPU overhead
>> in
>> this case is minimal.
>>
>> there's already relatively small Bloom filter file, which can be used for
>> saving network traffic instead of sha1 digest.
>>
>> Bloom filters are used to test if a row key may exist in an SSTable.
>>
>> any explanation ?
>>
>> If you can provide some more information on your use case we may be able
>> to help.
>>
>> Cheers
>>
>>
>> -
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 30/08/2012, at 5:18 AM, Илья Шипицин  wrote:
>>
>> in terms of our high-rate write load cassandra1.0.11 is about 3 (three!!)
>> times slower than cassandra-0.7.8
>> after some investigation carried out I noticed files with "sha1" extension
>> (which are missing for Cassandra-0.7.8)
>>
>> in maybeWriteDigest() function I see no option fot switching sha1 digests
>> off.
>>
>> I agree, such digests save some network IO, but they seem to be very bad
>> in terms of CPU and disk IO.
>> why to use one more digest (which have to be calculated), there's already
>> relatively small Bloom filter file, which can be used for saving network
>> traffic instead of sha1 digest.
>>
>> any explanation ?
>>
>> Ilya Shipitsin
>>
>>
>>
>>
>>
>
>

Re: find when minor compaction happened?

[Satoshi Yamada]

Tyler,
yeah, i found those.thanks for your help.
satoshi

--- On Tue, 2012/9/4, Tyler Hobbs  wrote:

There are several logs associated with each minor compaction.  Grep your logs 
for "Compacting".

On Mon, Sep 3, 2012 at 7:41 AM, Satoshi Yamada  
wrote:

hi,

i know minor compaction is caused when the num of SSTableexceeds the thresholds.
then, is there anyway to find what time minor compaction happened?is minor 
compaction output into log?

thanks,satoshi


-- 
Tyler Hobbs

DataStax

Re: Cassandra API Library.

[Brian O'Neill]

You got it.  (done)

-brian

On Tue, Sep 4, 2012 at 7:08 AM, Filipe Gonçalves
 wrote:
> @Brian: you can add the Cassandra::Simple Perl client
> http://fmgoncalves.github.com/p5-cassandra-simple/
>
>
> 2012/8/27 Paolo Bernardi 
>>
>> On 08/23/2012 01:40 PM, Thomas Spengler wrote:
>>>
>>> 4) pelops (Thrift,Java)
>>>
>>>
>> I've been using Pelops for quite some time with pretty good results; it
>> felt much cleaner than Hector.
>>
>> Paolo
>>
>> --
>> @bernarpa
>> http://paolobernardi.wordpress.com
>>
>
>
>
> --
> Filipe Gonçalves



-- 
Brian ONeill
Lead Architect, Health Market Science (http://healthmarketscience.com)
Apache Cassandra MVP
mobile:215.588.6024
blog: http://brianoneill.blogspot.com/
twitter: @boneill42

Re: Cassandra API Library.

[Nuba Princigalli]

check also http://search.cpan.org/dist/perlcassa/ :)


On Tue, Sep 4, 2012 at 10:05 AM, Brian O'Neill wrote:

> You got it.  (done)
>
> -brian
>
> On Tue, Sep 4, 2012 at 7:08 AM, Filipe Gonçalves
>  wrote:
> > @Brian: you can add the Cassandra::Simple Perl client
> > http://fmgoncalves.github.com/p5-cassandra-simple/
> >
> >
> > 2012/8/27 Paolo Bernardi 
> >>
> >> On 08/23/2012 01:40 PM, Thomas Spengler wrote:
> >>>
> >>> 4) pelops (Thrift,Java)
> >>>
> >>>
> >> I've been using Pelops for quite some time with pretty good results; it
> >> felt much cleaner than Hector.
> >>
> >> Paolo
> >>
> >> --
> >> @bernarpa
> >> http://paolobernardi.wordpress.com
> >>
> >
> >
> >
> > --
> > Filipe Gonçalves
>
>
>
> --
> Brian ONeill
> Lead Architect, Health Market Science (http://healthmarketscience.com)
> Apache Cassandra MVP
> mobile:215.588.6024
> blog: http://brianoneill.blogspot.com/
> twitter: @boneill42
>

Re: Repair has now effect

[Patricio Echagüe]

embarrassing.

Chef somehow ran in that box and updated the schema with a version of it
that had RF=1.

Sorry about that.

On Mon, Sep 3, 2012 at 1:45 AM, Radim Kolar  wrote:

>
>   INFO [AntiEntropySessions:6] 2012-09-02 15:46:23,022
>> AntiEntropyService.java (line 663) [repair #%s] No neighbors to repair with
>> on range %s: session completed
>>
> you have RF=1, or too many nodes are down.
>

anyone know how to lookup non-continguous columns BUT for prefixes?

[Hiller, Dean]

I have a row that is an index like so

Index row -> ., ., ., . 
, ., ., .

I would like to get all of the pks for
   which are pk32 and pk7
And
   which are pk54

This is a trimmed down example of course.  I am thinking maybe I might just use 
the astyanax async to send out 500 requests instead.

Thanks,
Dean

Re: anyone know how to lookup non-continguous columns BUT for prefixes?

[Dave Brosius]

You'd need to make n queries, or do a superset query from min>;-

Re: are asynchronous schema updates possible ?

[Илья Шипицин]

Is it ok multiple servers will create/update the same CF at once ? I'm
looking into dynamic schema update during application deploy/update.

вторник, 4 сентября 2012 г. пользователь Sylvain Lebresne писал:

> To add to Aaron response, you can update a CF concurrently in 1.1
> already. However, you cannot create multiple CF concurrently just yet,
> but that will be fixed in 1.2.
>
> --
> Sylvain
>
> On Sun, Aug 26, 2012 at 11:04 PM, aaron morton 
> >
> wrote:
> > Concurrent schema changes are coming in 1.2.
> >
> > I could not find a single issue that covered it, that may be my bad
> search
> > fu. The issues for 1.2 are here
> > https://issues.apache.org/jira/browse/CASSANDRA/fixforversion/12319262
> >
> > Cheers
> >
> > -
> > Aaron Morton
> > Freelance Developer
> > @aaronmorton
> > http://www.thelastpickle.com
> >
> > On 24/08/2012, at 7:06 PM, Илья Шипицин >
> wrote:
> >
> > Hello!
> >
> > we are looking into concurent schema updates (when multiple instances of
> > application create CFs at once.
> >
> > at the http://wiki.apache.org/cassandra/MultiTenant there's open ticket
> > 1391, it is said it is still open.
> > however, in jura is said "1.1.0 is fixed"
> >
> > can schema be updated asynchrously on 1.1.x ? or not ?
> > if multiple server create the same CF ?
> >
> > Cheers,
> > Ilya Shipitsin
> >
> >
>

Re: are asynchronous schema updates possible ?

[Sylvain Lebresne]

On Tue, Sep 4, 2012 at 8:23 PM, Илья Шипицин  wrote:
> Is it ok multiple servers will create/update the same CF at once ? I'm
> looking into dynamic schema update during application deploy/update.

As said above, it is ok for update the same CF concurrently in 1.1 but
*not* for creation (if you create CF concurrently, whether that is the
same CF or not, you might have problem). The last part will be fixed
in 1.2 however.

--
Sylvain

>
> вторник, 4 сентября 2012 г. пользователь Sylvain Lebresne писал:
>
>> To add to Aaron response, you can update a CF concurrently in 1.1
>> already. However, you cannot create multiple CF concurrently just yet,
>> but that will be fixed in 1.2.
>>
>> --
>> Sylvain
>>
>> On Sun, Aug 26, 2012 at 11:04 PM, aaron morton 
>> wrote:
>> > Concurrent schema changes are coming in 1.2.
>> >
>> > I could not find a single issue that covered it, that may be my bad
>> > search
>> > fu. The issues for 1.2 are here
>> > https://issues.apache.org/jira/browse/CASSANDRA/fixforversion/12319262
>> >
>> > Cheers
>> >
>> > -
>> > Aaron Morton
>> > Freelance Developer
>> > @aaronmorton
>> > http://www.thelastpickle.com
>> >
>> > On 24/08/2012, at 7:06 PM, Илья Шипицин  wrote:
>> >
>> > Hello!
>> >
>> > we are looking into concurent schema updates (when multiple instances of
>> > application create CFs at once.
>> >
>> > at the http://wiki.apache.org/cassandra/MultiTenant there's open ticket
>> > 1391, it is said it is still open.
>> > however, in jura is said "1.1.0 is fixed"
>> >
>> > can schema be updated asynchrously on 1.1.x ? or not ?
>> > if multiple server create the same CF ?
>> >
>> > Cheers,
>> > Ilya Shipitsin
>> >
>> >

keyspace not propagated to a new node?

[Yang]

I let a new node join the ring, and giving it a new token by
-Dinitial_token=

both the new node and old nodes on the ring discover each other, that's
good.

I already defined a Keyspace on the old ring, but this  KS is not
propagated to the new node.
Even after 10 minutes, the new node still does not see the KS.

do I have to do something? I thought the bootstrap process would get the
KS'es???

also I tried the -Dcassandra.replace_token, same issue


thanks
Yang

Re: are asynchronous schema updates possible ?

[Илья Шипицин]

what kind of  problems?

"nodes do not agree about schema" exception on later node ? something worse?

2012/9/5 Sylvain Lebresne 

> On Tue, Sep 4, 2012 at 8:23 PM, Илья Шипицин  wrote:
> > Is it ok multiple servers will create/update the same CF at once ? I'm
> > looking into dynamic schema update during application deploy/update.
>
> As said above, it is ok for update the same CF concurrently in 1.1 but
> *not* for creation (if you create CF concurrently, whether that is the
> same CF or not, you might have problem). The last part will be fixed
> in 1.2 however.
>
> --
> Sylvain
>
> >
> > вторник, 4 сентября 2012 г. пользователь Sylvain Lebresne писал:
> >
> >> To add to Aaron response, you can update a CF concurrently in 1.1
> >> already. However, you cannot create multiple CF concurrently just yet,
> >> but that will be fixed in 1.2.
> >>
> >> --
> >> Sylvain
> >>
> >> On Sun, Aug 26, 2012 at 11:04 PM, aaron morton  >
> >> wrote:
> >> > Concurrent schema changes are coming in 1.2.
> >> >
> >> > I could not find a single issue that covered it, that may be my bad
> >> > search
> >> > fu. The issues for 1.2 are here
> >> >
> https://issues.apache.org/jira/browse/CASSANDRA/fixforversion/12319262
> >> >
> >> > Cheers
> >> >
> >> > -
> >> > Aaron Morton
> >> > Freelance Developer
> >> > @aaronmorton
> >> > http://www.thelastpickle.com
> >> >
> >> > On 24/08/2012, at 7:06 PM, Илья Шипицин  wrote:
> >> >
> >> > Hello!
> >> >
> >> > we are looking into concurent schema updates (when multiple instances
> of
> >> > application create CFs at once.
> >> >
> >> > at the http://wiki.apache.org/cassandra/MultiTenant there's open
> ticket
> >> > 1391, it is said it is still open.
> >> > however, in jura is said "1.1.0 is fixed"
> >> >
> >> > can schema be updated asynchrously on 1.1.x ? or not ?
> >> > if multiple server create the same CF ?
> >> >
> >> > Cheers,
> >> > Ilya Shipitsin
> >> >
> >> >
>

Re: Practical node size limits

[Dustin Wenz]

I'm following up on this issue, which I've been monitoring for the last several 
weeks. I thought people might find my observations interesting.

Ever since increasing the heap size to 64GB, we've had no OOM conditions that 
resulted in a JVM termination. Our nodes have around 2.5TB of data each, and 
the replication factor is four. IO on the cluster seems to be fine, though I 
haven't been paying particular attention to any GC hangs.

The bottleneck now seems to be the repair time. If any node becomes too 
inconsistent, or needs to be replaced, the rebuilt time is over a week. That 
issue alone makes this cluster configuration unsuitable for production use.

- .Dustin

On Jul 30, 2012, at 2:04 PM, Dustin Wenz  wrote:

> Thanks for the pointer! It sounds likely that's what I'm seeing. CFStats 
> reports that the bloom filter size is currently several gigabytes. Is there 
> any way to estimate how much heap space a repair would require? Is it a 
> function of simply adding up the filter file sizes, plus some fraction of 
> neighboring nodes?
> 
> I'm still curious about the largest heap sizes that people are running with 
> on their deployments. I'm considering increasing ours to 64GB (with 96GB 
> physical memory) to see where that gets us. Would it be necessary to keep the 
> young-gen size small to avoid long GC pauses? I also suspect that I may need 
> to keep my memtable sizes small to avoid long flushes; maybe in the 1-2GB 
> range.
> 
>   - .Dustin
> 
> On Jul 29, 2012, at 10:45 PM, Edward Capriolo  wrote:
> 
>> Yikes. You should read:
>> 
>> http://wiki.apache.org/cassandra/LargeDataSetConsiderations
>> 
>> Essentially what it sounds like your are now running into is this:
>> 
>> The BloomFilters for each SSTable must exist in main memory. Repair
>> tends to create some extra data which normally gets compacted away
>> later.
>> 
>> Your best bet is to temporarily raise the Xmx heap and adjust the
>> index sampling size. If you need to save the data (if it is just test
>> data you may want to give up and start fresh)
>> 
>> Generally the issue with the large disk configurations it is hard to
>> keep a good ram/disk ratio. Then most reads turn into disk seeks and
>> the throughput is low. I get the vibe people believe large stripes are
>> going to help Cassandra. The issue is that stripes generally only
>> increase sequential throughput, but Cassandra is a random read system.
>> 
>> How much ram/disk you need is case dependent but 1/5 ratio of RAM to
>> disk is where I think most people want to be, unless their system is
>> carrying SSD disks.
>> 
>> Again you have to keep your bloom filters in java heap memory so and
>> design that tries to create a quatrillion small rows is going to have
>> memory issues as well.
>> 
>> On Sun, Jul 29, 2012 at 10:40 PM, Dustin Wenz  wrote:
>>> I'm trying to determine if there are any practical limits on the amount of 
>>> data that a single node can handle efficiently, and if so, whether I've hit 
>>> that limit or not.
>>> 
>>> We've just set up a new 7-node cluster with Cassandra 1.1.2 running under 
>>> OpenJDK6. Each node is 12-core Xeon with 24GB of RAM and is connected to a 
>>> stripe of 10 3TB disk mirrors (a total of 20 spindles each) and connected 
>>> via dual SATA-3 interconnects. I can read and write around 900MB/s 
>>> sequentially on the arrays. I started out with Cassandra tuned with 
>>> all-default values, with the exception of the compaction throughput which 
>>> was increased from 16MB/s to 100MB/s. These defaults will set the heap size 
>>> to 6GB.
>>> 
>>> Our schema is pretty simple; only 4 column families and each has one 
>>> secondary index. The replication factor was set to four, and compression 
>>> disabled. Our access patterns are intended to be about equal numbers of 
>>> inserts and selects, with no updates, and the occasional delete.
>>> 
>>> The first thing we did was begin to load data into the cluster. We could 
>>> perform about 3000 inserts per second, which stayed mostly flat. Things 
>>> started to go wrong around the time the nodes exceeded 800GB. Cassandra 
>>> began to generate a lot of "mutations messages dropped" warnings, and was 
>>> complaining that the heap was over 75% capacity.
>>> 
>>> At that point, we stopped all activity on the cluster and attempted a 
>>> repair. We did this so we could be sure that the data was fully consistent 
>>> before continuing. Our mistake was probably trying to repair all of the 
>>> nodes simultaneously - within an hour, Java terminated on one of the nodes 
>>> with a heap out-of-memory message. I then increased all of the heap sizes 
>>> to 8GB, and reduced the heap_newsize to 800MB. All of the nodes were 
>>> restarted, and there was no no outside activity on the cluster. I then 
>>> began a repair on a single node. Within a few hours, it OOMed again and 
>>> exited. I then increased the heap to 12GB, and attempted the same thing. 
>>> This time, the repair ran for about

Re: are asynchronous schema updates possible ?

[Hiller, Dean]

+1 What kinds of problems?
Thanks,
Dean

From: Илья Шипицин mailto:chipits...@gmail.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Tuesday, September 4, 2012 1:12 PM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Re: are asynchronous schema updates possible ?


what kind of  problems?

"nodes do not agree about schema" exception on later node ? something worse?

2012/9/5 Sylvain Lebresne mailto:sylv...@datastax.com>>
On Tue, Sep 4, 2012 at 8:23 PM, Илья Шипицин 
mailto:chipits...@gmail.com>> wrote:
> Is it ok multiple servers will create/update the same CF at once ? I'm
> looking into dynamic schema update during application deploy/update.

As said above, it is ok for update the same CF concurrently in 1.1 but
*not* for creation (if you create CF concurrently, whether that is the
same CF or not, you might have problem). The last part will be fixed
in 1.2 however.

--
Sylvain

>
> вторник, 4 сентября 2012 г. пользователь Sylvain Lebresne писал:
>
>> To add to Aaron response, you can update a CF concurrently in 1.1
>> already. However, you cannot create multiple CF concurrently just yet,
>> but that will be fixed in 1.2.
>>
>> --
>> Sylvain
>>
>> On Sun, Aug 26, 2012 at 11:04 PM, aaron morton 
>> mailto:aa...@thelastpickle.com>>
>> wrote:
>> > Concurrent schema changes are coming in 1.2.
>> >
>> > I could not find a single issue that covered it, that may be my bad
>> > search
>> > fu. The issues for 1.2 are here
>> > https://issues.apache.org/jira/browse/CASSANDRA/fixforversion/12319262
>> >
>> > Cheers
>> >
>> > -
>> > Aaron Morton
>> > Freelance Developer
>> > @aaronmorton
>> > http://www.thelastpickle.com
>> >
>> > On 24/08/2012, at 7:06 PM, Илья Шипицин 
>> > mailto:chipits...@gmail.com>> wrote:
>> >
>> > Hello!
>> >
>> > we are looking into concurent schema updates (when multiple instances of
>> > application create CFs at once.
>> >
>> > at the http://wiki.apache.org/cassandra/MultiTenant there's open ticket
>> > 1391, it is said it is still open.
>> > however, in jura is said "1.1.0 is fixed"
>> >
>> > can schema be updated asynchrously on 1.1.x ? or not ?
>> > if multiple server create the same CF ?
>> >
>> > Cheers,
>> > Ilya Shipitsin
>> >
>> >

Helenos - web based gui tool

[Tomek Kuprowski]

Dear all,

I'm happy to announce a first release of Helenos. This is a web based gui
tool to manage your data stored in Cassandra.

Project site: https://github.com/tomekkup/helenos

Some screens:  https://picasaweb.google.com/tomekkuprowski/Helenos

Hope you'll find it usefull. I'll be grateful for your comments and
opinions.

-- 
-- 
Regards !
Tomek Kuprowski

playOrm now supports N-level joins on cassandra (no limitations on where clause)

[Hiller, Dean]

There is no = or < limitations.  Joins are in beta and currently can only do 
inner joins at this time….Also, queries return a Cursor so you can page as well 
and keep the cursor in a web server session if needed for paging.

It also looks like joins may be faster with cassandra/playOrm vs. 
DBMS/hibernate.  We plan on running tests there to see.

The only limitation is you need to be able to partition your data into 
partitions and queries on these partitions are very fast.

Example SQL query ran in our JoinTest

@NoSqlQuery(name="findJoinOnPartition", query="PARTITIONS p(:partitionId) 
select p FROM TABLE as p INNER JOIN p.security as s where s.valuation > :value 
and p.numShares < :numShares)

The ad-hoc query tool supports the same syntax and is built off the same 
library, BUT the ad-hoc tool requires one of two things

 1.  You use the ORM so it writes meta data into cassandra OR
 2.  You write in meta data yourself into cassandra using the meta data apis in 
playOrm

Later,
Dean

Re: Order of the cyclic group of hashed partitioners

[aaron morton]

> I believe the question is why is the maximum 2**127 and not
> 0x
The maximum is the size of the digest created by MD5. 

Does that answer the question?


-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 3/09/2012, at 8:20 PM, Tim Wintle  wrote:

> On Tue, 2012-08-28 at 16:57 +1200, aaron morton wrote:
>> Sorry I don't understand your question. 
>> 
>> Can you explain it a bit more or maybe someone else knows.
> 
> I believe the question is why is the maximum 2**127 and not
> 0x
> 
> Tim
> 
>> 
>> Cheers
>> 
>> -
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>> 
>> On 27/08/2012, at 7:16 PM, Romain HARDOUIN  wrote:
>> 
>>> 
>>> Thank you Aaron. 
>>> This limit was pushed down in RandomPartitioner but the question still 
>>> exists... 
>>> 
>>> 
>>> aaron morton  a écrit sur 26/08/2012 23:35:50 :
>>> 
> AbstractHashedPartitioner 
 does not exist in the trunk. 
 https://git-wip-us.apache.org/repos/asf?p=cassandra.git;
 a=commitdiff;h=a89ef1ffd4cd2ee39a2751f37044dba3015d72f1
 
 
 Cheers
 
 -
 Aaron Morton
 Freelance Developer
 @aaronmorton
 http://www.thelastpickle.com
 
 On 24/08/2012, at 10:51 PM, Romain HARDOUIN  
 wrote:
 
> 
> Hi, 
> 
> AbstractHashedPartitioner defines a maximum of 2**127 hence an 
 order of (2**127)+1. 
> I'd say that tokens of such partitioners are intented to be 
 distributed in Z/(127), hence a maximum of (2**127)-1. 
> Could there be a mix up between maximum and order? 
> This is a detail but could someone confirm/invalidate? 
> 
> Regards, 
> 
> Romain
 
>> 
> 
> 

Re: Text searches and free form queries

[aaron morton]

AFAIk if you want to keep it inside cassandra then DSE, roll your own from 
scratch or start with https://github.com/tjake/Solandra . 

Outside of Cassandra I've heard of people using Elastic Search or Solr which I 
*think* is now faster at updating the index. 

Hope that helps. 

 
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 4/09/2012, at 3:00 AM, Andrey V. Panov  wrote:

> Some one did search on Lucene, but for very fresh data they build search 
> index in memory so data become available for search without delays.
> 
> On 3 September 2012 22:25, Oleg Dulin  wrote:
> Dear Distinguished Colleagues:
> 

Re: configure KeyCahce to use Non-Heap memory ?

[aaron morton]

> Is there any way I can configure KeyCahce to use Non-Heap memory ? 
No.
You could add a feature request here 
https://issues.apache.org/jira/browse/CASSANDRA

Could you post some stats on the current key cache size and hit rate ? (from 
nodetool info) 
It would be interesting to know how many keys it contains Vs the number of rows 
on the box and the hit rate. 

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 4/09/2012, at 3:01 PM, Ananth Gundabattula  
wrote:

> 
> Is there any way I can configure KeyCahce to use Non-Heap memory ? 
> 
> We have large memory nodes :  ~96GB memory per node and effectively using 
> only  8 GB configured for heap ( to avoid GC issues because of a large heap) 
> 
> We have a constraint with respect to : 
> Row cache models don't reflect our data query patterns and hence can only 
> optimize on the key cache 
> Time constrained to change our schema to be more NO-SQL specific
> 
> 
> Regards,
> Ananth 

Re: java.lang.NoClassDefFoundError when trying to do anything on one CF on one node

[aaron morton]

Hmmm, this looks like an error in ctor for NodeId$LocalNodeIdHistory. Are there 
any other ERROR log messages?

Do you see either of these two messages in the log:
"No saved local node id, using newly generated: {}"
or
"Saved local node id: {}"


Can you use cassandra-cli / cqlsh to print the contents of the NodeIdInfo and 
LocationInfo CF's from system ? 

I *think* a work around may be to:

* stop the node
* remove  LocationInfo and NodeInfo cfs. 
* restart

Note this will read the token from the yaml file again, so make sure it's right.

 cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 4/09/2012, at 9:51 PM, Thomas van Neerijnen  wrote:

> Hi
> 
> I have a single node in a 6 node Cassandra 1.0.11 cluster that seems to have 
> a single column family in a weird state.
> 
> Repairs, upgradesstables, anything that touches this CF crashes.
> I've drained the node, removed every file for this CF from said node, removed 
> the commit log, started it up and as soon as data is written to this CF on 
> this node I'm in the same situation again. Anyone have any suggestions for 
> how to fix this?
> I'm tempted to remove the node and re-add it but I was hoping for something a 
> little less disruptive.
> 
> $ nodetool -h localhost upgradesstables Player PlayerCounters
> Error occured while upgrading the sstables for keyspace Player
> java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: 
> Could not initialize class org.apache.cassandra.utils.NodeId$LocalIds
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at 
> org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:219)
> at 
> org.apache.cassandra.db.compaction.CompactionManager.performSSTableRewrite(CompactionManager.java:235)
> at 
> org.apache.cassandra.db.ColumnFamilyStore.sstablesRewrite(ColumnFamilyStore.java:999)
> at 
> org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:1652)
> at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at 
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
> at 
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
> at 
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
> at 
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
> at 
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
> at 
> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
> at 
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
> at 
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
> at 
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
> at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
> at sun.rmi.transport.Transport$1.run(Transport.java:159)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
> at 
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
> at 
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
> at 
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.NoClassDefFoundError: Could not initialize class 
> org.apache.cassandra.utils.NodeId$LocalIds
> at org.apache.cassandra.utils.NodeId.localIds(NodeId.java:49)
> at 
> org.apache.cassandra.utils.NodeId.getOldLocalNodeIds(NodeId.java:79)
> at 
> org.apache.cassandra.db.CounterColumn.computeOldShardMerge

Re: performance is drastically degraded after 0.7.8 --> 1.0.11 upgrade

[aaron morton]

That's slower.

the Recent* metrics are the best to look at. They recent each time you look at 
them. So read them, then run the test, then read them again.  

You'll need to narrow it down still. e.g. Is there a single test taking a very 
long time or are all tests running slower ?  The Histogram stats can help with 
that as they provide a spread of latencies. 

Cheers
 
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 5/09/2012, at 12:27 AM, Илья Шипицин  wrote:

> it was good idea to have a look at StorageProxy :-)
>  
>  
> 1.0.10 Performance Tests
> StorageProxy
> 
> RangeOperations: 546
> ReadOperations: 694563
> TotalHints: 0
> TotalRangeLatencyMicros: 4469484
> TotalReadLatencyMicros:245669679
> TotalWriteLatencyMicros: 57819722
> WriteOperations:208741
> 
> 
> 0.7.10 Performance Tests
> StorageProxy
> 
> RangeOperations: 520
> ReadOperations: 671476
> TotalRangeLatencyMicros: 2208902
> TotalReadLatencyMicros: 162186009
> TotalWriteLatencyMicros: 33911222
> WriteOperations: 204806
> 
> 
> 2012/9/3 aaron morton 
> The whole test run is taking longer ? So it could be slower queries or slower 
> test setup / tear down?
> 
> If you are creating and truncate the KS for each of the 500 tests is that 
> taking longer ? (Schema code has changed a lot 0.7 > 1.0)
> Can you log the execution time for tests and find ones that are taking longer 
> ?
>  
> There are full request metrics available on the StorageProxy JMX object. 
> 
> Cheers
> 
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 31/08/2012, at 4:45 PM, Илья Шипицин  wrote:
> 
>> we are using functional tests ( ~500 tests in time).
>> it is hard to tell which query is slower, it is "slower in general".
>>  
>> same hardware. 1 node, 32Gb RAM, 8Gb heap. default cassandra settings.
>> as we are talking about functional tests, so we recreate KS just before 
>> tests are run.
>>  
>> I do not know how to record queries (there are a lot of them), if you are 
>> interested, I can set up a special stand for you.
>> 
>> 2012/8/31 aaron morton 
 we are running somewhat queue-like with aggressive write-read patterns.
>> We'll need some more details…
>> 
>> How much data ?
>> How many machines ?
>> What is the machine spec ?
>> How many clients ?
>> Is there an example of a slow request ? 
>> How are you measuring that it's slow ? 
>> Is there anything unusual in the log ? 
>> 
>> Cheers
>> 
>> -
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>> 
>> On 31/08/2012, at 3:30 AM, Edward Capriolo  wrote:
>> 
>>> If you move from 7.X to 0.8X or 1.0X you have to rebuild sstables as
>>> soon as possible. If you have large bloomfilters you can hit a bug
>>> where the bloom filters will not work properly.
>>> 
>>> 
>>> On Thu, Aug 30, 2012 at 9:44 AM, Илья Шипицин  wrote:
 we are running somewhat queue-like with aggressive write-read patterns.
 I was looking for scripting queries from live Cassandra installation, but I
 didn't find any.
 
 is there something like thrift-proxy or other query logging/scripting 
 engine
 ?
 
 2012/8/30 aaron morton 
> 
> in terms of our high-rate write load cassandra1.0.11 is about 3 (three!!)
> times slower than cassandra-0.7.8
> 
> We've not had any reports of a performance drop off. All tests so far have
> show improvements in both read and write performance.
> 
> I agree, such digests save some network IO, but they seem to be very bad
> in terms of CPU and disk IO.
> 
> The sha1 is created so we can diagnose corruptions in the -Data component
> of the SSTables. They are not used to save network IO.
> It is calculated while streaming the Memtable to disk so has no impact on
> disk IO. While not the fasted algorithm I would assume it's CPU overhead 
> in
> this case is minimal.
> 
> there's already relatively small Bloom filter file, which can be used for
> saving network traffic instead of sha1 digest.
> 
> Bloom filters are used to test if a row key may exist in an SSTable.
> 
> any explanation ?
> 
> If you can provide some more information on your use case we may be able
> to help.
> 
> Cheers
> 
> 
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 30/08/2012, at 5:18 AM, Илья Шипицин  wrote:
> 
> in terms of our high-rate write load cassandra1.0.11 is about 3 (three!!)
> times slower than cassandra-0.7.8
> after some investigation carried out I noticed files with "sha1" extension
> (which are missing for Cassandra-0.7.8)
> 
> in maybeWriteDigest() function I see no option fot switching sha1 digests
> off.
> 
> I agree, such digests save some network IO, but they seem to be very bad
>

Re: anyone know how to lookup non-continguous columns BUT for prefixes?

[aaron morton]

It's not possible to get non contiguous column slices. 

> This is a trimmed down example of course.  I am thinking maybe I might just 
> use the astyanax async to send out 500 requests instead.
Keep an eye on what happens on the server side. 
Sometimes less is more. 

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 5/09/2012, at 4:56 AM, "Hiller, Dean"  wrote:

> I have a row that is an index like so
> 
> Index row -> ., ., ., 
> . , ., ., .
> 
> I would like to get all of the pks for
>which are pk32 and pk7
> And
>which are pk54
> 
> This is a trimmed down example of course.  I am thinking maybe I might just 
> use the astyanax async to send out 500 requests instead.
> 
> Thanks,
> Dean
> 
> 

Re: keyspace not propagated to a new node?

[aaron morton]

What version are you on ?

Anything in the logs from MigrationManager ?

You could try nodetool resetlocalschema .

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 5/09/2012, at 7:02 AM, Yang  wrote:

> I let a new node join the ring, and giving it a new token by -Dinitial_token=
> 
> both the new node and old nodes on the ring discover each other, that's good.
> 
> I already defined a Keyspace on the old ring, but this  KS is not propagated 
> to the new node.
> Even after 10 minutes, the new node still does not see the KS.
> 
> do I have to do something? I thought the bootstrap process would get the 
> KS'es???
> 
> also I tried the -Dcassandra.replace_token, same issue
> 
> 
> thanks
> Yang

Re: why do I have to use internal IP for EC2 nodes?

[aaron morton]

See http://aws.amazon.com/articles/1145?_encoding=UTF8&jiveRedirect=1#12

The external  dns will resolve to the internal IP when resolved internally. 

Using the internal IP means you are not charged for IO and it makes it clear 
you do not expect this service to be access from outside. 

Cheers
 
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 5/09/2012, at 7:37 AM, Yang  wrote:

> http://www.datastax.com/docs/1.1/initialize/cluster_init
> 
> 
> says:
> 
> " Note  In the - seeds list property, include the internal IP addresses of 
> each seed node."
> 
> why do I have to use internal IP?
> on a EC2 node,
> hostname resolution seems to directly give its internal IP:
> 
> $ host aws1devbic1.biqa.ctgrd.com
> aws1devbic1.biqa.ctgrd.com is an alias for 
> ec2-50-17-3-229.compute-1.amazonaws.com.
> ec2-50-17-3-229.compute-1.amazonaws.com has address 10.28.166.83
> 
> so using the public DNS or internal IP seems to be the same thing, or 
> something I'm missing ?
> 
> Thanks
> Yang
> 

Re: performance is drastically degraded after 0.7.8 --> 1.0.11 upgrade

[Илья Шипицин]

all tests use similar data access patterns, so every test on 1.0.11 is
slower than 0.7.8
recent micros confirms that.

2012/9/5 aaron morton 

> That's slower.
>
> the Recent* metrics are the best to look at. They recent each time you
> look at them. So read them, then run the test, then read them again.
>
> You'll need to narrow it down still. e.g. Is there a single test taking a
> very long time or are all tests running slower ?  The Histogram stats can
> help with that as they provide a spread of latencies.
>
> Cheers
>
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 5/09/2012, at 12:27 AM, Илья Шипицин  wrote:
>
> it was good idea to have a look at StorageProxy :-)
>
>
> 1.0.10 Performance Tests
> StorageProxy
>
> RangeOperations: 546
> ReadOperations: 694563
> TotalHints: 0
> TotalRangeLatencyMicros: 4469484
> TotalReadLatencyMicros:245669679
> TotalWriteLatencyMicros: 57819722
> WriteOperations:208741
>
>
> 0.7.10 Performance Tests
> StorageProxy
>
> RangeOperations: 520
> ReadOperations: 671476
> TotalRangeLatencyMicros: 2208902
> TotalReadLatencyMicros: 162186009
> TotalWriteLatencyMicros: 33911222
> WriteOperations: 204806
>
>
> 2012/9/3 aaron morton 
>
>> The whole test run is taking longer ? So it could be slower queries or
>> slower test setup / tear down?
>>
>> If you are creating and truncate the KS for each of the 500 tests is that
>> taking longer ? (Schema code has changed a lot 0.7 > 1.0)
>> Can you log the execution time for tests and find ones that are taking
>> longer ?
>>
>> There are full request metrics available on the StorageProxy JMX object.
>>
>> Cheers
>>
>>  -
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 31/08/2012, at 4:45 PM, Илья Шипицин  wrote:
>>
>> we are using functional tests ( ~500 tests in time).
>> it is hard to tell which query is slower, it is "slower in general".
>>
>> same hardware. 1 node, 32Gb RAM, 8Gb heap. default cassandra settings.
>> as we are talking about functional tests, so we recreate KS just before
>> tests are run.
>>
>> I do not know how to record queries (there are a lot of them), if you are
>> interested, I can set up a special stand for you.
>>
>> 2012/8/31 aaron morton 
>>
>>> we are running somewhat queue-like with aggressive write-read patterns.
>>>
>>> We'll need some more details...
>>>
>>> How much data ?
>>> How many machines ?
>>> What is the machine spec ?
>>> How many clients ?
>>> Is there an example of a slow request ?
>>> How are you measuring that it's slow ?
>>> Is there anything unusual in the log ?
>>>
>>> Cheers
>>>
>>>  -
>>> Aaron Morton
>>> Freelance Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 31/08/2012, at 3:30 AM, Edward Capriolo 
>>> wrote:
>>>
>>> If you move from 7.X to 0.8X or 1.0X you have to rebuild sstables as
>>> soon as possible. If you have large bloomfilters you can hit a bug
>>> where the bloom filters will not work properly.
>>>
>>>
>>> On Thu, Aug 30, 2012 at 9:44 AM, Илья Шипицин 
>>> wrote:
>>>
>>> we are running somewhat queue-like with aggressive write-read patterns.
>>> I was looking for scripting queries from live Cassandra installation,
>>> but I
>>> didn't find any.
>>>
>>> is there something like thrift-proxy or other query logging/scripting
>>> engine
>>> ?
>>>
>>> 2012/8/30 aaron morton 
>>>
>>>
>>> in terms of our high-rate write load cassandra1.0.11 is about 3 (three!!)
>>> times slower than cassandra-0.7.8
>>>
>>> We've not had any reports of a performance drop off. All tests so far
>>> have
>>> show improvements in both read and write performance.
>>>
>>> I agree, such digests save some network IO, but they seem to be very bad
>>> in terms of CPU and disk IO.
>>>
>>> The sha1 is created so we can diagnose corruptions in the -Data component
>>> of the SSTables. They are not used to save network IO.
>>> It is calculated while streaming the Memtable to disk so has no impact on
>>> disk IO. While not the fasted algorithm I would assume it's CPU overhead
>>> in
>>> this case is minimal.
>>>
>>> there's already relatively small Bloom filter file, which can be used for
>>> saving network traffic instead of sha1 digest.
>>>
>>> Bloom filters are used to test if a row key may exist in an SSTable.
>>>
>>> any explanation ?
>>>
>>> If you can provide some more information on your use case we may be able
>>> to help.
>>>
>>> Cheers
>>>
>>>
>>> -
>>> Aaron Morton
>>> Freelance Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 30/08/2012, at 5:18 AM, Илья Шипицин  wrote:
>>>
>>> in terms of our high-rate write load cassandra1.0.11 is about 3 (three!!)
>>> times slower than cassandra-0.7.8
>>> after some investigation carried out I noticed files with "sha1"
>>> extension
>>> (which are missing for Cassandra-0.7.8)
>>>
>>> in maybeWriteDigest() function I see no option fot switching sha1 digests
>>> off

Re: keyspace not propagated to a new node?

[Yang]

this is 1.1.2

unfortunately the logs does not show anything except for the initial
startup sequences.
on the siblings, somehow the /var/log/cassandra/system.log were wrongly
deleted, so I can't see their responses either

I'll try resetlocalschema next time

thanks!
yang

On Tue, Sep 4, 2012 at 7:03 PM, aaron morton wrote:

> What version are you on ?
>
> Anything in the logs from MigrationManager ?
>
> You could try nodetool resetlocalschema .
>
> Cheers
>
>   -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 5/09/2012, at 7:02 AM, Yang  wrote:
>
> I let a new node join the ring, and giving it a new token by
> -Dinitial_token=
>
> both the new node and old nodes on the ring discover each other, that's
> good.
>
> I already defined a Keyspace on the old ring, but this  KS is not
> propagated to the new node.
> Even after 10 minutes, the new node still does not see the KS.
>
> do I have to do something? I thought the bootstrap process would get the
> KS'es???
>
> also I tried the -Dcassandra.replace_token, same issue
>
>
> thanks
> Yang
>
>
>

Re: why do I have to use internal IP for EC2 nodes?

[Yang]

thanks, but if the communication between cluster nodes all resolve to
"internal to internal", amazon will not charge the traffic as external
traffic, right?

On Tue, Sep 4, 2012 at 7:08 PM, aaron morton wrote:

> See http://aws.amazon.com/articles/1145?_encoding=UTF8&jiveRedirect=1#12
>
> The external  dns will resolve to the internal IP when resolved internally.
>
> Using the internal IP means you are not charged for IO and it makes it
> clear you do not expect this service to be access from outside.
>
> Cheers
>
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 5/09/2012, at 7:37 AM, Yang  wrote:
>
> > http://www.datastax.com/docs/1.1/initialize/cluster_init
> >
> >
> > says:
> >
> > " Note  In the - seeds list property, include the internal IP addresses
> of each seed node."
> >
> > why do I have to use internal IP?
> > on a EC2 node,
> > hostname resolution seems to directly give its internal IP:
> >
> > $ host aws1devbic1.biqa.ctgrd.com
> > aws1devbic1.biqa.ctgrd.com is an alias for
> ec2-50-17-3-229.compute-1.amazonaws.com.
> > ec2-50-17-3-229.compute-1.amazonaws.com has address 10.28.166.83
> >
> > so using the public DNS or internal IP seems to be the same thing, or
> something I'm missing ?
> >
> > Thanks
> > Yang
> >
>
>

Re: why do I have to use internal IP for EC2 nodes?

[Robin Verlangen]

@Yang: Sounds legit, as internal is not the same as external. Beware of the
fact that internal traffic is only free when it's in the same availability
zone. In the same region is charged with a small amount (~ $0.01).

With kind regards,

Robin Verlangen
*Software engineer*
*
*
W http://www.robinverlangen.nl
E ro...@us2.nl

Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.



2012/9/5 Yang 

> thanks, but if the communication between cluster nodes all resolve to
> "internal to internal", amazon will not charge the traffic as external
> traffic, right?
>
> On Tue, Sep 4, 2012 at 7:08 PM, aaron morton wrote:
>
>> See http://aws.amazon.com/articles/1145?_encoding=UTF8&jiveRedirect=1#12
>>
>> The external  dns will resolve to the internal IP when resolved
>> internally.
>>
>> Using the internal IP means you are not charged for IO and it makes it
>> clear you do not expect this service to be access from outside.
>>
>> Cheers
>>
>> -
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 5/09/2012, at 7:37 AM, Yang  wrote:
>>
>> > http://www.datastax.com/docs/1.1/initialize/cluster_init
>> >
>> >
>> > says:
>> >
>> > " Note  In the - seeds list property, include the internal IP addresses
>> of each seed node."
>> >
>> > why do I have to use internal IP?
>> > on a EC2 node,
>> > hostname resolution seems to directly give its internal IP:
>> >
>> > $ host aws1devbic1.biqa.ctgrd.com
>> > aws1devbic1.biqa.ctgrd.com is an alias for
>> ec2-50-17-3-229.compute-1.amazonaws.com.
>> > ec2-50-17-3-229.compute-1.amazonaws.com has address 10.28.166.83
>> >
>> > so using the public DNS or internal IP seems to be the same thing, or
>> something I'm missing ?
>> >
>> > Thanks
>> > Yang
>> >
>>
>>
>