Re: migrating from 0.6 to 0.8, "java.io.IOError: ... cannot extend file to required size"

[Zhu Han]

On Wed, Aug 10, 2011 at 5:24 PM, aaron morton wrote:

> I remember seeing this once before upgrading a system from 0.6 to 0.7 on a
> Ubuntu EC2 (non data stax build) with EBS disks. I did the same thing and
> just assumed it was an EBS or 0.6 bug. From memory after the upgrade that
> node had some serialisation issues with a few SSTables that I had to
> eventually take off the node.
>
> I would recommend running nodetool scrub *before* the upgrade to ensure the
> files can be read.
>

But "nodetool scrub" is provided by 0.7.3 or later. How can we run scrub
before upgrade?


>
> Cheers
>
> -
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 10 Aug 2011, at 13:09, Jonathan Ellis wrote:
>
> > What OS was this?
> >
> > On Tue, Aug 9, 2011 at 6:32 PM, ian douglas  wrote:
> >> Thanks to 'driftx' in IRC, adding this line to my cassandra.yaml file
> seems
> >> to have fixed the problem:
> >>
> >> disk_access_mode: standard
> >>
> >>
> >>
> >>
> >> On 08/09/2011 04:06 PM, ian douglas wrote:
> >>
> >> updated stack trace after fixing the corrupted GameSave2 tables:
> >>
> >>
> >>
> >>  INFO 16:05:20,314 Logging initialized
> >>  INFO 16:05:20,330 Heap size: 6290407424/6291456000
> >>  INFO 16:05:20,336 JNA not found. Native methods will be disabled.
> >>  INFO 16:05:20,344 Loading settings from
> >> file:/data/apache-cassandra-0.8.1/conf/cassandra.yaml
> >>  INFO 16:05:20,474 DiskAccessMode 'auto' determined to be mmap,
> >> indexAccessMode is mmap
> >>  INFO 16:05:20,627 Global memtable threshold is enabled at 2000MB
> >>  INFO 16:05:20,628 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:20,633 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:20,634 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:20,635 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:20,636 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:20,637 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:20,737 Opening /data/cassandra8/data/system/Schema-g-13
> >>  INFO 16:05:20,766 Opening /data/cassandra8/data/system/Migrations-g-13
> >>  INFO 16:05:20,770 Opening /data/cassandra8/data/system/LocationInfo-g-2
> >>  INFO 16:05:20,772 Opening /data/cassandra8/data/system/LocationInfo-g-1
> >>  INFO 16:05:20,821 Loading schema version
> >> 8fbed060-c2ae-11e0--242d50cf1fff
> >>  INFO 16:05:21,028 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,030 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,032 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,033 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,035 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,039 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,040 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,044 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,045 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,047 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,047 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,048 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,049 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,049 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,050 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,054 Opening /data/cassandra8/data/armorgames/Feeds2-1777
> >>  INFO 16:05:21,774 Opening /data/cassandra8/data/armorgames/Status-764
> >>  INFO 16:05:21,800 Opening /data/cassandra8/data/armorgames/Comments-763
> >>  INFO 16:05:21,816 Opening
> >> /data/cassandra8/data/armorgames/UserGameshareData-8483
> >>  INFO 16:05:31,645 Opening
> /data/cassandra8/data/armorgames/GameSave2-7309
> >>  INFO 16:05:33,235 Opening
> /data/cassandra8/data/armorgames/GameSave2-7391
> >>  INFO 16:05:33,240 Opening
> /data/cassandra8/data/arm

AW: IndexSliceQuery issue - ReadStage piling up (looks like deadlock/infinite loop or similar)

[Roland Gude]

Yes, i can reproduce this behavior

If issue a query like this (on 0.7.8 with patch for CASSANDRA-2964 applied)
[default@demo]get users where birth_date = 1968 and state = 'UT';
with an index on birth_date but no index on state
I do not get results (actually I get '0 rows') even though there are rows which 
statisfy all clauses.
However, if I repeat this several times several of the nodes start piling up 
pending reads. (tpstats shows some 8 reads pending)
And even though the nodes are not able to fulfill (read) requests anymore, they 
are not marked as down by the gossiper. Overall this results in an unusable 
cluster.

If I do the same thing on a 0.7.5 cluster
Cassandra logs a nullpointerexception and the cli returns with null, but the 
cluster stays functional.


Von: aaron morton [mailto:aa...@thelastpickle.com]
Gesendet: Mittwoch, 10. August 2011 23:48
An: user@cassandra.apache.org
Betreff: Re: IndexSliceQuery issue - ReadStage piling up (looks like 
deadlock/infinite loop or similar)

Are you still having a problem ? I'm a bit confused about what you saying.

Cheers

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 10 Aug 2011, at 03:33, Roland Gude wrote:


Hi,

I experience issues when doing a indexslicequery with multiple expressions if 
one of the expressions is about a non index column

I did the equivalent of this example (but with my data) from
http://www.datastax.com/dev/blog/whats-new-cassandra-07-secondary-indexes


Secondary indexes automate this. Let's add some state data:

[default@demo] set users[bsanderson][state] = 'UT';
[default@demo] set users[prothfuss][state] = 'WI';
[default@demo] set users[htayler][state] = 'UT';

Note that even though state is not indexed yet, we can include the new state 
data in a query as long as another column in the query is indexed:

[default@demo] get users where state = 'UT';
No indexed columns present in index clause with operator EQ
[default@demo] get users where state = 'UT' and birth_date > 1970;
No indexed columns present in index clause with operator EQ
[default@demo]get users where birth_date = 1968 and state = 'UT';
---
RowKey: htayler
=> (column=birth_date, value=1968, timestamp=1291334765649000)
=> (column=full_name, value=Howard Tayler, timestamp=129133474916)
=> (column=state, value=5554, timestamp=1291334890708000)

On On 0.7.8 (with CASSANDRA-2964 applied)
This example will not return any data, but return "0 rows". I repeated the 
query multiple times with different variations for the values which should all 
have returned data, but eventually I ended up with the cluster having 8 
reads pending on some of the nodes

On 0.7.5 the query will result in a NullPointerException being thrown and 
"null" returned in the cli

ERROR [ReadStage:258] 2011-08-09 16:03:27,153 AbstractCassandraDaemon.java 
(line 113) Fatal exception in thread Thread[ReadStage:258,5,main]
java.lang.RuntimeException: java.lang.NullPointerException
at 
org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:51)
at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NullPointerException
at org.apache.cassandra.db.ColumnFamily.addAll(ColumnFamily.java:131)
at 
org.apache.cassandra.db.ColumnFamilyStore.scan(ColumnFamilyStore.java:1615)
at 
org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:42)
... 4 more
ERROR [ReadStage:258] 2011-08-09 16:03:27,153 AbstractCassandraDaemon.java 
(line 113) Fatal exception in thread Thread[ReadStage:258,5,main]
java.lang.RuntimeException: java.lang.NullPointerException
at 
org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:51)
at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NullPointerException
at org.apache.cassandra.db.ColumnFamily.addAll(ColumnFamily.java:131)
at 
org.apache.cassandra.db.ColumnFamilyStore.scan(ColumnFamilyStore.java:1615)
at 
org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:42)
... 4 more



Can anybody reproduce this?

Greetings,
Roland

--
YOOCHOOSE GmbH

Roland Gude
Software Engineer

Im Mediapark 8, 50670 Köln

+49 221 4544151 (Tel)
+49 221 4544159 (Fax)
+49 171 7894057 (Mobil)


Email: roland.g...@yoochoose.com
WWW: www

Re: SOLVED: Problem upgrading to 0.8.3 - "replication_factor is an option for SimpleStrategy, not NetworkTopologyStrategy"

[Martin Lansler]

Hi Eldad / All,

On Wed, Aug 10, 2011 at 8:32 AM, Eldad Yamin  wrote:
> Can you please explain how did you upgraded. something like step-by-step.
> Thanks!

I took the liberty of replying to the group as it would be interesting
to hear how other folks out there are doing it...

I'm *not* running a prod system, just a test system of three nodes on
my laptop. So it would be nice to hear about real setups. Here is my
test setup:

apache-cassandra -> apache-cassandra-0.8.3
apache-cassandra-0.8.2/
apache-cassandra-0.8.3/
node1/
node2/
node3/

All nodeX look like:
bin -> ../apache-cassandra/bin/
commitlog/
conf/
data/
interface -> ../apache-cassandra/interface/
lib -> ../apache-cassandra/lib/
saved_caches/

The 'conf' directory is copied into each node from the virgin
cassandra distribution. I then create a local GIT repo and add the
'conf' directory so I can track any configuration changes on a node.
Then relevant node specific configuration settings are set. The
'commitlog', 'data' and 'saved_caches' are created by cassandra and
must be configured in 'cassandra.yaml' for each node.

When I upgrade I do the following:

1.
Make a diff of the new conf files from the new version so that  get
new parameters etc... I use emacs ediff-mode.
2.
Remove the old "apache-cassandra" symlink and point it to the new cassandra dist
3.
In a rolling fashion stop one node, and then restart it... as the
symlink is changes it will then boot with the upgraded cassandra dist.
(remember to cd out & in of the bin/ dir otherwise you will still be
in the old directory).
(4).
Should something break... just re-create the old symlink and restart
the node (provided cassandra has not performed any non backwards
compatible changes to the db files, should be noted in the README)

That's pretty much it.

On a prod setup one would probably use a tool such as puppet
(www.puppetlabs.com/) to ease setting up on many nodes... But there
are many ways to do this, for instance pssh
(http://code.google.com/p/parallel-ssh/).

Regards,
-Martin

On Wed, Aug 10, 2011 at 8:32 AM, Eldad Yamin  wrote:
> Can you please explain how did you upgraded. something like step-by-step.
> Thanks!
>
> On Wed, Aug 10, 2011 at 12:11 AM, Martin Lansler 
> wrote:
>>
>> Hi,
>>
>> When I upgraded from 0.8.2 to 0.8.3 I encountered a exception during
>> startup:
>> ...
>> Caused by: org.apache.cassandra.config.ConfigurationException:
>> replication_factor is an option for SimpleStrategy, not
>> NetworkTopologyStrategy
>>        at
>> org.apache.cassandra.locator.NetworkTopologyStrategy.(NetworkTopologyStrategy.java:70)
>> ...
>>
>> This was caused by the following change:
>> ---
>> cassandra/trunk/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
>>  2011/07/18
>> 13:32:48        1147869
>> +++
>> cassandra/trunk/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
>>  2011/08/02
>> 17:07:22        1153200
>> @@ -67,7 +67,7 @@
>>             {
>>                 String dc = entry.getKey();
>>                 if (dc.equalsIgnoreCase("replication_factor"))
>> -                    continue; // TODO remove this for 1.0
>> +                    throw new
>> ConfigurationException("replication_factor is an option for
>> SimpleStrategy, not NetworkTopologyStrategy");
>>                 Integer replicas = Integer.valueOf(entry.getValue());
>>                 newDatacenters.put(dc, replicas);
>>             }
>>
>> To solve it use the 'cassandra-cli' on one of the other "up" nodes:
>>
>> [default@unknown] describe keyspace TestKeyspace;
>> WARNING: Could not connect to the JMX on 127.0.0.3:7199, information
>> won't be shown.
>> Keyspace: TestKeyspace:
>>  Replication Strategy:
>> org.apache.cassandra.locator.NetworkTopologyStrategy
>>  Durable Writes: true
>>    Options: [replication_factor:2]
>> ...
>>
>> default@unknown] update keyspace TestKeyspace with
>> strategy_options=[{datacenter:2}];
>> 47035d20-c2ca-11e0--de5e2f7aa3bf
>> Waiting for schema agreement...
>> ... schemas agree across the cluster
>>
>> [default@unknown] describe keyspace TestKeyspace;
>> Keyspace: TestKeyspace:
>>  Replication Strategy:
>> org.apache.cassandra.locator.NetworkTopologyStrategy
>>  Durable Writes: true
>>    Options: [datacenter:2]
>>
>> Then restart the 0.8.3 node and it should be fine.
>>
>> Regards,
>> -Martin
>
>

Re: Need help in CF design

[Benoit Perroud]

You can apply this query really simply using cassandra and secondary 
indexes.


You will have a CF "TABLE", where row keys are your PK. Just to be sure 
of my understanding, your SQL query will either return 1 row or no row, 
right ?


3) SliceQuery returns a range of columns for a given key, it may be your 
friend.






On 11. 08. 11 07:50, a...@9y.com wrote:

I recently started with Cassandra and found interesting.

I was curious in SQL we have

SELECT * from TABLE where PK="primary_key" and other_attribute
between 500 and 1000;

My questions are :
1) Is it possible to design to get equivalent results for above
query ( using CQL or Hector) with Cassandra.

2) How can we design CF in that case? Using secondary index is an
option but I am not clear how that can be applied here.

3) Is there any way we can have a range slice over columns names
instead of having range or row keys.

I am just a novice. So, Can anyone help me with these question

Re: Tuning a column family for archival

[Edward Capriolo]

On Thu, Aug 11, 2011 at 12:07 AM, aaron morton wrote:

> There's not much to do other than turn off the caches (which you have done)
> and leave it alone.
>
> If you want to poke around perhaps look at the compaction settings (from
> CLI help):
>
> - max_compaction_threshold: The maximum number of SSTables allowed before a
> minor compaction is forced. Default is 32, setting to 0 disables minor
> compactions.
>
> Decreasing this will cause minor compactions to start more frequently and
> be less intensive. The min_compaction_threshold and
> max_compaction_threshold
> boundaries are the number of tables Cassandra attempts to merge together at
> once.
>
> - min_compaction_threshold: The minimum number of SSTables needed
> to start a minor compaction. Default is 4, setting to 0 disables minor
> compactions.
>
> Increasing this will cause minor compactions to start less frequently and
> be more intensive. The min_compaction_threshold and
> max_compaction_threshold
> boundaries are the number of tables Cassandra attempts to merge together at
> once.
>
> You *could* disable compaction and then manually compact at the best time.
> If you are not doing many updates I'd wait and see.
>
> You could repair different CF's at different times. This would help with
> reducing the amount of data that is used to build the Merkle tree's, but
> there is a bug about streaming the differences that means extra data is
> streamed (can't remember the bug number now)
>
> I'd wait to see if there is an issue first.
>
> Cheers
>
> -
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 11 Aug 2011, at 13:18, Jason Baker wrote:
>
> > I have a column family that I'm using to archive records.  They're mostly
> kept around for historical purposes.  Aside from that, they're mostly
> considered deleted.  It's probably going to be very rare that anyone reads
> from this table *ever*.  I don't really even write to it that much.
> >
> > Does anyone have advice for me as far as how (or if) I should tune this
> table with that in mind?  My concern is less speeding up access to this
> table than it is making sure that it doesn't impact the performance of any
> other column families in any way.
> >
> > Here's the data from nodetool cfstat (although this table was just
> created a few days ago):
> >
> >   Column Family: ArchivedLinks
> >   SSTable count: 1
> >   Space used (live): 29580801
> >   Space used (total): 97838786
> >   Number of Keys (estimate): 93184
> >   Memtable Columns Count: 7497
> >   Memtable Data Size: 3223587
> >   Memtable Switch Count: 11
> >   Read Count: 0
> >   Read Latency: NaN ms.
> >   Write Count: 139091
> >   Write Latency: 0.007 ms.
> >   Pending Tasks: 0
> >   Key cache: disabled
> >   Row cache: disabled
> >   Compacted row minimum size: 259
> >   Compacted row maximum size: 372
> >   Compacted row mean size: 311
>
>
In many regards Cassandra automatically does the correct thing. Other then
the costs of the bloom filters for the table size being in ram, if you never
read or write to those sstables and you are not reusing the row key, the OS
will page out those tables and they will not take any cache space.

Coming soon compaction is going to change a lot, I know one of the tickets
in the works is that SSTables will have a max size, compaction should do
something like a one to one rewrite of these tables, which should not be
very intensive.

Best practices when deploying & upgrading a cassandra cluster

[Martin Lansler]

(Note: This is a repost from another thread which did not have a
relevant subject, sorry for the spamming)

Hi Eldad / All,

On Wed, Aug 10, 2011 at 8:32 AM, Eldad Yamin  wrote:
> Can you please explain how did you upgraded. something like step-by-step.
> Thanks!

I took the liberty of replying to the group as it would be interesting
to hear how other folks out there are doing it...

I'm *not* running a prod system, just a test system of three nodes on
my laptop. So it would be nice to hear about real setups. Here is my
test setup:

apache-cassandra -> apache-cassandra-0.8.3
apache-cassandra-0.8.2/
apache-cassandra-0.8.3/
node1/
node2/
node3/

All nodeX look like:
bin -> ../apache-cassandra/bin/
commitlog/
conf/
data/
interface -> ../apache-cassandra/interface/
lib -> ../apache-cassandra/lib/
saved_caches/

The 'conf' directory is copied into each node from the virgin
cassandra distribution. I then create a local GIT repo and add the
'conf' directory so I can track any configuration changes on a node.
Then relevant node specific configuration settings are set. The
'commitlog', 'data' and 'saved_caches' are created by cassandra and
must be configured in 'cassandra.yaml' for each node.

When I upgrade I do the following:

1.
Make a diff of the new conf files from the new version so that  get
new parameters etc... I use emacs ediff-mode.
2.
Remove the old "apache-cassandra" symlink and point it to the new cassandra dist
3.
In a rolling fashion stop one node, and then restart it... as the
symlink is changes it will then boot with the upgraded cassandra dist.
(remember to cd out & in of the bin/ dir otherwise you will still be
in the old directory).
(4).
Should something break... just re-create the old symlink and restart
the node (provided cassandra has not performed any non backwards
compatible changes to the db files, should be noted in the README)

That's pretty much it.

On a prod setup one would probably use a tool such as puppet
(www.puppetlabs.com/) to ease setting up on many nodes... But there
are many ways to do this, for instance pssh
(http://code.google.com/p/parallel-ssh/).

Regards,
-Martin

Re: SOLVED: Problem upgrading to 0.8.3 - "replication_factor is an option for SimpleStrategy, not NetworkTopologyStrategy"

[Martin Lansler]

Please continue discussion in new thread "Best practices when
deploying & upgrading a cassandra cluster".

-Martin

On Thu, Aug 11, 2011 at 12:43 PM, Martin Lansler
 wrote:
> Hi Eldad / All,
>
> On Wed, Aug 10, 2011 at 8:32 AM, Eldad Yamin  wrote:
>> Can you please explain how did you upgraded. something like step-by-step.
>> Thanks!
>
> I took the liberty of replying to the group as it would be interesting
> to hear how other folks out there are doing it...
>
> I'm *not* running a prod system, just a test system of three nodes on
> my laptop. So it would be nice to hear about real setups. Here is my
> test setup:
>
> apache-cassandra -> apache-cassandra-0.8.3
> apache-cassandra-0.8.2/
> apache-cassandra-0.8.3/
> node1/
> node2/
> node3/
>
> All nodeX look like:
> bin -> ../apache-cassandra/bin/
> commitlog/
> conf/
> data/
> interface -> ../apache-cassandra/interface/
> lib -> ../apache-cassandra/lib/
> saved_caches/
>
> The 'conf' directory is copied into each node from the virgin
> cassandra distribution. I then create a local GIT repo and add the
> 'conf' directory so I can track any configuration changes on a node.
> Then relevant node specific configuration settings are set. The
> 'commitlog', 'data' and 'saved_caches' are created by cassandra and
> must be configured in 'cassandra.yaml' for each node.
>
> When I upgrade I do the following:
>
> 1.
> Make a diff of the new conf files from the new version so that  get
> new parameters etc... I use emacs ediff-mode.
> 2.
> Remove the old "apache-cassandra" symlink and point it to the new cassandra 
> dist
> 3.
> In a rolling fashion stop one node, and then restart it... as the
> symlink is changes it will then boot with the upgraded cassandra dist.
> (remember to cd out & in of the bin/ dir otherwise you will still be
> in the old directory).
> (4).
> Should something break... just re-create the old symlink and restart
> the node (provided cassandra has not performed any non backwards
> compatible changes to the db files, should be noted in the README)
>
> That's pretty much it.
>
> On a prod setup one would probably use a tool such as puppet
> (www.puppetlabs.com/) to ease setting up on many nodes... But there
> are many ways to do this, for instance pssh
> (http://code.google.com/p/parallel-ssh/).
>
> Regards,
> -Martin
>
> On Wed, Aug 10, 2011 at 8:32 AM, Eldad Yamin  wrote:
>> Can you please explain how did you upgraded. something like step-by-step.
>> Thanks!
>>
>> On Wed, Aug 10, 2011 at 12:11 AM, Martin Lansler 
>> wrote:
>>>
>>> Hi,
>>>
>>> When I upgraded from 0.8.2 to 0.8.3 I encountered a exception during
>>> startup:
>>> ...
>>> Caused by: org.apache.cassandra.config.ConfigurationException:
>>> replication_factor is an option for SimpleStrategy, not
>>> NetworkTopologyStrategy
>>>        at
>>> org.apache.cassandra.locator.NetworkTopologyStrategy.(NetworkTopologyStrategy.java:70)
>>> ...
>>>
>>> This was caused by the following change:
>>> ---
>>> cassandra/trunk/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
>>>  2011/07/18
>>> 13:32:48        1147869
>>> +++
>>> cassandra/trunk/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
>>>  2011/08/02
>>> 17:07:22        1153200
>>> @@ -67,7 +67,7 @@
>>>             {
>>>                 String dc = entry.getKey();
>>>                 if (dc.equalsIgnoreCase("replication_factor"))
>>> -                    continue; // TODO remove this for 1.0
>>> +                    throw new
>>> ConfigurationException("replication_factor is an option for
>>> SimpleStrategy, not NetworkTopologyStrategy");
>>>                 Integer replicas = Integer.valueOf(entry.getValue());
>>>                 newDatacenters.put(dc, replicas);
>>>             }
>>>
>>> To solve it use the 'cassandra-cli' on one of the other "up" nodes:
>>>
>>> [default@unknown] describe keyspace TestKeyspace;
>>> WARNING: Could not connect to the JMX on 127.0.0.3:7199, information
>>> won't be shown.
>>> Keyspace: TestKeyspace:
>>>  Replication Strategy:
>>> org.apache.cassandra.locator.NetworkTopologyStrategy
>>>  Durable Writes: true
>>>    Options: [replication_factor:2]
>>> ...
>>>
>>> default@unknown] update keyspace TestKeyspace with
>>> strategy_options=[{datacenter:2}];
>>> 47035d20-c2ca-11e0--de5e2f7aa3bf
>>> Waiting for schema agreement...
>>> ... schemas agree across the cluster
>>>
>>> [default@unknown] describe keyspace TestKeyspace;
>>> Keyspace: TestKeyspace:
>>>  Replication Strategy:
>>> org.apache.cassandra.locator.NetworkTopologyStrategy
>>>  Durable Writes: true
>>>    Options: [datacenter:2]
>>>
>>> Then restart the 0.8.3 node and it should be fine.
>>>
>>> Regards,
>>> -Martin
>>
>>
>

Re: Ec2Snitch

[Viliam Holub]

Yup, work perfectly now.

Thanks,
V.

On 10. Aug (Wednesday) v 20:45:36 -0500 2011, Brandon Williams wrote:
> You probably have other nodes that are NOT using the snitch yet, so
> they haven't populated DC/RACK info yet.  The exceptions will stop
> when all snitches have been changed.
> 
> On Wed, Aug 10, 2011 at 7:55 PM, Viliam Holub  wrote:
> >
> > Hi,
> >
> > I tried to switch to Ec2Snith. Although it correctly found the region:
> >
> > INFO 23:18:00,643 EC2Snitch using region: eu-west, zone: 1a.
> >
> > it started to report NullPointerException every second:
> >
> > ERROR 00:23:40,268 Internal error processing get_slice
> > java.lang.NullPointerException
> >        at 
> > org.apache.cassandra.locator.Ec2Snitch.getDatacenter(Ec2Snitch.java:93)
> >        at 
> > org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:122)
> >        at 
> > org.apache.cassandra.locator.OldNetworkTopologyStrategy.calculateNaturalEndpoints(OldNetworkTopologyStrategy.java:64)
> >        at 
> > org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:99)
> >        at 
> > org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:1708)
> >        at 
> > org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:1702)
> >        at 
> > org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:511)
> >        at 
> > org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:480)
> >        at 
> > org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:126)
> >        at 
> > org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:280)
> >        at 
> > org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:362)
> >        at 
> > org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:323)
> >        at 
> > org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:3033)
> >        at 
> > org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
> >        at 
> > org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> >        at 
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >        at 
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >        at java.lang.Thread.run(Thread.java:636)
> >
> > Am I doing something wrong?
> >
> > Thanks,
> > Viliam
> >
> >

-- 
Viliam Holub
School of Computer Science and Informatics
University College Dublin, Ireland

performance problems on new cluster

[Anton Winter]

Hi,

I have recently been migrating to a small 12 node Cassandra cluster 
spanning across 4 DC's and have been encountering various issues with 
what I suspect to be a performance tuning issue with my data set.  I've 
learnt a few lessons along the way but I'm at a bit of a roadblock now 
where I have been experiencing frequent OutOfMemory exceptions, various 
other exceptions, poor performance and my ring is appearing to become 
imbalanced during repairs.  I've tried various different configurations 
but haven't been able to get to the bottom of my performance issues.  
I'm assuming this has something to do with my data and some performance 
tuning metric that I'm merely overlooking.


My ring was created as documented in the wiki & various other 
performance tuning guides, calculating the tokens at each DC and 
incrementing when in conflict.  It is as follows:


Address DC  RackStatus State   Load
OwnsToken

   
113427455640312821154458202477256070487
dc1host1  dc1  1a  Up Normal  88.62 GB33.33%  0
dc2host1  dc2  1   Up Normal  14.76 GB0.00%   1
dc3host1dc3  1   Up Normal  15.99 GB
0.00%   2
dc4host1cd4  1   Up Normal  14.52 GB
0.00%   3
dc1host2   dc1  1a  Up Normal  18.02 GB
33.33%  56713727820156410577229101238628035242
dc2host2  dc2  1   Up Normal  16.5 GB 
0.00%   56713727820156410577229101238628035243
dc3host2 dc3  1   Up Normal  16.37 GB
0.00%   56713727820156410577229101238628035244
dc4host2dc4  1   Up Normal  13.34 GB
0.00%   56713727820156410577229101238628035245
dc1host3  dc1  1a  Up Normal  16.59 GB
33.33%  113427455640312821154458202477256070484
dc2host3   dc2  1   Up Normal  15.22 GB
0.00%   113427455640312821154458202477256070485
dc3host3   dc3  1   Up Normal  15.59 GB
0.00%   113427455640312821154458202477256070486
dc4host3dc4  1   Up Normal  8.84 GB 
0.00%   113427455640312821154458202477256070487


The above ring was freshly created and fairly evenly distributed in load 
prior to a repair (which is still running at the time of the above 
command) on dc1host1, however with the exception of dc4host3 where a 
previous bulk data load timed out.  dc4host3 was responding poorly, was 
failing according to other nodes and judging from its heap usage was 
rather close to OOM'ing before it was restarted.


I'm also using NTS with RF2.

The primary issues I'm experiencing are:

Light load against nodes in dc1 was causing OutOfMemory exceptions 
across all Cassandra servers outside of dc1 which were all idle and 
eventually after several hours happened on one of the dc1 nodes.  This 
issue was produced using svn trunk r1153002 and an in house written 
Snitched which effectively combined PropertyFileSnitch with some 
components of Ec2Snitch.  While trying to resolve these issues I have 
moved to a r1156490 snapshot and have switched across to just the 
PropertyFileSnitch and simply utilising the broadcast_address 
configuration option available in trunk which seems to work quite well.


Since moving to r1156490 we have stopped getting OOM's, but that may 
actually be because we have been unable to send traffic to the cluster 
to be able to produce one.


The most current issues I have been experiencing are the following:

1) thrift timeouts & general degraded response times
2) *lots* of exception errors, such as:

ERROR [ReadRepairStage:1076] 2011-08-11 13:33:41,266 
AbstractCassandraDaemon.java (line 133) Fatal exception in thread 
Thread[ReadRepairStage:1076,5,main]

java.lang.AssertionError
at 
org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:73)
at 
org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

at java.lang.Thread.run(Thread.java:662)

3) ring imbalances during a repair (refer to the above nodetool ring output)
4) regular failure detection when any node does something only 
moderately stressful, such as a repair or are under light load etc. but 
the node itself thinks it is fine.


My hosts are all 32Gb with either 4 or 16 cores, I've set heaps 
appropriately to half physical memory (16G) and for the purpose of 
cluster simplicity set all younggen to 400Mb.  JNA is in use, commitlogs 
and data have been split onto different filesystems and so on.


My data set as des

Cassandra on Java SE 7?

[Martin Lansler]

Hi,

Has anybody run Cassandra on Java SE 7? Any issues/caveats?

I'm on a mac so I have to wait a while :-(

-Martin

Running cassandra on a Blades + SAN

[David McNelis]

Hey folks,

I was wondering if anyone has a cassandra cluster running on a server setup
using blades, with a SAN appliance as the data file storage medium.

I would expect that there would be performance  let-downs  if the SAN was
connected with anything other than a fiber channel, but are there other
drawbacks?  Imagine a system with something along the lines of 12 600G 10k
SAS drives in a single SAN, with a bunch of dual proc, quad core blades as
the hosts.  I would think that we'd have the commitlog live on the blade's
internal drives would allow us to write data with sufficient throughput.

Anyone have any thoughts on a Blade v. Rackable solution for spinning up a
cassandra cluster?

-- 
*David McNelis*
Lead Software Engineer
Agentis Energy
www.agentisenergy.com
o: 630.359.6395
c: 219.384.5143

*A Smart Grid technology company focused on helping consumers of energy
control an often under-managed resource.*

Re: Running cassandra on a Blades + SAN

[Jonathan Ellis]

The SAN model defeats some of the point of using Cassandra, but a
"hybrid" of commitlog-local would be better than "everything on san."

http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-on-iSCSI-td5945217.html
is a good thread for background.

On Thu, Aug 11, 2011 at 8:56 AM, David McNelis
 wrote:
> Hey folks,
> I was wondering if anyone has a cassandra cluster running on a server setup
> using blades, with a SAN appliance as the data file storage medium.
> I would expect that there would be performance  let-downs  if the SAN was
> connected with anything other than a fiber channel, but are there other
> drawbacks?  Imagine a system with something along the lines of 12 600G 10k
> SAS drives in a single SAN, with a bunch of dual proc, quad core blades as
> the hosts.  I would think that we'd have the commitlog live on the blade's
> internal drives would allow us to write data with sufficient throughput.
> Anyone have any thoughts on a Blade v. Rackable solution for spinning up a
> cassandra cluster?
>
> --
> David McNelis
> Lead Software Engineer
> Agentis Energy
> www.agentisenergy.com
> o: 630.359.6395
> c: 219.384.5143
> A Smart Grid technology company focused on helping consumers of energy
> control an often under-managed resource.
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Solandra/Cassandra ring issue

[Ashley Martens]

No shared seeds. Downright freaky.

Re: Solandra/Cassandra ring issue

[Jake Luciani]

seriously,  If you change the cluster name in cassandra.yaml they won't
join.

On Thu, Aug 11, 2011 at 12:31 PM, Ashley Martens wrote:

> No shared seeds. Downright freaky.
>
>


-- 
http://twitter.com/tjake

[RELEASE] Apache Cassandra 0.8.4 released

[Sylvain Lebresne]

The Cassandra team is pleased to announce the release of Apache Cassandra
version 0.8.4.

Cassandra is a highly scalable second-generation distributed database,
bringing together Dynamo's fully distributed design and Bigtable's
ColumnFamily-based data model. You can read more here:

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

Cassandra 0.8.3 is only a few days old, but we just fixed an important bug
with counters and 0.8.3 had a potential minor upgrade annoyance, so here is
Cassandra 0.8.4. This version is thus a bug fix release[1] and as usual,
please pay attention to the release notes[2] before upgrading.

If you were to encounter any problem, let us know[3].

Have fun!

[1]: http://goo.gl/lqbW7 (CHANGES.txt)
[2]: http://goo.gl/8q5DM (NEWS.txt)
[3]: https://issues.apache.org/jira/browse/CASSANDRA

Re: Cassandra on Java SE 7?

[Edward Capriolo]

On Thu, Aug 11, 2011 at 11:32 AM, Martin Lansler
wrote:

> Hi,
>
> Has anybody run Cassandra on Java SE 7? Any issues/caveats?
>
> I'm on a mac so I have to wait a while :-(
>
> -Martin
>


Now mind you I am running Cassandra 0.7, Tomcat 7.0, and Hive 0.7, so I love
anything once it hits version 7 (except windows 7 that is) but something is
telling me to wait a while on java 7...

http://www.lucidimagination.com/blog/2011/07/28/dont-use-java-7-for-anything/

:)

Edward

Re: ColumnFamilyOutputFormat problem

[Jian Fang]

53 seconds included the map phase to read and process the input file. The
records were updated at the end of the reduce phase.

I checked the sales ranks in the update file and the sales ranks in the
Cassandra, they are different and thus, the records
were not actually updated.

I remember I run the word count example for Cassandra 0.8.0 some time ago, I
saw the similar behavior, i.e., the results were not written to the
Cassandra column family. But If I changed hadoop to write to the file
system, I can see the results in the output file. I didn't try the word
count example for Cassandra 0.8.2 though.

Anyway to solve this problem?

Thanks,

John
On Thu, Aug 11, 2011 at 12:17 AM, aaron morton wrote:

> I'm a simple guy. My first step would be see if the expected data is in the
> data base, if not what's missing.
>
> 2.5M updates / 3 nodes = 833,333 per node
> 833,333 / 53 seconds = 15,723 per second
> 1 / 15,723  = 0.6 seconds / 0.06 milliseconds per mutation
>
> sounds reasonable to me.
>
> check the Write Latency in nodetool cfstats and the row count estimates.
>
> Cheers
>
> -
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 11 Aug 2011, at 14:50, Jian Fang wrote:
>
> There are data and each Cassandra cluster node holds about 100G. From the
> application point of view, if I run the job twice with the same input file,
> i.e., the sales rank update file, then I should see a much smaller number of
> products, whose rank change exceeds the threshold, in the output file for
> the second run because the sales ranks have been updated to be the same as
> the ranks in the input file during the first run. But actually, I saw the
> output files stay the same for the two runs. One explanation is that the
> ranks were not actually updated for the first run. Also, 53 seconds to run
> the whole hadoop job with 2.5 million Cassandra updates on three nodes, do
> you think that is possible? Each node is a regular Linux box with 8 CPUs.
>
> Thanks,
>
> John
>
> On Wed, Aug 10, 2011 at 5:40 PM, aaron morton wrote:
>
>>  Seems the data are not actually written to Cassandra.
>>
>>
>> Before jumping into the Hadoop side of things are you saying there is no
>> data in Cassandra ? Can you retrieve any using the CLI  ? Take a look at
>> cfstats on each node to see the estimated record count.
>>
>> Cheers
>>
>>  -
>> Aaron Morton
>> Freelance Cassandra Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 11 Aug 2011, at 08:20, Jian Fang wrote:
>>
>> Hi,
>>
>> I am using Cassandra 0.8.2 with Hadoop 0.20.2. My application read a file
>> and then write about 2.5 million records
>> to Cassandra. I used ColumnFamilyOutputFormat to write to Cassandra. My
>> Cassandra cluster has three nodes with
>> one Hadoop task tracker on each node. The wired problem is that I only saw
>> one map and one reducer tasks and job only took
>> 53 seconds to finish. Seems the data are not actually written to
>> Cassandra.
>>
>> Here is status from Hadoop web admin:
>>
>> User: hadoop
>> Job Name: SalesRankWriter
>> Job File:
>> hdfs://x:54310/hadoop/tmp/mapred/system/job_201108051329_0060/job.xml
>> Job Setup: Successful
>> Status: Succeeded
>> Started at: Wed Aug 10 15:24:43 EDT 2011
>> Finished at: Wed Aug 10 15:25:36 EDT 2011
>> Finished in: 52sec
>> Job Cleanup: Successful
>> Kind % Complete Num Tasks Pending Running Complete Killed Failed/Killed
>> Task Attempts
>> map 100.00%
>> 1 0 0 1 0 0 / 0
>> reduce 100.00%
>> 1 0 0 1 0 0 / 0
>>
>> Counter Map Reduce Total
>> Job Counters Launched reduce tasks 0 0 1
>> Launched map tasks 0 0 1
>> Data-local map tasks 0 0 1
>> FileSystemCounters FILE_BYTES_READ 50,698,700 50,698,646 101,397,346
>> HDFS_BYTES_READ 56,149,360 0 56,149,360
>> FILE_BYTES_WRITTEN 101,397,378 50,698,646 152,096,024
>> Map-Reduce Framework Reduce input groups 0 2,534,932 2,534,932
>> Combine output records 0 0 0
>> Map input records 2,534,932 0 2,534,932
>> Reduce shuffle bytes 0 0 0
>> Reduce output records 0 2,534,932 2,534,932
>> Spilled Records 5,069,864 2,534,932 7,604,796
>> Map output bytes 45,628,776 0 45,628,776
>> Map output records 2,534,932 0 2,534,932
>> Combine input records 0 0 0
>> Reduce input records 0 2,534,932 2,534,932
>>
>> and the log for the mapper
>>
>>  2011-08-10 15:24:48,717 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>> Initializing JVM Metrics with processName=MAP, sessionId=
>> 2011-08-10 15:24:48,857 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb
>> = 100 2011-08-10 15:24:48,917 INFO org.apache.hadoop.mapred.MapTask: data
>> buffer = 79691776/99614720 2011-08-10 15:24:48,917 INFO
>> org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680 2011-08-10
>> 15:24:49,760 INFO org.apache.hadoop.mapred.MapTask: Spilling map output:
>> record full = true 2011-08-10 15:24:49,760 INFO
>> org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 4718592; bufvoid =
>> 99614720 2011-08-10 15:24:49,760 I

Re: Tuning a column family for archival

[Jason Baker]

On Thu, Aug 11, 2011 at 6:14 AM, Edward Capriolo wrote:
>
> In many regards Cassandra automatically does the correct thing. Other then
> the costs of the bloom filters for the table size being in ram, if you never
> read or write to those sstables and you are not reusing the row key, the OS
> will page out those tables and they will not take any cache space.
>

Forgive me if I'm being dense, but the wiki says that an instance shouldn't
be swapping at all[1].  I presently have swappiness turned down to 0.  Are
you saying that there may be benefits to allowing some swap usage?

[1]
http://wiki.apache.org/cassandra/MemtableThresholds?highlight=%28swap%29#Virtual_Memory_and_Swap

Re: Tuning a column family for archival

[Jonathan Ellis]

No.  He's saying that one of the points of mmaping the data files is
that the OS is free to only keep files that are actually used, in the
page cache.  Since this data is backed by an actual file swap is not
involved.

On Thu, Aug 11, 2011 at 12:59 PM, Jason Baker  wrote:
> On Thu, Aug 11, 2011 at 6:14 AM, Edward Capriolo 
> wrote:
>>
>> In many regards Cassandra automatically does the correct thing. Other then
>> the costs of the bloom filters for the table size being in ram, if you never
>> read or write to those sstables and you are not reusing the row key, the OS
>> will page out those tables and they will not take any cache space.
>
> Forgive me if I'm being dense, but the wiki says that an instance shouldn't
> be swapping at all[1].  I presently have swappiness turned down to 0.  Are
> you saying that there may be benefits to allowing some swap usage?
> [1] http://wiki.apache.org/cassandra/MemtableThresholds?highlight=%28swap%29#Virtual_Memory_and_Swap



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: migrating from 0.6 to 0.8, "java.io.IOError: ... cannot extend file to required size"

[aaron morton]

I did a test upgrade first and ran the scrub as part of that process to make 
sure everything was working.

Cheers

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 11 Aug 2011, at 21:47, Zhu Han wrote:

> 
> 
> On Wed, Aug 10, 2011 at 5:24 PM, aaron morton  wrote:
> I remember seeing this once before upgrading a system from 0.6 to 0.7 on a 
> Ubuntu EC2 (non data stax build) with EBS disks. I did the same thing and 
> just assumed it was an EBS or 0.6 bug. From memory after the upgrade that 
> node had some serialisation issues with a few SSTables that I had to 
> eventually take off the node.
> 
> I would recommend running nodetool scrub *before* the upgrade to ensure the 
> files can be read.
> 
> But "nodetool scrub" is provided by 0.7.3 or later. How can we run scrub 
> before upgrade?
>  
> 
> Cheers
> 
> -
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 10 Aug 2011, at 13:09, Jonathan Ellis wrote:
> 
> > What OS was this?
> >
> > On Tue, Aug 9, 2011 at 6:32 PM, ian douglas  wrote:
> >> Thanks to 'driftx' in IRC, adding this line to my cassandra.yaml file seems
> >> to have fixed the problem:
> >>
> >> disk_access_mode: standard
> >>
> >>
> >>
> >>
> >> On 08/09/2011 04:06 PM, ian douglas wrote:
> >>
> >> updated stack trace after fixing the corrupted GameSave2 tables:
> >>
> >>
> >>
> >>  INFO 16:05:20,314 Logging initialized
> >>  INFO 16:05:20,330 Heap size: 6290407424/6291456000
> >>  INFO 16:05:20,336 JNA not found. Native methods will be disabled.
> >>  INFO 16:05:20,344 Loading settings from
> >> file:/data/apache-cassandra-0.8.1/conf/cassandra.yaml
> >>  INFO 16:05:20,474 DiskAccessMode 'auto' determined to be mmap,
> >> indexAccessMode is mmap
> >>  INFO 16:05:20,627 Global memtable threshold is enabled at 2000MB
> >>  INFO 16:05:20,628 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:20,633 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:20,634 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:20,635 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:20,636 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:20,637 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:20,737 Opening /data/cassandra8/data/system/Schema-g-13
> >>  INFO 16:05:20,766 Opening /data/cassandra8/data/system/Migrations-g-13
> >>  INFO 16:05:20,770 Opening /data/cassandra8/data/system/LocationInfo-g-2
> >>  INFO 16:05:20,772 Opening /data/cassandra8/data/system/LocationInfo-g-1
> >>  INFO 16:05:20,821 Loading schema version
> >> 8fbed060-c2ae-11e0--242d50cf1fff
> >>  INFO 16:05:21,028 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,030 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,032 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,033 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,035 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,039 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,040 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,044 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,045 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,047 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,047 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,048 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,049 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,049 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,050 Removing compacted SSTable files (see
> >> http://wiki.apache.org/cassandra/MemtableSSTable)
> >>  INFO 16:05:21,054 Opening /data/cassandra8/data/armorgames/Feeds2-1777
> >>  INFO 16:05:21,774 Opening /data/cassandra8/data/armorgames/Status-764
> >>  INFO 16:05:21,800 Opening /data/cassandra8/data/armorgames/Comments-763
> >>  INFO 16:05:21,816 Opening

Re: Need help in CF design

[aaron morton]

>> 1) Is it possible to design to get equivalent results for above
>> query ( using CQL or Hector) with Cassandra.
If this is a common query in your app it's god idea to design the data model to 
support the request. Seems safe to assume  the PK in your example is non 
unique, I'll call it the FKID below and use PKID as a unique id for the entity

Relational Model:
PKID int (unique)
FKID int (non unique)
other_attribute int (non unique)

CF: AttributeIndex
row key: FKID
column name: CompositeType(attr_value:IntegerType, pkid:IntegerType)
column value: nothing 

You would then do a SliceRange over the rows. 

For a more detailed look check out this presentation from Ed Anuff at the 
Cassandra SF conference http://www.slideshare.net/edanuff/indexing-in-cassandra

Cheers

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 11 Aug 2011, at 23:06, Benoit Perroud wrote:

> You can apply this query really simply using cassandra and secondary indexes.
> 
> You will have a CF "TABLE", where row keys are your PK. Just to be sure of my 
> understanding, your SQL query will either return 1 row or no row, right ?
> 
> 3) SliceQuery returns a range of columns for a given key, it may be your 
> friend.
> 
> 
> 
> 
> 
> On 11. 08. 11 07:50, a...@9y.com wrote:
>> I recently started with Cassandra and found interesting.
>> 
>> I was curious in SQL we have
>> 
>> SELECT * from TABLE where PK="primary_key" and other_attribute
>> between 500 and 1000;
>> 
>> My questions are :
>> 1) Is it possible to design to get equivalent results for above
>> query ( using CQL or Hector) with Cassandra.
>> 
>> 2) How can we design CF in that case? Using secondary index is an
>> option but I am not clear how that can be applied here.
>> 
>> 3) Is there any way we can have a range slice over columns names
>> instead of having range or row keys.
>> 
>> I am just a novice. So, Can anyone help me with these question
>> 

Re: [RELEASE] Apache Cassandra 0.8.4 released

[Ian Danforth]

Would you be so kind as to announce the rpm release as well?

On Thu, Aug 11, 2011 at 10:52 AM, Sylvain Lebresne wrote:

> The Cassandra team is pleased to announce the release of Apache Cassandra
> version 0.8.4.
>
> Cassandra is a highly scalable second-generation distributed database,
> bringing together Dynamo's fully distributed design and Bigtable's
> ColumnFamily-based data model. You can read more here:
>
>  http://cassandra.apache.org/
>
> Downloads of source and binary distributions are listed in our download
> section:
>
>  http://cassandra.apache.org/download/
>
> Cassandra 0.8.3 is only a few days old, but we just fixed an important bug
> with counters and 0.8.3 had a potential minor upgrade annoyance, so here is
> Cassandra 0.8.4. This version is thus a bug fix release[1] and as usual,
> please pay attention to the release notes[2] before upgrading.
>
> If you were to encounter any problem, let us know[3].
>
> Have fun!
>
> [1]: http://goo.gl/lqbW7 (CHANGES.txt)
> [2]: http://goo.gl/8q5DM (NEWS.txt)
> [3]: https://issues.apache.org/jira/browse/CASSANDRA
>

Re: Best practices when deploying & upgrading a cassandra cluster

[aaron morton]

In a non dev system it's a lot easier to use the packages 
http://wiki.apache.org/cassandra/DebianPackaging
http://www.datastax.com/docs/0.8/install/packaged_releases

Cheers

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 12 Aug 2011, at 02:30, Martin Lansler wrote:

> (Note: This is a repost from another thread which did not have a
> relevant subject, sorry for the spamming)
> 
> Hi Eldad / All,
> 
> On Wed, Aug 10, 2011 at 8:32 AM, Eldad Yamin  wrote:
>> Can you please explain how did you upgraded. something like step-by-step.
>> Thanks!
> 
> I took the liberty of replying to the group as it would be interesting
> to hear how other folks out there are doing it...
> 
> I'm *not* running a prod system, just a test system of three nodes on
> my laptop. So it would be nice to hear about real setups. Here is my
> test setup:
> 
> apache-cassandra -> apache-cassandra-0.8.3
> apache-cassandra-0.8.2/
> apache-cassandra-0.8.3/
> node1/
> node2/
> node3/
> 
> All nodeX look like:
> bin -> ../apache-cassandra/bin/
> commitlog/
> conf/
> data/
> interface -> ../apache-cassandra/interface/
> lib -> ../apache-cassandra/lib/
> saved_caches/
> 
> The 'conf' directory is copied into each node from the virgin
> cassandra distribution. I then create a local GIT repo and add the
> 'conf' directory so I can track any configuration changes on a node.
> Then relevant node specific configuration settings are set. The
> 'commitlog', 'data' and 'saved_caches' are created by cassandra and
> must be configured in 'cassandra.yaml' for each node.
> 
> When I upgrade I do the following:
> 
> 1.
> Make a diff of the new conf files from the new version so that  get
> new parameters etc... I use emacs ediff-mode.
> 2.
> Remove the old "apache-cassandra" symlink and point it to the new cassandra 
> dist
> 3.
> In a rolling fashion stop one node, and then restart it... as the
> symlink is changes it will then boot with the upgraded cassandra dist.
> (remember to cd out & in of the bin/ dir otherwise you will still be
> in the old directory).
> (4).
> Should something break... just re-create the old symlink and restart
> the node (provided cassandra has not performed any non backwards
> compatible changes to the db files, should be noted in the README)
> 
> That's pretty much it.
> 
> On a prod setup one would probably use a tool such as puppet
> (www.puppetlabs.com/) to ease setting up on many nodes... But there
> are many ways to do this, for instance pssh
> (http://code.google.com/p/parallel-ssh/).
> 
> Regards,
> -Martin

Client traffic encryption best practices....

[Chris Marino]

Hello, is there any consensus on how to secure client/cluster
communications???

I'm running an 8 node cluster across EC2 regions.  I'm running inter-node
encryption and I want to encrypt the traffic from the clients as well.

My options seem to be:

Have the client connect to only one node and encrypt that one connection
with OpenVPN/stunnel (or something similar). Or, set up an encrypted tunnel
from the client to each node. Is there a client library that could take care
of this for me??

Setting up tunnels to each node is a major pain, but pointing the client to
only one node is going to kill my performance.  I'm running 4 nodes in each
EC2 region with one client in each. Maybe I could connect the client only to
the local nodes, which should simplify things a bit, but I was wondering if
anyone had any experience with this or could suggest something that might be
better.

Please let me know.
Thanks.
CM

Re: tpstats confusion

[Ian Danforth]

I don't think so, perhaps more succinctly, why would ROW actions only be
performed on a single node?

Ian

On Wed, Aug 10, 2011 at 8:12 PM, Jonathan Ellis  wrote:

> are you i/o bound?  I believe counter ROW needs to perform a read of
> the old value.
>
> On Wed, Aug 10, 2011 at 7:17 PM, Ian Danforth 
> wrote:
> > Hello all!
> > Thanks for taking the time to read this, I'm new to Cassandra and really
> > want to get it working :)
> > Below you'll find the output from tpstats on the three nodes in my .8.1
> > cluster. As you'll note from ring (also below), my cluster appears
> balanced.
> > However, please note that NODE1 is the only node that has completed any
> > ReplicateOnWriteStage actions. I've been having a lot of trouble with RoW
> > and seen a single node in my various test clusters get hugely backed up
> on
> > this stage to the point where it slows all writes to a crawl.
> > I'm using counters in a column family defined like this:
> >  KEYSPACE/CF 
> > create keyspace Keyspace1
> > with strategy_options=[{replication_factor:3}]
> > and placement_strategy =
> 'org.apache.cassandra.locator.SimpleStrategy';
> > use Keyspace1;
> > create column family TwitterTest
> > with default_validation_class = CounterColumnType
> > and replicate_on_write = true;
> > 
> >
> > With any given batch_insert() I'm incrementing ~17,000 counters. I've
> backed
> > off a bit and started using ConsistencyLevel.QUORUM, as it was suggested
> > elsewhere that ANY or ONE could overwhelm the system.
> >
> > I hope I'm just doing something stupid and one you experienced folks can
> > point out my error.
> > Thanks in advance!
> > Ian
> >  MORE DATA 
> > Address DC  RackStatus State   Load
>  Owns
> >Token
> >
> >113427455640312821154458202477256070484
> > 10.87.9.57  datacenter1 rack1   Up Normal  193.86 MB
> > 33.33%  0
> > 10.86.223.32datacenter1 rack1   Up Normal  191.1 MB
> >  33.33%  56713727820156410577229101238628035242
> > 10.86.229.24datacenter1 rack1   Up Normal  193.63 MB
> > 33.33%  113427455640312821154458202477256070484
> >
> >
> >
> > SEED
> > Pool NameActive   Pending  Completed
> > ReadStage 0 0  7
> > RequestResponseStage  0 01479839
> > MutationStage 0 01479908
> > ReadRepairStage   0 0  0
> > ReplicateOnWriteStage 0 0  0
> > GossipStage   0 0   5176
> > AntiEntropyStage  0 0  0
> > MigrationStage0 0128
> > MemtablePostFlusher   0 0 28
> > StreamStage   0 0  0
> > FlushWriter   0 0 28
> > MiscStage 0 0  2
> > FlushSorter   0 0  0
> > InternalResponseStage 0 0363
> > HintedHandoff 0 0  0
> > NODE1
> > Pool NameActive   Pending  Completed
> > ReadStage 0 0  7
> > RequestResponseStage  0 02960208
> > MutationStage 0 01480104
> > ReadRepairStage   0 0  0
> > ReplicateOnWriteStage 0 01480104
> > GossipStage   0 0   4151
> > AntiEntropyStage  0 0  0
> > MigrationStage0 0128
> > MemtablePostFlusher   0 0 31
> > StreamStage   0 0  8
> > FlushWriter   0 0 31
> > MiscStage 0 0  8
> > FlushSorter   0 0  0
> > InternalResponseStage 0 0  1
> > HintedHandoff 0 0  0
> > NODE2
> > Pool NameActive   Pending  Completed
> > ReadStage 0 0  7
> > RequestResponseStage  0 0  0
> > MutationStage 0 01481530
> > ReadRepairStage   0 0  0
> > ReplicateOnWriteStage 0 0  0
> > GossipStage   0 0   4201
> > AntiEntropyStage  0 0  0
> > MigrationStage0 0128
> > MemtablePostFlusher   0 0 31
> > StreamStage   0 0  6
> > FlushWr

Re: tpstats confusion

[Jonathan Ellis]

https://issues.apache.org/jira/browse/CASSANDRA-2889

On Thu, Aug 11, 2011 at 2:55 PM, Ian Danforth  wrote:
> I don't think so, perhaps more succinctly, why would ROW actions only be
> performed on a single node?
> Ian
>
> On Wed, Aug 10, 2011 at 8:12 PM, Jonathan Ellis  wrote:
>>
>> are you i/o bound?  I believe counter ROW needs to perform a read of
>> the old value.
>>
>> On Wed, Aug 10, 2011 at 7:17 PM, Ian Danforth 
>> wrote:
>> > Hello all!
>> > Thanks for taking the time to read this, I'm new to Cassandra and really
>> > want to get it working :)
>> > Below you'll find the output from tpstats on the three nodes in my .8.1
>> > cluster. As you'll note from ring (also below), my cluster appears
>> > balanced.
>> > However, please note that NODE1 is the only node that has completed any
>> > ReplicateOnWriteStage actions. I've been having a lot of trouble with
>> > RoW
>> > and seen a single node in my various test clusters get hugely backed up
>> > on
>> > this stage to the point where it slows all writes to a crawl.
>> > I'm using counters in a column family defined like this:
>> >  KEYSPACE/CF 
>> > create keyspace Keyspace1
>> >     with strategy_options=[{replication_factor:3}]
>> >     and placement_strategy =
>> > 'org.apache.cassandra.locator.SimpleStrategy';
>> > use Keyspace1;
>> > create column family TwitterTest
>> >     with default_validation_class = CounterColumnType
>> >     and replicate_on_write = true;
>> > 
>> >
>> > With any given batch_insert() I'm incrementing ~17,000 counters. I've
>> > backed
>> > off a bit and started using ConsistencyLevel.QUORUM, as it was suggested
>> > elsewhere that ANY or ONE could overwhelm the system.
>> >
>> > I hope I'm just doing something stupid and one you experienced folks can
>> > point out my error.
>> > Thanks in advance!
>> > Ian
>> >  MORE DATA 
>> > Address         DC          Rack        Status State   Load
>> >  Owns
>> >    Token
>> >
>> >    113427455640312821154458202477256070484
>> > 10.87.9.57      datacenter1 rack1       Up     Normal  193.86 MB
>> > 33.33%  0
>> > 10.86.223.32    datacenter1 rack1       Up     Normal  191.1 MB
>> >  33.33%  56713727820156410577229101238628035242
>> > 10.86.229.24    datacenter1 rack1       Up     Normal  193.63 MB
>> > 33.33%  113427455640312821154458202477256070484
>> >
>> >
>> >
>> > SEED
>> > Pool Name                    Active   Pending      Completed
>> > ReadStage                         0         0              7
>> > RequestResponseStage              0         0        1479839
>> > MutationStage                     0         0        1479908
>> > ReadRepairStage                   0         0              0
>> > ReplicateOnWriteStage             0         0              0
>> > GossipStage                       0         0           5176
>> > AntiEntropyStage                  0         0              0
>> > MigrationStage                    0         0            128
>> > MemtablePostFlusher               0         0             28
>> > StreamStage                       0         0              0
>> > FlushWriter                       0         0             28
>> > MiscStage                         0         0              2
>> > FlushSorter                       0         0              0
>> > InternalResponseStage             0         0            363
>> > HintedHandoff                     0         0              0
>> > NODE1
>> > Pool Name                    Active   Pending      Completed
>> > ReadStage                         0         0              7
>> > RequestResponseStage              0         0        2960208
>> > MutationStage                     0         0        1480104
>> > ReadRepairStage                   0         0              0
>> > ReplicateOnWriteStage             0         0        1480104
>> > GossipStage                       0         0           4151
>> > AntiEntropyStage                  0         0              0
>> > MigrationStage                    0         0            128
>> > MemtablePostFlusher               0         0             31
>> > StreamStage                       0         0              8
>> > FlushWriter                       0         0             31
>> > MiscStage                         0         0              8
>> > FlushSorter                       0         0              0
>> > InternalResponseStage             0         0              1
>> > HintedHandoff                     0         0              0
>> > NODE2
>> > Pool Name                    Active   Pending      Completed
>> > ReadStage                         0         0              7
>> > RequestResponseStage              0         0              0
>> > MutationStage                     0         0        1481530
>> > ReadRepairStage                   0         0              0
>> > ReplicateOnWriteStage             0         0              0
>> > GossipStage                       0         0           4201
>> > AntiEntropyStage                  0    

Re: performance problems on new cluster

[aaron morton]

Is there a reason you are using the trunk and not one of the tagged releases? 
Official releases are a lot more stable than the trunk. 

> 1) thrift timeouts & general degraded response times
For read or writes ? What sort of queries are you running ? Check the local 
latency on each node using cfstats and cfhistogram, and a bit of iostat 
http://spyced.blogspot.com/2010/01/linux-performance-basics.html What does 
nodetool tpstats say, is there a stage backing up?

If the local latency is OK look at the cross DC situation. What CL are you 
using? Are nodes timing out waiting for nodes in other DC's ? 

> 2) *lots* of exception errors, such as:
Repair is trying to run on a response which is a digest response, this should 
not be happening. Can you provide some more info on the type of query you are 
running ? 

> 3) ring imbalances during a repair (refer to the above nodetool ring output)
You may be seeing this
https://issues.apache.org/jira/browse/CASSANDRA-2280
I think it's a mistake that is it marked as resolved. 

> 4) regular failure detection when any node does something only moderately 
> stressful, such as a repair or are under light load etc. but the node itself 
> thinks it is fine.
What version are you using ? 

I'd take a look at the exceptions first then move onto the performance issues. 

Cheers

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 12 Aug 2011, at 03:16, Anton Winter wrote:

> Hi,
> 
> I have recently been migrating to a small 12 node Cassandra cluster spanning 
> across 4 DC's and have been encountering various issues with what I suspect 
> to be a performance tuning issue with my data set.  I've learnt a few lessons 
> along the way but I'm at a bit of a roadblock now where I have been 
> experiencing frequent OutOfMemory exceptions, various other exceptions, poor 
> performance and my ring is appearing to become imbalanced during repairs.  
> I've tried various different configurations but haven't been able to get to 
> the bottom of my performance issues.  I'm assuming this has something to do 
> with my data and some performance tuning metric that I'm merely overlooking.
> 
> My ring was created as documented in the wiki & various other performance 
> tuning guides, calculating the tokens at each DC and incrementing when in 
> conflict.  It is as follows:
> 
> Address DC  RackStatus State   LoadOwns   
>  Token
>   
> 113427455640312821154458202477256070487
> dc1host1  dc1  1a  Up Normal  88.62 GB33.33%  0
> dc2host1  dc2  1   Up Normal  14.76 GB0.00%   1
> dc3host1dc3  1   Up Normal  15.99 GB0.00%   2
> dc4host1cd4  1   Up Normal  14.52 GB0.00%   3
> dc1host2   dc1  1a  Up Normal  18.02 GB33.33%  
> 56713727820156410577229101238628035242
> dc2host2  dc2  1   Up Normal  16.5 GB 0.00%   
> 56713727820156410577229101238628035243
> dc3host2 dc3  1   Up Normal  16.37 GB0.00%   
> 56713727820156410577229101238628035244
> dc4host2dc4  1   Up Normal  13.34 GB0.00%   
> 56713727820156410577229101238628035245
> dc1host3  dc1  1a  Up Normal  16.59 GB33.33%  
> 113427455640312821154458202477256070484
> dc2host3   dc2  1   Up Normal  15.22 GB0.00%   
> 113427455640312821154458202477256070485
> dc3host3   dc3  1   Up Normal  15.59 GB0.00%   
> 113427455640312821154458202477256070486
> dc4host3dc4  1   Up Normal  8.84 GB 0.00%   
> 113427455640312821154458202477256070487
> 
> The above ring was freshly created and fairly evenly distributed in load 
> prior to a repair (which is still running at the time of the above command) 
> on dc1host1, however with the exception of dc4host3 where a previous bulk 
> data load timed out.  dc4host3 was responding poorly, was failing according 
> to other nodes and judging from its heap usage was rather close to OOM'ing 
> before it was restarted.
> 
> I'm also using NTS with RF2.
> 
> The primary issues I'm experiencing are:
> 
> Light load against nodes in dc1 was causing OutOfMemory exceptions across all 
> Cassandra servers outside of dc1 which were all idle and eventually after 
> several hours happened on one of the dc1 nodes.  This issue was produced 
> using svn trunk r1153002 and an in house written Snitched which effectively 
> combined PropertyFileSnitch with some components of Ec2Snitch.  While trying 
> to resolve these issues I have moved to a r1156490 snapshot and have switched 
> across to just the PropertyFileSnitch and simply utilising the 
> broadcast_address configuration option available in trunk which seems to work 
>

Re: ColumnFamilyOutputFormat problem

[aaron morton]

Turn the logging up in cassandra or your MR job and make sure the 
batch_mutation is sent. Sounds like it's not. 

Cheers

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 12 Aug 2011, at 07:22, Jian Fang wrote:

> 53 seconds included the map phase to read and process the input file. The 
> records were updated at the end of the reduce phase.
> 
> I checked the sales ranks in the update file and the sales ranks in the 
> Cassandra, they are different and thus, the records
> were not actually updated.
> 
> I remember I run the word count example for Cassandra 0.8.0 some time ago, I 
> saw the similar behavior, i.e., the results were not written to the Cassandra 
> column family. But If I changed hadoop to write to the file system, I can see 
> the results in the output file. I didn't try the word count example for 
> Cassandra 0.8.2 though.
> 
> Anyway to solve this problem?
> 
> Thanks,
> 
> John 
> On Thu, Aug 11, 2011 at 12:17 AM, aaron morton  
> wrote:
> I'm a simple guy. My first step would be see if the expected data is in the 
> data base, if not what's missing. 
> 
> 2.5M updates / 3 nodes = 833,333 per node 
> 833,333 / 53 seconds = 15,723 per second 
> 1 / 15,723  = 0.6 seconds / 0.06 milliseconds per mutation 
>  
> sounds reasonable to me. 
> 
> check the Write Latency in nodetool cfstats and the row count estimates. 
> 
> Cheers
> 
> -
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 11 Aug 2011, at 14:50, Jian Fang wrote:
> 
>> There are data and each Cassandra cluster node holds about 100G. From the 
>> application point of view, if I run the job twice with the same input file, 
>> i.e., the sales rank update file, then I should see a much smaller number of 
>> products, whose rank change exceeds the threshold, in the output file for 
>> the second run because the sales ranks have been updated to be the same as 
>> the ranks in the input file during the first run. But actually, I saw the 
>> output files stay the same for the two runs. One explanation is that the 
>> ranks were not actually updated for the first run. Also, 53 seconds to run 
>> the whole hadoop job with 2.5 million Cassandra updates on three nodes, do 
>> you think that is possible? Each node is a regular Linux box with 8 CPUs. 
>> 
>> Thanks,
>> 
>> John  
>> 
>> On Wed, Aug 10, 2011 at 5:40 PM, aaron morton  
>> wrote:
>>>  Seems the data are not actually written to Cassandra.
>> 
>> Before jumping into the Hadoop side of things are you saying there is no 
>> data in Cassandra ? Can you retrieve any using the CLI  ? Take a look at 
>> cfstats on each node to see the estimated record count.
>> 
>> Cheers
>>  
>> -
>> Aaron Morton
>> Freelance Cassandra Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>> 
>> On 11 Aug 2011, at 08:20, Jian Fang wrote:
>> 
>>> Hi,
>>> 
>>> I am using Cassandra 0.8.2 with Hadoop 0.20.2. My application read a file 
>>> and then write about 2.5 million records
>>> to Cassandra. I used ColumnFamilyOutputFormat to write to Cassandra. My 
>>> Cassandra cluster has three nodes with
>>> one Hadoop task tracker on each node. The wired problem is that I only saw 
>>> one map and one reducer tasks and job only took
>>> 53 seconds to finish. Seems the data are not actually written to Cassandra.
>>> 
>>> Here is status from Hadoop web admin:
>>> 
>>> User: hadoop
>>> Job Name: SalesRankWriter
>>> Job File: 
>>> hdfs://x:54310/hadoop/tmp/mapred/system/job_201108051329_0060/job.xml
>>> Job Setup: Successful
>>> Status: Succeeded
>>> Started at: Wed Aug 10 15:24:43 EDT 2011
>>> Finished at: Wed Aug 10 15:25:36 EDT 2011
>>> Finished in: 52sec
>>> Job Cleanup: Successful
>>> Kind% Complete  Num Tasks   Pending Running Complete
>>> Killed  Failed/Killed
>>> Task Attempts
>>> map 100.00%
>>> 1   0   0   1   0   0 / 0
>>> reduce  100.00%
>>> 1   0   0   1   0   0 / 0
>>> 
>>> Counter Map Reduce  Total
>>> Job CountersLaunched reduce tasks   0   0   1
>>> Launched map tasks  0   0   1
>>> Data-local map tasks0   0   1
>>> FileSystemCounters  FILE_BYTES_READ 50,698,700  50,698,646  
>>> 101,397,346
>>> HDFS_BYTES_READ 56,149,360  0   56,149,360
>>> FILE_BYTES_WRITTEN  101,397,378 50,698,646  152,096,024
>>> Map-Reduce FrameworkReduce input groups 0   2,534,932   
>>> 2,534,932
>>> Combine output records  0   0   0
>>> Map input records   2,534,932   0   2,534,932
>>> Reduce shuffle bytes0   0   0
>>> Reduce output records   0   2,534,932   2,534,932
>>> Spilled Records 5,069,864   2,534,932   7,604,796
>>> Map output bytes45,628,776  0   45,628,776
>>> Map output records  2,534,932   0   2,534,932
>>> Combine input records  

Re: tpstats confusion

[Ian Danforth]

I'm writing at QUORUM though and (pardon me for being dense) it would seem
that the first replica shouldn't always be on the same server if I'm using
RP.

I very much appreciate your time, I'm sure there is something basic that
just hasn't clicked.

Ian

On Thu, Aug 11, 2011 at 2:56 PM, Jonathan Ellis  wrote:

> https://issues.apache.org/jira/browse/CASSANDRA-2889
>
> On Thu, Aug 11, 2011 at 2:55 PM, Ian Danforth 
> wrote:
> > I don't think so, perhaps more succinctly, why would ROW actions only be
> > performed on a single node?
> > Ian
> >
> > On Wed, Aug 10, 2011 at 8:12 PM, Jonathan Ellis 
> wrote:
> >>
> >> are you i/o bound?  I believe counter ROW needs to perform a read of
> >> the old value.
> >>
> >> On Wed, Aug 10, 2011 at 7:17 PM, Ian Danforth 
> >> wrote:
> >> > Hello all!
> >> > Thanks for taking the time to read this, I'm new to Cassandra and
> really
> >> > want to get it working :)
> >> > Below you'll find the output from tpstats on the three nodes in my
> .8.1
> >> > cluster. As you'll note from ring (also below), my cluster appears
> >> > balanced.
> >> > However, please note that NODE1 is the only node that has completed
> any
> >> > ReplicateOnWriteStage actions. I've been having a lot of trouble with
> >> > RoW
> >> > and seen a single node in my various test clusters get hugely backed
> up
> >> > on
> >> > this stage to the point where it slows all writes to a crawl.
> >> > I'm using counters in a column family defined like this:
> >> >  KEYSPACE/CF 
> >> > create keyspace Keyspace1
> >> > with strategy_options=[{replication_factor:3}]
> >> > and placement_strategy =
> >> > 'org.apache.cassandra.locator.SimpleStrategy';
> >> > use Keyspace1;
> >> > create column family TwitterTest
> >> > with default_validation_class = CounterColumnType
> >> > and replicate_on_write = true;
> >> > 
> >> >
> >> > With any given batch_insert() I'm incrementing ~17,000 counters. I've
> >> > backed
> >> > off a bit and started using ConsistencyLevel.QUORUM, as it was
> suggested
> >> > elsewhere that ANY or ONE could overwhelm the system.
> >> >
> >> > I hope I'm just doing something stupid and one you experienced folks
> can
> >> > point out my error.
> >> > Thanks in advance!
> >> > Ian
> >> >  MORE DATA 
> >> > Address DC  RackStatus State   Load
> >> >  Owns
> >> >Token
> >> >
> >> >113427455640312821154458202477256070484
> >> > 10.87.9.57  datacenter1 rack1   Up Normal  193.86 MB
> >> > 33.33%  0
> >> > 10.86.223.32datacenter1 rack1   Up Normal  191.1 MB
> >> >  33.33%  56713727820156410577229101238628035242
> >> > 10.86.229.24datacenter1 rack1   Up Normal  193.63 MB
> >> > 33.33%  113427455640312821154458202477256070484
> >> >
> >> >
> >> >
> >> > SEED
> >> > Pool NameActive   Pending  Completed
> >> > ReadStage 0 0  7
> >> > RequestResponseStage  0 01479839
> >> > MutationStage 0 01479908
> >> > ReadRepairStage   0 0  0
> >> > ReplicateOnWriteStage 0 0  0
> >> > GossipStage   0 0   5176
> >> > AntiEntropyStage  0 0  0
> >> > MigrationStage0 0128
> >> > MemtablePostFlusher   0 0 28
> >> > StreamStage   0 0  0
> >> > FlushWriter   0 0 28
> >> > MiscStage 0 0  2
> >> > FlushSorter   0 0  0
> >> > InternalResponseStage 0 0363
> >> > HintedHandoff 0 0  0
> >> > NODE1
> >> > Pool NameActive   Pending  Completed
> >> > ReadStage 0 0  7
> >> > RequestResponseStage  0 02960208
> >> > MutationStage 0 01480104
> >> > ReadRepairStage   0 0  0
> >> > ReplicateOnWriteStage 0 01480104
> >> > GossipStage   0 0   4151
> >> > AntiEntropyStage  0 0  0
> >> > MigrationStage0 0128
> >> > MemtablePostFlusher   0 0 31
> >> > StreamStage   0 0  8
> >> > FlushWriter   0 0 31
> >> > MiscStage 0 0  8
> >> > FlushSorter   0 0  0
> >> > InternalResponseStage 0 0  1
> >> > HintedHandoff 0 0  

RE: sstableloader throws storage_port error

[Tom Davidson]

I am trying to sstableloader and I do not want to access Cassandra on the same 
node. I haved edited my Cassandra.yaml to with appropriate values for the 
listen_address and rpc_address but I keep getting the error below. The 
Cassandra-cli tool, nodetool etc. works find when trying to connect to my 
Cassandra cluster, but sstableloader does not. Any suggestions?

[tdavidson@nadevsan06 ~]$ sstableloader --debug -v Demo
Starting client (and waiting 30 seconds for gossip) ...
org.apache.cassandra.config.ConfigurationException: Unable to bind to address 
nadevsan04/10.168.121.57:7000. Set listen_address in cassandra.yaml to an 
interface you can bind to, e.g., your private IP address on EC2
java.lang.RuntimeException: org.apache.cassandra.config.ConfigurationException: 
Unable to bind to address nadevsan04/10.168.121.57:7000. Set listen_address in 
cassandra.yaml to an interface you can bind to, e.g., your private IP address 
on EC2
at 
org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:225)
at 
org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:104)
at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:61)
Caused by: org.apache.cassandra.config.ConfigurationException: Unable to bind 
to address nadevsan04/10.168.121.57:7000. Set listen_address in cassandra.yaml 
to an interface you can bind to, e.g., your private IP address on EC2
at 
org.apache.cassandra.net.MessagingService.getServerSocket(MessagingService.java:220)
at 
org.apache.cassandra.net.MessagingService.listen(MessagingService.java:191)
at 
org.apache.cassandra.service.StorageService.initClient(StorageService.java:350)
at 
org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:182)
... 2 more

From: John Conwell [mailto:j...@iamjohn.me]
Sent: Tuesday, July 26, 2011 12:11 PM
To: user@cassandra.apache.org
Subject: Re: sstableloader throws storage_port error

After much research and experimentation, I figured out how to get sstableloader 
running on the same machine as a live cassandra node instance.

The key, as Jonathan stated is to configure sstableloader to use a different 
ipaddress than the running cassandra instance is using.  To do this, I ran this 
command, which created the loopback address for 127.0.0.2


sudo ifconfig lo0 alias 127.0.0.2

No you can have cassandra configured to listen to 127.0.0.1, and sstableloader 
configured to listen to 127.0.0.2

By the way, to remove this ipaddress, run

sudo ifconfig lo0 -alias 127.0.0.2

But thats not really all.  Because sstableloader reads the cassandra.yaml file 
to get the gossip ipaddress, you need to make a copy of the cassandra install 
directory (or at least the bin and conf folders).  Basically one folder with 
yaml configured for Cassandra, the other folder with yaml configured for 
sstableloader.

Hope this helps people. I've written an in depth description of how to do all 
this, and can post it if people want, but I'm not sure the etiquette of posting 
blog links in the email list.

Thanks,
John

On Tue, Jul 26, 2011 at 7:40 AM, John Conwell 
mailto:j...@iamjohn.me>> wrote:
If I have Cassandra already running on my machine, how do I configure 
sstableloader to run on a different IP (127.0.0.2).  Also, does that mean in 
order to use sstableloader on the same machine as an running Cassandra node, I 
have to have two NIC cards?

I looked around for any info about how to configure and run sstableloader, but 
other than what the cmdline spits out I cant find anything.  Are there any 
examples or best practices?  Is it designed to be run on a machine that isn't 
running a cassandra node?

On Mon, Jul 25, 2011 at 8:24 PM, Jonathan Ellis 
mailto:jbel...@gmail.com>> wrote:
sstableloader uses gossip to discover the Cassandra ring, so you'll
need to run it on a different IP (127.0.0.2 is fine).

On Mon, Jul 25, 2011 at 2:41 PM, John Conwell 
mailto:j...@iamjohn.me>> wrote:
> I'm trying to figure out how to use the sstableloader tool.  For my test I
> have a single node cassandra instance running on my local machine.  I have
> cassandra running, and validate this by connecting to it with cassandra-cli.
> I run sstableloader using the following command:
> bin/sstableloader /Users/someuser/cassandra/mykeyspace
> and I get the following error:
> org.apache.cassandra.config.ConfigurationException: 
> localhost/127.0.0.1:7000
> is in use by another process.  Change listen_address:storage_port in
> cassandra.yaml to values that do not conflict with other services
>
> I've played around with different ports, but nothing works.  It it because
> I'm trying to run sstableloader on the same machine that cassandra is
> running on?  It would be odd I think, but cant thing of another reason I
> would get that eror.
> Thanks,
> John


--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
htt

Re: [RELEASE] Apache Cassandra 0.8.4 released

[Nate McCall]

RPMs for 0.8.4 are available via http://rpm.datastax.com/

Note - the Apache Cassandra team has nothing to do with the above
RPMs. We (DataStax) maintain this repository independently and try to
deploy them as close to the official releases as possible.

On Thu, Aug 11, 2011 at 4:14 PM, Ian Danforth  wrote:
> Would you be so kind as to announce the rpm release as well?
> On Thu, Aug 11, 2011 at 10:52 AM, Sylvain Lebresne 
> wrote:
>>
>> The Cassandra team is pleased to announce the release of Apache Cassandra
>> version 0.8.4.
>>
>> Cassandra is a highly scalable second-generation distributed database,
>> bringing together Dynamo's fully distributed design and Bigtable's
>> ColumnFamily-based data model. You can read more here:
>>
>>  http://cassandra.apache.org/
>>
>> Downloads of source and binary distributions are listed in our download
>> section:
>>
>>  http://cassandra.apache.org/download/
>>
>> Cassandra 0.8.3 is only a few days old, but we just fixed an important bug
>> with counters and 0.8.3 had a potential minor upgrade annoyance, so here
>> is
>> Cassandra 0.8.4. This version is thus a bug fix release[1] and as usual,
>> please pay attention to the release notes[2] before upgrading.
>>
>> If you were to encounter any problem, let us know[3].
>>
>> Have fun!
>>
>> [1]: http://goo.gl/lqbW7 (CHANGES.txt)
>> [2]: http://goo.gl/8q5DM (NEWS.txt)
>> [3]: https://issues.apache.org/jira/browse/CASSANDRA
>
>

Re: performance problems on new cluster

[Anton Winter]

Is there a reason you are using the trunk and not one of the tagged 
releases? Official releases are a lot more stable than the trunk.


Yes, as we are using a combination of Ec2 and colo servers we are 
needing to use broadcast_address from CASSANDRA-2491.  The patch that is 
associated with that JIRA does not apply cleanly against 0.8 so this is 
why we are using trunk.



1) thrift timeouts & general degraded response times
For read or writes ? What sort of queries are you running ? Check the 
local latency on each node using cfstats and cfhistogram, and a bit of 
iostat 
http://spyced.blogspot.com/2010/01/linux-performance-basics.html What 
does nodetool tpstats say, is there a stage backing up?


If the local latency is OK look at the cross DC situation. What CL are 
you using? Are nodes timing out waiting for nodes in other DC's ?


iostat doesn't show a request queue bottleneck.  The timeouts we are 
seeing is for reads.  The latency on the nodes I have temporarily used 
for reads is around 2-45ms.  The next token in the ring at an alternate 
DC is showing ~4ms with everything else around 0.05ms.  tpstats desn't 
show any active/pending.  Reads are at CL.ONE & Writes using CL.ANY





2) *lots* of exception errors, such as:
Repair is trying to run on a response which is a digest response, this 
should not be happening. Can you provide some more info on the type of 
query you are running ?



The query being run is  get cf1['user-id']['seg']


3) ring imbalances during a repair (refer to the above nodetool ring 
output)

You may be seeing this
https://issues.apache.org/jira/browse/CASSANDRA-2280
I think it's a mistake that is it marked as resolved.

What can I do in regards to confirming this issue is still outstanding 
and/or we are affected by it?


4) regular failure detection when any node does something only 
moderately stressful, such as a repair or are under light load etc. 
but the node itself thinks it is fine.

What version are you using ?

Version of failure detection?  I've not seen anything on this so I 
suspect this is the default.



Thanks,
Anton

Re: sstableloader throws storage_port error

[Jonathan Ellis]

"Unable to bind to address nadevsan04/10.168.121.57:7000" means
something else is using that address/port.  netstat can tell you what
process that is, if you're not sure.

On Thu, Aug 11, 2011 at 4:24 PM, Tom Davidson  wrote:
> I am trying to sstableloader and I do not want to access Cassandra on the
> same node. I haved edited my Cassandra.yaml to with appropriate values for
> the listen_address and rpc_address but I keep getting the error below. The
> Cassandra-cli tool, nodetool etc. works find when trying to connect to my
> Cassandra cluster, but sstableloader does not. Any suggestions?
>
>
>
> [tdavidson@nadevsan06 ~]$ sstableloader --debug -v Demo
>
> Starting client (and waiting 30 seconds for gossip) ...
>
> org.apache.cassandra.config.ConfigurationException: Unable to bind to
> address nadevsan04/10.168.121.57:7000. Set listen_address in cassandra.yaml
> to an interface you can bind to, e.g., your private IP address on EC2
>
> java.lang.RuntimeException:
> org.apache.cassandra.config.ConfigurationException: Unable to bind to
> address nadevsan04/10.168.121.57:7000. Set listen_address in cassandra.yaml
> to an interface you can bind to, e.g., your private IP address on EC2
>
>     at
> org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:225)
>
>     at
> org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:104)
>
>     at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:61)
>
> Caused by: org.apache.cassandra.config.ConfigurationException: Unable to
> bind to address nadevsan04/10.168.121.57:7000. Set listen_address in
> cassandra.yaml to an interface you can bind to, e.g., your private IP
> address on EC2
>
>     at
> org.apache.cassandra.net.MessagingService.getServerSocket(MessagingService.java:220)
>
>     at
> org.apache.cassandra.net.MessagingService.listen(MessagingService.java:191)
>
>     at
> org.apache.cassandra.service.StorageService.initClient(StorageService.java:350)
>
>     at
> org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:182)
>
>     ... 2 more
>
>
>
> From: John Conwell [mailto:j...@iamjohn.me]
> Sent: Tuesday, July 26, 2011 12:11 PM
> To: user@cassandra.apache.org
> Subject: Re: sstableloader throws storage_port error
>
>
>
> After much research and experimentation, I figured out how to get
> sstableloader running on the same machine as a live cassandra node instance.
>
>
>
> The key, as Jonathan stated is to configure sstableloader to use a different
> ipaddress than the running cassandra instance is using.  To do this, I ran
> this command, which created the loopback address for 127.0.0.2
>
>
>
> sudo ifconfig lo0 alias 127.0.0.2
>
>
>
> No you can have cassandra configured to listen to 127.0.0.1, and
> sstableloader configured to listen to 127.0.0.2
>
>
>
> By the way, to remove this ipaddress, run
>
> sudo ifconfig lo0 -alias 127.0.0.2
>
>
>
> But thats not really all.  Because sstableloader reads the cassandra.yaml
> file to get the gossip ipaddress, you need to make a copy of the cassandra
> install directory (or at least the bin and conf folders).  Basically one
> folder with yaml configured for Cassandra, the other folder with yaml
> configured for sstableloader.
>
>
>
> Hope this helps people. I've written an in depth description of how to do
> all this, and can post it if people want, but I'm not sure the etiquette of
> posting blog links in the email list.
>
>
>
> Thanks,
> John
>
>
>
> On Tue, Jul 26, 2011 at 7:40 AM, John Conwell  wrote:
>
> If I have Cassandra already running on my machine, how do I configure
> sstableloader to run on a different IP (127.0.0.2).  Also, does that mean in
> order to use sstableloader on the same machine as an running Cassandra node,
> I have to have two NIC cards?
>
>
>
> I looked around for any info about how to configure and run sstableloader,
> but other than what the cmdline spits out I cant find anything.  Are there
> any examples or best practices?  Is it designed to be run on a machine that
> isn't running a cassandra node?
>
>
>
> On Mon, Jul 25, 2011 at 8:24 PM, Jonathan Ellis  wrote:
>
> sstableloader uses gossip to discover the Cassandra ring, so you'll
> need to run it on a different IP (127.0.0.2 is fine).
>
> On Mon, Jul 25, 2011 at 2:41 PM, John Conwell  wrote:
>> I'm trying to figure out how to use the sstableloader tool.  For my test I
>> have a single node cassandra instance running on my local machine.  I have
>> cassandra running, and validate this by connecting to it with
>> cassandra-cli.
>> I run sstableloader using the following command:
>> bin/sstableloader /Users/someuser/cassandra/mykeyspace
>> and I get the following error:
>> org.apache.cassandra.config.ConfigurationException:
>> localhost/127.0.0.1:7000
>> is in use by another process.  Change listen_address:storage_port in
>> cassandra.yaml to values that do not conflict with other services
>>
>> I've played around with different ports, but not

Re: column metadata and sstable

[Yi Yang]

Thanks Aaron,

This is the same as I've thought. But I'm wondering if there's come triggers 
that can hash the column name in order to save disk space. Or do you think it's 
better to have this feature?

Best,
Steve



On Aug 6, 2011, at 7:06 PM, aaron morton wrote:

> AFAIK it just makes it easier for client API's to understand what data type 
> to use. e.g. it can give your code a long rather than a str / byte array . 
> 
> Personally I'm on the fence about using it. It has some advantages to the 
> client, but given the server does not really need the information it feels a 
> little like additional coupling that's not needed . 
> 
> Cheers
> 
> -
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 6 Aug 2011, at 11:58, Yi Yang wrote:
> 
>> Dear all,
>> 
>> I'm wondering what's the advantage of assigning column metadata when NOT 
>> using secondary indices.
>> 
>> I've gone through the SSTable internal and found out it won't do such 
>> conversion.Thus I think the only advantage we got via column metadata is 
>> a data validation type, am I correct?
>> 
>> Thanks.
>> Steve
> 

RE: sstableloader throws storage_port error

[Tom Davidson]

I figured it out... I didn't realize the nuances of the sstableloader. Correct 
me if I am wrong, but the sstableloader instance needs to communicate with a 
Cassandra cluster via Gossip. In order to do that it sets up a mini gossip 
server bound to a local port. I was thinking the Cassandra.yaml listen and rpc 
ports could be the ports of other machines.

-Original Message-
From: Jonathan Ellis [mailto:jbel...@gmail.com] 
Sent: Thursday, August 11, 2011 6:04 PM
To: user@cassandra.apache.org
Subject: Re: sstableloader throws storage_port error

"Unable to bind to address nadevsan04/10.168.121.57:7000" means
something else is using that address/port.  netstat can tell you what
process that is, if you're not sure.

On Thu, Aug 11, 2011 at 4:24 PM, Tom Davidson  wrote:
> I am trying to sstableloader and I do not want to access Cassandra on the
> same node. I haved edited my Cassandra.yaml to with appropriate values for
> the listen_address and rpc_address but I keep getting the error below. The
> Cassandra-cli tool, nodetool etc. works find when trying to connect to my
> Cassandra cluster, but sstableloader does not. Any suggestions?
>
>
>
> [tdavidson@nadevsan06 ~]$ sstableloader --debug -v Demo
>
> Starting client (and waiting 30 seconds for gossip) ...
>
> org.apache.cassandra.config.ConfigurationException: Unable to bind to
> address nadevsan04/10.168.121.57:7000. Set listen_address in cassandra.yaml
> to an interface you can bind to, e.g., your private IP address on EC2
>
> java.lang.RuntimeException:
> org.apache.cassandra.config.ConfigurationException: Unable to bind to
> address nadevsan04/10.168.121.57:7000. Set listen_address in cassandra.yaml
> to an interface you can bind to, e.g., your private IP address on EC2
>
>     at
> org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:225)
>
>     at
> org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:104)
>
>     at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:61)
>
> Caused by: org.apache.cassandra.config.ConfigurationException: Unable to
> bind to address nadevsan04/10.168.121.57:7000. Set listen_address in
> cassandra.yaml to an interface you can bind to, e.g., your private IP
> address on EC2
>
>     at
> org.apache.cassandra.net.MessagingService.getServerSocket(MessagingService.java:220)
>
>     at
> org.apache.cassandra.net.MessagingService.listen(MessagingService.java:191)
>
>     at
> org.apache.cassandra.service.StorageService.initClient(StorageService.java:350)
>
>     at
> org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:182)
>
>     ... 2 more
>
>
>
> From: John Conwell [mailto:j...@iamjohn.me]
> Sent: Tuesday, July 26, 2011 12:11 PM
> To: user@cassandra.apache.org
> Subject: Re: sstableloader throws storage_port error
>
>
>
> After much research and experimentation, I figured out how to get
> sstableloader running on the same machine as a live cassandra node instance.
>
>
>
> The key, as Jonathan stated is to configure sstableloader to use a different
> ipaddress than the running cassandra instance is using.  To do this, I ran
> this command, which created the loopback address for 127.0.0.2
>
>
>
> sudo ifconfig lo0 alias 127.0.0.2
>
>
>
> No you can have cassandra configured to listen to 127.0.0.1, and
> sstableloader configured to listen to 127.0.0.2
>
>
>
> By the way, to remove this ipaddress, run
>
> sudo ifconfig lo0 -alias 127.0.0.2
>
>
>
> But thats not really all.  Because sstableloader reads the cassandra.yaml
> file to get the gossip ipaddress, you need to make a copy of the cassandra
> install directory (or at least the bin and conf folders).  Basically one
> folder with yaml configured for Cassandra, the other folder with yaml
> configured for sstableloader.
>
>
>
> Hope this helps people. I've written an in depth description of how to do
> all this, and can post it if people want, but I'm not sure the etiquette of
> posting blog links in the email list.
>
>
>
> Thanks,
> John
>
>
>
> On Tue, Jul 26, 2011 at 7:40 AM, John Conwell  wrote:
>
> If I have Cassandra already running on my machine, how do I configure
> sstableloader to run on a different IP (127.0.0.2).  Also, does that mean in
> order to use sstableloader on the same machine as an running Cassandra node,
> I have to have two NIC cards?
>
>
>
> I looked around for any info about how to configure and run sstableloader,
> but other than what the cmdline spits out I cant find anything.  Are there
> any examples or best practices?  Is it designed to be run on a machine that
> isn't running a cassandra node?
>
>
>
> On Mon, Jul 25, 2011 at 8:24 PM, Jonathan Ellis  wrote:
>
> sstableloader uses gossip to discover the Cassandra ring, so you'll
> need to run it on a different IP (127.0.0.2 is fine).
>
> On Mon, Jul 25, 2011 at 2:41 PM, John Conwell  wrote:
>> I'm trying to figure out how to use the sstableloader tool.  For my test I
>> have a single node ca

Re: tpstats confusion

[aaron morton]

I've not checked the code but from memory when the nodes are ordered in 
proximity to the coordinator the local node is always first if it's in the 
replica set. So with RF=3 and N=3 the closest node is always the local one. 

Cheers
 
-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 12 Aug 2011, at 10:10, Ian Danforth wrote:

> I'm writing at QUORUM though and (pardon me for being dense) it would seem 
> that the first replica shouldn't always be on the same server if I'm using 
> RP. 
> 
> I very much appreciate your time, I'm sure there is something basic that just 
> hasn't clicked.
> 
> Ian
> 
> On Thu, Aug 11, 2011 at 2:56 PM, Jonathan Ellis  wrote:
> https://issues.apache.org/jira/browse/CASSANDRA-2889
> 
> On Thu, Aug 11, 2011 at 2:55 PM, Ian Danforth  wrote:
> > I don't think so, perhaps more succinctly, why would ROW actions only be
> > performed on a single node?
> > Ian
> >
> > On Wed, Aug 10, 2011 at 8:12 PM, Jonathan Ellis  wrote:
> >>
> >> are you i/o bound?  I believe counter ROW needs to perform a read of
> >> the old value.
> >>
> >> On Wed, Aug 10, 2011 at 7:17 PM, Ian Danforth 
> >> wrote:
> >> > Hello all!
> >> > Thanks for taking the time to read this, I'm new to Cassandra and really
> >> > want to get it working :)
> >> > Below you'll find the output from tpstats on the three nodes in my .8.1
> >> > cluster. As you'll note from ring (also below), my cluster appears
> >> > balanced.
> >> > However, please note that NODE1 is the only node that has completed any
> >> > ReplicateOnWriteStage actions. I've been having a lot of trouble with
> >> > RoW
> >> > and seen a single node in my various test clusters get hugely backed up
> >> > on
> >> > this stage to the point where it slows all writes to a crawl.
> >> > I'm using counters in a column family defined like this:
> >> >  KEYSPACE/CF 
> >> > create keyspace Keyspace1
> >> > with strategy_options=[{replication_factor:3}]
> >> > and placement_strategy =
> >> > 'org.apache.cassandra.locator.SimpleStrategy';
> >> > use Keyspace1;
> >> > create column family TwitterTest
> >> > with default_validation_class = CounterColumnType
> >> > and replicate_on_write = true;
> >> > 
> >> >
> >> > With any given batch_insert() I'm incrementing ~17,000 counters. I've
> >> > backed
> >> > off a bit and started using ConsistencyLevel.QUORUM, as it was suggested
> >> > elsewhere that ANY or ONE could overwhelm the system.
> >> >
> >> > I hope I'm just doing something stupid and one you experienced folks can
> >> > point out my error.
> >> > Thanks in advance!
> >> > Ian
> >> >  MORE DATA 
> >> > Address DC  RackStatus State   Load
> >> >  Owns
> >> >Token
> >> >
> >> >113427455640312821154458202477256070484
> >> > 10.87.9.57  datacenter1 rack1   Up Normal  193.86 MB
> >> > 33.33%  0
> >> > 10.86.223.32datacenter1 rack1   Up Normal  191.1 MB
> >> >  33.33%  56713727820156410577229101238628035242
> >> > 10.86.229.24datacenter1 rack1   Up Normal  193.63 MB
> >> > 33.33%  113427455640312821154458202477256070484
> >> >
> >> >
> >> >
> >> > SEED
> >> > Pool NameActive   Pending  Completed
> >> > ReadStage 0 0  7
> >> > RequestResponseStage  0 01479839
> >> > MutationStage 0 01479908
> >> > ReadRepairStage   0 0  0
> >> > ReplicateOnWriteStage 0 0  0
> >> > GossipStage   0 0   5176
> >> > AntiEntropyStage  0 0  0
> >> > MigrationStage0 0128
> >> > MemtablePostFlusher   0 0 28
> >> > StreamStage   0 0  0
> >> > FlushWriter   0 0 28
> >> > MiscStage 0 0  2
> >> > FlushSorter   0 0  0
> >> > InternalResponseStage 0 0363
> >> > HintedHandoff 0 0  0
> >> > NODE1
> >> > Pool NameActive   Pending  Completed
> >> > ReadStage 0 0  7
> >> > RequestResponseStage  0 02960208
> >> > MutationStage 0 01480104
> >> > ReadRepairStage   0 0  0
> >> > ReplicateOnWriteStage 0 01480104
> >> > GossipStage   0 0   4151
> >> > AntiEntropyStage  0 0  0
> >> > MigrationStage0 0128
> >> > MemtablePostFlusher   0 0 31
> >> > Stre

Re: performance problems on new cluster

[aaron morton]

> 
> iostat doesn't show a request queue bottleneck.  The timeouts we are seeing 
> is for reads.  The latency on the nodes I have temporarily used for reads is 
> around 2-45ms.  The next token in the ring at an alternate DC is showing ~4ms 
> with everything else around 0.05ms.  tpstats desn't show any active/pending.  
> Reads are at CL.ONE & Writes using CL.ANY

OK, node latency is fine and you are using some pretty low consistency. You 
said NTS with RF 2, is that RF 2 for each DC ? 

The steps below may help get an idea of whats going on…

1) use nodetool getendpoints to determine which replicas a key is.  
2) connect directly to one of the endpoints with the CLI, ensure CL is ONE and 
do your test query. 
3) connect to another node in the same DC that is not a replica and do the 
same. 
4) connect to another node in a different DC and do the same 

Once you can repo it try turning up the logging not the coordinator to DEBUG 
you can do this via JConsole. Look for these lines….

* Command/ConsistencyLevel is….
* reading data locally... or reading data from…
* reading digest locally… or reading digest for from…
* Read timeout:…

You'll also see some lines about receiving messages from other nodes.  

Hopefully you can get an idea of which nodes are involved in a failing query. 
Getting a thrift TimedOutException on a read with CL ONE is pretty odd. 

> What can I do in regards to confirming this issue is still outstanding and/or 
> we are affected by it?
It's in 0.8 and will not be fixed. My unscientific approach was to repair a 
single CF at a time, hoping that the differences would be smaller and less data 
would be streamed. 
Minor compaction should help squish things down. If you want to get more 
aggressive reduce the min compaction threshold and trigger a minor compaction 
with nodetool flush.   

> Version of failure detection?  I've not seen anything on this so I suspect 
> this is the default.
Was asking so I could see if their were any fixed in Gossip or the 
FailureDetect that you were missing. Check the CHANGES.txt file. 

Hope that helps. 

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 12 Aug 2011, at 12:48, Anton Winter wrote:

> 
>> Is there a reason you are using the trunk and not one of the tagged 
>> releases? Official releases are a lot more stable than the trunk.
>> 
> Yes, as we are using a combination of Ec2 and colo servers we are needing to 
> use broadcast_address from CASSANDRA-2491.  The patch that is associated with 
> that JIRA does not apply cleanly against 0.8 so this is why we are using 
> trunk.
> 
>>> 1) thrift timeouts & general degraded response times
>> For read or writes ? What sort of queries are you running ? Check the local 
>> latency on each node using cfstats and cfhistogram, and a bit of iostat 
>> http://spyced.blogspot.com/2010/01/linux-performance-basics.html What does 
>> nodetool tpstats say, is there a stage backing up?
>> 
>> If the local latency is OK look at the cross DC situation. What CL are you 
>> using? Are nodes timing out waiting for nodes in other DC's ?
> 
> iostat doesn't show a request queue bottleneck.  The timeouts we are seeing 
> is for reads.  The latency on the nodes I have temporarily used for reads is 
> around 2-45ms.  The next token in the ring at an alternate DC is showing ~4ms 
> with everything else around 0.05ms.  tpstats desn't show any active/pending.  
> Reads are at CL.ONE & Writes using CL.ANY
> 
>> 
>>> 2) *lots* of exception errors, such as:
>> Repair is trying to run on a response which is a digest response, this 
>> should not be happening. Can you provide some more info on the type of query 
>> you are running ?
>> 
> The query being run is  get cf1['user-id']['seg']
> 
> 
>>> 3) ring imbalances during a repair (refer to the above nodetool ring output)
>> You may be seeing this
>> https://issues.apache.org/jira/browse/CASSANDRA-2280
>> I think it's a mistake that is it marked as resolved.
>> 
> What can I do in regards to confirming this issue is still outstanding and/or 
> we are affected by it?
> 
>>> 4) regular failure detection when any node does something only moderately 
>>> stressful, such as a repair or are under light load etc. but the node 
>>> itself thinks it is fine.
>> What version are you using ?
>> 
> Version of failure detection?  I've not seen anything on this so I suspect 
> this is the default.
> 
> 
> Thanks,
> Anton
> 

Re: Client traffic encryption best practices....

[Vijay]

https://issues.apache.org/jira/browse/THRIFT-106 seems to be the right way
to go but the cassandra server needs to support too which we might want
to add

Regards,




On Thu, Aug 11, 2011 at 2:54 PM, Chris Marino  wrote:

> Hello, is there any consensus on how to secure client/cluster
> communications???
>
> I'm running an 8 node cluster across EC2 regions.  I'm running inter-node
> encryption and I want to encrypt the traffic from the clients as well.
>
> My options seem to be:
>
> Have the client connect to only one node and encrypt that one connection
> with OpenVPN/stunnel (or something similar). Or, set up an encrypted tunnel
> from the client to each node. Is there a client library that could take care
> of this for me??
>
> Setting up tunnels to each node is a major pain, but pointing the client to
> only one node is going to kill my performance.  I'm running 4 nodes in each
> EC2 region with one client in each. Maybe I could connect the client only to
> the local nodes, which should simplify things a bit, but I was wondering if
> anyone had any experience with this or could suggest something that might be
> better.
>
> Please let me know.
> Thanks.
> CM
>