RE: FW: Very slow batch insert using version 0.7.2

2011-03-11 Thread Desimpel, Ignace 
That is the amount of records I need to add for each document. And we would 
like to test it with more than 100K or more documents. That's why we thought 
Cassandra could be a good database system.

At start I did the inserts one by one. Of course by doing it in batch the 
system was a lot faster, and it worked fine in version 0.6.x.
With your question in mind, I did some more tests (only on Windows XP):
1) Changed the code to insert two sets of about 50K. Same behavior in 0.7.x.
2) Then changed it to store 1000 records at a time. Seems a bit better. Now the 
rpc timeout is not throwed. But the number of flushing of Memtables and the 
number of generate commit logs is still large. And the total amount of time to 
write is still more than 10 minutes, although is used to be less than 10 
seconds.


I do not know the code of Cassandra, but I also have the system running in 
Eclipse. Thus if needed I can debug the code but I would need some input from 
your team.

Ignace


-Original Message-
From: Ryan King [mailto:r...@twitter.com] 
Sent: donderdag 10 maart 2011 18:18
To: user@cassandra.apache.org
Cc: Desimpel, Ignace
Subject: Re: FW: Very slow batch insert using version 0.7.2

Why use such a large batch size?

-ryan

On Thu, Mar 10, 2011 at 6:31 AM, Desimpel, Ignace
 wrote:
>
>
> Hello,
>
> I had a demo application with embedded cassandra version 0.6.x, inserting
> about 120 K  row mutations in one call.
>
> In version 0.6.x that usually took about 5 seconds, and I could repeat this
> step adding each time the same amount of data.
>
> Running on a single CPU computer, single hard disk, XP 32 bit OS, 1G memory
>
> I tested this again on CentOS 64 bit OS, 6G memory, different settings of
> memtable_throughput_in_mb and memtable_operations_in_millions.
>
> Also tried version 0.7.3. Also the same behavior.
>
>
>
> Now with version 0.7.2 the call returns with a timeout exception even using
> a timeout of 12 (2 minutes). I see the CPU time going to 100%, a lot of
> disk writing ( giga bytes), a lot of log messages  about compacting,
> flushing, commitlog, ...
>
>
>
> Below you can find some information using the nodetool at start of the batch
> mutation and also after 14 minutes. The MutationStage is clearly showing how
> slow the system handles the row mutations.
>
>
>
> Attached : Cassandra.yaml with at end the description of my database
> structure using yaml
>
> Attached : log file with cassandra output.
>
>
>
> Any idea what I could be doing wrong?
>
>
>
> Regards,
>
>
>
> Ignace Desimpel
>
>
>
> ignace.desim...@nuance.com
>
>
>
> At start of the insert (after inserting 124360 row mutations) I get the
> following info from the nodetool :
>
>
>
> C:\apache-cassandra-07.2\bin>nodetool --host ads.nuance.com info
>
> Starting NodeTool
>
> 34035877798200531112672274220979640561
>
> Gossip active    : true
>
> Load : 5.49 MB
>
> Generation No    : 1299502115
>
> Uptime (seconds) : 1152
>
> Heap Memory (MB) : 179,84 / 1196,81
>
>
>
> C:\apache-cassandra-07.2\bin>nodetool --host ads.nuance.com tpstats
>
> Starting NodeTool
>
> Pool Name    Active   Pending  Completed
>
> ReadStage 0 0  40637
>
> RequestResponseStage  0 0 30
>
> MutationStage    32    121679  72149
>
> GossipStage   0 0  0
>
> AntiEntropyStage  0 0  0
>
> MigrationStage    0 0  1
>
> MemtablePostFlusher   0 0  6
>
> StreamStage   0 0  0
>
> FlushWriter   0 0  5
>
> MiscStage 0     0  0
>
> FlushSorter   0 0  0
>
> InternalResponseStage 0 0  0
>
> HintedHandoff 0 0  0
>
>
>
> After 14 minutes (timeout exception after 2 minutes : see log file) I get :
>
>
>
> C:\apache-cassandra-07.2\bin>nodetool --host ads.nuance.com info
>
> Starting NodeTool
>
> 34035877798200531112672274220979640561
>
> Gossip active    : true
>
> Load : 10.31 MB
>
> Generation No    : 1299502115
>
> Uptime (seconds) : 2172
>
> Heap Memory (MB) : 733,82 / 1196,81
>
>
>
> C:\apache-cassandra-07.2\bin>nodetool --host ads.nuance.com tpstats
>
> Starting NodeTool
>
> Pool Name    Active   Pending  Completed
>
> ReadStage 0 0  40646
>
> RequestResponseStage  0 0 30
>
> MutationStage    32    103310  90526
>
> GossipStage   0 0  0
>
> AntiEntropyStage  0 0  0
>
> MigrationStage    0 0  1
>
> MemtablePostFlusher   0 0   

Re: problem with bootstrap

2011-03-11 Thread Patrik Modesto 
Unfortunately I can't provide the info, I deleted it. It was in wery
strange state.

I started with new cluster today, 2 nodes, each with
auto_bootstrap:true. I can create a keyspace with RF=3, but I can't
insert any data in it. It didn't happen with the old cluster which
made me think. How could I insert data in the old cluster in keyspace
with RF=3 but with just 2 nodes? I found out that the cluster had 3
nodes for short time in the past. We had to remove/return one node but
that was enough for the cluster to accept writes to keyspace with RF=3
even with just 2 nodes.

So I tried to recreate the cluster state:

I have 4 clean server, cassndra 0.7.3, auto_bootstrap:true

1) setup & run node1 - success

2) create keyspace Context with rf=3" and create CF Url via
cassandra-cli - success

3) list Url - Internal error processing get_range_slicesl
node1:
ERROR 09:46:28,725 Internal error processing get_range_slices
java.lang.IllegalStateException: replication factor (3) exceeds number
of endpoints (1)

4) setup & run node2 - success

5) list Url on node1 - Internal error processing get_range_slicesl
node1:
ERROR 09:46:28,725 Internal error processing get_range_slices
java.lang.IllegalStateException: replication factor (3) exceeds number
of endpoints (1)

6) list Url on node2 - Internal error processing get_range_slicesl
node2:
ERROR 09:50:54,231 Internal error processing get_range_slices
java.lang.IllegalStateException: replication factor (3) exceeds number
of endpoints (2)

7) insert on node1 - Internal error processing insert
node1:
ERROR 09:53:11,669 Internal error processing insert
java.lang.IllegalStateException: replication factor (3) exceeds number
of endpoints (2)

8) insert on node2 - Internal error processing insert
node2:
ERROR 09:53:54,833 Internal error processing insert
java.lang.IllegalStateException: replication factor (3) exceeds number
of endpoints (2)

9) setup & run node3 - success

10) list Url on node1 - success

11) insert in Url on node1 - success

12) stop cassandra on node3 - success

13) list & insert on node1&2 - success

14) loadbalance on node1 - Exception in thread "main"
java.lang.IllegalStateException: replication factor (3) exceeds number
of endpoints (2)

15) setup & run node4 - success

16) list Url on node4 - success BUT
node4:
ERROR 10:05:38,452 Fatal exception in thread
Thread[RequestResponseStage:1,5,main]
java.lang.AssertionError
at 
org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
at 
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR 10:05:38,462 Fatal exception in thread
Thread[RequestResponseStage:17,5,main]
java.lang.AssertionError
at 
org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
at 
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

17) loadbalance on node1 - success

18) list Url on node4 - success BUT
node4:
ERROR 10:09:58,251 Fatal exception in thread
Thread[RequestResponseStage:18,5,main]
java.lang.AssertionError
at 
org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
at 
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR 10:09:58,257 Fatal exception in thread
Thread[RequestResponseStage:5,5,main]
java.lang.AssertionError
at 
org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
at 
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

19) repair on node4 - after long long wait I killed it, non of the
nodes report any error

20) list Url on node1 - success BUT
node1:
ERROR 10:18:53,715 Fatal

Re: FW: Very slow batch insert using version 0.7.2

2011-03-11 Thread Zhu Han 
On Fri, Mar 11, 2011 at 10:40 AM, Erik Forkalsrud wrote:

>
> I see the same behavior with smaller batch sizes.  It appears to happen
> when starting Cassandra with the defaults on relatively large systems.
>  Attached is a script I created to reproduce the problem. (usage: mutate.sh
>  /path/to/apache-cassandra-0.7.3-bin.tar.gz)   It extracts a stock cassandra
> distribution to a temp dir and starts it up, (single node) then creates a
> keyspace with a column family and does a batch mutate to insert 5000
> columns.
>
> When I run it on my laptop (Fedora 14, 64-bit, 4 cores, 8GB RAM)  it
> flushes one Memtable with 5000 operations
> When I run it on a server  (RHEL5, 64-bit, 16 cores, 96GB RAM) it flushes
> 100 Memtables with anywhere between 1 operation and 359 operations (35 bytes
> and 12499 bytes)
>

What's the settings of commit log flush, periodic or in batch?


>
> I'm guessing I can override the JVM memory parameters to avoid the frequent
> flushing on the server, but I haven't experimented with that yet.  The only
> difference in the effective command line between the laptop and server is
> "-Xms3932M -Xmx3932M -Xmn400M" on the laptop and "-Xms48334M -Xmx48334M
> -Xmn1600M" on the server.
>
>
>
> --
> Erik Forkalsrud
> Commission Junction
>
>
>
> On 03/10/2011 09:18 AM, Ryan King wrote:
>
>> Why use such a large batch size?
>>
>> -ryan
>>
>> On Thu, Mar 10, 2011 at 6:31 AM, Desimpel, Ignace
>>   wrote:
>>
>>>
>>> Hello,
>>>
>>> I had a demo application with embedded cassandra version 0.6.x, inserting
>>> about 120 K  row mutations in one call.
>>>
>>> In version 0.6.x that usually took about 5 seconds, and I could repeat
>>> this
>>> step adding each time the same amount of data.
>>>
>>> Running on a single CPU computer, single hard disk, XP 32 bit OS, 1G
>>> memory
>>>
>>> I tested this again on CentOS 64 bit OS, 6G memory, different settings of
>>> memtable_throughput_in_mb and memtable_operations_in_millions.
>>>
>>> Also tried version 0.7.3. Also the same behavior.
>>>
>>>
>>>
>>> Now with version 0.7.2 the call returns with a timeout exception even
>>> using
>>> a timeout of 12 (2 minutes). I see the CPU time going to 100%, a lot
>>> of
>>> disk writing ( giga bytes), a lot of log messages  about compacting,
>>> flushing, commitlog, …
>>>
>>>
>>>
>>> Below you can find some information using the nodetool at start of the
>>> batch
>>> mutation and also after 14 minutes. The MutationStage is clearly showing
>>> how
>>> slow the system handles the row mutations.
>>>
>>>
>>>
>>> Attached : Cassandra.yaml with at end the description of my database
>>> structure using yaml
>>>
>>> Attached : log file with cassandra output.
>>>
>>>
>>>
>>> Any idea what I could be doing wrong?
>>>
>>>
>>>
>>> Regards,
>>>
>>>
>>>
>>> Ignace Desimpel
>>>
>>>
>>>
>>> ignace.desim...@nuance.com
>>>
>>>
>>>
>>> At start of the insert (after inserting 124360 row mutations) I get the
>>> following info from the nodetool :
>>>
>>>
>>>
>>> C:\apache-cassandra-07.2\bin>nodetool --host ads.nuance.com info
>>>
>>> Starting NodeTool
>>>
>>> 34035877798200531112672274220979640561
>>>
>>> Gossip active: true
>>>
>>> Load : 5.49 MB
>>>
>>> Generation No: 1299502115
>>>
>>> Uptime (seconds) : 1152
>>>
>>> Heap Memory (MB) : 179,84 / 1196,81
>>>
>>>
>>>
>>> C:\apache-cassandra-07.2\bin>nodetool --host ads.nuance.com tpstats
>>>
>>> Starting NodeTool
>>>
>>> Pool NameActive   Pending  Completed
>>>
>>> ReadStage 0 0  40637
>>>
>>> RequestResponseStage  0 0 30
>>>
>>> MutationStage32121679  72149
>>>
>>> GossipStage   0 0  0
>>>
>>> AntiEntropyStage  0 0  0
>>>
>>> MigrationStage0 0  1
>>>
>>> MemtablePostFlusher   0 0  6
>>>
>>> StreamStage   0 0  0
>>>
>>> FlushWriter   0 0  5
>>>
>>> MiscStage 0 0  0
>>>
>>> FlushSorter   0 0  0
>>>
>>> InternalResponseStage 0 0  0
>>>
>>> HintedHandoff 0 0  0
>>>
>>>
>>>
>>> After 14 minutes (timeout exception after 2 minutes : see log file) I get
>>> :
>>>
>>>
>>>
>>> C:\apache-cassandra-07.2\bin>nodetool --host ads.nuance.com info
>>>
>>> Starting NodeTool
>>>
>>> 34035877798200531112672274220979640561
>>>
>>> Gossip active: true
>>>
>>> Load : 10.31 MB
>>>
>>> Generation No: 1299502115
>>>
>>> Uptime (seconds) : 2172
>>>
>>> Heap Memory (MB) : 733,82 / 1196,81
>>>
>>>
>>>
>>> C:\apache-cassandra-07.2\bin>nodetool --host ads.nuance.com tpstats
>>>
>>> Starting NodeTool
>>>
>>> Pool NameActive   Pending  Completed
>>>
>>> ReadS

Re: memory utilization

2011-03-11 Thread Chris Burroughs 
On 03/10/2011 09:26 PM, Bill Hastings wrote:
> Hi All
> 
> Memory utilization reported by JCOnsole for Cassandra seems to be much
> lesser than that reported by top ("RES" memory). Can someone explain this?
> Maybe off topic but would appreciate a response.
> 

Is there an more or less constant amount of resident memory, or is it
growing over a period of days?

Poor performance on small data set

2011-03-11 Thread Vodnok 
Hi,

I'm facing poor performance issue getting a simple row

Here is my dev env :

- Windows 7
- PHPCassa [PHP 5.3.5]
- Cassandra 0.7.3

CF :
create column family docs with comparator = 'UTF8Type' and column_type =
'Standard' and rows_cached=10 and keys_cached=10;

There is less than 1000 rows and i've got a 75-100ms to get one row by id
With memcached it's 2ms

I don't know where is the problem. jvm ? cassandra ? phpcassa ?

What can i do to detect where is the problem ?

Thank you,

Vodnok,

ps: phpcassa say to use C Extension (but cannot use this C Extension made
for linux on Windows) but i'm not sure the issue is coming from phpcassa

Re: Poor performance on small data set

2011-03-11 Thread Peter Schuller 
> There is less than 1000 rows and i've got a 75-100ms to get one row by id
> With memcached it's 2ms
>
> I don't know where is the problem. jvm ? cassandra ? phpcassa ?
>
> What can i do to detect where is the problem ?

I'm not familiar with the PHP client, but this sounds suspiciously
like a nagle + delayed ACK problem. The PHP client probably isn't
setting the TCP_NODELAY flag (or the equivalent in Windows).

Google for "nagle delayed ack" for details.

-- 
/ Peter Schuller

Re: Poor performance on small data set

2011-03-11 Thread Edward Capriolo 
On Fri, Mar 11, 2011 at 11:44 AM, Peter Schuller
 wrote:
>> There is less than 1000 rows and i've got a 75-100ms to get one row by id
>> With memcached it's 2ms
>>
>> I don't know where is the problem. jvm ? cassandra ? phpcassa ?
>>
>> What can i do to detect where is the problem ?
>
> I'm not familiar with the PHP client, but this sounds suspiciously
> like a nagle + delayed ACK problem. The PHP client probably isn't
> setting the TCP_NODELAY flag (or the equivalent in Windows).
>
> Google for "nagle delayed ack" for details.
>
> --
> / Peter Schuller
>
Also you will find that setting rowsCached and keysCached not
effective. Chose one or the other. (that is not your problem but an
FYI)

Re: Pig output to Cassandra

2011-03-11 Thread Jeremy Hanna 
Yep - it's usable and separate so you should be able to download 0.7-branch and 
build the jar and use it against a 0.7.3 cluster.  I've been using it against a 
0.7.2 cluster actually.

http://svn.apache.org/repos/asf/cassandra/branches/cassandra-0.7/

To use it, check out the readme in the contrib/pig directory in the 0.7 branch. 
 One trivial example of copying one column family to another is:

rows = LOAD 'cassandra://Keyspace1/Standard1' USING CassandraStorage();
STORE rows INTO 'cassandra://Keyspace1/Standard2' USING CassandraStorage();

using the contrib/pig/pig_cassandra script (tx to brandon for that example).  
Also make sure that Standard2 exists too.

If you have any other questions talk to jeromatron (me), driftx (brandon), or 
mvdir (eldon) in #cassandra or #hadoop-pig on IRC on freenode.

On Mar 10, 2011, at 11:43 PM, Mark wrote:

> Sweet! This is exactly what I was looking for and it looks like it was just 
> resolved.
> 
> Are there any working examples or documentation on this feature?
> 
> Thanks
> 
> On 3/10/11 8:57 PM, Matt Kennedy wrote:
>> On its way... https://issues.apache.org/jira/browse/CASSANDRA-1828
>> 
>> On Mar 10, 2011, at 11:17 PM, Mark wrote:
>> 
>>> I thought I read somewhere that Pig has an output format that can write to 
>>> Cassandra but I am unable to find any documentation on this. Is this 
>>> possible and if so can someone please point me in the right direction. 
>>> Thanks

Re: Poor performance on small data set

2011-03-11 Thread Jonathan Ellis 
Also: https://issues.apache.org/jira/browse/THRIFT-638

On Fri, Mar 11, 2011 at 10:44 AM, Peter Schuller
 wrote:
>> There is less than 1000 rows and i've got a 75-100ms to get one row by id
>> With memcached it's 2ms
>>
>> I don't know where is the problem. jvm ? cassandra ? phpcassa ?
>>
>> What can i do to detect where is the problem ?
>
> I'm not familiar with the PHP client, but this sounds suspiciously
> like a nagle + delayed ACK problem. The PHP client probably isn't
> setting the TCP_NODELAY flag (or the equivalent in Windows).
>
> Google for "nagle delayed ack" for details.
>
> --
> / Peter Schuller
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Pig output to Cassandra

2011-03-11 Thread Mark 

I'll give it a try. Thanks alot!

On 3/11/11 9:30 AM, Jeremy Hanna wrote:

Yep - it's usable and separate so you should be able to download 0.7-branch and 
build the jar and use it against a 0.7.3 cluster.  I've been using it against a 
0.7.2 cluster actually.

http://svn.apache.org/repos/asf/cassandra/branches/cassandra-0.7/

To use it, check out the readme in the contrib/pig directory in the 0.7 branch. 
 One trivial example of copying one column family to another is:

rows = LOAD 'cassandra://Keyspace1/Standard1' USING CassandraStorage();
STORE rows INTO 'cassandra://Keyspace1/Standard2' USING CassandraStorage();

using the contrib/pig/pig_cassandra script (tx to brandon for that example).  
Also make sure that Standard2 exists too.

If you have any other questions talk to jeromatron (me), driftx (brandon), or 
mvdir (eldon) in #cassandra or #hadoop-pig on IRC on freenode.

On Mar 10, 2011, at 11:43 PM, Mark wrote:


Sweet! This is exactly what I was looking for and it looks like it was just 
resolved.

Are there any working examples or documentation on this feature?

Thanks

On 3/10/11 8:57 PM, Matt Kennedy wrote:

On its way... https://issues.apache.org/jira/browse/CASSANDRA-1828

On Mar 10, 2011, at 11:17 PM, Mark wrote:


I thought I read somewhere that Pig has an output format that can write to 
Cassandra but I am unable to find any documentation on this. Is this possible 
and if so can someone please point me in the right direction. Thanks

Re: FW: Very slow batch insert using version 0.7.2

2011-03-11 Thread Erik Forkalsrud 

On 03/11/2011 04:56 AM, Zhu Han wrote:


When I run it on my laptop (Fedora 14, 64-bit, 4 cores, 8GB RAM)
 it flushes one Memtable with 5000 operations
When I run it on a server  (RHEL5, 64-bit, 16 cores, 96GB RAM) it
flushes 100 Memtables with anywhere between 1 operation and 359
operations (35 bytes and 12499 bytes)


What's the settings of commit log flush, periodic or in batch?



It's whatever the default setting is, (in the cassandra.yaml that is 
packaged in the apache-cassandra-0.7.3-bin.tar.gz download) specifically:


   commitlog_rotation_threshold_in_mb: 128
   commitlog_sync: periodic
   commitlog_sync_period_in_ms: 1
   flush_largest_memtables_at: 0.75

If I describe keyspace I get:

   [default@unknown] describe keyspace Events;
   Keyspace: Events:
 Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
   Replication Factor: 1
 Column Families:
   ColumnFamily: Event
 Columns sorted by: org.apache.cassandra.db.marshal.TimeUUIDType
 Row cache size / save period: 0.0/0
 Key cache size / save period: 20.0/14400
 Memtable thresholds: 14.109375/3010/1440
 GC grace seconds: 864000
 Compaction min/max thresholds: 4/32
 Read repair chance: 1.0
 Built indexes: []


It turns out my suspicion was right. When I tried overriding the jvm 
memory parameters calculated in conf/cassandra-env.sh to use the values 
calculated on my 8GB laptop like this:


   MAX_HEAP_SIZE=3932m HEAP_NEWSIZE=400m ./mutate.sh

That made the server behave much nicer.  This time it kept all 5000 
operations in a single Memtable.  Also, when running with these memory 
settings the Memtable thresholds changed to "1.1390625/243/1440"   (from 
"14.109375/3010/1440")  (all the other output from "describe 
keyspace" remains the same)


So it looks like something goes wrong when cassandra gets too much memory.


--
Erik Forkalsrud
Commission Junstion

Re: FW: Very slow batch insert using version 0.7.2

2011-03-11 Thread Jonathan Ellis 
https://issues.apache.org/jira/browse/CASSANDRA-2158, fixed in 0.7.3

you could have saved a lot of time just by upgrading first. :)

On Fri, Mar 11, 2011 at 2:02 PM, Erik Forkalsrud  wrote:
> On 03/11/2011 04:56 AM, Zhu Han wrote:
>>
>> When I run it on my laptop (Fedora 14, 64-bit, 4 cores, 8GB RAM)  it
>> flushes one Memtable with 5000 operations
>> When I run it on a server  (RHEL5, 64-bit, 16 cores, 96GB RAM) it flushes
>> 100 Memtables with anywhere between 1 operation and 359 operations (35 bytes
>> and 12499 bytes)
>
> What's the settings of commit log flush, periodic or in batch?
>
>
> It's whatever the default setting is, (in the cassandra.yaml that is
> packaged in the apache-cassandra-0.7.3-bin.tar.gz download) specifically:
>
>    commitlog_rotation_threshold_in_mb: 128
>    commitlog_sync: periodic
>    commitlog_sync_period_in_ms: 1
>    flush_largest_memtables_at: 0.75
>
> If I describe keyspace I get:
>
>    [default@unknown] describe keyspace Events;
>    Keyspace: Events:
>      Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
>        Replication Factor: 1
>      Column Families:
>        ColumnFamily: Event
>      Columns sorted by: org.apache.cassandra.db.marshal.TimeUUIDType
>      Row cache size / save period: 0.0/0
>      Key cache size / save period: 20.0/14400
>      Memtable thresholds: 14.109375/3010/1440
>      GC grace seconds: 864000
>      Compaction min/max thresholds: 4/32
>      Read repair chance: 1.0
>      Built indexes: []
>
>
> It turns out my suspicion was right. When I tried overriding the jvm memory
> parameters calculated in conf/cassandra-env.sh to use the values calculated
> on my 8GB laptop like this:
>
>    MAX_HEAP_SIZE=3932m HEAP_NEWSIZE=400m ./mutate.sh
>
> That made the server behave much nicer.  This time it kept all 5000
> operations in a single Memtable.  Also, when running with these memory
> settings the Memtable thresholds changed to "1.1390625/243/1440"   (from
> "14.109375/3010/1440")  (all the other output from "describe keyspace"
> remains the same)
>
> So it looks like something goes wrong when cassandra gets too much memory.
>
>
> --
> Erik Forkalsrud
> Commission Junstion
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: FW: Very slow batch insert using version 0.7.2

2011-03-11 Thread Erik Forkalsrud 

On 03/11/2011 12:13 PM, Jonathan Ellis wrote:

https://issues.apache.org/jira/browse/CASSANDRA-2158, fixed in 0.7.3

you could have saved a lot of time just by upgrading first. :)



Hmm, I'm testing with 0.7.3 ...
but now I know at least which knob to turn.


- Erik -

Re: How long will all nodes data sync.

2011-03-11 Thread aaron morton 
The answer is eventually. There is no point in time. 

The simple case where your 3 nodes are up, and not under pressure, your write 
will  end up at every replica. 

Hope that helps . 
Aaron
On 11 Mar 2011, at 16:38, Vincent Lu (ECL) wrote:

> Hi all,
>  
> I have a question about eventually consistency.
> If there are 3 nodes and RF=3, Write-C=Quorum.
> How long will all 3 nodes data sync?
> Does any configuration can change that?
> Thanks in advance.
>  
> Vincent
> This correspondence is from Cyberlink Corp. and is intended only for use by 
> the recipient named herein, and may contain privileged, proprietary and/or 
> confidential information, and is intended only to be seen and used by named 
> addressee(s). You are notified that any discussion, dissemination, 
> distribution or copying of this correspondence and any attachments, is 
> strictly prohibited, unless otherwise authorized or consented to in writing 
> by the sender. If you have received this correspondence in error, please 
> notify the sender immediately, and please permanently delete the original and 
> any copies of it and any attachment and destroy any related printouts without 
> reading or copying them.

Re: How long will all nodes data sync.

2011-03-11 Thread mcasandra 
Is there a way to monitor how far behind the sync is? In case of hinted hand
off or when node is down for extended period of time it will probably be
helpful to know.

--
View this message in context: 
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/How-long-will-all-nodes-data-sync-tp6160221p6162784.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
Nabble.com.

Re: How long will all nodes data sync.

2011-03-11 Thread Jonathan Ellis 
If you're using ConsistencyLevel appropriately then it doesn't matter,
because you're guaranteed to see data as current as you need.

On Thu, Mar 10, 2011 at 9:38 PM, Vincent Lu (ECL)
 wrote:
> Hi all,
>
>
>
> I have a question about eventually consistency.
>
> If there are 3 nodes and RF=3, Write-C=Quorum.
>
> How long will all 3 nodes data sync?
>
> Does any configuration can change that?
>
> Thanks in advance.
>
>
>
> Vincent
>
> This correspondence is from Cyberlink Corp. and is intended only for use by
> the recipient named herein, and may contain privileged, proprietary and/or
> confidential information, and is intended only to be seen and used by named
> addressee(s). You are notified that any discussion, dissemination,
> distribution or copying of this correspondence and any attachments, is
> strictly prohibited, unless otherwise authorized or consented to in writing
> by the sender. If you have received this correspondence in error, please
> notify the sender immediately, and please permanently delete the original
> and any copies of it and any attachment and destroy any related printouts
> without reading or copying them.



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: On 0.6.6 to 0.7.3 migration, DC-aware traffic and minimising data transfer

2011-03-11 Thread Jonathan Ellis 
On Thu, Mar 10, 2011 at 6:06 AM, Jedd Rashbrooke  wrote:
>  My question is whether it's considered safer to upgrade via 0.6.12
>  to 0.7, or if a direct 0.6.6 -> 0.7 upgrade is safe enough?

You don't need latest 0.6 before upgrading.

>  Copying a cluster between AWS DC's:
>  We have ~ 150-250GB per node, with a Replication Factor of 4.
>  I ack that 0.6 -> 0.7 is necessarily STW, so in an attempt to
>  minimise that outage period I was wondering if it's possible to
>  drain & stop the cluster, then copy over only the 1st, 5th, 9th,
>  and 13th nodes' worth of data (which should be a full copy of
>  all our actual data - we are nicely partitioned, despite the
>  disparity in GB per node) and have Cassandra re-populate the
>  new destination 16 nodes from those four data sets.  If this is
>  feasible, is it likely to be more expensive (in terms of time the
>  new cluster is unresponsive as it rebuilds) than just copying
>  across all 16 sets of data - about 2.7TB.

I'm confused.  You're trying to upgrade and add a DC at the same time?

>  Chattiness / gossip traffic requirements on DC-aware:
>  I haven't pondered deeply on a 7 design yet, so this question is
>  even more nebulous.  We're seeing growth (raw) of about 100GB
>  per month on our 16 node RF4 cluster - say about 25GB of 'actual'
>  data growth.  We don't delete (much) data.  Amazon's calculator
>  suggests even 100GB in/out of a data center is modestly priced,
>  but I'm cautious in case the replication traffic is particularly chatty
>  or excessive.  And how expensive (in terms of traffic) a compaction
>  or repair would be across data centers.

Compactions are node-local.

Normal writes are optimized for the WAN (only one copy will be sent
between DCs; the recipient in the other DC will then forward it to
other replicas there).

Repairs is not yet WAN-optimized but is still cheap if your replicas
are close to consistent since only merkle trees + inconsistent ranges
are sent over the network.

-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: How long will all nodes data sync.

2011-03-11 Thread mcasandra 
Yes I understand that piece but my thought is that if node is down and came
up but at that point we want to know how long the sync will take in case
there were another node to fail in the replica set. It also is good data
point to see how long it takes to sync.

It's always good to have this data handy and is always useful when in
production, in my opinion.

--
View this message in context: 
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/How-long-will-all-nodes-data-sync-tp6160221p6162816.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
Nabble.com.

Seed

2011-03-11 Thread mcasandra 
I've read in some posts before and on the wiki that all the nodes in the
cluster should have same seed list and 2 is the no. of seeds recommended. My
question is it advisable to have node seed itself. Say for eg Node A, Node B
and Node C in a cluster have a seed list of A and B. Now according to the
recommendation even Node A will have seed of A and B, but it somehow looks
wrong. Am I reading it incorrectly?

--
View this message in context: 
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Seed-tp6162837p6162837.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
Nabble.com.

Re: problem with bootstrap

2011-03-11 Thread Aaron Morton 
IMHO creating a keyspace with RF higher than the number of nodes sounds like a 
bug. It puts the cluster into a bad place. It may even be a regression, will 
take a look at the code.

The assertion is interesting. Can you reproduce it with logging at debug and 
post the results? Could you try to reproduce it with a clean cluster?

Thanks
Aaron


On 11/03/2011, at 10:24 PM, Patrik Modesto  wrote:

> Unfortunately I can't provide the info, I deleted it. It was in wery
> strange state.
> 
> I started with new cluster today, 2 nodes, each with
> auto_bootstrap:true. I can create a keyspace with RF=3, but I can't
> insert any data in it. It didn't happen with the old cluster which
> made me think. How could I insert data in the old cluster in keyspace
> with RF=3 but with just 2 nodes? I found out that the cluster had 3
> nodes for short time in the past. We had to remove/return one node but
> that was enough for the cluster to accept writes to keyspace with RF=3
> even with just 2 nodes.
> 
> So I tried to recreate the cluster state:
> 
> I have 4 clean server, cassndra 0.7.3, auto_bootstrap:true
> 
> 1) setup & run node1 - success
> 
> 2) create keyspace Context with rf=3" and create CF Url via
> cassandra-cli - success
> 
> 3) list Url - Internal error processing get_range_slicesl
> node1:
> ERROR 09:46:28,725 Internal error processing get_range_slices
> java.lang.IllegalStateException: replication factor (3) exceeds number
> of endpoints (1)
> 
> 4) setup & run node2 - success
> 
> 5) list Url on node1 - Internal error processing get_range_slicesl
> node1:
> ERROR 09:46:28,725 Internal error processing get_range_slices
> java.lang.IllegalStateException: replication factor (3) exceeds number
> of endpoints (1)
> 
> 6) list Url on node2 - Internal error processing get_range_slicesl
> node2:
> ERROR 09:50:54,231 Internal error processing get_range_slices
> java.lang.IllegalStateException: replication factor (3) exceeds number
> of endpoints (2)
> 
> 7) insert on node1 - Internal error processing insert
> node1:
> ERROR 09:53:11,669 Internal error processing insert
> java.lang.IllegalStateException: replication factor (3) exceeds number
> of endpoints (2)
> 
> 8) insert on node2 - Internal error processing insert
> node2:
> ERROR 09:53:54,833 Internal error processing insert
> java.lang.IllegalStateException: replication factor (3) exceeds number
> of endpoints (2)
> 
> 9) setup & run node3 - success
> 
> 10) list Url on node1 - success
> 
> 11) insert in Url on node1 - success
> 
> 12) stop cassandra on node3 - success
> 
> 13) list & insert on node1&2 - success
> 
> 14) loadbalance on node1 - Exception in thread "main"
> java.lang.IllegalStateException: replication factor (3) exceeds number
> of endpoints (2)
> 
> 15) setup & run node4 - success
> 
> 16) list Url on node4 - success BUT
> node4:
> ERROR 10:05:38,452 Fatal exception in thread
> Thread[RequestResponseStage:1,5,main]
> java.lang.AssertionError
>at 
> org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
>at 
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
>at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>at java.lang.Thread.run(Thread.java:662)
> ERROR 10:05:38,462 Fatal exception in thread
> Thread[RequestResponseStage:17,5,main]
> java.lang.AssertionError
>at 
> org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
>at 
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
>at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>at java.lang.Thread.run(Thread.java:662)
> 
> 17) loadbalance on node1 - success
> 
> 18) list Url on node4 - success BUT
> node4:
> ERROR 10:09:58,251 Fatal exception in thread
> Thread[RequestResponseStage:18,5,main]
> java.lang.AssertionError
>at 
> org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
>at 
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
>at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>at java.lang.Thread.run(Thread.java:662)
> ERROR 10:09:58,257 Fatal exception in thread
> Thread[RequestResponseStage:5,5,main]
> java.lang.AssertionError
>at 
> org.apache.cassandra.servic

Re: FW: Very slow batch insert using version 0.7.2

2011-03-11 Thread Erik Forkalsrud 

On 03/11/2011 12:13 PM, Jonathan Ellis wrote:

https://issues.apache.org/jira/browse/CASSANDRA-2158, fixed in 0.7.3

you could have saved a lot of time just by upgrading first. :)



It looks like the fix isn't entirely correct.  The bug is still in 
0.7.3.   In Memtable.java, the line:


   THRESHOLD = cfs.getMemtableThroughputInMB() * 1024 * 1024;

should be changed to:

   THRESHOLD = cfs.getMemtableThroughputInMB() * 1024L * 1024L;


Here's some code that illustrates the difference:

public void testMultiplication() {
int memtableThroughputInMB = 2300;
long thresholdA = memtableThroughputInMB * 1024 * 1024;
long thresholdB = memtableThroughputInMB * 1024L * 1024L;
System.out.println("a=" + thresholdA + " b=" + thresholdB);
}



- Erik -



On Fri, Mar 11, 2011 at 2:02 PM, Erik Forkalsrud  wrote:

On 03/11/2011 04:56 AM, Zhu Han wrote:

When I run it on my laptop (Fedora 14, 64-bit, 4 cores, 8GB RAM)  it
flushes one Memtable with 5000 operations
When I run it on a server  (RHEL5, 64-bit, 16 cores, 96GB RAM) it flushes
100 Memtables with anywhere between 1 operation and 359 operations (35 bytes
and 12499 bytes)

What's the settings of commit log flush, periodic or in batch?


It's whatever the default setting is, (in the cassandra.yaml that is
packaged in the apache-cassandra-0.7.3-bin.tar.gz download) specifically:

commitlog_rotation_threshold_in_mb: 128
commitlog_sync: periodic
commitlog_sync_period_in_ms: 1
flush_largest_memtables_at: 0.75

If I describe keyspace I get:

[default@unknown] describe keyspace Events;
Keyspace: Events:
  Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Replication Factor: 1
  Column Families:
ColumnFamily: Event
  Columns sorted by: org.apache.cassandra.db.marshal.TimeUUIDType
  Row cache size / save period: 0.0/0
  Key cache size / save period: 20.0/14400
  Memtable thresholds: 14.109375/3010/1440
  GC grace seconds: 864000
  Compaction min/max thresholds: 4/32
  Read repair chance: 1.0
  Built indexes: []


It turns out my suspicion was right. When I tried overriding the jvm memory
parameters calculated in conf/cassandra-env.sh to use the values calculated
on my 8GB laptop like this:

MAX_HEAP_SIZE=3932m HEAP_NEWSIZE=400m ./mutate.sh

That made the server behave much nicer.  This time it kept all 5000
operations in a single Memtable.  Also, when running with these memory
settings the Memtable thresholds changed to "1.1390625/243/1440"   (from
"14.109375/3010/1440")  (all the other output from "describe keyspace"
remains the same)

So it looks like something goes wrong when cassandra gets too much memory.


--
Erik Forkalsrud
Commission Junstion

Write speed roughly 1/10 of expected.

2011-03-11 Thread Steven Liu 
We are using the latest phpcassa
(phpcassa-0.7.a.2.tar.gz
) and cassandra 0.7.3,
we have inserted 12+ million documents into one column family with the
following keyspace/columnfamily settings:

Keyspace: dffl:
  Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Replication Factor: 4
  Column Families:
ColumnFamily: product
  Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type
  Row cache size / save period: 0.0/0
  Key cache size / save period: 20.0/14400
  Memtable thresholds: 1.1578125/247/60
  GC grace seconds: 864000
  Compaction min/max thresholds: 4/32
  Read repair chance: 1.0
  Built indexes: []

It took 219 minutes to insert 12+ million docs which translates to about 913
docs/second using batch_insert in batches of 1250 documents per batch.

We have a cluster of 10 nodes all running 2 x Xeon 3.6ghz with 8GB memory
each and RAID 5 SCSI u320 and have expected better performance. We do have
the binary accel installed (actually, we went through the PHP client and
removed the $bin_accel checks so it always use the accel).

Does anybody have any ideas on the common gotcha's which could cause it to
be this slow?

Re: Write speed roughly 1/10 of expected.

2011-03-11 Thread Peter Schuller 
> It took 219 minutes to insert 12+ million docs which translates to about 913
> docs/second using batch_insert in batches of 1250 documents per batch.

How big are the documents and/or how big is the resulting data when loaded?

What is your data model - is each document a single column? Or a row
containing multiple columns? "913 docs/second" can be low or high or
expected, very much depending on what that means in terms of rows,
columns and sizes.

Did you observe what the bottleneck were during insertion? Were you
inserting using a single client or multiple concurrent clients to make
sure you're not bottlenecking there?

(I have no idea how fast phpcassa is.)

-- 
/ Peter Schuller

Re: Write speed roughly 1/10 of expected.

2011-03-11 Thread Tyler Hobbs 
>
> (I have no idea how fast phpcassa is.)
>

The current master branch (which has the benefit of
THRIFT-638,
while 0.7.a.3 does not) can insert about 3k individual rows a second against
a local Cassandra instance.

-- 
Tyler Hobbs
Software Engineer, DataStax 
Maintainer of the pycassa  Cassandra
Python client library

Re: FW: Very slow batch insert using version 0.7.2

2011-03-11 Thread Jonathan Ellis 
Absolutely right!  Thanks, fixed for 0.7.4.

On Fri, Mar 11, 2011 at 4:14 PM, Erik Forkalsrud  wrote:
> On 03/11/2011 12:13 PM, Jonathan Ellis wrote:
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-2158, fixed in 0.7.3
>>
>> you could have saved a lot of time just by upgrading first. :)
>
>
> It looks like the fix isn't entirely correct.  The bug is still in 0.7.3.
> In Memtable.java, the line:
>
>   THRESHOLD = cfs.getMemtableThroughputInMB() * 1024 * 1024;
>
> should be changed to:
>
>   THRESHOLD = cfs.getMemtableThroughputInMB() * 1024L * 1024L;
>
>
> Here's some code that illustrates the difference:
>
>    public void testMultiplication() {
>        int memtableThroughputInMB = 2300;
>        long thresholdA = memtableThroughputInMB * 1024 * 1024;
>        long thresholdB = memtableThroughputInMB * 1024L * 1024L;
>        System.out.println("a=" + thresholdA + " b=" + thresholdB);
>    }
>
>
>
> - Erik -
>
>
>> On Fri, Mar 11, 2011 at 2:02 PM, Erik Forkalsrud
>>  wrote:
>>>
>>> On 03/11/2011 04:56 AM, Zhu Han wrote:

 When I run it on my laptop (Fedora 14, 64-bit, 4 cores, 8GB RAM)  it
 flushes one Memtable with 5000 operations
 When I run it on a server  (RHEL5, 64-bit, 16 cores, 96GB RAM) it
 flushes
 100 Memtables with anywhere between 1 operation and 359 operations (35
 bytes
 and 12499 bytes)
>>>
>>> What's the settings of commit log flush, periodic or in batch?
>>>
>>>
>>> It's whatever the default setting is, (in the cassandra.yaml that is
>>> packaged in the apache-cassandra-0.7.3-bin.tar.gz download) specifically:
>>>
>>>    commitlog_rotation_threshold_in_mb: 128
>>>    commitlog_sync: periodic
>>>    commitlog_sync_period_in_ms: 1
>>>    flush_largest_memtables_at: 0.75
>>>
>>> If I describe keyspace I get:
>>>
>>>    [default@unknown] describe keyspace Events;
>>>    Keyspace: Events:
>>>      Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
>>>        Replication Factor: 1
>>>      Column Families:
>>>        ColumnFamily: Event
>>>          Columns sorted by: org.apache.cassandra.db.marshal.TimeUUIDType
>>>          Row cache size / save period: 0.0/0
>>>          Key cache size / save period: 20.0/14400
>>>          Memtable thresholds: 14.109375/3010/1440
>>>          GC grace seconds: 864000
>>>          Compaction min/max thresholds: 4/32
>>>          Read repair chance: 1.0
>>>          Built indexes: []
>>>
>>>
>>> It turns out my suspicion was right. When I tried overriding the jvm
>>> memory
>>> parameters calculated in conf/cassandra-env.sh to use the values
>>> calculated
>>> on my 8GB laptop like this:
>>>
>>>    MAX_HEAP_SIZE=3932m HEAP_NEWSIZE=400m ./mutate.sh
>>>
>>> That made the server behave much nicer.  This time it kept all 5000
>>> operations in a single Memtable.  Also, when running with these memory
>>> settings the Memtable thresholds changed to "1.1390625/243/1440"   (from
>>> "14.109375/3010/1440")      (all the other output from "describe
>>> keyspace"
>>> remains the same)
>>>
>>> So it looks like something goes wrong when cassandra gets too much
>>> memory.
>>>
>>>
>>> --
>>> Erik Forkalsrud
>>> Commission Junstion
>>>
>>>
>>
>>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Seed

2011-03-11 Thread Tyler Hobbs 
On Fri, Mar 11, 2011 at 2:55 PM, mcasandra  wrote:

> My question is it advisable to have node seed itself.
>

Yes, every node should have the same seed list, including the seeds
themselves.

-- 
Tyler Hobbs
Software Engineer, DataStax 
Maintainer of the pycassa  Cassandra
Python client library

Cassandra still won't start - in-use ports block it

2011-03-11 Thread Bob Futrelle 
My frustration continues, especially exasperating because so many people
just seem to download Cassandra and run it with no problems.
All my efforts have been stymied by one port-in-use problem after another.
People on this list have helped and their suggestions got me a little bit
further, but no further.

Platform,  MacBook Pro, OS 10.6.6

Java(TM) SE Runtime Environment (build 1.6.0_24-b07-334-10M3326)
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02-334, mixed mode)

Highlights of the problems:
...
WARN 07:48:28,482 Could not start register mbean in JMX
...
Caused by: java.net.BindException: Address already in use
...
ERROR 07:48:28,511 Exception encountered during startup.
java.lang.RuntimeException: Unable to create thrift socket to localhost/
10.0.1.3:9160
...
Caused by: org.apache.thrift.transport.TTransportException: Could not create
ServerSocket on address localhost/10.0.1.3:9160.
...

  - Bob Futrelle

Details:

apache-cassandra-0.7.3: sudo ./bin/cassandra -f -p pidfile
Password:
 INFO 07:48:27,851 Logging initialized
 INFO 07:48:27,862 Heap size: 1052770304/1052770304
 INFO 07:48:27,864 JNA not found. Native methods will be disabled.
 INFO 07:48:27,872 Loading settings from
file:/Users/robertfutrelle/Research/Cassandra/apache-cassandra-0.7.3/conf/cassandra.yaml
 INFO 07:48:27,993 DiskAccessMode 'auto' determined to be mmap,
indexAccessMode is mmap
 INFO 07:48:28,095 reading saved cache
/var/lib/cassandra/saved_caches/system-LocationInfo-KeyCache
 INFO 07:48:28,101 Opening /var/lib/cassandra/data/system/LocationInfo-f-9
 INFO 07:48:28,126 Opening /var/lib/cassandra/data/system/LocationInfo-f-10
 INFO 07:48:28,164 Couldn't detect any schema definitions in local storage.
 INFO 07:48:28,165 Found table data in data directories. Consider using JMX
to call org.apache.cassandra.service.StorageService.loadSchemaFromYaml().
 INFO 07:48:28,176 Creating new commitlog segment
/var/lib/cassandra/commitlog/CommitLog-1299847708176.log
 INFO 07:48:28,184 Replaying
/var/lib/cassandra/commitlog/CommitLog-1299809981227.log
 INFO 07:48:28,187 Finished reading
/var/lib/cassandra/commitlog/CommitLog-1299809981227.log
 INFO 07:48:28,187 Log replay complete
 INFO 07:48:28,207 Cassandra version: 0.7.3
 INFO 07:48:28,207 Thrift API version: 19.4.0
 INFO 07:48:28,209 Loading persisted ring state
 INFO 07:48:28,213 Starting up server gossip
 INFO 07:48:28,224 switching in a fresh Memtable for LocationInfo at
CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1299847708176.log',
position=148)
 INFO 07:48:28,224 Enqueuing flush of Memtable-LocationInfo@2098581612(29
bytes, 1 operations)
 INFO 07:48:28,225 Writing Memtable-LocationInfo@2098581612(29 bytes, 1
operations)
 INFO 07:48:28,305 Completed flushing
/var/lib/cassandra/data/system/LocationInfo-f-11-Data.db (80 bytes)
 INFO 07:48:28,336 Using saved token 68734258064819962813495316844051960711
 INFO 07:48:28,337 switching in a fresh Memtable for LocationInfo at
CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1299847708176.log',
position=444)
 INFO 07:48:28,337 Enqueuing flush of Memtable-LocationInfo@389001391(53
bytes, 2 operations)
 INFO 07:48:28,338 Writing Memtable-LocationInfo@389001391(53 bytes, 2
operations)
 INFO 07:48:28,404 Completed flushing
/var/lib/cassandra/data/system/LocationInfo-f-12-Data.db (163 bytes)
 INFO 07:48:28,405 Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-f-9-Data.db'),SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-f-10-Data.db'),SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-f-11-Data.db'),SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-f-12-Data.db')]
 INFO 07:48:28,472 Compacted to
/var/lib/cassandra/data/system/LocationInfo-tmp-f-13-Data.db.  770 to 447
(~58% of original) bytes for 3 keys.  Time: 65ms.
 WARN 07:48:28,482 Could not start register mbean in JMX
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.cassandra.utils.Mx4jTool.maybeLoad(Mx4jTool.java:66)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:203)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:316)
at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79)
Caused by: java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:383)
at java.net.ServerSocket.bind(ServerSocket.java:328)
at java.net.ServerSocket.(ServerSocket.java:194)
at
mx4j.tools.adaptor.PlainAdaptorServerSocketFactory.createServerSocket(PlainAdaptorServerSocketFactory.java:24)
at
mx4j.tools.adaptor.http.HttpAdaptor.cr

Re: Cassandra still won't start - in-use ports block it

2011-03-11 Thread Jeremy Hanna 
I don't know if others have asked this but do you have a firewall running that 
would prevent access to those ports or something like that?

On Mar 11, 2011, at 10:40 PM, Bob Futrelle wrote:

> My frustration continues, especially exasperating because so many people just 
> seem to download Cassandra and run it with no problems.
> All my efforts have been stymied by one port-in-use problem after another.
> People on this list have helped and their suggestions got me a little bit 
> further, but no further.
> 
> Platform,  MacBook Pro, OS 10.6.6
> 
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07-334-10M3326)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02-334, mixed mode)
> 
> Highlights of the problems:
> ...
> WARN 07:48:28,482 Could not start register mbean in JMX
> ...
> Caused by: java.net.BindException: Address already in use
> ...
> ERROR 07:48:28,511 Exception encountered during startup.
> java.lang.RuntimeException: Unable to create thrift socket to 
> localhost/10.0.1.3:9160
> ...
> Caused by: org.apache.thrift.transport.TTransportException: Could not create 
> ServerSocket on address localhost/10.0.1.3:9160.
> ...
> 
>   - Bob Futrelle
> 
> Details:
> 
> apache-cassandra-0.7.3: sudo ./bin/cassandra -f -p pidfile
> Password:
>  INFO 07:48:27,851 Logging initialized
>  INFO 07:48:27,862 Heap size: 1052770304/1052770304
>  INFO 07:48:27,864 JNA not found. Native methods will be disabled.
>  INFO 07:48:27,872 Loading settings from 
> file:/Users/robertfutrelle/Research/Cassandra/apache-cassandra-0.7.3/conf/cassandra.yaml
>  INFO 07:48:27,993 DiskAccessMode 'auto' determined to be mmap, 
> indexAccessMode is mmap
>  INFO 07:48:28,095 reading saved cache 
> /var/lib/cassandra/saved_caches/system-LocationInfo-KeyCache
>  INFO 07:48:28,101 Opening /var/lib/cassandra/data/system/LocationInfo-f-9
>  INFO 07:48:28,126 Opening /var/lib/cassandra/data/system/LocationInfo-f-10
>  INFO 07:48:28,164 Couldn't detect any schema definitions in local storage.
>  INFO 07:48:28,165 Found table data in data directories. Consider using JMX 
> to call org.apache.cassandra.service.StorageService.loadSchemaFromYaml().
>  INFO 07:48:28,176 Creating new commitlog segment 
> /var/lib/cassandra/commitlog/CommitLog-1299847708176.log
>  INFO 07:48:28,184 Replaying 
> /var/lib/cassandra/commitlog/CommitLog-1299809981227.log
>  INFO 07:48:28,187 Finished reading 
> /var/lib/cassandra/commitlog/CommitLog-1299809981227.log
>  INFO 07:48:28,187 Log replay complete
>  INFO 07:48:28,207 Cassandra version: 0.7.3
>  INFO 07:48:28,207 Thrift API version: 19.4.0
>  INFO 07:48:28,209 Loading persisted ring state
>  INFO 07:48:28,213 Starting up server gossip
>  INFO 07:48:28,224 switching in a fresh Memtable for LocationInfo at 
> CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1299847708176.log',
>  position=148)
>  INFO 07:48:28,224 Enqueuing flush of Memtable-LocationInfo@2098581612(29 
> bytes, 1 operations)
>  INFO 07:48:28,225 Writing Memtable-LocationInfo@2098581612(29 bytes, 1 
> operations)
>  INFO 07:48:28,305 Completed flushing 
> /var/lib/cassandra/data/system/LocationInfo-f-11-Data.db (80 bytes)
>  INFO 07:48:28,336 Using saved token 68734258064819962813495316844051960711
>  INFO 07:48:28,337 switching in a fresh Memtable for LocationInfo at 
> CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1299847708176.log',
>  position=444)
>  INFO 07:48:28,337 Enqueuing flush of Memtable-LocationInfo@389001391(53 
> bytes, 2 operations)
>  INFO 07:48:28,338 Writing Memtable-LocationInfo@389001391(53 bytes, 2 
> operations)
>  INFO 07:48:28,404 Completed flushing 
> /var/lib/cassandra/data/system/LocationInfo-f-12-Data.db (163 bytes)
>  INFO 07:48:28,405 Compacting 
> [SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-f-9-Data.db'),SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-f-10-Data.db'),SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-f-11-Data.db'),SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-f-12-Data.db')]
>  INFO 07:48:28,472 Compacted to 
> /var/lib/cassandra/data/system/LocationInfo-tmp-f-13-Data.db.  770 to 447 
> (~58% of original) bytes for 3 keys.  Time: 65ms.
>  WARN 07:48:28,482 Could not start register mbean in JMX
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.cassandra.utils.Mx4jTool.maybeLoad(Mx4jTool.java:66)
>   at 
> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:203)
>   at 
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:316)
>   at 
> org.apache.cassandra.thrift.CassandraDaemon.