This may not be directly related to the upgrade to 1.1.2, but I was running on
1.1.0 for a while with no issues, and I did the upgrade to 1.1.2 a few days ago.
2 of my nodes started throwing lots of promote exceptions, and then a lot of
the beforeAppend exceptions from then on every few minutes.
.
On 16.07.2012, at 20:32, Bryce Godfrey wrote:
This may not be directly related to the upgrade to 1.1.2, but I was running on
1.1.0 for a while with no issues, and I did the upgrade to 1.1.2 a few days ago.
2 of my nodes started throwing lots of promote exceptions, and then a lot of
the before
What is the process for joining a new data center to an existing cluster as DR?
We have a 5 node cluster in our primary DC, and want to bring up 5 more in our
2nd data center purely for DR. How should these new nodes be joined to the
cluster and be seen as the 2nd data center? Do the new nodes
We are in the process of building out a new DR system in another Data Center,
and we want to mirror our Cassandra environment to that DR. I have a couple
questions on the best way to do this after reading the documentation on the
Datastax website. We didn't initially plan for this to be a DR s
seful:
http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
Some comments inline:
On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey
mailto:bryce.godf...@azaleos.com>> wrote:
We are in the process of building out a new DR system in another Data Cent
management#adding-a-data-center-to-a-cluster
Some comments inline:
On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey
mailto:bryce.godf...@azaleos.com>> wrote:
We are in the process of building out a new DR system in another Data Center,
and we want to mirror our Cassandra environment to that D
, 10.20.8.3]
From: Mohit Anchlia [mailto:mohitanch...@gmail.com]
Sent: Friday, August 24, 2012 1:55 PM
To: user@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter
That's interesting can you do describe cluster?
On Fri, Aug 24, 2012 at 12:11 PM, Bryce Go
k the logs for errors ? Can you see your schema change in there ?
Also what is the output from show schema; in the cli ?
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 25/08/2012, at 6:53 PM, Bryce Godfrey
mailto:bryce.godf...@azaleos.c
Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey
mailto:bryce.godf...@azaleos.com>> wrote:
Show schema output show the simple strategy still
[default@unknown] show schema EBonding;
create keyspace EBonding
with placement_strategy = 'SimpleStrategy'
and strategy_options = {repli
Expanding cluster to include a new DR datacenter
Can you describe your schema again with TierPoint in it?
On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey
mailto:bryce.godf...@azaleos.com>> wrote:
Same results. I restarted the node also to see if it just wasn't picking up
the chang
n with TierPoint in it?
On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey
mailto:bryce.godf...@azaleos.com>> wrote:
Same results. I restarted the node also to see if it just wasn't picking up
the changes and it still shows Simple.
When I specify the DC for strategy_options I shoul
hard to say if it would work.
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 29/08/2012, at 9:10 AM, Bryce Godfrey
mailto:bryce.godf...@azaleos.com>> wrote:
I believe what may be really going on is that my schema is in a bad or co
Is there anything I can do on the configuration side to prevent nodes from
going OOM due to queries that will read large amounts of data and exceed the
heap available?
For the past few days of we had some nodes consistently freezing/crashing with
OOM. We got a heap dump into MAT and figured ou
We have a 5 node cluster, with a matching 5 nodes for DR in another data
center. With a replication factor of 3, does the node I send a write too
attempt to send it to the 3 servers in the DR also? Or does it send it to 1
and let it replicate locally in the DR environment to save bandwidth ac
AN. Though the
responses from all nodes in other DC will go over WAN.
On Thu, Oct 25, 2012 at 10:44 AM, Bryce Godfrey
mailto:bryce.godf...@azaleos.com>> wrote:
We have a 5 node cluster, with a matching 5 nodes for DR in another data
center. With a replication factor of 3, does the node I
I'm looking for some guidance on how to model some stat tracking over time,
bucketed to some type of interval (15 min, hour, etc).
As an example, let's say I would like to track network traffic throughput and
bucket it to 15 minute intervals. In our old model, using thrift I would
create a col
nf/log4j-server.properties.
maki
2011/6/1 Bryce Godfrey :
> Hi there, I'm a bit new to Linux and Cassandra so I'm hoping someone
> can help me with this.
>
>
>
> I've been evaluating Cassandra for the last few days and I'm now
> having a problem starting up the
I'm on 0.8.4
I have removed a dead node from the cluster using nodetool removetoken command,
and moved one of the remaining nodes to rebalance the tokens. Everything looks
fine when I run nodetool ring now, as it only lists the remaining 2 nodes and
they both look fine, owning 50% of the token
20.2
and 20.3 think 20.1 is still around ?
Cheers
-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 20/08/2011, at 9:48 AM, Bryce Godfrey wrote:
> I'm on 0.8.4
>
> I have removed a dead node from the cluster using n
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 22/08/2011, at 6:23 AM, Bryce Godfrey wrote:
> Both .2 and .3 list the same from the mbean that Unreachable is empty
> collection, and Live node lists all 3 nodes still:
> 192.168.20.2
> 192.168.20.3
> 192.168.2
Row cache: disabled
Compacted row minimum size: 2816160
Compacted row maximum size: 386857368
Compacted row mean size: 120432714
Is there a way for me to manually remove this dead node?
-Original Message-
From: Bryce Godfrey
Taking the cluster down completely did remove the phantom node. The
hintscolumnfamily is causing a lot of commit logs to back up and threaten the
commit log drive to run out of space. A manual flush of that column family
always clears out the files though.
-Original Message-
From: Br
I'm wondering what the consensus is for running a Cassandra cluster on top of
Windows boxes? We are currently running a small 5 node cluster on top of
CentOS without problems, so I have no desire to move. But we are a windows
shop, and I have an IT department that is scared of Linux since they
readPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Bryce Godfrey | Sr. Software Engineer | Azaleos
Corporation<http://www.azaleos.com/> | T: 206.926.1978 | M: 206.849.2477
Nope. I did alter two of my own column families to use Leveled compaction and
then ran scrub on each node, is the only change I have made from the upgrade.
Bryce Godfrey | Sr. Software Engineer | Azaleos Corporation | T: 206.926.1978 |
M: 206.849.2477
-Original Message-
From: Jonathan
Disk utilization is actually about 80% higher than what is reported for
nodetool ring across all my nodes on the data drive
Bryce Godfrey | Sr. Software Engineer | Azaleos
Corporation<http://www.azaleos.com/> | T: 206.926.1978 | M: 206.849.2477
From: Dan Hendry [mailto:dan.he
11 11:52 AM
To: user@cassandra.apache.org
Subject: Re: Problem after upgrade to 1.0.1
Does restarting the node fix this?
On Thu, Nov 3, 2011 at 1:51 PM, Bryce Godfrey wrote:
> Disk utilization is actually about 80% higher than what is reported
> for nodetool ring across all my nodes on the
M
To: user@cassandra.apache.org
Subject: Re: Problem after upgrade to 1.0.1
I found the problem and posted a patch on
https://issues.apache.org/jira/browse/CASSANDRA-3451. If you build with that
patch and rerun scrub the exception should go away.
On Thu, Nov 3, 2011 at 2:08 PM, Bryce Godfrey wrote:
> A
upgrade to 1.0.1
One possibility: If you're overloading the cluster, replicas will drop updates
to avoid OOMing. (This is logged at WARN level.) Before 1.x Cassandra would
just let that slide, but with w/ 1.0 it started recording hints for those.
On Thu, Nov 3, 2011 at 7:17 PM, Bryce
I'm seeing this same problem after upgrade to 1.0.3 from .8
Nothing changed with the column family storing the counters, but now it just
constantly times out trying to increment them. No errors in the event logs or
any other issues with my cluster.
Did you find a resolution?
From: Carlos Rolo
I have a dead node I need to remove from the cluster so that I can rebalance
among the existing servers (can't replace it for a while).
I used nodetool removetoken and it's been stuck in the "leaving" state for over
a day now. I've tried a rolling restart, which kicks of some streaming for a
w
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:81)
Now I see no streams going on between any nodes, and the node is still listed
as "moving" when viewing the ring.
From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com]
Sent: Sunday, December 11, 2011 11
I'm seeing this also, and my nodes have started crashing with "too many open
file errors". Running lsof I see lots of these open tmp files.
java 8185 root 911u REG 8,32 38
129108266
/opt/cassandra/data/MonitoringData/Properties-tmp-hc-268721-CompressionI
The system HintsColumnFamily seems large in my cluster, and I want to track
down why that is. I try invoking "listEndpointsPendingHints()" for
o.a.c.db.HintedHandoffManager and it never returns, and also freezes the node
that its invoked against. It's a 3 node cluster, and all nodes have been
Forgot to mention that this is on 1.0.8
From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com]
Sent: Wednesday, March 14, 2012 12:34 PM
To: user@cassandra.apache.org
Subject: Large hints column family
The system HintsColumnFamily seems large in my cluster, and I want to track
down why that is
at INFO level.
Also checking, do the hinted_handoff_* settings in cassandra.yaml have their
default settings ?
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 15/03/2012, at 8:35 AM, Bryce Godfrey wrote:
Forgot to mention that
://www.thelastpickle.com
On 16/03/2012, at 12:58 PM, Bryce Godfrey wrote:
We were having some occasional memory pressure issues, but we just added some
more RAM a few days ago to the nodes and things are running more smoothly now,
but in general nodes have not been going up and down.
I tried to
I keep running into this with my testing (on a windows box), Is this just a OOM
for RAM?
ERROR [COMMIT-LOG-ALLOCATOR] 2012-04-16 13:36:18,790
AbstractCassandraDaemon.java (line 134) Exception in thread
Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.io.IOError: java.io.IOException: Map failed
Mon, Apr 16, 2012 at 10:45 PM, Bryce Godfrey
wrote:
> I keep running into this with my testing (on a windows box), Is this just a
> OOM for RAM?
How much RAM do you have? Do you use completely standard settings? Do you also
OOM if you try the same test with Cassandra 1.0.9?
--
S
Per CF or per Row TTL would be very usefull for me also with our timeseries
data.
-Original Message-
From: Igor [mailto:i...@4friends.od.ua]
Sent: Wednesday, April 18, 2012 6:06 AM
To: user@cassandra.apache.org
Subject: Re: size tiered compaction - improvement
For my use case it would b
This is the second node I've joined to my cluster in the last few days, and so
far both have become stuck at 100% on a large file according to netstats. This
is on 1.0.9, is there anything I can do to make it move on besides restarting
Cassandra? I don't see any errors or warns in logs for eit
The commit log drives on my nodes keep slowly filling up. I don't see any
errors in my logs that are indicating any issues that I can map to this issue.
Is this how 1.1 is supposed to work now? Previous versions seemed to keep this
drive at a minimum as it flushed.
/dev/mapper/mpathf 25G
with files of 128MB.
> (https://issues.apache.org/jira/browse/CASSANDRA-3411) This should
> however not exceed your commitlog size in Cassandra.yaml.
>
>
>
> commitlog_total_space_in_mb: 4096
>
>
>
> Kind regards,
>
> Pieter Callewaert
>
>
>
> From: Bry
also the internal hard coded default for commitlog_total_space_in_mb
If you are seeing more that 4GB of commit log files let us know.
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 22/05/2012, at 6:35 AM, Bryce Godfrey wrote:
Thanks, I&#x
not reclaim the space either and delete any log files.
Bryce Godfrey | Sr. Software Engineer | Azaleos
Corporation<http://www.azaleos.com/>
From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com]
Sent: Tuesday, May 22, 2012 1:10 PM
To: user@cassandra.apache.org
Subject: RE: 1.1 not re
I'll try to get some log files for this with DEBUG enabled. Tough on
production though.
From: aaron morton [mailto:aa...@thelastpickle.com]
Sent: Monday, June 04, 2012 11:15 AM
To: user@cassandra.apache.org
Subject: Re: 1.1 not removing commit log files?
Apply the local hint mutation follows th
As the new node starts up I get this error before boostrap starts:
INFO 08:20:51,584 Enqueuing flush of Memtable-schema_columns@1493418651(0/0
serialized/live bytes, 1 ops)
INFO 08:20:51,584 Writing Memtable-schema_columns@1493418651(0/0
serialized/live bytes, 1 ops)
INFO 08:20:51,589 Completed
12 at 12:28 AM, Bryce Godfrey
wrote:
> As the new node starts up I get this error before boostrap starts:
>
>
>
> INFO 08:20:51,584 Enqueuing flush of
> Memtable-schema_columns@1493418651(0/0
> serialized/live bytes, 1 ops)
>
> INFO 08:20:51,584 Writing Memtab
48 matches
Mail list logo
