I tried using 'nodetool rebuild' after I add the datacenters,date same
outcome, and after I decommission my keyspaces are getting wiped out, I
don't understand this.
On Thu, Aug 7, 2014 at 1:54 PM, srmore wrote:
>
> Thanks for the detailed reply Ken, this really helps.
enter. So, does nodetool rebuild help here ?
[1] https://wiki.apache.org/cassandra/Operations#Removing_nodes_entirely
>
>
>
>
> On Thu, Aug 7, 2014 at 2:11 PM, srmore wrote:
>
>>
>>
>>
>> On Thu, Aug 7, 2014 at 12:27 PM, Robert Coli
>> wrote:
>&
On Thu, Aug 7, 2014 at 12:27 PM, Robert Coli wrote:
> On Thu, Aug 7, 2014 at 10:04 AM, srmore wrote:
>
>> Sorry for being ambiguous. By "deletes" I mean that running decommission
>> I can no longer see any keyspaces owned by this node or replicated by other
>&g
atacenters).
On Thu, Aug 7, 2014 at 11:43 AM, Robert Coli wrote:
> On Thu, Aug 7, 2014 at 8:26 AM, srmore wrote:
>
>>
>> tl;dr: Decommissioning datacenters by running nodetool decommission on a
>> node deletes the data on the decommissioned node - is this expected ?
>>
tl;dr: Decommissioning datacenters by running nodetool decommission on a
node deletes the data on the decommissioned node - is this expected ?
I am trying our some tests on my multi-datacenter setup. Somewhere in the
docs I read that decommissioning a node will stream its data to other nodes
but i
Sorry to hear that Robert, I ran into similar issue a while ago. I had an
extremely heavy write and update load, as a result Cassandra (1.2.9) was
constantly flushing to disk and used to GC, tried exactly the same steps
you tried (tuning memtable_flush_writers (to 2) and
memtable_flush_queue_size (
Hello Kyle,
For your first question, you need to create aliases to localhost e.g.
127.0.0.2,127.0.0.3 etc. this should get you going.
About the logging issue, I think if your instance failing before it gets to
long anything, as an example you can strart one instance and make sure it
logs correctly.
What version of Cassandra are you running ? I used to see them a lot with
1.2.9, I could correlate the dropped messages with the heap usage almost
every time, so check in the logs whether you are getting GC'd. In this
respect 1.2.12 appears to be more stable. Moving to 1.2.12 took care of
this for
What is your thread stack size (xss) ? try increasing that, that could
help. Sometimes the limitation is imposed by the host provider (e.g.
amazon ec2 etc.)
Thanks,
Sandeep
On Mon, Dec 16, 2013 at 6:53 AM, Oleg Dulin wrote:
> Hi guys!
>
> I beleive my limits settings are correct. Here is the o
, is there an easier way to update the data from RandomPartitioner to
Murmur3 ? (upgradesstable ?)
>
> On Fri, Dec 6, 2013 at 10:36 AM, srmore wrote:
>
>>
>>
>>
>> On Fri, Dec 6, 2013 at 9:59 AM, Vicky Kak wrote:
>>
>>> You have passed the JVM configu
to find something.
-sandeep
> Cheers
>
> -
> Aaron Morton
> New Zealand
> @aaronmorton
>
> Co-Founder & Principal Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> On 12/12/2013, at 3:39 pm, srmore wrote:
>
> Thanks Aaron
>
>
>
the number of threads, did not help much.
> Hope that helps.
>
> -
> Aaron Morton
> New Zealand
> @aaronmorton
>
> Co-Founder & Principal Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> On 7/12/2013, at 8:05 am, srmore w
1-0-improved-memory-and-disk-space-management
>
> The flushing of 2.6 gb to the disk might slow the performance if
> frequently called, may be you have lots of write operations going on.
>
>
>
> On Fri, Dec 6, 2013 at 10:06 PM, srmore wrote:
>
>>
>>
>&g
), capacity 0 (bytes), 0 hits, 0 requests,
NaN recent hit rate, 0 save period in seconds
On Fri, Dec 6, 2013 at 11:15 AM, Vicky Kak wrote:
> Since how long the server had been up, hours,days,months?
>
>
> On Fri, Dec 6, 2013 at 10:41 PM, srmore wrote:
>
>> Looks like I
:
> Hi srmore,
>
> Perhaps if you use jconsole and connect to the jvm using jmx. Then uner
> MBeans tab, start inspecting the GC metrics.
>
> /Jason
>
>
> On Fri, Dec 6, 2013 at 11:40 PM, srmore wrote:
>
>>
>>
>>
>> On Fri, Dec 6, 2013 at 9:32
> The spikes are not that significant in our case and we are running the
> cluster with 1.7 gb heap.
>
> Are these spikes causing any issue at your end?
>
There are no big spikes, the overall performance seems to be about 40% low.
>
>
>
>
> On Fri, Dec 6, 2013 at
hold=2
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
> Yes compactions/GC's could skipe the CPU, I had similar behavior with my
> setup.
>
Were you able to get around it ?
>
> -VK
>
>
> On Fri, Dec 6, 2013 at 7:40 PM, srmore wrote:
&g
We have a 3 node cluster running cassandra 1.2.12, they are pretty big
machines 64G ram with 16 cores, cassandra heap is 8G.
The interesting observation is that, when I send traffic to one node its
performance is 2x more than when I send traffic to all the nodes. We ran
1.0.11 on the same box and
lso did a read before a write.
>
> After we got through the initial data migration, our normal workload is
> *much* less writes (and reads for that matter) such that our cluster can
> easily handle it, so we didn't investigate further.
>
> -- C
>
>
> On Sat, Nov 23, 2
Hello,
We moved to cassandra 1.2.9 from 1.0.11 to take advantage of the off-heap
bloom filters and other improvements.
We see a lot of messages dropped under high load conditions. We noticed
that when we do heavy read AND write simultaneously (we read first and
check whether the key exists if not
(it was the
> default for a while in 1.2).
>
> Cheers
>
> -
> Aaron Morton
> New Zealand
> @aaronmorton
>
> Co-Founder & Principal Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> On 13/11/2013, at 3:09 am, srmore
to centos6.
>
>
>
> Here is someone having a similar issue.
>
>
> http://mail-archives.apache.org/mod_mbox/cassandra-commits/201307.mbox/%3CJIRA.12616012.1352862646995.6820.1373083550278@arcas%3E
>
>
>
>
>
> *From:* srmore [mailto:comom...@gmail.com]
>
I might be missing something obvious here, for some reason I cannot seem to
get internode_compression = all to work. I am getting the following
exception. I am using cassandra 1.2.9 and have snappy-java-1.0.5.jar in my
classpath. Google search did not return any useful result, has anyone seen
this
The problem was cross_node_timeout value,I had it set to true and my ntp
clocks were not synchronized as a result, some of the requests were
dropped.
Thanks,
Sandeep
On Sat, Nov 9, 2013 at 6:02 PM, srmore wrote:
> I recently upgraded to 1.2.9 and I am seeing a lot of REQUEST_RESPONSE
I recently upgraded to 1.2.9 and I am seeing a lot of REQUEST_RESPONSE and
MUTATION messages are being dropped.
This happens when I have multiple nodes in the cluster (about 3 nodes) and
I send traffic to only one node. I don't think the traffic is that high, it
is around 400 msg/sec with 100 thre
We ran into similar heap issues a while ago for 1.0.11, I am not sure
whether you are at the luxury of upgrading to at-least 1.2.9, we were not.
After a lot of various painful attempts and weeks of testing (just as in
your case) the following settings worked (did not completely relieve the
heap pre
Thanks Rob that helps !
On Fri, Oct 25, 2013 at 7:34 PM, Robert Coli wrote:
> On Fri, Oct 25, 2013 at 2:47 PM, srmore wrote:
>
>> I don't know whether this is possible but was just curious, can you query
>> for the data in the remote datacenter with a CL.ONE ?
>>
I don't know whether this is possible but was just curious, can you query
for the data in the remote datacenter with a CL.ONE ?
There could be a case where one might not have a QUORUM and would like to
read the most recent data which includes the data from the other
datacenter. AFAIK to reliably
t;
> W dniu 02.10.2013 16:32, srmore pisze:
>
> I changed my index_interval from 128 to index_interval: 128 to 512, does
>> it
>> make sense to increase more than this ?
>>
>>
>> On Wed, Oct 2, 2013 at 9:30 AM, cem wrote:
>>
>> Have a look to index_i
uld you tell more about your traffic
> characteristics?
>
> Cem
>
>
> On Wed, Oct 2, 2013 at 4:32 PM, srmore wrote:
>
>> I changed my index_interval from 128 to index_interval: 128 to 512, does
>> it make sense to increase more than this ?
>>
>&g
I changed my index_interval from 128 to index_interval: 128 to 512, does it
make sense to increase more than this ?
On Wed, Oct 2, 2013 at 9:30 AM, cem wrote:
> Have a look to index_interval.
>
> Cem.
>
>
> On Wed, Oct 2, 2013 at 2:25 PM, srmore wrote:
>
>> The vers
size is function of
> number of keys/metadata. In Cassandra 1.2 lot of the metadata like bloom
> filters were moved off heap.
>
>
> On Tue, Oct 1, 2013 at 9:34 PM, srmore wrote:
>
>> Does anyone know what would roughly be the heap size for cassandra with
>> 1TB of data ?
Does anyone know what would roughly be the heap size for cassandra with 1TB
of data ? We started with about 200 G and now on one of the nodes we are
already on 1 TB. We were using 8G of heap and that served us well up until
we reached 700 G where we started seeing failures and nodes flipping.
With
Was too fast on the send button, sorry.
The thing I wanted to add was the
pending signals (-i) 515038
that looks odd to me, could that be related.
On Thu, Sep 19, 2013 at 4:53 PM, srmore wrote:
>
> I hit this issue again today and looks like changing -Xss option do
(seconds, -t) unlimited
max user processes (-u) 515038
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Has anyone run into this ?
[1] http://www.datastax.com/docs/1.1/troubleshooting/index
On Wed, Sep 11, 2013 at 8:47 AM, srmore wrote:
istribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
>
> *From:* srmore [mailto:comom...@gmail.com]
> *Sent:* Tuesday, September 10, 2013
I would be interested to know that too, it would be great if anyone can
share how they do (or do not) track or monitor cross datacenter migrations.
Thanks !
On Wed, Sep 4, 2013 at 10:13 AM, Anand Somani wrote:
> Hi,
>
> Scenario is a cluster spanning across datacenters and we use Local_quorum
I have a 5 node cluster with a load of around 300GB each. A node went down
and does not come up. I can see the following exception in the logs.
ERROR [main] 2013-09-09 21:50:56,117 AbstractCassandraDaemon.java (line
139) Fatal exception in thread Thread[main,5,main]
java.lang.OutOfMemoryError: una
On Mon, Aug 12, 2013 at 2:49 PM, Robert Coli wrote:
> On Mon, Aug 12, 2013 at 12:31 PM, srmore wrote:
>
>> There are some operations that demand the use lock and I was wondering
>> whether Cassandra has a built in locking mechanism. After hunting the web
>> for a w
All,
There are some operations that demand the use lock and I was wondering
whether Cassandra has a built in locking mechanism. After hunting the web
for a while it appears that the answer is no, although I found this
outdated wiki page which describes the algorithm
http://wiki.apache.org/cassandra
Thanks Takenori,
Looks like the tool provides some good info that people can use. It would
be great if you can share it with the community.
On Thu, Jul 11, 2013 at 6:51 AM, Takenori Sato wrote:
> Hi,
>
> I think it is a common headache for users running a large Cassandra
> cluster in productio
On Fri, Jul 5, 2013 at 6:08 PM, Robert Coli wrote:
> On Thu, Jul 4, 2013 at 10:03 AM, srmore wrote:
>
>> We are planning to move data from a 2 node cluster to a 3 node cluster.
>> We are planning to copy the data from the two nodes (snapshot) to the new 2
>> nodes and ho
We are planning to move data from a 2 node cluster to a 3 node cluster. We
are planning to copy the data from the two nodes (snapshot) to the new 2
nodes and hoping that Cassandra will sync it to the third node. Will this
work ? are there any other commands to run after we are done migrating,
like
issues compacting deleted rows and get poor read
>> performance with a value that high. I'd guess that anything larger than
>> 0.1 might as well be 1.0.
>>
>> -Bryan
>>
>>
>>
>> On Fri, Jun 21, 2013 at 5:58 AM, srmore wrote:
>>
>&g
>
> It theoretically triggers a full GC and when it will happen depends on the
> JVM
>
> -Wei
>
> --
> *From: *"Robert Coli"
> *To: *user@cassandra.apache.org
> *Sent: *Tuesday, June 18, 2013 10:43:13 AM
> *Subject: *Re: Heap is not released and s
pgrade but gotto
work with this for now).
On Tue, Jun 18, 2013 at 12:13 PM, Robert Coli wrote:
> On Tue, Jun 18, 2013 at 8:25 AM, srmore wrote:
> > I see an issues when I run high traffic to the Cassandra nodes, the heap
> > gets full to about 94% (which is expected)
>
>
I see an issues when I run high traffic to the Cassandra nodes, the heap
gets full to about 94% (which is expected) but the thing that confuses me
is that the heap usage never goes down after the traffic is stopped
(at-least, it appears to be so) . I kept the nodes up for a day after
stopping the t
I am seeing the similar behavior, in my case I have 2 nodes in each
datacenter and one node always has high latency (equal to the latency
between the two datacenters). When one of the datacenters is shutdown the
latency drops.
I am curious to know whether anyone else has these issues and if yes ho
Hello All,
We are thinking of going with Cassandra on a 8 core machine, are there any
optimizations that can help us here ?
I have seen that during startup stage Cassandra uses only one core, is
there a way we can speed up the startup process ?
Thanks !
Mon, Jun 3, 2013 at 3:37 PM, Hiller, Dean wrote:
> Our badness threshold is 0.1 currently(just checked). Our website used to
> get slow during a slow node time until we rolled our own patch out.
>
> Dean
>
> From: srmore mailto:comom...@gmail.com>>
> Reply-To:
Yup, RF is 2 for both the datacenters.
On Mon, Jun 3, 2013 at 3:36 PM, Sylvain Lebresne wrote:
> What's your replication factor? Do you have RF=2 on both datacenters?
>
>
> On Mon, Jun 3, 2013 at 10:09 PM, srmore wrote:
>
>> I am a bit confused when using the c
other DC nodes out of the
> list of reading from for you as well. I need to circle back to with my
> teammate to check if he got his fix posted to the dev list or not.
>
> Later,
> Dean
>
> From: srmore mailto:comom...@gmail.com>>
> Reply-To: "user@cassandra.
With CL=TWO it appears that one node randomly picks the node from other
datacenter to get the data. i.e. one node in the datacenter consistently
underperforms.
On Mon, Jun 3, 2013 at 3:21 PM, Hiller, Dean wrote:
> What happens when you use CL=TWO.
>
> Dean
>
> From: srmo
I am a bit confused when using the consistency level for multi datacenter
setup. Following is my setup:
I have 4 nodes the way these are set up are
Node 1 DC 1 - N1DC1
Node 2 DC 1 - N2DC1
Node 1 DC 2 - N1DC2
Node 2 DC 2 - N2DC2
I setup a delay in between two datacenters (DC1 and DC2 around 1 sec
n2.nabble.com/Is-it-safe-to-stop-a-read-repair-and-any-suggestion-on-speeding-up-repairs-td6607367.html
>
> Thanks
>
> On May 29, 2013, at 9:32 PM, srmore wrote:
>
> Hello,
> I am observing that my performance is drastically decreasing when my data
> size grows. I have a 3
p://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2
>
> On Wed, May 29, 2013 at 11:32 PM, srmore wrote:
> > Hello,
> > I am observing that my performance is drastically decreasing when my data
> > size grows. I have a 3 node cluster with 64 GB of ram and
Hello,
I am observing that my performance is drastically decreasing when my data
size grows. I have a 3 node cluster with 64 GB of ram and my data size is
around 400GB on all the nodes. I also see that when I re-start Cassandra
the performance goes back to normal and then again starts decreasing af
Network_Time_Protocol
>
> Hope that helps.
>
> -
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 10/05/2013, at 9:16 AM, srmore wrote:
>
> Thanks Rob !
>
> Tried the steps, that did no
at 1:58 PM, Robert Coli wrote:
> On Wed, May 8, 2013 at 5:40 PM, srmore wrote:
> > After running the commands, I get back to the same issue. Cannot afford
> to
> > lose the data so I guess this is the only option for me. And
> unfortunately I
> > am using 1.0.12 ( cann
Hello,
I have a cluster of 4 nodes and two of them are on different schema. I
tried to run the commands described in the FAQ section but no luck (
http://wiki.apache.org/cassandra/FAQ#schema_disagreement) .
After running the commands, I get back to the same issue. Cannot afford to
lose the data s
60 matches
Mail list logo
