I agree with Jon. It's almost a statistical certainty that such updates
will be processed out of order some of the time because the clock sync
between machines will never be perfect.
Depending on how your actual code that shows this problem is structured,
there are ways to reduce or eliminate such
High volume updates to a single key in a distributed system that relies on
a timestamp for conflict resolution is not a particularly great idea. If
you ever do this from multiple clients you'll find unexpected results at
least some of the time.
On Tue, Dec 15, 2015 at 12:41 PM Paulo Motta
wrote:
> We are using 2.1.7.1
Then you should be able to use the java driver timestamp generators.
> So, we need to look for clock sync issues between nodes in our ring? How
close do they need to be?
millisecond precision since that is the server precision for timestamps, so
probably NTP should do the
On Tue, Dec 15, 2015 at 2:57 PM Paulo Motta
wrote:
> What cassandra and driver versions are you running?
>
>
We are using 2.1.7.1
> It may be that the second update is getting the same timestamp as the
> first, or even a lower timestamp if it's being processed by another server
> with unsynced
What cassandra and driver versions are you running?
It may be that the second update is getting the same timestamp as the
first, or even a lower timestamp if it's being processed by another server
with unsynced clock, so that update may be getting lost.
If you have high frequency updates in the s
We are encountering a situation in our environment (a 6-node Cassandra
ring) where we are trying to insert a row and then immediately update it,
using LOCAL_QUORUM consistency (replication factor = 3). I have replicated
the issue using the following code:
https://gist.github.com/jwcarman/72714e6d
It did, but a ran it again on one node – that node never recovered. ☹
From: Robert Coli [mailto:rc...@eventbrite.com]
Sent: 02 October 2015 21:20
To: user@cassandra.apache.org
Subject: Re: Consistency Issues
On Fri, Oct 2, 2015 at 1:32 AM, Walsh, Stephen
mailto:stephen.wa...@aspect.com>>
On Fri, Oct 2, 2015 at 1:32 AM, Walsh, Stephen
wrote:
> Sorry for the late reply, I ran the nodetool resetlocalschema on all
> nodes but in the end it just removed all the schemas and crashed the
> applications.
>
> I need to reset and try again. I’ll try get you the gc stats today J
>
FTR, runn
en.wa...@aspect.com]
Sent: 02 October 2015 09:32
To: user@cassandra.apache.org
Subject: RE: Consistency Issues
Sorry for the late reply, I ran the nodetool resetlocalschema on all nodes but
in the end it just removed all the schemas and crashed the applications.
I need to reset and try again. I’ll try ge
:01
To: user@cassandra.apache.org
Subject: Re: Consistency Issues
You say that you don't think GC is your issue... but did you actually check?
The reasons you suggest aren't very convincing. Can you provide your GC
settings, and take a look at jstat --gccause?
http://docs.oracle.co
t;
>
>
> But I really need to understand this best practice that was mentioned (on
> number of CF’s) by Jack Krupansky.
>
> Anyone more information on this?
>
>
>
>
>
> Many thanks for all your help guys keep it coming J
>
> Steve
>
>
>
> *From:*
k...@gmail.com]
> *Sent:* 01 October 2015 14:00
> *To:* user
> *Subject:* Re: Consistency Issues
>
>
>
> Onur, was responding to Stephen's issue.
>
>
>
>
>
> On Thu, Oct 1, 2015 at 8:56 AM, Onur Yalazı
> wrote:
>
> Thank you Jake.
>
>
ode 1 : has all CF’s
>
> Node 2 : has all CF’s
>
> Node 3 : has all CF’s
>
> Node 4 : has all CF’s
>
>
>
>
>
> This is indeed very strange….
>
>
>
>
>
> *From:* Carlos Alonso [mailto:i...@mrcalonso.com]
> *Sent:* 01 October 2015 12:05
> *To:*
Thanks Jake, I’ll try test out 2.1.9 to see if it resolved the issue and ill
try “nodetool resetlocalschema” now to see if it helps.
Cassandra is 2.1.6
OpsCenter is 5.2.1
From: Jake Luciani [mailto:jak...@gmail.com]
Sent: 01 October 2015 14:00
To: user
Subject: Re: Consistency Issues
Onur
Onur, was responding to Stephen's issue.
On Thu, Oct 1, 2015 at 8:56 AM, Onur Yalazı wrote:
> Thank you Jake.
>
> The issue is I do not have missing CF's and upgrading beyond 2.1.3 is not
> a possibility because of the deprecation of cql dialects. Our application
> is using Hector and migrating
Thank you Jake.
The issue is I do not have missing CF's and upgrading beyond 2.1.3 is
not a possibility because of the deprecation of cql dialects. Our
application is using Hector and migrating to cql3 is a huge refactoring.
On 01/10/15 15:48, Jake Luciani wrote:
Couple things to try.
1. n
Couple things to try.
1. nodetool resetlocalschema on the nodes with missing CFs. This will
refresh the schema on the local node.
2. upgrade to 2.1.9. There are some pretty major issues in 2.1.6 (nothing
specific to this problem but worth upgrading)
deed very strange….
>
>
>
>
>
> *From:* Carlos Alonso [mailto:i...@mrcalonso.com]
> *Sent:* 01 October 2015 12:05
> *To:* user@cassandra.apache.org
> *Subject:* Re: Consistency Issues
>
>
>
> And that's a stupid one, I know, but does the column you're tryi
CF’s
This is indeed very strange….
From: Carlos Alonso [mailto:i...@mrcalonso.com]
Sent: 01 October 2015 12:05
To: user@cassandra.apache.org
Subject: Re: Consistency Issues
And that's a stupid one, I know, but does the column you're trying to access
actually exist?
Carlos Alonso | Sof
J
>
>
>
>
>
> *From:* Carlos Alonso [mailto:i...@mrcalonso.com]
> *Sent:* 01 October 2015 10:11
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: Consistency Issues
>
>
>
> Hi Stephen.
>
>
>
> The UnknownColumnFamilyException made me tho
I did think of that and they are all the same version ☺
From: Carlos Alonso [mailto:i...@mrcalonso.com]
Sent: 01 October 2015 10:11
To: user@cassandra.apache.org
Subject: Re: Consistency Issues
Hi Stephen.
The UnknownColumnFamilyException made me thought of a possible schema
disagreement in
help guys keep it coming J
>
> Steve
>
>
>
> *From:* Ricardo Sancho [mailto:sancho.rica...@gmail.com]
> *Sent:* 01 October 2015 09:39
> *To:* user@cassandra.apache.org
> *Subject:* RE: Consistency Issues
>
>
>
> Can you tell us how much time your gcs are taking?
ancho [mailto:sancho.rica...@gmail.com]
Sent: 01 October 2015 09:39
To: user@cassandra.apache.org
Subject: RE: Consistency Issues
Can you tell us how much time your gcs are taking?
Do you see any especially long ones?
On 1 Oct 2015 09:37, "Walsh, Stephen"
mailto:stephen.wa...@aspect.com>> wr
ot the root causing of the
> inconsistency issue.
>
>
>
> Can anyone verify the best practice for number of CF’s?
>
>
>
>
>
> *From:* Robert Coli [mailto:rc...@eventbrite.com]
> *Sent:* 30 September 2015 18:45
> *To:* user@cassandra.apache.org
> *Subject:
September 2015 18:45
To: user@cassandra.apache.org
Subject: Re: Consistency Issues
On Wed, Sep 30, 2015 at 9:06 AM, Walsh, Stephen
mailto:stephen.wa...@aspect.com>> wrote:
We never had these issue with our first run. Its only when we added another 25%
of writes.
As Jack said, you are pr
On Wed, Sep 30, 2015 at 9:06 AM, Walsh, Stephen
wrote:
>
> We never had these issue with our first run. Its only when we added
> another 25% of writes.
>
As Jack said, you are probably pushing your GC over a threshold, leading to
long pause times and inability to meet quorum.
As Sebastian said,
t in best practices for the number of CF’s?
From: Sebastian Estevez [mailto:sebastian.este...@datastax.com]
Sent: 30 September 2015 17:29
To: user@cassandra.apache.org
Subject: Re: Consistency Issues
Can you provide exact details on where your load balancer is? Like Michael
said, you shouldn&#
.
>
>
>
> Many thanks for taking the time to reply Jack
>
>
>
>
>
>
>
> *From:* Jack Krupansky [mailto:jack.krupan...@gmail.com]
> *Sent:* 30 September 2015 16:53
> *To:* user@cassandra.apache.org
> *Subject:* Re: Consistency Issues
>
>
>
> M
these issue with our first run. Its only when we added another 25%
of writes.
Many thanks for taking the time to reply Jack
From: Jack Krupansky [mailto:jack.krupan...@gmail.com]
Sent: 30 September 2015 16:53
To: user@cassandra.apache.org
Subject: Re: Consistency Issues
More than "low hun
1e5-a435-e7be20c92086
>
>
>
> Any idea what this is related too?
>
> All these tests are run with a clean setup of Cassandra nodes followed by
> a nodetool repair.
>
> Before any data hits them.
>
>
>
>
>
> *From:* Walsh, Stephen [mailto:stephen.wa...@asp
gt; a nodetool repair.
>
> Before any data hits them.
>
>
>
>
>
> *From:* Walsh, Stephen [mailto:stephen.wa...@aspect.com]
> *Sent:* 30 September 2015 15:17
> *To:* user@cassandra.apache.org
> *Subject:* Consistency Issues
>
>
>
> Hi there,
>
>
>
stephen.wa...@aspect.com]
Sent: 30 September 2015 15:17
To: user@cassandra.apache.org
Subject: Consistency Issues
Hi there,
We are having some issues with consistency. I'll try my best to explain.
We have an application that was able to
Write ~1000 p/s
Read ~300 p/s
Total CF created: 400
Hi there,
We are having some issues with consistency. I'll try my best to explain.
We have an application that was able to
Write ~1000 p/s
Read ~300 p/s
Total CF created: 400
Total Keyspaces created : 80
On a 4 node Cassandra Cluster with
Version 2.1.6
Replication : 3
Consistency (Read & Write)
It looks like NTP was the problem. Thanks for the solution!!!
On Wed, May 13, 2015 at 9:20 AM, Robert Wille wrote:
> Timestamps have millisecond granularity. If you make multiple writes
> within the same millisecond, then the outcome is not deterministic.
>
> Also, make sure you are running n
Timestamps have millisecond granularity. If you make multiple writes within the
same millisecond, then the outcome is not deterministic.
Also, make sure you are running ntp. Clock skew will manifest itself similarly.
On May 13, 2015, at 3:47 AM, Jared Rodriguez
mailto:jrodrig...@kitedesk.com>>
Thanks for the feedback. We have dug in deeper and upgraded to Cassandra
2.0.14 and are seeing the same issue. What appears to be happening is that
if a record is initially written, then the first read is fine. But if we
immediately update that record with a second write, that then the second
re
On Tue, May 12, 2015 at 12:35 PM, Michael Shuler
wrote:
> This is a 4 node cluster running Cassandra 2.0.6
>>
>
> Can you reproduce the same issue on 2.0.14? (or better yet, the
> cassandra-2.0 branch HEAD, which will soon ship 2.0.15) If you get the same
> results, please, open a JIRA with the r
On 05/12/2015 04:50 AM, Jared Rodriguez wrote:
I have a specific update and query that I need to ensure has strong
consistency. To that end, when I do the write, I set the consistency
level to ALL. Shortly afterwards, I do a query for that record with a
consistency of ONE and somehow get back s
I have a specific update and query that I need to ensure has strong
consistency. To that end, when I do the write, I set the consistency level
to ALL. Shortly afterwards, I do a query for that record with a
consistency of ONE and somehow get back stale data.
This is a 4 node cluster running Cass
39 matches
Mail list logo
