Which versions of Cassandra and OpsCenter are you using? Because probably opscenter and your app are using cql and cqlsh is using thrift or vice versa and that's why depending on where you access from you see different things?
Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso> On 1 October 2015 at 13:06, Walsh, Stephen <stephen.wa...@aspect.com> wrote: > No such thing as a stupid questionJ > > I know they exist in some nodes, but if they replicated correctly is a > different story. > > I’m checking this one now, > > > > Ok, hooked up OpsCenter to see what it was saying, > > Out of the 100 keyspaces creted, > > 9 are missing one CF > > 2 are missing two CF’s > > 1 is missing three CF’s > > > > It looks like the replication of the tables did not complete to all nodes? > > > > Looking at each of the 4 nodes at the keyspace with 3 missing CF’s > > (via CQLSH_HOST=x.x.x.x cqlsh & “Describe keyspace XXX;”) > > > > Node 1 : has all CF’s > > Node 2 : has all CF’s > > Node 3 : has all CF’s > > Node 4 : has all CF’s > > > > > > This is indeed very strange…. > > > > > > *From:* Carlos Alonso [mailto:i...@mrcalonso.com] > *Sent:* 01 October 2015 12:05 > *To:* user@cassandra.apache.org > *Subject:* Re: Consistency Issues > > > > And that's a stupid one, I know, but does the column you're trying to > access actually exist? > > > Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso> > > > > On 1 October 2015 at 11:09, Walsh, Stephen <stephen.wa...@aspect.com> > wrote: > > I did think of that and they are all the same version J > > > > > > *From:* Carlos Alonso [mailto:i...@mrcalonso.com] > *Sent:* 01 October 2015 10:11 > > > *To:* user@cassandra.apache.org > *Subject:* Re: Consistency Issues > > > > Hi Stephen. > > > > The UnknownColumnFamilyException made me thought of a possible schema > disagreement in which any of your nodes has a different version and > therefore you cannot reach quorum? > > > > Can you run nodetool describecluster and see if all nodes have the same > schema versions? > > > > Cheers! > > > Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso> > > > > On 1 October 2015 at 09:49, Walsh, Stephen <stephen.wa...@aspect.com> > wrote: > > If you’re looking for the clean-up of the old gen in the jvm heap, it > doesn’t happen. > > We have a new gen turning 15 times before its pushed to old gen. > > Seems all our data only has a TTL of 10 seconds – very little data is sent > to the old gen. > > > > Add in heap size of 8GB with a new gen size of 2GB, I don’t think gc is > our issue. > > > > > > I’m more worried about error messages in the Cassandra log file that state. > > > > > > UnknownColumnFamilyException reading from socket; closing > > org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find > cfId=cf411b50-6785-11e5-a435-e7be20c92086 > > > > and > > > > cassandra OutboundTcpConnection.java:313 - error writing to Connection. > > > > > > > > But I really need to understand this best practice that was mentioned (on > number of CF’s) by Jack Krupansky. > > Anyone more information on this? > > > > > > Many thanks for all your help guys keep it coming J > > Steve > > > > *From:* Ricardo Sancho [mailto:sancho.rica...@gmail.com] > *Sent:* 01 October 2015 09:39 > *To:* user@cassandra.apache.org > *Subject:* RE: Consistency Issues > > > > Can you tell us how much time your gcs are taking? > Do you see any especially long ones? > > On 1 Oct 2015 09:37, "Walsh, Stephen" <stephen.wa...@aspect.com> wrote: > > There is no load balancer in front of Cassandra, it’s in front of our > application. > > Everyone seems hung up on this point? But it’s not the root causing of the > inconsistency issue. > > > > Can anyone verify the best practice for number of CF’s? > > > > > > *From:* Robert Coli [mailto:rc...@eventbrite.com] > *Sent:* 30 September 2015 18:45 > *To:* user@cassandra.apache.org > *Subject:* Re: Consistency Issues > > > > On Wed, Sep 30, 2015 at 9:06 AM, Walsh, Stephen <stephen.wa...@aspect.com> > wrote: > > > > We never had these issue with our first run. Its only when we added > another 25% of writes. > > > > As Jack said, you are probably pushing your GC over a threshold, leading > to long pause times and inability to meet quorum. > > > > As Sebastian said, you probably shouldn't need a load balancer in front of > Cassandra. > > > > =Rob > > > > This email (including any attachments) is proprietary to Aspect Software, > Inc. and may contain information that is confidential. If you have received > this message in error, please do not read, copy or forward this message. > Please notify the sender immediately, delete it from your system and > destroy any copies. You may not further disclose or distribute this email > or its attachments. > > This email (including any attachments) is proprietary to Aspect Software, > Inc. and may contain information that is confidential. If you have received > this message in error, please do not read, copy or forward this message. > Please notify the sender immediately, delete it from your system and > destroy any copies. You may not further disclose or distribute this email > or its attachments. > > > > This email (including any attachments) is proprietary to Aspect Software, > Inc. and may contain information that is confidential. If you have received > this message in error, please do not read, copy or forward this message. > Please notify the sender immediately, delete it from your system and > destroy any copies. You may not further disclose or distribute this email > or its attachments. > > > This email (including any attachments) is proprietary to Aspect Software, > Inc. and may contain information that is confidential. If you have received > this message in error, please do not read, copy or forward this message. > Please notify the sender immediately, delete it from your system and > destroy any copies. You may not further disclose or distribute this email > or its attachments. >
