Did anyone run into similar issues? On Thu, Sep 6, 2018 at 10:27 AM Jai Bheemsen Rao Dhanwada < jaibheem...@gmail.com> wrote:
> Here is the stacktrace from the failure, it looks like it's trying to > gather all the columfamily metrics and going OOM. Is this just for the JMX > metrics? > > > https://github.com/apache/cassandra/blob/cassandra-2.1.16/src/java/org/apache/cassandra/metrics/ColumnFamilyMetrics.java > > ERROR [MessagingService-Incoming-/10.133.33.57] 2018-09-06 15:43:19,280 > CassandraDaemon.java:231 - Exception in thread > Thread[MessagingService-Incoming-/x.x.x.x,5,main] > java.lang.OutOfMemoryError: Java heap space > at java.io.DataInputStream.<init>(DataInputStream.java:58) > ~[na:1.8.0_151] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:139) > ~[apache-cassandra-2.1.16.jar:2.1.16] > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:88) > ~[apache-cassandra-2.1.16.jar:2.1.16] > ERROR [InternalResponseStage:1] 2018-09-06 15:43:19,281 > CassandraDaemon.java:231 - Exception in thread > Thread[InternalResponseStage:1,5,main] > java.lang.OutOfMemoryError: Java heap space > at > org.apache.cassandra.metrics.ColumnFamilyMetrics$AllColumnFamilyMetricNameFactory.createMetricName( > *ColumnFamilyMetrics.java:784*) ~[apache-cassandra-2.1.16.jar:2.1.16] > at > org.apache.cassandra.metrics.ColumnFamilyMetrics.createColumnFamilyHistogram(ColumnFamilyMetrics.java:716) > ~[apache-cassandra-2.1.16.jar:2.1.16] > at > org.apache.cassandra.metrics.ColumnFamilyMetrics.<init>(ColumnFamilyMetrics.java:597) > ~[apache-cassandra-2.1.16.jar:2.1.16] > at > org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:361) > ~[apache-cassandra-2.1.16.jar:2.1.16] > at > org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:527) > ~[apache-cassandra-2.1.16.jar:2.1.16] > at > org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:498) > ~[apache-cassandra-2.1.16.jar:2.1.16] > at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:335) > ~[apache-cassandra-2.1.16.jar:2.1.16] > at > org.apache.cassandra.db.DefsTables.addColumnFamily(DefsTables.java:385) > ~[apache-cassandra-2.1.16.jar:2.1.16] > at > org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:293) > ~[apache-cassandra-2.1.16.jar:2.1.16] > at > org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.java:194) > ~[apache-cassandra-2.1.16.jar:2.1.16] > at > org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:166) > ~[apache-cassandra-2.1.16.jar:2.1.16] > at > org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:75) > ~[apache-cassandra-2.1.16.jar:2.1.16] > at > org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:54) > ~[apache-cassandra-2.1.16.jar:2.1.16] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) > ~[apache-cassandra-2.1.16.jar:2.1.16] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > ~[na:1.8.0_151] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > ~[na:1.8.0_151] > at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151] > > On Thu, Aug 30, 2018 at 12:51 PM Jai Bheemsen Rao Dhanwada < > jaibheem...@gmail.com> wrote: > >> thank you >> >> On Thu, Aug 30, 2018 at 11:58 AM Jeff Jirsa <jji...@gmail.com> wrote: >> >>> This is the closest JIRA that comes to mind (from memory, I didn't >>> search, there may be others): >>> https://issues.apache.org/jira/browse/CASSANDRA-8150 >>> >>> The best blog that's all in one place on tuning GC in cassandra is >>> actually Amy's 2.1 tuning guide: >>> https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html - >>> it's somewhat out of date as it's for 2.1, but since that's what you're >>> running, that works out in your favor. >>> >>> >>> >>> >>> >>> On Thu, Aug 30, 2018 at 10:53 AM Jai Bheemsen Rao Dhanwada < >>> jaibheem...@gmail.com> wrote: >>> >>>> Hi Jeff, >>>> >>>> Is there any JIRA that talks about increasing the HEAP will help? >>>> Also, any other alternatives than increasing the HEAP Size? last time >>>> when I tried increasing the heap, longer GC Pauses caused more damage in >>>> terms of latencies while gc pause. >>>> >>>> On Wed, Aug 29, 2018 at 11:07 PM Jai Bheemsen Rao Dhanwada < >>>> jaibheem...@gmail.com> wrote: >>>> >>>>> okay, thank you >>>>> >>>>> On Wed, Aug 29, 2018 at 11:04 PM Jeff Jirsa <jji...@gmail.com> wrote: >>>>> >>>>>> You’re seeing an OOM, not a socket error / timeout. >>>>>> >>>>>> -- >>>>>> Jeff Jirsa >>>>>> >>>>>> >>>>>> On Aug 29, 2018, at 10:56 PM, Jai Bheemsen Rao Dhanwada < >>>>>> jaibheem...@gmail.com> wrote: >>>>>> >>>>>> Jeff, >>>>>> >>>>>> any idea if this is somehow related to : >>>>>> https://issues.apache.org/jira/browse/CASSANDRA-11840? >>>>>> does increasing the value of streaming_socket_timeout_in_ms to a >>>>>> higher value helps? >>>>>> >>>>>> On Wed, Aug 29, 2018 at 10:52 PM Jai Bheemsen Rao Dhanwada < >>>>>> jaibheem...@gmail.com> wrote: >>>>>> >>>>>>> I have 72 nodes in the cluster, across 8 datacenters.. the moment I >>>>>>> try to increase the node above 84 or so, the issue starts. >>>>>>> >>>>>>> I am still using CMS Heap, assuming it will create more harm if I >>>>>>> increase the heap size beyond 8G(recommended). >>>>>>> >>>>>>> On Wed, Aug 29, 2018 at 6:53 PM Jeff Jirsa <jji...@gmail.com> wrote: >>>>>>> >>>>>>>> Given the size of your schema, you’re probably getting flooded with >>>>>>>> a bunch of huge schema mutations as it hops into gossip and tries to >>>>>>>> pull >>>>>>>> the schema from every host it sees. You say 8 DCs but you don’t say how >>>>>>>> many nodes - I’m guessing it’s a lot? >>>>>>>> >>>>>>>> This is something that’s incrementally better in 3.0, but a real >>>>>>>> proper fix has been talked about a few times - >>>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-11748 and >>>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-13569 for example >>>>>>>> >>>>>>>> In the short term, you may be able to work around this by >>>>>>>> increasing your heap size. If that doesn’t work, there’s an ugly ugly >>>>>>>> hack >>>>>>>> that’ll work on 2.1: limiting the number of schema blobs you can get >>>>>>>> at a >>>>>>>> time - in this case, that means firewall off all but a few nodes in >>>>>>>> your >>>>>>>> cluster for 10-30 seconds, make sure it gets the schema (watch the >>>>>>>> logs or >>>>>>>> file system for the tables to be created), then remove the firewall so >>>>>>>> it >>>>>>>> can start the bootstrap process (it needs the schema to setup the >>>>>>>> streaming >>>>>>>> plan, and it needs all the hosts up in gossip to stream successfully, >>>>>>>> so >>>>>>>> this is an ugly hack to give you time to get the schema and then heal >>>>>>>> the >>>>>>>> cluster so it can bootstrap). >>>>>>>> >>>>>>>> Yea that’s awful. Hopefully either of the two above JIRAs lands to >>>>>>>> make this less awful. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Jeff Jirsa >>>>>>>> >>>>>>>> >>>>>>>> On Aug 29, 2018, at 6:29 PM, Jai Bheemsen Rao Dhanwada < >>>>>>>> jaibheem...@gmail.com> wrote: >>>>>>>> >>>>>>>> It fails before bootstrap >>>>>>>> >>>>>>>> streaming throughpu on the nodes is set to 400Mb/ps >>>>>>>> >>>>>>>> On Wednesday, August 29, 2018, Jeff Jirsa <jji...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Is the bootstrap plan succeeding (does streaming start or does it >>>>>>>>> crash before it logs messages about streaming starting)? >>>>>>>>> >>>>>>>>> Have you capped the stream throughput on the existing hosts? >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Jeff Jirsa >>>>>>>>> >>>>>>>>> >>>>>>>>> On Aug 29, 2018, at 5:02 PM, Jai Bheemsen Rao Dhanwada < >>>>>>>>> jaibheem...@gmail.com> wrote: >>>>>>>>> >>>>>>>>> Hello All, >>>>>>>>> >>>>>>>>> We are seeing some issue when we add more nodes to the cluster, >>>>>>>>> where new node bootstrap is not able to stream the entire metadata and >>>>>>>>> fails to bootstrap. Finally the process dies with OOM >>>>>>>>> (java.lang.OutOfMemoryError: >>>>>>>>> Java heap space) >>>>>>>>> >>>>>>>>> But if I remove few nodes from the cluster we don't see this issue. >>>>>>>>> >>>>>>>>> Cassandra Version: 2.1.16 >>>>>>>>> # of KS and CF : 100, 3000 (approx) >>>>>>>>> # of DC: 8 >>>>>>>>> # of Vnodes per node: 256 >>>>>>>>> >>>>>>>>> Not sure what is causing this behavior, has any one come across >>>>>>>>> this scenario? >>>>>>>>> thanks in advance. >>>>>>>>> >>>>>>>>>
