Hi Bo, You might try sending the same question to the java driver mailing list. I haven't seen your particular error in several years of running Cassandra on AWS. The closest I saw in the past was due to a protocol error in the driver during the 2.0 beta timeframe.
Steve On Wed, Mar 16, 2016 at 1:30 PM, Bo Finnerup Madsen <bo.gunder...@gmail.com> wrote: > Hi, > > I ran another test with the following client setup: > final Cluster cluster = Cluster.builder() > .addContactPoints(key.getContactPoints()) > .withSocketOptions(new SocketOptions().setKeepAlive(true)) > .withReconnectionPolicy(Policies.defaultReconnectionPolicy()) > .withRetryPolicy(Policies.defaultRetryPolicy()) > .build(); > > But unfortunately the same issue happened. > I have uploaded the cassandra driver debug log here: > https://gist.github.com/anonymous/db6fa061298018c46954 > > It all goes south at about line 433 (time 19:46:43) where the client > throws a: > io.netty.handler.codec.DecoderException: > com.datastax.driver.core.exceptions.DriverInternalError: Adjusted frame > length exceeds 268435456: 462591744 - discarded > > Any ideas? > > ons. 16. mar. 2016 kl. 19.23 skrev Steve Robenalt <sroben...@highwire.org > >: > >> Hi Bo, >> >> I would suggest adding: >> >> .withReconnectionPolicy(new ExponentialReconnectionPolicy(1000,30000)) >> >> or something similar to your cluster builder. >> >> Steve >> >> >> On Wed, Mar 16, 2016 at 11:18 AM, Bo Finnerup Madsen < >> bo.gunder...@gmail.com> wrote: >> >>> Hi Sean, >>> >>> Thank you for taking the time to answer :) >>> We are using a very vanilla connection, without any sorts of tuning >>> policies. The cluster/session is constructed as follows: >>> final Cluster cluster = Cluster.builder() >>> >>> .addContactPoints(key.getContactPoints()) >>> .build(); >>> final Session session = >>> cluster.connect(key.getKeyspace()); >>> Perhaps it is too vanilla and we are missing something? >>> >>> Since I posted the question, I have tried downgrading to cassandra >>> v2.1.13 and java driver 2.1.9. But got the same error. So I suspect it is >>> something we are doing wrong. >>> >>> >>> >>> ons. 16. mar. 2016 kl. 18.59 skrev <sean_r_dur...@homedepot.com>: >>> >>>> Are you using any of the Tuning Policies ( >>>> https://docs.datastax.com/en/developer/java-driver/2.0/common/drivers/reference/tuningPolicies_c.html)? >>>> It could be that you are hitting some peak load and the driver is not >>>> retrying hosts once they are marked “down.” >>>> >>>> >>>> >>>> >>>> >>>> Sean Durity – Lead Cassandra Admin >>>> >>>> Big DATA Team >>>> >>>> For support, create a JIRA >>>> <https://portal.homedepot.com/sites/bigdata/Shared%20Documents/Jira%20Hadoop%20Support%20Workflow.pdf> >>>> >>>> >>>> >>>> *From:* Bo Finnerup Madsen [mailto:bo.gunder...@gmail.com] >>>> *Sent:* Tuesday, March 15, 2016 5:24 AM >>>> *To:* user@cassandra.apache.org >>>> *Subject:* Experiencing strange disconnect issue >>>> >>>> >>>> >>>> Hi, >>>> >>>> >>>> >>>> We are currently trying to convert an existing java web application to >>>> use cassandra, and while most of it works great :) we have a "small" issue. >>>> >>>> >>>> >>>> After some time, we all connectivity seems to be lost and we get the >>>> following errors: >>>> >>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All >>>> host(s) tried for query failed (tried: /10.61.70.107:9042 >>>> (com.datastax.driver.core.exceptions.TransportException: [/10.61.70.107] >>>> Connection has been closed), /10.61.70.108:9042 >>>> (com.datastax.driver.core.exceptions.TransportException: [/10.61.70.108] >>>> Connection has been closed)) >>>> >>>> >>>> >>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All >>>> host(s) tried for query failed (tried: /10.61.70.107:9042 >>>> (com.datastax.driver.core.exceptions.DriverException: Timeout while trying >>>> to acquire available connection (you may want to increase the driver number >>>> of per-host connections)), /10.61.70.108:9042 >>>> (com.datastax.driver.core.exceptions.TransportException: [/10.61.70.108] >>>> Connection has been closed), /10.61.70.110:9042 >>>> (com.datastax.driver.core.exceptions.TransportException: [/10.61.70.110] >>>> Connection has been closed)) >>>> >>>> >>>> >>>> The errors persists, and the application needs to be restarted to >>>> recover. >>>> >>>> >>>> >>>> At application startup we create a cluster and a session which we reuse >>>> through out the application as pr. the documentation. We don't specify any >>>> other options when connecting than the IP's of the three servers. We are >>>> running cassandra 3.0.3 tar ball in EC2 in a cluster of three machines. The >>>> connections are made using v3.0.0 java driver. >>>> >>>> >>>> >>>> I have uploaded the configuration and logs from our cassandra cluster >>>> here: https://gist.github.com/anonymous/452e736b401317b5b38d >>>> >>>> The issue happend at 00:44:46. >>>> >>>> >>>> >>>> I would greatly appreciate any ideas as to what we are doing wrong to >>>> experience this? :) >>>> >>>> >>>> >>>> Thank you in advance! >>>> >>>> >>>> >>>> Yours sincerely, >>>> >>>> Bo Madsen >>>> >>>> ------------------------------ >>>> >>>> The information in this Internet Email is confidential and may be >>>> legally privileged. It is intended solely for the addressee. Access to this >>>> Email by anyone else is unauthorized. If you are not the intended >>>> recipient, any disclosure, copying, distribution or any action taken or >>>> omitted to be taken in reliance on it, is prohibited and may be unlawful. >>>> When addressed to our clients any opinions or advice contained in this >>>> Email are subject to the terms and conditions expressed in any applicable >>>> governing The Home Depot terms of business or client engagement letter. The >>>> Home Depot disclaims all responsibility and liability for the accuracy and >>>> content of this attachment and for any damages or losses arising from any >>>> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other >>>> items of a destructive nature, which may be contained in this attachment >>>> and shall not be liable for direct, indirect, consequential or special >>>> damages in connection with this e-mail message or its attachment. >>>> >>> >> >> >> -- >> Steve Robenalt >> Software Architect >> sroben...@highwire.org <bza...@highwire.org> >> (office/cell): 916-505-1785 >> >> HighWire Press, Inc. >> 425 Broadway St, Redwood City, CA 94063 >> www.highwire.org >> >> Technology for Scholarly Communication >> > -- Steve Robenalt Software Architect sroben...@highwire.org <bza...@highwire.org> (office/cell): 916-505-1785 HighWire Press, Inc. 425 Broadway St, Redwood City, CA 94063 www.highwire.org Technology for Scholarly Communication
