I've found the problem, in cqlsh file, find all Cluster 's construct method
like ' conn = Cluster(xxx,xxx). At the end, add
parameter control_connection_timeout=float(_SOME_MS_VALUE_). As below:
conn = Cluster(contact_points=(self.hostname,), port=self.port,
cql_version=self.conn.cql_version,
protocol_version=DEFAULT_PROTOCOL_VERSION,
auth_provider=auth_provider,
ssl_options=self.conn.ssl_options,
load_balancing_policy=WhiteListRoundRobinPolicy([self.hostname]),
control_connection_timeout=float(36000))
this will fix the problem, thanks all.
2016-09-20 17:23 GMT+08:00 George Sigletos <sigle...@textkernel.nl>:
> Apologies. The entire error stack:
>
> ERROR [SharedPool-Worker-5] 2016-09-20 11:23:20,039 ErrorMessage.java:251
> - Unexpected exception during request
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.ReadTimeoutException:
> Operation timed out - received only 2 responses.
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
> ~[guava-16.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
> ~[guava-16.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
> ~[guava-16.0.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
> ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at
> org.apache.cassandra.service.ClientState.authorize(ClientState.java:353)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at
> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at
> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at
> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at org.apache.cassandra.cql3.statements.SelectStatement.
> checkAccess(SelectStatement.java:195) ~[apache-cassandra-2.1.14.jar:
> 2.1.14]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439)
> [apache-cassandra-2.1.14.jar:2.1.14]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335)
> [apache-cassandra-2.1.14.jar:2.1.14]
> at io.netty.channel.SimpleChannelInboundHandler.channelRead(
> SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.
> 0.23.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(
> AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.
> 0.23.Final]
> at io.netty.channel.AbstractChannelHandlerContext.access$700(
> AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.
> 0.23.Final]
> at io.netty.channel.AbstractChannelHandlerContext$8.run(
> AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.
> 0.23.Final]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source) [na:1.7.0_79]
> at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorSe
> rvice$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> [apache-cassandra-2.1.14.jar:2.1.14]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-2.1.14.jar:2.1.14]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_79]
> Caused by: java.lang.RuntimeException:
> org.apache.cassandra.exceptions.ReadTimeoutException:
> Operation timed out - received only 2 responses.
> at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at
> org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at
> org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at
> org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at
> org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at com.google.common.cache.LocalCache$LoadingValueReference.
> loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na]
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
> ~[guava-16.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.
> lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
> ~[guava-16.0.jar:na]
> ... 23 common frames omitted
>
> On Tue, Sep 20, 2016 at 11:22 AM, George Sigletos <sigle...@textkernel.nl>
> wrote:
>
>> This appears in the system log:
>>
>> Caused by: java.lang.RuntimeException:
>> org.apache.cassandra.exceptions.ReadTimeoutException:
>> Operation timed out - received only 2 responses.
>> at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at
>> org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at
>> org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at
>> org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at
>> org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at
>> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
>> ~[guava-16.0.jar:na]
>> at
>> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
>> ~[guava-16.0.jar:na]
>> at
>> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
>> ~[guava-16.0.jar:na]
>> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
>> ~[guava-16.0.jar:na]
>>
>>
>> On Tue, Sep 20, 2016 at 11:12 AM, George Sigletos <sigle...@textkernel.nl
>> > wrote:
>>
>>> I am also getting the same error:
>>> cqlsh <my host> -u cassandra -p cassandra
>>>
>>> Connection error: ('Unable to connect to any servers', {'<my host>':
>>> OperationTimedOut('errors=Timed out creating connection (5 seconds),
>>> last_host=None',)})
>>>
>>> But it is not consistent. Sometimes I manage to connect. It is random.
>>> Using 2.1.14
>>>
>>> On Tue, Jun 14, 2016 at 4:29 AM, joseph gao <gaojf.bok...@gmail.com>
>>> wrote:
>>>
>>>> hi, Patric, [image: 内嵌图片 1],
>>>> netstat -lepunt looks like above
>>>>
>>>> 2016-05-27 23:16 GMT+08:00 Patrick McFadin <pmcfa...@gmail.com>:
>>>>
>>>>> Can you do a netstat -lepunt and show the output? If Cassandra is
>>>>> running you aren't trying to connect to the ip/port it's bound to.
>>>>>
>>>>> Patrick
>>>>>
>>>>>
>>>>> On Monday, May 23, 2016, joseph gao <gaojf.bok...@gmail.com> wrote:
>>>>>
>>>>>> I used to think it's firewall/network issues too. So I make ufw to be
>>>>>> inactive. I really don't what's the reason.
>>>>>>
>>>>>> 2016-05-09 19:01 GMT+08:00 kurt Greaves <k...@instaclustr.com>:
>>>>>>
>>>>>>> Don't be fooled, despite saying tcp6 and :::*, it still listens on
>>>>>>> IPv4. As far as I'm aware this happens on all 2.1 Cassandra nodes, and
>>>>>>> may
>>>>>>> just be an oddity of netstat. It would be unrelated to your connection
>>>>>>> timeout issues, that's most likely related to firewall/network issues.
>>>>>>>
>>>>>>> On 9 May 2016 at 09:59, joseph gao <gaojf.bok...@gmail.com> wrote:
>>>>>>>
>>>>>>>> It doesn't work ,still using ipv6 [image: 内嵌图片 1]
>>>>>>>>
>>>>>>>> And I already set [image: 内嵌图片 2]
>>>>>>>>
>>>>>>>> Now I'm using 4.1.1 using 9160 port instead of 5.x.x。
>>>>>>>>
>>>>>>>> Hopefully this could be resolved, Thanks!
>>>>>>>>
>>>>>>>> 2016-03-30 22:13 GMT+08:00 Alain RODRIGUEZ <arodr...@gmail.com>:
>>>>>>>>
>>>>>>>>> Hi Joseph,
>>>>>>>>>
>>>>>>>>> why cassandra using tcp6 for 9042 port like :
>>>>>>>>>> tcp6 0 0 0.0.0.0:9042 :::*
>>>>>>>>>> LISTEN
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> if I remember correctly, in 2.1 and higher, cqlsh uses native
>>>>>>>>> transport, port 9042 (instead of thrift port 9160) and your clients
>>>>>>>>> (if
>>>>>>>>> any) are also probably using native transport (port 9042). So yes,
>>>>>>>>> this
>>>>>>>>> could be an issue indeed.
>>>>>>>>>
>>>>>>>>> You should have something like:
>>>>>>>>>
>>>>>>>>> tcp 0 0 1.2.3.4:9042 :::*
>>>>>>>>> LISTEN
>>>>>>>>>
>>>>>>>>> You are using IPv6 and no rpc address. Try setting it to the
>>>>>>>>> listen address and using IPv4.
>>>>>>>>>
>>>>>>>>> C*heers,
>>>>>>>>>
>>>>>>>>> -----------------------
>>>>>>>>>
>>>>>>>>> Alain Rodriguez - al...@thelastpickle.com
>>>>>>>>>
>>>>>>>>> France
>>>>>>>>>
>>>>>>>>> The Last Pickle - Apache Cassandra Consulting
>>>>>>>>>
>>>>>>>>> http://www.thelastpickle.com
>>>>>>>>>
>>>>>>>>> 2016-03-30 6:09 GMT+02:00 joseph gao <gaojf.bok...@gmail.com>:
>>>>>>>>>
>>>>>>>>>> why cassandra using tcp6 for 9042 port like :
>>>>>>>>>> tcp6 0 0 0.0.0.0:9042 :::*
>>>>>>>>>> LISTEN
>>>>>>>>>> would this be the problem
>>>>>>>>>>
>>>>>>>>>> 2016-03-30 11:34 GMT+08:00 joseph gao <gaojf.bok...@gmail.com>:
>>>>>>>>>>
>>>>>>>>>>> still have not fixed it . cqlsh: error: no such option:
>>>>>>>>>>> --connect-timeout
>>>>>>>>>>> cqlsh version 5.0.1
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2016-03-25 16:46 GMT+08:00 Alain RODRIGUEZ <arodr...@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Joseph.
>>>>>>>>>>>>
>>>>>>>>>>>> As I can't reproduce here, I believe you are having network
>>>>>>>>>>>> issue of some kind.
>>>>>>>>>>>>
>>>>>>>>>>>> MacBook-Pro:~ alain$ cqlsh --version
>>>>>>>>>>>> cqlsh 5.0.1
>>>>>>>>>>>> MacBook-Pro:~ alain$ echo 'DESCRIBE KEYSPACES;' | cqlsh
>>>>>>>>>>>> --connect-timeout=5 --request-timeout=10
>>>>>>>>>>>> system_traces system
>>>>>>>>>>>> MacBook-Pro:~ alain$
>>>>>>>>>>>>
>>>>>>>>>>>> It's been a few days, did you manage to fix it ?
>>>>>>>>>>>>
>>>>>>>>>>>> C*heers,
>>>>>>>>>>>> -----------------------
>>>>>>>>>>>> Alain Rodriguez - al...@thelastpickle.com
>>>>>>>>>>>> France
>>>>>>>>>>>>
>>>>>>>>>>>> The Last Pickle - Apache Cassandra Consulting
>>>>>>>>>>>> http://www.thelastpickle.com
>>>>>>>>>>>>
>>>>>>>>>>>> 2016-03-21 9:59 GMT+01:00 joseph gao <gaojf.bok...@gmail.com>:
>>>>>>>>>>>>
>>>>>>>>>>>>> cqlsh version 5.0.1. nodetool tpstats looks good, log looks
>>>>>>>>>>>>> good. And I used specified port 9042. And it immediately returns
>>>>>>>>>>>>> fail (less
>>>>>>>>>>>>> than 3 seconds). By the way where should I use
>>>>>>>>>>>>> '--connect-timeout', cqlsh
>>>>>>>>>>>>> seems don't have such parameters.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2016-03-18 17:29 GMT+08:00 Alain RODRIGUEZ <arodr...@gmail.com
>>>>>>>>>>>>> >:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is the node fully healthy or rejecting some requests ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What are the outputs for "grep -i "ERROR"
>>>>>>>>>>>>>> /var/log/cassandra/system.log" and "nodetool tpstats"?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Any error? Any pending / blocked or dropped messages?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also did you try using distinct ports (9160 for thrift, 9042
>>>>>>>>>>>>>> for native) - out of curiosity, not sure this will help.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What is your version of cqlsh "cqlsh --version" ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> doesn't work most times. But some time it just work fine
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Do you fill like this is due to a timeout (query being too
>>>>>>>>>>>>>> big, cluster being to busy)? Try setting this higher:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --connect-timeout=CONNECT_TIMEOUT
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Specify the connection timeout in
>>>>>>>>>>>>>> seconds (default: 5 seconds).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --request-timeout=REQUEST_TIMEOUT
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Specify the default request timeout
>>>>>>>>>>>>>> in seconds (default: 10 seconds).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> C*heers,
>>>>>>>>>>>>>> -----------------------
>>>>>>>>>>>>>> Alain Rodriguez - al...@thelastpickle.com
>>>>>>>>>>>>>> France
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The Last Pickle - Apache Cassandra Consulting
>>>>>>>>>>>>>> http://www.thelastpickle.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2016-03-18 4:49 GMT+01:00 joseph gao <gaojf.bok...@gmail.com>
>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Of course yes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2016-03-17 22:35 GMT+08:00 Vishwas Gupta <
>>>>>>>>>>>>>>> vishwas.gu...@snapdeal.com>:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Have you started the Cassandra service?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> sh cassandra
>>>>>>>>>>>>>>>> On 17-Mar-2016 7:59 pm, "Alain RODRIGUEZ" <
>>>>>>>>>>>>>>>> arodr...@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi, did you try with the address of the node rather than
>>>>>>>>>>>>>>>>> 127.0.0.1
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Is the transport protocol used by cqlsh (not sure if it is
>>>>>>>>>>>>>>>>> thrift or binary - native in 2.1) active ? What is the
>>>>>>>>>>>>>>>>> "nodetool info"
>>>>>>>>>>>>>>>>> output ?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> C*heers,
>>>>>>>>>>>>>>>>> -----------------------
>>>>>>>>>>>>>>>>> Alain Rodriguez - al...@thelastpickle.com
>>>>>>>>>>>>>>>>> France
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The Last Pickle - Apache Cassandra Consulting
>>>>>>>>>>>>>>>>> http://www.thelastpickle.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2016-03-17 14:26 GMT+01:00 joseph gao <
>>>>>>>>>>>>>>>>> gaojf.bok...@gmail.com>:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> hi, all
>>>>>>>>>>>>>>>>>> cassandra version 2.1.7
>>>>>>>>>>>>>>>>>> When I use cqlsh to connect cassandra, something is wrong
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Connection error: ( Unable to connect to any servers',
>>>>>>>>>>>>>>>>>> {'127.0.0.1': OperationTimedOut('errors=None,
>>>>>>>>>>>>>>>>>> last_host=None,)})
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This happens lots of times, but sometime it works just
>>>>>>>>>>>>>>>>>> fine. Anybody knows why?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> ------
>>>>>>>>>>>>>>>>>> Joseph Gao
>>>>>>>>>>>>>>>>>> PhoneNum:15210513582
>>>>>>>>>>>>>>>>>> QQ: 409343351
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> ------
>>>>>>>>>>>>>>> Joseph Gao
>>>>>>>>>>>>>>> PhoneNum:15210513582
>>>>>>>>>>>>>>> QQ: 409343351
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> ------
>>>>>>>>>>>>> Joseph Gao
>>>>>>>>>>>>> PhoneNum:15210513582
>>>>>>>>>>>>> QQ: 409343351
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> ------
>>>>>>>>>>> Joseph Gao
>>>>>>>>>>> PhoneNum:15210513582
>>>>>>>>>>> QQ: 409343351
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> ------
>>>>>>>>>> Joseph Gao
>>>>>>>>>> PhoneNum:15210513582
>>>>>>>>>> QQ: 409343351
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> ------
>>>>>>>> Joseph Gao
>>>>>>>> PhoneNum:15210513582
>>>>>>>> QQ: 409343351
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Kurt Greaves
>>>>>>> k...@instaclustr.com
>>>>>>> www.instaclustr.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> ------
>>>>>> Joseph Gao
>>>>>> PhoneNum:15210513582
>>>>>> QQ: 409343351
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> ------
>>>> Joseph Gao
>>>> PhoneNum:15210513582
>>>> QQ: 409343351
>>>>
>>>
>>>
>>
>
--
------
Joseph Gao
PhoneNum:15210513582
QQ: 409343351
