Batch Queries Timeout When Private IP of a Node Fails in Multi-DC Cassandra 4.1.4

2025-02-04 Thread manish khandelwal
, causing the timeout. - The node is not marked down in nodetool status but is unreachable in nodetool describecluster till node is not brought down. After restarting the node, nodetool status also shows the problematic node as Down. Is this expected behaviour that nodes in the datacenter

Re: COUNTER timeout

2021-09-15 Thread Joe Obernberger
Thank you! Clocks were out of sync; chronyd wasn't chrony'ding. Going so much faster now!  Cheers. -Joe On 9/15/2021 4:07 PM, Bowen Song wrote: Well, the log says cross node timeout, latency a bit over 44 seconds. Here's a few most likely causes: 1. The clocks are not i

Re: COUNTER timeout

2021-09-15 Thread Bowen Song
Well, the log says cross node timeout, latency a bit over 44 seconds. Here's a few most likely causes: 1. The clocks are not in sync - please check the time on each server, and ensure NTP client is running on all Cassandra servers 2. Long stop the world GC pauses - please check the GC

Re: COUNTER timeout

2021-09-15 Thread Joe Obernberger
Thank you Erick - looking through all the logs on the nodes I found this: INFO  [CompactionExecutor:17551] 2021-09-15 15:13:20,524 CompactionTask.java:245 - Compacted (fb0cdca0-1658-11ec-9098-dd70c3a3487a) 4 sstables to [/data/7/cassandra/data/doc/fieldcounts-03b67080ada111ebade9fdc1d34336d3/n

Re: COUNTER timeout

2021-09-15 Thread Bowen Song
data: com.datastax.oss.driver.api.core.servererrors.WriteTimeoutException: Cassandra timeout during COUNTER write query at consistency ONE (1 replica were required but only 0 acknowledged the write)     at com.datastax.oss.driver.api.core.servererrors.WriteTimeoutException.copy(WriteTimeoutException.java:96)  

Re: COUNTER timeout

2021-09-14 Thread Erick Ramirez
The obvious conclusion is to say that the nodes can't keep up so it would be interesting to know how often you're issuing the counter updates. Also, how are the commit log disks performing on the nodes? If you have monitoring in place, check the IO stats/metrics. And finally, review the logs on the

COUNTER timeout

2021-09-14 Thread Joe Obernberger
I'm getting a lot of the following errors during ingest of data: com.datastax.oss.driver.api.core.servererrors.WriteTimeoutException: Cassandra timeout during COUNTER write query at consistency ONE (1 replica were required but only 0 acknowledged the write)  

Operation timeout error from Apache Cassandra 3.11.6

2020-12-23 Thread MyWorld
Hi, We are currently using apache cassandra 3.11.6. We have a cluster of 4 nodes all in single DC. All of a sudden we started experiencing high connection timeout on the cluster. This timeout spikes are coming intermittently. As checked from grafana, read/write time, jvm threads count, client

Re: Cassandra timeout during read query

2020-10-27 Thread Attila Wind
gether the records (page) - thus helps you to avoid the timeout issue. Based on our measurements smaller page sizes does not add too much to the overall query time at all - but helps Cassandra a lot to eventually fulfill the full request as she can do much better load balancing too as you are

Re: Cassandra timeout during read query

2020-10-27 Thread Deepak Sharma
- in order being able to help (better) you need to share >> those details >> >> That 5 secs timeout comes from the coordinator node I think - see >> cassandra.yaml "read_request_timeout_in_ms" setting - that is influencing >> this >> >> But it does not mat

Re: Cassandra timeout during read query

2020-09-19 Thread Deepak Sharma
e > those details > > That 5 secs timeout comes from the coordinator node I think - see > cassandra.yaml "read_request_timeout_in_ms" setting - that is influencing > this > > But it does not matter too much... The point is that none of the replicas > could completed y

Re: Cassandra timeout during read query

2020-09-15 Thread Attila Wind
Hi Deepak, Aaron has right - in order being able to help (better) you need to share those details That 5 secs timeout comes from the coordinator node I think - see cassandra.yaml "read_request_timeout_in_ms" setting - that is influencing this But it does not matter too much...

Re: Cassandra timeout during read query

2020-09-14 Thread Aaron Ploetz
ep 14, 2020 at 10:58 AM Deepak Sharma wrote: > Hi There, > > We are running into a strange issue in our Cassandra Cluster where one > specific query is failing with following error: > > Cassandra timeout during read query at consistency QUORUM (3 responses > were required bu

Cassandra timeout during read query

2020-09-14 Thread Deepak Sharma
Hi There, We are running into a strange issue in our Cassandra Cluster where one specific query is failing with following error: Cassandra timeout during read query at consistency QUORUM (3 responses were required but only 0 replica responded) This is not a typical query read timeout that we

Re: Cassandra read requests not getting timeout

2019-08-05 Thread Jon Haddad
I think this might be because the timeout only applied to each request, and the driver is paginating in the background. Each page is a new request. On Mon, Aug 5, 2019, 12:08 AM Oleksandr Shulgin < oleksandr.shul...@zalando.de> wrote: > On Mon, Aug 5, 2019 at 8:50 AM nokia ceph > wr

Re: Cassandra read requests not getting timeout

2019-08-05 Thread Oleksandr Shulgin
On Mon, Aug 5, 2019 at 8:50 AM nokia ceph wrote: > Hi Community, > > I am using Cassanadra 3.0.13 . 5 node cluster simple topology. Following > are the timeout parameters in yaml file: > > # grep timeout /etc/cassandra/conf/cassandra.yaml > cas_contentio

Cassandra read requests not getting timeout

2019-08-04 Thread nokia ceph
Hi Community, I am using Cassanadra 3.0.13 . 5 node cluster simple topology. Following are the timeout parameters in yaml file: # grep timeout /etc/cassandra/conf/cassandra.yaml cas_contention_timeout_in_ms: 1000 counter_write_request_timeout_in_ms: 5000 cross_node_timeout: false

Re: cqlsh --request-timeout=3600 doesn't seems to work

2018-08-24 Thread Vitaliy Semochkin
ded to do count in cassandra better if u can avoid it > > > On Fri, Aug 24, 2018, 4:06 PM Vitaliy Semochkin wrote: >> >> Hi, >> >> i'm running count query for a very small table (less than 1000 000 records). >> When the amount of records gets to 800 000 i r

Re: cqlsh --request-timeout=3600 doesn't seems to work

2018-08-24 Thread Pranay akula
than 1000 000 > records). > When the amount of records gets to 800 000 i receive read timeout > error in cqlsh. > I tried to run cqlsh with option --request-timeout=3600, but receive same > error, > what should I do in order not to recieve timeout e

cqlsh --request-timeout=3600 doesn't seems to work

2018-08-24 Thread Vitaliy Semochkin
Hi, i'm running count query for a very small table (less than 1000 000 records). When the amount of records gets to 800 000 i receive read timeout error in cqlsh. I tried to run cqlsh with option --request-timeout=3600, but receive same error, what should I do in order not to recieve ti

Re: Timeout for only one keyspace in cluster

2018-07-23 Thread learner dba
, 10:44:29 PM CDT, wxn...@zjqunshuo.com wrote: Your partition key is foreignid. You may have a large partition. Why not use foreignid+timebucket as partition key? From: learner dbaDate: 2018-07-19 01:48To: User cassandra.apache.orgSubject:  Timeout for only one keyspace

Re: Timeout for only one keyspace in cluster

2018-07-23 Thread Jonathan Haddad
42510 > any thing else that we can check? > > On Wednesday, July 18, 2018, 10:44:29 PM CDT, wxn...@zjqunshuo.com < > wxn...@zjqunshuo.com> wrote: > > > Your partition key is foreignid. You may have a large partition. Why not > use foreignid+timebucket as par

Re: Timeout for only one keyspace in cluster

2018-07-23 Thread learner dba
check? On Wednesday, July 18, 2018, 10:44:29 PM CDT, wxn...@zjqunshuo.com wrote: Your partition key is foreignid. You may have a large partition. Why not use foreignid+timebucket as partition key?  From: learner dbaDate: 2018-07-19 01:48To: User cassandra.apache.orgSubject:  Timeout for onl

Re: Timeout for only one keyspace in cluster

2018-07-23 Thread James Shaw
00 88.15 0.00 7007506 > 42510 > any thing else that we can check? > > On Wednesday, July 18, 2018, 10:44:29 PM CDT, wxn...@zjqunshuo.com < > wxn...@zjqunshuo.com> wrote: > > > Your partition key is foreignid. You may have a

Re: Timeout for only one keyspace in cluster

2018-07-23 Thread learner dba
, wxn...@zjqunshuo.com wrote: Your partition key is foreignid. You may have a large partition. Why not use foreignid+timebucket as partition key?  From: learner dbaDate: 2018-07-19 01:48To: User cassandra.apache.orgSubject:  Timeout for only one keyspace in clusterHi, We have a cluster with mul

Re: Timeout for only one keyspace in cluster

2018-07-23 Thread learner dba
 01:48To: User cassandra.apache.orgSubject:  Timeout for only one keyspace in clusterHi, We have a cluster with multiple keyspaces. All queries are performing good but write operation on few tables in one specific keyspace gets write timeout. Table has counter column and counter update query times

Re: Timeout for only one keyspace in cluster

2018-07-21 Thread Ben Slater
; Max 0.00 88.15 0.00 7007506 >> 42510 >> any thing else that we can check? >> >> On Wednesday, July 18, 2018, 10:44:29 PM CDT, wxn...@zjqunshuo.com < >> wxn...@zjqunshuo.com> wrote: >> >> >&

Re: Timeout for only one keyspace in cluster

2018-07-21 Thread Goutham reddy
42510 > any thing else that we can check? > > On Wednesday, July 18, 2018, 10:44:29 PM CDT, wxn...@zjqunshuo.com < > wxn...@zjqunshuo.com> wrote: > > > Your partition key is foreignid. You may have a large partition. Why not > use foreignid+timebucket as partition k

Re: Timeout for only one keyspace in cluster

2018-07-20 Thread learner dba
div.yiv4986088840foxdiv20180719114102755131 {}#yiv4986088840 body {font-size:10.5pt;color:rgb(0, 0, 0);line-height:1.5;}Your partition key is foreignid. You may have a large partition. Why not use foreignid+timebucket as partition key?  From: learner dbaDate: 2018-07-19 01:48To: User cassandra.apache.orgSubject:  Timeout for only

Re: Timeout for only one keyspace in cluster

2018-07-19 Thread learner dba
+timebucket as partition key?  From: learner dbaDate: 2018-07-19 01:48To: User cassandra.apache.orgSubject:  Timeout for only one keyspace in clusterHi, We have a cluster with multiple keyspaces. All queries are performing good but write operation on few tables in one specific keyspace gets write

Re: Timeout for only one keyspace in cluster

2018-07-18 Thread wxn...@zjqunshuo.com
Your partition key is foreignid. You may have a large partition. Why not use foreignid+timebucket as partition key? From: learner dba Date: 2018-07-19 01:48 To: User cassandra.apache.org Subject: Timeout for only one keyspace in cluster Hi, We have a cluster with multiple keyspaces. All

Timeout for only one keyspace in cluster

2018-07-18 Thread learner dba
Hi, We have a cluster with multiple keyspaces. All queries are performing good but write operation on few tables in one specific keyspace gets write timeout. Table has counter column and counter update query times out always. Any idea? CREATE TABLE x.y (     foreignid uuid,     timebucket

Re: Safe read request timeout value

2018-06-15 Thread anil patimidi
Hi, Depends on how much latency can your app tolerate. You always want to make sure that client side timeouts are set greater than server side timeout value. So that client is n't timing out while server is still serving the requests. You might want to start with defaults and try bumpi

Safe read request timeout value

2018-06-15 Thread Vsevolod Filaretov
What are general practices on read request timeout settings? How large is my theoretical maximum of server-side user read timeout which does not compromise cluster availability, prioritizing inserts stability?

Single Node Timeout Error and High Dropped Mutation after Upgradesstables

2018-04-11 Thread hitesh dua
ache-cassandra-3.10.jar:3.10] > > at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144] > > INFO [HintsDispatcher:767] 2018-04-09 19:05:24,874 >> HintsDispatchExecutor.java:283 - Finished hinted handoff of file >> 68c7c130-6cf8-4864-bde8-1819f238045c-1523315072851

Re: "READ messages were dropped ... for internal timeout" after big amount of writes

2018-03-19 Thread Dmitry Simonov
nd contains 3 tables (current table sizes: >> 1Gb, 700Mb, 12Kb). >> >> Several times per day there are bursts of "READ messages were dropped ... >> for internal timeout" messages in logs (on every cassandra node). Duration: >> 5 - 15 minutes. >> >> D

Re: "READ messages were dropped ... for internal timeout" after big amount of writes

2018-03-16 Thread Nicolas Guyomar
pace has RF=3 and contains 3 tables (current table sizes: > 1Gb, 700Mb, 12Kb). > > Several times per day there are bursts of "READ messages were dropped ... > for internal timeout" messages in logs (on every cassandra node). Duration: > 5 - 15 minutes. > > During p

"READ messages were dropped ... for internal timeout" after big amount of writes

2018-03-15 Thread Dmitry Simonov
Hello! We are experiencing problems with Cassandra 2.2.8. There is a cluster with 3 nodes. Problematic keyspace has RF=3 and contains 3 tables (current table sizes: 1Gb, 700Mb, 12Kb). Several times per day there are bursts of "READ messages were dropped ... for internal timeout" messag

Re: MUTATION messages were dropped in last 5000 ms for cross node timeout

2017-08-15 Thread Erick Ramirez
tpstats output though there are bunch of MUTATIONS > dropped observed. > > > > > > INFO [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 > - MUTATION messages were dropped in last 5000 ms: 822 for internal timeout > and 2152 for cross node time

Re: MUTATION messages were dropped in last 5000 ms for cross node timeout

2017-08-04 Thread Akhil Mehra
Glad I could be of help :) Hopefully the partition size resize goes smoothly. Regards, Akhil > On 4/08/2017, at 5:41 AM, ZAIDI, ASAD A wrote: > > Hi Akhil, > > Thank you for your reply. > > I kept testing different timeout numbers over last week and eventually

RE: MUTATION messages were dropped in last 5000 ms for cross node timeout

2017-08-03 Thread ZAIDI, ASAD A
Hi Akhil, Thank you for your reply. I kept testing different timeout numbers over last week and eventually settled at setting *_request_timeout_in_ms parameters at 1.5minutes for coordinator wait time. That is the number where I donot see any dropped mutations. Also asked developers to tweak

Re: Timeout while setting keyspace

2017-07-26 Thread Peng Xiao
https://datastax-oss.atlassian.net/browse/JAVA-1002 This one says it's the driver issue,we will have a try. -- Original -- From: "";<2535...@qq.com>; Date: Wed, Jul 26, 2017 04:12 PM To: "user"; Subject: Timeout

Timeout while setting keyspace

2017-07-26 Thread Peng Xiao
-nio-worker-2] com.datastax.driver.core.Connection : Timeout while setting keyspace on Connection[/172.16.42.138:9042-3, inFlight=5, closed=false]. This should not happen but is not critical (it will be retried) 2017-07-26 15:49:20.677 WARN 11706 --- [-] [cluster1-nio-worker-3

Re: MUTATION messages were dropped in last 5000 ms for cross node timeout

2017-07-21 Thread Akhil Mehra
se the write_request_timeout_in_ms. The write_request_timeout_in_ms (http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html <http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html>) can be used to increase the mutation timeout. B

RE: MUTATION messages were dropped in last 5000 ms for cross node timeout

2017-07-21 Thread Anuj Wadehra
not be optimal.     Thank you again.       From: Anuj Wadehra [mailto:anujw_2...@yahoo.co.in] Sent: Thursday, July 20, 2017 12:17 PM To: ZAIDI, ASAD A ; user@cassandra.apache.org Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node timeout   Hi Asad

RE: MUTATION messages were dropped in last 5000 ms for cross node timeout

2017-07-21 Thread ZAIDI, ASAD A
before, none of any timeout parameter settings matches that #! Load is intermittently high but again cpu queue length never goes beyond medium depth. I wonder if there is some internal limit that I’m still not aware of. Thanks/Asad From: Akhil Mehra [mailto:akhilme...@gmail.com] Sent: Thursday

RE: MUTATION messages were dropped in last 5000 ms for cross node timeout

2017-07-21 Thread ZAIDI, ASAD A
cross node timeout In a cloud environment, cross_node_timeout = true can cause issues; we had this issue in our environment and it is set to false now. Dropped messages is an another issue Subroto On Jul 20, 2017, at 8:27 AM, ZAIDI, ASAD A mailto:az1...@att.com>> wrote: Hello Folks – I’m

RE: MUTATION messages were dropped in last 5000 ms for cross node timeout

2017-07-21 Thread ZAIDI, ASAD A
optimal. Thank you again. From: Anuj Wadehra [mailto:anujw_2...@yahoo.co.in] Sent: Thursday, July 20, 2017 12:17 PM To: ZAIDI, ASAD A ; user@cassandra.apache.org Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node timeout Hi Asad, You can do following things: 1

Re: MUTATION messages were dropped in last 5000 ms for cross node timeout

2017-07-20 Thread Akhil Mehra
cting > clock drift on 16node cluster. I do not see pending or blocked HintedHandoff > in tpstats output though there are bunch of MUTATIONS dropped observed. > > > INFO [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - > MUTATION messages were dropp

Re: MUTATION messages were dropped in last 5000 ms for cross node timeout

2017-07-20 Thread Subroto Barua
h there are bunch of MUTATIONS dropped observed. > > > INFO [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - > MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and > 2152 for cross node timeout > > > I’m seeking help here i

Re: MUTATION messages were dropped in last 5000 ms for cross node timeout

2017-07-20 Thread Anuj Wadehra
internal timeout and 2152 for cross node timeout   I’m seeking help here if you please let me know what I need to check in order to address these cross node timeouts.   Thank you, Asad  

MUTATION messages were dropped in last 5000 ms for cross node timeout

2017-07-20 Thread ZAIDI, ASAD A
stats output though there are bunch of MUTATIONS dropped observed. INFO [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 2152 for cross node timeout I'm seeking help here if you please let me

Re: Timeout while trying to acquire available connection

2017-06-26 Thread Jacob Shadix
pps being not able to communicate to the > Cassandra cluster with the following errors: > > All host(s) tried for query failed (tried: cassandra2-test:9042 > (com.datastax.driver.core.exceptions.DriverException: Timeout while > trying to acquire available connection (you may wan

Timeout while trying to acquire available connection

2017-06-26 Thread Ivan Iliev
Hello everyone! I am seeing recent behavior of apps being not able to communicate to the Cassandra cluster with the following errors: All host(s) tried for query failed (tried: cassandra2-test:9042 (com.datastax.driver.core.exceptions.DriverException: Timeout while trying to acquire available

Re: sasi index question (read timeout on many selects)

2017-02-17 Thread Benjamin Roth
Btw: They break incremental repair if you use CDC: https://issues.apache. org/jira/browse/CASSANDRA-12888 Not only when using CDC! You shouldn't use incremental repairs with MVs. Never (right now). 2017-02-16 17:42 GMT+01:00 Jonathan Haddad : > My advice to avoid them is based on the issues th

Re: sasi index question (read timeout on many selects)

2017-02-16 Thread Jonathan Haddad
My advice to avoid them is based on the issues that have been filed in Jira. Benjamin Roth is one of the only people talking about his MV usage, and has filed a few JIRAs discussing their problems when bootstrapping new nodes, as well as issues repairing. https://issues.apache.org/jira/browse/CAS

Re: sasi index question (read timeout on many selects)

2017-02-16 Thread Micha
On 16.02.2017 16:33, Jonathan Haddad wrote: > > Regarding MVs, do not use the ones that shipped with 3.x. They're not > ready for production. Manage it yourself by using a second table and > inserting a second record there. > Out of interest... there is a slight discrepance between the advic

Re: sasi index question (read timeout on many selects)

2017-02-16 Thread Micha
On 16.02.2017 16:33, Jonathan Haddad wrote: > I agree w/ DuyHai regarding the index. The use case described here is a > terrible one for SASI indexes. > > Regarding MVs, do not use the ones that shipped with 3.x. They're not > ready for production. Manage it yourself by using a second table a

Re: sasi index question (read timeout on many selects)

2017-02-16 Thread Jonathan Haddad
I agree w/ DuyHai regarding the index. The use case described here is a terrible one for SASI indexes. Regarding MVs, do not use the ones that shipped with 3.x. They're not ready for production. Manage it yourself by using a second table and inserting a second record there. On Thu, Feb 16, 201

Re: sasi index question (read timeout on many selects)

2017-02-16 Thread DuyHai Doan
Using MV and put id as partition key is your best bet right now. SASI will be too expensive for this simple use case On Thu, Feb 16, 2017 at 3:21 PM, Micha wrote: > > > it's like having a table (sha256 blob primary key, id timeuuid, data1 > text, ., ) > > So both, sha256 and id are unique. >

Re: sasi index question (read timeout on many selects)

2017-02-16 Thread Micha
it's like having a table (sha256 blob primary key, id timeuuid, data1 text, ., ) So both, sha256 and id are unique. I would like to query *either* with sha256 *or* with id. I thought this can be done with a sasi index, but it has to be done with a second table (manual way) or with a mv with

Re: sasi index question (read timeout on many selects)

2017-02-16 Thread DuyHai Doan
[image: Inline image 1] On Thu, Feb 16, 2017 at 3:08 PM, Micha wrote: > > > On 16.02.2017 14:30, DuyHai Doan wrote: > > Why indexing BLOB data ? It does not make any sense > > My partition key is a secure hash sum, I don't index a blob. > > > > >

Re: sasi index question (read timeout on many selects)

2017-02-16 Thread Benjamin Roth
No matter what has to be indexed here, the preferrable way is most probably denormalization instead of another index. 2017-02-16 15:09 GMT+01:00 DuyHai Doan : > [image: Inline image 1] > > On Thu, Feb 16, 2017 at 3:08 PM, Micha wrote: > >> >> >> On 16.02.2017 14:30, DuyHai Doan wrote: >> > Why i

Re: sasi index question (read timeout on many selects)

2017-02-16 Thread Micha
On 16.02.2017 14:30, DuyHai Doan wrote: > Why indexing BLOB data ? It does not make any sense My partition key is a secure hash sum, I don't index a blob.

Re: sasi index question (read timeout on many selects)

2017-02-16 Thread DuyHai Doan
Why indexing BLOB data ? It does not make any sense "I thought sasi index is globally held, in contrast to the normal secondary index.." --> Who said that ? It's just wrong On Thu, Feb 16, 2017 at 1:50 PM, Micha wrote: > Hi, > > > my table has (among others) three columns, which are unique blob

sasi index question (read timeout on many selects)

2017-02-16 Thread Micha
Hi, my table has (among others) three columns, which are unique blobs. So I made the first column the partition key and created two sasi indices for the two other columns. After inserting ca 90m records I'm not able to query a bunch of rows (sending 1 selects to the cluster) using only a sas

Re: Cqlsh timeout and schema refresh exceptions

2016-12-19 Thread Saumitra S
eturn isSchemaInAgreement = true. I am trying to find out why CF creation is not able to finish schema agreement in even 30 seconds? I can increase timeout, or just do manual pooling with checkSchemaAgreement, but then CF creation time will be too high for my application usecase. Even 30 seconds is too high.

Re: Cqlsh timeout and schema refresh exceptions

2016-12-19 Thread Matija Gobec
sting* > > > > On Mon, 19 Dec 2016 14:10:37 -0500 *Saumitra S > >* wrote > > > Hi Vladimir, > > Thanks for the response. > > When I see *"**com.datastax.driver.core.ControlConnection"* exceptions, I > see that keyspaces and CF are created

Re: Cqlsh timeout and schema refresh exceptions

2016-12-19 Thread Vladimir Yudovin
Vladimir, Thanks for the response. When I see "com.datastax.driver.core.ControlConnection" exceptions, I see that keyspaces and CF are created. But when I create CF with large number of columns(2400 cols) quickly one after the other(with 2 seconds gap between CREATE TABLE queries), I get schema agreem

Re: Cqlsh timeout and schema refresh exceptions

2016-12-19 Thread Saumitra S
en I see *"**com.datastax.driver.core.ControlConnection"* exceptions, I > see that keyspaces and CF are created. But when I create CF with large > number of columns(2400 cols) quickly one after the other(with 2 seconds gap > between CREATE TABLE queries), I get schema agreement ti

Re: Cqlsh timeout and schema refresh exceptions

2016-12-19 Thread Vladimir Yudovin
create CF with large number of columns(2400 cols) quickly one after the other(with 2 seconds gap between CREATE TABLE queries), I get schema agreement timeout errors ( com.datastax.driver.core.Cluster | Error while waiting for schema agreement). This happens even with a clean slate(empty data dire

Re: Cqlsh timeout and schema refresh exceptions

2016-12-19 Thread Saumitra S
es), I get schema agreement timeout errors (* com.datastax.driver.core.Cluster | Error while waiting for schema agreement). *This happens even with a clean slate(empty data directory), just after creating 4 keyspaces. Timeout is set to 30 seconds. Please note that CREATE TABLE queries are NOT fired i

Re: Cqlsh timeout and schema refresh exceptions

2016-12-19 Thread Vladimir Yudovin
Hi, Question: Does C* reads some schema/metadata on calling cqlsh, which is causing timeout with large number of keyspaces? A lot ). cqlsh reads schemas, cluster topology, each node tokens, etc. You can just capture TCP port 9042 (unless you use SSL) and view all negotiation between cqlsh

Cqlsh timeout and schema refresh exceptions

2016-12-19 Thread Saumitra S
Hi All, I have a 2 node cluster(32gb ram/8cpu) running 3.0.10 and I created 50 keyspaces in it. Each keyspace has 25 CF. Column count in each CF ranges between 5 to 30. I am getting few issues once keyspace count reaches ~50. *Issue 1:* When I try to use cqlsh, I get timeout. *$ cqlsh

Re: Debugging high tail read latencies (internal timeout)

2016-07-07 Thread daemeon reiydelle
pools are nearly empty - >> >> https://gist.github.com/nemothekid/28b2a8e8353b3e60d7bbf390ed17987c >> >> Relevant line: >> REQUEST_RESPONSE messages were dropped in last 5000 ms: 1 for internal >> timeout and 0 for cross node timeout. Mean internal dropped latency: 54930 >> ms and Mean cross-node dropped latency: 0 ms >> >> Are there any tools I can use to start to understand what is causing >> these issues? >> >> Nimi >> >> >

Re: Debugging high tail read latencies (internal timeout)

2016-07-07 Thread Bryan Cheng
ing > wrong. I don't think there are GC issues, but the logs mention dropped > messages due to timeouts while the threadpools are nearly empty - > > https://gist.github.com/nemothekid/28b2a8e8353b3e60d7bbf390ed17987c > > Relevant line: > REQUEST_RESPONSE messages were dro

Debugging high tail read latencies (internal timeout)

2016-07-06 Thread Nimi Wariboko Jr
t.github.com/nemothekid/28b2a8e8353b3e60d7bbf390ed17987c Relevant line: REQUEST_RESPONSE messages were dropped in last 5000 ms: 1 for internal timeout and 0 for cross node timeout. Mean internal dropped latency: 54930 ms and Mean cross-node dropped latency: 0 ms Are there any tools I can use to start to und

Re: Transaction failed because of timeout, retry failed because of the first try actually succeeded.

2016-06-30 Thread Robert Wille
I had this problem, and it was caused by my retry policy. For reasons I don’t remember (but is documented in a C* Jira ticket), when onWriteTimeout() is called, you cannot call RetryDecision.retry(cl), as it will be a CL that is incompatible with LWT. After the fix (2.1.?), you can pass null, an

Transaction failed because of timeout, retry failed because of the first try actually succeeded.

2016-06-30 Thread Justin Lin
Hi everyone, I recently encountered a problem wrt light weight transaction. My query is to insert a row to a table if the row doesn't exist. It goes like this: Insert Into mytable (key, col1, col2) Value("key1", 1, 2) If Not Exist My case is the driver somehow gets time out from waiting for coor

Timeout while waiting for workers when flushing pool

2016-06-01 Thread Zhang, Charles
auto-generation. It started indexing fine, for a while, and then one of the solr nodes went down. And the system log shows this: ERROR [NonPeriodicTasks:1] 2016-05-31 17:47:36,560 CassandraDaemon.java:229 - Exception in thread Thread[NonPeriodicTasks:1,5,main] java.lang.RuntimeException: Ti

Re: Session timeout

2016-02-12 Thread oleg yusim
Jack, I updated my document with all the security gaps I was able to discover (see the second table, below the fist one). I also moved the document to Google Docs from Word doc, shared on Google Drive, following Matt's suggestion. Please, see the updated link: https://docs.google.com/document/d/1

Re: Session timeout

2016-02-11 Thread oleg yusim
Jack, This document doesn't cover all the areas where user will need to get engaged in explicit mitigation, it only covers those, I wasn't sure about. But - you are making a good point here. Let me update the document with the rest of the gaps, so community would have a complete list here. Thanks

Re: Session timeout

2016-02-11 Thread Jack Krupansky
Thanks! A useful contribution, no matter what the outcome. I trust your ability to read of the doc, so I don't expect a lot of change to the responses, but we'll see. At a minimum, it will probably be good to have doc to highlight areas where users will need to engage in explicit mitigation efforts

Re: Session timeout

2016-02-11 Thread oleg yusim
Robert, Jack, Bryan, As you suggested, I put together document, titled Cassandra_Security_Topics_to_Discuss, put it on Google Drive and shared it with everybody on this list. The document contains list of questions I have on Cassandra, my take on it, and has a place for notes Community would like

Re: Session timeout

2016-02-01 Thread oleg yusim
ose companies will probably answer some of your questions for free if you > post on these mailing lists. They’ll likely answer even more if you pay > them. > > > > From: oleg yusim > Reply-To: "user@cassandra.apache.org" > Date: Friday, January 29, 2016 at 9:16 AM &g

Re: Session timeout

2016-01-29 Thread Bryan Cheng
To throw my (unsolicited) 2 cents into the ring, Oleg, you work for a well-funded and fairly large company. You are certainly free to continue using the list and asking for community support (I am definitely not in any position to tell you otherwise, anyway), but that community support is by defini

Re: Session timeout

2016-01-29 Thread Robert Coli
On Fri, Jan 29, 2016 at 3:12 PM, Jack Krupansky wrote: > One last time, I'll simply renew my objection to the way you are abusing > this list. > FWIW, while I appreciate that OP (Oleg) is attempting to do a service for the community, I agree that the flood of single topic, context-lacking posts

Re: Session timeout

2016-01-29 Thread Jack Krupansky
One last time, I'll simply renew my objection to the way you are abusing this list. You'll hear no further reply from me and I will begin marking any more of your excessive inquiries as spam. If others in the community wish to do your security review for you one item at a time, that is their prerog

Re: Session timeout

2016-01-29 Thread oleg yusim
Jack, I have to note, Cassandra documentation the way it stays now, is not nearly detailed enough. For instance: https://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configLoggingLevels_r.html is all Cassandra has to say about logging. The reason why I bring my questions to the maili

Re: Session timeout

2016-01-29 Thread Jack Krupansky
No offense, but my suggestion here is that you write up a preliminary list of your own answers based on your own reading of the doc, specs, and white papers (and source code) and post that list, like on Google Docs, for people to review in bulk, rather than force all Cassandra users on this list to

Re: Session timeout

2016-01-29 Thread oleg yusim
Jack, Appreciate the links. As I mentioned, I looked over both DSE and Cassandra sets of documentation, and ran some experiments on my Cassandra installation. What I'm bringing here is something I couldn't find definitive answer for in any of the above-mentioned sources. For instance, regarding l

Re: Session timeout

2016-01-29 Thread Jack Krupansky
There is some more detail on DSE Security in this white paper: http://www.datastax.com/wp-content/uploads/2014/04/WP-DataStax-Enterprise-SOX-Compliance.pdf It mentions auditing, for example. I think you were asking abut that earlier. There may be some additional info or discussion related to secu

Re: Session timeout

2016-01-29 Thread oleg yusim
Alex, No offense are taken, your question is absolutely legit. As we used to joke in security world "putting on my black hat"/"putting on my white hat" - i.e. same set of questions I would be asking for hacking and protecting the product. So, I commend you for being careful here. Now, at that par

Re: Session timeout

2016-01-29 Thread Alex Popescu
On Fri, Jan 29, 2016 at 8:17 AM, oleg yusim wrote: > Thanks for encouraging me, I kind of grew a bit desperate. I'm security > person, not a Cassandra expert, and doing security assessment of Cassandra > DB, I have to rely on community heavily. I will put together a composed > version of all my p

Re: Session timeout

2016-01-29 Thread oleg yusim
estions for free if you > post on these mailing lists. They’ll likely answer even more if you pay > them. > > > > From: oleg yusim > Reply-To: "user@cassandra.apache.org" > Date: Friday, January 29, 2016 at 9:16 AM > To: "user@cassandra.apache.org" > Subject: Re: Sessio

Re: Session timeout

2016-01-29 Thread Jeff Jirsa
e.org" Date: Friday, January 29, 2016 at 9:16 AM To: "user@cassandra.apache.org" Subject: Re: Session timeout Jon, I suspected something like that. I did a bit of learning on Cassandra before starting my assessment, and I understand that you are right, and it is generally no

Re: Session timeout

2016-01-29 Thread oleg yusim
> > > On Fri, Jan 29, 2016 at 6:19 AM oleg yusim wrote: > >> Not a problem, Carlos, at least you tried :) I have overall a big problem >> with my queries to Cassandra community. Most of them are not getting >> answered. >> >> Oleg >> >> On Fri,

Re: Session timeout

2016-01-29 Thread Jonathan Haddad
rote: > Not a problem, Carlos, at least you tried :) I have overall a big problem > with my queries to Cassandra community. Most of them are not getting > answered. > > Oleg > > On Fri, Jan 29, 2016 at 8:03 AM, Carlos Alonso wrote: > >> Oh, I thought you meant read/w

Re: Session timeout

2016-01-29 Thread oleg yusim
t;Security assessment questions" and will post it once again. As per the session timeout, my understanding, Cassandra currently doesn't support it. I didn't find any mention of it in documentation. Also, I just ran simple experiment on my installation (version 2.1.8, default settings): I o

Re: Session timeout

2016-01-29 Thread Carlos Alonso
t has quite a lot of activity and its easy sometimes to miss emails. About this session timeout thing, could you please reply to this thread if you find a solution? I'm curious about it. Cheers! Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso> On 29 January 2

  1   2   3   4   >