Re: Query timed out after PT2M
Update - I believe that for large tables, the spark.cassandra.read.timeoutMS needs to be very long; like 4 hours or longer. The job now runs much longer, but still doesn't complete. I'm now facing this all too familiar error: com.datastax.oss.driver.api.core.servererrors.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_ONE (1 responses were required but only 0 replica responded) In the past this has been due to clocks being out of sync (not the issue here), or a table that has been written to with LOCAL_ONE instead of LOCAL_QUORUM. I don't believe either of those are the case. To be sure, I ran a repair on the table overnight (about 17 hours to complete). For the next test, I set the spark.cassandra.connection.timeoutMS to 6 (default is 5000), and the spark.cassandra.query.retry.count to -1. Suggestions? Thoughts? Thanks all. -Joe On 2/7/2022 10:35 AM, Joe Obernberger wrote: Some more info. Tried different GC strategies - no luck. It only happens on large tables (more than 1 billion rows). Works fine on a 300million row table. There is very high CPU usage during the run. I've tried setting spark.dse.continuousPagingEnabled to false and I've tried setting spark.cassandra.input.readsPerSec to 10; no effect. Stats: nodetool cfstats doc.doc Total number of tables: 82 Keyspace : doc Read Count: 9620329 Read Latency: 0.5629605546754171 ms Write Count: 510561482 Write Latency: 0.02805177028806885 ms Pending Flushes: 0 Table: doc SSTable count: 77 Old SSTable count: 0 Space used (live): 82061188941 Space used (total): 82061188941 Space used by snapshots (total): 0 Off heap memory used (total): 317037065 SSTable Compression Ratio: 0.3816525125492022 Number of partitions (estimate): 101021793 Memtable cell count: 209646 Memtable data size: 44087966 Memtable off heap memory used: 0 Memtable switch count: 10 Local read count: 25665 Local read latency: NaN ms Local write count: 2459322 Local write latency: NaN ms Pending flushes: 0 Percent repaired: 0.0 Bytes repaired: 0.000KiB Bytes unrepaired: 184.869GiB Bytes pending repair: 0.000KiB Bloom filter false positives: 2063 Bloom filter false ratio: 0.01020 Bloom filter space used: 169249016 Bloom filter off heap memory used: 169248400 Index summary off heap memory used: 50863401 Compression metadata off heap memory used: 96925264 Compacted partition minimum bytes: 104 Compacted partition maximum bytes: 943127 Compacted partition mean bytes: 1721 Average live cells per slice (last five minutes): NaN Maximum live cells per slice (last five minutes): 0 Average tombstones per slice (last five minutes): NaN Maximum tombstones per slice (last five minutes): 0 Dropped Mutations: 0 nodetool tablehistograms doc.doc doc/doc histograms Percentile Read Latency Write Latency SSTables Partition Size Cell Count (micros) (micros) (bytes) 50% 0.00 0.00 0.00 1109 86 75% 0.00 0.00 0.00 3311 215 95% 0.00 0.00 0.00 3311 215 98% 0.00 0.00 0.00 3311 215 99% 0.00 0.00 0.00 3311 215 Min 0.00 0.00 0.00 104 5 Max 0.00 0.00 0.00 943127 2299 I'm stuck. -Joe On 2/3/2022 9:30 PM, manish khandelwal wrote: It maybe the case you have lots of tombstones in this table which is making reads slow and timeouts during bulk reads. On Fri, Feb 4, 2022, 03:23 Joe Obernberger wrote: So it turns out that number after PT is increments of 60 seconds. I changed the timeout to 96, and now I get PT16M (96/6). Since I'm still getting timeouts, something else must be wrong. Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 306 in stage 0.0 failed 4 times, most recent failure: Lost task 306.3 in stage 0.0 (TID 1180) (172.16.100.39 executor 0): com.datastax.oss.driver.api.core.DriverTimeoutException: Query timed out after PT16M
Re: Query timed out after PT2M
Update - the answer was spark.cassandra.input.split.sizeInMB. The default value is 512MBytes. Setting this to 50 resulted in a lot more splits and the job ran in under 11 minutes; no timeout errors. In this case the job was a simple count. 10 minutes 48 seconds for over 8.2 billion rows. Fast! Good times ahead. -Joe On 2/8/2022 10:00 AM, Joe Obernberger wrote: Update - I believe that for large tables, the spark.cassandra.read.timeoutMS needs to be very long; like 4 hours or longer. The job now runs much longer, but still doesn't complete. I'm now facing this all too familiar error: com.datastax.oss.driver.api.core.servererrors.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_ONE (1 responses were required but only 0 replica responded) In the past this has been due to clocks being out of sync (not the issue here), or a table that has been written to with LOCAL_ONE instead of LOCAL_QUORUM. I don't believe either of those are the case. To be sure, I ran a repair on the table overnight (about 17 hours to complete). For the next test, I set the spark.cassandra.connection.timeoutMS to 6 (default is 5000), and the spark.cassandra.query.retry.count to -1. Suggestions? Thoughts? Thanks all. -Joe On 2/7/2022 10:35 AM, Joe Obernberger wrote: Some more info. Tried different GC strategies - no luck. It only happens on large tables (more than 1 billion rows). Works fine on a 300million row table. There is very high CPU usage during the run. I've tried setting spark.dse.continuousPagingEnabled to false and I've tried setting spark.cassandra.input.readsPerSec to 10; no effect. Stats: nodetool cfstats doc.doc Total number of tables: 82 Keyspace : doc Read Count: 9620329 Read Latency: 0.5629605546754171 ms Write Count: 510561482 Write Latency: 0.02805177028806885 ms Pending Flushes: 0 Table: doc SSTable count: 77 Old SSTable count: 0 Space used (live): 82061188941 Space used (total): 82061188941 Space used by snapshots (total): 0 Off heap memory used (total): 317037065 SSTable Compression Ratio: 0.3816525125492022 Number of partitions (estimate): 101021793 Memtable cell count: 209646 Memtable data size: 44087966 Memtable off heap memory used: 0 Memtable switch count: 10 Local read count: 25665 Local read latency: NaN ms Local write count: 2459322 Local write latency: NaN ms Pending flushes: 0 Percent repaired: 0.0 Bytes repaired: 0.000KiB Bytes unrepaired: 184.869GiB Bytes pending repair: 0.000KiB Bloom filter false positives: 2063 Bloom filter false ratio: 0.01020 Bloom filter space used: 169249016 Bloom filter off heap memory used: 169248400 Index summary off heap memory used: 50863401 Compression metadata off heap memory used: 96925264 Compacted partition minimum bytes: 104 Compacted partition maximum bytes: 943127 Compacted partition mean bytes: 1721 Average live cells per slice (last five minutes): NaN Maximum live cells per slice (last five minutes): 0 Average tombstones per slice (last five minutes): NaN Maximum tombstones per slice (last five minutes): 0 Dropped Mutations: 0 nodetool tablehistograms doc.doc doc/doc histograms Percentile Read Latency Write Latency SSTables Partition Size Cell Count (micros) (micros) (bytes) 50% 0.00 0.00 0.00 1109 86 75% 0.00 0.00 0.00 3311 215 95% 0.00 0.00 0.00 3311 215 98% 0.00 0.00 0.00 3311 215 99% 0.00 0.00 0.00 3311 215 Min 0.00 0.00 0.00 104 5 Max 0.00 0.00 0.00 943127 2299 I'm stuck. -Joe On 2/3/2022 9:30 PM, manish khandelwal wrote: It maybe the case you have lots of tombstones in this table which is making reads slow and timeouts during bulk reads. On Fri, Feb 4, 2022, 03:23 Joe Obernberger wrote: So it turns out that number after PT is increments of 60 seconds. I changed the timeout to 96, and now I get PT16M (96/6). Since I'm still getting t
