Re: [External]Cross-Node Latency Issues

2024-10-23 Thread Jiri Steuer (EIT) 
Hi Naman,

To be honest, in case of allow filtering ... you can expect Langer processing 
time and you cannot keep SLA. Can you use indexes and remove allow filtering?

NOTE: you can use SAI in version 5

Regards

 Jiri

Zasláno z Outlooku pro Android



This item's classification is Internal. It was created by and is in property of 
the EmbedIT. Do not distribute outside of the organization.


Od: Naman kaushik 
Odesláno: čtvrtek, října 24, 2024 6:26:27 dop.
Komu: user@cassandra.apache.org 
Předmět: [External]Cross-Node Latency Issues

This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly with links 
and attachments.
Please report all suspicious e-mails to 
helpd...@embedit.com



Hello everyone,

We are currently using Cassandra 4.1.3 in a two-data-center cluster. Recently, 
we observed cross-node latency spikes of 3-4 seconds in one of our data 
centers. Below are the relevant logs from all three nodes in this DC:

DEBUG [ScheduledTasks:1] 2024-10-20 02:46:43,164 MonitoringTask.java:174 - 413 
operations were slow in the last 5001 msecs:
, time 3400 
msec - slow timeout 500 msec
, 
time 2309 msec - slow timeout 500 msec/cross-node
, time 3287 msec 
- slow timeout 500 msec/cross-node
, time 2878 msec - slow timeout 500 msec/cross-node
, time 3056 msec - slow timeout 500 msec/cross-node
, time 2353 msec - slow timeout 500 msec/cross-node
, time 2224 msec - slow timeout 500 msec/cross-node
, time 3396 
msec - slow timeout 500 msec

Upon investigation, we found no GC pauses at the time of the latency, and CPU 
and memory utilization across all nodes appeared normal. Additionally, latency 
metrics from Grafana also showed standard performance.

Given these observations, we are trying to identify the potential causes of 
this latency. Any insights or suggestions from the community would be greatly 
appreciated!

Thank you!

Re: Tombstone Generation in Cassandra 4.1.3 Despite No Update/Delete Operations

2024-10-23 Thread Naman kaushik 
Thanks everyone for your responses.
We have columns with list and list types, and after using
sstabledump, we found that the tombstones are being generated due to these
columns.

I’ve encountered another issue related to tombstones in a table that is not
involved in any write operations, as it is solely used for data processing.
Despite no update or delete operations occurring, I'm observing one
tombstone scanned per query. The TTL is set to 0, and I’ve manually
attempted to compact the table on each node, yet the tombstone remains.
What could be the possible reason for this behavior?

On Wed, Oct 9, 2024 at 10:32 AM Jon Haddad  wrote:

> Are you using collections?
>
> —
> Jon Haddad
> Rustyrazorblade Consulting
> rustyrazorblade.com
>
>
> On Tue, Oct 8, 2024 at 10:52 PM Naman kaushik 
> wrote:
>
>> Hi Community,
>>
>> We are currently using Cassandra version 4.1.3 and have encountered an
>> issue related to tombstone generation. We have two tables storing monthly
>> data: table_september and table_october. Each table has a TTL of 30 days.
>>
>> For the month of October, data is being inserted into the table_october,
>> and we are seeing the following warning at the start of the month:
>>
>> *WARN  [CompactionExecutor:22030] 2024-10-07 16:37:16,376 
>> BigTableWriter.java:274 - Writing 102594 tombstones to **table_october*
>>
>> *Here are a few things to note*:
>>
>>- No update or delete operations are being performed on the table.
>>- TTL is correctly set to 30 days, and the data being inserted is
>>within this time range, so the TTL shouldn't be the reason for tombstones.
>>- No null values are being inserted in any column.
>>
>> We are still seeing tombstones being generated for the October table.
>> Does anyone have insights into what could be causing these tombstones, or
>> how we can prevent this from happening?
>>
>> Any help would be greatly appreciated!
>>
>> Thanks in advance.
>>
>

Re: Cross-Node Latency Issues

2024-10-23 Thread Stéphane Alleaume 
Hi,

How many sstables per read ? 

Are your partitions unbounded ?

What max size are the partitions ?

Kind regards 
Stéphane 



Le 24 octobre 2024 06:25:05 GMT+02:00, Naman kaushik  
a écrit :
>Hello everyone,
>
>We are currently using Cassandra 4.1.3 in a two-data-center cluster.
>Recently, we observed cross-node latency spikes of 3-4 seconds in one of
>our data centers. Below are the relevant logs from all three nodes in this
>DC:
>
>DEBUG [ScheduledTasks:1] 2024-10-20 02:46:43,164
>MonitoringTask.java:174 - 413 operations were slow in the last 5001
>msecs:
>LastModified, ItemImages, ItemTitles, ItemCategories, ItemRating,
>ApprovalStatus, LocalName, UserIdentifier, IsDisplayed, VariantOptions
>FROM product_data.item_table WHERE item_table_display_id =
>2854462277448 LIMIT 5000 ALLOW FILTERING>, time 3400 msec - slow
>timeout 500 msec
>MasterGroupData, MasterSubCategoryData, MasterParentCategoryData FROM
>product_data.taxonomy_table WHERE master_id = 6402 LIMIT 5000 ALLOW
>FILTERING>, time 2309 msec - slow timeout 500 msec/cross-node
>LastModified, ItemImages, ItemTitles, ItemCategories, ItemRating,
>ApprovalStatus, LocalName, UserIdentifier, IsDisplayed, VariantOptions
>FROM product_data.item_table WHERE item_table_display_id = 24279823548
>LIMIT 5000 ALLOW FILTERING>, time 3287 msec - slow timeout 500
>msec/cross-node
>LastModified, ItemImages, ItemTitles, ItemCategories, ItemRating,
>ApprovalStatus, LocalName, UserIdentifier, IsDisplayed, VariantOptions
>FROM product_data.item_table WHERE item_table_display_id =
>2854486264330 LIMIT 5000 ALLOW FILTERING>, time 2878 msec - slow
>timeout 500 msec/cross-node
>MasterGroupData, MasterSubCategoryData, MasterParentCategoryData FROM
>product_data.taxonomy_table WHERE master_id = 27245 LIMIT 5000 ALLOW
>FILTERING>, time 3056 msec - slow timeout 500 msec/cross-node
>MasterGroupData, MasterSubCategoryData, MasterParentCategoryData FROM
>product_data.taxonomy_table WHERE master_id = 32856 LIMIT 5000 ALLOW
>FILTERING>, time 2353 msec - slow timeout 500 msec/cross-node
>MasterGroupData, MasterSubCategoryData, MasterParentCategoryData FROM
>product_data.taxonomy_table WHERE master_id = 95589 LIMIT 5000 ALLOW
>FILTERING>, time 2224 msec - slow timeout 500 msec/cross-node
>LastModified, ItemImages, ItemTitles, ItemCategories, ItemRating,
>ApprovalStatus, LocalName, UserIdentifier, IsDisplayed, VariantOptions
>FROM product_data.item_table WHERE item_table_display_id =
>2854514159012 LIMIT 5000 ALLOW FILTERING>, time 3396 msec - slow
>timeout 500 msec
>
>Upon investigation, we found no GC pauses at the time of the latency, and
>CPU and memory utilization across all nodes appeared normal. Additionally,
>latency metrics from Grafana also showed standard performance.
>
>Given these observations, we are trying to identify the potential causes of
>this latency. Any insights or suggestions from the community would be
>greatly appreciated!
>
>Thank you!

Re: Cross-Node Latency Issues

2024-10-23 Thread MyWorld 
How many sstables per read ? -- >> 1 to 4 sstables

Are your partitions unbounded ? >> No

What max size are the partitions ? >> P99 varies from few bytes to 70KB
while max partition of tables varies from few bytes to 900 KB

On Thu, Oct 24, 2024 at 11:00 AM Stéphane Alleaume 
wrote:

> Hi,
>
> How many sstables per read ?
>
> Are your partitions unbounded ?
>
> What max size are the partitions ?
>
> Kind regards
> Stéphane
>
>
>
> Le 24 octobre 2024 06:25:05 GMT+02:00, Naman kaushik <
> namankaush...@gmail.com> a écrit :
>
>> Hello everyone,
>>
>> We are currently using Cassandra 4.1.3 in a two-data-center cluster.
>> Recently, we observed cross-node latency spikes of 3-4 seconds in one of
>> our data centers. Below are the relevant logs from all three nodes in this
>> DC:
>>
>> DEBUG [ScheduledTasks:1] 2024-10-20 02:46:43,164 MonitoringTask.java:174 - 
>> 413 operations were slow in the last 5001 msecs:
>> > LastModified, ItemImages, ItemTitles, ItemCategories, ItemRating, 
>> ApprovalStatus, LocalName, UserIdentifier, IsDisplayed, VariantOptions FROM 
>> product_data.item_table WHERE item_table_display_id = 2854462277448 LIMIT 
>> 5000 ALLOW FILTERING>, time 3400 msec - slow timeout 500 msec
>> > MasterSubCategoryData, MasterParentCategoryData FROM 
>> product_data.taxonomy_table WHERE master_id = 6402 LIMIT 5000 ALLOW 
>> FILTERING>, time 2309 msec - slow timeout 500 msec/cross-node
>> > LastModified, ItemImages, ItemTitles, ItemCategories, ItemRating, 
>> ApprovalStatus, LocalName, UserIdentifier, IsDisplayed, VariantOptions FROM 
>> product_data.item_table WHERE item_table_display_id = 24279823548 LIMIT 5000 
>> ALLOW FILTERING>, time 3287 msec - slow timeout 500 msec/cross-node
>> > LastModified, ItemImages, ItemTitles, ItemCategories, ItemRating,
>> ApprovalStatus, LocalName, UserIdentifier, IsDisplayed, VariantOptions FROM 
>> product_data.item_table WHERE item_table_display_id = 2854486264330 LIMIT 
>> 5000 ALLOW FILTERING>, time 2878 msec - slow timeout 500 msec/cross-node
>> > MasterSubCategoryData, MasterParentCategoryData FROM 
>> product_data.taxonomy_table WHERE master_id = 27245 LIMIT 5000 ALLOW 
>> FILTERING>, time 3056 msec - slow timeout 500 msec/cross-node
>> > MasterSubCategoryData, MasterParentCategoryData FROM 
>> product_data.taxonomy_table WHERE master_id = 32856 LIMIT 5000 ALLOW 
>> FILTERING>, time 2353 msec - slow timeout 500 msec/cross-node
>> > MasterSubCategoryData, MasterParentCategoryData FROM 
>> product_data.taxonomy_table WHERE master_id = 95589 LIMIT 5000 ALLOW 
>> FILTERING>, time 2224 msec - slow timeout 500 msec/cross-node
>> > LastModified, ItemImages, ItemTitles, ItemCategories, ItemRating, 
>> ApprovalStatus, LocalName, UserIdentifier, IsDisplayed, VariantOptions FROM 
>> product_data.item_table WHERE item_table_display_id = 2854514159012 LIMIT 
>> 5000 ALLOW FILTERING>, time 3396 msec - slow timeout 500 msec
>>
>> Upon investigation, we found no GC pauses at the time of the latency, and
>> CPU and memory utilization across all nodes appeared normal. Additionally,
>> latency metrics from Grafana also showed standard performance.
>>
>> Given these observations, we are trying to identify the potential causes
>> of this latency. Any insights or suggestions from the community would be
>> greatly appreciated!
>>
>> Thank you!
>>
>