1. What was the read request? Are you fetching a single row, a million, something else? 2. What are your GC settings? 3. What's the hardware in use? What resources have been allocated to each instance? 4. Did you see this issue after a single request or is the cluster under heavy load?
If you're going to share a config it's much easier to read as an actual text file rather than a double spaced paste into the ML. In the future if you could share a link to the yaml you might get more eyes on it. Jon On Sat, Mar 16, 2019 at 3:57 PM Sundaramoorthy, Natarajan < natarajan_sundaramoor...@optum.com> wrote: > 3 pod deployed in openshift. Read request timed out due to GC collection. > Can you please look at below parameters and value to see if anything is out > of place? Thanks > > > > > > cat cassandra.yaml > > > > num_tokens: 256 > > > > > > > > hinted_handoff_enabled: true > > > > hinted_handoff_throttle_in_kb: 1024 > > > > max_hints_delivery_threads: 2 > > > > hints_directory: /cassandra_data/hints > > > > hints_flush_period_in_ms: 10000 > > > > max_hints_file_size_in_mb: 128 > > > > > > batchlog_replay_throttle_in_kb: 1024 > > > > authenticator: PasswordAuthenticator > > > > authorizer: AllowAllAuthorizer > > > > role_manager: CassandraRoleManager > > > > roles_validity_in_ms: 2000 > > > > > > permissions_validity_in_ms: 2000 > > > > > > > > > > partitioner: org.apache.cassandra.dht.Murmur3Partitioner > > > > data_file_directories: > > - /cassandra_data/data > > > > commitlog_directory: /cassandra_data/commitlog > > > > disk_failure_policy: stop > > > > commit_failure_policy: stop > > > > key_cache_size_in_mb: > > > > key_cache_save_period: 14400 > > > > > > > > row_cache_size_in_mb: 0 > > > > row_cache_save_period: 0 > > > > > > counter_cache_size_in_mb: > > > > counter_cache_save_period: 7200 > > > > > > saved_caches_directory: /cassandra_data/saved_caches > > > > commitlog_sync: periodic > > commitlog_sync_period_in_ms: 10000 > > > > commitlog_segment_size_in_mb: 32 > > > > > > seed_provider: > > - class_name: org.apache.cassandra.locator.SimpleSeedProvider > > parameters: > > - seeds: > "cassandra-0.cassandra.ihr-ei.svc.cluster.local,cassandra-1.cassandra.ihr-ei.svc.cluster.local" > > > > concurrent_reads: 32 > > concurrent_writes: 32 > > concurrent_counter_writes: 32 > > > > concurrent_materialized_view_writes: 32 > > > > > > > > > > disk_optimization_strategy: ssd > > > > > > > > memtable_allocation_type: heap_buffers > > > > commitlog_total_space_in_mb: 2048 > > > > > > index_summary_capacity_in_mb: > > > > index_summary_resize_interval_in_minutes: 60 > > > > trickle_fsync: false > > trickle_fsync_interval_in_kb: 10240 > > > > storage_port: 7000 > > > > ssl_storage_port: 7001 > > > > listen_address: 10.130.7.245 > > > > broadcast_address: 10.130.7.245 > > > > > > > > start_native_transport: true > > native_transport_port: 9042 > > > > > > > > start_rpc: true > > > > rpc_address: 0.0.0.0 > > > > rpc_port: 9160 > > > > broadcast_rpc_address: 10.130.7.245 > > > > rpc_keepalive: true > > > > rpc_server_type: sync > > > > > > > > > > thrift_framed_transport_size_in_mb: 15 > > > > incremental_backups: false > > > > snapshot_before_compaction: false > > > > auto_snapshot: true > > > > tombstone_warn_threshold: 1000 > > tombstone_failure_threshold: 100000 > > > > column_index_size_in_kb: 64 > > > > > > batch_size_warn_threshold_in_kb: 5 > > > > batch_size_fail_threshold_in_kb: 50 > > > > > > compaction_throughput_mb_per_sec: 16 > > > > compaction_large_partition_warning_threshold_mb: 100 > > > > sstable_preemptive_open_interval_in_mb: 50 > > > > > > > > read_request_timeout_in_ms: 50000 > > range_request_timeout_in_ms: 100000 > > write_request_timeout_in_ms: 20000 > > counter_write_request_timeout_in_ms: 5000 > > cas_contention_timeout_in_ms: 1000 > > truncate_request_timeout_in_ms: 60000 > > request_timeout_in_ms: 100000 > > > > cross_node_timeout: false > > > > > > phi_convict_threshold: 12 > > > > endpoint_snitch: GossipingPropertyFileSnitch > > > > dynamic_snitch_update_interval_in_ms: 100 > > dynamic_snitch_reset_interval_in_ms: 600000 > > dynamic_snitch_badness_threshold: 0.1 > > > > request_scheduler: org.apache.cassandra.scheduler.NoScheduler > > > > > > > > server_encryption_options: > > internode_encryption: none > > keystore: conf/.keystore > > truststore: conf/.truststore > > > > client_encryption_options: > > enabled: false > > optional: false > > keystore: conf/.keystore > > > > internode_compression: all > > > > inter_dc_tcp_nodelay: false > > > > tracetype_query_ttl: 86400 > > tracetype_repair_ttl: 604800 > > > > gc_warn_threshold_in_ms: 1000 > > > > enable_user_defined_functions: false > > > > enable_scripted_user_defined_functions: false > > > > windows_timer_interval: 1 > > > > > > auto_bootstrap: false > > > This e-mail, including attachments, may include confidential and/or > proprietary information, and may be used only by the person or entity > to which it is addressed. If the reader of this e-mail is not the intended > recipient or his or her authorized agent, the reader is hereby notified > that any dissemination, distribution or copying of this e-mail is > prohibited. If you have received this e-mail in error, please notify the > sender by replying to this message and delete this e-mail immediately. >
