Re: terrible read/write latency fluctuation

On Sun, Oct 25, 2015 at 3:47 PM, 曹志富 wrote: > >> I will try to trace a read that take > 20msec >> . >> >> just HDD.no delete just 60days ttl.value size is small ,max length is 140. >> >> >> My data like Time Series . date of 90% reads which timestamp < 7days. >> data almost just insert ,with a lit update. >> > >

Re: terrible read/write latency fluctuation

I will try to trace a read that take > 20msec . just HDD.no delete just 60days ttl.value size is small ,max length is 140. My data like Time Series . date of 90% reads which timestamp < 7days. data almost just insert ,with a lit update.

CqlOutputFormat with auth

Hadoop2.6,cassandra2.1.6. Here is the exception stack: Error: java.lang.RuntimeException: InvalidRequestException(why:You have not logged in) at org.apache.cassandra.hadoop.cql3.CqlRecordWriter.(CqlRecordWriter.java:121) at org.apache.cassandra.hadoop.cql3.CqlRecordWriter.(CqlRecordWriter.java:88

Re: unusual GC log

in > cassandra.yaml; are you running with stock GC settings in cassandra-env.sh? > what JDK/OS? > > On Oct 19, 2015, at 11:40 PM, 曹志富 wrote: > > INFO [Service Thread] 2015-10-20 10:42:47,854 GCInspector.java:252 - > ParNew GC in 476ms. CMS Old Gen: 4288526240 -> 4725514832; Par E

unusual GC log

INFO [Service Thread] 2015-10-20 10:42:47,854 GCInspector.java:252 - ParNew GC in 476ms. CMS Old Gen: 4288526240 -> 4725514832; Par Eden Space: 671088640 -> 0; INFO [Service Thread] 2015-10-20 10:42:50,870 GCInspector.java:252 - ParNew GC in 423ms. CMS Old Gen: 4725514832 -> 5114687560; Par Ede

Re: abnormal log after remove a node

is. > > C*heers, > > Alain > > 2015-06-19 11:40 GMT+02:00 曹志富 : > >> I have a C* 2.1.5 with 24 nodes.A few days ago ,I have remove a node from >> this cluster using nodetool decommission. >> >> But tody I find some log like this: >> >> INFO [G

Re: node join ant decommission exception

] -- Ranger Tsao 2015-06-29 21:33 GMT+08:00 曹志富 : > error when node join cluster: > > WARN 13:30:18 UnknownColumnFamilyException reading from socket; closing > org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find > cfId=917

Re: node join ant decommission exception

veMessages(IncomingTcpConnection.java:170) ~[apache-cassandra-2.1.6.jar:2.1.6] at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:88) ~[apache-cassandra-2.1.6.jar:2.1.6] -- Ranger Tsao 2015-06-29 18:01 GMT+08:00 曹志富 :

Range not found after nodetool decommission

ERROR [OptionalTasks:1] 2015-06-25 08:56:19,156 CassandraDaemon.java:223 - Exception in thread Thread[OptionalTasks:1,5,main] java.lang.AssertionError: -110036444293069784 not found in -- Ranger Tsao

Re: system-hints compaction all the time

this is unusual. -- Ranger Tsao 2015-06-22 16:06 GMT+08:00 Jason Wee : > what's your questions? > > On Mon, Jun 22, 2015 at 12:05 AM, 曹志富 wrote: > >> the logger like this : >> >> >> INFO [Compac

system-hints compaction all the time

the logger like this : INFO [CompactionExecutor:501] 2015-06-21 21:42:36,306 CompactionTask.java:140 - Compacting [SSTableReader(path='/home/ant/apache-cassandra-2.1.6/bin/../data/data/system/hints/system-hints-ka-365-Data.db')] INFO [CompactionExecutor:501] 2015-06-21 21:42:37,782 CompactionTa

cfstats ERROR

error: /home/ant/apache-cassandra-2.1.6/bin/../data/data/blogger/edgestore/blogger-edgestore-tmplink-ka-146100-Data.db -- StackTrace -- java.lang.AssertionError: /home/ant/apache-cassandra-2.1.6/bin/../data/data/blogger/edgestore/blogger-edgestore-tmplink-ka-146100-Data.db at org.apache.cassandra.i

exception when anti-entropy repair

ERROR [RepairJobTask:3] 2015-06-21 00:18:02,187 RepairJob.java:145 - Error occurred during snapshot phase java.lang.RuntimeException: Could not create snapshot at /172.19.104.107 at org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:78) ~[apache-cassandr

abnormal log after remove a node

I have a C* 2.1.5 with 24 nodes.A few days ago ,I have remove a node from this cluster using nodetool decommission. But tody I find some log like this: INFO [GossipStage:1] 2015-06-19 17:38:05,616 Gossiper.java:968 - InetAddress /172.19.105.41 is now DOWN INFO [GossipStage:1] 2015-06-19 17:38:0

seems a doc error

In ths doc :Enabling JMX authentication In the chapter ： $ nodetool status -u cassandra -pw cassandra should change to $ nodetool -u cassandra -pw cassandra status -

Re: auto clear data with ttl

it. > > Sent from my iPhone > > On Jun 8, 2015, at 7:26 PM, 曹志富 wrote: > > I have C* 2.1.5,store some data with ttl.Reduce the gc_grace_seconds to > zero. > > But it seems has no effect. > > Did I miss something? > -- > Ranger Tsao > >

auto clear data with ttl

I have C* 2.1.5,store some data with ttl.Reduce the gc_grace_seconds to zero. But it seems has no effect. Did I miss something? -- Ranger Tsao

Re: what this error mean

t; > jason > > > > On Fri, May 29, 2015 at 9:39 AM, 曹志富 wrote: > >> I have a 25 noedes C* cluster with C* 2.1.3. These days a node occur >> split brain many times。 >> >> check the log I found this: >> >> INFO [MemtableFlushWriter:118] 2

what this error mean

I have a 25 noedes C* cluster with C* 2.1.3. These days a node occur split brain many times。 check the log I found this: INFO [MemtableFlushWriter:118] 2015-05-29 08:07:39,176 Memtable.java:378 - Completed flushing /home/ant/apache-cassandra-2.1.3/bin/../data/data/system/sstable_activity-5a1

java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code

Hi guys: I havle C* 2.1.3 cluster,25 nodes ,running in JDK_1.7.0_71, CentOS 2.6.32-220.el6.x86_64,4 Core,32GB RAM. Today one of the nodes,has some error like this: java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code at org.apache.cassandra.i

Re: C* 2.0.9 Compaction Error

Have someone see this before ? How to deal with this error?remove this node first ,then add this node to cluster? -- Ranger Tsao 2015-03-10 10:57 GMT+08:00 曹志富 : > Hi,every one: > > I have a 12 nodes C* 2.0.9 cluster for titan.I found some error w

C* 2.0.9 Compaction Error

Hi,every one: I have a 12 nodes C* 2.0.9 cluster for titan.I found some error when doing compaction,the exception stack: java.lang.AssertionError: Added column does not sort as the last column at org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:115) at or

Re: how to clear data from disk

nodetool clearsnapshot -- Ranger Tsao 2015-03-10 10:47 GMT+08:00 鄢来琼 : > Hi ALL, > > > > After drop table, I found the data is not removed from disk, I should > reduce the gc_grace_seconds before the drop operation. > > I have to wait for 10 days, but there i

Run witch repair cmd when increase replication factor

I want fo increase replication factor in my C* 2.1.3 cluster(rf chang from 2 to 3 for some keyspaces). I read the doc of Updating the replication factor . The step two is run the nodetool repair.But as I know nodetoo

Re: Input/Output Error

; > > On Wed, Mar 4, 2015 at 2:18 AM, 曹志富 wrote: > >> Some times My C* 2.1.3 cluster compaction or streaming occur this error >> ,do this because of disk or filesystem problem?? >> >> Thanks All. >> >> -- >> Ranger Tsao >> > >

Input/Output Error

Some times My C* 2.1.3 cluster compaction or streaming occur this error ,do this because of disk or filesystem problem?? Thanks All. -- Ranger Tsao

Why nodetool netstats show this

lastest joined in my cluster,his status is Up Normal. I want to know why this message?Shoul I remove this node? -- 曹志富 手机：18611121927 邮箱：caozf.zh...@gmail.com 微博：http://weibo.com/boliza/

Re: Node joining take a long time

my listen_address as some as rpc_address,and rpc_interface not config -- 曹志富 手机：18611121927 邮箱：caozf.zh...@gmail.com 微博：http://weibo.com/boliza/ 2015-02-20 23:16 GMT+08:00 Jan Kesten : > Hi, > > a short hint for those upgrading: If you upgrade

Re: Node joining take a long time

So ,what can I do???Waiting for 2.1.4 or upgrade to 2.1.3?? -- 曹志富 手机：18611121927 邮箱：caozf.zh...@gmail.com 微博：http://weibo.com/boliza/ 2015-02-20 3:16 GMT+08:00 Robert Coli : > On Thu, Feb 19, 2015 at 7:34 AM, Mark Reddy > wrote: > >> I

Re: Node joining take a long time

First thank all of you. Almost three days,till right now the status is still Joining. My cluster per 650G a node. -- 曹志富 手机：18611121927 邮箱：caozf.zh...@gmail.com 微博：http://weibo.com/boliza/ 2015-02-20 3:16 GMT+08:00 Robert Coli : > On Thu, Feb 19, 2015 a

Re: can't delete tmp file

Thanks you Roland -- 曹志富 手机：18611121927 邮箱：caozf.zh...@gmail.com 微博：http://weibo.com/boliza/ 2015-02-19 20:32 GMT+08:00 Roland Etzenhammer : > Hi, > > try 2.1.3 - with 2.1.2 this is "normal". From the changelog: > > * Make sure we d

Re: can't delete tmp file

Just upgrade my cluster to 2.1.3??? -- 曹志富 手机：18611121927 邮箱：caozf.zh...@gmail.com 微博：http://weibo.com/boliza/ 2015-02-19 20:32 GMT+08:00 Roland Etzenhammer : > Hi, > > try 2.1.3 - with 2.1.2 this is "normal". From the changelog: > >

Node joining take a long time

total I want know ,is there some problem with my cluster? -- 曹志富 手机：18611121927 邮箱：caozf.zh...@gmail.com 微博：http://weibo.com/boliza/

can't delete tmp file

Hi guys: I have a 20 nodes C* cluster with vnodes,version is 2.1.2. when after compaction some nodes has some tmp files (like this: weibo-content1-tmp-ka-95117-Data.db),there can't remove after GC. I want know how to deal this,and It's normal? Thanks. ------

Re: How to deal with too many sstables

You are right.I have already change cold_reads_to_omit to 0.0. -- 曹志富 手机：18611121927 邮箱：caozf.zh...@gmail.com 微博：http://weibo.com/boliza/ 2015-02-03 14:15 GMT+08:00 Roland Etzenhammer : > Hi, > > maybe you are running into an issue that I also had o

Re: How to deal with too many sstables

Just run nodetool repair. The nodes witch has many sstables are newest in my cluster.Before add these nodes to my cluster ,my cluster have not compaction automaticly because my cluster is an only write cluster. thanks. -- 曹志富 手机：18611121927 邮箱：caozf.zh

How to deal with too many sstables

Hi,all: I have 18 nodes C* cluster with cassandra2.1.2.Some nodes have aboud 40,000+ sstables. my compaction strategy is STCS. Could someone give me some solution to deal with this situation. Thanks. -- 曹志富 手机：18611121927 邮箱：caozf.zh...@gmail.com 微博：http

Re: SStables can't compat automaticly

Your are right,currently My cluster only write ,when My cluster build after two month ,I will change to the default thresholds. Thanks for your reply. -- 曹志富 手机：18611121927 邮箱：caozf.zh...@gmail.com 微博：http://weibo.com/boliza/ 2015-01-26 22:40 GMT+08:00 Eric

Re: SStables can't compat automaticly

I read the CQL table properties again.This property could contorl the compaction . Right now My C* cluster only write without any read. -- 曹志富 手机：18611121927 邮箱：caozf.zh...@gmail.com 微博：http://weibo.com/boliza/ 2015-01-26 17:41 GMT+08:00 Roland Etzenhammer

Re: SStables can't compat automaticly

Yes I use cassandra 2.1.2 JDK is 1.7.0_71. I will try your solution. Thank you Roland Etzenhammer!!! -- 曹志富 手机：18611121927 邮箱：caozf.zh...@gmail.com 微博：http://weibo.com/boliza/ 2015-01-26 17:41 GMT+08:00 Roland Etzenhammer : > Hi, > > are you runn

Re: SStables can't compat automaticly

No,to confirm this I have run the command all my nodes:bin/nodetool enableautocompaction -- 曹志富 手机：18611121927 邮箱：caozf.zh...@gmail.com 微博：http://weibo.com/boliza/ 2015-01-26 16:49 GMT+08:00 Jason Wee : > Did you disable auto compaction through nodet

SStables can't compat automaticly

t to compact the sstables , it cost a long long time .Also the new sstables can't compat automatic. I trace the log , the CMS GC too often,almost 30 minute onetime. Could someone help me to solve this problem. -- 曹志富 手机：18611121927 邮箱：caozf.zh...@gmail.com 微博：http://weibo.com/boliza/