Re: Retrieving all row keys of a CF
If you're getting partial data back, then failing eventually, try setting .withCheckpointManager() - this will let you keep track of the token ranges you've successfully processed, and not attempt to reprocess them. This will also let you set up tasks on bigger data sets that take hours or days to run, and reasonably safely interrupt it at any time without losing progress. This is some *very* old code, but I dug this out of a git history. We don't use Astyanax any longer, but maybe an example implementation will help you. This is Scala instead of Java, but hopefully you can get the gist. https://gist.github.com/MightyE/83a79b74f3a69cfa3c4e If you're timing out talking to your cluster, then I don't recommend using the cluster to track your checkpoints, but some other data store (maybe just a flatfile). Again, this is just to give you a sense of what's involved. On Fri, Jan 16, 2015 at 6:31 PM, Mohammed Guller wrote: > Both total system memory and heap size can’t be 8GB? > > > > The timeout on the Astyanax client should be greater than the timeouts on > the C* nodes, otherwise your client will timeout prematurely. > > > > Also, have you tried increasing the timeout for the range queries to a > higher number? It is not recommended to set them very high, because a lot > of other problems may start happening, but then reading 800,000 partitions > is not a normal operation. > > > > Just as an experimentation, can you set the range timeout to 45 seconds on > each node and the timeout on the Astyanax client to 50 seconds? Restart the > nodes after increasing the timeout and try again. > > > > Mohammed > > > > *From:* Ravi Agrawal [mailto:ragra...@clearpoolgroup.com] > *Sent:* Friday, January 16, 2015 5:11 PM > > *To:* user@cassandra.apache.org > *Subject:* RE: Retrieving all row keys of a CF > > > > > > 1)What is the heap size and total memory on each node? 8GB, > 8GB > > 2)How big is the cluster? 4 > > 3)What are the read and range timeouts (in cassandra.yaml) on > the C* nodes? 10 secs, 10 secs > > 4)What are the timeouts for the Astyanax client? 2 secs > > 5)Do you see GC pressure on the C* nodes? How long does GC for > new gen and old gen take? occurs every 5 secs dont see huge gc pressure, > <50ms > > 6)Does any node crash with OOM error when you try > AllRowsReader? No > > > > *From:* Mohammed Guller [mailto:moham...@glassbeam.com > ] > *Sent:* Friday, January 16, 2015 7:30 PM > *To:* user@cassandra.apache.org > *Subject:* RE: Retrieving all row keys of a CF > > > > A few questions: > > > > 1) What is the heap size and total memory on each node? > > 2) How big is the cluster? > > 3) What are the read and range timeouts (in cassandra.yaml) on the > C* nodes? > > 4) What are the timeouts for the Astyanax client? > > 5) Do you see GC pressure on the C* nodes? How long does GC for new > gen and old gen take? > > 6) Does any node crash with OOM error when you try AllRowsReader? > > > > Mohammed > > > > *From:* Ravi Agrawal [mailto:ragra...@clearpoolgroup.com > ] > *Sent:* Friday, January 16, 2015 4:14 PM > *To:* user@cassandra.apache.org > *Subject:* Re: Retrieving all row keys of a CF > > > > Hi, > > I and Ruchir tried query using AllRowsReader recipe but had no luck. We > are seeing PoolTimeoutException. > > SEVERE: [Thread_1] Error reading RowKeys > > *com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException*: > *PoolTimeoutException*: [host=servername, latency=2003(2003), > attempts=4]Timed out waiting for connection > >at > com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection( > *SimpleHostConnectionPool.java:231*) > >at > com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection( > *SimpleHostConnectionPool.java:198*) > >at > com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.borrowConnection( > *RoundRobinExecuteWithFailover.java:84*) > >at > com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation( > *AbstractExecuteWithFailoverImpl.java:117*) > >at > com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover( > *AbstractHostPartitionConnectionPool.java:338*) > >at > com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$2.execute( > *ThriftColumnFamilyQueryImpl.java:397*) > >at com.netflix.astyanax.recipes.reader.AllRowsReader$1.call( > *AllRowsReader.java:447*) > >at com.netflix.astyanax.recipes.reader.AllRowsReader$1.call( > *AllRowsReader.java:419*) > >at java.util.concurrent.FutureTask.run(*FutureTask.java:262*) > >at java.util.concurrent.ThreadPoolExecutor.runWorker( > *ThreadPoolExecutor.java:1145*) > >at java.util.concurrent.ThreadPoolExecutor$Worker.run( > *ThreadPoolExecutor.java:615*) > >at java.lang.Thread.run(*Thread.java:745*) > > > >
Should one expect to see hints being stored/delivered occasionally?
Hello,
I thought hints are being stored on /node_A/ every time /node_B/ is
unavailable for whatever the reason. I also thought that these hints are
being delivered from /node_A/ to /node_B/ when /node_B/ is back and this
is true for a /period = max_hint_window_in_ms/. After that hints are
dropped and therefore never delivered to /node_B/.
Obviously I am wrong, because occasionally we get alerted from our
monitoring system that hints are being stored and delivered, which as
far as I know indicates a problem. Now, when that happens I cannot
correlate it with any network issues (all nodes are on the same LAN
anyway) or other problems. The output from /system.log/ looks like this:
INFO [CompactionExecutor:109085] 2015-01-17 15:35:13,536
CompactionTask.java (line 262) Compacted 2 sstables to
[/var/lib/cassandra/data/DataMining/quotebyquotereference/DataMining-quoteby
quotereference-ic-89765,]. 222,905,570 bytes to 222,881,859 (~99% of
original) in 91,850ms = 2.314172MB/s. 161,259 total rows, 161,253
unique. Row merge counts were {1:161247, 2:6, }
INFO [CompactionExecutor:109090] 2015-01-17 15:35:13,537
CompactionTask.java (line 105) Compacting
[SSTableReader(path='/var/lib/cassandra/data/DataMining/quotebyquotereference/DataMining-
quotebyquotereference-ic-89750-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/DataMining/quotebyquotereference/DataMining-quotebyquotereference-ic-89765-Data.db')]
INFO [HintedHandoff:2] 2015-01-17 15:35:38,564
HintedHandOffManager.java (line 294) Started hinted handoff for host:
2ae2c679-8769-44da-a713-3bc21c670620 with IP: /10.3.5.3
INFO [HintedHandoff:1] 2015-01-17 15:35:38,564
HintedHandOffManager.java (line 294) Started hinted handoff for host:
0bb63124-6333-43fa-b1c8-3a8f6627b85a with IP: /10.3.5.2
INFO [HintedHandoff:1] 2015-01-17 15:35:38,967
HintedHandOffManager.java (line 326) Finished hinted handoff of 17 rows
to endpoint /10.3.5.2
INFO [HintedHandoff:1] 2015-01-17 15:35:38,968 ColumnFamilyStore.java
(line 633) Enqueuing flush of Memtable-hints@1779218028(614406/2848765
serialized/live bytes, 220 ops)
INFO [FlushWriter:9360] 2015-01-17 15:35:38,969 Memtable.java (line
398) Writing Memtable-hints@1779218028(614406/2848765 serialized/live
bytes, 220 ops)
INFO [FlushWriter:9360] 2015-01-17 15:35:39,192 Memtable.java (line
436) Completed flushing
/var/lib/cassandra/data/system/hints/system-hints-ic-89-Data.db (176861
bytes) for commitlog position ReplayPosition(segmentId=1418136927153,
position=20201767)
INFO [CompactionExecutor:109094] 2015-01-17 15:35:39,194
CompactionTask.java (line 105) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/hints/system-hints-ic-89-Data.db')]
INFO [CompactionExecutor:109094] 2015-01-17 15:35:39,485
CompactionTask.java (line 262) Compacted 1 sstables to
[/var/lib/cassandra/data/system/hints/system-hints-ic-90,]. 176,861
bytes to 177,355 (~100% of original) in 290ms = 0.583238MB/s. 4 total
rows, 3 unique. Row merge counts were {1:4, }
INFO [HintedHandoff:1] 2015-01-17 15:35:39,485
HintedHandOffManager.java (line 294) Started hinted handoff for host:
6b99058f-ba48-42b9-baa1-a878a74338cc with IP: /10.3.5.1
INFO [HintedHandoff:1] 2015-01-17 15:35:40,084
HintedHandOffManager.java (line 326) Finished hinted handoff of 22 rows
to endpoint /10.3.5.1
INFO [HintedHandoff:1] 2015-01-17 15:35:40,085 ColumnFamilyStore.java
(line 633) Enqueuing flush of Memtable-hints@1204004752(2356/10923
serialized/live bytes, 62 ops)
INFO [FlushWriter:9360] 2015-01-17 15:35:40,085 Memtable.java (line
398) Writing Memtable-hints@1204004752(2356/10923 serialized/live bytes,
62 ops)
Is there any other occasion that hints are stored and then being sent in
a cluster, other than network or other temporary or permanent failure?
Could it be that the client responsible for establishing a connection is
causing this? We use the Datastax C# driver for connecting to the
cluster and we run C* 1.2.18 on Ubuntu 12.04.
Many thanks,
Vasilis
who owns the data?
Hey all, I've setup a 3 node cassandra ring. All the nodes are reporting in and appear to be working correctly. I have an RF setting of 3. However under the 'Owns' category all, all that I see if a '?'. [root@beta-new:~] #nodetool status Datacenter: datacenter1 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens OwnsHost ID Rack UN xx.xx.xx.xx 227.99 KB 256 ? 97c6e976-8cad-4f39-af21-00360f091b37 rack1 UN xx.xx.xx.xx 269.4 KB 256 ? 7014cf67-4632-41ee-b4f3-454874c3b402 rack1 UN xx.xx.xx.xx 422.91 KB 256 ? 14f0efab-3ca5-43f3-9576-b9054c5a2557 rack1 I was just wondering if this might mean I have a configuration error in my cassandra.yaml file. How can I see the % of data owned by each node? Thanks Tim -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
Re: who owns the data?
Try running Nodetool status system By specifying a keyspace (system) in the command, you should get more meaningful results. Using the command on keyspaces as you dev/test/etc will provide real results. -- Colin Clark +1-320-221-9531 > On Jan 17, 2015, at 7:22 PM, Tim Dunphy wrote: > > > Hey all, > > I've setup a 3 node cassandra ring. All the nodes are reporting in and > appear to be working correctly. I have an RF setting of 3. > > However under the 'Owns' category all, all that I see if a '?'. > > > [root@beta-new:~] #nodetool status > Datacenter: datacenter1 > === > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens OwnsHost ID > Rack > UN xx.xx.xx.xx 227.99 KB 256 ? > 97c6e976-8cad-4f39-af21-00360f091b37 rack1 > UN xx.xx.xx.xx 269.4 KB 256 ? > 7014cf67-4632-41ee-b4f3-454874c3b402 rack1 > UN xx.xx.xx.xx 422.91 KB 256 ? > 14f0efab-3ca5-43f3-9576-b9054c5a2557 rack1 > > I was just wondering if this might mean I have a configuration error in my > cassandra.yaml file. How can I see the % of data owned by each node? > > Thanks > Tim > -- > GPG me!! > > gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B >
Re: who owns the data?
> > Try running > Nodetool status system > By specifying a keyspace (system) in the command, you should get more > meaningful results. Using the command on keyspaces as you dev/test/etc > will provide real results. Thank you Colin! That was the answer. When I run [root@beta-new:~] #nodetool status joke_fire1 Datacenter: datacenter1 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 162.243.60.6 293.6 KB 256 100.0% 97c6e976-8cad-4f39-af21-00360f091b37 rack1 UN 104.131.199.203 270.26 KB 256 100.0% 7014cf67-4632-41ee-b4f3-454874c3b402 rack1 UN 162.243.109.94 385.8 KB 256 100.0% 14f0efab-3ca5-43f3-9576-b9054c5a2557 rack1 On my own keyspace I do see the % owned. Doing the same on the system keyspace doesn't get me the % owned, but I guess that's because system is unique to each node? [root@beta-new:~] #nodetool status system Datacenter: datacenter1 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens OwnsHost ID Rack UN 162.243.60.6 293.6 KB 256 ? 97c6e976-8cad-4f39-af21-00360f091b37 rack1 UN 104.131.199.203 270.26 KB 256 ? 7014cf67-4632-41ee-b4f3-454874c3b402 rack1 UN 162.243.109.94 385.8 KB 256 ? 14f0efab-3ca5-43f3-9576-b9054c5a2557 rack1 Thanks Tim On Sat, Jan 17, 2015 at 10:41 PM, Colin wrote: > Try running > > Nodetool status system > > By specifying a keyspace (system) in the command, you should get more > meaningful results. Using the command on keyspaces as you dev/test/etc > will provide real results. > > -- > *Colin Clark* > +1-320-221-9531 > > > On Jan 17, 2015, at 7:22 PM, Tim Dunphy wrote: > > > Hey all, > > I've setup a 3 node cassandra ring. All the nodes are reporting in and > appear to be working correctly. I have an RF setting of 3. > > However under the 'Owns' category all, all that I see if a '?'. > > > [root@beta-new:~] #nodetool status > Datacenter: datacenter1 > === > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens OwnsHost ID > Rack > UN xx.xx.xx.xx 227.99 KB 256 ? > 97c6e976-8cad-4f39-af21-00360f091b37 rack1 > UN xx.xx.xx.xx 269.4 KB 256 ? > 7014cf67-4632-41ee-b4f3-454874c3b402 rack1 > UN xx.xx.xx.xx 422.91 KB 256 ? > 14f0efab-3ca5-43f3-9576-b9054c5a2557 rack1 > > I was just wondering if this might mean I have a configuration error in my > cassandra.yaml file. How can I see the % of data owned by each node? > > Thanks > Tim > -- > GPG me!! > > gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B > > -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
