What's your memory / CPU usage at? And how much ram + cpu do you have on this server?
On Wed, Mar 18, 2015 at 2:31 PM, Mehak Mehta <meme...@cs.stonybrook.edu> wrote: > Currently there is only single node which I am calling directly with > around 150000 rows. Full data will be in around billions per node. > The code is working only for size 100/200. Also the consecutive fetching > is taking around 5-10 secs. > > I have a parallel script which is inserting the data while I am reading > it. When I stopped the script it worked for 500/1000 but not more than > that. > > > > On Wed, Mar 18, 2015 at 5:08 AM, Ali Akhtar <ali.rac...@gmail.com> wrote: > >> If even 500-1000 isn't working, then your cassandra node might not be >> up. >> >> 1) Try running nodetool status from shell on your cassandra server, make >> sure the nodes are up. >> >> 2) Are you calling this on the same server where cassandra is running? >> Its trying to connect to localhost . If you're running it on a different >> server, try passing in the direct ip of your cassandra server. >> >> On Wed, Mar 18, 2015 at 2:05 PM, Mehak Mehta <meme...@cs.stonybrook.edu> >> wrote: >> >>> Data won't change much but queries will be different. >>> I am not working on the rendering tool myself so I don't know much >>> details about it. >>> >>> Also as suggested by you I tried to fetch data in size of 500 or 1000 >>> with java driver auto pagination. >>> It fails when the number of records are high (around 100000) with >>> following error: >>> >>> Exception in thread "main" >>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) >>> tried for query failed (tried: localhost/127.0.0.1:9042 >>> (com.datastax.driver.core.exceptions.DriverException: Timed out waiting for >>> server response)) >>> >>> >>> On Wed, Mar 18, 2015 at 4:47 AM, Ali Akhtar <ali.rac...@gmail.com> >>> wrote: >>> >>>> How often does the data change? >>>> >>>> I would still recommend a caching of some kind, but without knowing >>>> more details (how often the data is changing, what you're doing with the 1m >>>> rows after getting them, etc) I can't recommend a solution. >>>> >>>> I did see your other thread. I would also vote for elasticsearch / solr >>>> , they are more suited for the kind of analytics you seem to be doing. >>>> Cassandra is more for storing data, it isn't all that great for complex >>>> queries / analytics. >>>> >>>> If you want to stick to cassandra, you might have better luck if you >>>> made your range columns part of the primary key, so something like PRIMARY >>>> KEY(caseId, x, y) >>>> >>>> On Wed, Mar 18, 2015 at 1:41 PM, Mehak Mehta <meme...@cs.stonybrook.edu >>>> > wrote: >>>> >>>>> The rendering tool renders a portion a very large image. It may fetch >>>>> different data each time from billions of rows. >>>>> So I don't think I can cache such large results. Since same results >>>>> will rarely fetched again. >>>>> >>>>> Also do you know how I can do 2d range queries using Cassandra. Some >>>>> other users suggested me using Solr. >>>>> But is there any way I can achieve that without using any other >>>>> technology. >>>>> >>>>> On Wed, Mar 18, 2015 at 4:33 AM, Ali Akhtar <ali.rac...@gmail.com> >>>>> wrote: >>>>> >>>>>> Sorry, meant to say "that way when you have to render, you can just >>>>>> display the latest cache." >>>>>> >>>>>> On Wed, Mar 18, 2015 at 1:30 PM, Ali Akhtar <ali.rac...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> I would probably do this in a background thread and cache the >>>>>>> results, that way when you have to render, you can just cache the latest >>>>>>> results. >>>>>>> >>>>>>> I don't know why Cassandra can't seem to be able to fetch large >>>>>>> batch sizes, I've also run into these timeouts but reducing the batch >>>>>>> size >>>>>>> to 2k seemed to work for me. >>>>>>> >>>>>>> On Wed, Mar 18, 2015 at 1:24 PM, Mehak Mehta < >>>>>>> meme...@cs.stonybrook.edu> wrote: >>>>>>> >>>>>>>> We have UI interface which needs this data for rendering. >>>>>>>> So efficiency of pulling this data matters a lot. It should be >>>>>>>> fetched within a minute. >>>>>>>> Is there a way to achieve such efficiency >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Mar 18, 2015 at 4:06 AM, Ali Akhtar <ali.rac...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Perhaps just fetch them in batches of 1000 or 2000? For 1m rows, >>>>>>>>> it seems like the difference would only be a few minutes. Do you have >>>>>>>>> to do >>>>>>>>> this all the time, or only once in a while? >>>>>>>>> >>>>>>>>> On Wed, Mar 18, 2015 at 12:34 PM, Mehak Mehta < >>>>>>>>> meme...@cs.stonybrook.edu> wrote: >>>>>>>>> >>>>>>>>>> yes it works for 1000 but not more than that. >>>>>>>>>> How can I fetch all rows using this efficiently? >>>>>>>>>> >>>>>>>>>> On Wed, Mar 18, 2015 at 3:29 AM, Ali Akhtar <ali.rac...@gmail.com >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Have you tried a smaller fetch size, such as 5k - 2k ? >>>>>>>>>>> >>>>>>>>>>> On Wed, Mar 18, 2015 at 12:22 PM, Mehak Mehta < >>>>>>>>>>> meme...@cs.stonybrook.edu> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Jens, >>>>>>>>>>>> >>>>>>>>>>>> I have tried with fetch size of 10000 still its not giving any >>>>>>>>>>>> results. >>>>>>>>>>>> My expectations were that Cassandra can handle a million rows >>>>>>>>>>>> easily. >>>>>>>>>>>> >>>>>>>>>>>> Is there any mistake in the way I am defining the keys or >>>>>>>>>>>> querying them. >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> Mehak >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Mar 18, 2015 at 3:02 AM, Jens Rantil < >>>>>>>>>>>> jens.ran...@tink.se> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> Try setting fetchsize before querying. Assuming you don't set >>>>>>>>>>>>> it too high, and you don't have too many tombstones, that should >>>>>>>>>>>>> do it. >>>>>>>>>>>>> >>>>>>>>>>>>> Cheers, >>>>>>>>>>>>> Jens >>>>>>>>>>>>> >>>>>>>>>>>>> – >>>>>>>>>>>>> Skickat från Mailbox <https://www.dropbox.com/mailbox> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Mar 18, 2015 at 2:58 AM, Mehak Mehta < >>>>>>>>>>>>> meme...@cs.stonybrook.edu> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I have requirement to fetch million row as result of my query >>>>>>>>>>>>>> which is giving timeout errors. >>>>>>>>>>>>>> I am fetching results by selecting clustering columns, then >>>>>>>>>>>>>> why the queries are taking so long. I can change the timeout >>>>>>>>>>>>>> settings but I >>>>>>>>>>>>>> need the data to fetched faster as per my requirement. >>>>>>>>>>>>>> >>>>>>>>>>>>>> My table definition is: >>>>>>>>>>>>>> *CREATE TABLE images.results (uuid uuid, >>>>>>>>>>>>>> analysis_execution_id varchar, analysis_execution_uuid uuid, x >>>>>>>>>>>>>> double, y >>>>>>>>>>>>>> double, loc varchar, w double, h double, normalized varchar, >>>>>>>>>>>>>> type varchar, >>>>>>>>>>>>>> filehost varchar, filename varchar, image_uuid uuid, image_uri >>>>>>>>>>>>>> varchar, >>>>>>>>>>>>>> image_caseid varchar, image_mpp_x double, image_mpp_y double, >>>>>>>>>>>>>> image_width >>>>>>>>>>>>>> double, image_height double, objective double, cancer_type >>>>>>>>>>>>>> varchar, Area >>>>>>>>>>>>>> float, submit_date timestamp, points list<double>, PRIMARY KEY >>>>>>>>>>>>>> ((image_caseid),Area,uuid));* >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here each row is uniquely identified on the basis of unique >>>>>>>>>>>>>> uuid. But since my data is generally queried based upon >>>>>>>>>>>>>> *image_caseid >>>>>>>>>>>>>> *I have made it partition key. >>>>>>>>>>>>>> I am currently using Java Datastax api to fetch the results. >>>>>>>>>>>>>> But the query is taking a lot of time resulting in timeout >>>>>>>>>>>>>> errors: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Exception in thread "main" >>>>>>>>>>>>>> com.datastax.driver.core.exceptions.NoHostAvailableException: >>>>>>>>>>>>>> All host(s) >>>>>>>>>>>>>> tried for query failed (tried: localhost/127.0.0.1:9042 >>>>>>>>>>>>>> (com.datastax.driver.core.exceptions.DriverException: Timed out >>>>>>>>>>>>>> waiting for >>>>>>>>>>>>>> server response)) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:289) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:205) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52) >>>>>>>>>>>>>> at QueryDB.queryArea(TestQuery.java:59) >>>>>>>>>>>>>> at TestQuery.main(TestQuery.java:35) >>>>>>>>>>>>>> Caused by: >>>>>>>>>>>>>> com.datastax.driver.core.exceptions.NoHostAvailableException: >>>>>>>>>>>>>> All host(s) >>>>>>>>>>>>>> tried for query failed (tried: localhost/127.0.0.1:9042 >>>>>>>>>>>>>> (com.datastax.driver.core.exceptions.DriverException: Timed out >>>>>>>>>>>>>> waiting for >>>>>>>>>>>>>> server response)) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:108) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:179) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:744) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Also when I try the same query on console even while using >>>>>>>>>>>>>> limit of 2000 rows: >>>>>>>>>>>>>> >>>>>>>>>>>>>> cqlsh:images> select count(*) from results where >>>>>>>>>>>>>> image_caseid='TCGA-HN-A2NL-01Z-00-DX1' and Area<100 and Area>20 >>>>>>>>>>>>>> limit 2000; >>>>>>>>>>>>>> errors={}, last_host=127.0.0.1 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>>>> Mehak >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
