The indexes were written about 1-2 months ago. No data has been added to the servers since the indexes were created. Additionally, the indexes appeared to be stable until I noticed the issue today. ... which occurred after a made a large query without setting a LIMIT
I set the consistency level and moved the select statement between different nodes. The results remained inconsistent, returning a random number between 0 and 8. It did not appear to make much difference between the different nodes or consistency level. However, after about 20 attempts, all servers started to only return 0 results. Lastly, this appeared in the logs during that time: INFO [IndexSummaryManager:1] 2016-08-02 22:11:43,245 IndexSummaryRedistribution.java:74 - Redistributing index summaries INFO [OptionalTasks:1] 2016-08-02 22:25:06,508 NoSpamLogger.java:91 - Maximum memory usage reached (536870912 bytes), cannot allocate chunk of 1048576 bytes On Tue, Aug 2, 2016 at 6:58 PM, DuyHai Doan <[email protected]> wrote: > One possible explanation is that you're querying data while the index > files are being built so that the result are different > The second possible explanation is the consistency level. > > Try the query again using CL = QUORUM, try on several nodes to see if the > results are different > > On Tue, Aug 2, 2016 at 6:32 PM, George Webster <[email protected]> > wrote: > >> Hey DuyHai, >> Thank you for your help. >> >> 1) Cassandra version >> [cqlsh 5.0.1 | Cassandra 3.5 | CQL spec 3.4.0 | Native protocol v4] >> >> >> 2) CREATE CUSTOM INDEX statement for your index >> >> CREATE CUSTOM INDEX objects_mime_idx ON test.objects (mime) USING >> 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {'analyzed' : >> 'true', 'analyzer_class' : >> 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer', >> 'tokenization_enable_stemming' : 'false', 'tokenization_locale' : 'en', >> 'tokenization_normalize_lowercase' : 'true', 'tokenization_skip_stop_words' >> : 'true'}; >> >> >> 3) Consistency level used for your SELECT >> I am using the default consistency >> cassandra@cqlsh> CONSISTENCY >> Current consistency level is ONE. >> >> >> 4) Replication factor >> >> CREATE KEYSPACE system_distributed WITH REPLICATION = { >> 'class' : 'org.apache.cassandra.locator.SimpleStrategy', >> 'replication_factor': '3' } >> AND DURABLE_WRITES = true; >> >> >> 5) Are you creating the index when the table is EMPTY or have you created >> the index when the table already contains some data ? >> I created the indexes after the tables contained data. >> >> >> On Tue, Aug 2, 2016 at 5:22 PM, DuyHai Doan <[email protected]> wrote: >> >>> Hello George >>> >>> Can you provide more details ? >>> >>> 1) Cassandra version >>> 2) CREATE CUSTOM INDEX statement for your index >>> 3) Consistency level used for your SELECT >>> 4) Replication factor >>> 5) Are you creating the index when the table is EMPTY or have you >>> created the index when the table already contains some data ? >>> >>> On Tue, Aug 2, 2016 at 4:05 PM, George Webster <[email protected]> >>> wrote: >>> >>>> Hey guys and gals, >>>> >>>> I am having a strange issue with Cassandra SASI and I was hoping you >>>> could help solve the mystery. My issue is inconsistency between returned >>>> results and strange log errors. >>>> >>>> The biggest issue is that when I perform a query I am getting back >>>> inconsistent results. First few times I received between 3 and 7 results >>>> and then I finally received 187 results. At no point in time did I change >>>> the query statement. However, after I received the 187 results, any on >>>> queries returned zero results. >>>> >>>> my query: >>>> SELECT * >>>> FROM test.objects >>>> WHERE mime LIKE 'ELF%'; >>>> >>>> When I look in the system.log file I see the following: >>>> WARN [SharedPool-Worker-1] 2016-08-02 15:58:53,256 >>>> SelectStatement.java:351 - Aggregation query used without partition key >>>> WARN [SharedPool-Worker-1] 2016-08-02 15:59:02,978 >>>> SelectStatement.java:351 - Aggregation query used without partition key >>>> >>>> >>>> When I look in the debug.log file I see the following when zero results >>>> are returned: >>>> WARN [SharedPool-Worker-1] 2016-08-02 15:58:53,256 >>>> SelectStatement.java:351 - Aggregation query used without partition key >>>> WARN [SharedPool-Worker-1] 2016-08-02 15:59:02,978 >>>> SelectStatement.java:351 - Aggregation query used without partition key >>>> >>>> Additionally, I see a lot of errors in the log that state: >>>> INFO [OptionalTasks:1] 2016-08-02 15:40:04,310 NoSpamLogger.java:91 - >>>> Maximum memory usage reached (536870912 bytes), cannot allocate chunk of >>>> 1048576 bytes >>>> INFO [OptionalTasks:1] 2016-08-02 15:55:04,387 NoSpamLogger.java:91 - >>>> Maximum memory usage reached (536870912 bytes), cannot allocate chunk of >>>> 1048576 bytes >>>> >>>> >>>> Any ideas? >>>> >>>> >>> >> >
