Fixing typo's Hi Tauzell,
Yeah our users want to query, do aggregations on Elastic Search directly and we cannot have inconsistent data because say the writes didn't make it into Cassandra but made it to Elastic search then a simple aggregations like count will lead to a wrong answer but again as @Hans pointed out this is no longer a Kafka question and also your solution has merits in its own way which I really appreciate it! your solution does make writes faster and probably some performance penalty on the read side given repairs happen during the read stage in Cassandra (We could check in both but since our users query elastic search directly there is no way for us to check it in Cassandra else we could go with your solution as well). Basically, we use ES as an index for Cassandra since secondary indexes in Cassandra (including the latest implementation SASI) doesn't work with our use case since we have high cardinality columns (which means every row in a column is unique so index on a high cardinality column is not very efficient given the underlying data structure used by SASI, but with inverted index which is used by ES is much faster). We do use Apache Spark along with Cassandra and I am trying to explore Succint http://succinct.cs.berkeley.edu/wp/wordpress/ and if everything works out with Succint we can get rid of elastic search. The only thing that I worry and still testing with Spark, Cassandra and Succint is whether If the aggregations/computations of a column or search on particular Cassandra field/column can happen in real time given a big dataset (with ES it does so the goal is to see if we can get somewhere close or perform even better). Thanks!
