As you may be aware the granularity that Spark streaming has is micro-batching and that is limited to 0.5 second. So if you have continuous ingestion of data then Spark streaming may not be granular enough for CEP. You may consider other products.
Worth looking at this old thread on mine "Spark support for Complex Event Processing (CEP) https://mail-archives.apache.org/mod_mbox/spark-user/201604.mbox/%3CCAJ3fcbB8eaf0JV84bA7XGUK5GajC1yGT3ZgTNCi8arJg56=l...@mail.gmail.com%3E HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 18 October 2017 at 20:52, anna stax <annasta...@gmail.com> wrote: > Hello all, > > Has anyone used spark streaming for CEP (Complex Event processing). Any > CEP libraries that works well with spark. I have a use case for CEP and > trying to see if spark streaming is a good fit. > > Currently we have a data pipeline using Kafka, Spark streaming and > Cassandra for data ingestion and near real time dashboard. > > Please share your experience. > Thanks much. > -Anna > > >
