> On Sep 29, 2016, at 09:54, Ali Akhtar <ali.rac...@gmail.com> wrote: > > I'd appreciate some thoughts / suggestions on which of these alternatives I > should go with (e.g, using raw Kafka consumers vs Spark for ETL, which > persistent data store to use, and how to query that data store in the > backend of the web UI, for displaying the reports).
Hi Ali, I’m no expert in any of this, but I’m working on a project that is broadly similar to yours, and FWIW I’m evaluating Druid as the datastore which would host the queryable data and, well, actually handle and fulfill queries. Since Druid has built-in support for streaming ingestion from Kafka topics, I’m tentatively thinking of doing my ETL in a stream processing topology (I’m using Kafka Streams, FWIW), which would write the events destined for Druid into certain topics, from which Druid would ingest those events. HTH, Avi ———— Software Architect @ Park Assist We’re hiring! http://tech.parkassist.com/jobs/
