use Spark streaming to receive topics from Kafka and process through some kind of rule engine based on incoming tickers.
Sound like a variation of Lambda architecture. You have the batch layer sorted out and now looking for speed layer with some form of dashboard. Something similar to below but much simplified [image: image.png] HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Tue, 9 Apr 2019 at 21:26, Nick Torenvliet <natorenvl...@gmail.com> wrote: > Hi all, > > Just looking for some general guidance. > > We have a kafka -> druid pipeline we intend to use in an industrial setting > to monitor process data. > > Our kafka system recieves messages on a single topic. > > The messages are {"timestamp": yy:mm:ddThh:mm:ss.mmm, "plant_equipment_id": > "id_string", "sensorvalue": float} > > For our POC there are about 2000 unique plant_equipment ids, this will > quickly grow to 20,000. > > The kafka topic streams into druid > > We are building some node.js/react browser based apps for analytics and > real time stream monitoring. > > We are thinking that for visualizing historical data sets we will hit druid > for data. > > For real time streaming we are wondering what our best option is. > > One option is to just hit druid semi regularly and update the on screen > visualization as data arrives from there. > > Another option is to stream subset of the topics (somehow) from kafka using > some streams interface. > > With all the stock ticker apps out there, I have to imagine this is a > really common use case. > > Anyone have any thoughts as to what we are best to do? > > Nick >
