Apache Kafka and Flink: Stateful Streaming Data Pipelines made easy with SQL

Hello, I would like to encourage you to participate in the free webinars in the field of real-time streaming, which will take place next Tuesday, March 15th. One of the speakers Francesco Tisiot <https://www.linkedin.com/> will talk about *Apache Kafka and Flink: Stateful Streaming Data Pip

Re: Streaming Data

Hi All, thanks for the recommendations. I really like staying close to the Apache.org product line - so I will likely go the route that puts kafka streams or spark into play. On Wed, Apr 10, 2019 at 11:45 AM Robin Moffatt wrote: > +1 for looking into Kafka Streams. You can build and maintain st

Re: Streaming Data

+1 for looking into Kafka Streams. You can build and maintain state within your app, and expose it on a REST endpoint for your node/react app to query. There's an example of this here: https://github.com/confluentinc/kafka-streams-examples/blob/5.2.1-post/src/main/java/io/confluent/examples/streams

Re: Streaming Data

Hi Nick, You could give Kafka Streams a try for your use case. It is included in Kafka, so probably you already have it! For details, see https://kafka.apache.org/documentation/streams/ . KSQL, which has already been mentioned by another member of the mailing list uses Kafka Streams under the ho

Re: Streaming Data

For more flexibility without the need for extensive coding I suggest Esper complex event processing (http://www.espertech.com/esper) -R Sent: Tuesday, April 09, 2019 at 4:26 PM From: "Nick Torenvliet" To: users@kafka.apache.org Subject: Streaming Data Hi all, Just looking for so

Re: Streaming Data

Nick, Have you looked at Apache Flink? It’s got very powerful API’s and you can stream aggregations, filters, etc right to druid and also it has very robust state management that might be a good fit for your use case. https://flink.apache.org/ https://github.com/druid-io/tranquility Thanks, Ke

Re: Streaming Data

use Spark streaming to receive topics from Kafka and process through some kind of rule engine based on incoming tickers. Sound like a variation of Lambda architecture. You have the batch layer sorted out and now looking for speed layer with some form of dashboard. Something similar to below but m

Re: Streaming Data

I would stream to influxdb and visualize with grafana. Works great for dashboards. But I would rethink your line format. It's very convenient to have tags (or labels) that are key/value pair on each metric if you ever want to aggregate over a group of similar metrics. Svante

Re: Streaming Data

Hi Nick, Have you looked into KSQL? Kind regards, Liam Clarke On Wed, 10 Apr. 2019, 8:26 am Nick Torenvliet, wrote: > Hi all, > > Just looking for some general guidance. > > We have a kafka -> druid pipeline we intend to use in an industrial setting > to monitor process data. > > Our kafka sy

Streaming Data

Hi all, Just looking for some general guidance. We have a kafka -> druid pipeline we intend to use in an industrial setting to monitor process data. Our kafka system recieves messages on a single topic. The messages are {"timestamp": yy:mm:ddThh:mm:ss.mmm, "plant_equipment_id": "id_string", "se

Loading and streaming data from Kafka to BigQuery

Hi guys, I would like to recommend the following post, discussing and comparing techniques for loading data from Kafka to BigQuery. https://medium.com/myheritage-engineering/kafka-to-bigquery-load-a-guide-for-streaming-billions-of-daily-events-cbbf31f4b737 Feedback is welcome. *Ofir Sharony* Ba