Machine Learning with window data

Hi, I have a use case where I like to analyze windows of sensordata. Currently I have a working case where I use Structured Streaming to process real-time streams of sensordata. Now I like to analyse windows of sensordata and use classification to predict the class of a whole window. For instanc

Re: [STRUCTURED STREAM] Join static dataset in state function (flatMapGroupsWithState)

onsole. BTW: I now implemented the approach to join this datasource with the streaming source before feeding it the state function. That works! But I am still curious how to do this in flatmapgroupswithstate? Or that the state functions have not been designed to do such things… Regards,

[Structured streaming] Merging streaming with semi-static datasets

Hi, I’m currently doing some tests with Structured Streaming and I’m wondering how I can merge the streaming dataset with a more-or-less static dataset (from a JDBC source). With more-or-less I mean a dataset which does not change that often and could be cached by Spark for a while. It is possi

Re: [Spark structured streaming] Use of (flat)mapgroupswithstate takes long time

does the cluster have? The cluster is small and has 2 workers, both using 1 core. I’m wondering how Spark determines the number of cores when using Docker (a single host with multiple spark containers). Regards, Chris From: Tathagata Das Date: Tuesday, 23 January 2018 at 00:04 To: Christiaa