Spark structured steaming from kafka - last message processed again after resume from checkpoint

Hi, I described my issue in full detail on http://stackoverflow.com/questions/41300223/spark-structured-steaming-from-kafka-last-message-processed-again-after-resume Any idea what's going wrong? Looking at the code base on https://github.com/apache/spark/blob/3f62e1b5d9e75dc07bac3aa4db3e8d0615cc

Re: Spark structured steaming from kafka - last message processed again after resume from checkpoint

Hi Niek, That's expected. Just answered on stackoverflow. On Sun, Dec 25, 2016 at 8:07 AM, Niek wrote: > Hi, > > I described my issue in full detail on > http://stackoverflow.com/questions/41300223/spark- > structured-steaming-from-kafka-last-message-processed-again-after-resume > > Any idea wh

Sharing data in columnar storage between two applications

Here is an interesting discussion to share data in columnar storage between two applications. https://github.com/apache/spark/pull/15219#issuecomment-265835049 One of the ideas is to prepare interfaces (or trait) only for read or write. Each application can implement only one class to want to do

Re: Sharing data in columnar storage between two applications

NOt so much about between applications, rather multiple frameworks within an application, but still related: https://cs.stanford.edu/~matei/papers/2017/cidr_weld.pdf On Sun, Dec 25, 2016 at 8:12 PM, Kazuaki Ishizaki wrote: > Here is an interesting discussion to share data in columnar storage > b