Best to qualify your thoughts with an example
By using the foreachBatch function combined with the update output mode in
Spark Structured Streaming, you can effectively handle and integrate
late-arriving data into your aggregations. This approach will allow you to
continuously update your aggregat
Hi Team,
Hope you all are doing well. I have run into a use case in which I want to
do the aggregation in foreachbatch and use update mode for handling late
data in structured streaming. Will this approach work in effectively
capturing late arriving data in the aggregations? Please help.
Thank
Hi All,
1. Is there a way to do partial sort of say (timestamp column) in update
mode? I am currently using Spark 2.1.1 and its looks like it is not
possible however I am wondering if this possible in 2.2?
2. can we do full sort in update mode with a specified watermark? since
after a specified wa
