Re: Checkpointing in Spark Structured Streaming

2021-03-22 Thread Rohit Agrawal
queries trying to > update to the same checkpoint which might possibly mess up the checkpoint. > You need to make sure there's only one streaming query running against a > specific checkpoint. > > 1. https://issues.apache.org/jira/browse/SPARK-34383 > > On Tue, Mar 23, 2021

Checkpointing in Spark Structured Streaming

2021-03-22 Thread Rohit Agrawal
Hi, I have been experimenting with the Continuous mode and the Micro batch mode in Spark Structured Streaming. When enabling checkpoint to S3 instead of the local File System we see that Continuous mode has no change in latency (expected due to async checkpointing) however the Micro-batch mode exp