Hi, I have 2spark structure streaming queries writing to the same outpath in object storage. Once in a while I am getting the "IllegalStateException: Race while writing batch 4". I found that this error is because there are two writers writing to the output path. The file streaming sink doesn't support multiple writers. It assumes there is only one writer writing to the path. Each query needs to use its own output directory.
Is there a way to write the output to the same path by both queries, as I need the output at the same path.? Regards Amit Joshi
