Hello Community, I need to delete old src data created by Spark Structured Streaming. Just deleting relevant folder throws an exception while reading batch dataframe from file-system:
java.io.FileNotFoundException: File file:/data/avro/year=2020/month=3/day=13/hour=12/part-00000-0cc84e65-3f49-4686-85e3-1ecf48952794.c000.avro does not exist Issue is actualy the same that described here: https://stackoverflow.com/questions/60773445/how-to-delete-old-data-that-was-created-by-spark-structured-streaming?newreg=5cc791c48358491c88d9b2dae1e436d9 Didn't find a way to delete it via Spark API. Are there any solutions to do it via API instead of editing metadata manually? Your help would be appreciated.
