[ https://issues.apache.org/jira/browse/SPARK-50854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913851#comment-17913851 ]
Vlad Rozov commented on SPARK-50854: ------------------------------------ Please assign it to me. > Usage of relative path in `DataStreamWriter` writes parquet files in > unexpected location > ---------------------------------------------------------------------------------------- > > Key: SPARK-50854 > URL: https://issues.apache.org/jira/browse/SPARK-50854 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 3.5.4 > Reporter: Vlad Rozov > Priority: Minor > > The following code writes {{_spark_metadata}} to a directory relative to the > spark driver, while parquet files are written to a directory relative to an > executor. Both should be written to a directory relative to the driver > working directory. > {code:java} > sparkSession.readStream() > .format("rate") > .load() > .writeStream() > .format("parquet") > .option("checkpointLocation", "checkpoints") > .trigger(AvailableNow()) > .start("test.parquet") > .awaitTermination(); > {code} > > Please see https://lists.apache.org/thread/ffzwn1y2fgyjw0j09cv4np9z00wymxwv -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org