[ 
https://issues.apache.org/jira/browse/SPARK-50854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913851#comment-17913851
 ] 

Vlad Rozov commented on SPARK-50854:
------------------------------------

Please assign it to me.

> Usage of relative path in `DataStreamWriter` writes parquet files in 
> unexpected location
> ----------------------------------------------------------------------------------------
>
>                 Key: SPARK-50854
>                 URL: https://issues.apache.org/jira/browse/SPARK-50854
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 3.5.4
>            Reporter: Vlad Rozov
>            Priority: Minor
>
> The following code writes {{_spark_metadata}} to a directory relative to the 
> spark driver, while parquet files are written to a directory relative to an 
> executor. Both should be written to a directory relative to the driver 
> working directory.
> {code:java}
> sparkSession.readStream()
>   .format("rate")
>   .load()
>   .writeStream()
>   .format("parquet")
>   .option("checkpointLocation", "checkpoints")
>   .trigger(AvailableNow())
>   .start("test.parquet")
>   .awaitTermination();
>  {code}
>  
> Please see https://lists.apache.org/thread/ffzwn1y2fgyjw0j09cv4np9z00wymxwv



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to