[
https://issues.apache.org/jira/browse/SPARK-61172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61172:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Add _set_opts to the readers and writers, which pandas-on-Spark's to_csv and
> to_json call, in the Python client as PySpark has it
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61172
> URL: https://issues.apache.org/jira/browse/SPARK-61172
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> PySpark Connect's DataFrameReader, DataFrameWriter, DataFrameWriterV2 and
> DataStreamReader have _set_opts(schema=None, **options) of OptionUtils: it
> sets the schema, if any, and the options that are not None, through schema
> and option. pandas-on-Spark's DataFrame.to_csv and to_json with a path call
> it on the writer of the DataFrame (builder._set_opts(sep=..., header=...,
> ...)).
> The Python client's readers and writers have no _set_opts:
> AttributeError: 'pyspark.sql.readwriter.DataFrameWriter' object has no
> attribute '_set_opts'
> Seven tests of Apache Spark's pandas-on-Spark io/test_parity_csv.py and
> io/test_parity_dataframe_conversion.py (to_csv and to_json with a path or
> partition columns) fail on this client.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
