Yicong Huang created SPARK-56165:
------------------------------------
Summary: Use temporary directory instead of saveAsTable in
test_data_source_segfault
Key: SPARK-56165
URL: https://issues.apache.org/jira/browse/SPARK-56165
Project: Spark
Issue Type: Improvement
Components: PySpark, Tests
Affects Versions: 4.2.0
Reporter: Yicong Huang
In `test_data_source_segfault`, the two write/commit subtests use
`saveAsTable("test_table")` to exercise the Python worker write and commit code
paths. This can fail when the catalog rejects the custom data source format.
We should replace `saveAsTable("test_table")` with `.save(tempdir)` using
`tempfile.TemporaryDirectory`, which avoids catalog format validation while
still exercising the same Python worker write/commit code paths. This is also
consistent with how other write tests in the same file are structured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
