Yen-Hsiang, Wang created SPARK-60142:
----------------------------------------
Summary: Add unit tests for vector field type in
_SparkPartitionTorchDataset._get_field_converter
Key: SPARK-60142
URL: https://issues.apache.org/jira/browse/SPARK-60142
Project: Spark
Issue Type: Test
Components: ML
Affects Versions: 4.4.0
Reporter: Yen-Hsiang, Wang
Problem
_SparkPartitionTorchDataset._get_field_converter in
python/pyspark/ml/torch/data.py handles three branches: vector, numeric/array
types, and
unsupported types (raises ValueError). The vector branch — which converts
both dense and sparse MLlib vectors from their Arrow/pandas dict
representation — is currently not covered by any test.
The only existing test (TorchDistributorDataLoaderUnitTests.test_data_loader)
is an integration test that exercises the full TorchDistributor
pipeline and has been skipped (@unittest.skip) since SPARK-50864 due to
slowness, leaving the vector converter logic without any automated
coverage.
What
Add lightweight unit tests directly against
_SparkPartitionTorchDataset._get_field_converter for the vector field type,
covering:
- Dense vector (type == 1)
- Sparse vector (type == 0) with non-empty indices
- Sparse vector with empty indices (edge case)
Since _get_field_converter is a @staticmethod and the vector converter only
depends on NumPy, no SparkSession is needed — the tests run fast
and in isolation.
Why
The vector branch contains non-trivial logic (sparse → dense expansion via
index assignment). A bug there would silently produce wrong training
data without any test catching it.
---
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
