Holden Karau created SPARK-60189:
------------------------------------
Summary: Validate reported category strings at construction
Key: SPARK-60189
URL: https://issues.apache.org/jira/browse/SPARK-60189
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 4.4.0
Reporter: Holden Karau
`optionMatchesTypes` matches four categories and `case _ => false` drops the
rest -- a variety reporting an unknown category is silently neutered (perf
bug, not correctness). `_variety_option` validates shape and length but not
vocabulary. Refuse unknown category strings loudly at construction, in the
same spirit as the branch's other refusals. Note the interplay with
SPARK-55219: if the vocabulary grows there, the accepted set should live
somewhere both sides agree on, and unknown strings should still fail at
construction rather than drop at matching.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
