Hyukjin Kwon created SPARK-61177:
------------------------------------
Summary: Quote the field names of DataType::to_ddl, and keep the
case of the collation of DataType::from_ddl, in the Rust client
Key: SPARK-61177
URL: https://issues.apache.org/jira/browse/SPARK-61177
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Two defects of the DDL of the Rust client's DataType:
1. DataType::to_ddl writes the name of each field as it is, with format!("{}
{}", name, type): a field "my col" gives "my col int", which Spark cannot
parse, and a backtick in a name is not escaped. Spark's StructType.toDDL quotes
the names that need it ("`my col` INT"), and so do nested struct fields.
2. DataType::from_ddl lowercases its input before it reads the collation of a
string, so "a STRING COLLATE UTF8_LCASE" gives the collation "utf8_lcase",
where Spark gives UTF8_LCASE. The json() of the type and its equality with the
types of the server then differ.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
