Hyukjin Kwon created SPARK-61178:
------------------------------------
Summary: Make StructType.fields the list of the struct, and the
name, dataType and nullable of StructField assignable, so that their edits show
in the struct, in the Python client as PySpark does
Key: SPARK-61178
URL: https://issues.apache.org/jira/browse/SPARK-61178
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark's StructType holds its fields in a list, fields, which it gives back
each time: edits of it, such as st.fields.append(f), st.fields[0] = f,
st.fields.pop() and st.fields.sort(...), change the struct, and a list kept
before st.add(...) sees the new field. The name, dataType and nullable of a
StructField are plain attributes, so f.nullable = False changes the field and
the struct it is in (a common way to change the nullability of a schema).
The Python client's StructType.fields gives a new list on each access, so these
edits are lost, and StructField's name, dataType and nullable cannot be set:
AttributeError: attribute 'nullable' of 'pyspark.sql.types.StructField' objects
is not writable
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
