Hyukjin Kwon created SPARK-61171:
------------------------------------
Summary: Let the objects of the Python client be referred to
weakly and take attributes, as those of PySpark
Key: SPARK-61171
URL: https://issues.apache.org/jira/browse/SPARK-61171
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark's classes of Spark Connect are Python classes: their objects can be
referred to weakly (weakref.ref, WeakKeyDictionary, WeakValueDictionary) and
take attributes of their own, as libraries that cache by weak reference or tag
a DataFrame do.
The Python client's SparkSession, DataFrame, Column, GroupedData, WindowSpec,
DataFrameReader, DataFrameWriter, DataFrameWriterV2, MergeIntoWriter,
DataStreamReader, DataStreamWriter, StreamingQuery, StreamingQueryManager,
Catalog, RuntimeConf, DataFrameNaFunctions, DataFrameStatFunctions,
TableValuedFunction, TableArg, DataSourceRegistration, Observation, the data
types, StructField, VariantVal, Geometry, Geography and the resource profile
classes are classes of the Rust extension, which refuse both:
TypeError: cannot create weak reference to 'DataFrame' object
AttributeError: 'pyspark.sql.connect.dataframe.DataFrame' object has no
attribute 'x'
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
