Hi! Currently when creating DataFrame, we cannot use a class that extends StructType as schema. This is due to the type_verifier using `type(obj) == StructType`. If this check is replaced with a `isinstance`, the extended types can be supported.
Here is a small test that fails currently: > import unittest > from pyspark.sql.types import StructType, _make_type_verifier > > class ExtendedStructType(StructType): ... > > class SampleTest(unittest.TestCase): > def test_extended_struct_type(self): > schema = ExtendedStructType([]) > _make_type_verifier(schema)([]) Is this something that should get fixed? If so, I have created a PR for same, but this does not have a jira ticket assigned yet: https://github.com/apache/spark/pull/50726. PS: I have sent this email twice to d...@spark.apache.org, requested to create a JIRA account so I could create a bug for same. Thanks! Vaibhaw --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
