[ https://issues.apache.org/jira/browse/SPARK-50839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh Motwani updated SPARK-50839: ---------------------------------- Description: In classic Spark shell (PySpark): {code:java} >>> spark.createDataFrame([1], "v int").show() Traceback (most recent call last): File "<stdin>", line 1, in <module> ... pyspark.errors.exceptions.base.PySparkTypeError: [FIELD_DATA_TYPE_UNACCEPTABLE] StructType([StructField('v', IntegerType(), True)]) can not accept object 1 in type <class 'int'>. {code} In Spark Connect shell (PySpark): {code:java} >>> spark.createDataFrame([1], "v int").show() 25/01/15 14:05:01 WARN CheckAllocator: More than one DefaultAllocationManager on classpath. Choosing first found +---+ | v| +---+ | 1| +---+ {code} was: In classic Spark shell (PySpark): ``` >>> spark.createDataFrame([1], "v int").show() Traceback (most recent call last): File "<stdin>", line 1, in <module> ... pyspark.errors.exceptions.base.PySparkTypeError: [FIELD_DATA_TYPE_UNACCEPTABLE] StructType([StructField('v', IntegerType(), True)]) can not accept object 1 in type <class 'int'>. ``` In Spark Connect shell (PySpark): ``` >>> spark.createDataFrame([1], "v int").show() 25/01/15 14:05:01 WARN CheckAllocator: More than one DefaultAllocationManager on classpath. Choosing first found +---+ | v| +---+ | 1| +---+ ``` > Spark Connect createDataFrame behavior does not match with classic Spark > ------------------------------------------------------------------------ > > Key: SPARK-50839 > URL: https://issues.apache.org/jira/browse/SPARK-50839 > Project: Spark > Issue Type: Bug > Components: Connect, PySpark > Affects Versions: 4.0.0 > Reporter: Harsh Motwani > Priority: Major > > In classic Spark shell (PySpark): > {code:java} > >>> spark.createDataFrame([1], "v int").show() > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > ... > pyspark.errors.exceptions.base.PySparkTypeError: > [FIELD_DATA_TYPE_UNACCEPTABLE] StructType([StructField('v', IntegerType(), > True)]) can not accept object 1 in type <class 'int'>. > {code} > In Spark Connect shell (PySpark): > {code:java} > >>> spark.createDataFrame([1], "v int").show() > 25/01/15 14:05:01 WARN CheckAllocator: More than one DefaultAllocationManager > on classpath. Choosing first found > +---+ > | v| > +---+ > | 1| > +---+ > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org