Here is the code snippet:
var df = session.read().parquet(basepath);
for(Column partition : partitionColumnsList){
df = df.withColumn(partition.getName(),
df.col(partition.getName()).cast(partition.getType()));
}
Column is a class containing Schema Information, like for example the name of
the column and the data type of the column.
Best, Rico.
> Am 17.02.2022 um 03:17 schrieb Morven Huang <[email protected]>:
>
> Hi Rico, you have any code snippet? I have no problem casting int to string.
>
>> 2022年2月17日 上午12:26，Rico Bergmann <[email protected]> 写道：
>>
>> Hi!
>>
>> I am reading a partitioned dataFrame into spark using automatic type
>> inference for the partition columns. For one partition column the data
>> contains an integer, therefor Spark uses IntegerType for this column. In
>> general this is supposed to be a StringType column. So I tried to cast this
>> column to StringType. But this fails with AnalysisException “cannot cast int
>> to string”.
>>
>> Is this a bug? Or is it really not allowed to cast an int to a string?
>>
>> I’m using Spark 3.1.1
>>
>> Best regards
>>
>> Rico.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [email protected]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [email protected]
>
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]
