alamb commented on issue #2326:
URL: https://github.com/apache/datafusion/issues/2326#issuecomment-2291105584
> Right now datafusion doesn't support struct evolution very well. Imagine
you have a struct named `customData` with field `someOptionEnabled` in one
parquet file, later down the line you add a new field `newAddedOption` to the
`customData` struct in another parquet file. Currently when you try and `SELECT
* FROM table` you'll get this error:
>
> ```
> {"message":"Failed to collect DataFrame batches: Plan(\"Cannot cast file
schema field customData of type Struct([Field { name:
\\\"someOptionEnabled\\\", data_type: Boolean, nullable: true, dict_id: 0,
dict_is_ordered: false, metadata: {} }]) to table schema field of type
Struct([Field { name: \\\"someOptionEnabled\\\", data_type: Boolean, nullable:
true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name:
\\\"newAddedOption\\\", data_type: Float64, nullable: true, dict_id: 0,
dict_is_ordered: false, metadata: {} }])\")","status":"error"}
> ```
>
> Feels like we should handle this more gracefully. cc @alamb
I agree
> I'm happy to make contributions if someone can point me to the right
places to look.
My suggestion is to start with filing a ticket with a self contained
reproducer (either rust code or SQL) that shows what you are trying to do.
This would likely become part of the test of any code improvement we make,
as well as providing some more detail for other contributors to help point to
the right place in the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
