comphead commented on issue #15162:
URL: https://github.com/apache/datafusion/issues/15162#issuecomment-2714869946
Thanks @tustvold the requirement is to customize hardcoded inner field for
ListType which is hardcoded `item` now.
So DF and arrow-rs creates data arrays for ListType
```
List(Field { name: "item", data_type: Int8, nullable: true, dict_id: 0,
dict_is_ordered: false, metadata:
{} })
```
Comet using the physical plan directly, there is no coercion phase and
schema from Apache Spark for the same comes as
```
List(Field { name: "element", data_type: Int8, nullable: true, dict_id: 0,
dict_is_ordered: false, metadata:
{} })
```
When RecordBatch created
```
https://github.com/apache/arrow-rs/blob/f4fde769ab6e1a9b75f890b7f8b47bc22800830b/arrow-array/src/record_batch.rs#L331
```
it checks both schema(easy to modify) and column arrays schema and the error
is thrown if both schemas doesn't match, in this specific case it doesn't match
by inner list field name `item` vs `element`. Do you see any solution on
Arrow-rs side to let downstream projects to redefine `LIST_FIELD_DEFAULT_NAME`
value to the custom one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
