TheBuilderJR commented on issue #13270:
URL: https://github.com/apache/datafusion/issues/13270#issuecomment-2667008491
@zhuqi-lucas here's one current failure scenario with evolution:
https://github.com/apache/datafusion/issues/14755
--
This is an automated message from the Apache Git Ser
TheBuilderJR commented on issue #13270:
URL: https://github.com/apache/datafusion/issues/13270#issuecomment-2666309651
I just created an issue for this:
https://github.com/apache/datafusion/issues/14753
--
This is an automated message from the Apache Git Service.
To respond to the message
TheBuilderJR commented on issue #13270:
URL: https://github.com/apache/datafusion/issues/13270#issuecomment-2666296334
@zhuqi-lucas here's a concrete repro for what I expect datafusion to be able
to do but can't
```
use std::fs;
use std::sync::Arc;
use datafusion::prelude::*;
TheBuilderJR commented on issue #13270:
URL: https://github.com/apache/datafusion/issues/13270#issuecomment-2664700070
+1 I'm also blocked on this. It'd be nice if schema evolution could be a
first class citizen in datafusion. It's been pretty painful/stressful running
into schema evolution
zhuqi-lucas commented on issue #13270:
URL: https://github.com/apache/datafusion/issues/13270#issuecomment-2664731578
Just noticed we have a solution for partition evolution, see details PR, may
be we need some improvement based on it?
https://github.com/apache/datafusion/pull/12683/f
adriangb commented on issue #13270:
URL: https://github.com/apache/datafusion/issues/13270#issuecomment-2651549536
I think the fundamental issue is that the partition columns are specified on
a per-exec basis via `FileScanConfig`. The only solutions I can think of are:
- Change the APIs t
zhuqi-lucas commented on issue #13270:
URL: https://github.com/apache/datafusion/issues/13270#issuecomment-2650443328
First round investigation:
We need runtime to infer partitions result to overwrite the empty
FileScanConfig table_partition_cols, i can't find a good way until now.
zhuqi-lucas commented on issue #13270:
URL: https://github.com/apache/datafusion/issues/13270#issuecomment-2649754388
@adriangb Sorry for the delay, i am starting to investigate this issue this
week.
--
This is an automated message from the Apache Git Service.
To respond to the message,
logan-keede commented on issue #13270:
URL: https://github.com/apache/datafusion/issues/13270#issuecomment-2648393486
@adriangb my focus has been on refactoring `FileScanConfig` to move it out
of core. I cant say I understand the internals that much, but I will look into
it and mention it h
adriangb commented on issue #13270:
URL: https://github.com/apache/datafusion/issues/13270#issuecomment-2648246475
@logan-keede I see you're doing some work on `FileScanConfig`. Would it be
relevant to consider what needs to be changed to fix this?
--
This is an automated message from the
zhuqi-lucas commented on issue #13270:
URL: https://github.com/apache/datafusion/issues/13270#issuecomment-2552702754
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
T
alamb commented on issue #13270:
URL: https://github.com/apache/datafusion/issues/13270#issuecomment-2460247447
Thanks @adriangb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comm
adriangb commented on issue #13270:
URL: https://github.com/apache/datafusion/issues/13270#issuecomment-2458552653
cc @alamb I had promised you this a long time ago but only got around to it
now
--
This is an automated message from the Apache Git Service.
To respond to the message, please
adriangb opened a new issue, #13270:
URL: https://github.com/apache/datafusion/issues/13270
### Describe the bug
With CSV:
```shell
echo "a,b\n1,2" > data1.csv
mkdir a=2
echo "b\n3" > a=2/data2.csv
datafusion-cli
> SELECT * FROM '**/*.csv';
Arrow error: Csv err
14 matches
Mail list logo
