Hi, godfrey and Jark, thanks for joining the discussion.
The implications for FileSource
Sorry about that, I have missed an important feature that the FileSource
supports continuous reading. I think we can do the same thing for the
FileSource,
which is using the partition pruning function to f
Hi zoucao,
Regarding the pruning function, maybe a simple filter function is enough,
e.g. `FilterFunction partitionFilter`.
Besides, it would be better to state clearly how the new
`applyPartitionPuringFunction`
method affects batch/bounded table sources. From my understanding,
this method won't
Hi zoucao,
The FileSource does support streaming reading [1].
Best regards,
Martijn
[1]
https://nightlies.apache.org/flink/flink-docs-release-1.15/api/java/org/apache/flink/connector/file/src/FileSource.html
Op ma 4 jul. 2022 om 05:58 schreef godfrey he :
> Hi zoucao,
>
> Look forward your FL
Hi zoucao,
Look forward your FLIP.
>For Batch reading, the 'remainingPartitions' will be seen as the partitions
>needed to consume, for streaming reading, we use the
>'partitionPruningFunction' to ignore the unneeded partitions.
There should be for bounded source(maybe batch or streaming),
`apply
Hi Martijn, thanks for your attention, I'm glad to create a FLIP, and could
you help give me the permission?
My Id is zoucao, and my mail is zoucao...@gmail.com.
The implications for FileSource
In the above discussion, only HiveSource has been involved, because it
holds a continuous partition fet
Hi zoucao,
I think this topic deserves a proper FLIP and a vote. This approach is
focussed only on Hive, but I would also like to understand the implications
for FileSource. Can you create one?
Best regards,
Martijn
Op wo 22 jun. 2022 om 18:50 schreef cao zou :
> Hi devs, I want to start a dis
Hi devs, I want to start a discussion to find a way to support partition
pruning for streaming reading.
Now, Flink has supported the partition pruning, the implementation consists
of *Source Ability*, *Logical Rule*, and the interface
*SupportsPartitionPushDown*, but they all only take effect in
