Hi,
@Jingsong, I agree that adding a new feature (ParquetAvroInputFormat) to
an old source API is a maintenance burden. But IMHO I think that while
the new DataStream batch/streaming convergent API is not 100% functional
we still need to maintain older sources and add missing features to them.
Indeed, I realized that DataStream API in batch mode (1) does not
support aggregations yet (2) so in such a case a user would stick to the
DataSet API. And the new FileSource API with
ParquetColumnarRowInputFormat is not available in DataSet API (3).
So, long story short, in some cases a user will have no other choice
than using ParquetInputFormat and legacy source.
WDYT ?
[1] https://issues.apache.org/jira/browse/FLINK-19316
[2] https://issues.apache.org/jira/browse/FLINK-22587
[3]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface#FLIP27:RefactorSourceInterface-Compatibility,Deprecation,andMigrationPlan
Best,
Etienne
On 24/02/2021 03:35, Jingsong Li wrote:
Hi Etienne,
Thanks for your reporting.
There are indeed many problems. There is no doubt that we need to improve
our current format implementation.
But ParquetTableSource and ParquetInputFormat are legacy implementations
with legacy interfaces. We have introduced new interfaces for execution and
SQL. You can see:
- ParquetColumnarRowInputFormat with BulkFormat interface. It is just for
columnar row reading, not support complex types, we need
migrate ParquetInputFormat to the new BulkFormat interface.
- FileSystemTableSource with DynamicTableSource interface, It is a generic
FileSystem source for all formats, we can just use it for parquet too.
Considering ParquetTableSource and ParquetInputFormat are legacy
interfaces, I think we can finish migration work first, what do you think?
Best,
Jingsong
On Wed, Feb 24, 2021 at 12:46 AM Etienne Chauchot <echauc...@apache.org>
wrote:
Hi all,
I've been playing with Parquet with SQL and Avro lately. I've found some
bugs:
1. https://issues.apache.org/jira/browse/FLINK-21388 : I already
submitted a PR on this one (https://github.com/apache/flink/pull/14961)
2. https://issues.apache.org/jira/browse/FLINK-21389
3. https://issues.apache.org/jira/browse/FLINK-21468
I've already started to work on this ticket:
https://issues.apache.org/jira/browse/FLINK-21393
I'd be happy to receive your comments on these tickets
Best
Etienne Chauchot
