[
https://issues.apache.org/jira/browse/FLINK-9814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549347#comment-16549347
]
François Lacombe edited comment on FLINK-9814 at 7/19/18 3:19 PM:
------------------------------------------------------------------
Hi Fabian,
Ok to add option to check the header.
Regarding the overhead, can't it be done prior to split the file and give it to
workers?
I think about reading the first line of the file with a dedicated file system
access in the source class at its creation maybe.
This would be ok for many sources, since headers often contains columns and
types descriptors and prevent doing a lot of work with bad formatted files
All the best
was (Author: flacombe):
Hi Fabian,
Ok to add option to check the header.
Regarding the overhead, can't it be done prior to split the file and give it
workers?
I think about reading the first line of the file with a dedicated file system
access in the source class at the source creation maybe.
This would be ok for many sources, since headers often contains columns and
types descriptors and prevent doing a lot of work with bad formatted files
All the best
> CsvTableSource "lack of column" warning
> ---------------------------------------
>
> Key: FLINK-9814
> URL: https://issues.apache.org/jira/browse/FLINK-9814
> Project: Flink
> Issue Type: Wish
> Components: Table API & SQL
> Affects Versions: 1.5.0
> Reporter: François Lacombe
> Assignee: vinoyang
> Priority: Minor
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The CsvTableSource class is built by defining expected columns to be find in
> the corresponding csv file.
>
> It would be great to throw an Exception when the csv file doesn't have the
> same structure as defined in the source. For retro-compatibility sake,
> developers should explicitly set the builder to define columns stricly and
> expect Exception to be thrown in case of structure difference.
> It can be easilly checked with file header if it exists.
> Is this possible ?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
