swuferhong commented on code in PR #20007:
URL: https://github.com/apache/flink/pull/20007#discussion_r905699266
##########
flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFileFormatFactory.java:
##########
@@ -136,6 +147,45 @@ public BulkFormat<RowData, FileSourceSplit>
createRuntimeDecoder(
public ChangelogMode getChangelogMode() {
return ChangelogMode.insertOnly();
}
+
+ @Override
+ public TableStats reportStatistics(List<Path> files, DataType
producedDataType) {
+ final int totalSampleLineCnt = 100;
+ try {
+ long totalSize = 0;
+ int sampledLineCnt = 0;
+ long sampledTotalSize = 0;
+ for (Path file : files) {
+ FileSystem fs = FileSystem.get(file.toUri());
+ FileStatus status = fs.getFileStatus(file);
+ totalSize += status.getLen();
+
+ // sample the line size
+ if (sampledLineCnt < totalSampleLineCnt) {
+ try (InputStreamReader isr =
+ new InputStreamReader(
+ Files.newInputStream(new
File(file.toUri()).toPath()))) {
+ BufferedReader br = new BufferedReader(isr);
+ String line;
+ while (sampledLineCnt < totalSampleLineCnt
+ && (line = br.readLine()) != null) {
+ sampledLineCnt += 1;
+ sampledTotalSize += line.length();
+ }
+ }
+ }
+ }
+ if (sampledTotalSize == 0) {
+ return TableStats.UNKNOWN;
+ }
+
+ int realSampledLineCnt = Math.min(totalSampleLineCnt,
sampledLineCnt);
+ int estimatedRowCount = (int) (totalSize * realSampledLineCnt
/ sampledTotalSize);
+ return new TableStats(estimatedRowCount);
Review Comment:
> Why the estimation doesn't give the column stat?
Not likes Orc and Parquet format, Csv format not support collect stats. The
only way to collect column stats is to build a csv reader and read all files
which is a heavy operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
