Hi! Each parallelism of the lookup operation will load all data from the lookup table source, so you're loading 10GB of data to each parallelism and storing them in JVM memory. That is not only slow but also very memory-consuming.
Have you tried joining your main stream with the hive table directly (that is, using streaming joins instead of lookup joins)? Does that meet your need or why do you have to use lookup joins? Jason Yi <93t...@gmail.com> 于2022年2月5日周六 08:01写道： > Hello, > > I created external tables on Hive with data in s3 and wanted to use those > tables as a lookup table in Flink. > > When I used an external table containing a small size of data as a lookup > table, Flink quickly loaded the data into TM memory and did a Temporal join > to an event stream. But, when I put an external table containing ~10GB of > data, Flink took so long to load the data and finally returned a timeout > error. (I set the heartbeat.timeout to 200000) > > Is there a way to make Flink read Hive data faster? Or is this normal? > MySQL lookup tables would be recommended when we have a large size of > dimension data? > > Here's the test environment: > - 1.14.0 Flink > - EMR 6.5 > - Hive 3.1.2 installed on EMR > - Hive with a default MetaStore on EMR used. (Not MySQL or Glue Metastore) > - Parquet source data in s3 for the external table on Hive > > Below is part of the Flink log produced while loading the Hive table data. > Flink seemed to open one parquet file multiple times and moved to another > parquet file to open. I wonder if this is normal. Why Flink didn't read > data from multiple files in parallel. I'm not sure if this is a problem > caused by the default Hive Metastore. > > ...... > 2022-02-04 22:42:54,839 INFO > org.apache.flink.table.filesystem.FileSystemLookupFunction [] - > Populating lookup join cache > 2022-02-04 22:42:54,839 INFO > org.apache.flink.table.filesystem.FileSystemLookupFunction [] - > Populating lookup join cache > 2022-02-04 22:42:55,083 INFO org.apache.hadoop.mapred.FileInputFormat > [] - Total input files to process : 12 > 2022-02-04 22:42:55,084 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:42:55,096 INFO org.apache.hadoop.mapred.FileInputFormat > [] - Total input files to process : 12 > 2022-02-04 22:42:55,097 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:42:55,105 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:42:55,116 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:42:55,169 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:42:55,172 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:42:57,782 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:42:57,783 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:42:57,799 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:42:57,801 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:42:57,851 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:42:57,851 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:42:57,853 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:42:57,897 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:42:57,898 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:42:57,899 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:42:57,908 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:42:57,950 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:01,581 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:01,582 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:01,592 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:01,594 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:01,678 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:01,679 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:01,680 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:01,682 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:01,682 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:01,684 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:01,727 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:01,732 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:03,210 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:03,211 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:03,268 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:03,268 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:03,269 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:03,313 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:03,315 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:03,316 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:03,377 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:03,377 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:03,378 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:03,427 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:07,845 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:07,846 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:07,908 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:07,909 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:07,910 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:07,942 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:07,943 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:07,999 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:08,002 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:08,003 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:08,004 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:08,053 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:10,404 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:10,406 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:11,908 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:11,910 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:11,945 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:11,945 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:11,946 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:11,963 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:11,964 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:11,964 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:11,996 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:12,013 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:15,101 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:15,102 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:15,117 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:15,118 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:15,168 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:15,175 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:18,337 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:18,338 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:18,410 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:18,467 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:18,468 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:18,523 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00000-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:18,651 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:18,676 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00001-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:18,722 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00001-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:18,778 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:18,779 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00001-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:18,835 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00001-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:19,903 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:19,904 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00001-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:19,952 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00001-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:19,952 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:19,953 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00001-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:19,996 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00001-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:23,308 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:23,309 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00001-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:23,363 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00001-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:23,364 INFO > org.apache.flink.connectors.hive.read.HiveTableInputFormat [] - Use > flink parquet ColumnarRowData reader. > 2022-02-04 22:43:23,364 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00001-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > 2022-02-04 22:43:23,406 INFO > com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem [] - Opening > 's3://bucket/path/to/files/part-00001-55f0ff62-bf83-4eac-8ce8-308bd9efda24-c000.snappy.parquet' > for reading > ...... >
