Re: Flink JDBC connector behavior

2022-02-08 Thread Roman Khachatryan
Hi, scan.partition.num (the number of partitions [1]) translates into parallel queries to the database (with different to/from). Batch size is further calculated from lower and upper bounds and the number of partitions. scan.fetch-size hints JDBC driver to adjust the fetch size (see [2]). The fi

Flink JDBC connector behavior

2022-02-08 Thread Qihua Yang
Hi, We are using flink jdbc connector to read whole database table line by line. A few things I don't quite understand. We configured BATCH_SIZE=100 and PARTITION_NUM=1000, table is pretty big. What is the flink internal behavior to read data from table? Flink read BATCH_SIZE data each time? Or it