Hi Fabian, I opened the following issue to track the improvement proposed: https://issues.apache.org/jira/browse/FLINK-12198
Best, Konstantinos From: Papadopoulos, Konstantinos <konstantinos.papadopou...@iriworldwide.com> Sent: Δευτέρα, 15 Απριλίου 2019 12:30 μμ To: Fabian Hueske <fhue...@gmail.com> Cc: Rong Rong <walter...@gmail.com>; user <user@flink.apache.org> Subject: RE: Flink JDBC: Disable auto-commit mode Hi Fabian, Glad to hear that you agree for such an improvement. Of course, I can handle it. Best, Konstantinos From: Fabian Hueske <fhue...@gmail.com<mailto:fhue...@gmail.com>> Sent: Δευτέρα, 15 Απριλίου 2019 11:56 πμ To: Papadopoulos, Konstantinos <konstantinos.papadopou...@iriworldwide.com<mailto:konstantinos.papadopou...@iriworldwide.com>> Cc: Rong Rong <walter...@gmail.com<mailto:walter...@gmail.com>>; user <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: Flink JDBC: Disable auto-commit mode Hi Konstantinos, This sounds like a useful extension to me. Would you like to create a Jira issue and contribute the improvement? In the meantime, you can just fork the code of JDBCInputFormat and adjust it to your needs. Best, Fabian Am Mo., 15. Apr. 2019 um 08:53 Uhr schrieb Papadopoulos, Konstantinos <konstantinos.papadopou...@iriworldwide.com<mailto:konstantinos.papadopou...@iriworldwide.com>>: Hi Rong, We have already tried to set the fetch size with no success. According to PG documentation we have to set both configuration parameters (i.e., auto-commit to false and limit fetch) to achieve our purpose. Thanks, Konstantinos From: Rong Rong <walter...@gmail.com<mailto:walter...@gmail.com>> Sent: Παρασκευή, 12 Απριλίου 2019 6:50 μμ To: Papadopoulos, Konstantinos <konstantinos.papadopou...@iriworldwide.com<mailto:konstantinos.papadopou...@iriworldwide.com>> Cc: user <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: Flink JDBC: Disable auto-commit mode Hi Konstantinos, Seems like setting for auto commit is not directly possible in the current JDBCInputFormatBuilder. However there's a way to specify the fetch size [1] for your DB round-trip, doesn't that resolve your issue? Similarly in JDBCOutputFormat, a batching mode was also used to stash upload rows before flushing to DB. -- Rong [1] https://docs.oracle.com/cd/E18283_01/java.112/e16548/resltset.htm#insertedID4<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.oracle.com%2Fcd%2FE18283_01%2Fjava.112%2Fe16548%2Fresltset.htm%23insertedID4&data=02%7C01%7C%7C2cc5777737b740b23dcd08d6c185070b%7C43728c2044474b27ac2e4bdabb3c0121%7C0%7C0%7C636909174427110104&sdata=vHwPnVT%2BI41Xxkp1Zfl%2BOgTReZ0ILL5RkhDez72jJvM%3D&reserved=0> On Fri, Apr 12, 2019 at 6:23 AM Papadopoulos, Konstantinos <konstantinos.papadopou...@iriworldwide.com<mailto:konstantinos.papadopou...@iriworldwide.com>> wrote: Hi all, We are facing an issue when trying to integrate PostgreSQL with Flink JDBC. When you establish a connection to the PostgreSQL database, it is in auto-commit mode. It means that each SQL statement is treated as a transaction and is automatically committed, but this functionality results in unexpected behavior (e.g., out-of-memory errors) when executed for large result sets. In order to bypass such issues, we must disable the auto-commit mode. To do this, in a simple Java application, we call the setAutoCommit() method of the Connection object. So, my question is: How can we achieve this by using JDBCInputFormat of Flink? Thanks in advance, Konstantinos
