Re: JDBCInputFormat GC overhead limit exceeded error

@Max: If you're too busy right now to open a PR for this, I can also do it ... just let me know. On Wed, Jan 20, 2016 at 7:28 PM, Stephan Ewen wrote: > Super, thanks for finding this. Makes a lot of sense to have a result set > that does hold onto data. > > Would be great if you could open a pul

Re: JDBCInputFormat GC overhead limit exceeded error

Super, thanks for finding this. Makes a lot of sense to have a result set that does hold onto data. Would be great if you could open a pull request with this fix, as other users will benefit from that as well! Cheers, Stephan On Wed, Jan 20, 2016 at 6:03 PM, Maximilian Bode < maximilian.b...@tn

Re: JDBCInputFormat GC overhead limit exceeded error

Hi Stephan, thanks for your fast answer. Just setting the Flink-managed memory to a low value would not have worked for us, as we need joins etc. in the same job. After investigating the JDBCInputFormat, we found the line statement = dbConn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, Re

Re: JDBCInputFormat GC overhead limit exceeded error

Hi! This kind of error (GC overhead exceeded) usually means that the system is reaching a state where it has very many still living objects and frees little memory during each collection. As a consequence, it is basically busy with only garbage collection. Your job probably has about 500-600 MB o

Re: JDBCInputFormat GC overhead limit exceeded error

Hi Robert, I am using 0.10.1. > Am 19.01.2016 um 17:42 schrieb Robert Metzger : > > Hi Max, > > which version of Flink are you using? > > On Tue, Jan 19, 2016 at 5:35 PM, Maximilian Bode > wrote: > Hi everyone, > > I am facing a problem using the JDBCInput

Re: JDBCInputFormat GC overhead limit exceeded error

Hi Max, which version of Flink are you using? On Tue, Jan 19, 2016 at 5:35 PM, Maximilian Bode < maximilian.b...@tngtech.com> wrote: > Hi everyone, > > I am facing a problem using the JDBCInputFormat which occurred in a larger > Flink job. As a minimal example I can reproduce it when just writin

JDBCInputFormat GC overhead limit exceeded error

Hi everyone, I am facing a problem using the JDBCInputFormat which occurred in a larger Flink job. As a minimal example I can reproduce it when just writing data into a csv after having read it from a database, i.e. DataSet> existingData = env.createInput( JDBCInputFormat.buildJDBCInput