Good to hear it works.
Libraries, class-loading, and initialization seems to be one of the things
that remains tricky once one switches to distributed processed.
On Thu, Jun 25, 2015 at 10:58 AM, Flavio Pompermaier <pomperma...@okkam.it>
wrote:
> Sorry for the late response but I was on vacation the last 2 weeks..
>
> Calling Class.forName("com.mysql.jdbc.Driver") in the main() of my class
> made the things work!
>
> Thanks for the support,
> Flavio
>
> On Fri, Jun 5, 2015 at 11:13 PM, Stephan Ewen <se...@apache.org> wrote:
>
>> Can you manually load the driver class, with "Class.forName(...)", or
>> does that yield a "ClassNotFoundException" ?
>>
>> On Fri, Jun 5, 2015 at 11:10 PM, Flavio Pompermaier <pomperma...@okkam.it
>> > wrote:
>>
>>> in the fat jar
>>> On 5 Jun 2015 19:28, "Stephan Ewen" <se...@apache.org> wrote:
>>>
>>>> In which way is the driver in the classpath?
>>>>
>>>> - fat jar?
>>>> - in the nested /out folder in the slim jar?
>>>>
>>>>
>>>>
>>>> On Fri, Jun 5, 2015 at 7:23 PM, Flavio Pompermaier <
>>>> pomperma...@okkam.it> wrote:
>>>>
>>>>> Actually I just need to load it in the main method (job manager)
>>>>> before calling any flink operation, I retrieve the records in a mysql
>>>>> table
>>>>> because they contain the path of files I'll need to read. Nothing more
>>>>> nothing less
>>>>> On 5 Jun 2015 19:06, "Robert Metzger" <rmetz...@apache.org> wrote:
>>>>>
>>>>>> Sure.
>>>>>>
>>>>>> So the DriverManager has a static variable called
>>>>>> "registeredDrivers".
>>>>>> When DriverManager.getConnection() is called, the method is looking
>>>>>> up if an registered driver for that connection (in this case "mysql") is
>>>>>> available.
>>>>>>
>>>>>> For drivers to be in that list, they have to register themselves
>>>>>> using the DriverManager.registerDriver() method.
>>>>>>
>>>>>> Drivers can register themselves with a static constructor (which is
>>>>>> executed when Java loads a class):
>>>>>>
>>>>>> public class PoolingDriver implements Driver {
>>>>>> /** Register myself with the {@link DriverManager}. */
>>>>>> static {
>>>>>> try {
>>>>>> DriverManager.registerDriver(new PoolingDriver());
>>>>>> } catch(Exception e) {
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> To execute that driver registration, you need to do:
>>>>>> Class.forName("org.datanucleus.store.rdbms.datasource.dbcp.
>>>>>> PoolingDriver");
>>>>>> because then Java is loading the class and executing the static
>>>>>> constructor which is registering the driver at the connection manager.
>>>>>>
>>>>>> When executing Flink locally, you are using only one JVM. By calling
>>>>>> the MySQL driver manually in the main() method of your flink job, you are
>>>>>> registering the MySQL driver at the DriverManager of that JVM.
>>>>>>
>>>>>> However, when you run Flink in a distributed cluster, at the
>>>>>> TaskManager JVMs, the MySQL driver is not loaded at the DriverManager
>>>>>> there.
>>>>>> Therefore, you have to make sure that Class.forName("
>>>>>> org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver"); (this
>>>>>> is not the correct class for the MySQL driver)
>>>>>> has been called.
>>>>>> One approach to do that is to call Class.forName() in the open()
>>>>>> method of your function.
>>>>>>
>>>>>> Best,
>>>>>> Robert
>>>>>>
>>>>>>
>>>>>> On Fri, Jun 5, 2015 at 6:54 PM, Flavio Pompermaier <
>>>>>> pomperma...@okkam.it> wrote:
>>>>>>
>>>>>>> HI Robert,
>>>>>>> In the main method I connect to a mysql table that acts as a
>>>>>>> data-source repository that I use to know which dataset I need to load.
>>>>>>> All
>>>>>>> mysql classes are present in the shaded jar.
>>>>>>> Could you explain a little bit more in detail the solution to fix
>>>>>>> this problem please? Sorry but I didn't understand it :(
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Flavio
>>>>>>> On 5 Jun 2015 18:33, "Robert Metzger" <rmetz...@apache.org> wrote:
>>>>>>>
>>>>>>>> Hi Stefano,
>>>>>>>>
>>>>>>>> I doubt that there are conflicting dependencies because Flink does
>>>>>>>> not contain MySQL dependencies.
>>>>>>>> Are you using Flink's JDBCInputFormat or custom code?
>>>>>>>>
>>>>>>>> For drivers to register at java.sql's DriverManager, their classes
>>>>>>>> need to be loaded first. To load a class, you need to call
>>>>>>>> Class.forName("<classname>");
>>>>>>>>
>>>>>>>> Maybe you are loading the class in the application's main() method
>>>>>>>> (thats why it is working from eclipse) but not on the cluster instances
>>>>>>>> which are supposed to read the data.
>>>>>>>>
>>>>>>>> On Fri, Jun 5, 2015 at 5:16 PM, Stefano Bortoli <
>>>>>>>> s.bort...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Robert,
>>>>>>>>>
>>>>>>>>> I answer on behalf of Flavio. He told me the driver jar was
>>>>>>>>> included. Smells lik class-loading issue due to 'conflicting'
>>>>>>>>> dependencies. Is it possible?
>>>>>>>>>
>>>>>>>>> Saluti,
>>>>>>>>> Stefano
>>>>>>>>>
>>>>>>>>> 2015-06-05 16:24 GMT+02:00 Robert Metzger <rmetz...@apache.org>:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> is the MySQL driver part of the Jar file that you've build?
>>>>>>>>>>
>>>>>>>>>> On Fri, Jun 5, 2015 at 4:11 PM, Flavio Pompermaier <
>>>>>>>>>> pomperma...@okkam.it> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi to all,
>>>>>>>>>>>
>>>>>>>>>>> I'm using a fresh build of flink-0.9-SNAPSHOT and in my flink
>>>>>>>>>>> job I set up a mysql connection.
>>>>>>>>>>> When I run the job from Eclipse everything is fine,
>>>>>>>>>>> while when running the job from the Web UI I get the following
>>>>>>>>>>> exception:
>>>>>>>>>>>
>>>>>>>>>>> java.sql.SQLException: No suitable driver found for
>>>>>>>>>>> jdbc:mysql:/localhost:3306/mydb?autoReconnect=true
>>>>>>>>>>> at java.sql.DriverManager.getConnection(DriverManager.java:596)
>>>>>>>>>>> at java.sql.DriverManager.getConnection(DriverManager.java:215)
>>>>>>>>>>>
>>>>>>>>>>> How can I fix that?
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Flavio
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>
>
>
