Why do the JDBC jars need to be on the job manager node though? On Mon, May 2, 2022 at 9:36 AM Chesnay Schepler <ches...@apache.org> wrote:
> yes. > But if you can ensure that the driver isn't bundled by any user-jar you > can also skip the pattern configuration step. > > The pattern looks correct formatting-wise; you could try whether > com.microsoft.sqlserver.jdbc. is enough to solve the issue. > > On 02/05/2022 14:41, John Smith wrote: > > Oh, so I should copy the jars to the lib folder and > set classloader.parent-first-patterns.additional: > "org.apache.ignite.;com.microsoft.sqlserver.jdbc." to both the task > managers and job managers? > > Also is my pattern correct? > "org.apache.ignite.;com.microsoft.sqlserver.jdbc." > > Just to be sure I'm running a standalone cluster using zookeeper. So I > have 3 zookeepers, 3 job managers and 3 task managers. > > > On Mon, May 2, 2022 at 2:57 AM Chesnay Schepler <ches...@apache.org> > wrote: > >> And you do should make sure that it is set for both processes! >> >> On 02/05/2022 08:43, Chesnay Schepler wrote: >> >> The setting itself isn't taskmanager specific; it applies to both the >> job- and taskmanager process. >> >> On 02/05/2022 05:29, John Smith wrote: >> >> Also just to be sure this is a Task Manager setting right? >> >> On Thu, Apr 28, 2022 at 11:13 AM John Smith <java.dev....@gmail.com> >> wrote: >> >>> I assume you will take action on your side to track and fix the doc? :) >>> >>> On Thu, Apr 28, 2022 at 11:12 AM John Smith <java.dev....@gmail.com> >>> wrote: >>> >>>> Ok so to summarize... >>>> >>>> - Build my job jar and have the JDBC driver as a compile only >>>> dependency and copy the JDBC driver to flink lib folder. >>>> >>>> Or >>>> >>>> - Build my job jar and include JDBC driver in the shadow, plus copy the >>>> JDBC driver in the flink lib folder, plus make an entry in config for >>>> classloader.parent-first-patterns-additional >>>> <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#classloader-parent-first-patterns-additional> >>>> >>>> >>>> On Thu, Apr 28, 2022 at 10:17 AM Chesnay Schepler <ches...@apache.org> >>>> wrote: >>>> >>>>> I think what I meant was "either add it to /lib, or [if it is already >>>>> in /lib but also bundled in the jar] add it to the parent-first patterns." >>>>> >>>>> On 28/04/2022 15:56, Chesnay Schepler wrote: >>>>> >>>>> Pretty sure, even though I seemingly documented it incorrectly :) >>>>> >>>>> On 28/04/2022 15:49, John Smith wrote: >>>>> >>>>> You sure? >>>>> >>>>> - >>>>> >>>>> *JDBC*: JDBC drivers leak references outside the user code >>>>> classloader. To ensure that these classes are only loaded once you >>>>> should >>>>> either add the driver jars to Flink’s lib/ folder, or add the >>>>> driver classes to the list of parent-first loaded class via >>>>> classloader.parent-first-patterns-additional >>>>> >>>>> <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#classloader-parent-first-patterns-additional> >>>>> . >>>>> >>>>> It says either or >>>>> >>>>> >>>>> On Wed, Apr 27, 2022 at 3:44 AM Chesnay Schepler <ches...@apache.org> >>>>> wrote: >>>>> >>>>>> You're misinterpreting the docs. >>>>>> >>>>>> The parent/child-first classloading controls where Flink looks for a >>>>>> class *first*, specifically whether we first load from /lib or the >>>>>> user-jar. >>>>>> It does not allow you to load something from the user-jar in the >>>>>> parent classloader. That's just not how it works. >>>>>> >>>>>> It must be in /lib. >>>>>> >>>>>> On 27/04/2022 04:59, John Smith wrote: >>>>>> >>>>>> Hi Chesnay as per the docs... >>>>>> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/debugging/debugging_classloading/ >>>>>> >>>>>> You can either put the jars in task manager lib folder or use >>>>>> classloader.parent-first-patterns-additional >>>>>> <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#classloader-parent-first-patterns-additional> >>>>>> >>>>>> I prefer the latter like this: the dependency stays with the user-jar >>>>>> and not on the task manager. >>>>>> >>>>>> On Tue, Apr 26, 2022 at 9:52 PM John Smith <java.dev....@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Ok so I should put the Apache ignite and my Microsoft drivers in the >>>>>>> lib folders of my task managers? >>>>>>> >>>>>>> And then in my job jar only include them as compile time >>>>>>> dependencies? >>>>>>> >>>>>>> >>>>>>> On Tue, Apr 26, 2022 at 10:42 AM Chesnay Schepler < >>>>>>> ches...@apache.org> wrote: >>>>>>> >>>>>>>> JDBC drivers are well-known for leaking classloaders unfortunately. >>>>>>>> >>>>>>>> You have correctly identified your alternatives. >>>>>>>> >>>>>>>> You must put the jdbc driver into /lib instead. Setting only the >>>>>>>> parent-first pattern shouldn't affect anything. >>>>>>>> That is only relevant if something is in both in /lib and the >>>>>>>> user-jar, telling Flink to prioritize what is in lib. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 26/04/2022 15:35, John Smith wrote: >>>>>>>> >>>>>>>> So I put classloader.parent-first-patterns.additional: >>>>>>>> "org.apache.ignite." in the task config and so far I don't think I'm >>>>>>>> getting "java.lang.OutOfMemoryError: Metaspace" any more. >>>>>>>> >>>>>>>> Or it's too early to tell. >>>>>>>> >>>>>>>> Though now, the task managers are shutting down due to some >>>>>>>> other failures. >>>>>>>> >>>>>>>> So maybe because tasks were failing and reloading often the task >>>>>>>> manager was running out of Metspace. But now maybe it's just >>>>>>>> cleanly shutting down. >>>>>>>> >>>>>>>> On Wed, Apr 20, 2022 at 11:35 AM John Smith <java.dev....@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Or I can put in the config to treat org.apache.ignite. classes as >>>>>>>>> first class? >>>>>>>>> >>>>>>>>> On Tue, Apr 19, 2022 at 10:18 PM John Smith < >>>>>>>>> java.dev....@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Ok, so I loaded the dump into Eclipse Mat and followed: >>>>>>>>>> https://cwiki.apache.org/confluence/display/FLINK/Debugging+ClassLoader+leaks >>>>>>>>>> >>>>>>>>>> - On the Histogram, I got over 30 entries for: >>>>>>>>>> ChildFirstClassLoader >>>>>>>>>> - Then I clicked on one of them "Merge Shortest Path..." and >>>>>>>>>> picked "Exclude all phantom/weak/soft references" >>>>>>>>>> - Which then gave me: SqlDriverManager > Apache Ignite JdbcThin >>>>>>>>>> Driver >>>>>>>>>> >>>>>>>>>> So i'm guessing anything JDBC based. I should copy into the task >>>>>>>>>> manager libs folder and my jobs make the dependencies as compile >>>>>>>>>> only? >>>>>>>>>> >>>>>>>>>> On Tue, Apr 19, 2022 at 12:18 PM Yaroslav Tkachenko < >>>>>>>>>> yaros...@goldsky.io> wrote: >>>>>>>>>> >>>>>>>>>>> Also >>>>>>>>>>> https://shopify.engineering/optimizing-apache-flink-applications-tips >>>>>>>>>>> might be helpful (has a section on profiling, as well as >>>>>>>>>>> classloading). >>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 19, 2022 at 4:35 AM Chesnay Schepler < >>>>>>>>>>> ches...@apache.org> wrote: >>>>>>>>>>> >>>>>>>>>>>> We have a very rough "guide" in the wiki (it's just the >>>>>>>>>>>> specific steps I took to debug another leak): >>>>>>>>>>>> >>>>>>>>>>>> https://cwiki.apache.org/confluence/display/FLINK/Debugging+ClassLoader+leaks >>>>>>>>>>>> >>>>>>>>>>>> On 19/04/2022 12:01, huweihua wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi, John >>>>>>>>>>>> >>>>>>>>>>>> Sorry for the late reply. You can use MAT[1] to analyze the >>>>>>>>>>>> dump file. Check whether have too many loaded classes. >>>>>>>>>>>> >>>>>>>>>>>> [1] https://www.eclipse.org/mat/ >>>>>>>>>>>> >>>>>>>>>>>> 2022年4月18日 下午9:55，John Smith <java.dev....@gmail.com> 写道： >>>>>>>>>>>> >>>>>>>>>>>> Hi, can anyone help with this? I never looked at a dump file >>>>>>>>>>>> before. >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Apr 14, 2022 at 11:59 AM John Smith < >>>>>>>>>>>> java.dev....@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, so I have a dump file. What do I look for? >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Mar 31, 2022 at 3:28 PM John Smith < >>>>>>>>>>>>> java.dev....@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Ok so if there's a leak, if I manually stop the job and >>>>>>>>>>>>>> restart it from the UI multiple times, I won't see the issue >>>>>>>>>>>>>> because >>>>>>>>>>>>>> because the classes are unloaded correctly? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Mar 31, 2022 at 9:20 AM huweihua < >>>>>>>>>>>>>> huweihua....@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The difference is that manually canceling the job stops the >>>>>>>>>>>>>>> JobMaster, but automatic failover keeps the JobMaster running. >>>>>>>>>>>>>>> But looking >>>>>>>>>>>>>>> on TaskManager, it doesn't make much difference >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2022年3月31日 上午4:01，John Smith <java.dev....@gmail.com> 写道： >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Also if I manually cancel and restart the same job over and >>>>>>>>>>>>>>> over is it the same as if flink was restarting a job due to >>>>>>>>>>>>>>> failure? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I.e: When I click "Cancel Job" on the UI is the job >>>>>>>>>>>>>>> completely unloaded vs when the job scheduler restarts a job >>>>>>>>>>>>>>> because if >>>>>>>>>>>>>>> whatever reason? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Lile this I'll stop and restart the job a few times or maybe >>>>>>>>>>>>>>> I can trick my job to fail and have the scheduler restart it. >>>>>>>>>>>>>>> Ok let me >>>>>>>>>>>>>>> think about this... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Mar 30, 2022 at 10:24 AM 胡伟华 <huweihua....@gmail.com> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> So if I run the same jobs in my dev env will I still be >>>>>>>>>>>>>>>> able to see the similar dump? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I think running the same job in dev should be reproducible, >>>>>>>>>>>>>>>> maybe you can have a try. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> If not I would have to wait at a low volume time to do it >>>>>>>>>>>>>>>> on production. Aldo if I recall the dump is as big as the JVM >>>>>>>>>>>>>>>> memory right >>>>>>>>>>>>>>>> so if I have 10GB configed for the JVM the dump will be 10GB >>>>>>>>>>>>>>>> file? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Yes, JMAP will pause the JVM, the time of pause depends on >>>>>>>>>>>>>>>> the size to dump. you can use "jmap -dump:live" to dump only >>>>>>>>>>>>>>>> the reachable >>>>>>>>>>>>>>>> objects, this will take a brief pause >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2022年3月30日 下午9:47，John Smith <java.dev....@gmail.com> 写道： >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I have 3 task managers (see config below). There is total >>>>>>>>>>>>>>>> of 10 jobs with 25 slots being used. >>>>>>>>>>>>>>>> The jobs are 100% ETL I.e; They load Json, transform it and >>>>>>>>>>>>>>>> push it to JDBC, only 1 job of the 10 is pushing to Apache >>>>>>>>>>>>>>>> Ignite cluster. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> FOR JMAP. I know that it will pause the task manager. So if >>>>>>>>>>>>>>>> I run the same jobs in my dev env will I still be able to see >>>>>>>>>>>>>>>> the similar >>>>>>>>>>>>>>>> dump? I I assume so. If not I would have to wait at a low >>>>>>>>>>>>>>>> volume time to do >>>>>>>>>>>>>>>> it on production. Aldo if I recall the dump is as big as the >>>>>>>>>>>>>>>> JVM memory >>>>>>>>>>>>>>>> right so if I have 10GB configed for the JVM the dump will be >>>>>>>>>>>>>>>> 10GB file? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> # Operating system has 16GB total. >>>>>>>>>>>>>>>> env.ssh.opts: -l flink -oStrictHostKeyChecking=no >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> cluster.evenly-spread-out-slots: true >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> taskmanager.memory.flink.size: 10240m >>>>>>>>>>>>>>>> taskmanager.memory.jvm-metaspace.size: 2048m >>>>>>>>>>>>>>>> taskmanager.numberOfTaskSlots: 16 >>>>>>>>>>>>>>>> parallelism.default: 1 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> high-availability: zookeeper >>>>>>>>>>>>>>>> high-availability.storageDir: >>>>>>>>>>>>>>>> file:///mnt/flink/ha/flink_1_14/ >>>>>>>>>>>>>>>> high-availability.zookeeper.quorum: ... >>>>>>>>>>>>>>>> high-availability.zookeeper.path.root: /flink_1_14 >>>>>>>>>>>>>>>> high-availability.cluster-id: /flink_1_14_cluster_0001 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> web.upload.dir: /mnt/flink/uploads/flink_1_14 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> state.backend: rocksdb >>>>>>>>>>>>>>>> state.backend.incremental: true >>>>>>>>>>>>>>>> state.checkpoints.dir: >>>>>>>>>>>>>>>> file:///mnt/flink/checkpoints/flink_1_14 >>>>>>>>>>>>>>>> state.savepoints.dir: >>>>>>>>>>>>>>>> file:///mnt/flink/savepoints/flink_1_14 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Mar 30, 2022 at 2:16 AM 胡伟华 <huweihua....@gmail.com> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi, John >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Could you tell us you application scenario? Is it a flink >>>>>>>>>>>>>>>>> session cluster with a lot of jobs? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Maybe you can try to dump the memory with jmap and use >>>>>>>>>>>>>>>>> tools such as MAT to analyze whether there are abnormal >>>>>>>>>>>>>>>>> classes and >>>>>>>>>>>>>>>>> classloaders >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> > 2022年3月30日 上午6:09，John Smith <java.dev....@gmail.com> >>>>>>>>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > Hi running 1.14.4 >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > My tasks manager still fails with >>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: Metaspace. The metaspace >>>>>>>>>>>>>>>>> out-of-memory error >>>>>>>>>>>>>>>>> has occurred. This can mean two things: either the job >>>>>>>>>>>>>>>>> requires a larger >>>>>>>>>>>>>>>>> size of JVM metaspace to load classes or there is a class >>>>>>>>>>>>>>>>> loading leak. >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > I have 2GB of metaspace configed >>>>>>>>>>>>>>>>> taskmanager.memory.jvm-metaspace.size: 2048m >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > But the task nodes still fail. >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > When looking at the UI metrics, the metaspace starts >>>>>>>>>>>>>>>>> low. Now I see 85% usage. It seems to be a class loading leak >>>>>>>>>>>>>>>>> at this >>>>>>>>>>>>>>>>> point, how can we debug this issue? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>> >>>>> >>>>> >> >> >
