Hi Luke, I can send you a code snippet with more details if it helps.
BTW found similar issue here: http://mail-archives.apache.org/mod_mbox/beam-user/201801.mbox/%3ccaf9t7_74pkr7fj51-6_tbsycz9aiz_xsm7rcali5kmkd1ng...@mail.gmail.com%3E Thanks and Regards Mohil On Thu, Jun 25, 2020 at 10:58 AM Mohil Khare <[email protected]> wrote: > Hi Luke, > Thanks for your response, I tried looking at worker logs using the logging > service of GCP and unable to get a clear picture. Not sure if its due to > memory pressure or low number of harness threads. > Attaching a few more screenshots of crash logs that I found as wells json > dump of logs. > > Let me know if you still think opening a ticket is a right way to go. > > Thanks and regards > Mohil > > On Thu, Jun 25, 2020 at 10:00 AM Luke Cwik <[email protected]> wrote: > >> Try looking at the worker logs to get a full stack trace. Take a look at >> this page for some debugging guidance[1] or consider opening a support case >> with GCP. >> >> 1: >> https://cloud.google.com/dataflow/docs/guides/troubleshooting-your-pipeline >> >> On Thu, Jun 25, 2020 at 1:42 AM Mohil Khare <[email protected]> wrote: >> >>> BTW, just to make sure that there is no issue with any individual >>> PTransform, I enabled each one of them one by one and the pipeline started >>> successfully. Issue happens as soon as I enable more than one new >>> aforementioned PTransform. >>> >>> Thanks and regards >>> Mohil >>> >>> On Thu, Jun 25, 2020 at 1:26 AM Mohil Khare <[email protected]> wrote: >>> >>>> Hello All, >>>> >>>> I am using the BEAM java 2.19.0 version on google dataflow. >>>> >>>> Need urgent help in debugging one issue. >>>> >>>> I recently added 3-4 new PTransformations. to an existing pipeline >>>> where I read data from BQ for a certain timestamp and create >>>> PCollectionView<Map<Key,value>> to be used as side input in other >>>> PTransforms. >>>> >>>> i.e. something like this: >>>> >>>> /** >>>> * Get PCollectionView Stats1 >>>> */ >>>> PCollectionView<Map<Stats1Key, Stats1>> stats1View = >>>> jobCompleteStatus >>>> .apply("Reload_MonthlyS2Stats_FromBQ", new ReadStatsS1()) >>>> .apply("View_S1STATS", View.asSingleton()); >>>> >>>> /** >>>> * Get PCollectionView of Stats2 >>>> */ >>>> PCollectionView<Map<Stats2Key, Stats2>> stats2View = >>>> jobCompleteStatus >>>> .apply("Reload_OptimalAppCharsInfo_FromBQ", new ReadStatsS2()) >>>> .apply("View_S2STATS", View.asSingleton()); >>>> >>>> >>>> and a couple more like these PTransforms. Here jobCompleteStatus is a >>>> message >>>> >>>> received from PubSub that act as a trigger to reload these views. >>>> >>>> The moment I deployed the above pipeline, it didn't start and >>>> >>>> error reporting gave weird exceptions(see attached screenshot1 and >>>> screenshot) which I don't know how to debug. >>>> >>>> >>>> Then as an experiment I made a change where I enabled only one new >>>> transformation >>>> >>>> and disabled others. This time I didn't see any issue. >>>> >>>> So it looks like some memory issue. >>>> >>>> I also compared worker logs between working case and non working case >>>> >>>> and it looks resources were not granted in non working case. >>>> >>>> (See attached working-workerlogs and nonworking-workerlogs) >>>> >>>> I could't find any other log. >>>> >>>> >>>> I would really appreciate quick help here. >>>> >>>> >>>> Thanks and Regards >>>> >>>> Mohil >>>> >>>> >>>>
