Hi Chiwan, I’m not expert of Scala but It seems about closure cleaning problem. Scala > App trait extends DelayedInit trait to initialize object. But Flink > serialization stack doesn’t handle this special initialization. (It is just > my opinion, not verified.) >
I arrived at the same conclusion with the DelayedInit trait. > > To run TFIDFNPE safely, you need to just change tokenize and uniqueWords > to method like following: > > ``` > def tokenize = … > def uniqueWords = ... > ``` > > With this change, I tested that TFIDFNPE works safely in Flink 0.10.1 > cluster. > Yeah, it works and i knew it. My aim it was to use tricky (functional) Scala mechanisms to test how much the Scala APIs are robust and idomitic-friendly for Scala users. > > About TFIDF object, you should avoid overriding main method if the object > is derived by App trait. It is also related DelaytedInit mechanism [1]. > Yeah, App trait is into a comment 'cause I am overriding the main method. TFIDF is equal to TFIDFNPE with App trait and not overriding main method. Thanks, Andrea > > [1]: > https://github.com/scala/scala/blob/2.10.x/src/library/scala/App.scala#L31 > > > On Jan 19, 2016, at 12:08 AM, Andrea Sella <[email protected]> > wrote: > > > > Hi, > > > > I was implementing TF-IDF example of flink-training when I faced a > problem with NPE during the deploy of my Job. > > > > Source code: https://github.com/alkagin/flink-tfidf-example > > > > I used 0.10.1 version and started in local mode. > > During the deploy of TFIDFNPE Job, which it extends App, Flink throws > NullPointerException on both flatMap functions. > > If I include the tokenize function into the closures of flatMap > functions, the Job works fine; see example TFIDFApp. > > To avoid this unexpected behavior I don't have use Scala App trait, see > TFIDF, but why? > > > > > > Thanks, > > Andrea > > > > > Regards, > Chiwan Park > >
