I created a JIRA ticket for this problem ( https://issues.apache.org/jira/browse/FLINK-1828) and I just made a PR that fix that (thanks Fabian and Robert for the great support!) Last question: is the "mapred.output.dir" parameter really necessary? At the end of my job that writes to hbase I found only _SUCCESSn and ._SUCCESS.crc files..
Best. Flavio On Sat, Apr 4, 2015 at 12:18 PM, <fhue...@gmail.com> wrote: > Yes, reusing output objects is a good practice but optional. It can help > to bring down GC overhead. > > You could make your function a RichFunction and initialize the output > object in open(). > > > > Switching function serialization to Kryo is on our TODO list (FLINK-1256). > Would be good to fix that soon, IMO. > > > Cheers, Fabian > > > From: Flavio Pompermaier > Sent: Saturday, 4. April, 2015 11:23 > To: dev@flink.apache.org > > > > > > Ok graeat!this was not prtfectly clear to me! I'll try that now. > > About reuse variable instead..I use it because I saw that this is a common > practice ib the examples but I'd like to know whether there's a real > benefit in reusing it for return tuple with respect to returning a brand > new one each time. Any insight about this? > On Apr 4, 2015 11:17 AM, "Fabian Hueske" <fhue...@gmail.com> wrote: > > > User functions are still serialized using Java serialization, not Kryo. > > Kryo is only used for data exchange at runtime between tasks. > > > > If a function such as your MapFunction has a non-serializable member > > variable, you need to declare it as transient and initialize it before it > > is executed, e.g., via open() or the first invocation of the functions > > processing method such as map(). > > > > 2015-04-04 10:59 GMT+02:00 Flavio Pompermaier <pomperma...@okkam.it>: > > > > > There's no way to register put with kryo for example? > > > On Apr 4, 2015 10:06 AM, "Robert Metzger" <rmetz...@apache.org> wrote: > > > > > > > Hey Flavio, > > > > > > > > I checked out your "master" branch and started the HBaseWriteExample. > > > > It started without errors (there were some errors connecting to > > > Zookeeper, > > > > but thats probably because I don't have HBase running). > > > > Am I using the right code ( > > > > > > > > > > > > > > https://github.com/fpompermaier/flink/commit/c1934da379dba360ad61d18bf921fae08822795a > > > > ) > > > > to reproduce this error? > > > > Maybe the error is also happening when the mapper is starting. > > > > > > > > Can you try making the following changes to your code? > > > > https://gist.github.com/rmetzger/a218beca4b0442f3c1f3 > > > > This is basically making the field that contains the non-serializable > > > "Put" > > > > element transient. > > > > > > > > > > > > > > > > On Sat, Apr 4, 2015 at 8:40 AM, Flavio Pompermaier < > > pomperma...@okkam.it > > > > > > > > wrote: > > > > > > > > > Any fix for this? > > > > > On Apr 3, 2015 7:43 AM, "Flavio Pompermaier" <pomperma...@okkam.it > > > > > > wrote: > > > > > > > > > > > > > > >
