Flume Sink, sometimes transaction not being commited/rolledback & closed

2014-09-09 Thread Ahmed Vila
Hi Group, We've implemented custom Flume Sink. We do some DB lookups for sanity checks within it and bunch of stuff in order to slightly prepare data for insertion into the postgresql via copy command. It gets the channel, transaction and it executes begin() method. Of course, there is try-catch-

Re: Flume Sink, sometimes transaction not being commited/rolledback & closed

2014-09-09 Thread Roshan Naik
you may want to rollback immediately after the log messages in the catch block and not at the end. On Tue, Sep 9, 2014 at 7:51 AM, Ahmed Vila wrote: > Hi Group, > > We've implemented custom Flume Sink. > We do some DB lookups for sanity checks within it and bunch of stuff in > order to slightly

Re: Flume Sink, sometimes transaction not being commited/rolledback & closed

2014-09-09 Thread Ahmed Vila
Hi Roshan, I'm puzzled because there should be at least a note about exception in catch block as I've already confirmed that flume logs those. Is it possible that SInkRunner just mixes up transactions ? Anyone already found himself in similar situation ? On Tue, Sep 9, 2014 at 11:15 PM, Roshan

Re: Flume Sink, sometimes transaction not being commited/rolledback & closed

2014-09-09 Thread Ahmed Vila
maybe I haven't expressed my self in a god way... I've already confirmed that flume logs all unhandled exceptions, so it should log the ones emerging from within that catch block. Actually, I've mixed something before with that cacheDBPool being null, so I can confirm that flume does log that exce

Re: Avro source and sink

2014-09-09 Thread Ed Judge
Joey, Thank you very much for the info. Let me look this over and digest it. Thanks, Ed. Sent from my iPhone > On Sep 8, 2014, at 4:27 PM, Joey Echeverria wrote: > > Hi Ed! > > This is definitely doable. What you want is an intercepter on source A that > will do the conversion from log li