question on sideoutput from ProcessWindow function

2017-09-08 Thread Prabhu V
Hi, Can we have a side output from a process window function ? I am currently genrating a side output from a process function. The main output of the process function is then Windowed and a ProcessWindowFunction is applied on the windows. Can I add to the SideOutpuStream from the ProcessWindowFu

Re: [EXTERNAL] Re: Fink application failing with kerberos issue after running successfully without any issues for few days

2017-08-17 Thread Prabhu V
+1 on the 7 day expiry explanation, This is most likely the cause. I faced the 7 day expiry issue with a previous version of flink that dint support keytabs, I am currently running flink-1.3 with keytabs (it has been going okay for 2 days now), I will update after the 7 day mark. Thanks, Prabhu

Re: kerberos yarn - failure in long running streaming application

2017-08-15 Thread Prabhu V
if you prefer to share the the logs privately. > > -Eron > > On Mon, Aug 14, 2017 at 12:32 AM, Ted Yu wrote: > >> bq. security.kerberos.login.contexts: Client,KafkaClien >> >> Just curious: there is missing 't' at the end of the above line. >> >> Maybe a

kerberos yarn - failure in long running streaming application

2017-08-13 Thread Prabhu V
Hi, I am running Flink-1.3.2 on yarn (Cloudera 2.6.0-cdh5.7.6). The application stream data from kafka, groups by key, creates a session window and writes to HDFS using a rich window function in the "window.apply" method. The rich window function creates the sequence file thus SequenceFile.creat

Re: set flink yarn jvm options

2016-08-04 Thread Prabhu V
n Thu, Aug 4, 2016 at 4:07 PM, Prabhu V wrote: > >> >> This property i believe affects only the yarn client. I want to set jvm >> opts on application-manager and task-manager containers. >> >> Thanks, >> Prabhu >> >> On Thu, Aug 4, 2016 at 3:07 PM, Jami

Re: set flink yarn jvm options

2016-08-04 Thread Prabhu V
n Thu, Aug 4, 2016 at 11:10 AM, Prabhu V wrote: > >> The docs mention that >> >> env.java.opts.jobmanager >> env.java.opts.taskmanager >> >> parameters are available but are ignored by the yarn client, is there a >> way to set the jvm opts for yarn ? &g

Re: set flink yarn jvm options

2016-08-04 Thread Prabhu V
The docs mention that env.java.opts.jobmanager env.java.opts.taskmanager parameters are available but are ignored by the yarn client, is there a way to set the jvm opts for yarn ? Thanks, Prabhu On Wed, Aug 3, 2016 at 7:03 PM, Prabhu V wrote: > Hi, > > Is there a way to set jvm o

set yarn jvm options

2016-08-03 Thread Prabhu V
Hi, Is there a way to set jvm options on the yarn application-manager and task-manager with flink ? Thanks, Prabhu

Re: Kakfa batches

2016-08-03 Thread Prabhu V
PM, Prabhu V wrote: > > Obeservations with Streaming. > > > > 1) Long running kerberos fails in 7 days (the data that is held in the > > window buffer is lost and restart results in event loss) > > This is a known issue I think. Looping in Max who knows the details I &

Re: Flink Batch Processing with Kafka

2016-08-03 Thread Prabhu V
If your environment is not kerberized (or if you can offord to restart the job every 7 days), a checkpoint enabled, flink job with windowing and the count trigger, would be ideal for your requirement. Check the api's on flink windows. I had something like this that worked stream.keyBy(0).countW

Kakfa batches

2016-08-03 Thread Prabhu V
I understand flink does steaming, but i feel my requirement is more batch oriented. Read froma kafka cluster, Do a little data massaging Bucket data into hadoop files that are atleast one hdfs block in size. Our environment is Yarn and kerberized (kafka and hadoop, i am currently allowed pass the