Re: Rich Window Function - When does close(tear down) method executes ?

Can I get any update please ? Regards, Swapnil

Re: How can I prove ....

Hi Again & sorry to take your time. But am puzzled by what I cannot explain why.The parallelism is set to 448. There are 112 tasks per TM. Why is Flink NOT allocating ALL 448 slots? It allocates only 1/2 of it.I also bumped up the # buffers to equate a 2GiB in each TM & see no difference :-(So I

Re: Sharing Java Collections within Flink Cluster

Hi Team, Will you be able to guide me on this? Is this a known issue with checkpointing ? CVP On 22 Sep 2016 15:57, "Chakravarthy varaga" wrote: > PFA, Flink_checkpoint_time.png in relation to this issue. > > On Thu, Sep 22, 2016 at 3:38 PM, Chakravarthy varaga < > chakravarth...@gmail.c

Fwd: Flink Scala - data load from CSV to postgress database.

Hi Guys, We have a requirement like – loading data from local CSV file to Postgress database using Flink Scala…We have tried number of ways all failed Do you have any example for this? With dependency libraries to understand how to load data from CSV to postgres We have tried and searched in Goo

Re: AW: REST Interface to JobManager

It looks like I was able to finally find what I needed. WebRuntimeMonitor.java contains the routes, with a little fiddling in the browser I think I can make the rest happen. https://github.com/apache/flink/blob/master/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/WebRuntimeM

AW: REST Interface to JobManager

Hi Curtis, we implemented this today. But without a REST-Interface. We transfer out artifacts and a script with a scp call from out Bamboo server and execute the script. This script kills the yarn application, start a new flink application in yarn and submit all routes to the cluster. Work

Rich Window Function - When does close(tear down) method executes ?

I am using rich window function in my streaming project. I want "close" method to get triggered after each window interval. In my case, open gets executed life time once & close method doesn't get executed ? Can anybody help to sort out same ? I want tear down method after each window interval. T

Re: How can I prove ....

Hi Again, following is from the dashboard while wverything is supposedlt running.No real-time change in send/received/#of records...but one node is definitely producing a *.out file...And all TMs are reporting in their *.log files. And the process will eventually end , but very slow.Thanks again

REST Interface to JobManager

I would like to be able to use Jenkins to deploy jobs to Flink. I’ve seen talk of a REST interface that might allow me to do this https://issues.apache.org/jira/browse/FLINK-1228 Is there any documentation around this feature?

Re: How can I prove ....

Thanks Aljoscha,Thats why I am wondering about this. I dont see send/receive columns change at alljust 0's all the time.The only thing that changes is time stamp.Is this an indication that the nodes in the cluster are not participating in execution of the data?Thanks again.Amir- From:

Re: Sharing Java Collections within Flink Cluster

PFA, Flink_checkpoint_time.png in relation to this issue. On Thu, Sep 22, 2016 at 3:38 PM, Chakravarthy varaga < chakravarth...@gmail.com> wrote: > Hi Aljoscha & Fabian, > > Finally I got this working. Thanks for your help. In terms persisting > the state (for S2), I tried to use checkpoint e

Re: Sharing Java Collections within Flink Cluster

Hi Aljoscha & Fabian, Finally I got this working. Thanks for your help. In terms persisting the state (for S2), I tried to use checkpoint every 10 Secs using a FsStateBackend... What I notice is that the checkpoint duration is almost 2 minutes for many cases, while for the other cases it vari

Re: RawSchema as deserialization schema

/cc Robert, he is looking into extending the Kafka Connectors to support more of Kafka's direct utilities On Thu, Sep 22, 2016 at 3:17 PM, Swapnil Chougule wrote: > It will be good to have RawSchema as one of the deserialization schema in > streaming framework (like SimpleStringSchema). > Many u

Re: RawSchema as deserialization schema

It will be good to have RawSchema as one of the deserialization schema in streaming framework (like SimpleStringSchema). Many use cases needs data in byte array format after reading from source like kafka. Any inputs for same ? On Mon, Sep 12, 2016 at 11:42 AM, Swapnil Chougule wrote: > Thanks

Re: How can I prove ....

Hi, depending on the data source you might not be able to stress CPU/MEM because the source might be to slow. As long as you see the numbers increasing in the Flink Dashboard for all operators you should be good. Cheers, Aljoscha On Thu, 22 Sep 2016 at 00:26 amir bahmanyari wrote: > That all no

Re: Daily/hourly TumblingEventTimeWindows

I have just noticed that this is exactly what it currently does. Reading the docs I assumed all windows would be of the same size. > Am 22.09.2016 um 13:35 schrieb Maximilian Bode : > > Hi everyone, > > is there an easy way to implement a tumbling event time window that tumbles > at a certain

Daily/hourly TumblingEventTimeWindows

Hi everyone, is there an easy way to implement a tumbling event time window that tumbles at a certain time? Examples could be daily or hourly (tumbling at exactly 00:00, 01:00, 02:00 etc.) windows. So in particular, for a daily window, the first window would be shorter than the rest, tumble at

Re: Flink JDBCOutputFormat logs wrong WARN message

Thanks Marton !! On Thu, Sep 22, 2016 at 4:36 PM, Márton Balassi wrote: > Done. Go ahead, Swapnil. > > Best, > Marton > > On Thu, Sep 22, 2016 at 1:03 PM, Swapnil Chougule > wrote: > >> Hi Fabian/ Chesnay >> Can anybody give me permission to assign JIRA (created for same.)? >> >> Thanks, >> Swa

Re: Custom(application) Metrics - Piggyback on Flink's metrics infra or not?

Actually i was wrong on the UDF point. By variables i meant the information that is encoded in the scope, like the subtask index, task name, taskmanager ID etc., however all these can be accessed from the MetricGroup that is returned by RuntimeContext#getMetricGroup(), which you can of course u

Re: Flink JDBCOutputFormat logs wrong WARN message

Done. Go ahead, Swapnil. Best, Marton On Thu, Sep 22, 2016 at 1:03 PM, Swapnil Chougule wrote: > Hi Fabian/ Chesnay > Can anybody give me permission to assign JIRA (created for same.)? > > Thanks, > Swapnil > > On Tue, Sep 20, 2016 at 6:18 PM, Swapnil Chougule > wrote: > >> Thanks Chesnay & Fa

Re: Flink JDBCOutputFormat logs wrong WARN message

Hi Fabian/ Chesnay Can anybody give me permission to assign JIRA (created for same.)? Thanks, Swapnil On Tue, Sep 20, 2016 at 6:18 PM, Swapnil Chougule wrote: > Thanks Chesnay & Fabian for update. > I will create JIRA issue & open a pull request to fix it. > > Thanks, > Swapnil > > On Tue, Sep

Re: Simple batch job hangs if run twice

Can you try running with DEBUG logging level? Then you should see if input splits are assigned. Also, you could try to use a debugger to see what's going on. On Mon, Sep 19, 2016 at 2:04 PM, Yassine MARZOUGUI < y.marzou...@mindlytix.com> wrote: > Hi Chensey, > > I am running Flink 1.1.2, and usin

Re: how to unit test streaming window jobs?

Hi Luis, using Event Time windows, you should be able to generate some test data and get predictable results. Flink is internally using similar tests to ensure correctness of the windowing implementation (for example the EventTimeWindowCheckpointingITCase). Regards, Robert On Mon, Sep 12, 2016 a

Re: emit watermarks

Hi Radu, Watermarks cannot be emitted by the triggers. Their responsibility is just to tell the operator to do (or not do) sth with the data it has gathered. Thus the name Trigger :P. Eviction policies allow you to also filter some elements out before applying your function. But that is as far

emit watermarks

Hi, Is there some way to emit a watermark in the trigger? I see that in the evictor there is the option to check the StreamRecord if it is a watermark..so I would hope that there is some option also to create them

Re: Flink job fails to restore RocksDB state after upgrading to 1.2-SNAPSHOT

Hi, to me, this looks like you are running into the problem described under [FLINK-4603] : KeyedStateBackend cannot restore user code classes. I have opened a pull request (PR 2533) this morning that should fix this behavior as soon as it is merged into master. Best, Stefan > Am 21.09.2016 um