Re: Error handling

Was referring to the original email thread: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Error-handling-td3448.html On Tue, May 8, 2018 at 5:29 PM, vishnuviswanath < vishnu.viswanat...@gmail.com> wrote: > Hi, > > Wondering if any of these ideas were implemented after the

Re: Apache Flink - Evictor interface clarification

Hi Mans, Have a look at this: http://apache-flink-mailing-list-archive.1008284. n3.nabble.com/DISCUSS-Enhance-Window-Evictor-in-Flink-tp12406p12442.html Thanks, Vishnu On Sat, Nov 18, 2017 at 11:28 AM, M Singh wrote: > Hi: > > I am looking at the documentation and

Re: Job submission timeout

Never mind, it was a silly mistake, I used "=" instead of ":" while setting akka.ask.timeout. Now it works fine! On Tue, Aug 22, 2017 at 5:10 PM, Vishnu Viswanath < vishnu.viswanat...@gmail.com> wrote: > Hi, > > After I submit the job the client timeout after

Job submission timeout

Hi, After I submit the job the client timeout after 10 seconds( Guess Job manager is taking long time to build the graph, it is a pretty big JobGraph). *Caused by: org.apache.flink.runtime.client.JobTimeoutException: JobManager did not respond within 1 milliseconds* * at org.apache.flink.runt

Re: Re: AVRO Union type support in Flink

etreff: Re: AVRO Union type support in Flink > Datum: Wed, 19 Jul 2017 10:26:24 -0400 > Von: Vishnu Viswanath > > An: Timo Walther > > > Hi Timo, > > Thanks for checking that. I did not try yet. My current application uses > Cascading and it has the limitati

AVRO Union type support in Flink

Hi All, Does Flink support AVRO union types - Documentation says it supports nullable types: {"name": "type_double_test", "type": ["null", "double"]} But my schema has something like : {"name": "union_field", "type": ["string", "double"]} Thanks Vishnu

Re: Checkpointing with RocksDB as statebackend

Hi Stephan, Is there a ticket number/link to track this, My job has all the conditions you mentioned. Thanks, Vishnu On Tue, Mar 14, 2017 at 7:13 AM, Stephan Ewen wrote: > Hi Vinay! > > We just discovered a bug in RocksDB. The bug affects windows without > reduce() or fold(), windows with evic

Incremental checkpoint branch

Hi, Can someone point me to the branch where the ongoing work for incremental checkpoint is going on, I would like to try it out even if the work is not complete. I have a use case where the state size increase about ~1gb every 5 minutes. Thanks, Vishnu

Re: Flink survey by data Artisans

Works for me also. On Fri, Nov 18, 2016 at 12:35 PM, Stephan Ewen wrote: > Just checked it, the link works for me. > > > On Fri, Nov 18, 2016 at 7:20 PM, amir bahmanyari > wrote: > >> [image: Inline image] >> >> >> -- >> *From:* Kostas Tzoumas >> *To:* "d...@flink.a

Re: Firing windows multiple times

Hi, Sorry, I was little busy lately. I have most part of FLIP-4 ready except ProcessingTimeEvictor case. I was trying to find the best way to set the ProcessingTime as per the suggestion from Aljoscha then I got busy with some other work. We can definitely work together on that. Thanks Vishnu O

Re: [SUGGESTION] Stack Overflow Documentation for Apache Flink

14:22, Maximilian Michels > > <mailto:m...@apache.org>> wrote: >> > >>> >> > >>> Hi! >> > >>> >> > >>> This looks neat. Let's try it out. I just voted. >> > >>> >> > >>> Cheers, >> > >>> Max >> > >>> >> > >>> On Sun, Sep 4, 2016 at 8:11 PM, Vishnu Viswanath >> > >>> > > <mailto:vishnu.viswanat...@gmail.com>> wrote: >> > >>> > Hi All, >> > >>> > >> > >>> > Why don't we make use of Stackoverflow's new documentation >> > feature to >> > >>> > do >> > >>> > some documentation of Apache Flink. >> > >>> > >> > >>> > To start, at least 5 SO users should commit to document, who >> > has at >> > >>> > least150 >> > >>> > reputation and have at least 1 positively scored answer in >> > Flink tag. >> > >>> > >> > >>> > http://stackoverflow.com/documentation/apache-flink >> > <http://stackoverflow.com/documentation/apache-flink> >> > >>> > >> > >>> > Regards, >> > >>> > Vishnu Viswanath >> > >> >> > >> >> > > >> > >> > >> >> >

[SUGGESTION] Stack Overflow Documentation for Apache Flink

erflow.com/documentation/apache-flink Regards, Vishnu Viswanath

Re: Not able to query : Queryable State

Hi, Are you sure the job id matches with running job? The error clearly says JobID not found. How are you setting the JobID, If I remember correctly, I used something like val jobId = JobID.fromHexString("myjob_id_here") to get the JobID object and used it to call client.getKvState() Thanks, Vis

Re: Firing windows multiple times

Hi Aljoscha, This looks like the bug that we discussed, as part of Enhance window evictor JIRA Thanks, Vishnu On Wed, Aug 10, 2016 at 1:18 PM, Aljoscha Krettek wrote: > Hi, > from your mail I'm gathering that you are in fact using an Evictor, is > that correct? If not, then the window operator

Re: Regarding QueryableState

Hi Ufuk, I was able to create a QueryableState stream and query it when running in cluster mode in my local machine, but I am unable to Query the stream when running on AWS cluster, getting error AbstractMethodError at kvState.getSerializedValue(serializedKeyAndNamespace) in KvStateServerHandler.j

Re: how does flink assign windows to task

old all the elements >> that can arrive for all the active windows (not triggered) for two keys in >> a task. For count windows this is easy to estimate. But for times windows >> it is less clear if you receive elements out of order. >> >> Let's see what Max re

Re: how does flink assign windows to task

but nodes might not have enough memory to hold the entire window in memory (since I can have a large number of Panes). Thanks, Vishnu On Fri, Jul 29, 2016 at 4:29 AM, Maximilian Michels wrote: > Hi Vishnu Viswanath, > > The keyed elements are spread across the 50 task slots (as

how does flink assign windows to task

slots and wait for 20 of them to finish before assigning the remaining 20 to the slots? Thanks, Vishnu Viswanath

Re: Processing windows in event time order

Hi David, You are right, the events in the window are not sorted according to the EventTime hence the processing is not done in an increasing order of timestamp. As you said, you will have to do the sorting yourself in your window function to make sure that you are processing the events in order.

Re: countWindow custom WindowFunction

], out: Collector[String]): Unit = { val previous: Double = input.head.observation val current: Double = input.last.observation val delta: Double = current - previous out.collect(s"TEST-DELTA: $window, $delta") } } Thanks and Regards, Vishnu Viswanath, www.vishnuviswana

Re: Checkpointing very large state in RocksDB?

Hi, Is there any other disadvantage of using fullyAsyncSnapshot, other than being slower. And would the slowness really matter since it is async anyways? Thanks and Regards, Vishnu Viswanath, On Thu, Jun 30, 2016 at 8:07 AM, Aljoscha Krettek wrote: > Hi, > are you taking

Re: maximum size of window

. > > Cheers, > Aljoscha > > On Tue, 28 Jun 2016 at 17:26 Vishnu Viswanath < > vishnu.viswanat...@gmail.com > > wrote: > >> Hi, >> >> Thank you for the responses. >> I am not sure if I will be able to use Fold/Reduce function, but I will &g

Re: maximum size of window

and Regards, Vishnu Viswanath, On Tue, Jun 28, 2016 at 5:04 AM, Aljoscha Krettek wrote: > Hi, > one thing to add: if you use a ReduceFunction or a FoldFunction for your > window the state will not grow with bigger window sizes or larger numbers > of elements because the result is eager

Re: maximum size of window

g > re-written to explain new features introduced in Flink 1.1, > which include more flexible handling of late events and more explicit > state garbage collection. > > So please stay tuned! > > I hope this helps at answering your question, > Kostas > > > On Jun 27, 2016,

maximum size of window

Hi All, - Is there any restriction on the size of a window in Flink with respect to the memory of the nodes? - What happens if a window size grows more than size of a node, will it be split into multiple nodes? if I am going to have a huge window, should I have fewer nodes with more memory. Is th

Re: dataset dataframe join

aStream program. That way you can keep > the data in your operator and then join with incoming stream records. > > Cheers, > Till > > On Wed, Jun 15, 2016 at 11:43 PM, Vishnu Viswanath < > vishnu.viswanat...@gmail.com> wrote: > >> Hi All, >> >> Is th

dataset dataframe join

Hi All, Is there any workaround/hack to join a dataset with datastream since https://issues.apache.org/jira/browse/FLINK-2320 is still in progress. Regards, Vishnu

Re: Behavior of SlidingProessingTimeWindow with CountTrigger

Hi Aljoscha, Thank you for the explanation and the link on IBM infosphere. That explains whey I am seeing (a,3) and (b,3) in my example. Yes, the name Evictor is confusing. Thanks and Regards, Vishnu Viswanath, www.vishnuviswanath.com On Mon, Mar 14, 2016 at 11:24 AM, Aljoscha Krettek wrote

Re: Behavior of SlidingProessingTimeWindow with CountTrigger

be evicted from the window, which can also be part of the next processing of that window. But here it looks like the sum is calculated only on the items that were evicted from the window. Could you please explain what is going on here. Thanks and Regards, Vishnu Viswanath, *www.vishnu

Behavior of SlidingProessingTimeWindow with CountTrigger

printed twice(a,c,e) and for some characters it is printed only once (b,d). I am not able to figure out what is going on. I think it may have something to do with the SlidingProcessingTimeWindow but I am not sure. Can someone explain me what is going on? Thanks and Regards, Vishnu Viswanath www.vishnuviswanath.com ​

Re: NoSuchMethodError flatMap

Fixed this, error was due to miss match between my flink version installed and the one I mentioned in my sbt file. On Mon, Mar 7, 2016 at 2:33 PM, Vishnu Viswanath < vishnu.viswanat...@gmail.com> wrote: > Hi All, > > After successfully writing the wordcount program, I was try

NoSuchMethodError flatMap

Hi All, After successfully writing the wordcount program, I was trying to create a streaming application but is getting below error when submitting the job in local mode. Vishnus-MacBook-Pro:flink vishnu$ flink run target/scala-2.11/flink-vishnu_2.11-1.0.jar java.lang.NoSuchMethodError: