Hi,
We went a different way and used the same underlying ClusterClient classes
from the command line tool, to connect to the JobManager and submit JARs
that way.
>From the YARN RM address, we search for the Flink application and use the
YarnClusterClient to get the connection details from the disc
Hi!
Let's try to figure that one out. Can you give us a bit more information?
- What source are you using for the slow input?
- How large is the state that you are checkpointing?
- Can you try to see in the log if actually the state snapshot takes that
long, or if it simply takes long for t
Hi CVP,
I'm not so much familiar with the internals of the checkpointing system,
but maybe Stephan (in CC) has an idea what's going on here.
Best, Fabian
2016-09-23 11:33 GMT+02:00 Chakravarthy varaga :
> Hi Aljoscha & Fabian,
>
> I have a stream application that has 2 stream source as belo
Are you sure you have the parallelism set to 448? You can see the
parallelism of operators in the web UI.
On Fri, Sep 23, 2016 at 12:15 AM, amir bahmanyari
wrote:
> Hi Again & sorry to take your time. But am puzzled by what I cannot
> explain why.
> The parallelism is set to 448. There are 112
Hi Swapnil,
you can just have all your logic in the apply() method of your
WindowFunction. This is called once the window fires so any code that you
put at the end there will be executed effectively after each window firing.
Cheers,
Aljoscha
On Fri, 23 Sep 2016 at 10:51 Stefan Richter
wrote:
>
I found out how to dump the stacktrace (using jps & jtrace). Please find
attached the stacktrace I got when the job got stuck.
Thanks,
Yassine
2016-09-23 11:48 GMT+02:00 Fabian Hueske :
> Yes, log files and stacktraces are different things.
> A stacktrace shows the call hierarchy of all threads
Yes, log files and stacktraces are different things.
A stacktrace shows the call hierarchy of all threads in a JVM at the time
when it is taken. So you can see the method that is currently executed (and
from where it was called) when the stacktrace is taken. In case of a
deadlock, you see where the
Hi Fabian,
Not sure if this answers your question, here is the stack I got when
debugging the combine and datasource operators when the job got stuck:
"DataSource (at main(BatchTest.java:28)
(org.apache.flink.api.java.io.TupleCsvInputFormat)) (1/8)"
at java.lang.Object.wait(Object.java)
at
org.ap
Hi Aljoscha & Fabian,
I have a stream application that has 2 stream source as below.
KeyedStream *ks1* = ds1.keyBy("*") ;
KeyedStream, String> *ks2* = ds2.flatMap(split T
into k-v pairs).keyBy(0);
ks1.connect(ks2).flatMap(X);
//X is a CoFlatMapFunction that inserts and re
Hi Yassine, can you share a stacktrace of the job when it got stuck?
Thanks, Fabian
2016-09-22 14:03 GMT+02:00 Yassine MARZOUGUI :
> The input splits are correctly assgined. I noticed that whenever the job
> is stuck, that is because the task *Combine (GroupReduce at
> first(DataSet.java:573)) *
Hi,
from the documentation of close(), the method is only called once at the end of
the lifecycle of a user function:
„Tear-down method for the user code. It is called after the last call to the
main working methods (e.g. map or join).“
If you want to perform tasks whenever a window triggers,
Hey,
I'm no expert at all, but for me this sounds like a use case for Complex Event
Processing (CEP). I don't know if you're aware of Flinks CEP Library [1, 2]?
Maybe that solves your problem of multiple firings. But best to wait for the
experts to answer your questions on handling state and fi
Hi,
in our project we're dealing with a stream of billing events. Each has
customerId and charge amount
We want to have a process that will trigger event (alarm) when sum of
charges for customer during last 4 hours exceeds certain threshold, say
- 10.
The triggered event should contain data fr
13 matches
Mail list logo
