Hi Bruno,
>From the code I conclude that "akka.client.timeout" setting is what affects
this. It defaults to 60 seconds.
I'm not sure why this setting is not documented though as well as many
other "akka.*" settings - maybe there are some good reasons behind.
Regards,
Yury
2017-01-31 17:47 GMT+0
link 1.2 with respect to class loading
>> and standalone clusters. Did you put any JARs into the lib folder of
>> Flink before submitting the job?
>>
>> – Ufuk
>>
>> On Thu, Jan 12, 2017 at 7:16 PM, Yury Ruchin
>> wrote:
>> > Hi,
>> >
>>
Hi,
I'd like to chime in since I've faced the same issue running Flink 1.1.4. I
have a long-running YARN session which I use to run multiple streaming jobs
concurrently. Once after cancelling and resubmitting the job I saw the "X
cannot be cast to X" ClassCastException exception in logs. I restart
Hi,
Is there a strict requirement that elements must proceed along the
processing pipeline exactly after being accounted by the reduce function?
If not, you could derive two streams from the original one to be processed
concurrently, something like this:
val protoStream = kafka source -> keyBy
v
Hi,
You containers got killed by YARN for exceeding virtual memory limits. For
some reason your container intensively allocate virtual memory while having
free physical memory.
There are some gotchas regarding such issue on CentOS, caused by
OS-specific aggressive virtual memory allocation: [1],
Hi,
I've faced a similar issue recently. Hope sharing my findings will help.
The problem can be split into 2 parts:
*Source of container failures*
The logs you provided indicate that YARN kills its containers for exceeding
memory limits. Important point here is that memory limit = JVM heap memory
Hi Stephan,
It looks like you have libraries from different versions of Flink
distribution on the same classpath.
ForkableFlinkMiniCluster resides in flink-test-utils. As of distribution
version 1.1.3 it invokes JobManager.startJobManagerActors() with 6
arguments. The signature changed by 1.1.4,
My bad, the "Records Out" in the previous message should be read "Records
sent" as per Flink UI.
2016-12-20 18:42 GMT+03:00 Yury Ruchin :
> Well, it seems I figured it out. You're right, Fabian, it works the way
> you described. I wrote
was some
inaccuracy of sampled numbers - "Records In" values in successor operators
were not exactly equal which made me think they receive different portions
of the stream. I believe the inaccuracy is somewhat intrinsic to live
stream sampling, so that's fine.
2016-12-20 14:35 GMT+03
should not be any non-deterministic behavior or splitting of records.
>
> Can you share some example code that produces the non-deterministic
> behavior?
>
> Best, Fabian
>
>
> 2016-12-20 10:50 GMT+01:00 Yury Ruchin :
>
>> Hi all,
>>
>> I have a streamin
Hi all,
I have a streaming job that I want at some point to duplicate stream, so
that one copy of it ends up in a sink, and another one goes down the
processing pipeline. This way I want to implement something similar to
"tee" Linux utility.
I tried a naive approach that looked like this:
val st
tor in your job aggregates the data, only the
> aggregates will be buffered.
> This might explain why the first operators of job are able to start
> processing while the FlatMap operator is still setting up itself.
>
> Best,
> Fabian
>
> 2016-12-17 13:42 GMT+01:00 Yury Ruchin :
>
Hi all,
I have a streaming job that essentially looks like this: KafkaSource -> Map
-> EventTimeWindow -> RichFlatMap -> CustomSink. The RichFlatMap part does
some heavy lifting in open(), so that the open() call blocks for several
minutes. I assumed that until open() returns the backpressure mech
er reason?
>
> On Thu, Dec 15, 2016 at 8:04 PM, Yury Ruchin
> wrote:
>
>> Hi,
>>
>> I'm seeing an issue with the load speed of Flink Web UI when running in
>> YARN session. Initial load takes several minutes or even more, although
>> according to the brows
Hi,
I'm seeing an issue with the load speed of Flink Web UI when running in
YARN session. Initial load takes several minutes or even more, although
according to the browser console there are only a couple of MBs to
download. When the loading is complete, the UI itself is quite responsive.
I don't
gt;>
>> *:*
>>
>>org/datanucleus/**
>>META-INF/*.SF
>>META-INF/*.DSA
>>META-INF/*.RSA
>>
>>
>>
>>
>>
Hi,
I have run into a classpath issue when running Flink streaming job in YARN
session. I package my app into a fat jar with all the dependencies needed.
One of them is Google Guava. I then submit the jar to the session. The task
managers pre-created by the session build their classpath from the
F
of each
> TaskManager.
>
>
> On Fri, Nov 25, 2016 at 10:53 AM, Yury Ruchin
> wrote:
>
>> Thanks Stefan! I think this would help if I had just one container per
>> node. But that's not my case - there are multiple TaskManagers running on
>> the same node,
you try adding the following to your flink.yaml?
>
> env.java.opts: -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.port=
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
>
> Best,
> Stefan
>
>
&g
Hi,
I want to enable JMX for my Flink streaming app running in YARN session.
How can I specify which ports containers will listen to? If I cannot
control it (e. g. they will be chosen randomly) - how can I detect which
ports were picked by containers: inspecting logs, looking at the Web UI
etc.?
Hello,
I'm writing a custom source function for my streaming job. The source
function manages some connection pool. I want to close that pool once my
job is "finished" (since the stream is unbounded, the only way I see is to
cancel the streaming job). Since I inherit RichSourceFunction, there are
21 matches
Mail list logo
