Re: Flink 1.4.0 can not override JAVA_HOME for single-job deployment on YARN

2017-12-17 Thread 杨光
Hi Stephan I tested my case again and found the parameters "-yD containerized.taskmanager.env.JAVA_HOME=/opt/jdk1.8.0_121 -yD containerized.master.env.JAVA_HOME=/opt/jdk1.8.0_121" and "-yD yarn.taskmanager.env.JAVA_HOME=/opt/jdk1.8.0_121 -yD containerized.master.env.JAVA_HOME=/opt/jdk1.8.0_1

Re: Can flink aggregate in local TM,then aggregate in global TM?

2017-12-17 Thread zhaifengwei
Hi Fabian Hueske-2, Thanks for your reply.This is exactly what I was looking for. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Flink vs Spark streaming benchmark

2017-12-17 Thread Flavio Pompermaier
This is very interesting and "educative" :) On Sun, Dec 17, 2017 at 4:39 PM, Kenny Gorman wrote: > Nice write up! > > -kg > > On Dec 16, 2017, at 4:59 PM, Fabian Hueske wrote: > > Hi, > > In case you haven't seen it yet. > Here's an analysis and response to Databricks' benchmark [1]. > > Best,

Re: aggregate does not allow RichAggregateFunction ?

2017-12-17 Thread Vishal Santoshi
Hello Fabian, We decided that it does not make sense to create partitioned kakka partitions b'coz of hot spot considerations. So we created a way to keep trimmed state in the Accumulator provided we know the current watermark to keep the trimmed state time correct. In essen

Re: JVM crash - SIGSEGV in ZIP_GetEntry

2017-12-17 Thread Gyula Fóra
Hi, I have seen similar errors when trying to serialize Kryo-typeserializers with Flink type infos accidentally. Maybe that helps :) Gyula On Sun, Dec 17, 2017, 15:52 Dawid Wysakowicz wrote: > Just as a follow-up I tried disabling mmap with > sun.zip.disableMemoryMapping, but it did not help.

Re: A question about Triggers

2017-12-17 Thread Vishal Santoshi
An addendum Is the element reference IN in onElement(IN element.. ) in Trigger, the same as IN the one provided to add(IN value) in Accumulator. It seems that any mutations to IN in the onElement() is not visible to the Accumulator that is carrying it as a previous element reference albeit in th

Re: Flink vs Spark streaming benchmark

2017-12-17 Thread Kenny Gorman
Nice write up! -kg > On Dec 16, 2017, at 4:59 PM, Fabian Hueske wrote: > > Hi, > > In case you haven't seen it yet. > Here's an analysis and response to Databricks' benchmark [1]. > > Best, Fabian > > [1] > https://data-artisans.com/blog/curious-case-broken-benchmark-revisiting-apache-flin

Re: JVM crash - SIGSEGV in ZIP_GetEntry

2017-12-17 Thread Dawid Wysakowicz
Just as a follow-up I tried disabling mmap with sun.zip.disableMemoryMapping, but it did not help. This time I got only Java stack: Stack: [0x7f9060757000,0x7f9060858000], sp=0x7f9060856350, free space=1020k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native

JVM crash - SIGSEGV in ZIP_GetEntry

2017-12-17 Thread Dawid Wysakowicz
Hi, Recently we observe regular taskmanager's JVM crashes just about a minute from the start of our Flink job. We run flink 1.3.2 on YARN (2.6.2.0-205). Java version: JRE version: Java(TM) SE Runtime Environment (8.0_112-b15) (build 1.8.0_112-b15) # Java VM: Java HotSpot(TM) 64-Bit Server VM (2

Re: Flink CEP state store

2017-12-17 Thread Dawid Wysakowicz
Hi Mahesh, You are right that it based on the Pattern you write there might be a big amount of intermittent states created. Flink CEP library keeps all its State in State Backend. So if you use e.g. RocksDBStateBackend it will be backed by disc. Nevertheless it is vital to allow clearing the s

Re: Flink vs Spark streaming benchmark

2017-12-17 Thread G.S.Vijay Raajaa
Thats awesome. Thanks for sharing. Regards, Vijay Raajaa G S On Sun, Dec 17, 2017 at 4:29 AM, Fabian Hueske wrote: > Hi, > > In case you haven't seen it yet. > Here's an analysis and response to Databricks' benchmark [1]. > > Best, Fabian > > [1] https://data-artisans.com/blog/curious-case-brok