RE: Stream enrichment with ingest mode

Hello, You’re right, one of our main use cases consist of adding missing fields, stored in a “small” reference table, periodically refreshed, to a stream. Using a broadcast stream and flink join was not the choice we made, because we didn’t want to add tricky watermarks and hold one stream (it

RE: Deploying the K8S operator sample on GKE Autopilot : Association with remote system [akka.tcp://flink@basic-example.default:6123] has failed,

hand on? Thanks, Arnaud From: LINZ, Arnaud Sent: Friday, January 12, 2024 2:07 PM To: user@flink.apache.org Subject: FW: Deploying the K8S operator sample on GKE Autopilot : Association with remote system [akka.tcp://flink@basic-example.default:6123] has failed, Hello, I am trying to follow the

FW: Deploying the K8S operator sample on GKE Autopilot : Association with remote system [akka.tcp://flink@basic-example.default:6123] has failed,

Hello, I am trying to follow the “quickstart” guide on a GKE Autopilot k8s cluster. https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/try-flink-kubernetes-operator/quick-start/ I could install the operator (without webhook) without issue ; however, when running kubectl c

RE: "Authentication failed" in "ConnectionState" when enabling internal SSL on Yarn with self signed certificate

Last update : My flink version is 1.14.3 in fact. The application works when enabling internal SSL in “local” intra-jvm cluster mode, so the certificate seems correct. I see no log in Yarn server side, only that the application get killed. I will try to take stack traces… De : LINZ, Arnaud

RE: "Authentication failed" in "ConnectionState" when enabling internal SSL on Yarn with self signed certificate

only error I get is a “timed out error” like the one you get when you don’t have enough ressources : (NoResourceAvailableException: Slot request bulk is not fulfillable! Could not allocate the required slot within slot request timeout) But I do have enough resources. De : LINZ, Arnaud Envoyé

"Authentication failed" in "ConnectionState" when enabling internal SSL on Yarn with self signed certificate

Hello, I use Flink 1.11.2 in Yarn cluster mode. I’ve followed the instructions listed here (https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/security/security-ssl/ ) to turn on

RE: [DISCUSS] FLIP-185: Shorter heartbeat timeout and interval default values

at some points… Somehow breaking the “slot number” contract). For the RAM cache, I believe that the hearbeat timeout may also times out because of a busy network. Cheers, Arnaud De : Till Rohrmann Envoyé : jeudi 22 juillet 2021 11:33 À : LINZ, Arnaud Cc : Gen Luo ; Yang Wang ; dev ; user

RE: [DISCUSS] FLIP-185: Shorter heartbeat timeout and interval default values

Hello, From a user perspective: we have some (rare) use cases where we use “coarse grain” datasets, with big beans and tasks that do lengthy operation (such as ML training). In these cases we had to increase the time out to huge values (heartbeat.timeout: 50) so that our app is not killed.

RE: Random Task executor shutdown

Hi, It’s 3.4.10 and does contain the bug. I’ll patch my flink client and see if it happens again. Best regards, Arnaud De : LINZ, Arnaud Envoyé : mercredi 18 novembre 2020 10:35 À : 'Guowei Ma' Cc : 'user' Objet : RE: Random Task executor shutdown Hello, We are won

RE: Random Task executor shutdown

Hello, We are wondering whether it is related to https://issues.apache.org/jira/browse/ZOOKEEPER-2775 or not. What is the version of the shaded zookeeper client in Flink 1.10.0 ? Best, Arnaud De : LINZ, Arnaud Envoyé : mercredi 18 novembre 2020 09:39 À : 'Guowei Ma' Cc : user

RE: Random Task executor shutdown

1) and the last one, there is only one minute. Is there other parameters to adjust to make the Zookeeper synchronization more robust when the network is slowed down ? Best, Arnaud De : Guowei Ma Envoyé : mardi 17 novembre 2020 00:49 À : LINZ, Arnaud Cc : user Objet : Re: Random Task executo

Random Task executor shutdown

(reposted with proper subject line -- sorry for the copy/paste) -Original message- Hello, I'm running Flink 1.10 on a yarn cluster. I have a streaming application, that, when under heavy load, fails from time to time with this unique error message in the whole yarn log: (...) 2020-11-15

RE: Re: Flink 1.11 not showing logs

Hello, I'm running Flink 1.10 on a yarn cluster. I have a streaming application, that, when under heavy load, fails from time to time with this unique error message in the whole yarn log: (...) 2020-11-15 16:18:42,202 WARN org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Receiv

RE: Best way to "emulate" a rich Partitioner with open() and close() methods ?

“intermediate” technical info in the stream payload. Best, Arnaud De : Robert Metzger Envoyé : vendredi 29 mai 2020 13:10 À : LINZ, Arnaud Cc : user Objet : Re: Best way to "emulate" a rich Partitioner with open() and close() methods ? Hi Arnaud, Maybe I don't fully

Best way to "emulate" a rich Partitioner with open() and close() methods ?

Hello, I would like to upgrade the performance of my Apache Kudu Sink by using the new “KuduPartitioner” of Kudu API to match Flink stream partitions with Kudu partitions to lower the network shuffling. For that, I would like to implement something like stream.partitionCustom(new KuduFlinkPa

RE: Building with Hadoop 3

Hello, Have you shared it somewhere on the web already? Best, Arnaud De : vino yang Envoyé : mercredi 4 décembre 2019 11:55 À : Márton Balassi Cc : Chesnay Schepler ; Foster, Craig ; user@flink.apache.org; d...@flink.apache.org Objet : Re: Building with Hadoop 3 Hi Marton, Thanks for your exp

How to fully re-aggregate a keyed windowed aggregate in the same window ?

Hello, I would like to compute statistics on a stream every hour. For that, I need to compute statistics on the keyed stream, then to reaggregate them. I’ve tried the following thing : stream.keyBy(mykey) .window(1 hour process time) .aggregate(my per-key aggregate)

RE: No yarn option in self-built flink version

-Pinclude-hadoop stated in the above paragraph… Arnaud De : Ufuk Celebi Envoyé : vendredi 7 juin 2019 12:00 À : LINZ, Arnaud Cc : user ; ches...@apache.org Objet : Re: No yarn option in self-built flink version Hey Arnaud, I think you need to active the Hadoop profile via -Pinclude-hadoop (the

No yarn option in self-built flink version

Hello, I am trying to build my own flink distribution with proper Cloudera dependencies. Reading https://ci.apache.org/projects/flink/flink-docs-stable/flinkDev/building.html I've done : git clone https://github.com/apache/flink cd flink git checkout tags/release-1.8.0 $MAVEN_HOME/bin/mvn clean

RE: Checkpoints and catch-up burst (heavy back pressure)

Hi, I like the idea, will give it a try. Thanks, Arnaud De : Stephen Connolly Envoyé : mardi 5 mars 2019 13:55 À : LINZ, Arnaud Cc : zhijiang ; user Objet : Re: Checkpoints and catch-up burst (heavy back pressure) On Tue, 5 Mar 2019 at 12:48, Stephen Connolly mailto:stephen.alan.conno

Re: Checkpoints and catch-up burst (heavy back pressure)

01, 2019 7:05 PM +0100 A : "LINZ, Arnaud" CC : zhijiang , user Objet : Re: Checkpoints and catch-up burst (heavy back pressure) Hi Arnaud, 1. What’s your checkpoint configuration? Wondering if you’re writing to HDFS, and thus the load you’re putting on it while catching up & ch

RE: Checkpoints and catch-up burst (heavy back pressure)

detecting the catchup mode and a max file per sec limitation when this occupe, slightly lower than the natural catchup rate. Message d'origine De : "LINZ, Arnaud" Date : ven., mars 01, 2019 2:04 PM +0100 A : zhijiang , user Objet : RE: Checkpoints and catch-

RE: Checkpoints and catch-up burst (heavy back pressure)

the ungraceful source sleep(), is there a way, programmatically, to know the “load” of the app, or to determine if checkpointing takes too much time, so that I can do it only on purpose? Thanks, Arnaud De : zhijiang Envoyé : vendredi 1 mars 2019 04:59 À : user ; LINZ, Arnaud Objet : Re: Checkpo

RE: Checkpoints and catch-up burst (heavy back pressure)

Update : Option 1 does not work. It still fails at the end of the timeout, no matter its value. Should I implement a “bandwidth” management system by using artificial Thread.sleep in the source depending on the back pressure ? De : LINZ, Arnaud Envoyé : jeudi 28 février 2019 15:47 À

RE: Checkpoints and catch-up burst (heavy back pressure)

names, everything else is local to the node). Should I adjust their values nonetheless ? To higher or lower values ? Best, Arnaud De : zhijiang Envoyé : jeudi 28 février 2019 10:58 À : user ; LINZ, Arnaud Objet : Re: Checkpoints and catch-up burst (heavy back pressure) Hi Arnaud, I think there

Checkpoints and catch-up burst (heavy back pressure)

Hello, I have a simple streaming app that get data from a source and store it to HDFS using a sink similar to the bucketing file sink. Checkpointing mode is “exactly once”. Everything is fine on a “normal” course as the sink is faster than the source; but when we stop the application for a whil

RE: Kerberos error when restoring from HDFS backend after 24 hours

Hi, I've managed to correct this by implementing my own FsStateBackend based on the original one with proper Kerberos relogin in createCheckpointStorage(). Regards, Arnaud -Message d'origine- De : LINZ, Arnaud Envoyé : vendredi 4 janvier 2019 11:32 À : user Objet : Kerberos

Kerberos error when restoring from HDFS backend after 24 hours

Hello and happy new year to all flink users, I have a streaming application (flink v1.7.0) on a Kerberized cluster, using a flink configuration file where the following parameters are set : security.kerberos.login.use-ticket-cache: false security.kerberos.login.keytab: X security.kerberos.lo

RE: Apache Flink 1.7.0 jar complete ?

two different computers… De : Till Rohrmann Envoyé : vendredi 30 novembre 2018 15:04 À : LINZ, Arnaud Cc : user Objet : Re: Apache Flink 1.7.0 jar complete ? Hi Arnaud, I tried to setup the same testing project as you've described and it worked for me. Could you maybe try to clear

Apache Flink 1.7.0 jar complete ?

Hi, When trying to update to 1.7.0, a simple local cluster test fails with : 12:03:55.182 [main] DEBUG o.a.f.s.a.graph.StreamGraphGenerator - Transforming SinkTransformation{id=2, name='Print to Std. Out', outputType=GenericType, parallelism=1} 12:03:55.182 [main] DEBUG o.a.f.s.a.graph.StreamGr

RE: Stopping a streaming app from its own code : behaviour change from 1.3 to 1.6

1.FLINK-10832<https://issues.apache.org/jira/browse/FLINK-10832> Created (with heavy difficulties as typing java code in a jira description was an awful experience ☺) De : LINZ, Arnaud Envoyé : mercredi 7 novembre 2018 11:43 À : 'user' Objet : RE: Stopping a streaming app f

RE: Stopping a streaming app from its own code : behaviour change from 1.3 to 1.6

FYI, the code below ends with version 1.6.0, do not end in 1.6.1. I suspect it’s a bug instead of a new feature. De : LINZ, Arnaud Envoyé : mercredi 7 novembre 2018 11:14 À : 'user' Objet : RE: Stopping a streaming app from its own code : behaviour change from 1.3 to 1.6 Hello,

RE: Stopping a streaming app from its own code : behaviour change from 1.3 to 1.6

ager leader changed to new address null. (org.apache.flink.runtime.jobmaster.JobMaster:1355) [2018-11-07 11:11:23,607] INFO Stop job leader service. (org.apache.flink.runtime.taskexecutor.JobLeaderService:135) [2018-11-07 11:11:23,608] INFO Stopped TaskExecutor akka://flink/user/taskmanager

Stopping a streaming app from its own code : behaviour change from 1.3 to 1.6

Hello, In flink 1.3, I was able to make a clean stop of a HA streaming application just by ending the source “run()” method (with an ending condition). I try to update my code to flink 1.6.2, but that is no longer working. Even if there are no sources and no item to process, the cluster con

RE: How to handle multiple yarn sessions and choose at runtime the one to submit a ha streaming job ?

want, but I think it should be made simpler... I'll open a Jira. Best regards, Arnaud De : LINZ, Arnaud Envoyé : jeudi 1 février 2018 16:23 À : user@flink.apache.org Objet : How to handle multiple yarn sessions and choose at runtime the one to submit a ha streaming job ? Hello, I am

How to handle multiple yarn sessions and choose at runtime the one to submit a ha streaming job ?

Hello, I am using Flink 1.3.2 and I'm struggling to achieve something that should be simple. For isolation reasons, I want to start multiple long living yarn session containers (with the same user) and choose at run-time, when I start a HA streaming app, which container will hold it. I start m

RE: How to stop FlinkKafkaConsumer and make job finished?

Hi, My 2 cents: not being able to programmatically nicely stop a Flink stream is what lacks most to the framework IMHO. It's a very common use case: each time you want to update the application or change its configuration you need to nicely stop & restart it, without triggering alerts, data lo

RE: OutOfMemory when looping on dataset filter

when starting the YARN session, and not having them in the user code jar file. That way, they need not be reloaded for each job. Greetings, Stephan On Fri, Dec 9, 2016 at 11:30 AM, LINZ, Arnaud mailto:al...@bouyguestelecom.fr>> wrote: Hi, Caching could have been a solution. Another o

RE: OutOfMemory when looping on dataset filter

all days of the same size? 3. Is the OOM happening in Flink or in the client process which fetches the result? Best, Fabian 2016-12-09 10:35 GMT+01:00 LINZ, Arnaud mailto:al...@bouyguestelecom.fr>>: Hello, I have a non-distributed treatment to apply to a DataSet of timed events, one da

OutOfMemory when looping on dataset filter

Hello, I have a non-distributed treatment to apply to a DataSet of timed events, one day after another in a flink batch. My algorithm is: // wholeSet is too big to fit in RAM with a collect(), so we cut it in pieces DataSet wholeSet = [Select WholeSet]; for (day 1 to 31) { List<>

RE: Collect() freeze on yarn cluster on strange recover/deserialization error

re 2016 14:43 À : LINZ, Arnaud ; user@flink.apache.org Cc : rmetz...@apache.org Objet : RE: Collect() freeze on yarn cluster on strange recover/deserialization error Good point with the collect() docs. Would you mind opening a JIRA issue for that? I'm not sure whether you can specify it via th

RE: Collect() freeze on yarn cluster on strange recover/deserialization error

-Message d'origine- De : Ufuk Celebi [mailto:u...@apache.org] Envoyé : jeudi 8 décembre 2016 10:49 À : LINZ, Arnaud ; user@flink.apache.org Objet : RE: Collect() freeze on yarn cluster on strange recover/deserialization error I also don't get why the job is recovering, but the oversized

RE: Collect() freeze on yarn cluster on strange recover/deserialization error

ment() but ExecutionEnvironment.getExecutionEnvironment(). Do I have to change the way I'm doing things ? How ? Thanks, Arnaud -Message d'origine----- De : LINZ, Arnaud Envoyé : mercredi 30 novembre 2016 08:59 À : user@flink.apache.org Objet : RE: Collect() freeze on yarn cluste

RE: Collect() freeze on yarn cluster on strange recover/deserialization error

g. Full log is availlable here : https://ftpext.bouyguestelecom.fr/?u=JDhCUdcAImsANZQdys86yID6UNq8H2r Thanks, Arnaud -Message d'origine- De : Ufuk Celebi [mailto:u...@apache.org] Envoyé : mardi 29 novembre 2016 18:43 À : LINZ, Arnaud ; user@flink.apache.org Objet : Re: Collect() freez

Collect() freeze on yarn cluster on strange recover/deserialization error

Hello, I have a Flink 1.1.3 batch application that makes a simple aggregation but freezes when collect() is called when the app is deployed on a ha-enabled yarn cluster (it works on a local cluster). Just before it hangs, I have the following deserialization error in the logs : (...) 2016-11-29

RE: How to stop job through java API

Hi, I have a similar issue. Here is how I deal with programmatically stopping permanent streaming jobs, and I’m interested in knowing if there is a better way now. Currently, I use hand-made streaming sources that periodically check for some flag and end if a stop request was made. Stopping th

RE: Flink 1.1.0 : Hadoop 1/2 compatibility mismatch

blem with the Maven > artifacts of 1.1.0 :-( I've added a warning to the release note and > will start a emergency vote for 1.1.1 which only updates the Maven > artifacts. > > On Tue, Aug 9, 2016 at 9:45 AM, LINZ, Arnaud wrote: >> Hello, >> >> >> >&g

RE: Classloader issue using AvroParquetInputFormat via HadoopInputFormat

Okay, That would also solve my issue. Greetings, Arnaud De : Stephan Ewen [mailto:se...@apache.org] Envoyé : mardi 9 août 2016 12:41 À : user@flink.apache.org Objet : Re: Classloader issue using AvroParquetInputFormat via HadoopInputFormat Hi Shannon! It seams that the something in the maven dep

Flink 1.1.0 : Hadoop 1/2 compatibility mismatch

Hello, I’ve switched to 1.1.0, but part of my code doesn’t work any longer. Despite the fact that I have no Hadoop 1 jar in my dependencies (2.7.1 clients & flink-hadoop-compatibility_2.10 1.1.0), I have a weird JobContext version mismatch error, that I was unable to understand. Code is a hive

RE: Yarn batch not working with standalone yarn job manager once a persistent, HA job manager is launched ?

set at the containers. We might change this for the 1.1.0 release. It should work if you put "yarn.properties-file.location: /custom/location" in your flink-conf.yaml before you execute "./bin/flink". Cheers, Max On Wed, Jun 15, 2016 at 3:14 PM, LINZ, Arnaud wrote: > Ooopsss

RE: Yarn batch not working with standalone yarn job manager once a persistent, HA job manager is launched ?

Ooopsss My mistake, snapshot/restore do works in a local env, I've had a weird configuration issue! But I still have the property file path issue :) -Message d'origine- De : LINZ, Arnaud Envoyé : mercredi 15 juin 2016 14:35 À : 'user@flink.apache.org' Objet 

RE: Yarn batch not working with standalone yarn job manager once a persistent, HA job manager is launched ?

does not lead to recovery but ends the execution, despite the streamExecEnv.enableCheckpointing(timeout) call. Is there a way to locally test this mechanism (other than poorly simulating it by explicitly calling snapshot & restore in a overridden source) ? Thanks, Arnaud -Message d'

RE: Yarn batch not working with standalone yarn job manager once a persistent, HA job manager is launched ?

HA job manager is launched ? Thanks for clarification. I think it might be related to the YARN properties file, which is still being used for the batch jobs. Can you try to delete it between submissions as a temporary workaround to check whether it's related? – Ufuk On Mon, Jun 6, 2016 at 3:18 PM,

RE: Yarn batch not working with standalone yarn job manager once a persistent, HA job manager is launched ?

the batch jobs, e.g. -yDrecovery.mode=standalone Can you try this? – Ufuk On Mon, Jun 6, 2016 at 2:19 PM, LINZ, Arnaud wrote: > Hi, > > > > I use Flink 1.0.0. I have a persistent yarn container set (a > persistent flink job manager) that I use for streaming jobs ; and I > use the

Yarn batch not working with standalone yarn job manager once a persistent, HA job manager is launched ?

Hi, I use Flink 1.0.0. I have a persistent yarn container set (a persistent flink job manager) that I use for streaming jobs ; and I use the “yarn-cluster” mode to launch my batches. I’ve just switched “HA” mode on for my streaming persistent job manager and it seems to works ; however my batc

RE: TimeWindow not getting last elements any longer with flink 1.0 vs 0.10.1

Flink. I hope this helps you to understand the failing test cases. Cheers, Till ​ On Mon, Mar 14, 2016 at 1:14 PM, LINZ, Arnaud mailto:al...@bouyguestelecom.fr>> wrote: Hello, I’ve switched my Flink version from 0.10.1 to 1.0 and I have a regression in some of my unit tests. To narr

TimeWindow not getting last elements any longer with flink 1.0 vs 0.10.1

Hello, I’ve switched my Flink version from 0.10.1 to 1.0 and I have a regression in some of my unit tests. To narrow the problem, here is what I’ve figured out: - I use a simple Streaming application with a source defined as “fromElements("Element 1", "Element 2", "Element 3") -

Quick question about enableObjectReuse()

Hi, I just want to be sure : when I set enableObjectReuse, I don’t need to create copies of objects that I get as input and return as output but which I don’t keep inside my user function ? For instance, if I want to join Tuple2(A,B) with C into Tuple3(A,B,C) using a Join function, I can write

RE: Left join with unbalanced dataset

Message d'origine- De : Ufuk Celebi [mailto:u...@apache.org] Envoyé : mardi 2 février 2016 15:27 À : user@flink.apache.org Objet : Re: Left join with unbalanced dataset > On 02 Feb 2016, at 15:15, LINZ, Arnaud wrote: > > Hi, > > Running again with more RAM made the treate

RE: Left join with unbalanced dataset

Hi, Running again with more RAM made the treatement go further, but Yarn still killed one container for memory consumption. I will experiment various memory parameters. How do I retrieve the log of a specific task manager post-mortem? I don't use a permanent Flink/Yarn container (it's killed u

RE: Left join with unbalanced dataset

nel.nio.NioEventLoop.run(NioEventLoop.java:357) -Message d'origine- De : Ufuk Celebi [mailto:u...@apache.org] Envoyé : mardi 2 février 2016 13:52 À : user@flink.apache.org Objet : Re: Left join with unbalanced dataset > On 02 Feb 2016, at 13:28, LINZ, Arnaud wrote: > > Thanks, > I’

RE: Left join with unbalanced dataset

0.10-SNAPSHOT. If that turns out to be the cause, the good news is that we started discussing a 0.10.2 maintenance release that should also have a fix for that. Greetings, Stephan On Tue, Feb 2, 2016 at 11:12 AM, LINZ, Arnaud mailto:al...@bouyguestelecom.fr>> wrote: Hi, Changing for a ou

RE: Left join with unbalanced dataset

:58:54,471 INFO org.apache.flink.yarn.YarnJobManager De : LINZ, Arnaud Envoyé : lundi 1 février 2016 09:40 À : user@flink.apache.org Objet : RE: Left join with unbalanced dataset Hi, Thanks, I can’t believe I missed the outer join operators… Will try them and will keep you informed. I use the

RE: Left join with unbalanced dataset

ache.org/projects/flink/flink-docs-release-0.10/apis/programming_guide.html#transformations Regards, Chiwan Park > On Jan 30, 2016, at 6:43 PM, LINZ, Arnaud > mailto:al...@bouyguestelecom.fr>> wrote: > > Hello, > > I have a very big dataset A to left join with a dataset B

Left join with unbalanced dataset

Hello, I have a very big dataset A to left join with a dataset B that is half its size. That is to say, half of A records will be matched with one record of B, and the other half with null values. I used a CoGroup for that, but my batch fails because yarn kills the container due to memory prob

RE: Crash in a simple "mapper style" streaming app likely due to a memory leak ?

eap memory in the config? - What parallelism do you run the job with? - What Java and Flink versions are you using? Even better, can you paste the first part of the TaskManager's log (where it prints the environment) here? Thanks, Stephan On Mon, Dec 14, 2015 at 9:57 AM, LINZ, Ar

RE: Crash in a simple "mapper style" streaming app likely due to a memory leak ?

have added a “PartitionBy” in my stream process before the sink and my app is no longer a simple “mapper style” app. Do you known the cause of the error and how to correct it ? Best regards, Arnaud De : LINZ, Arnaud Envoyé : vendredi 13 novembre 2015 15:49 À : 'user@flink.apache.org&#x

RE: HA Mode and standalone containers compatibility ?

ith the next 0.10.2 release. @Robert: are you aware of another workaround for the time being? On Thu, Dec 3, 2015 at 1:20 PM, LINZ, Arnaud mailto:al...@bouyguestelecom.fr>> wrote: Hi, It works fine with that file renamed. Is there a way to specify its path for a specific execution to

RE: HA Mode and standalone containers compatibility ?

e: HA Mode and standalone containers compatibility ? > On 03 Dec 2015, at 11:47, LINZ, Arnaud wrote: > > Oopss... False joy. OK, I think this is a bug in the YARN Client and the way it uses the .properties files to submit jobs. As a work around: Can you mv the /tmp/.yarn-properties-vo

RE: HA Mode and standalone containers compatibility ?

tating that I have only 24 slots on my 48 container cluster ! -Message d'origine- De : LINZ, Arnaud Envoyé : jeudi 3 décembre 2015 11:26 À : user@flink.apache.org Objet : RE: HA Mode and standalone containers compatibility ? Hi, The batch job does not need to be HA. I stopped everythin

RE: HA Mode and standalone containers compatibility ?

.root: /flink-streaming-1 # for the streaming session recovery.zookeeper.path.root: /flink-batch # for the batch session – Ufuk > On 03 Dec 2015, at 11:01, LINZ, Arnaud wrote: > > Yes, it does interfere, I do have additional task managers. My batch > application comes in my streaming cl

RE: HA Mode and standalone containers compatibility ?

\=h1r1en01\:2181@@recovery.zookeeper.path.root\=/flink@@state.backend\=filesystem@@state.backend.fs.checkpointdir\=hdfs\:///tmp/flink/checkpoints@@recovery.zookeeper.storageDir\=hdfs\:///tmp/flink/recovery/ jobManager=172.21.125.28\:36700 De : LINZ, Arnaud Envoyé : jeudi 3 décembre 2015 11:01 À

RE: HA Mode and standalone containers compatibility ?

nnected to the streaming cluster when you execute the batch job. Could you check that? Furthermore, could you check that actually a second yarn application is started when you run the batch jobs? Cheers, Till On Thu, Dec 3, 2015 at 9:57 AM, LINZ, Arnaud mailto:al...@bouyguestelecom.fr>> wro

HA Mode and standalone containers compatibility ?

Hello, I have both streaming applications & batch applications. Since the memory needs are not the same, I was using a long-living container for my streaming apps and new short-lived containers for hosting each batch execution. For that, I submit streaming jobs with "flink run" and batch j

Way to get accumulators values *during* job execution ?

Hello, I use Grafana/Graphite to monitor my applications. The Flink GUI is really nice, but it disappears after the job completes and consequently is not suitable to long-term monitoring. For batch applications, I simply send the accumulator’s values at the end of the job to my Graphite base.

Flink Streaming Core 0.10 in maven repos

Hello, Small question: I can't find the Streaming Core component in 0.10 version in the maven repo : http://mvnrepository.com/artifact/org.apache.flink/flink-streaming-core Thus in my pom file this artifact is the only part of my Flink's dependencies to stay in 0.10-SNAPSHOT version. Is there s

RE: Crash in a simple "mapper style" streaming app likely due to a memory leak ?

bit too high .. but I want to make sure that we first identify the issue. If the job is running with 50% cutoff, you can try to reduce it again towards 25% (that's the default value, unlike the documentation says). I hope that helps. Regards, Robert On Fri, Nov 13, 2015 at 2:58 PM, LINZ,

RE: Join Stream with big ref table

problem and with a lot of free mem… Greetings, Arnaud De : LINZ, Arnaud Envoyé : jeudi 12 novembre 2015 17:48 À : 'user@flink.apache.org' Objet : Join Stream with big ref table Hello, I have to enrich a stream with a big reference table (11,000,000 rows). I cannot use “join” because I can

Join Stream with big ref table

Hello, I have to enrich a stream with a big reference table (11,000,000 rows). I cannot use “join” because I cannot window the stream ; so in the “open()” function of each mapper I read the content of the table and put it in a HashMap (stored on the heap). 11M rows is quite big but it should t

RE: Multiple keys in reduceGroup ?

Thu, Oct 22, 2015 at 1:30 PM, LINZ, Arnaud mailto:al...@bouyguestelecom.fr>> wrote: Hi, I was using primitive types, and EnableObjectReuse was turned on. My next move was to turn it off, and it did solved the problem. It also increased execution time by 10%, but it’s hard to say i

RE: Multiple keys in reduceGroup ?

als()" ? > > The key objects will most certainly be different in each record (as they are > deserialized individually), but they should be equal. > > Stephan > > > On Thu, Oct 22, 2015 at 12:20 PM, LINZ, Arnaud > mailto:al...@bouyguestelecom.fr>> wrote: >

Multiple keys in reduceGroup ?

Hello, Trying to understand why my code was giving strange results, I’ve ended up adding “useless” controls in my code and came with what seems to me a bug. I group my dataset according to a key, but in the reduceGroup function I am passed values with different keys. My code has the following

RE: Flink batch runs OK but Yarn container fails in batch mode with -m yarn-cluster

obre 2015 14:34 À : user@flink.apache.org; LINZ, Arnaud Objet : Re: Flink batch runs OK but Yarn container fails in batch mode with -m yarn-cluster Hi Arnaud, I've looked into the problem but I couldn't reproduce it using Flink 0.9.0, Flink 0.9.1 and the current master snapshot (f3

Flink batch runs OK but Yarn container fails in batch mode with -m yarn-cluster

Hi, I am wondering why, despite the fact that my java main() methods runs OK and exit with 0 code value, the Yarn container status set by the englobing flink execution is FAILED with diagnostic "Flink YARN Client requested shutdown."? Command line : flink run -m yarn-cluster -yn 20 -ytm 81

RE: How to force the parallelism on small streams?

t. It distributes the emitted tuples in a round robin fashion to all consumer tasks. -Matthias On 09/02/2015 05:41 PM, LINZ, Arnaud wrote: > Hi, > > > > I have a source that provides few items since it gives file names to > the mappers. The mapper opens the file and proces

How to force the parallelism on small streams?

Hi, I have a source that provides few items since it gives file names to the mappers. The mapper opens the file and process records. As the files are huge, one input line (a filename) gives a consequent work to the next stage. My topology looks like : addSource(myFileSource).rebalance().setParal

RE: Best way for simple logging in jobs?

ly (when retrieving the logs from the log aggregation). Regards, Robert On Fri, Aug 28, 2015 at 3:55 PM, LINZ, Arnaud mailto:al...@bouyguestelecom.fr>> wrote: Hi, I am wondering if it’s possible to get my own logs inside the job functions (sources, mappers, sinks…). It would be nice

RE: "Flink YARN Client requested shutdown" in flink -m yarn-cluster mode?

licationId . We need to find out why the TaskManagers are shutting down. That is most likely logged in the TaskManager logs. On Fri, Aug 28, 2015 at 10:57 AM, LINZ, Arnaud mailto:al...@bouyguestelecom.fr>> wrote: Hello, I’ve moved my version from 0.9.0 and tried both 0.9-SNAPSHOT &a

Best way for simple logging in jobs?

Hi, I am wondering if it’s possible to get my own logs inside the job functions (sources, mappers, sinks…). It would be nice if I could get those logs in the Yarn’s logs, but writing System.out/System.err has no effect. For now I’m using a “StringBuffer” accumulator but it does not work in

"Flink YARN Client requested shutdown" in flink -m yarn-cluster mode?

Hello, I’ve moved my version from 0.9.0 and tried both 0.9-SNAPSHOT & 0.10-SNAPSHOT to continue my batch execution on my secured cluster thanks to [FLINK-2555]. My application works nicely in local mode and also in yarn mode using a job container started with yarn-session.sh, but it fails in

RE: HadoopDataOutputStream maybe does not expose enough methods of org.apache.hadoop.fs.FSDataOutputStream

this? Greetings, Stephan On Wed, Aug 26, 2015 at 6:23 PM, LINZ, Arnaud mailto:al...@bouyguestelecom.fr>> wrote: Hi, I’ve noticed that when you use org.apache.flink.core.fs.FileSystem to write into a hdfs file, calling org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(), i

HadoopDataOutputStream maybe does not expose enough methods of org.apache.hadoop.fs.FSDataOutputStream

Hi, I’ve noticed that when you use org.apache.flink.core.fs.FileSystem to write into a hdfs file, calling org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(), it returns a HadoopDataOutputStream that wraps a org.apache.hadoop.fs.FSDataOutputStream (under its org.apache.hadoop.hdfs.clie

RE: [0.10-SNAPSHOT ] When naming yarn application (yarn-session -nm), flink run without -m fails.

ument, because it is reading it from the conf/.yarn-session.properties file. Give me a few minutes to reproduce the issue. On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud mailto:al...@bouyguestelecom.fr>> wrote: Hi, Using last nightly build, it seems that if you call yarn-session.sh with -nm

[0.10-SNAPSHOT ] When naming yarn application (yarn-session -nm), flink run without -m fails.

Hi, Using last nightly build, it seems that if you call yarn-session.sh with -nm option to give a nice application name, then you cannot submit a job with flink run without specify the ever changing -m address since it does not find it any longer. Regards, Arnaud _

Source & job parallelism

Hi, I have a streaming source that extends RichParallelSourceFunction, but for some reason I don’t want parallelism at the source level, so I use : Env.setSource(mySource).setParrellelism(1).map(mymapper) I do want parallelism at the mapper level, because it’s a long task, and I would like the

Using HadoopInputFormat files from Flink/Yarn in a secure cluster gives an error

Hello, My application handles as input and output some HDFS files in the jobs and in the driver application. It works in local cluster mode, but when I’m trying to submit it to a yarn client, when I try to use a HadoopInputFormat (that comes from a HCatalog request), I have the following error:

Streaming window : count with timeout ?

Hello, The data in my stream have a timestamp that may be slightly out of order, but I need to process the data in the proper order. To do this, I use a windowing function and sort the items in a flatMap. However, the source may sometimes send data in “bulk batches” and sometimes “on the fly”.

getIndexOfThisSubtask : starts at 0 or 1 ?

Hello, According to the documentation, getIndexOfThisSubtask starts from 1; /** * Gets the number of the parallel subtask. The numbering starts from 1 and goes up to the parallelism, * as returned by {@link #getNumberOfParallelSubtasks()}. * * @return

No accumulator results in streaming

Hello, I’m struggling with this simple issue for hours now : I am unable to get the accumulator result of a streaming context result, the accumulator map in the JobExecutionResult is always empty. Simple test code (directly inspired from the documentation) : My source = public static clas

RE: How to cancel a Flink DataSource from the driver code?

http://localhost:8081/jobsInfo?get=cancel&job=68c53a77f11d34695ac1aea4f098af82 But I don't think there is a way to submit a topology in a non-blocking way, so that env.execute() returns immediately with the JobId. On Thu, Jul 2, 2015 at 9:35 AM, LINZ, Arnaud mailto:al...@bouyguestelec

  1   2   >