Re: Flink SSL Setup on a standalone cluster

How are you creating the remote environment? In particular, are passing a configuration to the RemoteEnvironment? Have you set the SSL options in the config? On 15.03.2018 22:46, Vinay Patil wrote: Hi, Even tried with ip-address for JobManager.host.name property,

Re: Change conf.yaml properties flink docker

Hi Miki, A custom image is not needed to do that. You can mount a directory containing a custom flink-conf.yaml [1], and set the environment variable FLINK_CONF_DIR to point to that directory [2][3]. Best, Gary [1] https://docs.docker.com/storage/volumes/ [2] https://docs.docker.com/engine/refer

Re: Can not cancel with savepoint with Flink 1.3.2

Is this related? 2018-03-16 03:43:42,557 INFO akka.actor.EmptyLocalActorRef - Message [org.apache.flink.runtime.metrics.dump.MetricDumpSerialization$MetricSerializationResult] from Actor[akka.tcp://flink@fps-flink-taskmanager-120318156-9sw8l:43048/user/MetricQueryService_dff29d2

Can not cancel with savepoint with Flink 1.3.2

Hi, I am running flink on K8S and store states in s3 with rocksdb backend. I used to be able to cancel and savepointing through the rest api. But sometimes the process never finish. No matter how many time I try. Is there a way to figure out what is going wrong? Why "isStoppable"=>false? Thanks

Slow flink checkpoint

Hi, I'm run a job on Flink streaming. I found with the increase in the number of 'InternalTimer' object the checkpoint more and more slowly. Is there any way to solve this problem ? such as make the "timeServiceManager" snapshot async. Thanks

documentation SEO

any google or duckduckgo search results in flink 1.3 version of the doc at The top of the results instead of 1.4 or latest.

Re: Flink SSL Setup on a standalone cluster

Hi, Even tried with ip-address for JobManager.host.name property, but did not work. When I tried netstat -anp | grep 6123 , I see 3 TM connection state as established, however when I submit the job , I see two more entries with state as TIME_WAIT and after some time these entries are gone and I ge

Re: Dependency Injection and Flink

Would it help to be able to register "initializers", meaning some classes/methods that will be called at every process entry point, to set up something like this? On Tue, Mar 13, 2018 at 7:56 PM, Steven Wu wrote: > Xiaochuan, > > We are doing exactly as you described. We keep the injector as a

[ANNOUNCE] Apache Flink 1.3.3 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.3.3, which is the third bugfix release for the Apache Flink 1.3 series.  Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming

Calling close() on Failure

Hello! I had a program lose a task manager the other day. The fail over back to a checkpoint and recovery worked like a charm. However, on one of my ProcessFunctions I defined a close() method and I noticed that it did not get called. To be clear, the task manager that failed was running that Proce

Re: flink sql: "slow" performance on Over Window aggregation with time attribute set to event time

Great! Thanks for reporting back :-) Buffer timeout of 0ms is quite aggressive. You might sending buffers of (by default) 32KB that just contain a single record. Anyway, now you know the nobs to tune the latency. Cheers, Fabian 2018-03-15 21:00 GMT+01:00 Yan Zhou [FDS Science] : > Hi Fabian, >

Re: flink sql: "slow" performance on Over Window aggregation with time attribute set to event time

Hi Fabian, Thank you for the information. After setting the watermark interval to 10ms and buffer timeout to 0 ms, the end-to-end latency is reduced to 5ms. I am very happy with the result and will go from there. Best Yan From: Fabian Hueske Sent: Thursda

Re: flink sql: "slow" performance on Over Window aggregation with time attribute set to event time

I see... Another issue might be the frequency with which you emit watermarks (in case you use a periodic watermark assigner). You can set the interval with StreamExecutionEnvironment.getConfig.setAutoWatermarkInterval() [1]. However, keep in mind that each watermark is an additional record which mi

Migration to Flip6 Kubernetes

Hello, Currently I have a Flink 1.4 cluster running on kubernetes based on the configuration describe on https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/deployment/kubernetes.html with additional config for HA with Zookeeper. With this I have several Taskmanagers, a single Jobmana

Re: flink sql: "slow" performance on Over Window aggregation with time attribute set to event time

Hi Fabian, Yes, it is typically not a good idea to generate watermark based on system time. I was setting the watermark based on system time with very little delay to see how fast my application could process the data. All the servers are sync with ntp and only 1ms difference with each other.

Re: TaskManager crashes with PageRank algorithm in Gelly

Termination of the TaskManager by the Linux OOM killer indicates an overallocation of memory and you have set "taskmanager.heap.mb: 139264” on machines with 136 GB. Even though you were able to (temporarily?) resolve the issue by enabling preallocation, you may see degraded performance if syste

Re: activemq connector not working..

I tried getting this in logs.. 2018-03-15 20:59:38,154 INFO org.apache.flink.streaming.runtime.tasks.StreamTask - No state backend has been configured, using default state backend (Memory / JobManager) 2018-03-15 20:59:38,296 INFO org.apache.flink.streaming.api.functions.source.Message

Re: activemq connector not working..

I tried in cluster as well . On Wed, Mar 14, 2018 at 10:01 PM, Timo Walther wrote: > Hi Puneet, > > are you running this job on the cluster or locally in your IDE? > > Regards, > Timo > > > Am 14.03.18 um 13:49 schrieb Puneet Kinra: > > Hi > > I used apache bahir connector below is the code.the

Re: Flink SSL Setup on a standalone cluster

Just an update, I am submitting the job from the master node, not using the normal flink run command to submit the job , but using Remote Execution Environment in code to do this. And in that I am passing the hostname which is same as provided in flink-conf.yaml Regards, Vinay Patil On Thu, Mar

Re: PartitionNotFoundException when restarting from checkpoint

Just to double check: We are talking about a Flink PartitionNotFoundException , I assume? The split brain situation is a good hint - the minority p

Re: Deserializing the InputFormat (org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormat@50bb10fd) failed: unread block data after upgrade to 1.4.2

we were jumping from version 1.3.1 (where everything worked fine) -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Deserializing the InputFormat (org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormat@50bb10fd) failed: unread block data after upgrade to 1.4.2

Hello, We have recently upgraded flink to version 1.4.2. Now our jobs that rely on Parquet/Avro files located on HDFS stopped working. I get exception: Caused by: org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task 'CHAIN DataSource (READING_RECORDS) -> Map (MAPPING_RECO

Change conf.yaml properties flink docker

HI , I want to change some flink-conf (jvm parameters ) properties that are default configured In the latest docker image . The only option that i can think of is creating custom image that will overwrite the current flink-conf file with new properties ? I basically want to create kubern

Re: Flink SSL Setup on a standalone cluster

Hi Guys, Any suggestions here Regards, Vinay Patil On Wed, Mar 14, 2018 at 8:08 PM, Vinay Patil wrote: > Hi, > > After waiting for some time I got the exception as Lost Connection to Job > Manager. Message: Could not retrieve the JobExecutionResult from Job Manager > > I am submitting the job

Continuous aggregation of results until end events matched CEP / ProcessWindowFunction ?

I watched a video which was published a while back by Matt Zimmer in Berlin Flink Forward (Sep 2017) titled Custom, Complex Windows at Scale using Apache Flink. https://www.youtube.com/watch?v=XUvqnsWm8yo In this window he analyzes a custom implementation of window that he implemented in order

Re: Implement a sort inside the WindowFunction

Hi Felipe, Just like the ReduceFunction, the WindowFunction is applied in the context of a single key. So, it will be called for each key and always just see a single record (the reduced record of the key). You'd have to add a non-keyed window (allWindow) for your sorting WindowFunction. Note that

Re: Too many open files on Bucketing sink

Hi, There is an open similar issue: https://issues.apache.org/jira/browse/FLINK-8707 It’s still under investigation and it would be helpful if you could follow up the discussion there, run same diagnostics commands as Alexander Gardner did (m

Re: Restart hook and checkpoint

If I understand fine-grained recovery correctly, one would still need to take checkpoints. Ashish would like to avoid checkpointing and accept to lose the state of the failed task. However, he would like to avoid losing more state than necessary due to restarting of tasks that did not fail. Best,