Why assignTimestampsAndWatermarks parallelism same as map，it will not fired？

Hi all, I use the same parallelism between map and assignTimestampsAndWatermarks , and it not fired, I saw the extractTimestamp and generateWatermark all is fine, but watermark is always not change and keep as min long value. And then I changed parallelism and different with map, and windows f

Class loading issues when using Remote Execution Environment

I am trying to get gradoop_demo (a gradoop based graph visualization app) working on Flink with *Remote* Execution Environment. This app, which is based on Gradoop, submits a job to the *preconfigured* execution environment, collects the results and se

Testing Metrics

Hey guys and gals, Just wondering: Does anyone have an idea how to test if metrics are being generated? I have an integration test and I just added a processor to count elements in late arrivals (the general idea is to capture those, count and get an average, so we can adjust the allowedLateness)

elasticsearch5 , java.lang.NoClassDefFoundError on mesos

Hi , Im having some weird issue when running some stream job to ELK . The job i starting fine but after few hours im getting this exception and the TM/JB is crashed . this is the config for the elesticserch sink , may by 1 sec flush can cause the deadlock ?? config.put("bulk.flush.max.act

Trigger state clear

Hi I have some issue possibly memory issue that causing the task manager to crash . full code : https://gist.github.com/miko-code/6d7010505c3cb95be122364b29057237 I defined fire_and_purge on element and also evictor so state should be very small ... Any suggestion how figure this issue ? T

Re: Flink State monitoring

Anything to add? Is there a Jira ticket for this yet? On Fri, Apr 20, 2018 at 1:03 PM, Stefan Richter wrote: > If estimates are good enough, we should be able to expose something. Would > still like to double check the guarantees to see if the estimates of > RocksDB are helpful or could be misle

Flink + HDInsight Cluster Deployment

Hello everyone! My task is to install Flink on an HDInsight cluster in Azure. More specifically, I have installed a Kafka cluster (with preconfigured Kafka) as I would like to also combine Flink and Kafka. Unfortunately, Azure does not provide preconfigured cluster for Flink. So I have to in

Re: Use gradle with flink

You can certainly setup and build Flink applications with Gradle. However the bad news is, the Flink project does not provide a pre-configured Gradle project/configuration yet. The good news is, the community is working on that [1] and there's already a PR [2] (opened 19 hours ago). Btw. besides

Re: Unsure how to further debug - operator threads stuck on java.lang.Thread.State: WAITING

Hi James, it is unlikely that your issue is the same as the one Miguel is having. His one https://issues.apache.org/jira/browse/FLINK-9242 is probably the same as https://issues.apache.org/jira/browse/FLINK-9144 and happens only in batch programs spilling data in Flink 1.5 and 1.6 versions before l

Re: Use gradle with flink

Currently only maven build is supported. Original message From: Georgi Stoyanov Date: 4/24/18 2:17 AM (GMT-08:00) To: user@flink.apache.org Subject: Use gradle with flink Hi guys, I’m wondering is it possible to setup my java flink application with gradle? I’m confused c

Re: KafkaJsonTableSource purpose

Hi Sebastien, I think you can do that with Flink's Table API / SQL and the KafkaJsonTableSource. Note that in Flink 1.4.2, the KafkaJsonTableSource does not support flat JSON yet. You'd also need a table-valued UDFs for the parsing of the message and joining the result with the original row. Depen

Re: data enrichment with SQL use case

Hi Alex, An operator that has to join two input streams obviously requires two inputs. In case of an enrichment join, the operator should first read the meta-data stream and build up a data structure as state against which the other input is joined. If the meta data is (infrequently) updated, thes

Re: Checkpointing barriers

Hi Alex, That's correct. The n refers to the n-th checkpoint. The checkpoint ID is important, because operators need to align the barriers to ensure that they consumed all inputs up to the point, where the barriers were injected into the stream. Each operator checkpoints its own state. For sources

Use gradle with flink

Hi guys, I’m wondering is it possible to setup my java flink application with gradle? I’m confused cause everywhere win the stackoverflow/documentation is used maven only :/ Kind Regards, Georgi Stoyanov

Install Flink on Microsoft Azure HDInsight

Hello everyone! My task is to install Flink on an HDInsight cluster in Azure. More specifically, I have installed a Kafka cluster (with preconfigured Kafka) as I would like to also combine Flink and Kafka. Unfortunately, Azure does not provide preconfigured cluster for Flink. So I have to install

Re: Checkpointing barriers

ok, I got it. Barrier-n is an indicator or n-th checkpoint. My first impression was that barriers are carrying offset information, but it was wrong. Thanks for unblocking ;-) Alex

Re: How to run flip-6 on mesos

Its 1.4.2 ... Any approximate date for 1.5 release ? Thanks allot for your help . On Tue, Apr 24, 2018 at 10:39 AM, Gary Yao wrote: > Hi Miki, > > The stacktrace you posted looks familiar [1]. We have fixed the issue in > Flink > 1.5. What is the Flink version you are using? FLIP-6 before Fli

Re: How to run flip-6 on mesos

Hi Miki, The stacktrace you posted looks familiar [1]. We have fixed the issue in Flink 1.5. What is the Flink version you are using? FLIP-6 before Flink 1.5 is very experimental, and I doubt that it is in a usable state. Since 1.5 is not out yet, you can either compile the release branch yourself

Re: Testing on Flink 1.5

Hi Amit, web.timeout should only affect RPC calls originating from the REST API. In FLIP-6, the submission of the job graph happens via HTTP. The value under akka.ask.timeout is still used as the default timeout for RPC calls [1][2]. Since you also had custom heartbeats settings, you should consid

Re: How to run flip-6 on mesos

The problem is that the Web UI hasn't started at all Im using the sane config file that i used for none flip-6 is that ok ? Also i got this error in the logs . 2018-04-24 10:16:05,466 ERROR > org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Could not > recover the job graph for 4a

Re: How to run flip-6 on mesos

Hi Miki, IIRC the port on which the Web UI is listening is not allocated dynamically when deploying on Mesos, and should be 8081 by default (you can override the default by setting rest.port in flink-conf.yaml). If you can find out the hostname/IP of the JobManager, you can submit as usual via the