Passing vm options

Hi , I am trying to pass some vm options e.g bin/flink run foo-assembly-0.0.1-SNAPSHOT.jar -Dflink.stateDir=file:///tmp/ -Dkafka.bootstrap.servers="localhost:9092" -Dkafka.security.ssl.enabled=false but it doesn't seem to override the values in application.conf . Am I missing something? BTW is it p

Building Flink from source according to vendor-specific version but causes protobuf conflict

Hi guys, Good day. I rebuilt flink from the source and specified the vendor specific Hadoop version. It works well when i just submit a streaming application without '-d'(--detached) option as follows: bin/flink run -m yarn-cluster -yqu root.streaming -yn 5 -yjm 2048 -ytm 3096 -ynm CJVForma

onTimer function is not getting executed and job is marked as finished.

Hi All Facing some issue with context to onTimer method in processfunction class TimerTest extends ProcessFunction,String>{ /** * */ private static final long serialVersionUID = 1L; @Override public void processElement(Tuple2 arg0, ProcessFunction, String>.Context ctx, Collector arg2) throws Ex

Re: Buffer stats when Back Pressure is high

Hi Gagan, What flink version do you use? And have you checked the buffers.inputQueueLength for all the related parallelism (connected with A) of B? It may exist the scenario that only one parallelim B is full of inqueue buffers which back pressure A, and the input queue for other parallelism B

Re: Unable to restore the checkpoint on restarting the application!!

Hi Till Its Working for me know ,but *context.isRestored() **is always returning false.* On Fri, Jan 4, 2019 at 7:42 PM Till Rohrmann wrote: > When starting a job from within the IDE using the LocalEnvironment, it is > not possible to specify a checkpoint from which to resume. That's why your >

Re: ConnectTimeoutException when createPartitionRequestClient

Hi Wenrui, the code to set the connect timeout looks ok to me [1]. I also tested it locally and checked that the timeout is correctly registered in Netty's AbstractNioChannel [2]. Increasing the number of threads to 128 should not be necessary. But it could indicate that there is some long lastin

Re: Unable to restore the checkpoint on restarting the application!!

Hi Puneet, if context.isRestored returns false, then Flink did not resume from a checkpoint/savepoint. Please make sure that you specify the correct path the an existing checkpoint. Cheers, Till On Mon, Jan 7, 2019 at 11:04 AM Puneet Kinra < puneet.ki...@customercentria.com> wrote: > Hi Till >

Re: onTimer function is not getting executed and job is marked as finished.

Hi Puneet, The value of the registered timer should within startTime and endTime of your job. For example, job starts at processing time t1 and stops at processing time t2. You have to make sure t1< `parseLong + 5000` < t2. Best, Hequn On Mon, Jan 7, 2019 at 5:50 PM Puneet Kinra < puneet.ki...@c

Reducing runtime of Flink planner

Hi everybody, I have a question concerning the planner for the Flink Table / Batch API. At the moment I try to use a library called Cypher for Apache Flink, a project that tries to implement the graph database query language Cypher on Apache Flink (CAPF, https://github.com/soerenreichardt/cyp

Re: onTimer function is not getting executed and job is marked as finished.

Hi Hequn Its a streaming job . On Mon, Jan 7, 2019 at 5:51 PM Hequn Cheng wrote: > Hi Puneet, > > The value of the registered timer should within startTime and endTime of > your job. For example, job starts at processing time t1 and stops at > processing time t2. You have to make sure t1< `pars

Re: Buffer stats when Back Pressure is high

Flink Version is 1.7. Thanks Zhijiang for your pointer. Initially I was checking only for few. However I just checked for all and found couple of them having queue length of 40+ which seems to be due to skewness in data. Is there any general guide lines on how to handle skewed data? In my case I am

How to get the temp result of each dynamic table when executing Flink-SQL?

Hi Expert, Usually when we write Flink-SQL program, usually we need to use multiple tables to get the final result, this is due to sometimes it is not possible to implement complicated logic in one SQL, sometimes due to the clarity of logic. For example: create view A as

Re: Passing vm options

Hey, AFAIK, Flink supports dynamic properties currently only on YARN and not really in standalone mode. If You are using YARN it should indeed be possible to set such configuration. If not, then I am afraid it is not possible. Best Regards, Dom. pon., 7 sty 2019 o 09:01 Avi Levi napisał(a): >

Re: The way to write a UDF with generic type

Currently, there is no more flexible approch for aggregate functions. Scalar functions can be overloaded but aggregate functions do not support this so far. Regards, Timo Am 07.01.19 um 02:27 schrieb yinhua.dai: Hi Timo, But getResultType should only return a concrete type information, righ

Re: How to get the temp result of each dynamic table when executing Flink-SQL?

Hi Henry, such a feature is currently under discussion [1] feel free to participate here and give feedback. So far you need to have some intermediate store usually this could be Kafka or a filesystem. I would recommend to write little unit tests that test each SQL step like it is done here [

Re: Reducing runtime of Flink planner

Hi Niklas, it would be interesting to know which planner caused the long runtime. Could you use a debugger to figure out more details? Is it really the Flink Table API planner or the under DataSet planner one level deeper? There was an issue that was recently closed [1] about the DataSet opt

Re: Buffer stats when Back Pressure is high

Hi Gagan, a typical solution to such a problem is to introduce an artifical key (enrichment id + some additional suffix), you can then keyBy on this artificial key and thus spread the workload more evenly. Of course you need to make sure that records of the second stream are duplicated to all

Re: onTimer function is not getting executed and job is marked as finished.

Hi Puneet, maybe you can show or explain us a bit more about your pipeline. From what I see your ProcessFunction looks correct. Are you sure the registering takes place? Regards, Timo Am 07.01.19 um 14:15 schrieb Puneet Kinra: Hi Hequn Its a streaming job . On Mon, Jan 7, 2019 at 5:51 PM

Re: S3 StreamingFileSink never completes multipart uploads

The not seeing the final post from the logs was what initially clued me into the same issue, so I figured it was the same issue, but I should have worded it better that it is one possible explanation. Sorry for the confusion! Addison On Fri, Jan 4, 2019 at 11:24 PM Kostas Kloudas wrote: >

Re: Building Flink from source according to vendor-specific version but causes protobuf conflict

Hi Wei, did you play around with classloading options mentioned here [1]. The -d option might impact how classes are loaded when the job is deployed on the cluster. I will loop in Gary that might now more about the YARN behavior. Regards, Timo [1] https://ci.apache.org/projects/flink/flink

Re: onTimer function is not getting executed and job is marked as finished.

I checked the same the function is getting exited when i am calling ctx.getTimeservice () function. On Mon, Jan 7, 2019 at 10:27 PM Timo Walther wrote: > Hi Puneet, > > maybe you can show or explain us a bit more about your pipeline. From what > I see your ProcessFunction looks correct. Are you

Re: How to migrate Kafka Producer ?

Hi Piotr, Thank you for looking into this. Do you have an idea when next version (1.7.2) will be available ? Also, could you validate / invalidate the approach I proposed in the previous comment ? Edward Rojas wrote > Regarding the kafka producer I am just updating the job with the new > conne

Re: S3 StreamingFileSink never completes multipart uploads

No problem and thanks for looking into the problem! I also commented on the JIRA. Cheers, Kostas On Mon, Jan 7, 2019, 18:13 Addison Higham The not seeing the final post from the logs was what initially clued me > into the same issue, so I figured it was the same issue, but I should have > worde

Kafka 2.0 quotas

https://www.cloudera.com/documentation/kafka/2-2-x/topics/kafka_performance.html#quotas Any thoughts around feasibility ( in fact is this even supported ) of using quotas in flink based kafka connectors to throttle consumption and possibly mitigating back pressure if the optimal consumption rate i

Re: Kafka 2.0 quotas

Aah, quotas are broker side "throttles" so I guess this is any client API specific including "flink" ? On Mon, Jan 7, 2019 at 2:17 PM Vishal Santoshi wrote: > > https://www.cloudera.com/documentation/kafka/2-2-x/topics/kafka_performance.html#quotas > Any thoughts around feasibility ( in fact is

Re: onTimer function is not getting executed and job is marked as finished.

Hi puneet, Could you print `parseLong + 5000` and `ctx.timerService().currentProcessingTime()` out and check the value? I know it is a streaming program. What I mean is the timer you have registered is not within the interval of your job, so the timer has not been triggered. For example, parseLong

Re: onTimer function is not getting executed and job is marked as finished.

Hi hequan Weird behaviour when i m calling ctx.timeservice() function is getting exited even not throwing error On Tuesday, January 8, 2019, Hequn Cheng wrote: > Hi puneet, > > Could you print `parseLong + 5000` and > `ctx.timerService().currentProcessingTime()` > out and check the value? > I

Re: Building Flink from source according to vendor-specific versionbut causes protobuf conflict

Hi,Timo Good day! Thank you for your help! This issue has been solved with the rebuilt flink version. But I found that does not work with the 'Apache Flink 1.7.1 only' version even if i configure the class path like export HADOOP_CLASSPATH=`hadoop classpath` . I will check it later. Thanks

Re: The way to write a UDF with generic type

Hi Timo, Can you let me know how the build-in "MAX" function able to support different types? Thanks. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: How to get the temp result of each dynamic table when executing Flink-SQL?

In our case, we wrote a console table sink which print everything on the console, and use "insert into" to write the interim result to console. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/