Deterministic Update

Is there any possibility to trigger sink operator on completion of checkpoint? -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Deterministic-Update-tp13580.html Sent from the Apache Flink User Mailing List archive. mailing list archive at N

Re: Flink streaming Python

There's an open PR that adds a streaming Python API. Would be great if you could check it out and provide some feedback! On 08.06.2017 05:43, yunfan123 wrote: Vote for python +1. I find it can't support kafka source from code. -- View this message

Re: Flink streaming Python

Vote for python +1. I find it can't support kafka source from code. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-streaming-Python-tp13573p13578.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabbl

回复：Question regarding configuring number of network buffers

Hi Ray, For your question : Does that say that each parallel task inside the  TaskManager talk to all parallel tasks inside the same TaskManager or to all  parallel tasks across all task managers? Each task will talk to all parallel upstream and downstream tasks that both include the same TaskMana

Re: How to run a Flink job in EMR?

Ah, maybe (1) wasn’t entirely clear so here’s the copy/pasted example with what I suggested: HadoopJarStepConfig copyJar = new HadoopJarStepConfig() .withJar("command-runner.jar") .withArgs("bash","-c", "aws s3 cp s3://mybucket/myjar.jar /home/hadoop" ); From: "Foster, Craig

Re: How to run a Flink job in EMR?

1) Since the jar is only required on the master node you should be able to just run a step with a very simple script like ‘bash –c “aws s3 cp s3://mybucket/myjar.jar .”’ So if you were to do that using the step similar to outlined in the EMR documentation, but replacing withArgs with the a

How to run a Flink job in EMR?

Hi Gang, I’ve been trying to get some Flink code running in Amazon Web Services’s Elastic MapReduce, but so far the only success I’ve had required me to log into the master node, download my jar from S3 to there, and then run it on the master node from the command line using something like the

Flink streaming Python

Hi I have asked the same question back in Jan 2016 and checking again with community to see if there is any update or plan for supporting streaming Flink in python.

Re: Question regarding configuring number of network buffers

Hi! I’ve looped in Nico (cc’ed) who knows about the configuration of network buffers the best, and could probably explain it to you in more detail. Cheers, Gordon On 7 June 2017 at 5:59:08 PM, Ray Ruvinskiy (ray.ruvins...@arcticwolf.com) wrote: The documentation provides the formula #slots-p

Re: Guava version conflict

Yes, those should not be in the flink-dist jar, so the root reason should be that the shading isn’t working properly for your custom build. If possible, could you try building Flink again with a lower Maven version as specified in the doc, and see if that works? If so, it could be that Maven 3.3

Re: Guava version conflict

What I did was to take the sources of the new ES connector and I took them into my code. Flink was compiled with maven 3.3+ but I did the double compilation as specified in the Flink build section. In flink dist I see guava classes, e.g.: com/google/common/util/concurrent/MoreExecutors$1.class com

Re: Painful KryoException: java.lang.IndexOutOfBoundsException on Flink Batch Api scala

Hi Andrea, I did some quick issue searching, and it seems like this is a frequently asked issue on Kryo: https://github.com/EsotericSoftware/kryo/issues/428. I can’t be sure at the moment if the resolution / workaround mentioned in there makes sense, I’ll have to investigate a bit more. Also,

Question regarding configuring number of network buffers

The documentation provides the formula #slots-per-TM^2 * #TMs * 4 to determine the number of network buffers we should configure. The documentation also says, “A logical network connection exists for each point-to-point exchange of data over the network, which typically happens at repartitioning

Re: Queries regarding FlinkCEP

Hi Dawid, Yes, now I understood what you meant. Although I added exactly the input you asked me to and I still get no alerts. I also observed that I am not getting alerts even with normal ordering of timestamp and with ascedingTimestampExtractor. I am adding an image where I entered the data fr

Re: Guava version conflict

Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES connector). Another thing to check, if you built Flink yourself, make sure you’re not using Maven 3.3+. There are shading problems when Flink is built with Maven versions higher then that. The flink-dist jar should not contain an

Re: Guava version conflict

I shaded the Elasticsearch dependency [1] and now the job works. So I cannot run a job that needs guava 18 on Flink 1.2.1... [1] https://www.elastic.co/blog/to-shade-or-not-to-shade On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai wrote: > Hi Flavio, > > Could there be another dependency in

Re: Guava version conflict

Hi Flavio, Could there be another dependency in your job that requires a conflicting version (w.r.t. ES 2.4.1) of Guava? I’ve just double checked the flink-dist jar, there doesn’t seem to be any non-shaded Guava dependencies there, so the conflict should not have been caused by Flink. Cheers,

Re: Queryable State Client with 1.3.0-rc0

Sorry for yet another update but this is the complete settings for making it work on Flink 1.3: Configuration configuration = new Configuration(); configuration.setInteger(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, 8); configuration.setString(JobManagerOptions.ADDRESS, "localhost"); configurat

Re: Queries regarding FlinkCEP

Sorry I have not responded earlier. Did you try input like this?: > 1,a,1,0 > 5,a,3,2 > 6,a,2,1 > 8,b,5,1 > 12,b,4,6 > 12,b,7,6 In response to your PS, in this example no events are late(in regards to maxOutOfOrderness). If after the last event there was an event like (12,b, , 6) it would

Guava version conflict

Hi to all, I'm trying to use the new ES connector to index data from Flink (with ES 2.4.1). When I try to run it from Eclipse everything is ok, when I run it from the cluster I get the following exception: java.lang.NoSuchMethodError: com.google.common.util. concurrent.MoreExecutors.directExecutor

Flink on kubernetes -> shell deployment

Hi together, I run Flink on kubernetes and I’m able to can access the web dashboard inside k8n. My question is: How do I get access to the shell? I don’t want to deploy my job on the UI. Thanks! Best, Marc

Re: Amazon Athena

Seems straight forward. The biggest challenge is that that you don’t want Athena picking up on partially written files or for whatever reason corrupt files. The issue with S3 is you cannot allow Flink to perform delete, truncate, or rename operations because it moves faster than S3 can become co

Painful KryoException: java.lang.IndexOutOfBoundsException on Flink Batch Api scala

Good afternoon dear Community, Since few days I'm really struggling to understand the reason behind this KryoException. Here the stack trace. 2017-06-07 10:18:52,514 ERROR org.apache.flink.runtime.operators.BatchTask - Error in task code: CHAIN GroupReduce (GroupReduce at my.or

Re: Queryable State Client with 1.3.0-rc0

Hi Claudio, Quick question: what exactly was your call for getting the local environment with web UI? Did you also have a custom Configuration where you specified, for example, that the queryable state server should be enabled? I can make an example work where I start a local cluster in one pro

Re: AsyncCollector Does not release the thread (1.2.1)

Hi Steve, I’m assuming you are using Flink 1.2.x? If yes, then I’m afraid you re-discovered this issue: https://issues.apache.org/jira/browse/FLINK-6435 . It was fixed in Flink 1.3.0. Is it possible for you to update to that version or do you th

Re: Flink and swapping question

I forgot to mention that my jobs are all batch (at the moment). Do you think that this problem could be related to - http://www.evanjones.ca/java-bytebuffer-leak.html#comment-3240054880 - and http://www.evanjones.ca/java-native-leak-bug.html Kurt told me also to add "env.java.opts: -Dio.ne

Re: Running job in "dry mode"?

On 07/06/2017 10:27, Maciek Próchniak wrote: On 07/06/2017 10:07, Tzu-Li (Gordon) Tai wrote: Hi Maciek, Is there any particular reason why you do not wish to start running the Kafka sources on the test run? Otherwise, it would be perfectly fine to start the test job for testing to see if

[DISCUSS] Removal of twitter-inputformat

Hello, I'm proposing to remove the Twitter-InputFormat in FLINK-6710 , with an open PR you can find here . The PR currently has a +1 from Robert, but Timo raised some concerns saying that it is useful

Re: Methods that trigger execution

Hi, I’m afraid I don’t know that part well enough. What’s the percentage in slowdown? (7 seconds alone doesn’t say anything) Maybe Till (in cc) knows more since he used to work on the ML part. Best, Aljoscha > On 6. Jun 2017, at 17:45, Borja wrote: > > *Thank so much Aljoscha* :) > I was stu

Re: Running job in "dry mode"?

Hi Maciek, Is there any particular reason why you do not wish to start running the Kafka sources on the test run? Otherwise, it would be perfectly fine to start the test job for testing to see if everything works, and keep that savepoint eventually for the non-dry run. Also, what our integratio