Re: Unable to submit job from the web UI in flink 4.0

yep working from cli. On Wed, Feb 14, 2018 at 1:41 AM, Vishal Santoshi wrote: > This I think is a known issue with 1.4. Good submissions work on UI ( > one's that do not have errors to report ) and CLI works always ( in our > case ) > > On Tue, Feb 13, 2018 at 1:02 PM, Gary Yao wrote: > >> c

Deploying Flink with JobManager HA on Docker Swarm/Kubernetes

Hi, I am trying to deploy a Flink cluster (1 JM, 2TM) on a Docker Swarm. For JobManager HA, I have started a 3 node zookeeper service on the same swarm network and configured Flink's zookeeper quorum with zookeeper service instances.  JobManager gets started with the LeaderElectionService and ge

Flink + Consul as HA backend. What do you think?

I'd like to get your opinion about this idea. I found related JIRA issue FLINK-2366, but it seems to be dead. To attract your attention I copy my comment here. As an experiment I've implemented Flink HA on top of Consul. The implementation is working fine in the "lab" but is not battle tested yet.

Re: Unable to submit job from the web UI in flink 4.0

This I think is a known issue with 1.4. Good submissions work on UI ( one's that do not have errors to report ) and CLI works always ( in our case ) On Tue, Feb 13, 2018 at 1:02 PM, Gary Yao wrote: > cc'ing user mailing list > > On Mon, Feb 12, 2018 at 12:40 PM, Puneet Kinra < > puneet.ki...@

Re: Unable to submit job from the web UI in flink 4.0

cc'ing user mailing list On Mon, Feb 12, 2018 at 12:40 PM, Puneet Kinra < puneet.ki...@customercentria.com> wrote: > Hi Gary > > Thanks for the response i am able to upload the jar but i when am pressing > the submit button > It keep on running but jar is not getting deployed. & even not getting

Re: Python and Scala

Hi And what about the differences between Scala REPL and Scala (compiled) ? Esa Piotr Nowojski kirjoitti 13.2.2018 klo 15:14: Hi, 1. Flink’s Python Batch API is not complete and it’s not on pair with Scala. 2. As for know there is no Python API for Flink Streaming, however there is some ong

Re: CoProcess() VS union.Process() & Timers in them

Thanks a lot Fabian and Xingcan! @ Fabian : Nice answer. It enhanced my intuition on the topic. So, how one may change the parallelism while the Flink job is running, e.g. lower the parallelism during the weekend? Also, it is not clear to me how to use the rescale() operator. If you may provide

RE: CEP for time series in csv-file

Hi I have little problem with this example about Scala maven. I created new project by command: “mvn archetype:generate …” as stated in the instructions. What is the correct build command ? “mvn package –Pbuild-jar” in same directory with pom.xml ? For example, it does not generate WordCount.ja

Re: Python and Scala

Hi, 1. Flink’s Python Batch API is not complete and it’s not on pair with Scala. 2. As for know there is no Python API for Flink Streaming, however there is some ongoing work with that: https://issues.apache.org/jira/browse/FLINK-5886 3. CEP doe

Re: CoProcess() VS union.Process() & Timers in them

You might also want to change the parallelism if the rate of your input streams varies, e.g., you scale an application down over night or the weekend. 2018-02-13 13:43 GMT+01:00 Xingcan Cui : > Hi Max, > > Currently, the timers can only be used with keyed streams. As @Fabian > suggested, you can

Re: CoProcess() VS union.Process() & Timers in them

Hi Max, Currently, the timers can only be used with keyed streams. As @Fabian suggested, you can “forge” a keyed stream with the special KeySelector, which maps all the records to the same key. IMO, Flink uses keyed streams/states as it’s a deterministic distribution mechanism. Here, “the para

Python and Scala

What the difference is to use Python and Scala in Flink ? Can I do all the same things with Python and Scala ? For examples CEP with files.

Re: Ceph configuration for checkpoints?

Hello Julio, how we deploy Ceph: 1) make sure Ceph is mounted through all the nodes that make up your Flink cluster; ensure path(s) are the same for each node 2) simply reference the Ceph mountpoint(s) via file:// Martin On Tue, Feb 13, 2018 at 10:18 AM Piotr Nowojski wrote: > Hi, > > Have

Re: Ceph configuration for checkpoints?

Hi, Have you tried to refer to ceph documentation? http://docs.ceph.com/docs/jewel/cephfs/hadoop/ It claims to be: > a drop-in replacement for the Hadoop File System (HDFS) So I would first try to configure ceph according to their documentation

Re: CoProcess() VS union.Process() & Timers in them

OK Great! Thanks a lot for the super ultra fast answer Fabian! One intuitive follow-up question. So, keyed state is the most preferable one, as it is easy for the Flink System to perform the re-distribution in case of change in parallelism, if we have a scale-up or scale-down. Also, it is useful

Re: CoProcess() VS union.Process() & Timers in them

Hi Max, you can use keyed state on an operator with parallelism 1 if you assign a default key with a KeySelector: stream.keyBy(new NullByteKeySelector) with NullByteKeySelector defined as public class NullByteKeySelector implements KeySelector { private static final long serialVersionUID =

Re: CoProcess() VS union.Process() & Timers in them

Hello XingCan, Finally, I did it with union. Now inside the processElement() function of my CoProcessFunction I am setting a timer and periodically I want to print out some data through the onTimer() function. Below I attach the image stating the following: "Caused by: java.lang.UnsupportedOpera

Re: Classloader leak with Kafka Mbeans / JMX Reporter

I looked through the kafka JmxReporter class (and other kafka classes that we use) and couldn't spot anything that would prevent the Mbeans from being unregistered. The obvious implication would be that we don't properly clean up kafka classes, but that code looks good as well. So at the moment