Flink on kubernetes: taskmanager error

Hello, I am trying to run flink on a kubernetes cluster using minikube and kubectl. I am following this example , which runs a flink 1.2 cluster ok. I am interested in running flink 1.5.1, but when I modify the flink version, I start to see these

Re: Checkpointing not happening in Standalone HA mode

Hi Vino, Yes I am enabling checkpoint in the code as follows : StreamExecutionEnvironment env = StreamExecutionEnvironment.createRemoteEnvironment(",,getJobConfiguration(),jarPath"); env.enableCheckpointing(1000); env.setSateBackend(new FsStateBackend("file:///")); env.getCheckpointConfig().s

Re: Checkpointing not happening in Standalone HA mode

Hi Vinay, Oh! You use a collection source? That's the problem. Please use a general source like Kafka or others. Maybe your checkpoint has not be triggered, your job has stopped. Thanks, vino. 2018-07-27 16:07 GMT+08:00 Vinay Patil : > Hi Vino, > > Yes I am enabling checkpoint in the code as f

Re: Checkpointing not happening in Standalone HA mode

Hi Vino, Yes, Job runs successfully, however, no checkpoints are successful. I will update the source Regards, Vinay Patil On Fri, Jul 27, 2018 at 2:00 PM vino yang wrote: > Hi Vinay, > > Oh! You use a collection source? That's the problem. Please use a general > source like Kafka or others.

Flink TaskManager binding data port to single interface while rpc port binds to all?

We're seeing an issue with our Flink TMs (1.4.2) where we explicitly set the TM data and RPC ports. When the TM comes up, we see the following bindings: == tcp6 0 0 :::9249 :::*LISTEN 2284/java tcp6 0 0 1

Custom Window example (data-based)

I want to get some experience implementing a custom window assigner, trigger, evictor, etc. Does anyone have an example of a custom window implementation that I could look at, or an idea for one to implement? The goal is to learn the custom window API. I'm looking for something besides a time or co

Re: How to connect more than 2 hetrogenous Streams!!

Hi Puneet, Flink doesn't support connecting more than 2 streams with different schema. There are ways I think might help you. 1. unify the schema and use union. 2. use multi join to join different streams. Hope this helps. Hequn On Thu, Jul 26, 2018 at 2:29 PM, Puneet Kinra < puneet.ki...@custom

Re: Custom Window example (data-based)

Hi Chris, I just find some resource you can have a look, list below: - Flink official documentation : https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/stream/operators/windows.html#built-in-and-custom-triggers - Custom, Complex Windows at Scale using Apache Flink(video):

Re: Flink Cluster and Pipeline Version Compatibility?

Hi, jlist9 > Is it so that the pipeline jars must be build with the same version of the cluster they'll be running on? Most interfaces are supported for backward comparability. And the closer the flink version is, the smaller the differences between interfaces. However, it is not for sure. Hence,

Re: Committing Kafka Transactions during Savepoint

Hi, this has been in the back of my head for a while now. I finally created a Jira issue: https://issues.apache.org/jira/browse/FLINK-9983 In there, I also outline a better fix that will take a bit longer to implement. Best, Aljoscha > On 26.

Re: Committing Kafka Transactions during Savepoint

Thank you, Aljoscha! Are Kafka transactions committed when a running job has been instructed to cancel with a savepoint (e.g. `flink cancel -s `)? This is my primary use for savepoints. I would expect that when a new job is submitted with the savepoint, as in the case of an application upgrade,

Does Flink release Hadoop(R) 2.8 work with Hadoop 3?

Hi, I can run Flink without bundled Hadoop fine. I was wondering if Flink with Hadoop 2.8 works with Hadoop 3 as well? Thanks, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Order of events in a Keyed Stream

Hi, We are currently using Flink to process financial data. We are getting position data from Kafka and we enrich the positions with account and product information. We are using Ingestion time while processing events. The question I have is: say I key the position datasream by account number. If

RuntimeException with valve output watermark when using CoGroup

Morning everyone, I'm getting the following exception in my Flink job (Flink version is 1.5.0): java.lang.RuntimeException: Exception occurred while processing valve output watermark: at org.apache.flink.streaming.runtime.io.StreamInputProcessor$ForwardingValveOutputHandler.handleWatermark(Stream

Re: RuntimeException with valve output watermark when using CoGroup

At first glance this looks like a bug. Is the nothing in the stack trace after the NullPointerException? How reliably can you reproduce this? On 27.07.2018 19:00, Taneli Saastamoinen wrote: Morning everyone, I'm getting the following exception in my Flink job (Flink version is 1.5.0): java

Re: AsyncFunction used twice with Asyncdatastream.unorderedWait - 2nd function's asyncInvoke not getting called

Hi, Turns out the issue was with the RichParallelSourceFunction I was using was resulting in the Sink not getting called after the connect SyncLatchFunction. Need to figure out the issue there but the 2 asyncInvoke functions work fine now after I replaced the *ParallelCameraSource (* RichParallelS

Re: Order of events in a Keyed Stream

Hi Harshvardhan, There are a number of factors to consider. 1. the consecutive Kafka messages must exist in a same topic of kafka. 2. the data should not been rebalanced. For example, operators should be chained in order to avoid rebalancing. 3. if you perform keyBy(), you should keyBy on a field

Re: How to connect more than 2 hetrogenous Streams!!

Hi Puneet, Hequn gave you two good solutions. If you have a good knowledge of Flink DataStream API. You can also customize it to connect more than two streams, you must know : - DataStream#connect API - ConnectedStreams - CoMapFunction, CoFlatMapFunction... Referring to them, you can go

Re: Flink Cluster and Pipeline Version Compatibility?

Hi Jlist9: Most of Flink's APIs try to ensure backward compatibility. But no documentation gives all APIs to do this. With the development of Flink, some features have changed dramatically, such as State, so Flink's official website gives the migration guidance document[1]. So, my personal suggest

Re: Does Flink release Hadoop(R) 2.8 work with Hadoop 3?

Hi Mich, I think this depends on the backward compatibility of the Hadoop client API. In theory, there is no problem. Hadoop 2.8 to Hadoop 3.0 is a very large upgrade, and personally recommend using a client version that is consistent with the Hadoop cluster. By compiling and packaging from source