Re: Discard message on deserialization errors.

Hi John, It should work with a *null* return value. In the java doc of DeserializationSchema#deserialize it says that > *@return The deserialized message as an object (null if the message cannot > be deserialized).* I also checked the Kafka fetcher in Flink and it can correctly handle a null de

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

Hi Stephan, big +1 for this contribution. It provides another user interface that is easy to use and popular at this time. these functions, It's hard for users to write in SQL/TableApi, while using DataStream is too complex. (We've done some stateFun kind jobs using DataStream before). With statef

Re: Batch Job in a Flink 1.9 Standalone Cluster

Thanks Xintong! In my case both of those parameters are set to false (default). I think I am sort of following what's happening here. I have one TM with heap size set to 1GB. When the cluster is started the TM doesn't use that 1GB (no allocations). Once the first batch job is submitted I can

StreamingFileSink

Hello, Question 1 I don’t see any reference material showing how to write compressed (gzip) files with StreamingFileSink. Can someone point me in the right direction? Question 2 We currently have a use case for a “StreamingFileProcessFunction”. Basically we need an output for the StreamingFileS

Discard message on deserialization errors.

Hi using Flink 1.8.0. I am ingesting data from Kafka, unfortunately for the time being I have not looked into using the schema registry. So for now I would like to write a simple deserialization schema that discards the data if deserialization fails. The other option is to do in flat map with ma

add() method of AggregateFunction not called even though new watermark is emitted

Hi, Here is my issue with *Event Processing* with the *add() method of MGroupingWindowAggregate not being called* even though a new watermark is fired 1. *Ingest data from Kinesis (works fine)* 2. *Deserialize* in MonitoringMapKinesisSchema(*works fine* and get json back) 3. I do *assign Monitoring

Re: Where are uploaded Job jars stored?

Hi Zhu Zhu, as per my original post. If you have 3 Job nodes and you upload the job through the web UI on a specific node, then only the web UI for that node will show it. If you browse another job node you will not see it. As per Stevens suggestion, I already have a Gluster FS mounted folder on t

Re: [DISCUSS] Best practice to run flink on kubernetes

Thanks for bumping this thread Yang. I've left some minor comments on your design document. All in all it looks very good to me. I think the next step could be to open the missing PRs to complete FLINK-9953. That would allow people to already try your implementation out and maybe someone from the c

Re: Flink restoring a job from a checkpoint

Hi Flavio If you did not even trigger a savepoint but meet this problem. First of all, please ensure your checkpoint would be retained [1]. Once your job fails due to a problematic message, you need to cancel your job and modify the job to not failover when meeting that problematic message agai

Re: Flink restoring a job from a checkpoint

I don't think schedule savepoint periodically is better than periodic checkpoint(which flink have out of box). 1. Savepoint and checkpoint have the same code path exception savepoint will do a full snapshot, and checkpoint can do an incremental snapshot. If the checkpoint can not be done, then the

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

Hi Stephan, I think this is a nice library, but what I like more about it is that it suggests exploring different use-cases. I think it definitely makes sense for the Flink community to explore more lightweight applications that reuses resources. Therefore I definitely think it is a good idea for

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

Hi Stephan, big +1 for adding statefun to Apache Flink. I think we can add it in the main repository as a library just like State Process API. Having it in the main repository can closely co-develop with Apache Flink which be beneficial for both side. Regards, Jark On Fri, 11 Oct 2019 at 19:12

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

Hi Stephan, +1 for adding stateful functions to Flink. I believe the new set of applications this feature will unlock will be super interesting for new and existing Flink users alike. One reason for not including it in the main repository would to not being bound to Flink's release cadence. This

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

+1 non-binding on contribution. Separate repo, or feature branch to start maybe? I just feel like in the beginning this thing is going to have lots of breaking changes that maybe aren't going to fit well with tests / other "v1+" release code. Just my .02. On Fri, Oct 11, 2019 at 4:38 AM Stephan

Re: Difference between windows in Spark and Flink

that is nice. So, only by this Flink shuffles fewer data them Spark. Now I need to plug Prometheus and Grafana to show it. Thanks Yun for your help! *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com * On

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

Hi Stephan, +1 for adding this to Apache Flink! Regarding the question if this should be committed to the main repository or as a separate one, I think it should be the main one. Previously we were discussing the idea of splitting Apache Flink into multiple repositories and I think the consen

Re: Difference between windows in Spark and Flink

Hi Felipe >From what I remember, Spark still use micro-batch to shuffle data in structed >streaming. For Flink, it actually process elements per record, there is no actual disk-io shuffle in Flink streaming. And record would emit to downstream by select specific channel through network[1]. That

Re: Challenges Deploying Flink With Savepoints On Kubernetes

Apart from these we have other environment and there check point worked fine in HA mode with complete cluster restart. But one of the job we are seeing an issue, in zookeeper the check point path is retrieved and its unable to find the check point path in persistent storage. I am wondering why this

[PROPOSAL] Contribute Stateful Functions to Apache Flink

Dear Flink Community! Some of you probably heard it already: On Tuesday, at Flink Forward Berlin, we announced **Stateful Functions**. Stateful Functions is a library on Flink to implement general purpose applications. It is built around stateful functions (who would have thunk) that can communic

Re: Flink restoring a job from a checkpoint

If I understood correctly you're saying that in this case I'd need to reprocess all messages from scratch (unless I retain my checkpoints..), right? Could it be a good strategy to schedule savepoints periodically to avoid such situations? Is there any smarter solution to this? On Fri, Oct 11, 2019

Re: Challenges Deploying Flink With Savepoints On Kubernetes

We are seeing below logs in production sometime ago, after that we stopped HA. Do you people think HA is enabled properly from the below logs? Regards Bhaskar 2019-09-24 17:40:17,675 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionSe

Re: Fencing token exceptions from Job Manager High Availability mode

Hi, @Joshua Fan , I think the logs you provided me substantiated my suspicion. I think you are running into FLINK-11843. It happens the following way: After regaining the leadership, the Dispatcher tries to start the JobManager. Recovering the state of the job takes a while (could also be the temp

Re: Difference between windows in Spark and Flink

Hi Yun, that is a very complete answer. Thanks! I was also wondering about the mini-batches that Spark creates when we have to create a SparkStream context. It still remains for all versions of stream processing in Spark, isn't it? And because that I Spark shuffles data [1] to wide-dependent oper