Issues in Batch Jobs Submission for a Session Cluster

Hi Flink Team, Greetings from GE Healthcare team. Here is a stackoverflow post for the same too posted by fellow dev here : https://stackoverflow.com/questions/70068336/flink-job-not-getting-submitted-java-io-ioexception-cannot-allocate-memory Summary of the post: Here is the usecase and relev

Re: How do you configure setCommitOffsetsOnCheckpoints in Flink 1.12 when using KafkaSourceBuilder?

Sorry, I spell it wrong, which I mean the PR. Here it is https://github.com/apache/flink/pull/17276 . Marco Villalobos 于2021年12月1日周三 下午9:18写道： > Thank you. One last question. What is an RP? Where can I read it? > > Marco > > On Nov 30, 2021, at 11:06 PM, Hang Ruan wrote: > > Hi, > > In 1.12.0-

Re: [DISCUSS] Deprecate Java 8 support

Flink can be built with Java 11 since 1.10. If I recall correctly we solved the tools.jar issue, which Hadoop depends on, by excluding that dependency. As far as we could tell it's not actually required. On 01/12/2021 19:56, Nicolás Ferrario wrote: Hi all, this would be awesome, I'm so tired of

Re: [DISCUSS] Deprecate Java 8 support

Hi all, this would be awesome, I'm so tired of seeing Java 8 everywhere (reminds me of Python 2.7). We're currently building our code against Java 11 because that's the latest version of Java available as a Flink Docker image, but it'd be great to use newer versions. I think it would also help to

Re: REST API for detached minicluster based integration test

so i went ahead and put some logging in the WatermarkGeneartor.onEvent and .onPeriodicEmit functions in the test source watermark generator, and i do see the watermarks come by with values through those functions. they're just not being returned as expected via the rest api. On Tue, Nov 30, 2021

Re: How do you configure setCommitOffsetsOnCheckpoints in Flink 1.12 when using KafkaSourceBuilder?

Thank you. One last question. What is an RP? Where can I read it? Marco > On Nov 30, 2021, at 11:06 PM, Hang Ruan wrote: > > Hi, > > In 1.12.0-1.12.5 versions, committing offset to kafka when the checkpoint is > open is the default behavior in KafkaSourceBuilder. And it can not be changed >

Re: [DISCUSS] Deprecate Java 8 support

Hello Gavin, If you run into any issues with Java 17, please report them in FLINK-15736 . I recently did some experiments with Java 17 myself; I would think that you will run into some blockers (like ASM requiring an upgrade

Re: [DISCUSS] Deprecate Java 8 support

Thanks for sharing this info with us Chesnay. We've been using Flink for 5 years, and upgraded to 1.13.2 months ago. The java version is still 8. Currently we're testing with java 17 in our staging environment. There are no special concerns. Will update when tests complete. On Tue, Nov 30, 2021

Re: how to run streaming process after batch process is completed?

Hi Alexander, This is my ideal data pipeline. - 1. Sqoop transfer bounded data from database to hive. And I think flink batch process is more efficient than streaming process, so i want to process this bounded data in batch mode and write result in HiveTable2. - 2. There ares some tools to tr

Re: [ANNOUNCE] Open source of remote shuffle project for Flink batch data processing

Hi Jiangang, Great to hear that, welcome to work together to make the project better. Best, Yingjie 刘建刚 于2021年12月1日周三 下午3:27写道： > Good work for flink's batch processing! > Remote shuffle service can resolve the container lost problem and reduce > the running time for batch jobs once failover.

Watermark behavior when connecting streams

Hi everyone, Based on what I know, a single operator with parallelism > 1 checks the watermarks from all its streams and uses the smallest one out of the non-idle streams. My first question is whether watermarks are forwarded as long as a different watermark strategy is not applied downstream?

RE: Windows and data loss.

Hi John, Sorry for the delay … I’m a little tight on spare time for user@flink currently. If you are still interested we could pick up the discussion and continue. However I’m don’t exactly understand what you want to achieve: 1. Would processing time windows be enough for you (and misplacemen