Re: [ANNOUNCE] Apache Flink 1.10.2 released

Thanks a lot for being the release manager Zhu Zhu! Congrats to all others who have contributed to the release! Best, Yangze Guo On Tue, Aug 25, 2020 at 2:42 PM Dian Fu wrote: > > Thanks ZhuZhu for managing this release and everyone else who contributed to > this release! > > Regards, > Dian >

Re: [ANNOUNCE] Apache Flink 1.10.2 released

Thanks ZhuZhu for managing this release and everyone else who contributed to this release! Regards, Dian > 在 2020年8月25日，下午2:22，Till Rohrmann 写道： > > Great news. Thanks a lot for being our release manager Zhu Zhu and to all > others who have contributed to the release! > > Cheers, > Till > >

Re: [ANNOUNCE] Apache Flink 1.10.2 released

Great news. Thanks a lot for being our release manager Zhu Zhu and to all others who have contributed to the release! Cheers, Till On Tue, Aug 25, 2020 at 5:37 AM Zhu Zhu wrote: > The Apache Flink community is very happy to announce the release of > Apache Flink 1.10.2, which is the first bugfi

Re: How jobmanager and task manager communicates with each other ?

Hi Sidhant, the cluster components use tcp to communicate with each other. If you are not using Flink's HA services, then the TaskManager nodes need to be configured with the JobManager's address to connect to them. If you are using HA services, then the service discovery happens through the HA se

Re: Flink Couchbase

Thanks Yang that helped. Sent from my iPhone > On Aug 24, 2020, at 8:44 PM, Yang Wang wrote: > >  > I think at least you have two different exceptions. > > > java.lang.Exception: Container released on a *lost* node > This usually means a Yarn nodemanager is down. So all the containers running

Re: Flink Couchbase

I think at least you have two different exceptions. > java.lang.Exception: Container released on a *lost* node This usually means a Yarn nodemanager is down. So all the containers running on this node will be released and rescheduled to a new one. If you want to figure out the root cause, you need

答复: How to sink invalid data from flatmap

Thanks a lot Jake for the quick response 发件人: Jake [mailto:ft20...@qq.com] 发送时间: 2020年8月25日 星期二 11:31 收件人: 范超 抄送: user 主题: Re: How to sink invalid data from flatmap Hi fanchao Yes. I suggest that. Jake On Aug 25, 2020, at 11:20 AM, 范超 mailto:fanc...@mgtv.com>> wrote: Thanks Jake. But can

[ANNOUNCE] Apache Flink 1.10.2 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.10.2, which is the first bugfix release for the Apache Flink 1.10 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming a

Re: How to sink invalid data from flatmap

Hi fanchao Yes. I suggest that. Jake > On Aug 25, 2020, at 11:20 AM, 范超 wrote: > > Thanks Jake. But can I just want to implement the ouput-tag function in my > flatmap function not in the process function. I check the parameters for the > flatmap ,there is no ‘context’, so is it means I’ve

答复: How to sink invalid data from flatmap

Thanks Jake. But can I just want to implement the ouput-tag function in my flatmap function not in the process function. I check the parameters for the flatmap ,there is no ‘context’, so is it means I’ve to use process to rewrite my flatmap function? 发件人: Jake [mailto:ft20...@qq.com] 发送时间: 202

答复: How to sink invalid data from flatmap

Thanks , Using the ctx.output() inside the process method solved my problem, but my custom flatmap function has to be retired? 发件人: Yun Tang [mailto:myas...@live.com] 发送时间: 2020年8月25日 星期二 10:58 收件人: 范超 ; user 主题: Re: How to sink invalid data from flatmap Hi Chao I think side output [1] might m

Re: How to sink invalid data from flatmap

Hi fanchao use side output, see[1] [1]https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/stream/side_output.html Jake > On Aug 25, 2020, at 10:54 AM, 范超 wrote: > > Hi, > I’m using the c

Re: [DISCUSS] Remove Kafka 0.10.x connector (and possibly 0.11.x)

Hi Aljoscha, I'm lightly leaning towards keeping the 0.10 connector, for Kafka 0.10 still has a steady user base in my observation. But if we drop 0.10 connector, can we ensure the users would be able to smoothly migrate to 0.11 connector/universal connector? If I remember correctly, the univ

Re: How to sink invalid data from flatmap

Hi Chao I think side output [1] might meet your requirements. [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/side_output.html Best Yun Tang From: 范超 Sent: Tuesday, August 25, 2020 10:54 To: user Subject: How to sink invalid data from fla

How to sink invalid data from flatmap

Hi, I’m using the custom flatmap function to validate the kafka json string message, if the kafka message is valid to transform to a pojo (using GSON), then go on with the next sink step. If it can not be parsed as a POJO, the GSON will throw the “com.google.gson.JsonSyntaxException”, and in my

Re: Setting job/task manager memory management in kubernetes

Hi, You need to define them in "flink-configuration-configmap.yaml". Please also make sure you've created the config map by executing "kubectl create -f flink-configuration-configmap.yaml". [1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/deployment/kubernetes.html Best, Yang

Re: Flink checkpointing with Azure block storage

Thanks Plyush, The thing that I was missing is this. Now it all works > On Aug 24, 2020, at 2:44 PM, Piyush Narang wrote: > > We had something like this when we were setting it in our code (now we’re > passing it via config). There’s likely a better /cleaner way: > private def configureCheckpo

Re: Flink Couchbase

Another one - Exception in thread "FileCache shutdown hook" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "FileCache shutdown hook" Regards, Vijay On Mon, Aug 24, 2020 at 1:04 PM Vijayendra Yadav wrote: > Actually got this message in rolledover contai

Re: Flink Couchbase

Actually got this message in rolledover container logs: [org.slf4j.impl.Log4jLoggerFactory] Exception in thread "cb-timer-1-1" java.lang.OutOfMemoryError: Metaspace Exception in thread "Thread-16" java.lang.OutOfMemoryError: Metaspace Exception in thread "TransientBlobCache shutdown hook" java.lan

Flink Couchbase

Hi Team, Running a flink job on Yarn, I am trying to make connections to couchbase DB in one of my map functions in Flink Streaming job. But my task manager containers keep failing and keep assigning new containers and not giving me an opportunity to get any useful logs. val cluster = Cluster.co

Re: Flink checkpointing with Azure block storage

We had something like this when we were setting it in our code (now we’re passing it via config). There’s likely a better /cleaner way: private def configureCheckpoints(env: StreamExecutionEnvironment, checkpointPath: String): Unit = { if (checkpointPath.startsWi

Re: Debezium Flink EMR

Thank you so much for the help! On Mon, Aug 24, 2020 at 4:08 AM Marta Paes Moreira wrote: > Yes — you'll get the full row in the payload; and you can also access the > change operation, which might be useful in your case. > > About performance, I'm summoning Kurt and @Jark Wu to > the thread, w

Re: How jobmanager and task manager communicates with each other ?

++d...@flink.apache.org On Mon, Aug 24, 2020, 7:31 PM sidhant gupta wrote: > Hi User > > How jobmanager and task manager communicates with each other ? How to set > connection between jobmanager and task manager running in different/same > ec2 instance ? Is it http or tcp ? How the service disco

Flink OnCheckpointRollingPolicy streamingfilesink

Hi Team, Bulk Formats can only have `OnCheckpointRollingPolicy`, which rolls (ONLY) on every checkpoint. *.withRollingPolicy(OnCheckpointRollingPolicy.build())* Question: What are recommended values related to checkpointing to fsstate, should it be more frequent checkpoints, or longer intervals,

[DISCUSS] Remove Kafka 0.10.x connector (and possibly 0.11.x)

Hi all, this thought came up on FLINK-17260 [1] but I think it would be a good idea in general. The issue reminded us that Kafka didn't have an idempotent/fault-tolerant Producer before Kafka 0.11.0. By now we have had the "modern" Kafka connector that roughly follows new Kafka releases for a

How jobmanager and task manager communicates with each other ?

Hi User How jobmanager and task manager communicates with each other ? How to set connection between jobmanager and task manager running in different/same ec2 instance ? Is it http or tcp ? How the service discovery works ? Thanks Sidhant Gupta

Idle stream does not advance watermark in connected stream

Hi all, We are testing the new Idleness detection feature in Flink 1.11, however, it does not work as we expected: When we connect two data streams, of which one is idle, the output watermark CoProcessOperator does not increase, hence the program cannot progress. I've made a small project to illus

Re: Setting job/task manager memory management in kubernetes

Hi, Sakshi Could you provide more information about: - What is the Flink version you are using? "taskmanager.heap.size" is deprecated since 1.10[1]. - How do you deploy the cluster? In the approach of native k8s[2] or the standalone k8s[3]? [1] https://ci.apache.org/projects/flink/flink-docs-rel

Why consecutive calls of orderBy are forbidden?

Hi, all. I tried calling two consecutive orderBy for a Table, but got an exception. Can anyone explain why this happens? In my mind, orderBy should be able to be called by any Tables. But obviously, it is not with no explanation. Here is a simplified version of code: Table table = btenv.scan("s

Re: Debezium Flink EMR

Yes — you'll get the full row in the payload; and you can also access the change operation, which might be useful in your case. About performance, I'm summoning Kurt and @Jark Wu to the thread, who will be able to give you a more complete answer and likely also some optimization tips for your spe

Re: Watermark generation issues with File sources in Flink 1.11.1

Hi Arti, what exactly do you mean by "checkpoints do not work"? Are there exceptions being thrown? How are you writing your file-based sources, what API methods are you using? Best, Aljoscha On 20.08.20 16:21, Arti Pande wrote: Hi Till, Thank you for your quick response. Both the AssignerW

Setting job/task manager memory management in kubernetes

Hello, I am trying to set the heap size of job and task manager when deploying the job in kubernetes. I have set the jobmanager.heap.size and taskmanager.heap.size. However, the custom values are not being used and it is creating its own values and starting the job. How can I set custom values? -

Re: Ververica Flink training resources

Piper, I'm happy to know that the exercises are working for you. > The new exercises are running well but I could not adjust the > servingspeedfactor to speed up the serving of data events. I'm guessing > this feature was removed in the new repo. > That's right. The feature of adjusting the ser

Re: AWS EMR deployment error : NoClassDefFoundError org/apache/flink/api/java/typeutils/ResultTypeQueryable

Thanks Prasanna and Chesnay. Changing the dependency scope worked and I also had to add a maven shaded plugin transformer to resolve another error. On Fri, Aug 21, 2020 at 11:38 PM Prasanna kumar < prasannakumarram...@gmail.com> wrote: > Manas, > > One option you could try is to set the scope in

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

Thanks a lot for the discussion! I will open a voting thread shortly! Kostas On Mon, Aug 24, 2020 at 9:46 AM Kostas Kloudas wrote: > > Hi Guowei, > > Thanks for the insightful comment! > > I agree that this can be a limitation of the current runtime, but I > think that this FLIP can go on as it

Re: Performance issue associated with managed RocksDB memory

The issue can be reproduced by using a certain combinations of the value of RocksDBOptions.WRITE_BUFFER_RATIO (default 0.5) and the Flink job parallelism. Examples that break: * Parallelism 1 and WRITE_BUFFER_RATIO 0.1 * Parallelism 5 and the default WRITE_BUFFER_RATIO 0.5 Examples that work: * P

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

Hi Guowei, Thanks for the insightful comment! I agree that this can be a limitation of the current runtime, but I think that this FLIP can go on as it discusses mainly the semantics that the DataStream API will expose when applied on bounded data. There will definitely be other FLIPs that will ac

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

Hi, Klou Thanks for your proposal. It's a very good idea. Just a little comment about the "Batch vs Streaming Scheduling". In the AUTOMATIC execution mode maybe we could not pick BATCH execution mode even if all sources are bounded. For example some applications would use the `CheckpointListener`

Re: ERROR : RocksDBStateBackend

Great to hear that you fixed the problem! Cheers, Till On Mon, Aug 24, 2020 at 2:53 AM Vijayendra Yadav wrote: > Thank You Till. I had an old hadoop version dependency in one of the > dependent jars causing conflict. > > On Fri, Aug 21, 2020 at 12:24 AM Till Rohrmann > wrote: > >> Hi Vijay, >