Flink Kafka consumer with low latency requirement

Dear Flink experts, I am experimenting Flink for a use case where there is a tight latency requirements. A stackoverflow article suggests that I can use setParallism(n) to process a Kafka partition in a multi-threaded way. My understanding is there is still one kafka consumer per partition, but b

[External] Using scala parallel collection with ForkJoinPool

Hi, We are doing a computationally expensive window aggregation in our flink job. Expectedly, the aggregation takes a lot of time. We are experimenting with scala parallel collections in order to speed up our computation. Are there any guidelines regarding this ? As per my understanding, ForkJoin

Re: dynamic metric

Thanks Till. Ok, I've got it. So, to prevent from register the metric twice I have to keep an index (Hasmap for example) to check if the metric already exists ? Le ven. 21 juin 2019 à 01:27, Till Rohrmann a écrit : > Hi David, > > I think it is not strictly required that you register the metric

Re: dynamic metric

Hi David, I think it is not strictly required that you register the metric in the open method. It is just convenient because otherwise you have to make sure that you register the metric only once (e.g. when doing it in the map function). What you need in order to register a metric is the runtime

dynamic metric

Hi, I want to create one metric related to the number of errors but in fact I would like to add some contextual labels ? What is the best way to do that ? gauge ? How to create this kind of metric dynamically during the run of the task (because open method is not possible because too early) ? Than

Re: Unable to set S3 like object storage for state backend.

Hi Vishwas, It might be that you’ve got a legacy bucket name (“aip_featuretoolkit”), as AWS no longer allows bucket names to contain an underscore . I’m guessing that the Hadoop S3 code is trying to treat your path as a va

Unable to set S3 like object storage for state backend.

Hi, I am using flink version 1.7.2 , I am trying to use S3 like object storage EMC ECS( https://www.emc.com/techpubs/ecs/ecs_s3_supported_features-1.htm) . I am using the flink-s3-fs-hadoop-1.7.2.jar file as a dependency for s3 filesystem and I have placed it under the lib folder and is available

Re: Updated Flink Documentation and tutorials

Thank you! On Thu, Jun 20, 2019, 5:49 AM Chesnay Schepler wrote: > There is no version of the documentation that is more up-to-date. The > documentation was simply not updated yet for the new architecture. > > On 20/06/2019 11:45, Pankaj Chand wrote: > > Based on the below conversation (reverse

metrics for checking whether a broker throttles requests based on its quota limits?

Hi, Recently we enabled Kafka quota management for our Kafka clusters. We are looking for Kafka metrics that can be used for alerting on whether a Kafka broker throttles requests based on quota. There are a few throttle related metrics on Kafka. But none of them can tell accurately whether the br

Re: Apache Flink - Question about metric registry and reporter and context information

You cannot access context information in the metric reporter itself /in a nice way/. You can wait for the first metric to be registered, and then extract arbitrary metric variables from the metric group. On 15/06/2019 19:31, Yun Tang wrote: Hi 1) Yes, the metrics reporter was instantiated per

Re: [ANNOUNCEMENT] March 2019 Bay Area Apache Flink Meetup

Hey Xuefu, thanks a lot for organizing the Bay Area Flink Meetup. For others following this email thread, here is the (deep) link to the event: https://www.meetup.com/Bay-Area-Apache-Flink-Meetup/events/262216929/ I see that only 19 people have RSVPed so far. In my experience, it's possible to a

Re: [DISCUSS] Deprecate previous Python APIs

I have created a JIRA and PR for removing the Python APIs. On 11/06/2019 15:30, Stephan Ewen wrote: Hi all! I would suggest to deprecating the existing python APIs for DataSet and DataStream API w

CoFlatMapFunction vs BroadcastProcessFunction

Hi guys, I read about http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Dynamic-Rule-Evaluation-in-Flink-td21125.html#a21241 http://apache-flink-user-maili

Re: Updated Flink Documentation and tutorials

There is no version of the documentation that is more up-to-date. The documentation was simply not updated yet for the new architecture. On 20/06/2019 11:45, Pankaj Chand wrote: Based on the below conversation (reverse chronological order) regarding my previous question on the role of Job Manag

Re: Updated Flink Documentation and tutorials

Based on the below conversation (reverse chronological order) regarding my previous question on the role of Job Manager in Flink: Hi Biao, Thank you for your reply! Please let me know the url of the updated Flink documentation. The url of the outdated document is: https://ci.apache.org/project

Re: [External] checkpoint metadata not in configured directory state.checkpoints.dir

Hi Congxian, I am not sure how can I track the checkpoint path. Can you suggestion regarding this ? Thanks, Vishal Sharma On Thu, Jun 20, 2019 at 11:17 AM Congxian Qiu wrote: > Hi, Vishal > If you want to restart from the last competed external checkpoint of the > previous stoped job, you need

Re: Updated Flink Documentation and tutorials

What makes you believe that they are out-dated? On 19/06/2019 19:17, Pankaj Chand wrote: Hello, Please let me know how to get the updated documentation and tutorials of Apache Flink. The stable v1.8 and v1.9-snapshot release of the documentation seems to be outdated. Thanks! Pankaj