Re: Securing Keytab File in Flink

2023-09-26 Thread Chirag Dewan via user
Thanks Gabor, reducing the attack vector looks a fair call here. However, I am still thinking of other ways to eliminate this security concern. Is there a way I can use ticketCache inside my pods somehow? Maybe something like Yarn?  Just thinking out loud, but would there be a case of automating

Custom Prometheus metrics disappeared in 1.16.2 => 1.17.1 upgrade

2023-09-26 Thread Javier Vegas
I implemented some custom Prometheus metrics that were working on 1.16.2, with my configuration metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory metrics.reporter.prom.port: I could see both Flink metrics and my custom metrics on port of

Re: Is there any example that use hbase connector in stream API

2023-09-26 Thread 碳酸君
HI Антон,Hang Thanks Reply I implemented the RichSinkFunction with HbaseClient ,it works well. I plan to copy some code from org.apache.flink.connector.hbase.sink.HBaseSinkFunction mainly the scheduled batch submit ,but except the converter part . I think I can read from TableAPI and write with

回复: backpressure for just one subtask

2023-09-26 Thread Chen Zhanghao
Hi Kenan, You may check the stack trace and task-level flame graph [1] to investigate it. [1] https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/debugging/flame_graphs/ Best, Zhanghao Chen 发件人: Kenan Kılıçtepe 发送时间: 2023年9月26日 23:06 收件人: user

backpressure for just one subtask

2023-09-26 Thread Kenan Kılıçtepe
Hi, I have a job running with parallelism=24. I see that only one subtask is %100 busy and the others are %100 idle. When I checked the received message counts, I saw that they are almost identical. How can I figure out why this task causes backpressure and why only one subtask is %100 busy. Than

回复: ProcessWindowFunction Parallelism

2023-09-26 Thread Chen Zhanghao
Hi Patricia, You are using the .windowall API which generates a global window. This operation is inherently non-parallel since all elements have to pass through the same operator instance so it cannot be set to any parallelism larger than 1. Best, Zhanghao Chen

ProcessWindowFunction Parallelism

2023-09-26 Thread patricia lee
Hi, Are processwindowfunctions cannot have more than 1 parallelism? Whenever I set it to 2, I am receiving an error message, "The parallelism of non parallel operator must be 1." dataKafka = Kafkasource (datasource) .parallelism(2) .rebalance(); dataKafka.windowAll(GlobalWindows.create()).trigge

Re: Is there any example that use hbase connector in stream API

2023-09-26 Thread Антон
Hi, I wrote to hbase using map extending RichMapFunction. I initialized the connection in overrided open method and wrote to hbase in overrided map method. 22.09.2023, 05:01, "碳酸君" :hi community: I'm trying to write some data to hbase in a stream job ,withflink-connector-hbase-2.2 . I have

Re: Side outputs documentation

2023-09-26 Thread Alexis Sarda-Espinosa
I see, sounds good, thanks for the clarification. Am Di., 26. Sept. 2023 um 03:29 Uhr schrieb Yunfeng Zhou < flink.zhouyunf...@gmail.com>: > Hi Alexis, > > Thanks for the clarification. I found the second constructor on > Flink's master branch here[1], and maybe it was that we had been > commenti