Integarting Apache Flink with Apache Ranger

2023-08-08 Thread arjun s
Hi, I'm interested to know if there is an available feature for integrating Apache Flink with Apache Ranger. If so, could you kindly share the relevant documentation with me? Thanks and regards, Arjun S

Re: About[jobmanager.memory.off-heap.size] not work on Flink(flink-session on yarn)

2023-08-08 Thread liu ron
Hi, wenjiang It does seem a bit odd that it could theoretically work either way. I think you can check the Flink JobManager log to find more information. Best, Ron 傅文江 于2023年7月24日周一 17:37写道: > > When I use ESSink, I find that I need to set Flink’s > jobmanager.memory.off-heap.size to 256MB.

Re: Questions related to Autoscaler

2023-08-08 Thread liu ron
Hi, Lijuan > 1 - It seems for flink job using flink operator to realize autoscaling, the only option to realize autoscaling is to enable the Autoscaler feature, and KEDA won’t work, right? What is KEDA mean? > 2 - I noticed from the document that we need to upgrade to flink version of 1.17 to u

Re: Streaming join performance

2023-08-08 Thread liu ron
Hi, David Regarding the N-way join, this feature aims to address the issue of state simplification, it is on the roadmap. Technically there are no limitations, but we'll need some time to find a sensible solution. Best, Ron David Anderson 于2023年8月9日周三 10:38写道: > This join optimization sounds p

Re: Easiest way to do a batch outer join

2023-08-08 Thread liu ron
Hi, Flavio IMO, the current DataStream API is not aligned with DataSet in terms of capabilities, I think you can try it with GlobalWindow. Another possible solution is to convert the DataStream to a table[1] first and then try it with a join on the Table API. [1] https://nightlies.apache.org/flin

Re: Streaming join performance

2023-08-08 Thread David Anderson
This join optimization sounds promising, but I'm wondering why Flink SQL isn't taking advantage of the N-Ary Stream Operator introduced in FLIP-92 [1][2] to implement a n-way join in a single operator. Is there something that makes this impossible/impractical? [1] https://cwiki.apache.org/confluen

Re: Investigating use of Custom Trigger to smooth CPU usage

2023-08-08 Thread Tucker Harvey via user
Hi David and Xiangyu, For more context, We have a job running on our cluster aggregating StatsD metrics in tumbling window, which causes CPU usage spikes due to concurrent executions. While staggering the windows using StaggerWindow could help this will impact the job's accuracy. Instead, we

Questions related to Autoscaler

2023-08-08 Thread Hou, Lijuan via user
Hi Flink team, This is Lijuan. I am working on our flink job to realize autoscaling. We are currently using flink version of 1.16.1, and using flink operator version of 1.5.0. I have some questions need to confirm with you. 1 - It seems for flink job using flink operator to realize autoscaling,