Re: Built-in functions to manipulate MULTISET type

2021-09-18 Thread Yuval Itzchakov
Hi Jing, I recall there is already an open ticket for built-in aggregate functions On Sat, Sep 18, 2021, 15:08 JING ZHANG wrote: > Hi Yuval, > You could open a JIRA to track this if you think some functions should be > added as built-in functions in Flink. > > Best, > JING ZHANG > > Yuval Itzch

Re: Flink Native Kubernetes - Configuration kubernetes.flink.log.dir not working

2021-09-18 Thread bat man
I was able to verify that this works fine Dkubernetes.flink.log.dir="/var/log/containers" with Flink 1.13.1. Initially there were some issues with the deployment, once fixed it worked fine. Cheers, Hemant On Sat, Sep 18, 2021 at 1:58 PM Yang Wang wrote: > I think it might be a bug that "kuberne

Re: Built-in functions to manipulate MULTISET type

2021-09-18 Thread JING ZHANG
Hi Yuval, You could open a JIRA to track this if you think some functions should be added as built-in functions in Flink. Best, JING ZHANG Yuval Itzchakov 于2021年9月18日周六 下午3:33写道: > The problem with defining a UDF is that you have to create one overload > per key type in the MULTISET. It would b

throughput reduced when mini-batch is enabled; how to concat multiset using separator

2021-09-18 Thread vtygoss
Hi, Flink community! i have two problems, 1. how to concat multiset using separator? In spark sql: concat_ws(seperator, collect_set(column)). But in flink, the result data type of function 'collect(distinct column) ' is multiset, the corresponding class of multiset is org.apache.flink.

Re: IAM Roles with Service Account on Flink 1.12 Running on Kubernetes - Seeing Errors

2021-09-18 Thread Yang Wang
It seems that the application failed to submit the job due to akka timeout, not about the service account. It will help a lot to debug the root cause if you could share the full JobManager logs. If the JobManager does not have enough permissions to create TaskManager pods and watch these pods, you

Re: Job manager crash

2021-09-18 Thread Yang Wang
The GC log looks quite normal. Maybe the K8s APIServer is overloaded. Best, Yang houssem 于2021年9月13日周一 下午5:11写道: > hello, > > here's some of full GC log: > > OpenJDK 64-Bit Server VM (25.232-b09) for linux-amd64 JRE (1.8.0_232-b09), > built on Oct 18 2019 15:04:46 by "jenkins" with gcc 4.8.2 20

Re: Flink Native Kubernetes - Configuration kubernetes.flink.log.dir not working

2021-09-18 Thread Yang Wang
I think it might be a bug that "kubernetes.flink.log.dir" could not take effect. I have created a ticket[1]. Could you please try with "-Denv.log.dir=/var/log/containers"? [1]. https://issues.apache.org/jira/browse/FLINK-24334 Best, Yang Guowei Ma 于2021年9月14日周二 下午4:48写道: > Hi > > Maybe you co

Re: Built-in functions to manipulate MULTISET type

2021-09-18 Thread Yuval Itzchakov
The problem with defining a UDF is that you have to create one overload per key type in the MULTISET. It would be very convenient to have functions like Snowflakes ARRAY_AGG. On Sat, Sep 18, 2021, 05:43 JING ZHANG wrote: > Hi Kai, > AFAIK, there is no built-in function to extract the keys in MUL