Re: Is there a way to avoid submit hive-udf's resources when we submit a job?

2020-09-22 Thread Rui Li
Hi Timo, I believe the blocker for this feature is that we don't support dynamically adding user jars/resources at the moment. We're able to read the path to the function jar from Hive metastore, but we cannot load the jar after the user session is started. On Tue, Sep 22, 2020 at 3:43 PM Timo Wa

Re: Is there a way to avoid submit hive-udf's resources when we submit a job?

2020-09-22 Thread Husky Zeng
Hi Timo, Thanks for your attention,As what I say in this comment, this feature can surely solve our problem, but it seems that the workload is much larger than the solution in my scenario. Our project urgently needs to solve the problem of reusing hive UDF in hive metastore, so we are more incline

Re: Is there a way to avoid submit hive-udf's resources when we submit a job?

2020-09-22 Thread Timo Walther
Hi Husky, I guess https://issues.apache.org/jira/browse/FLINK-14055 is what is needed to make this feature possible. @Rui: Do you know more about this issue and current limitations. Regards, Timo On 18.09.20 09:11, Husky Zeng wrote: When we submit a job which use udf of hive , the job will

Is there a way to avoid submit hive-udf's resources when we submit a job?

2020-09-18 Thread Husky Zeng
When we submit a job which use udf of hive , the job will dependent on udf's jars and configuration files. We have already store udf's jars and configuration files in hive metadata store,so we excpet that flink could get those files hdfs paths by hive-connector,and get those files in hdfs by paths