Re: Set job specific resources in one StreamTableEnvironment

2021-07-20 Thread Paul Lam
Hi Yun, Thanks a lot for your reply! Regarding the parallelism, I think `table.exec.resource.default-parallelism` that you mentioned is a good alternative, but it requires a set operation before running each query. And since it’s a `default` value, I suppose there should be an option with highe

Re: Set job specific resources in one StreamTableEnvironment

2021-07-19 Thread Yun Gao
Hi Paul, For parallelism, it should be able to be set with `table.exec.resource.default-parallelism` [1] , and an example to set the parameter is at the first several paragraph. But Regarding the total process memory, I think it should be only set in the cluster level since it is per-cluster

Set job specific resources in one StreamTableEnvironment

2021-07-15 Thread Paul Lam
Hi, I’m reusing the same StreamTableEnvironment to submit multiple table/sql jobs to a session cluster, but I couldn’t find a proper way to specify job resources for each job (like parallelism and total process memory), and they all uses the cluster default. I have considered overriding reso