Re: How to rebalance a table without converting to dataset

2018-04-17 Thread Shuyi Chen
Hi Darshan, thanks for raising the problem. We do have similar use of rebalancing in Flink SQL, where we want to rebalance the Kafka input with more partitions to increase parallelism in streaming. As Fabian suggests, rebalancing is not relation algebra. The closest use of the operation I can find

Re: How to rebalance a table without converting to dataset

2018-04-16 Thread Fabian Hueske
Hi Darshan, You are right. there's currently no rebalancing operation on the Table API. I see that this might be a good feature, not sure though how easy it would be to integrate because we need to pass it through the Calcite optimizer and rebalancing is not a relational operation. For now, conve

How to rebalance a table without converting to dataset

2018-04-13 Thread Darshan Singh
Hi I have a table and I want to rebalance the data so that each partition is equal. I cna convert to dataset and rebalance and then convert to table. I couldnt find any rebalance on table api. Does anyone know any better idea to rebalance table data? Thanks