Re: Complex batch workflow needs (too) much time to create executionPlan

2016-09-26 Thread Fabian Hueske
Hi Markus, thanks for the stacktraces! The client is indeed stuck in the optimizer. I have to look a bit more into this. Did you try to set JoinHints in your plan? That should reduce the plan space that is enumerated and therefore reduce the optimization time (maybe enough to run your application

Re: Complex batch workflow needs (too) much time to create executionPlan

2016-09-26 Thread Markus Nentwig
Hi Fabian, at first, sorry for the late answer. The given execution plan was created after 20 minutes, only one vertex centric iteration is missing. I can optimize the program because some operators are only needed to create intermediate debug results, still, it's not enough to run as one Flink j

Re: Complex batch workflow needs (too) much time to create executionPlan

2016-08-22 Thread Fabian Hueske
Hi Markus, you might be right, that a lot of time is spend in optimization. The optimizer enumerates all alternatives and chooses the plan with the least estimated cost. The degrees of freedom of the optimizer are rather restricted (execution strategies and the used partitioning & sorting keys. Th