Re: Parallelizing a tumbling group window

2018-01-11 Thread Colin Williams
Thanks for the reply. Unfortunately that project was unexpectedly cancelled but for other reasons. I was happy to work on it, and hopefully gained some insight. I have another question today unrelated towards Elasticsearch sinks, and will ask there. On Fri, Jan 5, 2018 at 2:52 AM, Fabian Hueske w

Re: Parallelizing a tumbling group window

2018-01-05 Thread Fabian Hueske
Hi Colin, There are two things that come to my mind: 1) You mentioned "suspect jobs are grouping by a field of constant values". Does that mean that the grouping key is always constant? Flink parallelizes the window computation per key, i.e., there is one thread per key. Although it would be poss

Re: Parallelizing a tumbling group window

2017-12-29 Thread Colin Williams
Hi Timo and flink-user, It's been a few weeks and we've made some changes to the application mentioned on this email. we've also updated for flink 1.4 . We are using the SQL / Table API with a tumbling window and user defined agg to generate a SQL query string like: SELECT measurement, `tag_App

Re: Parallelizing a tumbling group window

2017-12-11 Thread Timo Walther
Hi Colin, unfortunately, selecting the parallelism for parts of a SQL query is not supported yet. By default, tumbling window operators use the default parallelism of the environment. Simple project and select operations have the same parallelism as the inputs they are applied on. I think th

Parallelizing a tumbling group window

2017-12-08 Thread Colin Williams
Hello, I've inherited some flink application code. We're currently creating a table using a Tumbling SQL query similar to the first example in https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/table/sql. html#group-windows Where each generated SQL query looks something like SELE