Request multiple subpartitions of one partition

2019-01-09 Thread Chris Miller
Hello, let's image we do a hash join of two DataSources. For the join operation we choose parallelism=5. This way Flink uses 5 TaskManagerRunners to do the join job. In particular, the DataSource tasks, EACH ARE CREATING 5 SUBPARTITIONS. Every worker, now requests ONE SUBPARTITION from both

Identify the task within PartitionRequestQueue

2019-01-12 Thread Chris Miller
Hi all, I just made some changes within the flink/runtime/io/network/netty/PartitionRequestQueue.java These changes should only be aplied to Join-Tasks. Is there a way to figure out which Task the current PartitionRequestQueue object relates to? Something like: if(this.task.getTaskInfo()

Disable local data transportation

2019-01-13 Thread Chris Miller
Hi all, let's have a look at a simple Join with two DataSources and parallelism p=5. The whole Job consists of 3 parts: 1. DataSource Task 2. Join Task 3. DataSink Task In the first task, the data is provided and prepared for the Join task. In particular each DataSource task creates

Re: Request multiple subpartitions of one partition

2019-01-23 Thread Chris Miller
Hi Zhijang, thank you for your replay. I was playing around a little in the last days and ended up in a solution where I change the ResultPartitionView's subpartitionIndex as soon as it returns an EndOfPartition Event. This way I can, sequentially, receive multiple subpartitions at one single

[jira] [Created] (FLINK-15112) NPE when trying to join on two temporal tables in a single query

2019-12-06 Thread Chris Miller (Jira)
Chris Miller created FLINK-15112: Summary: NPE when trying to join on two temporal tables in a single query Key: FLINK-15112 URL: https://issues.apache.org/jira/browse/FLINK-15112 Project: Flink