Re: connection timeout during shuffle initialization

------------------ > From:Piotr Nowojski > Send Time:2020 Jan. 30 (Thu.) 19:29 > To:dev ; zhijiang > Subject:Re: connection timeout during shuffle initialization > > One more thing. Could you create a JIRA ticket for this issue? We could also > m

Re: connection timeout during shuffle initialization

. Let's forward the further solutions on the jira page as Piotr suggested. :) Best, Zhijiang -- From:Piotr Nowojski Send Time:2020 Jan. 30 (Thu.) 19:29 To:dev ; zhijiang Subject:Re: connection timeout during shuffle initializ

Re: connection timeout during shuffle initialization

One more thing. Could you create a JIRA ticket for this issue? We could also move the discussion there. Piotrek > On 30 Jan 2020, at 12:14, Piotr Nowojski wrote: > > Hi, > >>> I think it's perfectly ok to perform IO ops in netty threads, > (…) >>> Removing synchronization *did solve* the prob

Re: connection timeout during shuffle initialization

Hi, >> I think it's perfectly ok to perform IO ops in netty threads, (…) >> Removing synchronization *did solve* the problem for me, because it >> allows flink to leverage the whole netty event loop pool and it's ok to >> have a single thread blocked for a little while (we still can accept >> conn

Re: connection timeout during shuffle initialization

Just to clarify, these are bare metal nodes (128G ram, 16 cpus + hyperthreading, 4xHDDS, 10g network), which run yarn, hdfs and hbase. D. On Wed, Jan 29, 2020 at 5:03 PM David Morávek wrote: > Hi Piotr, > > removal of buffer prefetch in BoundedBlockingSubpartitionReader did not > help, I've alr

Re: connection timeout during shuffle initialization

Hi Piotr, removal of buffer prefetch in BoundedBlockingSubpartitionReader did not help, I've already tried that (there are still other problematic code paths, eg. releasePartition). I think it's perfectly ok to perform IO ops in netty threads, we just have to make sure, we can leverage multiple th

Re: connection timeout during shuffle initialization

Hi David, > with high load and io waits How high values are talking about? Could you attach a CPU profiler and post the results somehow? Which threads are busy on what call trees? Regarding the idea of removing of the locks in the `ResultPartitionManager`. I guess it could help a bit, but

Re: connection timeout during shuffle initialization

/CC Piotr and Zhijiang Sounds reasonable at first glance. Would like to hear Piotr's and Zhijiang's take, though, they know that code better than me. On Wed, Jan 29, 2020 at 1:58 PM David Morávek wrote: > Hi Stephan, > > I've actually managed to narrow problem down to blocked netty server > thr

Re: connection timeout during shuffle initialization

Hi Stephan, I've actually managed to narrow problem down to blocked netty server threads. I'm using 1.9.1 with few custom patches , that are not relevant to this issue. To highlight the problem, I've added these checks to ResultPartitionManager

Re: connection timeout during shuffle initialization

Hi! Concerning JAR files: I think this has nothing to do with it, it is a batch shuffle after all. The previous stage must have completed already. A few things that come to my mind: - What Flink version are you using? 1.9? - Are you sure that the source TaskManager is still running? Earlier F

Re: connection timeout during shuffle initialization

Hi, > In case of large jar, wouldn't this happen in previous stages as well (if > so this should not be the case)? I’m not exactly sure how jars are distributed, but if they are being sent/uploaded from one (or some other static/fixed number, like uploading to and reading from a DFS) node to al

Re: connection timeout during shuffle initialization

Hi Piotr, thanks for suggestions! In case of large jar, wouldn't this happen in previous stages as well (if so this should not be the case)? Also there shouldn't be any state involved (unless Beam IO's use it internally). I'll get back to you with with the results after checking TM's io stats.

Re: connection timeout during shuffle initialization

Hi David, The usual cause for connection time out is long deployment. For example if your Job's jar is large and takes long time to distribute across the cluster. I’m not sure if large state could affect this as well or not. Are you sure that’s not the case? The think you are suggesting, I hav

Re: connection timeout during shuffle initialization

Hi David, I'm unfortunately not familiar with these parts of Flink but I'm pulling Piotr in who might be able to tell you more. Cheers, Till On Mon, Jan 27, 2020 at 5:58 PM David Morávek wrote: > Hello community, > > I'm currently struggling with an Apache Beam batch pipeline on top of > Flink

connection timeout during shuffle initialization

Hello community, I'm currently struggling with an Apache Beam batch pipeline on top of Flink. The pipeline runs smoothly in smaller environments, but in production it always ends up with `connection timeout` in one of the last shuffle phases. org.apache.flink.runtime.io.network.partition.consumer