Hi, you provide not very much information for this question, e.g. what and how exactly your measure, if this is a local or distributed setting etc. I assume that it is distributed and that the cause for your observation is the buffer timeout, i.e. the maximum time that Flink waits until sending a buffer with just one element, which happens to be 100ms by default. You can decrease this value to some extend, at to cost of potential loss in throughput, but I think even values around 5-10ms are ok-ish. See [1] for more details. If you want to reduce latency between chained operators, you can also try to disable object-reuse:
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.getConfig().enableObjectReuse(); Best, Stefan > Am 20.09.2018 um 16:03 schrieb James Yu <cyu...@gmail.com>: > > The previous email seems unable to display embedded images, let me put on the > links. > Hi, > > My team and I try to measure total time spent on our flink job and found out > that Flink takes 40ms ~ 100ms to proceed from one operator to another. I > wonder how can we reduce this transition time. > > Following DAG represents our job: > https://drive.google.com/open?id=1wNR8po-SooAfYtCxU3qUDm4-hm16tclV > <https://drive.google.com/open?id=1wNR8po-SooAfYtCxU3qUDm4-hm16tclV> > > > and here is the screenshot of our log: > https://drive.google.com/open?id=14PTViZMkhagxeNjOb4R8u3BEr7Ym1xBi > <https://drive.google.com/open?id=14PTViZMkhagxeNjOb4R8u3BEr7Ym1xBi> > > at 19:37:04.564, the job is leaving "Source: Custom Source -> Flat Map" > at 19:37:04.605, the job is entering "Co-Flat Map" > at 19:37:04.605, the job is leaving "Co-Flat Map" > at 19:37:04.705, the job is entering "Co-Flat Map -> ...." > at 19:37:04.708, the job is leaving "Co-Flat Map -> ..." > > both "Co-Flat Map" finishes merely instantly, while most of the execution > time is spent on the transition. Any idea? > > > This is a UTF-8 formatted mail > ----------------------------------------------- > James C.-C.Yu > +886988713275 > +8615618429976 >
