RE: flink memory management / temp-io dir question

tas Kloudas Cc: user; Till Rohrmann Subject: RE: flink memory management / temp-io dir question Hi Till, Thanks for the reply. I don’t use batch, so I assume what I am seeing is streaming related. I thought rocksdb writes to a different dir though ( as defined by checkpoint.data.uri)? Regard

RE: flink memory management / temp-io dir question

: Kostas Kloudas Cc: Gopinath, Anand; user; Till Rohrmann Subject: Re: flink memory management / temp-io dir question Hi Anand, spilling using the io directories is only relevant for Flink's batch processing. This happens, for example if you enable blocking data exchange where the produced data c

Re: flink memory management / temp-io dir question

Hi Anand, spilling using the io directories is only relevant for Flink's batch processing. This happens, for example if you enable blocking data exchange where the produced data cannot be kept in memory. Moreover, it is used by many of Flink's out-of-core data structures to enable exactly this fea

Re: flink memory management / temp-io dir question

Sorry, I forgot to cc’ Till. > On Oct 8, 2018, at 2:17 PM, Kostas Kloudas > wrote: > > Hi Anand, > > I think that Till is the best person to answer your question. > > Cheers, > Kostas > >> On Oct 5, 2018, at 3:44 PM, anand.gopin...@ubs.com >> wrote: >> >> Hi

Re: flink memory management / temp-io dir question

Hi Anand, I think that Till is the best person to answer your question. Cheers, Kostas > On Oct 5, 2018, at 3:44 PM, anand.gopin...@ubs.com wrote: > > Hi , > I had a question with respect flink memory management / overspill to /tmp. > > In the docs > (https://ci.apache.org/projects/flink/fl

flink memory management / temp-io dir question

Hi , I had a question with respect flink memory management / overspill to /tmp. In the docs (https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/config.html#configuring-temporary-io-directories) it says: Although Flink aims to process as much data in main memory as possible, it is n