PS: I forgot to mention that also constant iteration input is cached.
On Mon, Jul 18, 2016 at 11:27 AM, Ufuk Celebi <u...@apache.org> wrote: > Hey Saliya, > > the result of each iteration (super step) that is fed back to the > iteration is cached. For the iterate operator that is the last partial > solution and for the delta iterate operator it's the current solution > set > (https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/batch/iterations.html). > > Internally, this works via custom iteration operator implementations > for head and tail tasks, which are co-located and share a hash table. > I think that the internals of this are not documented, you would have > to look into the code for this. Most of the relevant implementations > are found in the "org.apache.flink.runtime.iterative.task" package. > > Hope this helps... > > Ufuk > > > On Sun, Jul 17, 2016 at 9:36 PM, Saliya Ekanayake <esal...@gmail.com> wrote: >> Hi, >> >> I am trying to understand what's the intermediate caching support in Flink. >> For example, when there's an iterative dataset what's being cached between >> iterations. Is there some documentation on this? >> >> Thank you, >> Saliya >> >> -- >> Saliya Ekanayake >> Ph.D. Candidate | Research Assistant >> School of Informatics and Computing | Digital Science Center >> Indiana University, Bloomington >>
