[
https://issues.apache.org/jira/browse/FLINK-785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319938#comment-14319938
]
ASF GitHub Bot commented on FLINK-785:
--------------------------------------
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/370#issuecomment-74236482
I think for the combiner, we cannot do this. The way it is done here easily
blows up the memory, by collecting so many records. Bear in mind that a record
is not necessarily small, in the ALS case, they were 500MB large. Times 10k,
that is 5TB of buffered records. The non-chaining combiner variant makes sure
that only two objects are ever around by passing records/fragments through the
buffers in the local channels.
> Add Chained operators for AllReduce and AllGroupReduce
> ------------------------------------------------------
>
> Key: FLINK-785
> URL: https://issues.apache.org/jira/browse/FLINK-785
> Project: Flink
> Issue Type: Improvement
> Reporter: GitHub Import
> Assignee: Chesnay Schepler
> Labels: github-import
> Fix For: pre-apache
>
>
> Because the operators `AllReduce` and `AllGroupReduce` are used both for the
> pre-reduce (combiner side) and the final reduce, they would greatly benefit
> from a chained version.
> ---------------- Imported from GitHub ----------------
> Url: https://github.com/stratosphere/stratosphere/issues/785
> Created by: [StephanEwen|https://github.com/StephanEwen]
> Labels: runtime,
> Milestone: Release 0.6 (unplanned)
> Created at: Sun May 11 17:41:12 CEST 2014
> State: open
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
