Chiwan has a good point. Once the data that needs to be available to all machines is too large for one machine, there is no good solution any more. The best approach is an external store to which all nodes have access. It is not going to be terribly fast, though.
If you are in the situation that you need to broadcast so much data, you may want to rethink your approach to the problem in the first place. Is there no solution that can work with partitioned data? Even at the cost of re-partitioning twice or so? On Thu, Sep 3, 2015 at 10:35 AM, Chiwan Park <chiwanp...@apache.org> wrote: > Hi hagersaleh, > > Sorry for late reply. > > I think using an external system could be a solution for large scale data. > To use an external system, you have to implement rich functions such as > RichFilterFunction, RichMapFunction, …, etc. > > > Regards, > Chiwan Park > > > > On Aug 30, 2015, at 1:30 AM, hagersaleh <loveallah1...@yahoo.com> wrote: > > > > where are any ways for use broadcast variable with bigdata > > > > > > > > -- > > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/when-use-broadcast-variable-and-run-on-bigdata-display-this-error-please-help-tp2455p2566.html > > Sent from the Apache Flink User Mailing List archive. mailing list > archive at Nabble.com. > > > > >
