--
From:Jark Wu
Send Time:2019 Aug. 27 (Tue.) 16:27
To:dev
Cc:Yun Gao
Subject:Re: [DISCUSS] Enhance Support for Multicast Communication Pattern
Hi all,
Thanks Yun for bringing this topic. I missed this discussion because of the
"multicast" title.
After reading the design, if I
s on the Runtime,
> >> like broadcasting events, to have a complete design. Therefore, I think
> we
> >> may still first have the broadcasting problem settled in this thread?
> Based
> >> on the points learned in the discussion, now I think that we might be
> able
&g
equirements and more generalized
>> multicasting mechanism. :)
>>
>> Best,
>> Yun
>>
>>
>>
>> ------
>> From:SHI Xiaogang
>> Send Time:2019 Aug. 27 (Tue.) 09:16
>> To:dev
at is it possible for us to partition to multiple steps and
> supports broadcasting events first ? At the same time we could also
> continue working on other options to support more scenarios like non-key
> join and they seems to requires more thoughts.
>
> Best,
> Yun
>
&g
n
--
From:Zhu Zhu mailto:reed...@gmail.com>>
Send Time:2019 Aug. 23 (Fri.) 17:25
To:dev mailto:dev@flink.apache.org>>
Cc:Yun Gao mailto:yungao...@aliyun.com>>
Subject:Re: [DISCUSS] Enhance Support for Multicast Communica
partitioner should be required since users may create any subgraph
> for
> the
> iteration body, including the operators with key. I think a possible
> solution to this issue is to introduce another data type for
> 'broadcastEmit'. For example, for an operator Operator, it may
> broadcast
>
suitable for both
keyed
or
non-keyed partitioner.
Best,
Yun
--
From:Piotr Nowojski
Send Time:2019 Aug. 23 (Fri.) 22:29
To:Zhu Zhu
Cc:dev ; Yun Gao
Subject:Re: [DISCUSS] Enhance Support for Multicast Communication
Pattern
Hi,
If the primary motivation is broadcasting (for the iterations) and
w
o 于2019年8月23日周五 下午11:56写道：
> >>>>>
> >>>>>>Hi Piotr,
> >>>>>>
> >>>>>> Very thanks for the suggestions!
> >>>>>>
> >>>>>> Totally agree with that we could first focus on the broadca
d
>>>>>> records to all the tasks may be confused considering the semantics
>> of
>>>> keyed
>>>>>> partitioner. However, in the iteration case supporting broadcast
>> over
>>>> keyed
>>>>>> partitioner should b
t; the
> > > >> iteration body, including the operators with key. I think a possible
> > > >> solution to this issue is to introduce another data type for
> > > >> 'broadcastEmit'. For example, for an operator Operator, it may
> > > broadcast
> > >> my side on solving this issue somehow. But before we start discussing
> > how
> > >> to solve it one last control question:
> > >>>
> > >>> I guess this multicast is intended to be used in blink planner,
> right?
> >
keyed context. This should result in the design
> to
> >> introduce customized operator event (option 1 in the document). The
> cost of
> >> this method is that we need to introduce a new type of StreamElement and
> >> new interface for this type, but it should be suitable for both keyed or
>
>>
>>>> On 23 Aug 2019, at 11:38, Yun Gao
>> wrote:
>>>>
>>>>Hi Piotr,
>>>>
>>>> Thanks a lot for sharing the thoughts!
>>>>
>>>> For the iteration, agree with that multicasting is not
ould be suitable for both keyed or
> non-keyed partitioner.
>
> Best,
> Yun
>
>
>
> --
> From:Piotr Nowojski
> Send Time:2019 Aug. 23 (Fri.) 22:29
> To:Zhu Zhu
> Cc:dev ; Yun Gao
> Subject:Re: [DISCUSS] Enhance Support for Multicast Communication Patt
Best,
Yun
--
From:Piotr Nowojski
Send Time:2019 Aug. 23 (Fri.) 22:29
To:Zhu Zhu
Cc:dev ; Yun Gao
Subject:Re: [DISCUSS] Enhance Support for Multicast Communication Pattern
Hi,
If the primary motivation is broadcasting (for the iterations) and we have no
immediate need for multicast (
ChannelSelector#selectChannels()`, I'm wondering
> >> if we introduce a new MulticastRecordWriter and left the current
> >> RecordWriter untouched, could we avoid the performance degradation ? Since
> >> with such a modification the normal RecordWriter does not n
gt; > --------------------------
> > From:Zhu Zhu
> > Send Time:2019 Aug. 23 (Fri.) 17:25
> > To:dev
> > Cc:Yun Gao
> > Subject:Re: [DISCUSS] Enhance Support for Multicast Communication Pattern
> >
> > Hi Pio
e cross join case!
> Best,
> Yun
>
>
>
> --
> From:Zhu Zhu
> Send Time:2019 Aug. 23 (Fri.) 17:25
> To:dev
> Cc:Yun Gao
> Subject:Re: [DISCUSS] Enhance Support for Multicast Communication Pattern
>
> Hi Piotr,
>
&
wojski
> Send Time:2019 Aug. 23 (Fri.) 15:20
> To:dev
> Cc:Yun Gao
> Subject:Re: [DISCUSS] Enhance Support for Multicast Communication Pattern
>
> Hi,
>
> Yun:
>
> Thanks for proposing the idea. I have checked the document and left couple
> of questi
> > Best,
> > Yun
> >
> > --
> > From:Piotr Nowojski
> > Send Time:2019 Aug. 23 (Fri.) 15:20
> > To:dev
> > Cc:Yun Gao
> > Subject:Re: [DISCUSS] Enhance Support for Multicast
er, and accessing the first element of the returned array
> instead of reading the integer directly.
>
> Best,
> Yun
>
> --
> From:Piotr Nowojski
> Send Time:2019 Aug. 23 (Fri.) 15:20
> To:dev
&
Best,
Yun
--
From:Piotr Nowojski
Send Time:2019 Aug. 23 (Fri.) 15:20
To:dev
Cc:Yun Gao
Subject:Re: [DISCUSS] Enhance Support for Multicast Communication Pattern
Hi,
Yun:
Thanks for proposing the idea. I have checked the documen
Hi Piotr,
The case is about a broadcast join:
A--\
+--(join)--> C
B--/
Usually we can broadcast A(the result that JobVertex A produces) to all
subtasks of C.
But in this case the size of A is too large to fit in one subtask of C.
Thus we have to partition A to (A_0, A_1, A_2, ..., A_m-1).
Th
Hi,
Yun:
Thanks for proposing the idea. I have checked the document and left couple of
questions there, but it might be better to answer them here.
What is the exact motivation and what problems do you want to solve? We have
dropped multicast support from the network stack [1] for two reasons:
Thanks Yun for starting this discussion.
I think the multicasting can be very helpful in certain cases.
I have received requirements from users that they want to do broadcast
join, while the data set to broadcast is too large to fit in one task.
Thus the requirement turned out to be to support car
Hi everyone,
In some scenarios we met a requirement that some operators want to send
records to theirs downstream operators with an multicast communication pattern.
In detail, for some records, the operators want to send them according to the
partitioner (for example, Rebalance), and for s
26 matches
Mail list logo
