On Mon, Mar 18, 2019 at 6:54 PM Ivan Kelly wrote:
> Sorry it took me so long to take a look at this, last few weeks have
> been hectic.
>
> I still haven't gone through it fully, but putting the transaction
> buffer outside of the partition is fine with me. The thing I objected
> most to in previ
Sorry it took me so long to take a look at this, last few weeks have
been hectic.
I still haven't gone through it fully, but putting the transaction
buffer outside of the partition is fine with me. The thing I objected
most to in previous discussions was having a separate transaction
buffer per tr
Any other more comments on this topic?
- Sijie
On Sun, Mar 10, 2019 at 8:57 PM Jia Zhai wrote:
> Thanks @Sijie for the PIP.
> It has with enough details for me, It looks great, especially for the
> sidecar
> approach. Left some comments.
>
> Best Regards.
>
>
> Jia Zhai
>
> Beijing, China
>
> M
Thanks @Sijie for the PIP.
It has with enough details for me, It looks great, especially for the sidecar
approach. Left some comments.
Best Regards.
Jia Zhai
Beijing, China
Mobile: +86 15810491983
On Fri, Mar 8, 2019 at 9:58 PM Sijie Guo wrote:
> Hi Team,
>
> I have written down all my t
Hi Team,
I have written down all my thoughts around supporting transactional
streaming at Pulsar.
https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx
Please take a look and feel free to comment on the google doc. We can start
from there.
A
Will send the detailed proposal. We can go from there.
One interesting question I would like to reply here.
> But this is more microbatching than streaming.
I think people usually have a wrong impression about "microbatching" vs
"streaming".
The "microbatching" vs "streaming" are usually found i
> > My replies inline assume the above, so if you have a different view of
> > the general shape let me know.
> >
>
> Yes. We are on the same view of the general shape. I will write down the
> details of my proposal and will share it with the community tomorrow.
Please do. I think there's a lot of
On Tue, Mar 5, 2019 at 12:37 AM Ivan Kelly wrote:
> I think we agree on the general shape of the design as follows.
>
> - Transactions store message data somewhere that it is not
> materialized to the client immediately.
> - On commit, a single message is written to some log
> - Commit messages a
I think we agree on the general shape of the design as follows.
- Transactions store message data somewhere that it is not
materialized to the client immediately.
- On commit, a single message is written to some log
- Commit messages are then written to the topics of the logs in the
transaction, w
On Mon, Mar 4, 2019 at 6:15 PM Ivan Kelly wrote:
> > > Transactional acknowledgement also needs to be taken into account
> >
> > I don't think we have to treat `transactional acknowledgement` as a
> special
> > case. currently `acknowledgment` are actually "append" operations into
> > cursor ledg
> > Transactional acknowledgement also needs to be taken into account
>
> I don't think we have to treat `transactional acknowledgement` as a special
> case. currently `acknowledgment` are actually "append" operations into
> cursor ledgers.
> So the problem set can be reduced as `atomic append` to
;>
> > > >>>>>>
> > > >>>>>> On Thu, Feb 28, 2019 at 4:29 PM Richard Yu <
> > > >> yohan.richard...@gmail.com
> > > >>>>>
> > > >>>>>> wrote:
> > > >>>
t; an
> > >>>>>>> organized manner:
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://docs.google.com/document/d/1mSU
unknown", do you mean the broker will cache
> >>>> the
> >>>>>>>> message, not writing it to any log?
> >>>>>>>
> >>>>>>> We wouldn't cache the message from my interpretation of the steps.
> >
>>> message
>>>>>> is
>>>>>>> on its way. So all we have to do is store the message id and its
>>>>>>> corresponding status in a map, and depending on the producer's
>>>>> response,
>>>>>>> the status
age itself. This step basically notifies the broker that the
> > >>> message
> > >>>> is
> > >>>>> on its way. So all we have to do is store the message id and its
> > >>>>> corresponding status in a map, and depending o
>> In designs we've discussed previously, this was handled
> >>>>>> by a component called the transaction coordinator, which is a
> >> logical
> >>>>>> component which each broker knows how to talk to. For a transaction
> >>>&g
e is sent to the coordinator, which writes it to
>> its
>>>>>> own log, and then goes through each topic in the commit and marks
>> the
>>>>>> transaction as completed.
>>>>>
>>>>> I wasn't aware of previous disc
> Hope we can get this PIP rolling.
> > > >
> > > >
> > > > On Thu, Feb 28, 2019 at 2:53 AM Sijie Guo
> wrote:
> > > >
> > > >> Richard,
> > > >>
> > > >> Thank you for putting this put and pushing the
rward.
> > >>
> > >> I think this is a very large feature. It might be worth creating a
> > google
> > >> doc for it (which is better for collaboration). And I believe Ivan has
> > >> some
> > >> thoughts as well. If you can put up
c (make it
> world-editable),
> >> Ivan can probably dump his thoughts there and we can finalize the
> >> discussion and break down into tasks. So the whole community can
> actually
> >> work together at collaborating this.
> >>
> >> Thanks,
&g
t;> On Thu, Feb 28, 2019 at 1:08 PM Richard Yu
>> wrote:
>>
>> > Hi all,
>> >
>> > I would like to create a PIP for issue #2664 on Github. The details of
>> the
>> > PIP are below.
>> > I hope we could discuss this thoroughly.
>
e
>
> On Thu, Feb 28, 2019 at 1:08 PM Richard Yu
> wrote:
>
> > Hi all,
> >
> > I would like to create a PIP for issue #2664 on Github. The details of
> the
> > PIP are below.
> > I hope we could discuss this thoroughly.
> >
> > Cheers,
a PIP for issue #2664 on Github. The details of the
> PIP are below.
> I hope we could discuss this thoroughly.
>
> Cheers,
> Richard
>
> PIP-31: Add support for transactional messaging
>
> Motivation: Pulsar currently could improve upon their system of sending
> packet
Hi Richard,
This is something that has been discussed a few times, but the
outcomes of the discussions never written down. Thanks for starting
the conversation.
Responses to your PIP inline.
> Motivation: Pulsar currently could improve upon their system of sending
> packets of data by implementi
t; Hi all,
>
> I would like to create a PIP for issue #2664 on Github. The details of the
> PIP are below.
> I hope we could discuss this thoroughly.
>
> Cheers,
> Richard
>
> PIP-31: Add support for transactional messaging
>
> Motivation: Pulsar currently could impr
Hi all,
I would like to create a PIP for issue #2664 on Github. The details of the
PIP are below.
I hope we could discuss this thoroughly.
Cheers,
Richard
PIP-31: Add support for transactional messaging
Motivation: Pulsar currently could improve upon their system of sending
packets of data by
27 matches
Mail list logo
