Hi Lucas, Thanks for the explanation. It might be a nitpick, but it seems better to mention in the motivation part that today the client requests and controller requests are not only sharing the same queue, but also a bunch of things else, so that we can avoid asking people to read the rejected alternatives.
Thanks, Jiangjie (Becket) Qin On Fri, Aug 10, 2018 at 6:23 AM, Lucas Wang <lucasatu...@gmail.com> wrote: > @Becket, > > I've asked for review by Jun and Joel in the vote thread. > Regarding the separate thread and port, I did talk about it in the rejected > alternative design 1. > Please let me know if you'd like more elaboration or moving it to the > motivation, etc. > > Thanks, > Lucas > > On Wed, Aug 8, 2018 at 3:59 PM, Becket Qin <becket....@gmail.com> wrote: > > > Hi Lucas, > > > > Yes, a separate Jira is OK. > > > > Since the proposal has significantly changed since the initial vote > > started. We probably should let the others who have already voted know > and > > ensure they are happy with the updated proposal. > > Also, it seems the motivation part of the KIP wiki is still just talking > > about the separate queue and not fully cover the changes we make now, > e.g. > > separate thread, port, etc. We might want to explain a bit more so for > > people who did not follow the discussion mail thread also understand the > > whole proposal. > > > > Thanks, > > > > Jiangjie (Becket) Qin > > > > On Wed, Aug 8, 2018 at 12:44 PM, Lucas Wang <lucasatu...@gmail.com> > wrote: > > > > > Hi Becket, > > > > > > Thanks for the review. The current write up in the KIP won’t change the > > > ordering behavior. Are you ok with addressing that as a separate > > > independent issue (I’ll create a separate ticket for it)? > > > If so, can you please give me a +1 on the vote thread? > > > > > > Thanks, > > > Lucas > > > > > > On Tue, Aug 7, 2018 at 7:34 PM Becket Qin <becket....@gmail.com> > wrote: > > > > > > > Thanks for the updated KIP wiki, Lucas. Looks good to me overall. > > > > > > > > It might be an implementation detail, but do we still plan to use the > > > > correlation id to ensure the request processing order? > > > > > > > > Thanks, > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > On Tue, Jul 31, 2018 at 3:39 AM, Lucas Wang <lucasatu...@gmail.com> > > > wrote: > > > > > > > > > Thanks for your review, Dong. > > > > > Ack that these configs will have a bigger impact for users. > > > > > > > > > > On the other hand, I would argue that the request queue becoming > full > > > > > may or may not be a rare scenario. > > > > > How often the request queue gets full depends on the request > incoming > > > > rate, > > > > > the request processing rate, and the size of the request queue. > > > > > When that happens, the dedicated endpoints design can better handle > > > > > it than any of the previously discussed options. > > > > > > > > > > Another reason I made the change was that I have the same taste > > > > > as Becket that it's a better separation of the control plane from > the > > > > data > > > > > plane. > > > > > > > > > > Finally, I want to clarify that this change is NOT motivated by the > > > > > out-of-order > > > > > processing discussion. The latter problem is orthogonal to this > KIP, > > > and > > > > it > > > > > can happen in any of the design options we discussed for this KIP > so > > > far. > > > > > So I'd like to address out-of-order processing separately in > another > > > > > thread, > > > > > and avoid mentioning it in this KIP. > > > > > > > > > > Thanks, > > > > > Lucas > > > > > > > > > > On Fri, Jul 27, 2018 at 7:51 PM, Dong Lin <lindon...@gmail.com> > > wrote: > > > > > > > > > > > Hey Lucas, > > > > > > > > > > > > Thanks for the update. > > > > > > > > > > > > The current KIP propose new broker configs > > "listeners.for.controller" > > > > and > > > > > > "advertised.listeners.for.controller". This is going to be a big > > > change > > > > > > since listeners are among the most important configs that every > > user > > > > > needs > > > > > > to change. According to the rejected alternative section, it > seems > > > that > > > > > the > > > > > > reason to add these two configs is to improve performance when > the > > > data > > > > > > request queue is full rather than for correctness. It should be a > > > very > > > > > rare > > > > > > scenario and I am not sure we should add configs for all users > just > > > to > > > > > > improve the performance in such rare scenario. > > > > > > > > > > > > Also, if the new design is based on the issues which are > discovered > > > in > > > > > the > > > > > > recent discussion, e.g. out of order processing if we don't use a > > > > > dedicated > > > > > > thread for controller request, it may be useful to explain the > > > problem > > > > in > > > > > > the motivation section. > > > > > > > > > > > > Thanks, > > > > > > Dong > > > > > > > > > > > > On Fri, Jul 27, 2018 at 1:28 PM, Lucas Wang < > lucasatu...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > A kind reminder for review of this KIP. > > > > > > > > > > > > > > Thank you very much! > > > > > > > Lucas > > > > > > > > > > > > > > On Wed, Jul 25, 2018 at 10:23 PM, Lucas Wang < > > > lucasatu...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > I've updated the KIP by adding the dedicated endpoints for > > > > controller > > > > > > > > connections, > > > > > > > > and pinning threads for controller requests. > > > > > > > > Also I've updated the title of this KIP. Please take a look > and > > > let > > > > > me > > > > > > > > know your feedback. > > > > > > > > > > > > > > > > Thanks a lot for your time! > > > > > > > > Lucas > > > > > > > > > > > > > > > > On Tue, Jul 24, 2018 at 10:19 AM, Mayuresh Gharat < > > > > > > > > gharatmayures...@gmail.com> wrote: > > > > > > > > > > > > > > > >> Hi Lucas, > > > > > > > >> I agree, if we want to go forward with a separate controller > > > plane > > > > > and > > > > > > > >> data > > > > > > > >> plane and completely isolate them, having a separate port > for > > > > > > controller > > > > > > > >> with a separate Acceptor and a Processor sounds ideal to me. > > > > > > > >> > > > > > > > >> Thanks, > > > > > > > >> > > > > > > > >> Mayuresh > > > > > > > >> > > > > > > > >> > > > > > > > >> On Mon, Jul 23, 2018 at 11:04 PM Becket Qin < > > > becket....@gmail.com > > > > > > > > > > > > wrote: > > > > > > > >> > > > > > > > >> > Hi Lucas, > > > > > > > >> > > > > > > > > >> > Yes, I agree that a dedicated end to end control flow > would > > be > > > > > > ideal. > > > > > > > >> > > > > > > > > >> > Thanks, > > > > > > > >> > > > > > > > > >> > Jiangjie (Becket) Qin > > > > > > > >> > > > > > > > > >> > On Tue, Jul 24, 2018 at 1:05 PM, Lucas Wang < > > > > > lucasatu...@gmail.com> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > > >> > > Thanks for the comment, Becket. > > > > > > > >> > > So far, we've been trying to avoid making any request > > > handler > > > > > > thread > > > > > > > >> > > special. > > > > > > > >> > > But if we were to follow that path in order to make the > > two > > > > > planes > > > > > > > >> more > > > > > > > >> > > isolated, > > > > > > > >> > > what do you think about also having a dedicated > processor > > > > > thread, > > > > > > > >> > > and dedicated port for the controller? > > > > > > > >> > > > > > > > > > >> > > Today one processor thread can handle multiple > > connections, > > > > > let's > > > > > > > say > > > > > > > >> 100 > > > > > > > >> > > connections > > > > > > > >> > > > > > > > > > >> > > represented by connection0, ... connection99, among > which > > > > > > > >> connection0-98 > > > > > > > >> > > are from clients, while connection99 is from > > > > > > > >> > > > > > > > > > >> > > the controller. Further let's say after one selector > > > polling, > > > > > > there > > > > > > > >> are > > > > > > > >> > > incoming requests on all connections. > > > > > > > >> > > > > > > > > > >> > > When the request queue is full, (either the data request > > > being > > > > > > full > > > > > > > in > > > > > > > >> > the > > > > > > > >> > > two queue design, or > > > > > > > >> > > > > > > > > > >> > > the one single queue being full in the deque design), > the > > > > > > processor > > > > > > > >> > thread > > > > > > > >> > > will be blocked first > > > > > > > >> > > > > > > > > > >> > > when trying to enqueue the data request from > connection0, > > > then > > > > > > > >> possibly > > > > > > > >> > > blocked for the data request > > > > > > > >> > > > > > > > > > >> > > from connection1, ... etc even though the controller > > request > > > > is > > > > > > > ready > > > > > > > >> to > > > > > > > >> > be > > > > > > > >> > > enqueued. > > > > > > > >> > > > > > > > > > >> > > To solve this problem, it seems we would need to have a > > > > separate > > > > > > > port > > > > > > > >> > > dedicated to > > > > > > > >> > > > > > > > > > >> > > the controller, a dedicated processor thread, a > dedicated > > > > > > controller > > > > > > > >> > > request queue, > > > > > > > >> > > > > > > > > > >> > > and pinning of one request handler thread for controller > > > > > requests. > > > > > > > >> > > > > > > > > > >> > > Thanks, > > > > > > > >> > > Lucas > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > On Mon, Jul 23, 2018 at 6:00 PM, Becket Qin < > > > > > becket....@gmail.com > > > > > > > > > > > > > > >> > wrote: > > > > > > > >> > > > > > > > > > >> > > > Personally I am not fond of the dequeue approach > simply > > > > > because > > > > > > it > > > > > > > >> is > > > > > > > >> > > > against the basic idea of isolating the controller > plane > > > and > > > > > > data > > > > > > > >> > plane. > > > > > > > >> > > > With a single dequeue, theoretically speaking the > > > controller > > > > > > > >> requests > > > > > > > >> > can > > > > > > > >> > > > starve the clients requests. I would prefer the > approach > > > > with > > > > > a > > > > > > > >> > separate > > > > > > > >> > > > controller request queue and a dedicated controller > > > request > > > > > > > handler > > > > > > > >> > > thread. > > > > > > > >> > > > > > > > > > > >> > > > Thanks, > > > > > > > >> > > > > > > > > > > >> > > > Jiangjie (Becket) Qin > > > > > > > >> > > > > > > > > > > >> > > > On Tue, Jul 24, 2018 at 8:16 AM, Lucas Wang < > > > > > > > lucasatu...@gmail.com> > > > > > > > >> > > wrote: > > > > > > > >> > > > > > > > > > > >> > > > > Sure, I can summarize the usage of correlation id. > But > > > > > before > > > > > > I > > > > > > > do > > > > > > > >> > > that, > > > > > > > >> > > > it > > > > > > > >> > > > > seems > > > > > > > >> > > > > the same out-of-order processing can also happen to > > > > Produce > > > > > > > >> requests > > > > > > > >> > > sent > > > > > > > >> > > > > by producers, > > > > > > > >> > > > > following the same example you described earlier. > > > > > > > >> > > > > If that's the case, I think this probably deserves a > > > > > separate > > > > > > > doc > > > > > > > >> and > > > > > > > >> > > > > design independent of this KIP. > > > > > > > >> > > > > > > > > > > > >> > > > > Lucas > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > On Mon, Jul 23, 2018 at 12:39 PM, Dong Lin < > > > > > > lindon...@gmail.com > > > > > > > > > > > > > > > >> > > wrote: > > > > > > > >> > > > > > > > > > > > >> > > > > > Hey Lucas, > > > > > > > >> > > > > > > > > > > > > >> > > > > > Could you update the KIP if you are confident with > > the > > > > > > > approach > > > > > > > >> > which > > > > > > > >> > > > > uses > > > > > > > >> > > > > > correlation id? The idea around correlation id is > > kind > > > > of > > > > > > > >> scattered > > > > > > > >> > > > > across > > > > > > > >> > > > > > multiple emails. It will be useful if other > reviews > > > can > > > > > read > > > > > > > the > > > > > > > >> > KIP > > > > > > > >> > > to > > > > > > > >> > > > > > understand the latest proposal. > > > > > > > >> > > > > > > > > > > > > >> > > > > > Thanks, > > > > > > > >> > > > > > Dong > > > > > > > >> > > > > > > > > > > > > >> > > > > > On Mon, Jul 23, 2018 at 12:32 PM, Mayuresh Gharat > < > > > > > > > >> > > > > > gharatmayures...@gmail.com> wrote: > > > > > > > >> > > > > > > > > > > > > >> > > > > > > I like the idea of the dequeue implementation by > > > > Lucas. > > > > > > This > > > > > > > >> will > > > > > > > >> > > > help > > > > > > > >> > > > > us > > > > > > > >> > > > > > > avoid additional queue for controller and > > additional > > > > > > configs > > > > > > > >> in > > > > > > > >> > > > Kafka. > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > Thanks, > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > Mayuresh > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > On Sun, Jul 22, 2018 at 2:58 AM Becket Qin < > > > > > > > >> becket....@gmail.com > > > > > > > >> > > > > > > > > > >> > > > > wrote: > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > Hi Jun, > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > The usage of correlation ID might still be > > useful > > > to > > > > > > > address > > > > > > > >> > the > > > > > > > >> > > > > cases > > > > > > > >> > > > > > > > that the controller epoch and leader epoch > check > > > are > > > > > not > > > > > > > >> > > sufficient > > > > > > > >> > > > > to > > > > > > > >> > > > > > > > guarantee correct behavior. For example, if > the > > > > > > controller > > > > > > > >> > sends > > > > > > > >> > > a > > > > > > > >> > > > > > > > LeaderAndIsrRequest followed by a > > > > StopReplicaRequest, > > > > > > and > > > > > > > >> the > > > > > > > >> > > > broker > > > > > > > >> > > > > > > > processes it in the reverse order, the replica > > may > > > > > still > > > > > > > be > > > > > > > >> > > wrongly > > > > > > > >> > > > > > > > recreated, right? > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > Thanks, > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > Jiangjie (Becket) Qin > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > On Jul 22, 2018, at 11:47 AM, Jun Rao < > > > > > > j...@confluent.io > > > > > > > > > > > > > > > >> > > wrote: > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > Hmm, since we already use controller epoch > and > > > > > leader > > > > > > > >> epoch > > > > > > > >> > for > > > > > > > >> > > > > > > properly > > > > > > > >> > > > > > > > > caching the latest partition state, do we > > really > > > > > need > > > > > > > >> > > correlation > > > > > > > >> > > > > id > > > > > > > >> > > > > > > for > > > > > > > >> > > > > > > > > ordering the controller requests? > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > Thanks, > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > Jun > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > On Fri, Jul 20, 2018 at 2:18 PM, Becket Qin > < > > > > > > > >> > > > becket....@gmail.com> > > > > > > > >> > > > > > > > wrote: > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > >> Lucas and Mayuresh, > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> Good idea. The correlation id should work. > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> In the ControllerChannelManager, a request > > will > > > > be > > > > > > > resent > > > > > > > >> > > until > > > > > > > >> > > > a > > > > > > > >> > > > > > > > response > > > > > > > >> > > > > > > > >> is received. So if the controller to broker > > > > > > connection > > > > > > > >> > > > disconnects > > > > > > > >> > > > > > > after > > > > > > > >> > > > > > > > >> controller sends R1_a, but before the > > response > > > of > > > > > > R1_a > > > > > > > is > > > > > > > >> > > > > received, > > > > > > > >> > > > > > a > > > > > > > >> > > > > > > > >> disconnection may cause the controller to > > > resend > > > > > > R1_b. > > > > > > > >> i.e. > > > > > > > >> > > > until > > > > > > > >> > > > > R1 > > > > > > > >> > > > > > > is > > > > > > > >> > > > > > > > >> acked, R2 won't be sent by the controller. > > > > > > > >> > > > > > > > >> This gives two guarantees: > > > > > > > >> > > > > > > > >> 1. Correlation id wise: R1_a < R1_b < R2. > > > > > > > >> > > > > > > > >> 2. On the broker side, when R2 is seen, R1 > > must > > > > > have > > > > > > > been > > > > > > > >> > > > > processed > > > > > > > >> > > > > > at > > > > > > > >> > > > > > > > >> least once. > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> So on the broker side, with a single thread > > > > > > controller > > > > > > > >> > request > > > > > > > >> > > > > > > handler, > > > > > > > >> > > > > > > > the > > > > > > > >> > > > > > > > >> logic should be: > > > > > > > >> > > > > > > > >> 1. Process what ever request seen in the > > > > controller > > > > > > > >> request > > > > > > > >> > > > queue > > > > > > > >> > > > > > > > >> 2. For the given epoch, drop request if its > > > > > > correlation > > > > > > > >> id > > > > > > > >> > is > > > > > > > >> > > > > > smaller > > > > > > > >> > > > > > > > than > > > > > > > >> > > > > > > > >> that of the last processed request. > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> Thanks, > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> Jiangjie (Becket) Qin > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> On Fri, Jul 20, 2018 at 8:07 AM, Jun Rao < > > > > > > > >> j...@confluent.io> > > > > > > > >> > > > > wrote: > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >>> I agree that there is no strong ordering > > when > > > > > there > > > > > > > are > > > > > > > >> > more > > > > > > > >> > > > than > > > > > > > >> > > > > > one > > > > > > > >> > > > > > > > >>> socket connections. Currently, we rely on > > > > > > > >> controllerEpoch > > > > > > > >> > and > > > > > > > >> > > > > > > > leaderEpoch > > > > > > > >> > > > > > > > >>> to ensure that the receiving broker picks > up > > > the > > > > > > > latest > > > > > > > >> > state > > > > > > > >> > > > for > > > > > > > >> > > > > > > each > > > > > > > >> > > > > > > > >>> partition. > > > > > > > >> > > > > > > > >>> > > > > > > > >> > > > > > > > >>> One potential issue with the dequeue > > approach > > > is > > > > > > that > > > > > > > if > > > > > > > >> > the > > > > > > > >> > > > > queue > > > > > > > >> > > > > > is > > > > > > > >> > > > > > > > >> full, > > > > > > > >> > > > > > > > >>> there is no guarantee that the controller > > > > requests > > > > > > > will > > > > > > > >> be > > > > > > > >> > > > > enqueued > > > > > > > >> > > > > > > > >>> quickly. > > > > > > > >> > > > > > > > >>> > > > > > > > >> > > > > > > > >>> Thanks, > > > > > > > >> > > > > > > > >>> > > > > > > > >> > > > > > > > >>> Jun > > > > > > > >> > > > > > > > >>> > > > > > > > >> > > > > > > > >>> On Fri, Jul 20, 2018 at 5:25 AM, Mayuresh > > > > Gharat < > > > > > > > >> > > > > > > > >>> gharatmayures...@gmail.com > > > > > > > >> > > > > > > > >>>> wrote: > > > > > > > >> > > > > > > > >>> > > > > > > > >> > > > > > > > >>>> Yea, the correlationId is only set to 0 > in > > > the > > > > > > > >> > NetworkClient > > > > > > > >> > > > > > > > >> constructor. > > > > > > > >> > > > > > > > >>>> Since we reuse the same NetworkClient > > between > > > > > > > >> Controller > > > > > > > >> > and > > > > > > > >> > > > the > > > > > > > >> > > > > > > > >> broker, > > > > > > > >> > > > > > > > >>> a > > > > > > > >> > > > > > > > >>>> disconnection should not cause it to > reset > > to > > > > 0, > > > > > in > > > > > > > >> which > > > > > > > >> > > case > > > > > > > >> > > > > it > > > > > > > >> > > > > > > can > > > > > > > >> > > > > > > > >> be > > > > > > > >> > > > > > > > >>>> used to reject obsolete requests. > > > > > > > >> > > > > > > > >>>> > > > > > > > >> > > > > > > > >>>> Thanks, > > > > > > > >> > > > > > > > >>>> > > > > > > > >> > > > > > > > >>>> Mayuresh > > > > > > > >> > > > > > > > >>>> > > > > > > > >> > > > > > > > >>>> On Thu, Jul 19, 2018 at 1:52 PM Lucas > Wang > > < > > > > > > > >> > > > > lucasatu...@gmail.com > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > >>> wrote: > > > > > > > >> > > > > > > > >>>> > > > > > > > >> > > > > > > > >>>>> @Dong, > > > > > > > >> > > > > > > > >>>>> Great example and explanation, thanks! > > > > > > > >> > > > > > > > >>>>> > > > > > > > >> > > > > > > > >>>>> @All > > > > > > > >> > > > > > > > >>>>> Regarding the example given by Dong, it > > > seems > > > > > even > > > > > > > if > > > > > > > >> we > > > > > > > >> > > use > > > > > > > >> > > > a > > > > > > > >> > > > > > > queue, > > > > > > > >> > > > > > > > >>>> and a > > > > > > > >> > > > > > > > >>>>> dedicated controller request handling > > > thread, > > > > > > > >> > > > > > > > >>>>> the same result can still happen because > > > R1_a > > > > > will > > > > > > > be > > > > > > > >> > sent > > > > > > > >> > > on > > > > > > > >> > > > > one > > > > > > > >> > > > > > > > >>>>> connection, and R1_b & R2 will be sent > on > > a > > > > > > > different > > > > > > > >> > > > > connection, > > > > > > > >> > > > > > > > >>>>> and there is no ordering between > different > > > > > > > >> connections on > > > > > > > >> > > the > > > > > > > >> > > > > > > broker > > > > > > > >> > > > > > > > >>>> side. > > > > > > > >> > > > > > > > >>>>> I was discussing with Mayuresh offline, > > and > > > it > > > > > > seems > > > > > > > >> > > > > correlation > > > > > > > >> > > > > > id > > > > > > > >> > > > > > > > >>>> within > > > > > > > >> > > > > > > > >>>>> the same NetworkClient object is > > > monotonically > > > > > > > >> increasing > > > > > > > >> > > and > > > > > > > >> > > > > > never > > > > > > > >> > > > > > > > >>>> reset, > > > > > > > >> > > > > > > > >>>>> hence a broker can leverage that to > > properly > > > > > > reject > > > > > > > >> > > obsolete > > > > > > > >> > > > > > > > >> requests. > > > > > > > >> > > > > > > > >>>>> Thoughts? > > > > > > > >> > > > > > > > >>>>> > > > > > > > >> > > > > > > > >>>>> Thanks, > > > > > > > >> > > > > > > > >>>>> Lucas > > > > > > > >> > > > > > > > >>>>> > > > > > > > >> > > > > > > > >>>>> On Thu, Jul 19, 2018 at 12:11 PM, > Mayuresh > > > > > Gharat > > > > > > < > > > > > > > >> > > > > > > > >>>>> gharatmayures...@gmail.com> wrote: > > > > > > > >> > > > > > > > >>>>> > > > > > > > >> > > > > > > > >>>>>> Actually nvm, correlationId is reset in > > > case > > > > of > > > > > > > >> > connection > > > > > > > >> > > > > > loss, I > > > > > > > >> > > > > > > > >>>> think. > > > > > > > >> > > > > > > > >>>>>> > > > > > > > >> > > > > > > > >>>>>> Thanks, > > > > > > > >> > > > > > > > >>>>>> > > > > > > > >> > > > > > > > >>>>>> Mayuresh > > > > > > > >> > > > > > > > >>>>>> > > > > > > > >> > > > > > > > >>>>>> On Thu, Jul 19, 2018 at 11:11 AM > Mayuresh > > > > > Gharat > > > > > > < > > > > > > > >> > > > > > > > >>>>>> gharatmayures...@gmail.com> > > > > > > > >> > > > > > > > >>>>>> wrote: > > > > > > > >> > > > > > > > >>>>>> > > > > > > > >> > > > > > > > >>>>>>> I agree with Dong that out-of-order > > > > processing > > > > > > can > > > > > > > >> > happen > > > > > > > >> > > > > with > > > > > > > >> > > > > > > > >>>> having 2 > > > > > > > >> > > > > > > > >>>>>>> separate queues as well and it can > even > > > > happen > > > > > > > >> today. > > > > > > > >> > > > > > > > >>>>>>> Can we use the correlationId in the > > > request > > > > > from > > > > > > > the > > > > > > > >> > > > > controller > > > > > > > >> > > > > > > > >> to > > > > > > > >> > > > > > > > >>>> the > > > > > > > >> > > > > > > > >>>>>>> broker to handle ordering ? > > > > > > > >> > > > > > > > >>>>>>> > > > > > > > >> > > > > > > > >>>>>>> Thanks, > > > > > > > >> > > > > > > > >>>>>>> > > > > > > > >> > > > > > > > >>>>>>> Mayuresh > > > > > > > >> > > > > > > > >>>>>>> > > > > > > > >> > > > > > > > >>>>>>> > > > > > > > >> > > > > > > > >>>>>>> On Thu, Jul 19, 2018 at 6:41 AM Becket > > > Qin < > > > > > > > >> > > > > > becket....@gmail.com > > > > > > > >> > > > > > > > >>> > > > > > > > >> > > > > > > > >>>>> wrote: > > > > > > > >> > > > > > > > >>>>>>> > > > > > > > >> > > > > > > > >>>>>>>> Good point, Joel. I agree that a > > > dedicated > > > > > > > >> controller > > > > > > > >> > > > > request > > > > > > > >> > > > > > > > >>>> handling > > > > > > > >> > > > > > > > >>>>>>>> thread would be a better isolation. > It > > > also > > > > > > > solves > > > > > > > >> the > > > > > > > >> > > > > > > > >> reordering > > > > > > > >> > > > > > > > >>>>> issue. > > > > > > > >> > > > > > > > >>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>> On Thu, Jul 19, 2018 at 2:23 PM, Joel > > > > Koshy < > > > > > > > >> > > > > > > > >> jjkosh...@gmail.com> > > > > > > > >> > > > > > > > >>>>>> wrote: > > > > > > > >> > > > > > > > >>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>> Good example. I think this scenario > > can > > > > > occur > > > > > > in > > > > > > > >> the > > > > > > > >> > > > > current > > > > > > > >> > > > > > > > >>> code > > > > > > > >> > > > > > > > >>>> as > > > > > > > >> > > > > > > > >>>>>>>> well > > > > > > > >> > > > > > > > >>>>>>>>> but with even lower probability > given > > > that > > > > > > there > > > > > > > >> are > > > > > > > >> > > > other > > > > > > > >> > > > > > > > >>>>>>>> non-controller > > > > > > > >> > > > > > > > >>>>>>>>> requests interleaved. It is still > > > sketchy > > > > > > though > > > > > > > >> and > > > > > > > >> > I > > > > > > > >> > > > > think > > > > > > > >> > > > > > a > > > > > > > >> > > > > > > > >>>> safer > > > > > > > >> > > > > > > > >>>>>>>>> approach would be separate queues > and > > > > > pinning > > > > > > > >> > > controller > > > > > > > >> > > > > > > > >> request > > > > > > > >> > > > > > > > >>>>>>>> handling > > > > > > > >> > > > > > > > >>>>>>>>> to one handler thread. > > > > > > > >> > > > > > > > >>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>> On Wed, Jul 18, 2018 at 11:12 PM, > Dong > > > > Lin < > > > > > > > >> > > > > > > > >> lindon...@gmail.com > > > > > > > >> > > > > > > > >>>> > > > > > > > >> > > > > > > > >>>>>> wrote: > > > > > > > >> > > > > > > > >>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>> Hey Becket, > > > > > > > >> > > > > > > > >>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>> I think you are right that there > may > > be > > > > > > > >> out-of-order > > > > > > > >> > > > > > > > >>> processing. > > > > > > > >> > > > > > > > >>>>>>>> However, > > > > > > > >> > > > > > > > >>>>>>>>>> it seems that out-of-order > processing > > > may > > > > > > also > > > > > > > >> > happen > > > > > > > >> > > > even > > > > > > > >> > > > > > > > >> if > > > > > > > >> > > > > > > > >>> we > > > > > > > >> > > > > > > > >>>>>> use a > > > > > > > >> > > > > > > > >>>>>>>>>> separate queue. > > > > > > > >> > > > > > > > >>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>> Here is the example: > > > > > > > >> > > > > > > > >>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>> - Controller sends R1 and got > > > > disconnected > > > > > > > before > > > > > > > >> > > > > receiving > > > > > > > >> > > > > > > > >>>>>> response. > > > > > > > >> > > > > > > > >>>>>>>>> Then > > > > > > > >> > > > > > > > >>>>>>>>>> it reconnects and sends R2. Both > > > requests > > > > > now > > > > > > > >> stay > > > > > > > >> > in > > > > > > > >> > > > the > > > > > > > >> > > > > > > > >>>>> controller > > > > > > > >> > > > > > > > >>>>>>>>>> request queue in the order they are > > > sent. > > > > > > > >> > > > > > > > >>>>>>>>>> - thread1 takes R1_a from the > request > > > > queue > > > > > > and > > > > > > > >> then > > > > > > > >> > > > > thread2 > > > > > > > >> > > > > > > > >>>> takes > > > > > > > >> > > > > > > > >>>>>> R2 > > > > > > > >> > > > > > > > >>>>>>>>> from > > > > > > > >> > > > > > > > >>>>>>>>>> the request queue almost at the > same > > > > time. > > > > > > > >> > > > > > > > >>>>>>>>>> - So R1_a and R2 are processed in > > > > parallel. > > > > > > > >> There is > > > > > > > >> > > > > chance > > > > > > > >> > > > > > > > >>> that > > > > > > > >> > > > > > > > >>>>>> R2's > > > > > > > >> > > > > > > > >>>>>>>>>> processing is completed before R1. > > > > > > > >> > > > > > > > >>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>> If out-of-order processing can > happen > > > for > > > > > > both > > > > > > > >> > > > approaches > > > > > > > >> > > > > > > > >> with > > > > > > > >> > > > > > > > >>>>> very > > > > > > > >> > > > > > > > >>>>>>>> low > > > > > > > >> > > > > > > > >>>>>>>>>> probability, it may not be > worthwhile > > > to > > > > > add > > > > > > > the > > > > > > > >> > extra > > > > > > > >> > > > > > > > >> queue. > > > > > > > >> > > > > > > > >>>> What > > > > > > > >> > > > > > > > >>>>>> do > > > > > > > >> > > > > > > > >>>>>>>> you > > > > > > > >> > > > > > > > >>>>>>>>>> think? > > > > > > > >> > > > > > > > >>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>> Thanks, > > > > > > > >> > > > > > > > >>>>>>>>>> Dong > > > > > > > >> > > > > > > > >>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>> On Wed, Jul 18, 2018 at 6:17 PM, > > Becket > > > > > Qin < > > > > > > > >> > > > > > > > >>>> becket....@gmail.com > > > > > > > >> > > > > > > > >>>>>> > > > > > > > >> > > > > > > > >>>>>>>>> wrote: > > > > > > > >> > > > > > > > >>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>> Hi Mayuresh/Joel, > > > > > > > >> > > > > > > > >>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>> Using the request channel as a > > dequeue > > > > was > > > > > > > >> bright > > > > > > > >> > up > > > > > > > >> > > > some > > > > > > > >> > > > > > > > >>> time > > > > > > > >> > > > > > > > >>>>> ago > > > > > > > >> > > > > > > > >>>>>>>> when > > > > > > > >> > > > > > > > >>>>>>>>>> we > > > > > > > >> > > > > > > > >>>>>>>>>>> initially thinking of prioritizing > > the > > > > > > > request. > > > > > > > >> The > > > > > > > >> > > > > > > > >> concern > > > > > > > >> > > > > > > > >>>> was > > > > > > > >> > > > > > > > >>>>>> that > > > > > > > >> > > > > > > > >>>>>>>>> the > > > > > > > >> > > > > > > > >>>>>>>>>>> controller requests are supposed > to > > be > > > > > > > >> processed in > > > > > > > >> > > > > order. > > > > > > > >> > > > > > > > >>> If > > > > > > > >> > > > > > > > >>>> we > > > > > > > >> > > > > > > > >>>>>> can > > > > > > > >> > > > > > > > >>>>>>>>>> ensure > > > > > > > >> > > > > > > > >>>>>>>>>>> that there is one controller > request > > > in > > > > > the > > > > > > > >> request > > > > > > > >> > > > > > > > >> channel, > > > > > > > >> > > > > > > > >>>> the > > > > > > > >> > > > > > > > >>>>>>>> order > > > > > > > >> > > > > > > > >>>>>>>>> is > > > > > > > >> > > > > > > > >>>>>>>>>>> not a concern. But in cases that > > there > > > > are > > > > > > > more > > > > > > > >> > than > > > > > > > >> > > > one > > > > > > > >> > > > > > > > >>>>>> controller > > > > > > > >> > > > > > > > >>>>>>>>>> request > > > > > > > >> > > > > > > > >>>>>>>>>>> inserted into the queue, the > > > controller > > > > > > > request > > > > > > > >> > order > > > > > > > >> > > > may > > > > > > > >> > > > > > > > >>>> change > > > > > > > >> > > > > > > > >>>>>> and > > > > > > > >> > > > > > > > >>>>>>>>>> cause > > > > > > > >> > > > > > > > >>>>>>>>>>> problem. For example, think about > > the > > > > > > > following > > > > > > > >> > > > sequence: > > > > > > > >> > > > > > > > >>>>>>>>>>> 1. Controller successfully sent a > > > > request > > > > > R1 > > > > > > > to > > > > > > > >> > > broker > > > > > > > >> > > > > > > > >>>>>>>>>>> 2. Broker receives R1 and put the > > > > request > > > > > to > > > > > > > the > > > > > > > >> > head > > > > > > > >> > > > of > > > > > > > >> > > > > > > > >> the > > > > > > > >> > > > > > > > >>>>>> request > > > > > > > >> > > > > > > > >>>>>>>>>> queue. > > > > > > > >> > > > > > > > >>>>>>>>>>> 3. Controller to broker connection > > > > failed > > > > > > and > > > > > > > >> the > > > > > > > >> > > > > > > > >> controller > > > > > > > >> > > > > > > > >>>>>>>>> reconnected > > > > > > > >> > > > > > > > >>>>>>>>>> to > > > > > > > >> > > > > > > > >>>>>>>>>>> the broker. > > > > > > > >> > > > > > > > >>>>>>>>>>> 4. Controller sends a request R2 > to > > > the > > > > > > broker > > > > > > > >> > > > > > > > >>>>>>>>>>> 5. Broker receives R2 and add it > to > > > the > > > > > head > > > > > > > of > > > > > > > >> the > > > > > > > >> > > > > > > > >> request > > > > > > > >> > > > > > > > >>>>> queue. > > > > > > > >> > > > > > > > >>>>>>>>>>> Now on the broker side, R2 will be > > > > > processed > > > > > > > >> before > > > > > > > >> > > R1 > > > > > > > >> > > > is > > > > > > > >> > > > > > > > >>>>>> processed, > > > > > > > >> > > > > > > > >>>>>>>>>> which > > > > > > > >> > > > > > > > >>>>>>>>>>> may cause problem. > > > > > > > >> > > > > > > > >>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>> Thanks, > > > > > > > >> > > > > > > > >>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>> Jiangjie (Becket) Qin > > > > > > > >> > > > > > > > >>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>> On Thu, Jul 19, 2018 at 3:23 AM, > > Joel > > > > > Koshy > > > > > > < > > > > > > > >> > > > > > > > >>>>> jjkosh...@gmail.com> > > > > > > > >> > > > > > > > >>>>>>>>> wrote: > > > > > > > >> > > > > > > > >>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>> @Mayuresh - I like your idea. It > > > > appears > > > > > to > > > > > > > be > > > > > > > >> a > > > > > > > >> > > > simpler > > > > > > > >> > > > > > > > >>>> less > > > > > > > >> > > > > > > > >>>>>>>>> invasive > > > > > > > >> > > > > > > > >>>>>>>>>>>> alternative and it should work. > > > > > > > >> Jun/Becket/others, > > > > > > > >> > > do > > > > > > > >> > > > > > > > >> you > > > > > > > >> > > > > > > > >>>> see > > > > > > > >> > > > > > > > >>>>>> any > > > > > > > >> > > > > > > > >>>>>>>>>>> pitfalls > > > > > > > >> > > > > > > > >>>>>>>>>>>> with this approach? > > > > > > > >> > > > > > > > >>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>> On Wed, Jul 18, 2018 at 12:03 PM, > > > Lucas > > > > > > Wang > > > > > > > < > > > > > > > >> > > > > > > > >>>>>>>> lucasatu...@gmail.com> > > > > > > > >> > > > > > > > >>>>>>>>>>>> wrote: > > > > > > > >> > > > > > > > >>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>> @Mayuresh, > > > > > > > >> > > > > > > > >>>>>>>>>>>>> That's a very interesting idea > > that > > > I > > > > > > > haven't > > > > > > > >> > > thought > > > > > > > >> > > > > > > > >>>>> before. > > > > > > > >> > > > > > > > >>>>>>>>>>>>> It seems to solve our problem at > > > hand > > > > > > pretty > > > > > > > >> > well, > > > > > > > >> > > > and > > > > > > > >> > > > > > > > >>>> also > > > > > > > >> > > > > > > > >>>>>>>>>>>>> avoids the need to have a new > size > > > > > metric > > > > > > > and > > > > > > > >> > > > capacity > > > > > > > >> > > > > > > > >>>>> config > > > > > > > >> > > > > > > > >>>>>>>>>>>>> for the controller request > queue. > > In > > > > > fact, > > > > > > > if > > > > > > > >> we > > > > > > > >> > > were > > > > > > > >> > > > > > > > >> to > > > > > > > >> > > > > > > > >>>>> adopt > > > > > > > >> > > > > > > > >>>>>>>>>>>>> this design, there is no public > > > > > interface > > > > > > > >> change, > > > > > > > >> > > and > > > > > > > >> > > > > > > > >> we > > > > > > > >> > > > > > > > >>>>>>>>>>>>> probably don't need a KIP. > > > > > > > >> > > > > > > > >>>>>>>>>>>>> Also implementation wise, it > seems > > > > > > > >> > > > > > > > >>>>>>>>>>>>> the java class > LinkedBlockingQueue > > > can > > > > > > > readily > > > > > > > >> > > > satisfy > > > > > > > >> > > > > > > > >>> the > > > > > > > >> > > > > > > > >>>>>>>>>> requirement > > > > > > > >> > > > > > > > >>>>>>>>>>>>> by supporting a capacity, and > also > > > > > > allowing > > > > > > > >> > > inserting > > > > > > > >> > > > > > > > >> at > > > > > > > >> > > > > > > > >>>>> both > > > > > > > >> > > > > > > > >>>>>>>> ends. > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>> My only concern is that this > > design > > > is > > > > > > tied > > > > > > > to > > > > > > > >> > the > > > > > > > >> > > > > > > > >>>>> coincidence > > > > > > > >> > > > > > > > >>>>>>>> that > > > > > > > >> > > > > > > > >>>>>>>>>>>>> we have two request priorities > and > > > > there > > > > > > are > > > > > > > >> two > > > > > > > >> > > ends > > > > > > > >> > > > > > > > >>> to a > > > > > > > >> > > > > > > > >>>>>>>> deque. > > > > > > > >> > > > > > > > >>>>>>>>>>>>> Hence by using the proposed > > design, > > > it > > > > > > seems > > > > > > > >> the > > > > > > > >> > > > > > > > >> network > > > > > > > >> > > > > > > > >>>>> layer > > > > > > > >> > > > > > > > >>>>>>>> is > > > > > > > >> > > > > > > > >>>>>>>>>>>>> more tightly coupled with upper > > > layer > > > > > > logic, > > > > > > > >> e.g. > > > > > > > >> > > if > > > > > > > >> > > > > > > > >> we > > > > > > > >> > > > > > > > >>>> were > > > > > > > >> > > > > > > > >>>>>> to > > > > > > > >> > > > > > > > >>>>>>>> add > > > > > > > >> > > > > > > > >>>>>>>>>>>>> an extra priority level in the > > > future > > > > > for > > > > > > > some > > > > > > > >> > > > reason, > > > > > > > >> > > > > > > > >>> we > > > > > > > >> > > > > > > > >>>>>> would > > > > > > > >> > > > > > > > >>>>>>>>>>> probably > > > > > > > >> > > > > > > > >>>>>>>>>>>>> need to go back to the design of > > > > > separate > > > > > > > >> queues, > > > > > > > >> > > one > > > > > > > >> > > > > > > > >>> for > > > > > > > >> > > > > > > > >>>>> each > > > > > > > >> > > > > > > > >>>>>>>>>> priority > > > > > > > >> > > > > > > > >>>>>>>>>>>>> level. > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>> In summary, I'm ok with both > > designs > > > > and > > > > > > > lean > > > > > > > >> > > toward > > > > > > > >> > > > > > > > >>> your > > > > > > > >> > > > > > > > >>>>>>>> suggested > > > > > > > >> > > > > > > > >>>>>>>>>>>>> approach. > > > > > > > >> > > > > > > > >>>>>>>>>>>>> Let's hear what others think. > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>> @Becket, > > > > > > > >> > > > > > > > >>>>>>>>>>>>> In light of Mayuresh's suggested > > new > > > > > > design, > > > > > > > >> I'm > > > > > > > >> > > > > > > > >>> answering > > > > > > > >> > > > > > > > >>>>>> your > > > > > > > >> > > > > > > > >>>>>>>>>>> question > > > > > > > >> > > > > > > > >>>>>>>>>>>>> only in the context > > > > > > > >> > > > > > > > >>>>>>>>>>>>> of the current KIP design: I > think > > > > your > > > > > > > >> > suggestion > > > > > > > >> > > > > > > > >> makes > > > > > > > >> > > > > > > > >>>>>> sense, > > > > > > > >> > > > > > > > >>>>>>>> and > > > > > > > >> > > > > > > > >>>>>>>>>> I'm > > > > > > > >> > > > > > > > >>>>>>>>>>>> ok > > > > > > > >> > > > > > > > >>>>>>>>>>>>> with removing the capacity > config > > > and > > > > > > > >> > > > > > > > >>>>>>>>>>>>> just relying on the default > value > > of > > > > 20 > > > > > > > being > > > > > > > >> > > > > > > > >> sufficient > > > > > > > >> > > > > > > > >>>>>> enough. > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>> Thanks, > > > > > > > >> > > > > > > > >>>>>>>>>>>>> Lucas > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>> On Wed, Jul 18, 2018 at 9:57 AM, > > > > > Mayuresh > > > > > > > >> Gharat > > > > > > > >> > < > > > > > > > >> > > > > > > > >>>>>>>>>>>>> gharatmayures...@gmail.com > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> wrote: > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> Hi Lucas, > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> Seems like the main intent here > > is > > > to > > > > > > > >> prioritize > > > > > > > >> > > the > > > > > > > >> > > > > > > > >>>>>>>> controller > > > > > > > >> > > > > > > > >>>>>>>>>>> request > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> over any other requests. > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> In that case, we can change the > > > > request > > > > > > > queue > > > > > > > >> > to a > > > > > > > >> > > > > > > > >>>>> dequeue, > > > > > > > >> > > > > > > > >>>>>>>> where > > > > > > > >> > > > > > > > >>>>>>>>>> you > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> always insert the normal > requests > > > > > > (produce, > > > > > > > >> > > > > > > > >>>> consume,..etc) > > > > > > > >> > > > > > > > >>>>>> to > > > > > > > >> > > > > > > > >>>>>>>> the > > > > > > > >> > > > > > > > >>>>>>>>>> end > > > > > > > >> > > > > > > > >>>>>>>>>>>> of > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> the dequeue, but if its a > > > controller > > > > > > > request, > > > > > > > >> > you > > > > > > > >> > > > > > > > >>> insert > > > > > > > >> > > > > > > > >>>>> it > > > > > > > >> > > > > > > > >>>>>> to > > > > > > > >> > > > > > > > >>>>>>>>> the > > > > > > > >> > > > > > > > >>>>>>>>>>> head > > > > > > > >> > > > > > > > >>>>>>>>>>>>> of > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> the queue. This ensures that > the > > > > > > controller > > > > > > > >> > > request > > > > > > > >> > > > > > > > >>> will > > > > > > > >> > > > > > > > >>>>> be > > > > > > > >> > > > > > > > >>>>>>>> given > > > > > > > >> > > > > > > > >>>>>>>>>>>> higher > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> priority over other requests. > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> Also since we only read one > > request > > > > > from > > > > > > > the > > > > > > > >> > > socket > > > > > > > >> > > > > > > > >>> and > > > > > > > >> > > > > > > > >>>>> mute > > > > > > > >> > > > > > > > >>>>>>>> it > > > > > > > >> > > > > > > > >>>>>>>>> and > > > > > > > >> > > > > > > > >>>>>>>>>>>> only > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> unmute it after handling the > > > request, > > > > > > this > > > > > > > >> would > > > > > > > >> > > > > > > > >>> ensure > > > > > > > >> > > > > > > > >>>>> that > > > > > > > >> > > > > > > > >>>>>>>> we > > > > > > > >> > > > > > > > >>>>>>>>>> don't > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> handle controller requests out > of > > > > > order. > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> With this approach we can avoid > > the > > > > > > second > > > > > > > >> queue > > > > > > > >> > > and > > > > > > > >> > > > > > > > >>> the > > > > > > > >> > > > > > > > >>>>>>>>> additional > > > > > > > >> > > > > > > > >>>>>>>>>>>>> config > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> for the size of the queue. > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> What do you think ? > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> Thanks, > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> Mayuresh > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> On Wed, Jul 18, 2018 at 3:05 AM > > > > Becket > > > > > > Qin > > > > > > > < > > > > > > > >> > > > > > > > >>>>>>>> becket....@gmail.com > > > > > > > >> > > > > > > > >>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>> wrote: > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> Hey Joel, > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> Thank for the detail > > explanation. > > > I > > > > > > agree > > > > > > > >> the > > > > > > > >> > > > > > > > >>> current > > > > > > > >> > > > > > > > >>>>>> design > > > > > > > >> > > > > > > > >>>>>>>>>> makes > > > > > > > >> > > > > > > > >>>>>>>>>>>>> sense. > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> My confusion is about whether > > the > > > > new > > > > > > > config > > > > > > > >> > for > > > > > > > >> > > > > > > > >> the > > > > > > > >> > > > > > > > >>>>>>>> controller > > > > > > > >> > > > > > > > >>>>>>>>>>> queue > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> capacity is necessary. I > cannot > > > > think > > > > > > of a > > > > > > > >> case > > > > > > > >> > > in > > > > > > > >> > > > > > > > >>>> which > > > > > > > >> > > > > > > > >>>>>>>> users > > > > > > > >> > > > > > > > >>>>>>>>>>> would > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> change > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> it. > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> Thanks, > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> Jiangjie (Becket) Qin > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> On Wed, Jul 18, 2018 at 6:00 > PM, > > > > > Becket > > > > > > > Qin > > > > > > > >> < > > > > > > > >> > > > > > > > >>>>>>>>>> becket....@gmail.com> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> wrote: > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> Hi Lucas, > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> I guess my question can be > > > > rephrased > > > > > to > > > > > > > >> "do we > > > > > > > >> > > > > > > > >>>> expect > > > > > > > >> > > > > > > > >>>>>>>> user to > > > > > > > >> > > > > > > > >>>>>>>>>>> ever > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> change > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> the controller request queue > > > > > capacity"? > > > > > > > If > > > > > > > >> we > > > > > > > >> > > > > > > > >>> agree > > > > > > > >> > > > > > > > >>>>> that > > > > > > > >> > > > > > > > >>>>>>>> 20 > > > > > > > >> > > > > > > > >>>>>>>>> is > > > > > > > >> > > > > > > > >>>>>>>>>>>>> already > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> a > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> very generous default number > > and > > > we > > > > > do > > > > > > > not > > > > > > > >> > > > > > > > >> expect > > > > > > > >> > > > > > > > >>>> user > > > > > > > >> > > > > > > > >>>>>> to > > > > > > > >> > > > > > > > >>>>>>>>>> change > > > > > > > >> > > > > > > > >>>>>>>>>>>> it, > > > > > > > >> > > > > > > > >>>>>>>>>>>>> is > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> it > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> still necessary to expose > this > > > as a > > > > > > > config? > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> Thanks, > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> Jiangjie (Becket) Qin > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> On Wed, Jul 18, 2018 at 2:29 > > AM, > > > > > Lucas > > > > > > > >> Wang < > > > > > > > >> > > > > > > > >>>>>>>>>>> lucasatu...@gmail.com > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> wrote: > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> @Becket > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> 1. Thanks for the comment. > You > > > are > > > > > > right > > > > > > > >> that > > > > > > > >> > > > > > > > >>>>> normally > > > > > > > >> > > > > > > > >>>>>>>> there > > > > > > > >> > > > > > > > >>>>>>>>>>>> should > > > > > > > >> > > > > > > > >>>>>>>>>>>>> be > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> just > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> one controller request > because > > > of > > > > > > > muting, > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> and I had NOT intended to > say > > > > there > > > > > > > would > > > > > > > >> be > > > > > > > >> > > > > > > > >> many > > > > > > > >> > > > > > > > >>>>>>>> enqueued > > > > > > > >> > > > > > > > >>>>>>>>>>>>> controller > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> requests. > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> I went through the KIP > again, > > > and > > > > > I'm > > > > > > > not > > > > > > > >> > sure > > > > > > > >> > > > > > > > >>>> which > > > > > > > >> > > > > > > > >>>>>> part > > > > > > > >> > > > > > > > >>>>>>>>>>> conveys > > > > > > > >> > > > > > > > >>>>>>>>>>>>> that > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> info. > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> I'd be happy to revise if > you > > > > point > > > > > it > > > > > > > out > > > > > > > >> > the > > > > > > > >> > > > > > > > >>>>> section. > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> 2. Though it should not > happen > > > in > > > > > > normal > > > > > > > >> > > > > > > > >>>> conditions, > > > > > > > >> > > > > > > > >>>>>> the > > > > > > > >> > > > > > > > >>>>>>>>>> current > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> design > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> does not preclude multiple > > > > > controllers > > > > > > > >> > running > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> at the same time, hence if > we > > > > don't > > > > > > have > > > > > > > >> the > > > > > > > >> > > > > > > > >>>>> controller > > > > > > > >> > > > > > > > >>>>>>>>> queue > > > > > > > >> > > > > > > > >>>>>>>>>>>>> capacity > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> config and simply make its > > > > capacity > > > > > to > > > > > > > be > > > > > > > >> 1, > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> network threads handling > > > requests > > > > > from > > > > > > > >> > > > > > > > >> different > > > > > > > >> > > > > > > > >>>>>>>> controllers > > > > > > > >> > > > > > > > >>>>>>>>>>> will > > > > > > > >> > > > > > > > >>>>>>>>>>>> be > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> blocked during those > > troublesome > > > > > > times, > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> which is probably not what > we > > > > want. > > > > > On > > > > > > > the > > > > > > > >> > > > > > > > >> other > > > > > > > >> > > > > > > > >>>>> hand, > > > > > > > >> > > > > > > > >>>>>>>>> adding > > > > > > > >> > > > > > > > >>>>>>>>>>> the > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> extra > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> config with a default value, > > say > > > > 20, > > > > > > > >> guards > > > > > > > >> > us > > > > > > > >> > > > > > > > >>> from > > > > > > > >> > > > > > > > >>>>>>>> issues > > > > > > > >> > > > > > > > >>>>>>>>> in > > > > > > > >> > > > > > > > >>>>>>>>>>>> those > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> troublesome times, and IMO > > there > > > > > isn't > > > > > > > >> much > > > > > > > >> > > > > > > > >>>> downside > > > > > > > >> > > > > > > > >>>>> of > > > > > > > >> > > > > > > > >>>>>>>>> adding > > > > > > > >> > > > > > > > >>>>>>>>>>> the > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> extra > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> config. > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> @Mayuresh > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> Good catch, this sentence is > > an > > > > > > obsolete > > > > > > > >> > > > > > > > >>> statement > > > > > > > >> > > > > > > > >>>>>> based > > > > > > > >> > > > > > > > >>>>>>>> on > > > > > > > >> > > > > > > > >>>>>>>>> a > > > > > > > >> > > > > > > > >>>>>>>>>>>>> previous > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> design. I've revised the > > wording > > > > in > > > > > > the > > > > > > > >> KIP. > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> Thanks, > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> Lucas > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> On Tue, Jul 17, 2018 at > 10:33 > > > AM, > > > > > > > Mayuresh > > > > > > > >> > > > > > > > >>> Gharat < > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> gharatmayures...@gmail.com> > > > > wrote: > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> Hi Lucas, > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> Thanks for the KIP. > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> I am trying to understand > why > > > you > > > > > > think > > > > > > > >> "The > > > > > > > >> > > > > > > > >>>> memory > > > > > > > >> > > > > > > > >>>>>>>>>>> consumption > > > > > > > >> > > > > > > > >>>>>>>>>>>>> can > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> rise > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> given the total number of > > > queued > > > > > > > requests > > > > > > > >> > can > > > > > > > >> > > > > > > > >>> go > > > > > > > >> > > > > > > > >>>> up > > > > > > > >> > > > > > > > >>>>>> to > > > > > > > >> > > > > > > > >>>>>>>> 2x" > > > > > > > >> > > > > > > > >>>>>>>>>> in > > > > > > > >> > > > > > > > >>>>>>>>>>>> the > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> impact > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> section. Normally the > > requests > > > > from > > > > > > > >> > > > > > > > >> controller > > > > > > > >> > > > > > > > >>>> to a > > > > > > > >> > > > > > > > >>>>>>>> Broker > > > > > > > >> > > > > > > > >>>>>>>>>> are > > > > > > > >> > > > > > > > >>>>>>>>>>>> not > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> high > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> volume, right ? > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> Thanks, > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> Mayuresh > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> On Tue, Jul 17, 2018 at > 5:06 > > AM > > > > > > Becket > > > > > > > >> Qin < > > > > > > > >> > > > > > > > >>>>>>>>>>>> becket....@gmail.com> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> wrote: > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>>> Thanks for the KIP, Lucas. > > > > > > Separating > > > > > > > >> the > > > > > > > >> > > > > > > > >>>> control > > > > > > > >> > > > > > > > >>>>>>>> plane > > > > > > > >> > > > > > > > >>>>>>>>>> from > > > > > > > >> > > > > > > > >>>>>>>>>>>> the > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> data > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> plane > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>>> makes a lot of sense. > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>>> In the KIP you mentioned > > that > > > > the > > > > > > > >> > > > > > > > >> controller > > > > > > > >> > > > > > > > >>>>>> request > > > > > > > >> > > > > > > > >>>>>>>>> queue > > > > > > > >> > > > > > > > >>>>>>>>>>> may > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> have > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> many > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>>> requests in it. Will this > > be a > > > > > > common > > > > > > > >> case? > > > > > > > >> > > > > > > > >>> The > > > > > > > >> > > > > > > > >>>>>>>>> controller > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> requests > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> still > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>>> goes through the > > SocketServer. > > > > The > > > > > > > >> > > > > > > > >>> SocketServer > > > > > > > >> > > > > > > > >>>>>> will > > > > > > > >> > > > > > > > >>>>>>>>> mute > > > > > > > >> > > > > > > > >>>>>>>>>>> the > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> channel > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> once > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>>> a request is read and put > > into > > > > the > > > > > > > >> request > > > > > > > >> > > > > > > > >>>>> channel. > > > > > > > >> > > > > > > > >>>>>>>> So > > > > > > > >> > > > > > > > >>>>>>>>>>>> assuming > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> there > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> is > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>>> only one connection > between > > > > > > controller > > > > > > > >> and > > > > > > > >> > > > > > > > >>> each > > > > > > > >> > > > > > > > >>>>>>>> broker, > > > > > > > >> > > > > > > > >>>>>>>>> on > > > > > > > >> > > > > > > > >>>>>>>>>>> the > > > > > > > >> > > > > > > > > > > >
