Thanks everyone for the discussion, very helpful. I've updated the KIP to make the delay such that it is extended as new members join the group and that it never exceeds the groups rebalance timeout.
If everyone is ok with this I'll kick off the voting thread. Thanks again, Damian On Tue, 28 Mar 2017 at 23:18 Becket Qin <becket....@gmail.com> wrote: > I think separating leave/join makes sense. The scenario I can think of for > delaying a rebalance on LeaveGroupRequest is rolling bounce of a service. > But that scenario could be tricky because there may be mixture of joining > and leaving. What happens if a consumer left the group right after another > consumer joins the group? Which delay should be applied? > > Jason, if I understand correctly, the actual delay of the FIRST rebalance > for each group could be anywhere between group.initial.rebalance.delay.ms > and > the rebalance timeout, depending on how many times the delay is applied. > For example, if the delay is set to 3 seconds and rebalance timeout is set > to 10 seconds. At time T a consumer joins the group, the targeting > rebalance point would be T+3 if no other consumer joins. If another > consumer joins the group at T+2 then the targeting delay point would become > T+5, etc. However, no matter how many times the delay was extended, at T+10 > the rebalance will kick off even if at T+9 a new consumer joined the group. > > I also agree that we should set the default delay to some meaningful value > instead of setting it to 0. > > Thanks, > > Jiangjie (Becket) Qin > > On Tue, Mar 28, 2017 at 12:32 PM, Jason Gustafson <ja...@confluent.io> > wrote: > > > Hey Damian, > > > > Thanks for the KIP. I think the proposal makes sense as a workaround > maybe > > for some advanced users. However, I'm not sure we can depend on average > > users knowing that the config exists, much less setting it to something > > that makes sense. It's kind of a trend in streams that I'm not too > thrilled > > about to try and control these rebalances through careful tuning of > various > > timeouts. For example, the patch to avoid sending LeaveGroup depends on > the > > session timeout being set at least as long as the time for an average > > rolling restart. If the expectation is that these settings are only > needed > > for advanced users, it may be sufficient, but if the problems are > affecting > > average users, it seems less than ideal. That said, if we can get some > real > > benefit from low-hanging fruit like this, then it's probably worthwhile. > > > > This relates to the choice of default value, by the way. If we use 0 as > the > > default, my guess is that most users won't change it and the benefit > could > > be marginal. The choice of 3 seconds that you've documented seems fine to > > me. It matches the default consumer heartbeat interval, which controls > > typical rebalance latency, so there's some consistency there. > > > > Also, one minor comment: I guess the actual delay for each group will be > > the minimum of the group's rebalance timeout and > > group.initial.rebalance.delay.ms. Is that right? > > > > -Jason > > > > On Tue, Mar 28, 2017 at 8:29 AM, Damian Guy <damian....@gmail.com> > wrote: > > > > > @Ismael - yeah sure we can reduce the default, though i'm not sure what > > the > > > "right" default would be. > > > > > > On Tue, 28 Mar 2017 at 15:40 Ismael Juma <ism...@juma.me.uk> wrote: > > > > > > > Is 3 seconds the right default if the timer gets reset after each > > > consumer > > > > joins? Maybe we can lower the default value given the new approach. > > > > > > > > Ismael > > > > > > > > On Tue, Mar 28, 2017 at 9:53 AM, Damian Guy <damian....@gmail.com> > > > wrote: > > > > > > > > > All, > > > > > I'd like to get this back to the original discussion about Delaying > > > > initial > > > > > consumer group rebalance. > > > > > I think i'm leaning towards sticking with the broker config and > > > changing > > > > > the delay so that the timer starts again when a new consumer joins > > the > > > > > group. What are peoples thoughts on that? > > > > > > > > > > Doing something similar on leave is valid, but i'd prefer to > consider > > > it > > > > > separately from this. > > > > > > > > > > Thanks, > > > > > Damian > > > > > > > > > > On Tue, 28 Mar 2017 at 09:48 Damian Guy <damian....@gmail.com> > > wrote: > > > > > > > > > > > Matthias, > > > > > > > > > > > > Yes i know. > > > > > > > > > > > > Thanks, > > > > > > Damian > > > > > > > > > > > > On Mon, 27 Mar 2017 at 18:17 Matthias J. Sax < > > matth...@confluent.io> > > > > > > wrote: > > > > > > > > > > > > Damian, > > > > > > > > > > > > about "rebalance immediately" on timeout -- I guess, that's a > > > different > > > > > > case as no LeaveGroupRequest will be sent. Thus, the broker > should > > be > > > > > > able to distinguish both cases easily, and apply the delay only > if > > it > > > > > > received the LeaveGroupRequest but not if a consumer times out. > > > > > > > > > > > > Does this make sense? > > > > > > > > > > > > -Matthias > > > > > > > > > > > > On 3/27/17 1:56 AM, Damian Guy wrote: > > > > > > > @Becket > > > > > > > > > > > > > > Thanks for the feedback. Yes, i like the idea of extending the > > > delay > > > > as > > > > > > > each new consumer joins the group. Though, i think this could > be > > > done > > > > > > with > > > > > > > either a consumer or broker side config. But i get your point > > that > > > > some > > > > > > > consumers in the group can be misconfigured. > > > > > > > > > > > > > > @Matthias & @Eno - yes we could probably do something similar > if > > > the > > > > > > member > > > > > > > has sent the LeaveGroupRequest. I'm not sure it would be valid > if > > > the > > > > > > > member crashed, hence session.timeout would come into play, > we'd > > > > > probably > > > > > > > want to rebalance immediately. I'd be interested in hearing > > > thoughts > > > > > from > > > > > > > other core kafka folks on this one. > > > > > > > > > > > > > > Thanks, > > > > > > > Damian > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, 24 Mar 2017 at 23:01 Becket Qin <becket....@gmail.com> > > > > wrote: > > > > > > > > > > > > > >> Hi Matthias, > > > > > > >> > > > > > > >> Yes, that was what I was thinking. We will keep delay it until > > > > either > > > > > > >> reaching the rebalance timeout or no new consumer joins in > that > > > > small > > > > > > delay > > > > > > >> which is configured on the broker side. > > > > > > >> > > > > > > >> Thanks, > > > > > > >> > > > > > > >> Jiangjie (Becket) Qin > > > > > > >> > > > > > > >> On Fri, Mar 24, 2017 at 1:39 PM, Matthias J. Sax < > > > > > matth...@confluent.io > > > > > > > > > > > > > >> wrote: > > > > > > >> > > > > > > >>> @Becket: > > > > > > >>> > > > > > > >>> I am not sure, if I understand this correctly. Instead of > > > applying > > > > a > > > > > > >>> fixed delay, that starts when the first consumer of an > (empty) > > > > group > > > > > > >>> joins, you suggest to re-trigger/re-set the delay each time a > > new > > > > > > >>> consumer joins? > > > > > > >>> > > > > > > >>> This sound like a good strategy to me, if the config is on > the > > > > broker > > > > > > >> side. > > > > > > >>> > > > > > > >>> @Eno: > > > > > > >>> > > > > > > >>> I think that's a valid point and I like this idea! > > > > > > >>> > > > > > > >>> > > > > > > >>> -Matthias > > > > > > >>> > > > > > > >>> > > > > > > >>> On 3/24/17 1:23 PM, Eno Thereska wrote: > > > > > > >>>> Thanks Damian, > > > > > > >>>> > > > > > > >>>> This KIP deals with the initial phase only. What about the > > cases > > > > > when > > > > > > >>> several consumers leave a group? Won't there be several > > expensive > > > > > > >>> rebalances then as well? I'm wondering if it makes sense for > > the > > > > > delay > > > > > > to > > > > > > >>> hold anytime the "set" of consumers in a group changes, be it > > > > > addition > > > > > > to > > > > > > >>> the group or removal from group. > > > > > > >>>> > > > > > > >>>> Thanks > > > > > > >>>> Eno > > > > > > >>>> > > > > > > >>>> > > > > > > >>>>> On 24 Mar 2017, at 20:04, Becket Qin <becket....@gmail.com > > > > > > wrote: > > > > > > >>>>> > > > > > > >>>>> Thanks for the KIP, Damian. > > > > > > >>>>> > > > > > > >>>>> My two cents on this. It seems there are two things worth > > > > thinking > > > > > > >> here: > > > > > > >>>>> > > > > > > >>>>> 1. Better rebalance timing. We will try to rebalance only > > when > > > > all > > > > > > the > > > > > > >>>>> consumers in a group have joined. The challenge would be > > > someone > > > > > has > > > > > > >> to > > > > > > >>>>> define what does ALL consumers mean, it could either be a > > time > > > or > > > > > > >>> number of > > > > > > >>>>> consumers, etc. > > > > > > >>>>> > > > > > > >>>>> 2. Avoid frequent rebalance. For example, if there are 100 > > > > > consumers > > > > > > >> in > > > > > > >>> a > > > > > > >>>>> group, today, in the worst case, we may end up with 100 > > > > rebalances > > > > > > >> even > > > > > > >>> if > > > > > > >>>>> all the consumers joined the group in a reasonably small > > amount > > > > of > > > > > > >> time. > > > > > > >>>>> Frequent rebalance is also a bad thing for brokers. > > > > > > >>>>> > > > > > > >>>>> Having a client side configuration may solve problem 1 > better > > > > > because > > > > > > >>> each > > > > > > >>>>> consumer group can potentially configure their own timing. > > > > However, > > > > > > it > > > > > > >>> does > > > > > > >>>>> not really prevent frequent rebalance in general because > some > > > of > > > > > the > > > > > > >>>>> consumers can be misconfigured. (This may have something to > > do > > > > with > > > > > > >>> KIP-124 > > > > > > >>>>> as well. But if quota is applied on the JoinGroup/SyncGroup > > > > request > > > > > > it > > > > > > >>> may > > > > > > >>>>> cause some unwanted cascading effects.) > > > > > > >>>>> > > > > > > >>>>> Having a broker side configuration may result in less > > > flexibility > > > > > for > > > > > > >>> each > > > > > > >>>>> consumer group, but it can prevent frequent rebalance > > better. I > > > > > think > > > > > > >>> with > > > > > > >>>>> some reasonable design, the rebalance timing issue can be > > > > resolved > > > > > on > > > > > > >>> the > > > > > > >>>>> broker side as well. Matthias had a good point on extending > > the > > > > > delay > > > > > > >>> when > > > > > > >>>>> a new consumer joins a group (we actually did something > > similar > > > > to > > > > > > >> batch > > > > > > >>>>> ISR change propagation). For example, let's say on the > broker > > > > side, > > > > > > we > > > > > > >>> will > > > > > > >>>>> always delay 2 seconds each time we see a new consumer > > joining > > > a > > > > > > >>> consumer > > > > > > >>>>> group. This would probably work for most of the consumer > > groups > > > > and > > > > > > >> will > > > > > > >>>>> also limit the rebalance frequency to protect the brokers. > > > > > > >>>>> > > > > > > >>>>> I am not sure about the streams use case here, but if > > something > > > > > like > > > > > > 2 > > > > > > >>>>> seconds of delay is acceptable for streams, I would prefer > > > adding > > > > > the > > > > > > >>>>> configuration to the broker so that we can address both > > > problems. > > > > > > >>>>> > > > > > > >>>>> Thanks, > > > > > > >>>>> > > > > > > >>>>> Jiangjie (Becket) Qin > > > > > > >>>>> > > > > > > >>>>> > > > > > > >>>>> On Fri, Mar 24, 2017 at 5:30 AM, Damian Guy < > > > > damian....@gmail.com> > > > > > > >>> wrote: > > > > > > >>>>> > > > > > > >>>>>> Thanks for the feedback. > > > > > > >>>>>> > > > > > > >>>>>> Ewen: I'm happy to make it a client side config. Other > than > > > the > > > > > > >>> protocol > > > > > > >>>>>> bump i think the effort is almost the same. Personally i > see > > > no > > > > > > other > > > > > > >>>>>> issues, but based on discussions with others this is what > we > > > > came > > > > > up > > > > > > >>> with. > > > > > > >>>>>> > > > > > > >>>>>> True, it can probably be tested easily via an integration > > > test. > > > > > > >>>>>> > > > > > > >>>>>> Matthias: Yes i agree, the delay could be extended as each > > new > > > > > > member > > > > > > >>> joins > > > > > > >>>>>> the group. > > > > > > >>>>>> > > > > > > >>>>>> Thanks, > > > > > > >>>>>> Damian > > > > > > >>>>>> > > > > > > >>>>>> On Fri, 24 Mar 2017 at 05:14 Ewen Cheslack-Postava < > > > > > > >> e...@confluent.io> > > > > > > >>>>>> wrote: > > > > > > >>>>>> > > > > > > >>>>>>> I have the same initial response as Ismael re: broker vs > > > > consumer > > > > > > >>>>>> settings. > > > > > > >>>>>>> The global setting seems questionable. > > > > > > >>>>>>> > > > > > > >>>>>>> Could we maybe summarize what the impact of making this a > > > > client > > > > > > >>> config > > > > > > >>>>>>> would be? Protocol bump is obvious, but is there any > other > > > > > > >> significant > > > > > > >>>>>>> issue? For the protocol bump in particular, I think this > > > change > > > > > is > > > > > > >>>>>>> currently really critical for streams; it will be > valuable > > > > > > >> elsewhere, > > > > > > >>> but > > > > > > >>>>>>> the immediate demand is streams, so a protocol bump while > > > being > > > > > > >>> backwards > > > > > > >>>>>>> compatible wouldn't affect any other clients. Is this > still > > > > > > actually > > > > > > >>>>>>> compatible with different clients given that they would > now > > > > > expect > > > > > > >>>>>>> different timeouts? (I think it's strictly compatible if > > you > > > > wait > > > > > > >> for > > > > > > >>>>>>> responses, but if you enforce any client side timeouts, > I'm > > > not > > > > > so > > > > > > >>> sure.) > > > > > > >>>>>>> > > > > > > >>>>>>> re: test plan, I'm sure this will come as a surprise, but > > is > > > > the > > > > > > >>> system > > > > > > >>>>>>> test even necessary? Validating # of rebalances seems > messy > > > as > > > > > > other > > > > > > >>>>>> things > > > > > > >>>>>>> can cause rebalances (though admittedly not in a "clean" > > > case). > > > > > But > > > > > > >>>>>> really > > > > > > >>>>>>> it seems like an integration test could validate this by > > > making > > > > > > sure > > > > > > >>>>>> only 1 > > > > > > >>>>>>> rebalance occurred when 2 members joined with a > sufficient > > > time > > > > > > gap. > > > > > > >>>>>>> > > > > > > >>>>>>> -Ewen > > > > > > >>>>>>> > > > > > > >>>>>>> On Thu, Mar 23, 2017 at 3:53 PM, Matthias J. Sax < > > > > > > >>> matth...@confluent.io> > > > > > > >>>>>>> wrote: > > > > > > >>>>>>> > > > > > > >>>>>>>> Thanks for the KIP Damian! > > > > > > >>>>>>>> > > > > > > >>>>>>>> My two cents: > > > > > > >>>>>>>> > > > > > > >>>>>>>> - we should have an explicit parameter for this -- > > implicit > > > > > > setting > > > > > > >>>>>> are > > > > > > >>>>>>>> always tricky (the "importance" of this parameter would > be > > > > LOW) > > > > > > >>>>>>>> > > > > > > >>>>>>>> - the config should be different for each consumer > group: > > > > > > >>>>>>>> * assume you have a stateless app, you want to > rebalance > > > > > > >>> immediately > > > > > > >>>>>>>> * if you start-up in an visualized environment using > > some > > > > > tools > > > > > > >>> like > > > > > > >>>>>>>> Mesos you might need a different value that on bare > metal > > > (no > > > > VM > > > > > > to > > > > > > >>> be > > > > > > >>>>>>>> started) > > > > > > >>>>>>>> * it also depends, how many consumer instanced you > > expect > > > -- > > > > > > it's > > > > > > >>>>>>>> harder to start up 100 instances in 3 seconds than 5 > > > > > > >>>>>>>> > > > > > > >>>>>>>> - the default value should be zero > > > > > > >>>>>>>> > > > > > > >>>>>>>> > > > > > > >>>>>>>> One more thought: what about scaling scenarios? If a > > > consumer > > > > > > group > > > > > > >>> has > > > > > > >>>>>>>> 10 instanced and should be scaled up to 20, it would > make > > > > sense > > > > > to > > > > > > >> do > > > > > > >>>>>>>> this with a single rebalance, too. Thus, I am wondering, > > if > > > it > > > > > > >> would > > > > > > >>>>>>>> make sense to apply this delay each time a new consumer > > > joins > > > > > > >> group, > > > > > > >>>>>>>> even if the group is not empty? > > > > > > >>>>>>>> > > > > > > >>>>>>>> > > > > > > >>>>>>>> -Matthias > > > > > > >>>>>>>> > > > > > > >>>>>>>> > > > > > > >>>>>>>> On 3/23/17 10:19 AM, Damian Guy wrote: > > > > > > >>>>>>>>> Thanks Gouzhang - i think another problem with this is > > that > > > > is > > > > > > >>>>>>>> overloading > > > > > > >>>>>>>>> session.timeout.ms to mean multiple things. I'm not > sure > > > > that > > > > > is > > > > > > >> a > > > > > > >>>>>>> good > > > > > > >>>>>>>>> thing. > > > > > > >>>>>>>>> > > > > > > >>>>>>>>> On Thu, 23 Mar 2017 at 17:14 Guozhang Wang < > > > > wangg...@gmail.com > > > > > > > > > > > > >>>>>> wrote: > > > > > > >>>>>>>>> > > > > > > >>>>>>>>>> The downside of it, though, is that although it > "hides" > > > this > > > > > > from > > > > > > >>>>>> most > > > > > > >>>>>>>> of > > > > > > >>>>>>>>>> the users needing to be aware of it, by default > session > > > > > timeout > > > > > > >>> i.e. > > > > > > >>>>>>> the > > > > > > >>>>>>>>>> rebalance timeout is 10 seconds which could arguably > too > > > > long. > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>>> Guozhang > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>>> On Thu, Mar 23, 2017 at 10:12 AM, Guozhang Wang < > > > > > > >>> wangg...@gmail.com > > > > > > >>>>>>> > > > > > > >>>>>>>>>> wrote: > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>>>> Just throwing another alternative idea here: we can > > > > consider > > > > > > >> using > > > > > > >>>>>>> the > > > > > > >>>>>>>>>>> rebalance timeout value which is already included in > > the > > > > join > > > > > > >>>>>> request > > > > > > >>>>>>>>>>> protocol (and on the current Java client it is always > > > > written > > > > > > as > > > > > > >>>>>> the > > > > > > >>>>>>>>>>> session timeout value), that the first member joining > > > will > > > > > > >> always > > > > > > >>>>>>> force > > > > > > >>>>>>>>>> the > > > > > > >>>>>>>>>>> coordinator to wait that long. By doing this we do > not > > > need > > > > > to > > > > > > >>> bump > > > > > > >>>>>>> up > > > > > > >>>>>>>>>> the > > > > > > >>>>>>>>>>> protocol either. > > > > > > >>>>>>>>>>> > > > > > > >>>>>>>>>>> > > > > > > >>>>>>>>>>> Guozhang > > > > > > >>>>>>>>>>> > > > > > > >>>>>>>>>>> On Thu, Mar 23, 2017 at 5:49 AM, Damian Guy < > > > > > > >> damian....@gmail.com > > > > > > >>>> > > > > > > >>>>>>>>>> wrote: > > > > > > >>>>>>>>>>> > > > > > > >>>>>>>>>>>> Hi Ismael, > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> Mostly to avoid the protocol bump. > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> I agree that it may be difficult to choose the right > > > delay > > > > > for > > > > > > >>> all > > > > > > >>>>>>>>>>>> consumer > > > > > > >>>>>>>>>>>> groups, but we wanted to make this something that > most > > > > users > > > > > > >>> don't > > > > > > >>>>>>>>>> really > > > > > > >>>>>>>>>>>> need to think about, i.e., a small enough default > > delay > > > > that > > > > > > >>> works > > > > > > >>>>>>> in > > > > > > >>>>>>>>>> the > > > > > > >>>>>>>>>>>> majority of cases. However it would be much more > > > flexible > > > > > as a > > > > > > >>>>>>>> consumer > > > > > > >>>>>>>>>>>> config, which i'm happy to pursue if this change is > > > worthy > > > > > of > > > > > > a > > > > > > >>>>>>>> protocol > > > > > > >>>>>>>>>>>> bump. > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> Thanks, > > > > > > >>>>>>>>>>>> Damian > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> On Thu, 23 Mar 2017 at 12:35 Ismael Juma < > > > > ism...@juma.me.uk > > > > > > > > > > > > >>>>>> wrote: > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>>> Thanks for the KIP, Damian. It makes sense to avoid > > > > > multiple > > > > > > >>>>>>>>>> rebalances > > > > > > >>>>>>>>>>>>> during start-up. One issue with having this as a > > broker > > > > > > config > > > > > > >>> is > > > > > > >>>>>>>> that > > > > > > >>>>>>>>>>>> it > > > > > > >>>>>>>>>>>>> may be difficult to choose the right delay for all > > > > consumer > > > > > > >>>>>> groups. > > > > > > >>>>>>>>>> Can > > > > > > >>>>>>>>>>>> you > > > > > > >>>>>>>>>>>>> elaborate a little more on why the first > alternative > > > > (add a > > > > > > >>>>>>> consumer > > > > > > >>>>>>>>>>>>> config) was rejected? We bump protocol versions > > > regularly > > > > > > >> (when > > > > > > >>>>>> it > > > > > > >>>>>>>>>> makes > > > > > > >>>>>>>>>>>>> sense), so it would be good to get a bit more > detail. > > > > > > >>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>> Thanks, > > > > > > >>>>>>>>>>>>> Ismael > > > > > > >>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>> On Thu, Mar 23, 2017 at 12:24 PM, Damian Guy < > > > > > > >>>>>> damian....@gmail.com > > > > > > >>>>>>>> > > > > > > >>>>>>>>>>>> wrote: > > > > > > >>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>> Hi All, > > > > > > >>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>> I've prepared a KIP to add a configurable delay to > > the > > > > > > >> initial > > > > > > >>>>>>>>>>>> consumer > > > > > > >>>>>>>>>>>>>> group rebalance. > > > > > > >>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>> Please have look here: > > > > > > >>>>>>>>>>>>>> https://cwiki.apache.org/ > > > confluence/display/KAFKA/KIP- > > > > > > >>>>>>>>>>>>>> 134%3A+Delay+initial+consumer+group+rebalance > > > > > > >>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>> Thanks, > > > > > > >>>>>>>>>>>>>> Damian > > > > > > >>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>> BTW, i apologize if this appears twice. Seems the > > > first > > > > > one > > > > > > >> may > > > > > > >>>>>>> have > > > > > > >>>>>>>>>>>> not > > > > > > >>>>>>>>>>>>>> made it. > > > > > > >>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>> > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>> > > > > > > >>>>>>>>>>> > > > > > > >>>>>>>>>>> > > > > > > >>>>>>>>>>> -- > > > > > > >>>>>>>>>>> -- Guozhang > > > > > > >>>>>>>>>>> > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>>> -- > > > > > > >>>>>>>>>> -- Guozhang > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>> > > > > > > >>>>>>>> > > > > > > >>>>>>>> > > > > > > >>>>>>> > > > > > > >>>>>> > > > > > > >>>> > > > > > > >>> > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
