Hi Guozhang, Only evicting due to memory pressure (or if the metadata is stale, to prevent infinitely refreshing no longer used topics when memory's not an issue) makes sense, and would be the most intuitive way of going about it.
>From an implementation perspective, it's a little more difficult because it'd require the producer's BufferPool to feed back into the Metadata object, introduce a sane limit on the consumer side, and add the concept of potentially evicting stale metadata instead of refreshing it. So it's a little involved, and I'd prefer to save that for a round 2, if necessary. I've updated the KIP to change the default eviction time to 2 minutes. Thanks, Brian On Thu, Jan 2, 2020 at 1:28 PM Guozhang Wang <wangg...@gmail.com> wrote: > On Thu, Jan 2, 2020 at 12:42 PM Brian Byrne <bby...@confluent.io> wrote: > > > Hi Guozhang, > > > > You're correct in that, with the current defined default values, it's > > likely we'll be more proactive in refreshing metadata for > > once/rarely-touched topics, which isn't ideal. It'd be at most 2x the > rate, > > but we can certainly do better. I made the default eviction rate to > mirror > > the current hard-coded value in effort to minimize surprise, but perhaps > > there's a more suitable middle ground. > > > > Your mental model is good. I also agree that having a more aggressive > > eviction rate, or at least one that's a fair amount shorter than the > topic > > refresh rate, behaves better for the once/rarely-touched topics and is > more > > appropriate overall. One thing to note though is that, if metadata is > > evicted + refetched, there'll be a small latency bubble for the producer. > > > > The concern I had with shortening the eviction rate dramatically was that > > it could have unexpected consequences for clusters that keep the default, > > but whose P1 is slightly longer than the value. For example, if the > > metadata.evict.ms is 30 seconds, a periodic access rate of 40 seconds > > could > > request the topic metadata up to 7 times during a 5-minute interval. Of > > course, this is easily fixed by setting the eviction rate appropriately > by > > the client, but perhaps lowering the default to a moderate ~2 minutes > might > > be a better balance to avoid surprise of worst-case behavior. > > > > Something else that could be considered is defining a "soft topic > metadata > > cache size", which would only evict unused topics (last used >= > > metadata.evict.ms) if the soft cache size was exceeded or the metadata > was > > stale. However, it still doesn't save the client from degenerate behavior > > where P1 + e ~= metadata.evict.ms, so I'm more partial to keeping the > > eviction rate to ~2+ minutes. > > > > This makes sense. > > I think since the scenario that a producer sends to a topic once and never > did again is not very common, and more commonly a single producer would > only send to a very small subset of topics hosted on the cluster during its > life time, we can actually use a cache size to guide when should we > consider eviction (my understanding is that the memory pressure is the only > reason we want to evict, lmk if there are other motivations). A wild > thought is that we do not even need to introduce a new config since we have > a `buffer.size` already and we can calculate the exact bytes-size of each > topic metadata entry as well, so we can, say, consider evicting only when > that buffer size is reached together with buffered data AND metadata in > bytes. Then in practice we would not need to evict metadata anyways, AND if > we do need to, having some aggressive eviction threshold is more valid (btw > I think I also agree that something like 2minutes is fine). > > > > Yet another option is to have topics in the "U" set (i.e. topics > requesting > > non-urgent metadata update) exclude those that are nearing their eviction > > duration. I fear this might be over-engineering a potential non-problem > > though. Thoughts? > > > > Yeah I also feel it is a bit overkill for this. > > > > > > Thanks, > > Brian > > > > > > > > On Thu, Jan 2, 2020 at 11:44 AM Guozhang Wang <wangg...@gmail.com> > wrote: > > > > > Hello Brain, > > > > > > For the newly added `metadata.evict.ms` config, since its default > value > > is > > > set as the same as `metadata.max.age.ms` and most users would not > > override > > > default config values, would it be possible for scenarios that producer > > > send to a specific topic once and never send again, we may end up > > > refreshing the metadata that are about to be removed soon? In other > words > > > I'm wondering if for common cases the eviction period should be larger > > than > > > max.age or vice versa. This is my mental model: > > > > > > 1) If we evict a topic eagerly compared with refreshing it, e.g. after > > time > > > T1 has elapsed, and then after T2 (> T1) elapsed that topic is > requested > > > for sending again, we would force a metadata update which includes this > > > topic (remember we now always batch a set of topics in each metadata > > > request) > > > 2) if we evict a topic more lazily compared with refreshing it, say > after > > > T2 has elapsed, but before that after T1 (< T2) we have to refresh it > > > already, it is included in that metadata request. > > > > > > So from the number of metadata requests it may incur, evicting more > > eagerly > > > seems okay --- i.e. even if we do not evict this topic, it would still > > > cause a metadata request after the max.age.ms anyways so evicting it > > early > > > does not make things worse. More generally, if we know a topic's > sending > > > frequency is P1 (i.e. every P1 seconds we send once for that topic), > and > > > eviction period is P2, and the metadata.max.age is P3. Then the optimal > > > scenario in terms of metadata update frequency seems to be P1 = P2 < > P3. > > So > > > I feel making the default value for metadata.evict.ms smaller is fine, > > > e.g. > > > say 1minute or 30seconds. WDYT? > > > > > > > > > Guozhang > > > > > > On Thu, Dec 26, 2019 at 7:08 AM Brian Byrne <bby...@confluent.io> > wrote: > > > > > > > Hi Stanislav, > > > > > > > > Appreciate the feedback! > > > > > > > > 1. You're correct. I've added notes to the KIP to clarify. > > > > > > > > 2. Yes it should. Fixed. > > > > > > > > 3. So I made a mistake when generalizing the target refresh size, > which > > > > should have been using `metadata.max.age.ms` instead of ` > > > metadata.evict.ms > > > > `. > > > > Therefore, `metadata.max.age.ms` still controls the target refresh > > time > > > of > > > > active topic metadata, and doesn't change in that regard. You're > right > > in > > > > that the U set will make metadata refreshing more proactive in some > > > cases, > > > > which is done to smooth out the metadata RPCs over time and lessen > the > > > > potential bulk of some of the RPC responses by working in smaller > > units. > > > > The T set is what logically exists today and doesn't change. > > > > > > > > Thanks, > > > > Brian > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Dec 26, 2019 at 2:16 AM Stanislav Kozlovski < > > > > stanis...@confluent.io> > > > > wrote: > > > > > > > > > Hey Brian, > > > > > > > > > > 1. Could we more explicitly clarify the behavior of the algorithm > > when > > > > `|T| > > > > > > TARGET_METADATA_FETCH SIZE` ? I assume we ignore the config in > that > > > > > scenario > > > > > 2. Should `targetMetadataFetchSize = Math.max(topicsPerSec / 10, > 20)` > > > be > > > > > `topicsPerSec * 10` ? > > > > > 3. When is this new algorithm applied? To confirm my understanding > - > > > what > > > > > is the behavior of `metadata.max.age.ms` after this KIP? Are we > > > adding a > > > > > new, more proactive metadata fetch for topics in |U|? > > > > > > > > > > Thanks, > > > > > Stanislav > > > > > > > > > > On Thu, Dec 19, 2019 at 11:37 PM Brian Byrne <bby...@confluent.io> > > > > wrote: > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > For all interested, please take a look at the proposed algorithm > as > > > I'd > > > > > > like to get more feedback. I'll call for a vote once the break is > > > over. > > > > > > > > > > > > Thanks, > > > > > > Brian > > > > > > > > > > > > On Mon, Dec 9, 2019 at 10:18 PM Guozhang Wang < > wangg...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > Sounds good, I agree that should not make a big difference in > > > > practice. > > > > > > > > > > > > > > On Mon, Dec 9, 2019 at 2:07 PM Brian Byrne < > bby...@confluent.io> > > > > > wrote: > > > > > > > > > > > > > > > Hi Guozhang, > > > > > > > > > > > > > > > > I see, we agree on the topic threshold not applying to urgent > > > > topics, > > > > > > but > > > > > > > > differ slightly on what should be considered urgent. I would > > > argue > > > > > that > > > > > > > we > > > > > > > > should consider topics nearing the metadata.max.age.ms to be > > > > urgent > > > > > > > since > > > > > > > > they may still be well within the metadata.expiry.ms. That > is, > > > the > > > > > > > client > > > > > > > > still considers these topics to be relevant (not expired), > but > > > > > doesn't > > > > > > > want > > > > > > > > to incur the latency bubble of having to wait for the > metadata > > to > > > > be > > > > > > > > re-fetched if it's stale. This could be a frequent case if > the > > > > > > > > metadata.max.age.ms << metadata.expiry.ms. > > > > > > > > > > > > > > > > In practice, I wouldn't expect this to make a noticeable > > > difference > > > > > so > > > > > > I > > > > > > > > don't have a strong leaning, but the current behavior today > is > > to > > > > > > > > aggressively refresh the metadata of stale topics by > ensuring a > > > > > refresh > > > > > > > is > > > > > > > > triggered before that metadata.max.age.ms duration elapses. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Brian > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Dec 9, 2019 at 11:57 AM Guozhang Wang < > > > wangg...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hello Brian, > > > > > > > > > > > > > > > > > > Thanks for your explanation, could you then update the wiki > > > page > > > > > for > > > > > > > the > > > > > > > > > algorithm part since when I read it, I thought it was > > different > > > > > from > > > > > > > the > > > > > > > > > above, e.g. urgent topics should not be added just because > of > > > > > max.age > > > > > > > > > expiration, but should only be added if there are sending > > data > > > > > > pending. > > > > > > > > > > > > > > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > > > > On Mon, Dec 9, 2019 at 10:57 AM Brian Byrne < > > > bby...@confluent.io > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Guozhang, > > > > > > > > > > > > > > > > > > > > Thanks for the feedback! > > > > > > > > > > > > > > > > > > > > On Sun, Dec 8, 2019 at 6:25 PM Guozhang Wang < > > > > wangg...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > 1. The addition of *metadata.expiry.ms < > > > > > > http://metadata.expiry.ms> > > > > > > > > > > *should > > > > > > > > > > > be included in the public interface. Also its semantics > > > needs > > > > > > more > > > > > > > > > > > clarification (since previously it is hard-coded > > internally > > > > we > > > > > do > > > > > > > not > > > > > > > > > > need > > > > > > > > > > > to explain it publicly, but now with the configurable > > value > > > > we > > > > > do > > > > > > > > > need). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This was an oversight. Done. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. There are a couple of hard-coded parameters like 25 > > and > > > > 0.5 > > > > > in > > > > > > > the > > > > > > > > > > > proposal, maybe we need to explain why these magic > values > > > > makes > > > > > > > sense > > > > > > > > > in > > > > > > > > > > > common scenarios. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > So these are pretty fuzzy numbers, and seemed to be a > > decent > > > > > > balance > > > > > > > > > > between trade-offs. I've updated the target size to > account > > > for > > > > > > > setups > > > > > > > > > with > > > > > > > > > > a large number of topics or a shorter refresh time, as > well > > > as > > > > > > added > > > > > > > > some > > > > > > > > > > light rationale. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3. In the Urgent set condition, do you actually mean > > "with > > > no > > > > > > > cached > > > > > > > > > > > metadata AND there are existing data buffered for the > > > topic"? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes, fixed. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > One concern I have is whether or not we may introduce a > > > > > > regression, > > > > > > > > > > > especially during producer startup such that since we > > only > > > > > > require > > > > > > > up > > > > > > > > > to > > > > > > > > > > 25 > > > > > > > > > > > topics each request, it may cause the send data to be > > > > buffered > > > > > > more > > > > > > > > > time > > > > > > > > > > > than now due to metadata not available. I understand > this > > > is > > > > a > > > > > > > > > > acknowledged > > > > > > > > > > > trade-off in our design but any regression that may > > surface > > > > to > > > > > > > users > > > > > > > > > need > > > > > > > > > > > to be very carefully considered. I'm wondering, e.g. if > > we > > > > can > > > > > > > tweak > > > > > > > > > our > > > > > > > > > > > algorithm for the Urgent set, e.g. to consider those > with > > > non > > > > > > > cached > > > > > > > > > > > metadata have higher priority than those who have > elapsed > > > > > max.age > > > > > > > but > > > > > > > > > not > > > > > > > > > > > yet have been called for sending. More specifically: > > > > > > > > > > > > > > > > > > > > > > Urgent: topics that have been requested for sending but > > no > > > > > cached > > > > > > > > > > metadata, > > > > > > > > > > > and topics that have send request failed with e.g. > > > > NOT_LEADER. > > > > > > > > > > > Non-urgent: topics that are not in Urgent but have > > expired > > > > > > max.age. > > > > > > > > > > > > > > > > > > > > > > Then when sending metadata, we always send ALL in the > > > urgent > > > > > > (i.e. > > > > > > > > > ignore > > > > > > > > > > > the size limit), and only when they do not exceed the > > size > > > > > limit, > > > > > > > > > > consider > > > > > > > > > > > fill in more topics from Non-urgent up to the size > limit. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think we're on the same page here. Urgent topics ignore > > the > > > > > > target > > > > > > > > > > metadata RPC size, and are not bound by it in any way, > i.e. > > > if > > > > > > > there's > > > > > > > > > 100 > > > > > > > > > > urgent topics, we'll fetch all 100 in a single RPC. Like > > you > > > > say, > > > > > > > > > however, > > > > > > > > > > if a topic transitions to urgent and there's several > > > non-urgent > > > > > > ones, > > > > > > > > > we'll > > > > > > > > > > piggyback the non-urgent updates up to the target size. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Brian > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Nov 20, 2019 at 7:00 PM deng ziming < > > > > > > > dengziming1...@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > I think it's ok, and you can add another issue about > > > > > > > `asynchronous > > > > > > > > > > > > metadata` if `topic expiry` is not enough. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Nov 21, 2019 at 6:20 AM Brian Byrne < > > > > > > bby...@confluent.io > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hello all, > > > > > > > > > > > > > > > > > > > > > > > > > > I've refactored the KIP to remove implementing > > > > asynchronous > > > > > > > > > metadata > > > > > > > > > > > > > fetching in the producer during send(). It's now > > > > > exclusively > > > > > > > > > focused > > > > > > > > > > on > > > > > > > > > > > > > reducing the topic metadata fetch payload and > > proposes > > > > > > adding a > > > > > > > > new > > > > > > > > > > > > > configuration flag to control topic expiry > behavior. > > > > Please > > > > > > > take > > > > > > > > a > > > > > > > > > > look > > > > > > > > > > > > > when possible. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-526 > > > > > > > > > > > > > > > > > > > %3A+Reduce+Producer+Metadata+Lookups+for+Large+Number+of+Topics > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > Brian > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 4, 2019 at 10:04 AM Brian Byrne < > > > > > > > bby...@confluent.io > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Lucas, Guozhang, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you for the comments. Good point on > > > > > > > > > METADATA_MAX_AGE_CONFIG - > > > > > > > > > > > it > > > > > > > > > > > > > > looks like the ProducerMetadata was > differentiating > > > > > between > > > > > > > > > expiry > > > > > > > > > > > and > > > > > > > > > > > > > > refresh, but it should be unnecessary to do so > once > > > the > > > > > > cost > > > > > > > of > > > > > > > > > > > > fetching > > > > > > > > > > > > > a > > > > > > > > > > > > > > single topic's metadata is reduced. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I've updated the rejected alternatives and > removed > > > the > > > > > > config > > > > > > > > > > > > variables. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Brian > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 4, 2019 at 9:20 AM Guozhang Wang < > > > > > > > > wangg...@gmail.com > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hello Brian, > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Thanks for the KIP. > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> I think using asynchronous metadata update to > > > address > > > > 1) > > > > > > > > > metadata > > > > > > > > > > > > update > > > > > > > > > > > > > >> blocking send, but for other issues, currently > at > > > > > producer > > > > > > > we > > > > > > > > do > > > > > > > > > > > have > > > > > > > > > > > > a > > > > > > > > > > > > > >> configurable `METADATA_MAX_AGE_CONFIG` similar > to > > > > > > consumer, > > > > > > > by > > > > > > > > > > > default > > > > > > > > > > > > > is > > > > > > > > > > > > > >> 5min. So maybe we do not need to introduce new > > > configs > > > > > > here, > > > > > > > > but > > > > > > > > > > > only > > > > > > > > > > > > > >> change the semantics of that config from global > > > expiry > > > > > > > (today > > > > > > > > we > > > > > > > > > > > just > > > > > > > > > > > > > >> enforce a full metadata update for the whole > > > cluster) > > > > to > > > > > > > > > > > single-topic > > > > > > > > > > > > > >> expiry, and we can also extend its expiry > deadline > > > > > > whenever > > > > > > > > that > > > > > > > > > > > > > metadata > > > > > > > > > > > > > >> is successfully used to send a produce request. > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Guozhang > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> On Thu, Oct 3, 2019 at 6:51 PM Lucas Bradstreet > < > > > > > > > > > > lu...@confluent.io > > > > > > > > > > > > > > > > > > > > > > > > > >> wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Hi Brian, > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > This looks great, and should help reduce > > blocking > > > > and > > > > > > high > > > > > > > > > > > metadata > > > > > > > > > > > > > >> request > > > > > > > > > > > > > >> > volumes when the producer is sending to large > > > > numbers > > > > > of > > > > > > > > > topics, > > > > > > > > > > > > > >> especially > > > > > > > > > > > > > >> > at low volumes. I think the approach to make > > > > metadata > > > > > > > > fetching > > > > > > > > > > > > > >> asynchronous > > > > > > > > > > > > > >> > and batch metadata requests together will help > > > > > > > > significantly. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > The only other approach I can think of is to > > allow > > > > > users > > > > > > > to > > > > > > > > > > supply > > > > > > > > > > > > the > > > > > > > > > > > > > >> > producer with the expected topics upfront, > > > allowing > > > > > the > > > > > > > > > producer > > > > > > > > > > > to > > > > > > > > > > > > > >> perform > > > > > > > > > > > > > >> > a single initial metadata request before any > > sends > > > > > > occur. > > > > > > > I > > > > > > > > > see > > > > > > > > > > no > > > > > > > > > > > > > real > > > > > > > > > > > > > >> > advantages to this approach compared to the > > async > > > > > method > > > > > > > > > you’ve > > > > > > > > > > > > > >> proposed, > > > > > > > > > > > > > >> > but maybe we could add it to the rejected > > > > alternatives > > > > > > > > > section? > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > Thanks, > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > Lucas > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > On Fri, 20 Sep 2019 at 11:46, Brian Byrne < > > > > > > > > > bby...@confluent.io> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > I've updated the 'Proposed Changes' to > include > > > two > > > > > new > > > > > > > > > > producer > > > > > > > > > > > > > >> > > configuration variables: topic.expiry.ms > and > > > > > > > > > topic.refresh.ms > > > > > > > > > > . > > > > > > > > > > > > > Please > > > > > > > > > > > > > >> > take > > > > > > > > > > > > > >> > > a look. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > Thanks, > > > > > > > > > > > > > >> > > Brian > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > On Tue, Sep 17, 2019 at 12:59 PM Brian > Byrne < > > > > > > > > > > > bby...@confluent.io > > > > > > > > > > > > > > > > > > > > > > > > > > >> > wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > Dev team, > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > Requesting discussion for improvement to > the > > > > > > producer > > > > > > > > when > > > > > > > > > > > > dealing > > > > > > > > > > > > > >> > with a > > > > > > > > > > > > > >> > > > large number of topics. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > KIP: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-526%3A+Reduce+Producer+Metadata+Lookups+for+Large+Number+of+Topics > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > JIRA: > > > > > > > https://issues.apache.org/jira/browse/KAFKA-8904 > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > Thoughts and feedback would be > appreciated. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > Thanks, > > > > > > > > > > > > > >> > > > Brian > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> -- > > > > > > > > > > > > > >> -- Guozhang > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Best, > > > > > Stanislav > > > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > -- > -- Guozhang >
