date:20220823

[VOTE] [PIP-169] Support split bundle by flow or qps

2022-08-23 Thread lordcheng10

Hi Pulsar Community, I would like to start a VOTE on "Support split bundle by 
flow or qps."(PIP-169)
Here is the design 
detail  https://github.com/apache/pulsar/issues/16782 ;


and the discussion 
thread https://lists.apache.org/thread/cshyt10fwcjjxs93g8yf0svgwcgnshmg ;
 

Thanks, 

lordcheng10

[VOTE] [PIP-169] Support split bundle by flow or qps

2022-08-23 Thread lordcheng10

Hi Pulsar Community, I would like to start a VOTE on "Support split bundle by 
flow or qps."(PIP-169)
Here is the design detail: https://github.com/apache/pulsar/issues/16782


and the discussion thread: 
https://lists.apache.org/thread/cshyt10fwcjjxs93g8yf0svgwcgnshmg


Thanks,
lordcheng10

Re: [Discuss][PIP-164] Support split bundle by flow or qps

2022-08-23 Thread PengHui Li

It's a good idea to improve the bundle split for the case that the traffic
of the topic doesn't change drastically
Otherwise, we should not use this policy. or can we use it for all cases?
I think It should be documented in the proposal.

I have some questions

- do we need to consider the consumer rate
- the `flow or qps` is based on the entries or messages?
  IMO, based on the entries and throughput is more reasonable. Usually, the
entry rate affects the CPU usage, the throughput affects the network usage.

Thanks,
Penghui

On Mon, Aug 22, 2022 at 3:53 PM lordcheng10 <1572139...@qq.com.invalid>
wrote:

> The implementation logic has been modified, and the corresponding example
> has also been modified as follows:
>
>
> ## Motivation
> As we all know, Bundle split has 3 algorithms:
> - range_equally_divide
> - topic_count_equally_divide
> - specified_positions_divide
>
>
> However, none of these algorithms can divide bundles according to flow or
> qps, which may cause bundles to be split multiple times.
>
>
> ## Goal
> Our goal is to split bundles according to flow or QPS, so we propose a PIP
> to introduce a split algorithm based on flow or QPS.
> The main idea is that we can get the flow or qps information of a topic
> contained in a bundle,
> then split according to loadBalancerNamespaceBundleMaxMsgRate or
> loadBalancerNamespaceBundleMaxBandwidthMbytes configuration
>
>
> For example, there is bundle with boundaries 0x to 0x0200, and
> six topics : t1 , t2 , t3 , t4, t5, t6.
> loadBalancerNamespaceBundleMaxMsgRate=1100
> loadBalancerNamespaceBundleMaxBandwidthMbytes=110
>
>
> **Step 1: Get their hash position and corresponding flow and QPS:**
>
>
> > t1 with hashcode 10 msgRate 100/s throughput 10M/s
> >
> > t2 with hashcode 20 msgRate 200/s throughput 20M/s
> >
> > t3 with hashcode 80 msgRate 300/s throughput 30M/s
> >
> > t4 with hashcode 90 msgRate 400/s throughput 40M/s
> >
> > t5 with hashcode 100 msgRate 500/s throughput 50M/s
> >
> > t6 with hashcode 110 msgRate 600/s throughput 60M/s
>
>
>
>
> **Step 2: Calculate the total flow and qps of the bundle:**
>
>
> > bundleMsgRate = 100 + 200 + 300 + 400 + 500 + 600 = 2100
> > bundleThroughput = 10 + 20 + 30 + 40 + 50 + 60 = 210MB
>
>
> **Step 3: Calculate the position to split and split:**
>
>
> > QPS: (100 + 200 + 300 + 400 ) <
> loadBalancerNamespaceBundleMaxMsgRate=1100 & (100+200+300+400+500) >
> loadBalancerNamespaceBundleMaxMsgRate=1100
> > flow: (10 + 20 + 30 + 40 ) <
> loadBalancerNamespaceBundleMaxBandwidthMbytes=110 & (10 + 20 + 30 + 40
> + 50 ) > loadBalancerNamespaceBundleMaxBandwidthMbytes=110
> >
> > Split between t4 and t5:
> > splitStartPosition = 90
> > splitEndPosition = 100
> > splitPosition = (90 + 100) / 2 = 95
>
>
>
>
> ## API Changes
>
>
> 1. Add a split algorithm class based on flow or qps:
>
>
> `public class FlowOrQpsEquallyDivideBundleSplitAlgorithm implements
> NamespaceBundleSplitAlgorithm `
>
>
> 2. update the default configuration:
> ```
> private List Lists.newArrayList("range_equally_divide", "topic_count_equally_divide",
>    "specified_positions_divide", "flow_count_equally_divide");
> ```
> 3. added configuration
> ```
> @FieldContext(
>         dynamic = true,
>         category = CATEGORY_LOAD_BALANCER,
>         doc = "Acceptable difference between qps and
> loadBalancerNamespaceBundleMaxMsgRate "
>                 + " or flow and
> loadBalancerNamespaceBundleMaxBandwidthMbytes "
> )
> private int flowOrQpsDifferenceThresholdPercentage = 10;
> ```
>
>
>
>
> ## Implementation
> The execution steps of the
> FlowOrQpsEquallyDivideBundleSplitAlgorithm#getSplitBoundary method are as
> follows:
> 1. Get the hash position of each topic and the corresponding msgRate and
> msgThroughput, and sort them according to the position size：
>
>
> ```
> List Map Map ```
>
>
> 2. According to the topic hash position, traverse all topics from small to
> large,
> and split the bundle according to the configured
> loadBalancerNamespaceBundleMaxMsgRate or
> loadBalancerNamespaceBundleMaxBandwidthMbytes:
>
>
> ```
>   double bundleMsgRateTmp = 0;
>   double bundleThroughputTmp = 0;
>   for (int i = 0; i < topicNameHashList.size(); i++) {
>       long topicHashCode = topicNameHashList.get(i);
>       bundleThroughputTmp +=
> hashAndThroughput.get(topicHashCode);
>       bundleMsgRateTmp += hashAndMsgMap.get(topicHashCode);
>
>
>       if (bundleMsgRateTmp >
> loadBalancerNamespaceBundleMaxMsgRate
>             || bundleThroughputTmp >
> loadBalancerNamespaceBundleMaxBandwidthBytes) {
>             long splitStart = i > 0 ?
> topicNameHashList.get(i - 1) : topicHashCode;
>             long splitEnd = i > 0 ?
> topicHashCode : topicNameHashList.get(i + 1);
>             long splitMiddle = splitStart +
> (splitEnd - splitStart) / 2;
>             splitResults.add(splitMiddle);
>             bundleMsgRateTmp = 
> hashAndMsgMap.get(topicHashCode);
>             bundleThroughputTmp =
> hashAndThroughput.get(topicHashCode);
>

[DISCUSS] Create a robust and inline Client Feature Matrix page

2022-08-23 Thread Jun Ma

Hi, Pulsar community,

As you may notice, the Client Feature Matrix [0] has been linked on the Pulsar
doc site [1] for quite a while, providing an overview of feature supportability
on language-specific clients. As the outcome of PIP-108, it has addressed the
initial community request [2] for technology evaluation and selection.

However, it has the following limitations to continually serving the purpose
over time:
1. Visibility: Not prominent for users/maintainers to notice it.
2. Process: No required review/approval or version control.
3. Accuracy: A bit out-of-dated with limited chances to get it updated
(possibly caused by 1&2).

To make it more robust and prominent to better serve the adoption purpose, I
think we can make the following improvements:
1. Deliver a more robust Client Feature Matrix and required documentation
through a thorough review and update.
2. Move the matrix to the Pulsar repo and display it inline on a web page.
Refer to this reference [3].

A quick question is about the granularity of the feature sets presented in the
new matrix.
Generally, we have two options:
1. Display a full version as we do in the existing feature matrix to provide
more detailed tech capabilities.
2. Display a compact version with high-level features to provide better
readability.

Feel free to share your thoughts.

[0]
https://docs.google.com/spreadsheets/d/1YHYTkIXR8-Ql103u-IMI18TXLlGStK8uJjDsOOA0T20/edit?usp=sharing
[1] https://pulsar.apache.org/docs/next/client-libraries#feature-matrix
[2] https://github.com/apache/pulsar/issues/9723
[3]
https://beam.apache.org/documentation/runners/capability-matrix/when-in-processing-time/

Cheers,
momo-jun

47 matches

Mail list logo