Good job!
Expecting your result!
--
此致！Best Regards
陈明雨 Mingyu Chen
Email:
chenmin...@apache.org
At 2019-06-21 17:43:00, "Miao,Ling" <miaolin...@baidu.com> wrote:
>Hi all,
>
>I already finished the new load framework(streaming) of broker load and mini
>load. It’s time to verify the performance of new framework.
>
>This test will ignore the effect of rollup and query. The result of test will
>only depends on the performance of new load framework.
>
>
>The environment:
>Node 1:
>
> * Memory: about 3G
> * Disk: SATA about 2T
> * Network: about 10000Mb/s
>
>Node 2,3,4:
>
> * Memory: about 27G
> * Disk: SSD about 369G
> * Network: about 10000Mb/s
>
>The prepare of test:
>
> 1. Install doris:
>
> * One frontend and three backends.
> * Three backends are located in different node(2,3,4).
> * The frontend is located Node1.
>
> 1. Create table:
>
> * Situation 1: The table has only one tablet and 3 replicas.
>SQL: create table xxx (about 34 columns: 16 sum aggregation columns)
>ENGINE=OLAP AGGREGATE KEY(2 columns) DISTRIBUTED BY HASH(2 columns) BUCKETS 1
>PROPERTIES ("storage_type" = "COLUMN");
> * Situation 2: The table has 100 tablet and every tablet has 3 replicase.
>SQL: create table xxx (about 34 columns: sum 16) ENGINE=OLAP AGGREGATE KEY(2
>columns) DISTRIBUTED BY HASH(2 columns) BUCKETS 100 PROPERTIES ("storage_type"
>= "COLUMN");
>
>The data of test:
>Load different size (2K, 40M, 20G, 280G) of file: catalog_sales.dat
>
>The step of test:
>
> 1. Load data by non-streaming broker load
> 2. Load data by streaming broker load
> 3. Load data by non-streaming mini load
> 4. Load data by streaming mini load
>
>Is the plan of test precise？ I am looking forward to your suggestions of
>’Testing the performance of new load framework(streaming)’
>
>Best Regards,
>Emmy Miao
>
>
>发件人: Ling Miao <miaolin...@baidu.com<mailto:miaolin...@baidu.com>>
>日期: 2019年4月9日 星期二 下午4:39
>至: "dev@doris.apache.org<mailto:dev@doris.apache.org>"
><dev@doris.apache.org<mailto:dev@doris.apache.org>>
>抄送: palo-rd <palo...@baidu.com<mailto:palo...@baidu.com>>, Ling Miao
><miaolin...@baidu.com<mailto:miaolin...@baidu.com>>
>主题: [DISCUSS] Unify the load framework in Doris
>
>Hi all,
>
>I prepare to unify the load framework in Doris.
>
>The background is following:
>There are different load frameworks belong to different load methods in Doris.
>On the one hand, the performance of broker and mini load which use the old
>framework is lower than stream load. On the other hand, the loading of large
>file is not supported by the framework of broker load which includes
>non-streaming sink.
>
>The purposes of ‘Unify the Load Framework’ are following:
>
> 1. Improve the performance of broker and mini load by using the framework
> of stream load.
> 2. The loading of large file is supported by the unified framework.
>
>I am looking forward to your suggestions of ‘Unify the Load Framework'.
>
>Best Regards,
>Emmy Miao
