huangmengbin edited a comment on pull request #5369: URL: https://github.com/apache/incubator-doris/pull/5369#issuecomment-879117362
> ## Proposed changes > In the previous broker load, multiple OlapTableSinks would send data to the same LoadChannel, > and because of the lock granularity problem, LoadChannel could only process these requests serially, > which made it impossible to make full use of cluster resources. > > This CL modifies the related locks so that LoadChannel can process these requests in parallel. > > In the test, with a size of 20G, the load speed of 334 million rows of data in 3 nodes has been > increased from 9min to 5min, and after enabling 2 concurrency, it can be increased to 3min. > > Also modify the profile of load job. > > ## Types of changes > * [x] Breaking change (fix or feature that would cause existing functionality to not work as expected) > > ## Checklist > * [x] I have created an issue on (Fix [[Performance] Import the load performace #5281](https://github.com/apache/incubator-doris/issues/5281) ) and described the bug/feature there in detail Hi, morningman ! 请问一下可以更详细地描述一下当时测试时的信息吗？比如列的数量、哪种数据模型(Uniq、Aggregate、Duplicate)，等等？我在uniq模型上进行测试(9G数据、9台机器、1.3亿行、246列) 没有发现其有较明显的提速(13.5min->10min)，不知道是否符合代码的预期。（暂时还没找到原因 Thanks ！ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
