xiaokang commented on pull request #8451:
URL: https://github.com/apache/incubator-doris/pull/8451#issuecomment-1065797757
Hi @morningman, Thanks for your quick response!
>
> 1. It is better to provide some data to illustrate the memory usage before
and after the modification in order to give more information to the reviewer.
Test environment:
A single node cluster with 32g memory and 8 cpu cores. All configuration is
default.
Table schema:
CREATE TABLE `tableA` (
`uuid` text NULL,
`no` text NULL,
`f` text NULL,
`m` text NULL,
`data` text NULL,
`time1` datetime NULL,
`time2` datetime NULL,
`p` text NOT NULL
) ENGINE=OLAP
DUPLICATE KEY(`uuid`)
PARTITION BY LIST(`p`)
(PARTITION p1 VALUES IN ("1"),
...
PARTITION p98 VALUES IN ("98"),
PARTITION p99 VALUES IN ("99"))
DISTRIBUTED BY HASH(`uuid`) BUCKETS 10
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"in_memory" = "false",
"storage_format" = "V2"
)
Test steps:
1. create tableA and tableB with 100 partition.
2. insert 1 million rows with large average size 100KB to tableA.
3. execute query: 'set batch_size = 128; set exec_mem_limit=8147483648;
insert into tableB select * from table A;'
Result:
- master branch: be process quickly consume all 32g memory in about 20
seconds and killed by Linux oom-killer.
- fix branch: The memory usage of be process is 14g memory at peak and fall
back to 8g when query finished. 8g is for be storage page cache (32g*20%=6g)
and chunk_reserved_bytes (2g).
> 2. Also need to modify the `src/vec/exec/volap_scan_node.cpp`
I will try to do the same things for the vectorized code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
