Re: [PR] [performance](pipelineX) optimize get local state [doris]

2023-10-20 Thread via GitHub


github-actions[bot] commented on PR #25675:
URL: https://github.com/apache/doris/pull/25675#issuecomment-1772190418

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Bug](samples)fix IndexOutOfBoundsException [doris]

2023-10-20 Thread via GitHub


github-actions[bot] commented on PR #25608:
URL: https://github.com/apache/doris/pull/25608#issuecomment-1772191924

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Bug](samples)fix IndexOutOfBoundsException [doris]

2023-10-20 Thread via GitHub


github-actions[bot] commented on PR #25608:
URL: https://github.com/apache/doris/pull/25608#issuecomment-1772192000

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [I] [Bug] About the limit query, limit 1, the background will still perform a large number of calculations after the output of 1 result.关于limit查询,limit 1,输出结果1条后,后台仍会进行大量计算的问题。 [doris]

2023-10-20 Thread via GitHub


shaotianfeng commented on issue #25604:
URL: https://github.com/apache/doris/issues/25604#issuecomment-1772194450

   昨天有试着升级2.0.2还是存在这个问题,然后又降级了
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](stats) analyze spec col only [doris]

2023-10-20 Thread via GitHub


Kikyou1997 commented on PR #25660:
URL: https://github.com/apache/doris/pull/25660#issuecomment-1772194480

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [typo](doc)Modify the default value of Stale rowset cleanup policy [doris]

2023-10-20 Thread via GitHub


JNSimba merged PR #25517:
URL: https://github.com/apache/doris/pull/25517


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Bug](samples)fix IndexOutOfBoundsException [doris]

2023-10-20 Thread via GitHub


JNSimba merged PR #25608:
URL: https://github.com/apache/doris/pull/25608


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[doris] branch master updated: [typo](doc)Modify the default value of Stale rowset cleanup policy(#25517)

2023-10-20 Thread diwu
This is an automated email from the ASF dual-hosted git repository.

diwu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 68d3c25f263 [typo](doc)Modify the default value of Stale rowset 
cleanup policy(#25517)
68d3c25f263 is described below

commit 68d3c25f263916b2aff569f44de1b0b8de61816f
Author: caoliang-web <71004656+caoliang-...@users.noreply.github.com>
AuthorDate: Fri Oct 20 15:03:49 2023 +0800

[typo](doc)Modify the default value of Stale rowset cleanup policy(#25517)
---
 docs/en/docs/admin-manual/config/be-config.md| 2 +-
 docs/zh-CN/docs/admin-manual/config/be-config.md | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/en/docs/admin-manual/config/be-config.md 
b/docs/en/docs/admin-manual/config/be-config.md
index 4fc3106fbe9..05a8a5d16a0 100644
--- a/docs/en/docs/admin-manual/config/be-config.md
+++ b/docs/en/docs/admin-manual/config/be-config.md
@@ -1266,7 +1266,7 @@ BaseCompaction:546859:
 * Type: int64
 * Description: It is used to control the expiration time of cleaning up the 
merged rowset version. When the current time now() minus the max created 
rowset‘s create time in a version path is greater than 
tablet_rowset_stale_sweep_time_sec, the current path is cleaned up and these 
merged rowsets are deleted, the unit is second.
   - When writing is too frequent and the disk time is insufficient, you can 
configure less tablet_rowset_stale_sweep_time_sec. However, if this time is 
less than 5 minutes, it may cause fe to query the version that has been merged, 
causing a query -230 error.
-* Default value: 1800
+* Default value: 300
 
  `tablet_writer_open_rpc_timeout_sec`
 
diff --git a/docs/zh-CN/docs/admin-manual/config/be-config.md 
b/docs/zh-CN/docs/admin-manual/config/be-config.md
index ac0263491ae..4a1476a980e 100644
--- a/docs/zh-CN/docs/admin-manual/config/be-config.md
+++ b/docs/zh-CN/docs/admin-manual/config/be-config.md
@@ -1293,7 +1293,7 @@ BaseCompaction:546859:
 * 类型:int64
 * 描述:用来表示清理合并版本的过期时间,当当前时间 now() 
减去一个合并的版本路径中rowset最近创建创建时间大于tablet_rowset_stale_sweep_time_sec时,对当前路径进行清理,删除这些合并过的rowset,
 单位为s。
   - 当写入过于频繁,磁盘空间不足时,可以配置较少这个时间。不过这个时间过短小于5分钟时,可能会引发fe查询不到已经合并过的版本,引发查询-230错误。
-* 默认值:1800
+* 默认值:300
 
  `tablet_writer_open_rpc_timeout_sec`
 


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Improvements](status) catch some error status [doris]

2023-10-20 Thread via GitHub


github-actions[bot] commented on PR #25677:
URL: https://github.com/apache/doris/pull/25677#issuecomment-1772196114

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[doris] branch master updated: [Bug](samples)fix IndexOutOfBoundsException (#25608)

2023-10-20 Thread diwu
This is an automated email from the ASF dual-hosted git repository.

diwu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 42e5a3324c2 [Bug](samples)fix IndexOutOfBoundsException (#25608)
42e5a3324c2 is described below

commit 42e5a3324c2a9426df9b464c509063e98e53f195
Author: caoliang-web <71004656+caoliang-...@users.noreply.github.com>
AuthorDate: Fri Oct 20 15:04:44 2023 +0800

[Bug](samples)fix IndexOutOfBoundsException (#25608)
---
 .../src/main/java/org/apache/doris/sdk/DorisReaderExample.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/samples/doris-demo/doris-source-demo/src/main/java/org/apache/doris/sdk/DorisReaderExample.java
 
b/samples/doris-demo/doris-source-demo/src/main/java/org/apache/doris/sdk/DorisReaderExample.java
index 5c4a5c101b5..03bebef58d8 100644
--- 
a/samples/doris-demo/doris-source-demo/src/main/java/org/apache/doris/sdk/DorisReaderExample.java
+++ 
b/samples/doris-demo/doris-source-demo/src/main/java/org/apache/doris/sdk/DorisReaderExample.java
@@ -166,7 +166,7 @@ public class DorisReaderExample {
 if(!eos){
 int i = convertArrow(next, selectedColumns);
 offset += i;
-readRowCount += offset;
+readRowCount = offset;
 }
 }
 //close


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](decimal) support decimal256 [doris]

2023-10-20 Thread via GitHub


doris-robot commented on PR #25386:
URL: https://github.com/apache/doris/pull/25386#issuecomment-1772197714

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.42 seconds
stream load tsv:  552 seconds loaded 74807831229 Bytes, about 129 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  32 seconds loaded 861443392 Bytes, about 25 
MB/s
insert into select:  28.7 seconds inserted 1000 Rows, about 
348K ops/s
storage size: 17162108934 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [I] [bug] Doris2.0.2 执行固定SQL后,BE报错,甚至导致所有BE宕机。 [doris]

2023-10-20 Thread via GitHub


DA1OOO commented on issue #25644:
URL: https://github.com/apache/doris/issues/25644#issuecomment-1772211533

   Which version is OK to run this SQL? 2.0.1.1 ? Tableau will generate many 
SQL like this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[doris] branch dev_join updated (45d1d16f3e9 -> 16a2f7d0907)

2023-10-20 Thread panxiaolei
This is an automated email from the ASF dual-hosted git repository.

panxiaolei pushed a change to branch dev_join
in repository https://gitbox.apache.org/repos/asf/doris.git


omit 45d1d16f3e9 update dev_join to pre calculate bucket num (#25663)
omit dbd3454abe4 support batch size (#25629)
omit c3089031635 update rf
omit 032555667fb merge block to single block on join/set node
 add b0e0a0569a3 [Fix](row store) Real default value should be used instead 
of default… (#25230)
 add 6f6264693ff [fix](Nereids) can't choosing best plan for join that 
could only broadcast (#25511)
 add ef9cbc4c64e [enhancement](priv) Clarify ccr releated 
FrontendServiceImpl call privs (#25530)
 add 64aeeb971b5 [Fix](partial-update) Correct the alignment process when 
the table has sequence column and add cases (#25346)
 add 6cb947f72b2 [refactor](unused code) delete unused method from field.h 
(#25554)
 add d2400d1d7bf [feature](profile) profilev2 distinguish Sink and Operator 
in pipelineX (#25491)
 add db16a149704 [fix](backup) fix backup fail on s3 (#25496)
 add 76abbfc5556 [fix](mvn source) fix audit compile java-cup and 
cup-maven-plugin not found (#25564)
 add 0ec537edef4 [fix](column-id) fix null conn ctx  in column id flusher 
and parser for `database` field in  corresponding show stmt (#25393)
 add 8a8e6edba98 [enhancement](stats) Unify sample algorithm between olap 
table and external table (#25472)
 add 62d06584f1e [feature](fe) add function 'BitmapAgg' in nereids (#25508)
 add 0533911275b [Enhance](regression)change external default docker list 
(#25573)
 add c77590414ea [fix](pipeline)fix case  (#25567)
 add 80e5e722024 [fix](scanner) coredump caused by 
'prune_predicates_by_zone_map' (#2)
 add e4a83a22d14 [opt](error msg) Make data codec error clearly when load 
csv data can't display (#25540)
 add 85e83b9c6c5 [fix](auth)fix not check udf auth in nereids (#25541)
 add ea97f2d6e51 [refactor](nereids)make enable_stats session var (#25550)
 add 9e07291e65a [fix](stats) internal stats table tablet replica num not 
update as expected (#25563)
 add 2ddd2e50797 [feature](Nereids) add map_agg function (#25246)
 add 46158a6555a [tpcds-tools](nereids) remove cascades specific control 
for global config (#25595)
 add 9c9fc84f399 [feature](merge-cloud) Abstract BaseTablet for CloudTablet 
(#24929)
 add 2a442972a82 [Fix](merge-on-write) Fix some bugs about sequence column 
(#24915)
 add c21eb315b06 [feature](thrift api) support expr in MemoryScratchSink 
and make arrow::Schema recalculate with block info (#24603)
 add 32fc8a17998 [chore](compaction) Do not print the stack trace when the 
compaction task already exists (#25597)
 add 11fecafb749 [fix](move-memtable) fallback if target table contains 
inverted index (#25498)
 add dbf57876823 [fix](be) Make DorisCallOnce's function exception-safe 
(#25579)
 add 5b814e6e653 [cases](regression-test) Add create and drop S3 repository 
test (#25568)
 add 45178f5664d [improvement](disk balance) impr disk rebalancer sched 
with partition rebalancer (#25549)
 add 63c89df4741 [enhencement](RowsetWriter) Don't delete files when beta 
rowset writer destructed (#25578)
 add 8d65a627058 [Fix](show-frontends-disk)Fix NPE and macOS compatibility 
(#25565)
 add 4752b800b2d [typo](doc)update config (#25425)
 add fcf7bdc9e0e [typo](docs) Rename Import Advanced to CN Version (#25374)
 add b45f501e511 [improvement](nereids) Support aggregate functions without 
from clause (#25500)
 add ed713c2e2d6 [FIX](func) fix count distinct do not support 
arr/map/struct (#25483)
 add 3d1206d3259 [date](fix) modify push-down predicate for datev1 type 
(#25571)
 add e77b98be883 [fix](months_diff) fix wrong result of months_diff (#25577)
 add 159be51ea6a [bugfix](schema_change) Fix the coredump when doubly write 
during schema change (#22557)
 add 23535824934 [enhancement](load) support for broker load, routine load, 
mysql load and add docs (#25528)
 add d0d9c4267dd [fix](move-memtable) fix use-after-free in load stream 
stub (#25618)
 add 68eaba7220f [DOC](fix) fix hyperlink to create tpch table (#25561)
 add 4d2e7d7c86b [improvement](statistics)Set min max to NULL when collect 
stats with sample (#25593)
 add 54780c62e0d [improvement](executor)Using cgroup to implement cpu hard 
limit (#25489)
 add 7385602b190 [bug](rf) fix only min/max rf return error when has remote 
target (#25588)
 add b0e70bd02bc [fix](pipelineX) fix error set delta_urls (#25631)
 add fa5dd61563f [feature](pipelineX) operator id use negative numbers 
(#25529)
 add b964ab76b33 [refactor](shuffle) Simplify hash partitioning strategy 
(#25596)
 add 8b5fd616321 [pipelineX](profile) Add pending finish dependency timer 
(#25635)
 add a42536efc11  [feature](pipelineX) release dependency when task close 
(#25633)
 add 9ea8efe5fae [cover

[doris] 02/04: update rf

2023-10-20 Thread panxiaolei
This is an automated email from the ASF dual-hosted git repository.

panxiaolei pushed a commit to branch dev_join
in repository https://gitbox.apache.org/repos/asf/doris.git

commit d15c3f064a26199c7c66c5605e2f638daed1b622
Author: BiteThet 
AuthorDate: Wed Oct 18 14:22:02 2023 +0800

update rf
---
 be/src/exprs/runtime_filter_slots.h| 35 -
 be/src/pipeline/exec/hashjoin_build_sink.h |  2 +-
 be/src/vec/exec/join/vhash_join_node.h | 61 --
 3 files changed, 35 insertions(+), 63 deletions(-)

diff --git a/be/src/exprs/runtime_filter_slots.h 
b/be/src/exprs/runtime_filter_slots.h
index e0ff2cb0067..307253f430c 100644
--- a/be/src/exprs/runtime_filter_slots.h
+++ b/be/src/exprs/runtime_filter_slots.h
@@ -161,7 +161,7 @@ public:
 return Status::OK();
 }
 
-void insert(std::unordered_map>& datas) {
+void insert(const std::unordered_set& datas) {
 for (int i = 0; i < _build_expr_context.size(); ++i) {
 auto iter = _runtime_filters.find(i);
 if (iter == _runtime_filters.end()) {
@@ -169,30 +169,31 @@ public:
 }
 
 int result_column_id = 
_build_expr_context[i]->get_last_result_column_id();
-for (auto it : datas) {
-auto& column = 
it.first->get_by_position(result_column_id).column;
+for (const auto* it : datas) {
+auto column = it->get_by_position(result_column_id).column;
 
-if (auto* nullable =
+std::vector indexs;
+if (const auto* nullable =
 
vectorized::check_and_get_column(*column)) {
-auto& column_nested = nullable->get_nested_column_ptr();
-auto& column_nullmap = nullable->get_null_map_column_ptr();
-std::vector indexs;
-for (int row_num : it.second) {
-if (assert_cast(column_nullmap.get())
-->get_bool(row_num)) {
+column = nullable->get_nested_column_ptr();
+const uint8_t* null_map = assert_cast(
+  
nullable->get_null_map_column_ptr().get())
+  ->get_data()
+  .data();
+for (int i = 0; i < column->size(); i++) {
+if (null_map[i]) {
 continue;
 }
-indexs.push_back(row_num);
+indexs.push_back(i);
 }
-for (auto filter : iter->second) {
-filter->insert_batch(column_nested, indexs);
-}
-
 } else {
-for (auto filter : iter->second) {
-filter->insert_batch(column, it.second);
+for (int i = 0; i < column->size(); i++) {
+indexs.push_back(i);
 }
 }
+for (auto* filter : iter->second) {
+filter->insert_batch(column, indexs);
+}
 }
 }
 }
diff --git a/be/src/pipeline/exec/hashjoin_build_sink.h 
b/be/src/pipeline/exec/hashjoin_build_sink.h
index 9cf559588cc..49c1a459b70 100644
--- a/be/src/pipeline/exec/hashjoin_build_sink.h
+++ b/be/src/pipeline/exec/hashjoin_build_sink.h
@@ -101,7 +101,7 @@ protected:
 bool _has_set_need_null_map_for_build = false;
 bool _build_side_ignore_null = false;
 size_t _build_rf_cardinality = 0;
-std::unordered_map> 
_inserted_rows;
+std::unordered_set _inserted_blocks;
 std::shared_ptr _shared_hash_table_dependency;
 
 RuntimeProfile::Counter* _build_table_timer;
diff --git a/be/src/vec/exec/join/vhash_join_node.h 
b/be/src/vec/exec/join/vhash_join_node.h
index ef5a61eae17..c0d964fd66c 100644
--- a/be/src/vec/exec/join/vhash_join_node.h
+++ b/be/src/vec/exec/join/vhash_join_node.h
@@ -86,10 +86,10 @@ struct ProcessRuntimeFilterBuild {
 RETURN_IF_ERROR(parent->_runtime_filter_slots->init(
 state, hash_table_ctx.hash_table->size(), 
parent->_build_rf_cardinality));
 
-if (!parent->_runtime_filter_slots->empty() && 
!parent->_inserted_rows.empty()) {
+if (!parent->_runtime_filter_slots->empty() && 
!parent->_inserted_blocks.empty()) {
 {
 SCOPED_TIMER(parent->_push_compute_timer);
-parent->_runtime_filter_slots->insert(parent->_inserted_rows);
+
parent->_runtime_filter_slots->insert(parent->_inserted_blocks);
 }
 }
 {
@@ -117,54 +117,25 @@ struct ProcessHashTableBuild {
 
 template 
 Status run(HashTableContext& hash_table_ctx, ConstNullMapPtr null_map, 
bool* has_null_key) {
-using KeyGetter

[doris] 03/04: support batch size (#25629)

2023-10-20 Thread panxiaolei
This is an automated email from the ASF dual-hosted git repository.

panxiaolei pushed a commit to branch dev_join
in repository https://gitbox.apache.org/repos/asf/doris.git

commit a0d727cccac81ec4761e4de9d63913839979a7b2
Author: HappenLee 
AuthorDate: Fri Oct 20 10:33:31 2023 +0800

support batch size (#25629)

* support batch size

* support left anti/semi/outer join
---
 be/src/olap/delta_writer.cpp   |  2 +-
 be/src/olap/delta_writer.h |  2 +-
 be/src/olap/delta_writer_v2.cpp|  2 +-
 be/src/olap/delta_writer_v2.h  |  2 +-
 be/src/olap/memtable.cpp   |  4 +-
 be/src/olap/memtable.h |  2 +-
 be/src/olap/memtable_writer.cpp|  2 +-
 be/src/olap/memtable_writer.h  |  2 +-
 be/src/olap/tablet.cpp |  2 +-
 be/src/pipeline/exec/exchange_sink_operator.cpp|  2 +-
 .../exec/nested_loop_join_probe_operator.cpp   |  2 +-
 be/src/runtime/tablets_channel.cpp |  7 +-
 be/src/vec/columns/column.h|  6 +-
 be/src/vec/columns/column_array.cpp| 10 +--
 be/src/vec/columns/column_array.h  |  4 +-
 be/src/vec/columns/column_complex.h| 10 +--
 be/src/vec/columns/column_const.h  |  4 +-
 be/src/vec/columns/column_decimal.h|  6 +-
 be/src/vec/columns/column_dictionary.h |  4 +-
 be/src/vec/columns/column_fixed_length_object.h| 12 +--
 be/src/vec/columns/column_map.cpp  | 10 +--
 be/src/vec/columns/column_map.h|  4 +-
 be/src/vec/columns/column_nullable.cpp |  5 +-
 be/src/vec/columns/column_nullable.h   |  4 +-
 be/src/vec/columns/column_object.cpp   | 15 ++--
 be/src/vec/columns/column_object.h |  8 +-
 be/src/vec/columns/column_string.cpp   | 28 +++
 be/src/vec/columns/column_string.h |  4 +-
 be/src/vec/columns/column_struct.cpp   |  4 +-
 be/src/vec/columns/column_struct.h |  4 +-
 be/src/vec/columns/column_vector.cpp   | 13 ++--
 be/src/vec/columns/column_vector.h |  4 +-
 be/src/vec/columns/predicate_column.h  |  4 +-
 be/src/vec/common/hash_table/hash_map.h| 91 --
 be/src/vec/core/block.cpp  |  3 +-
 be/src/vec/core/block.h|  2 +-
 be/src/vec/exec/join/process_hash_table_probe.h|  2 +-
 .../vec/exec/join/process_hash_table_probe_impl.h  |  9 ++-
 be/src/vec/exec/join/vhash_join_node.h |  2 +-
 be/src/vec/exec/join/vnested_loop_join_node.cpp|  2 +-
 be/src/vec/exec/scan/pip_scanner_context.h |  6 +-
 be/src/vec/sink/vdata_stream_sender.cpp|  6 +-
 be/src/vec/sink/vdata_stream_sender.h  |  8 +-
 be/src/vec/sink/vtablet_sink_v2.h  |  2 +-
 44 files changed, 197 insertions(+), 130 deletions(-)

diff --git a/be/src/olap/delta_writer.cpp b/be/src/olap/delta_writer.cpp
index 23e1718cb7d..4098f512120 100644
--- a/be/src/olap/delta_writer.cpp
+++ b/be/src/olap/delta_writer.cpp
@@ -108,7 +108,7 @@ Status DeltaWriter::append(const vectorized::Block* block) {
 return write(block, {}, true);
 }
 
-Status DeltaWriter::write(const vectorized::Block* block, const 
std::vector& row_idxs,
+Status DeltaWriter::write(const vectorized::Block* block, const 
std::vector& row_idxs,
   bool is_append) {
 if (UNLIKELY(row_idxs.empty() && !is_append)) {
 return Status::OK();
diff --git a/be/src/olap/delta_writer.h b/be/src/olap/delta_writer.h
index 303f17f14f8..4b2db0c9cc3 100644
--- a/be/src/olap/delta_writer.h
+++ b/be/src/olap/delta_writer.h
@@ -67,7 +67,7 @@ public:
 
 Status init();
 
-Status write(const vectorized::Block* block, const std::vector& 
row_idxs,
+Status write(const vectorized::Block* block, const std::vector& 
row_idxs,
  bool is_append = false);
 
 Status append(const vectorized::Block* block);
diff --git a/be/src/olap/delta_writer_v2.cpp b/be/src/olap/delta_writer_v2.cpp
index ef3ff23f9d8..47723f26d72 100644
--- a/be/src/olap/delta_writer_v2.cpp
+++ b/be/src/olap/delta_writer_v2.cpp
@@ -138,7 +138,7 @@ Status DeltaWriterV2::append(const vectorized::Block* 
block) {
 return write(block, {}, true);
 }
 
-Status DeltaWriterV2::write(const vectorized::Block* block, const 
std::vector& row_idxs,
+Status DeltaWriterV2::write(const vectorized::Block* block, const 
std::vector& row_idxs,
 bool is_append) {
 if (UNLIKELY(row_idxs.empty() && !is_append)) {
 return Status::OK();
diff --git a/be/src/olap/delta_writer_v2.h b/be/src/olap/delta_writer_v2.h
index b2b1f5f1c19..8f92ec74273 100644
--- a/be/src/olap/delta_writer_v2.h
+++

[doris] 04/04: update dev_join to pre calculate bucket num (#25663)

2023-10-20 Thread panxiaolei
This is an automated email from the ASF dual-hosted git repository.

panxiaolei pushed a commit to branch dev_join
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 16a2f7d090731d7fd6ebf0d695fe933c153d41d5
Author: Pxl 
AuthorDate: Fri Oct 20 11:50:08 2023 +0800

update dev_join to pre calculate bucket num (#25663)
---
 be/src/vec/common/hash_table/hash_map.h| 51 +++--
 be/src/vec/common/hash_table/hash_map_context.h|  8 
 be/src/vec/exec/join/process_hash_table_probe.h|  6 ---
 .../vec/exec/join/process_hash_table_probe_impl.h  | 53 ++
 be/src/vec/exec/join/vhash_join_node.h |  2 +
 5 files changed, 39 insertions(+), 81 deletions(-)

diff --git a/be/src/vec/common/hash_table/hash_map.h 
b/be/src/vec/common/hash_table/hash_map.h
index 53cb01dbfaa..89dfe7f8aac 100644
--- a/be/src/vec/common/hash_table/hash_map.h
+++ b/be/src/vec/common/hash_table/hash_map.h
@@ -216,18 +216,23 @@ public:
 return phmap::priv::NormalizeCapacity(expect_bucket_size) + 1;
 }
 
+void reserve(int num_elem) {
+bucket_size = calc_bucket_size(num_elem + 1);
+first.resize(bucket_size, 0);
+next.resize(num_elem);
+}
+
 void build(const Key* __restrict keys, const size_t* __restrict 
hash_values, size_t num_elem,
int batch_size) {
-max_batch_size = batch_size;
-bucket_size = calc_bucket_size(num_elem + 1);
+_batch_size = batch_size;
+bucket_size = calc_bucket_size(num_elem);
 first.resize(bucket_size, 0);
 next.resize(num_elem);
 
 build_keys = keys;
 for (size_t i = 1; i < num_elem; i++) {
-uint32_t bucket_num = hash_values[i] & (bucket_size - 1);
-next[i] = first[bucket_num];
-first[bucket_num] = i;
+next[i] = first[hash_values[i]];
+first[hash_values[i]] = i;
 }
 }
 
@@ -248,18 +253,16 @@ public:
 return std::pair {0, 0};
 }
 
+size_t get_bucket_mask() { return bucket_size - 1; }
+
 private:
 template 
 auto _find_batch_left_semi_anti(const Key* __restrict keys,
 const size_t* __restrict hash_values, int 
probe_idx,
 int probe_rows, std::vector& 
probe_idxs) {
-auto matched_cnt = 0;
-const auto batch_size = max_batch_size;
-
-while (LIKELY(probe_idx < probe_rows && matched_cnt < batch_size)) {
-uint32_t bucket_num = hash_values[probe_idx] & (bucket_size - 1);
-auto build_idx = first[bucket_num];
-
+int matched_cnt = 0;
+while (LIKELY(probe_idx < probe_rows && matched_cnt < _batch_size)) {
+uint32_t build_idx = first[hash_values[probe_idx]];
 while (build_idx) {
 if (keys[probe_idx] == build_keys[build_idx]) {
 break;
@@ -279,12 +282,11 @@ private:
   const size_t* __restrict hash_values, 
int probe_idx,
   int probe_rows, std::vector& 
probe_idxs,
   std::vector& build_idxs) {
-auto matched_cnt = 0;
-const auto batch_size = max_batch_size;
+int matched_cnt = 0;
 uint32_t build_idx = 0;
 
 auto do_the_probe = [&]() {
-while (build_idx && LIKELY(matched_cnt < batch_size)) {
+while (build_idx && LIKELY(matched_cnt < _batch_size)) {
 if (keys[probe_idx] == build_keys[build_idx]) {
 probe_idxs[matched_cnt] = probe_idx;
 build_idxs[matched_cnt] = build_idx;
@@ -302,12 +304,7 @@ private:
 }
 }
 
-if (matched_cnt == max_batch_size && build_idx) {
-current_probe_idx = probe_idx;
-current_build_idx = build_idx;
-} else {
-probe_idx++;
-}
+probe_idx++;
 };
 
 // some row over the batch_size, need dispose first
@@ -317,17 +314,21 @@ private:
 current_build_idx = 0;
 do_the_probe();
 }
-while (LIKELY(probe_idx < probe_rows && matched_cnt < batch_size)) {
-uint32_t bucket_num = hash_values[probe_idx] & (bucket_size - 1);
-build_idx = first[bucket_num];
+while (LIKELY(probe_idx < probe_rows && matched_cnt < _batch_size)) {
+build_idx = first[hash_values[probe_idx]];
 do_the_probe();
 }
+
+if (matched_cnt == _batch_size && build_idx) {
+current_probe_idx = probe_idx - 1;
+current_build_idx = build_idx;
+}
 return std::pair {probe_idx, matched_cnt};
 }
 
 const Key* __restrict build_keys;
 uint32_t bucket_size = 0;
-int max_batch_size = 0;
+int _batch_size = 0;
 
 int current_probe_idx = -1;
 uint32_t current_b

[doris] 01/04: merge block to single block on join/set node

2023-10-20 Thread panxiaolei
This is an automated email from the ASF dual-hosted git repository.

panxiaolei pushed a commit to branch dev_join
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 2e0778a3d87f7a0af33dd11ed3e914ddfd922ed6
Author: BiteThet 
AuthorDate: Mon Oct 16 18:43:26 2023 +0800

merge block to single block on join/set node

update

update

update
---
 be/src/pipeline/exec/hashjoin_build_sink.cpp   | 101 ++--
 be/src/pipeline/exec/hashjoin_build_sink.h |   5 +-
 be/src/pipeline/exec/hashjoin_probe_operator.h |   4 +-
 be/src/pipeline/exec/set_sink_operator.cpp |  13 +-
 be/src/pipeline/exec/set_sink_operator.h   |   2 +-
 be/src/pipeline/exec/set_source_operator.cpp   |   6 +-
 be/src/pipeline/pipeline_x/dependency.h|   7 +-
 be/src/vec/common/hash_table/hash_map.h|  67 ++
 be/src/vec/common/hash_table/hash_map_context.h|   6 +-
 .../vec/common/hash_table/hash_table_set_build.h   |   9 +-
 be/src/vec/exec/join/join_op.h |  72 +++---
 be/src/vec/exec/join/process_hash_table_probe.h|   9 +-
 .../vec/exec/join/process_hash_table_probe_impl.h  | 257 +++--
 be/src/vec/exec/join/vhash_join_node.cpp   |  72 ++
 be/src/vec/exec/join/vhash_join_node.h | 100 ++--
 be/src/vec/exec/vset_operation_node.cpp|  69 +++---
 be/src/vec/exec/vset_operation_node.h  |   5 +-
 be/src/vec/runtime/shared_hash_table_controller.h  |  11 +-
 18 files changed, 256 insertions(+), 559 deletions(-)

diff --git a/be/src/pipeline/exec/hashjoin_build_sink.cpp 
b/be/src/pipeline/exec/hashjoin_build_sink.cpp
index 153882075b6..e7da32b0340 100644
--- a/be/src/pipeline/exec/hashjoin_build_sink.cpp
+++ b/be/src/pipeline/exec/hashjoin_build_sink.cpp
@@ -39,10 +39,7 @@ Overload(Callables&&... callables) -> Overload;
 
 
HashJoinBuildSinkLocalState::HashJoinBuildSinkLocalState(DataSinkOperatorXBase* 
parent,
  RuntimeState* state)
-: JoinBuildSinkLocalState(parent, state),
-  _build_block_idx(0),
-  _build_side_mem_used(0),
-  _build_side_last_mem_used(0) {}
+: JoinBuildSinkLocalState(parent, state) {}
 
 Status HashJoinBuildSinkLocalState::init(RuntimeState* state, 
LocalSinkStateInfo& info) {
 RETURN_IF_ERROR(JoinBuildSinkLocalState::init(state, info));
@@ -52,13 +49,7 @@ Status HashJoinBuildSinkLocalState::init(RuntimeState* 
state, LocalSinkStateInfo
 auto& p = _parent->cast();
 _shared_state->join_op_variants = p._join_op_variants;
 if (p._is_broadcast_join && 
state->enable_share_hash_table_for_broadcast_join()) {
-_shared_state->build_blocks = p._shared_hash_table_context->blocks;
-} else {
-_shared_state->build_blocks.reset(new 
std::vector());
-// avoid vector expand change block address.
-// one block can store 4g data, _build_blocks can store 128*4g data.
-// if probe data bigger than 512g, runtime filter maybe will core dump 
when insert data.
-
_shared_state->build_blocks->reserve(vectorized::HASH_JOIN_MAX_BUILD_BLOCK_COUNT);
+_shared_state->build_block = p._shared_hash_table_context->block;
 }
 _shared_state->is_null_safe_eq_join = p._is_null_safe_eq_join;
 _shared_state->store_null_in_hash_table = p._store_null_in_hash_table;
@@ -82,11 +73,6 @@ Status HashJoinBuildSinkLocalState::init(RuntimeState* 
state, LocalSinkStateInfo
 if (!_should_build_hash_table) {
 _shared_hash_table_dependency->block_writing();
 p._shared_hashtable_controller->append_dependency(p.id(), 
_shared_hash_table_dependency);
-} else if (p._is_broadcast_join) {
-// avoid vector expand change block address.
-// one block can store 4g data, _build_blocks can store 128*4g data.
-// if probe data bigger than 512g, runtime filter maybe will core dump 
when insert data.
-
_shared_state->build_blocks->reserve(vectorized::HASH_JOIN_MAX_BUILD_BLOCK_COUNT);
 }
 
 _memory_usage_counter = ADD_LABEL_COUNTER(profile(), "MemoryUsage");
@@ -156,25 +142,24 @@ void 
HashJoinBuildSinkLocalState::init_short_circuit_for_probe() {
 _shared_state->short_circuit_for_probe =
 (_shared_state->_has_null_in_build_side &&
  p._join_op == TJoinOp::NULL_AWARE_LEFT_ANTI_JOIN && 
!p._is_mark_join) ||
-(_shared_state->build_blocks->empty() && p._join_op == 
TJoinOp::INNER_JOIN &&
+(!_shared_state->build_block && p._join_op == TJoinOp::INNER_JOIN 
&&
  !p._is_mark_join) ||
-(_shared_state->build_blocks->empty() && p._join_op == 
TJoinOp::LEFT_SEMI_JOIN &&
+(!_shared_state->build_block && p._join_op == 
TJoinOp::LEFT_SEMI_JOIN &&
  !p._is_mark_join) ||
-(_shared_state->build_blocks->empty() && p._join_op == 
TJoinOp::RIGHT_OUTER_JOIN) ||
- 

Re: [PR] [improvement](insert) Modify insert into table_id(xxx) to doris_inter… [doris]

2023-10-20 Thread via GitHub


doris-robot commented on PR #25662:
URL: https://github.com/apache/doris/pull/25662#issuecomment-1772217857

   TeamCity be ut coverage result:
Function Coverage: 37.04% (8299/22408) 
Line Coverage: 29.16% (9/228594)
Region Coverage: 27.81% (34596/124407)
Branch Coverage: 24.41% (17577/72018)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/32f2b9e86808aa9e455b82d700e24186c3c33e35_32f2b9e86808aa9e455b82d700e24186c3c33e35/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [FIX](resize) fix array and map offsets resize with default value [doris]

2023-10-20 Thread via GitHub


doris-robot commented on PR #25669:
URL: https://github.com/apache/doris/pull/25669#issuecomment-1772217514

   TeamCity be ut coverage result:
Function Coverage: 37.08% (8309/22409) 
Line Coverage: 29.20% (66744/228598)
Region Coverage: 27.84% (34639/124407)
Branch Coverage: 24.43% (17591/72018)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/e4709fd4bff8637b4422595677e2b236a8ae9d05_e4709fd4bff8637b4422595677e2b236a8ae9d05/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](function) support ip function ipv4stringtonum(ordefault, ornull), inet_aton [doris]

2023-10-20 Thread via GitHub


superdiaodiao commented on code in PR #25510:
URL: https://github.com/apache/doris/pull/25510#discussion_r1365776957


##
gensrc/script/doris_builtins_functions.py:
##
@@ -2004,6 +2004,12 @@
 [['ipv4numtostring','inet_ntoa'], 'VARCHAR', ['SMALLINT'], 
'ALWAYS_NULLABLE'],
 [['ipv4numtostring','inet_ntoa'], 'VARCHAR', ['INT'], 
'ALWAYS_NULLABLE'],
 [['ipv4numtostring','inet_ntoa'], 'VARCHAR', ['BIGINT'], 
'ALWAYS_NULLABLE'],
+[['ipv4stringtonum','inet_aton'], 'BIGINT', ['VARCHAR'], 
'ALWAYS_NOT_NULLABLE'],

Review Comment:
   Yes, but this BE part is running so much time, so I have a plan to make the 
Nereids in another PR. 
   Is it alright?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](function) support ip function ipv4stringtonum(ordefault, ornull), inet_aton [doris]

2023-10-20 Thread via GitHub


superdiaodiao commented on code in PR #25510:
URL: https://github.com/apache/doris/pull/25510#discussion_r1365776957


##
gensrc/script/doris_builtins_functions.py:
##
@@ -2004,6 +2004,12 @@
 [['ipv4numtostring','inet_ntoa'], 'VARCHAR', ['SMALLINT'], 
'ALWAYS_NULLABLE'],
 [['ipv4numtostring','inet_ntoa'], 'VARCHAR', ['INT'], 
'ALWAYS_NULLABLE'],
 [['ipv4numtostring','inet_ntoa'], 'VARCHAR', ['BIGINT'], 
'ALWAYS_NULLABLE'],
+[['ipv4stringtonum','inet_aton'], 'BIGINT', ['VARCHAR'], 
'ALWAYS_NOT_NULLABLE'],

Review Comment:
   Yes, but this BE part runs so much time, so I have a plan to make the 
Nereids in another PR. 
   Is it alright?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](ut) fix some uts [doris]

2023-10-20 Thread via GitHub


hello-stephen commented on PR #25674:
URL: https://github.com/apache/doris/pull/25674#issuecomment-1772224944

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.68 seconds
stream load tsv:  576 seconds loaded 74807831229 Bytes, about 123 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  32 seconds loaded 861443392 Bytes, about 25 
MB/s
insert into select:  29.6 seconds inserted 1000 Rows, about 
337K ops/s
storage size: 17162848744 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[PR] [fix](auth)Fix no priv to insert to mysql catalog [doris]

2023-10-20 Thread via GitHub


zddr opened a new pull request, #25678:
URL: https://github.com/apache/doris/pull/25678

   ## Proposed changes
   
   Issue Number: close #xxx
   
   fix user can not insert to mysql catalog even if he has load permission
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](nereids)do not generate min-max runtime filter for A<=>B and A!=B [doris]

2023-10-20 Thread via GitHub


hello-stephen commented on PR #25676:
URL: https://github.com/apache/doris/pull/25676#issuecomment-1772226533

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Improvements](status) catch some error status [doris]

2023-10-20 Thread via GitHub


doris-robot commented on PR #25677:
URL: https://github.com/apache/doris/pull/25677#issuecomment-1772228576

   TeamCity be ut coverage result:
Function Coverage: 37.04% (8301/22409) 
Line Coverage: 29.17% (66693/228605)
Region Coverage: 27.80% (34640/124591)
Branch Coverage: 24.41% (17585/72028)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/4926afbe7c75ccf2c71540669d37d6c353e80032_4926afbe7c75ccf2c71540669d37d6c353e80032/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[PR] [fix](regression-test) fix bad case in s3 tvf load test [doris]

2023-10-20 Thread via GitHub


TangSiyang2001 opened a new pull request, #25679:
URL: https://github.com/apache/doris/pull/25679

   ## Proposed changes
   
   Issue Number: close #xxx
   
   
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](regression-test) fix bad case in s3 tvf load test [doris]

2023-10-20 Thread via GitHub


TangSiyang2001 commented on PR #25679:
URL: https://github.com/apache/doris/pull/25679#issuecomment-1772228802

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [FIX](resize) fix array and map offsets resize with default value [doris]

2023-10-20 Thread via GitHub


xiaokang commented on code in PR #25669:
URL: https://github.com/apache/doris/pull/25669#discussion_r1366585254


##
be/src/vec/columns/column_array.cpp:
##
@@ -422,7 +422,8 @@ void ColumnArray::reserve(size_t n) {
 
 //please check you real need size in data column, because it's maybe need 
greater size when data is string column
 void ColumnArray::resize(size_t n) {
-get_offsets().resize(n);
+auto last_off = get_offsets().back();
+get_offsets().resize_fill(n, last_off);

Review Comment:
   We need to check if resize_fill will modify old offsets value.



##
be/src/vec/columns/column_array.cpp:
##
@@ -422,7 +422,8 @@ void ColumnArray::reserve(size_t n) {
 
 //please check you real need size in data column, because it's maybe need 
greater size when data is string column
 void ColumnArray::resize(size_t n) {

Review Comment:
   Is there the same problem for ColumnStruct?



##
be/src/vec/columns/column_map.cpp:
##
@@ -450,7 +450,8 @@ void ColumnMap::reserve(size_t n) {
 }
 
 void ColumnMap::resize(size_t n) {
-get_offsets().resize(n);
+auto last_off = get_offsets().back();
+get_offsets().resize_fill(n, last_off);
 keys_column->resize(n);

Review Comment:
   should not call resize on keys and values column



##
be/src/vec/columns/column_array.cpp:
##
@@ -422,7 +422,8 @@ void ColumnArray::reserve(size_t n) {
 
 //please check you real need size in data column, because it's maybe need 
greater size when data is string column
 void ColumnArray::resize(size_t n) {
-get_offsets().resize(n);
+auto last_off = get_offsets().back();
+get_offsets().resize_fill(n, last_off);
 get_data().resize(n);

Review Comment:
   I think it should not call get_data().resize(n). Just append offsets with 
the same value.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] test [doris]

2023-10-20 Thread via GitHub


github-actions[bot] commented on code in PR #25493:
URL: https://github.com/apache/doris/pull/25493#discussion_r1366588162


##
be/src/vec/columns/column_struct.cpp:
##
@@ -225,8 +225,8 @@ void ColumnStruct::update_crcs_with_value(uint32_t* 
__restrict hash, PrimitiveTy
 }
 }
 
-void ColumnStruct::insert_indices_from(const IColumn& src, const int* 
indices_begin,
-   const int* indices_end) {
+void ColumnStruct::insert_indices_from(const IColumn& src, const uint32_t* 
__restrict indices_begin,

Review Comment:
   warning: method 'insert_indices_from' can be made static 
[readability-convert-member-functions-to-static]
   
   be/src/vec/columns/column_struct.h:123:
   ```diff
   - void insert_indices_from(const IColumn& src, const uint32* __restrict 
indices_begin,
   + static void insert_indices_from(const IColumn& src, const uint32* 
__restrict indices_begin,
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix][branch-2.0](regression-test) fix bad case in s3 tvf load test [doris]

2023-10-20 Thread via GitHub


TangSiyang2001 commented on PR #25680:
URL: https://github.com/apache/doris/pull/25680#issuecomment-1772233211

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[doris-thirdparty] branch clucene updated: [Fix](PFOR) fix PFOR 'illegal operand' error for none SSE4.2/AVX cpu (#129)

2023-10-20 Thread kxiao
This is an automated email from the ASF dual-hosted git repository.

kxiao pushed a commit to branch clucene
in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git


The following commit(s) were added to refs/heads/clucene by this push:
 new dd2a9c92 [Fix](PFOR) fix PFOR 'illegal operand' error for none 
SSE4.2/AVX cpu (#129)
dd2a9c92 is described below

commit dd2a9c929275a0091a59dd7acd2cdf8ad7419da6
Author: airborne12 
AuthorDate: Fri Oct 20 15:35:10 2023 +0800

[Fix](PFOR) fix PFOR 'illegal operand' error for none SSE4.2/AVX cpu (#129)
---
 src/core/CLucene/util/PFORUtil.cpp | 64 +++---
 1 file changed, 60 insertions(+), 4 deletions(-)

diff --git a/src/core/CLucene/util/PFORUtil.cpp 
b/src/core/CLucene/util/PFORUtil.cpp
index a9a32dfb..e290ea5b 100644
--- a/src/core/CLucene/util/PFORUtil.cpp
+++ b/src/core/CLucene/util/PFORUtil.cpp
@@ -29,6 +29,61 @@ ENC_FUNC g_p4nd1enc;
 ENC_FUNC g_p4nzenc;
 } // anonymous namespace
 
+size_t DefaultDEC(unsigned char *__restrict in, size_t n, uint32_t *__restrict 
out) {
+size_t bufferSize = 0;
+for (uint32_t i = 0; i < n; i++) {
+uint8_t b = in[bufferSize++];
+int32_t docCode = b & 0x7F;
+for (int32_t shift = 7; (b & 0x80) != 0; shift += 7) {
+b = in[bufferSize++];
+docCode |= (b & 0x7F) << shift;
+}
+out[i] = docCode;
+}
+}
+
+size_t DefaultDDEC(unsigned char *__restrict in, size_t n, uint32_t 
*__restrict out) {
+uint32_t docDelta = 0;
+size_t bufferSize = 0;
+for (uint32_t i = 0; i < n; i++) {
+uint8_t b = in[bufferSize++];
+int32_t docCode = b & 0x7F;
+for (int32_t shift = 7; (b & 0x80) != 0; shift += 7) {
+b = in[bufferSize++];
+docCode |= (b & 0x7F) << shift;
+}
+docDelta += (docCode >> 1);
+out[i] = docDelta;
+}
+}
+
+size_t DefaultDENC(uint32_t *__restrict in, size_t n, unsigned char 
*__restrict out) {
+int outIndex = 0;
+uint32_t lastDoc = 0;
+for (int32_t i = 0; i < n; i++) {
+uint32_t curDoc = in[i];
+uint32_t delta = curDoc - lastDoc;
+while ((delta & ~0x7F) != 0) {
+out[outIndex++] = (uint8_t)((delta & 0x7f) | 0x80);
+delta >>= 7; //doing unsigned shift
+}
+out[outIndex++] = (uint8_t)delta;
+lastDoc = curDoc;
+}
+}
+
+size_t DefaultENC(uint32_t *__restrict in, size_t n, unsigned char *__restrict 
out) {
+int outIndex = 0;
+for (int32_t i = 0; i < n; i++) {
+uint32_t curDoc = in[i];
+while ((curDoc & ~0x7F) != 0) {
+out[outIndex++] = (uint8_t)((curDoc & 0x7f) | 0x80);
+curDoc >>= 7; //doing unsigned shift
+}
+out[outIndex++] = (uint8_t)curDoc;
+}
+}
+
 __attribute__((constructor)) void SelectPFORFunctions() {
 #if (defined(__i386) || defined(__x86_64__))
 uint32_t eax, ebx, ecx, edx;
@@ -48,10 +103,10 @@ __attribute__((constructor)) void SelectPFORFunctions() {
 g_p4nd1enc = p4nd1enc128v32;
 g_p4nzenc = p4nzenc128v32;
 } else {
-g_p4nd1dec = p4nd1dec32;
-g_p4nzdec = p4nzdec32;
-g_p4nd1enc = p4nd1enc32;
-g_p4nzenc = p4nzenc32;
+g_p4nd1dec = DefaultDDEC;
+g_p4nzdec = DefaultDEC;
+g_p4nd1enc = DefaultDENC;
+g_p4nzenc = DefaultENC;
 }
 #endif
 #else
@@ -65,6 +120,7 @@ __attribute__((constructor)) void SelectPFORFunctions() {
 size_t P4DEC(unsigned char *__restrict in, size_t n, uint32_t *__restrict out) 
{
 return g_p4nd1dec(in, n, out);
 }
+
 size_t P4NZDEC(unsigned char *__restrict in, size_t n, uint32_t *__restrict 
out) {
 return g_p4nzdec(in, n, out);
 }


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[PR] [fix][branch-2.0](regression-test) fix bad case in s3 tvf load test [doris]

2023-10-20 Thread via GitHub


TangSiyang2001 opened a new pull request, #25680:
URL: https://github.com/apache/doris/pull/25680

   ## Proposed changes
   
   Issue Number: close #xxx
   
   
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Fix](PFOR) fix PFOR 'illegal operand' error for none SSE4.2/AVX cpu [doris-thirdparty]

2023-10-20 Thread via GitHub


xiaokang merged PR #129:
URL: https://github.com/apache/doris-thirdparty/pull/129


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [I] [Bug] can not compile on macos [doris]

2023-10-20 Thread via GitHub


felixwluo commented on issue #24825:
URL: https://github.com/apache/doris/issues/24825#issuecomment-1772233643

   To update xcode, run the "softwareupdate --install -a" command


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](nereids)fix bug of duplicate name of inline view [doris]

2023-10-20 Thread via GitHub


starocean999 merged PR #25627:
URL: https://github.com/apache/doris/pull/25627


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[doris] branch master updated: [fix](nereids)fix bug of duplicate name of inline view (#25627)

2023-10-20 Thread starocean999
This is an automated email from the ASF dual-hosted git repository.

starocean999 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new a11cde7beec [fix](nereids)fix bug of duplicate name of inline view 
(#25627)
a11cde7beec is described below

commit a11cde7beec413b9760cdf9130dd93ff25e36a5a
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Fri Oct 20 15:37:44 2023 +0800

[fix](nereids)fix bug of duplicate name of inline view (#25627)
---
 .../apache/doris/nereids/analyzer/UnboundSlot.java |   5 +
 .../nereids/rules/analysis/BindExpression.java |   4 +-
 .../doris/nereids/trees/expressions/Alias.java |   4 +-
 .../trees/expressions/ArrayItemReference.java  |   3 +-
 .../doris/nereids/trees/expressions/Slot.java  |   4 +
 .../nereids/trees/expressions/SlotReference.java   |  41 +---
 .../subquery/test_duplicate_name_in_view.groovy| 108 -
 7 files changed, 153 insertions(+), 16 deletions(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundSlot.java 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundSlot.java
index 2c8b03e647e..8fa85966dc0 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundSlot.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundSlot.java
@@ -63,6 +63,11 @@ public class UnboundSlot extends Slot implements Unbound, 
PropagateNullable {
 return nameParts.subList(0, nameParts.size() - 1);
 }
 
+@Override
+public String getInternalName() {
+return getName();
+}
+
 @Override
 public String toSql() {
 return nameParts.stream().map(Utils::quoteIfNeeded).reduce((left, 
right) -> left + "." + right).orElse("");
diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java
 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java
index d09432029c7..4c51578ef92 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java
@@ -709,11 +709,11 @@ public class BindExpression implements 
AnalysisRuleFactory {
 private void checkSameNameSlot(List childOutputs, String 
subQueryAlias) {
 Set nameSlots = new HashSet<>();
 for (Slot s : childOutputs) {
-if (nameSlots.contains(s.getName())) {
+if (nameSlots.contains(s.getInternalName())) {
 throw new AnalysisException("Duplicated inline view column 
alias: '" + s.getName()
 + "'" + " in inline view: '" + subQueryAlias + "'");
 } else {
-nameSlots.add(s.getName());
+nameSlots.add(s.getInternalName());
 }
 }
 }
diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Alias.java
 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Alias.java
index 0e91dec26c3..b02c968baba 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Alias.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Alias.java
@@ -27,6 +27,7 @@ import com.google.common.collect.ImmutableList;
 
 import java.util.List;
 import java.util.Objects;
+import java.util.Optional;
 
 /**
  * Expression for alias, such as col1 as c1.
@@ -73,7 +74,8 @@ public class Alias extends NamedExpression implements 
UnaryExpression {
 return new SlotReference(exprId, name, child().getDataType(), 
child().nullable(), qualifier,
 child() instanceof SlotReference
 ? ((SlotReference) child()).getColumn().orElse(null)
-: null);
+: null,
+nameFromChild ? Optional.of(child().toString()) : 
Optional.of(name));
 }
 
 @Override
diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/ArrayItemReference.java
 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/ArrayItemReference.java
index afebaa3016d..95c29eac87a 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/ArrayItemReference.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/ArrayItemReference.java
@@ -28,6 +28,7 @@ import com.google.common.collect.ImmutableList;
 
 import java.util.List;
 import java.util.Objects;
+import java.util.Optional;
 
 /**
  * it is item from array, which used in lambda function
@@ -138,7 +139,7 @@ public class ArrayItemReference extends NamedExpression 
implements ExpectsInputT
  * @param nullable true if nullable
  */
 public ArrayItemSlot(ExprId exprId, String name, DataType dataType, 
boolean nullable) {
-  

Re: [PR] [improvement](insert) Modify insert into table_id(xxx) to doris_inter… [doris]

2023-10-20 Thread via GitHub


doris-robot commented on PR #25662:
URL: https://github.com/apache/doris/pull/25662#issuecomment-1772244041

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.27 seconds
stream load tsv:  551 seconds loaded 74807831229 Bytes, about 129 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  31 seconds loaded 861443392 Bytes, about 26 
MB/s
insert into select:  29.1 seconds inserted 1000 Rows, about 
343K ops/s
storage size: 17162064231 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [improvement](outfile)support underscore prefix when select outfile [doris]

2023-10-20 Thread via GitHub


doris-robot commented on PR #25395:
URL: https://github.com/apache/doris/pull/25395#issuecomment-1772244323

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.27 seconds
stream load tsv:  565 seconds loaded 74807831229 Bytes, about 126 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  32 seconds loaded 861443392 Bytes, about 25 
MB/s
insert into select:  28.9 seconds inserted 1000 Rows, about 
346K ops/s
storage size: 17162544885 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [pipeline](load) disable pipeline load [doris]

2023-10-20 Thread via GitHub


doris-robot commented on PR #25667:
URL: https://github.com/apache/doris/pull/25667#issuecomment-1772244629

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.34 seconds
stream load tsv:  573 seconds loaded 74807831229 Bytes, about 124 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  32 seconds loaded 861443392 Bytes, about 25 
MB/s
insert into select:  28.9 seconds inserted 1000 Rows, about 
346K ops/s
storage size: 17162364730 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [FIX](resize) fix array and map offsets resize with default value [doris]

2023-10-20 Thread via GitHub


doris-robot commented on PR #25669:
URL: https://github.com/apache/doris/pull/25669#issuecomment-1772244414

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.54 seconds
stream load tsv:  579 seconds loaded 74807831229 Bytes, about 123 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  31 seconds loaded 861443392 Bytes, about 26 
MB/s
insert into select:  28.6 seconds inserted 1000 Rows, about 
349K ops/s
storage size: 17162350704 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Fix](create-table-like)Fix create table like error, the converted table field COMMENT contains extra characters [doris]

2023-10-20 Thread via GitHub


doris-robot commented on PR #25587:
URL: https://github.com/apache/doris/pull/25587#issuecomment-1772245047

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.54 seconds
stream load tsv:  556 seconds loaded 74807831229 Bytes, about 128 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  31 seconds loaded 861443392 Bytes, about 26 
MB/s
insert into select:  28.7 seconds inserted 1000 Rows, about 
348K ops/s
storage size: 17162250921 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](paimon)paimon catalog supports complex types [doris]

2023-10-20 Thread via GitHub


doris-robot commented on PR #25364:
URL: https://github.com/apache/doris/pull/25364#issuecomment-1772245437

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.65 seconds
stream load tsv:  577 seconds loaded 74807831229 Bytes, about 123 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  32 seconds loaded 861443392 Bytes, about 25 
MB/s
insert into select:  29.1 seconds inserted 1000 Rows, about 
343K ops/s
storage size: 17162353741 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[PR] [fix](restore tablet)fix restore_tablet() null_ptr [doris]

2023-10-20 Thread via GitHub


xuwei0912 opened a new pull request, #25682:
URL: https://github.com/apache/doris/pull/25682

   use StorageEngine::instance()
   
   ## Proposed changes
   
   Issue Number: close #xxx
   
   
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](auth)Fix no priv to insert to mysql catalog [doris]

2023-10-20 Thread via GitHub


zddr commented on PR #25678:
URL: https://github.com/apache/doris/pull/25678#issuecomment-1772251178

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](restore tablet)fix restore_tablet() null_ptr [doris]

2023-10-20 Thread via GitHub


xuwei0912 commented on PR #25682:
URL: https://github.com/apache/doris/pull/25682#issuecomment-1772251484

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](stats) analyze spec col only [doris]

2023-10-20 Thread via GitHub


doris-robot commented on PR #25660:
URL: https://github.com/apache/doris/pull/25660#issuecomment-1772253022

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.22 seconds
stream load tsv:  554 seconds loaded 74807831229 Bytes, about 128 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  32 seconds loaded 861443392 Bytes, about 25 
MB/s
insert into select:  28.7 seconds inserted 1000 Rows, about 
348K ops/s
storage size: 17162307643 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](restore tablet)fix restore_tablet() null_ptr [doris]

2023-10-20 Thread via GitHub


github-actions[bot] commented on PR #25682:
URL: https://github.com/apache/doris/pull/25682#issuecomment-1772260460

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](decimal) support decimal256 [doris]

2023-10-20 Thread via GitHub


jacktengg commented on PR #25386:
URL: https://github.com/apache/doris/pull/25386#issuecomment-1772265653

run external


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[PR] [fix](planner) collect_list and collect_set should always not null [doris]

2023-10-20 Thread via GitHub


morrySnow opened a new pull request, #25683:
URL: https://github.com/apache/doris/pull/25683

   cherry-pick from master
   PR: #25592
   commit id: 26f8c7e35260010da5132848e1fc80f6ec430736
   
   ## Proposed changes
   
   Issue Number: close #xxx
   
   
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](planner) collect_list and collect_set should always not null [doris]

2023-10-20 Thread via GitHub


morrySnow commented on PR #25683:
URL: https://github.com/apache/doris/pull/25683#issuecomment-1772266281

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](nereids)do not generate min-max runtime filter for A<=>B and A!=B [doris]

2023-10-20 Thread via GitHub


doris-robot commented on PR #25676:
URL: https://github.com/apache/doris/pull/25676#issuecomment-1772268087

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.38 seconds
stream load tsv:  555 seconds loaded 74807831229 Bytes, about 128 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  32 seconds loaded 861443392 Bytes, about 25 
MB/s
insert into select:  29.3 seconds inserted 1000 Rows, about 
341K ops/s
storage size: 17162106271 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](restore tablet)fix restore_tablet() null_ptr [doris]

2023-10-20 Thread via GitHub


xuwei0912 closed pull request #25682: [fix](restore tablet)fix restore_tablet() 
null_ptr
URL: https://github.com/apache/doris/pull/25682


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [refactor](jni) unified jni framework for java udaf [doris]

2023-10-20 Thread via GitHub


github-actions[bot] commented on PR #25591:
URL: https://github.com/apache/doris/pull/25591#issuecomment-1772271467

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Improvements](status) catch some error status [doris]

2023-10-20 Thread via GitHub


doris-robot commented on PR #25677:
URL: https://github.com/apache/doris/pull/25677#issuecomment-1772270094

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.17 seconds
stream load tsv:  560 seconds loaded 74807831229 Bytes, about 127 
MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc:  64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  32 seconds loaded 861443392 Bytes, about 25 
MB/s
insert into select:  28.7 seconds inserted 1000 Rows, about 
348K ops/s
storage size: 17162134711 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [refactor](jni) unified jni framework for java udaf [doris]

2023-10-20 Thread via GitHub


github-actions[bot] commented on PR #25591:
URL: https://github.com/apache/doris/pull/25591#issuecomment-1772271547

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [improvement](insert) Modify insert into table_id(xxx) to doris_inter… [doris]

2023-10-20 Thread via GitHub


morrySnow commented on code in PR #25662:
URL: https://github.com/apache/doris/pull/25662#discussion_r1366622347


##
fe/fe-core/src/main/cup/sql_parser.cup:
##
@@ -347,6 +347,7 @@ terminal String
 KW_DISTRIBUTION,
 KW_DIV,
 KW_DO,
+KW_DORIS_INTERNAL_TABLE_ID,

Review Comment:
   should add into non-reserved keyword list, and add same keyword for Nereids 
in `src/main/antlr4/org/apache/doris/nereids/DorisParser.g4`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](planner) collect_list and collect_set should always not null [doris]

2023-10-20 Thread via GitHub


github-actions[bot] commented on PR #25683:
URL: https://github.com/apache/doris/pull/25683#issuecomment-1772276480

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [performance](pipelineX) optimize get local state [doris]

2023-10-20 Thread via GitHub


doris-robot commented on PR #25675:
URL: https://github.com/apache/doris/pull/25675#issuecomment-1772278716

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.66 seconds
stream load tsv:  562 seconds loaded 74807831229 Bytes, about 126 
MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc:  66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet:  34 seconds loaded 861443392 Bytes, about 24 
MB/s
insert into select:  29.0 seconds inserted 1000 Rows, about 
344K ops/s
storage size: 17162120518 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[PR] [fix](move-memtable) fix inverted index condition [doris]

2023-10-20 Thread via GitHub


kaijchen opened a new pull request, #25684:
URL: https://github.com/apache/doris/pull/25684

   ## Proposed changes
   
   Fix reversed inverted index check condition.
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [refactor](jni) unified jni framework for java udaf [doris]

2023-10-20 Thread via GitHub


AshinGau merged PR #25591:
URL: https://github.com/apache/doris/pull/25591


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](move-memtable) inverted index condition is reversed [doris]

2023-10-20 Thread via GitHub


github-actions[bot] commented on PR #25684:
URL: https://github.com/apache/doris/pull/25684#issuecomment-1772285767

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](move-memtable) inverted index condition is reversed [doris]

2023-10-20 Thread via GitHub


dataroaring commented on PR #25684:
URL: https://github.com/apache/doris/pull/25684#issuecomment-1772285120

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](move-memtable) inverted index condition is reversed [doris]

2023-10-20 Thread via GitHub


github-actions[bot] commented on PR #25684:
URL: https://github.com/apache/doris/pull/25684#issuecomment-1772285715

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](nereids) Set to use nereids planner in select aggregation expr without from scene when regression-test [doris]

2023-10-20 Thread via GitHub


github-actions[bot] commented on PR #25665:
URL: https://github.com/apache/doris/pull/25665#issuecomment-1772287597

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](nereids) Set to use nereids planner in select aggregation expr without from scene when regression-test [doris]

2023-10-20 Thread via GitHub


github-actions[bot] commented on PR #25665:
URL: https://github.com/apache/doris/pull/25665#issuecomment-1772287548

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [nereids](array_agg)support array_agg in nereids [doris]

2023-10-20 Thread via GitHub


morrySnow commented on code in PR #25630:
URL: https://github.com/apache/doris/pull/25630#discussion_r1366635484


##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/ArrayAgg.java:
##
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.agg;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.AlwaysNotNullable;
+import 
org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.shape.UnaryExpression;
+import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.coercion.AnyDataType;
+import org.apache.doris.nereids.types.coercion.FollowToAnyDataType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * AggregateFunction 'array_agg'.
+ */
+public class ArrayAgg extends AggregateFunction
+implements UnaryExpression, ExplicitlyCastableSignature, 
AlwaysNotNullable {
+public static final List SIGNATURES = ImmutableList.of(
+FunctionSignature.ret(ArrayType.of(new 
FollowToAnyDataType(0))).args(new AnyDataType(0))
+);
+
+public ArrayAgg(Expression arg0) {
+super("array_agg", arg0);
+}
+
+public ArrayAgg(boolean distinct, Expression arg0) {
+super("array_agg", distinct, arg0);
+}
+
+@Override
+public List getSignatures() {
+return SIGNATURES;
+}
+
+@Override
+public AggregateFunction withDistinctAndChildren(boolean distinct, 
List children) {
+Preconditions.checkArgument(children.size() == 1);
+return new ArrayAgg(distinct, children.get(0));
+}

Review Comment:
   need override accept function



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[PR] [Fix](segment) need to rebuild col_id_to_predicates when true predicates encountered [doris]

2023-10-20 Thread via GitHub


airborne12 opened a new pull request, #25685:
URL: https://github.com/apache/doris/pull/25685

   
   ## Proposed changes
   
   Issue Number: close #xxx
   
   relate pr: #25366 
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Fix](segment) need to rebuild col_id_to_predicates when true predicates encountered [doris]

2023-10-20 Thread via GitHub


airborne12 commented on PR #25685:
URL: https://github.com/apache/doris/pull/25685#issuecomment-1772292082

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](auth)Fix no priv to insert to mysql catalog [doris]

2023-10-20 Thread via GitHub


doris-robot commented on PR #25678:
URL: https://github.com/apache/doris/pull/25678#issuecomment-1772293001

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.06 seconds
stream load tsv:  552 seconds loaded 74807831229 Bytes, about 129 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  31 seconds loaded 861443392 Bytes, about 26 
MB/s
insert into select:  29.0 seconds inserted 1000 Rows, about 
344K ops/s
storage size: 17161990921 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[doris] branch branch-2.0 updated: [fix](regression-test) fix bad case in s3 tvf load test in 2.0 (#25680)

2023-10-20 Thread dataroaring
This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new fe4a06a0096 [fix](regression-test) fix bad case in s3 tvf load test in 
2.0 (#25680)
fe4a06a0096 is described below

commit fe4a06a00967780ab51372007f05a18e6a1c5cf2
Author: Siyang Tang <82279870+tangsiyang2...@users.noreply.github.com>
AuthorDate: Fri Oct 20 16:21:11 2023 +0800

[fix](regression-test) fix bad case in s3 tvf load test in 2.0 (#25680)
---
 regression-test/suites/load_p0/tvf/test_s3_tvf.groovy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/regression-test/suites/load_p0/tvf/test_s3_tvf.groovy 
b/regression-test/suites/load_p0/tvf/test_s3_tvf.groovy
index 097c551ea2b..391b76d37eb 100644
--- a/regression-test/suites/load_p0/tvf/test_s3_tvf.groovy
+++ b/regression-test/suites/load_p0/tvf/test_s3_tvf.groovy
@@ -150,7 +150,7 @@ suite("test_s3_tvf", "load_p0") {
 .addProperty("column_separator", "|"))
 
 for(String table : arrayTables) {
-attributeList.add(new TvfAttribute(table, ["k00", "k01", "k02", "k03", 
"k04", "k05", "k06", "k07", "k08", "k09", "k10", "k11", "k12", "k13", "k14", 
"k15", "k16", "k17"], "", "", true)
+attributeList.add(new TvfAttribute(table, ["k00", "k01", "k02", "k03", 
"k04", "k05", "k06", "k07", "k08", "k09", "k10", "k11", "k12", "k13", "k14", 
"k15", "k16", "k17", "kd01", "kd02", "kd03", "kd04", "kd05", "kd06", "kd07", 
"kd08", "kd09", "kd10", "kd11", "kd12", "kd13", "kd14", "kd15", "kd16"], "", 
"", true)
 .addProperty("uri", 
"s3://doris-build-1308700295.cos.ap-beijing.myqcloud.com/regression/load/data/basic_array_data_with_errors.csv")
 .addProperty("format", "csv")
 .addProperty("column_separator", "|"))


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](move-memtable) inverted index filter is reversed [doris]

2023-10-20 Thread via GitHub


github-actions[bot] commented on PR #25684:
URL: https://github.com/apache/doris/pull/25684#issuecomment-1772295311

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](regression-test) fix bad case in s3 tvf load test [doris]

2023-10-20 Thread via GitHub


github-actions[bot] commented on PR #25679:
URL: https://github.com/apache/doris/pull/25679#issuecomment-1772295688

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](regression-test) fix bad case in s3 tvf load test [doris]

2023-10-20 Thread via GitHub


github-actions[bot] commented on PR #25679:
URL: https://github.com/apache/doris/pull/25679#issuecomment-1772295764

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](regression-test) fix bad case in s3 tvf load test in 2.0 [doris]

2023-10-20 Thread via GitHub


dataroaring merged PR #25680:
URL: https://github.com/apache/doris/pull/25680


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[PR] [opt](profile) set datalake profile level as 1 [doris]

2023-10-20 Thread via GitHub


AshinGau opened a new pull request, #25686:
URL: https://github.com/apache/doris/pull/25686

   ## Proposed changes
   
   Follow https://github.com/apache/doris/pull/25491, only the profile marked 
as 1 will be shown in simplified profile.
   
   
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [opt](profile) set datalake profile level as 1 [doris]

2023-10-20 Thread via GitHub


AshinGau commented on PR #25686:
URL: https://github.com/apache/doris/pull/25686#issuecomment-1772298043

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[I] [Bug] doris版本 1.2.7.1 版本 全新安装fe 可以正常启动,但是be启动不了 [doris]

2023-10-20 Thread via GitHub


AndyYanQS opened a new issue, #25687:
URL: https://github.com/apache/doris/issues/25687

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no 
similar issues.
   
   
   ### Version
   
   [fe.log](https://github.com/apache/doris/files/13051889/fe.log)
   [be.INFO.log](https://github.com/apache/doris/files/13051907/be.INFO.log)
   
   配置如下
   [be.conf.log](https://github.com/apache/doris/files/13051931/be.conf.log)
   [fe.conf.log](https://github.com/apache/doris/files/13051932/fe.conf.log)
   
   
   ### What's Wrong?
   
   doris版本 1.2.7.1 版本 全新安装fe 可以正常启动,但是be启动不了,已经手动添加了ALTER SYSTEM ADD BACKEND  
三个节点了,还是不行
   
   ### What You Expected?
   
   辛苦帮忙看看哈
   
   ### How to Reproduce?
   
   _No response_
   
   ### Anything Else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[PR] (selectdb-cloud) Rebuild failed channel to avoid connection refused [doris]

2023-10-20 Thread via GitHub


w41ter opened a new pull request, #25688:
URL: https://github.com/apache/doris/pull/25688

   We cannot judge the progress of reconnecting the underlying channel of grpc.
   In the worst case, it may take two minutes. But we couldn't stand the 
connection
   refused for two minutes, so we rebuilt the channel directly.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] (selectdb-cloud) Rebuild failed channel to avoid connection refused [doris]

2023-10-20 Thread via GitHub


w41ter commented on PR #25688:
URL: https://github.com/apache/doris/pull/25688#issuecomment-1772301163

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](auth)Fix no priv to insert to mysql catalog [doris]

2023-10-20 Thread via GitHub


zddr commented on PR #25678:
URL: https://github.com/apache/doris/pull/25678#issuecomment-1772301589

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Fix](segment) need to rebuild col_id_to_predicates when true predicates encountered [doris]

2023-10-20 Thread via GitHub


github-actions[bot] commented on PR #25685:
URL: https://github.com/apache/doris/pull/25685#issuecomment-1772303037

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [opt](profile) set datalake profile level as 1 [doris]

2023-10-20 Thread via GitHub


github-actions[bot] commented on PR #25686:
URL: https://github.com/apache/doris/pull/25686#issuecomment-1772308965

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] (selectdb-cloud) Rebuild failed channel to avoid connection refused [doris]

2023-10-20 Thread via GitHub


github-actions[bot] commented on PR #25688:
URL: https://github.com/apache/doris/pull/25688#issuecomment-1772310221

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](planner) collect_list and collect_set should always not null [doris]

2023-10-20 Thread via GitHub


hello-stephen commented on PR #25683:
URL: https://github.com/apache/doris/pull/25683#issuecomment-1772310727

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.54 seconds
stream load tsv:  578 seconds loaded 74807831229 Bytes, about 123 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  32 seconds loaded 861443392 Bytes, about 25 
MB/s
insert into select:  29.7 seconds inserted 1000 Rows, about 
336K ops/s
storage size: 17163060685 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](restore tablet)fix restore_tablet() null_ptr [doris]

2023-10-20 Thread via GitHub


hello-stephen commented on PR #25682:
URL: https://github.com/apache/doris/pull/25682#issuecomment-1772310964

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.34 seconds
stream load tsv:  583 seconds loaded 74807831229 Bytes, about 122 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  33 seconds loaded 861443392 Bytes, about 24 
MB/s
insert into select:  29.6 seconds inserted 1000 Rows, about 
337K ops/s
storage size: 17162582060 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [typo](doc)Adjust tablet_rowset_stale_sweep_time_sec parameter defaul… [doris]

2023-10-20 Thread via GitHub


DongLiang-0 commented on PR #25584:
URL: https://github.com/apache/doris/pull/25584#issuecomment-1772312332

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [regression-test](modify) enable and disable suitable test cases [doris]

2023-10-20 Thread via GitHub


shuke987 commented on PR #25658:
URL: https://github.com/apache/doris/pull/25658#issuecomment-1772313266

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [pipeline](load) disable pipeline load [doris]

2023-10-20 Thread via GitHub


Gabriel39 closed pull request #25667: [pipeline](load) disable pipeline load
URL: https://github.com/apache/doris/pull/25667


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [nereids](array_agg)support array_agg in nereids [doris]

2023-10-20 Thread via GitHub


LemonLiTree commented on code in PR #25630:
URL: https://github.com/apache/doris/pull/25630#discussion_r1366658973


##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/ArrayAgg.java:
##
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.agg;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.AlwaysNotNullable;
+import 
org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.shape.UnaryExpression;
+import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.coercion.AnyDataType;
+import org.apache.doris.nereids.types.coercion.FollowToAnyDataType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * AggregateFunction 'array_agg'.
+ */
+public class ArrayAgg extends AggregateFunction
+implements UnaryExpression, ExplicitlyCastableSignature, 
AlwaysNotNullable {
+public static final List SIGNATURES = ImmutableList.of(
+FunctionSignature.ret(ArrayType.of(new 
FollowToAnyDataType(0))).args(new AnyDataType(0))
+);
+
+public ArrayAgg(Expression arg0) {
+super("array_agg", arg0);
+}
+
+public ArrayAgg(boolean distinct, Expression arg0) {
+super("array_agg", distinct, arg0);
+}
+
+@Override
+public List getSignatures() {
+return SIGNATURES;
+}
+
+@Override
+public AggregateFunction withDistinctAndChildren(boolean distinct, 
List children) {
+Preconditions.checkArgument(children.size() == 1);
+return new ArrayAgg(distinct, children.get(0));
+}

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Feature](datatype) Add IPv4/v6 data type for doris [doris]

2023-10-20 Thread via GitHub


sjyango commented on PR #24965:
URL: https://github.com/apache/doris/pull/24965#issuecomment-1772317959

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](move-memtable) inverted index filter is reversed [doris]

2023-10-20 Thread via GitHub


doris-robot commented on PR #25684:
URL: https://github.com/apache/doris/pull/25684#issuecomment-1772320936

   TeamCity be ut coverage result:
Function Coverage: 37.03% (8299/22410) 
Line Coverage: 29.19% (66675/228388)
Region Coverage: 27.82% (34601/124360)
Branch Coverage: 24.41% (17572/71996)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/b2ecf5788b549af4bbc75da1c7f37f5e58945cba_b2ecf5788b549af4bbc75da1c7f37f5e58945cba/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[PR] set default docker list [doris]

2023-10-20 Thread via GitHub


zhangguoqiang666 opened a new pull request, #25689:
URL: https://github.com/apache/doris/pull/25689

   ## Proposed changes
   
   
   set default docker list
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Feature](datatype) Add IPv4/v6 data type for doris [doris]

2023-10-20 Thread via GitHub


sjyango commented on PR #24965:
URL: https://github.com/apache/doris/pull/24965#issuecomment-1772323345

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Fix](segment) need to rebuild col_id_to_predicates when true predicates encountered [doris]

2023-10-20 Thread via GitHub


doris-robot commented on PR #25685:
URL: https://github.com/apache/doris/pull/25685#issuecomment-1772326707

   TeamCity be ut coverage result:
Function Coverage: 37.03% (8299/22410) 
Line Coverage: 29.19% (66672/228399)
Region Coverage: 27.82% (34594/124363)
Branch Coverage: 24.41% (17576/72000)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/56e60ccfb471136aae64843365672c6df471672c_56e60ccfb471136aae64843365672c6df471672c/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[PR] [fix](load) patch a quick fix of loading failure [doris]

2023-10-20 Thread via GitHub


zclllyybb opened a new pull request, #25690:
URL: https://github.com/apache/doris/pull/25690

   ## Proposed changes
   
   Issue Number: close #xxx
   
   This will decrease the load fail probability
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Feature](datatype) Add IPv4/v6 data type for doris [doris]

2023-10-20 Thread via GitHub


github-actions[bot] commented on code in PR #24965:
URL: https://github.com/apache/doris/pull/24965#discussion_r1366675497


##
be/src/olap/types.h:
##
@@ -957,6 +969,103 @@ struct 
FieldTypeTraits
 }
 };
 
+template <>
+struct FieldTypeTraits
+: public BaseFieldtypeTraits {
+static Status from_string(void* buf, const std::string& scan_key, const 
int precision,
+  const int scale) {
+StringParser::ParseResult result = StringParser::PARSE_SUCCESS;
+uint32_t value = 
StringParser::string_to_unsigned_int(scan_key.c_str(), 
scan_key.size(), &result);
+
+if (result == StringParser::PARSE_FAILURE) {
+return Status::Error(
+"FieldTypeTraits::from_string meet 
PARSE_FAILURE");
+}
+*reinterpret_cast(buf) = value;
+return Status::OK();
+}
+
+static std::string to_string(const void* src) {
+uint32_t value = *reinterpret_cast(src);
+std::stringstream ss;
+ss << ((value >> 24) & 0xFF) << '.'
+   << ((value >> 16) & 0xFF) << '.'
+   << ((value >> 8) & 0xFF) << '.'
+   << (value & 0xFF);
+return ss.str();
+}
+};
+
+
+template <>
+struct FieldTypeTraits
+: public BaseFieldtypeTraits {
+static Status from_string(void* buf, const std::string& scan_key, const 
int precision,
+  const int scale) {
+std::istringstream iss(scan_key);
+std::string token;
+uint128_t result = 0;
+int count = 0;
+
+while (std::getline(iss, token, ':')) {
+if (token.empty()) {
+count += 8 - count;
+break;
+}
+
+if (count > 8) {
+return Status::Error(
+"FieldTypeTraits::from_string 
meet PARSE_FAILURE");
+}
+
+uint16_t value = 0;
+std::istringstream ss(token);
+if (!(ss >> std::hex >> value)) {
+return Status::Error(
+"FieldTypeTraits::from_string 
meet PARSE_FAILURE");
+}
+
+result = (result << 16) | value;
+count++;
+}
+
+if (count < 8) {
+return Status::Error(
+"FieldTypeTraits::from_string meet 
PARSE_FAILURE");
+}
+
+*reinterpret_cast(buf) = result;
+return Status::OK();
+}
+
+static std::string to_string(const void* src) {
+std::stringstream result;
+uint128_t ipv6 = *reinterpret_cast(src);
+
+for (int i = 0; i < 8; i++) {
+uint16_t part = static_cast((ipv6 >> (112 - i * 16)) & 
0x);
+result << std::to_string(part);
+if (i != 7) {
+result << ":";
+}
+}
+
+return result.str();
+}
+
+static void set_to_max(void* buf) {
+*reinterpret_cast(buf) =
+static_cast(99ll) * 
10ll * 1000ll +
+static_cast(9ll) * 1000ll + 999ll;

Review Comment:
   warning: 1000ll is a magic number; consider replacing it with a named 
constant [readability-magic-numbers]
   ```cpp
   static_cast(9ll) * 1000ll + 999ll;
^
   ```
   



##
be/src/olap/types.h:
##
@@ -957,6 +969,103 @@ struct 
FieldTypeTraits
 }
 };
 
+template <>
+struct FieldTypeTraits
+: public BaseFieldtypeTraits {
+static Status from_string(void* buf, const std::string& scan_key, const 
int precision,
+  const int scale) {
+StringParser::ParseResult result = StringParser::PARSE_SUCCESS;
+uint32_t value = 
StringParser::string_to_unsigned_int(scan_key.c_str(), 
scan_key.size(), &result);
+
+if (result == StringParser::PARSE_FAILURE) {
+return Status::Error(
+"FieldTypeTraits::from_string meet 
PARSE_FAILURE");
+}
+*reinterpret_cast(buf) = value;
+return Status::OK();
+}
+
+static std::string to_string(const void* src) {
+uint32_t value = *reinterpret_cast(src);
+std::stringstream ss;
+ss << ((value >> 24) & 0xFF) << '.'
+   << ((value >> 16) & 0xFF) << '.'
+   << ((value >> 8) & 0xFF) << '.'
+   << (value & 0xFF);
+return ss.str();
+}
+};
+
+
+template <>
+struct FieldTypeTraits
+: public BaseFieldtypeTraits {
+static Status from_string(void* buf, const std::string& scan_key, const 
int precision,
+  const int scale) {
+std::istringstream iss(scan_key);
+std::string token;
+uint128_t result = 0;
+int count = 0;
+
+while (std::getline(iss, token, ':')) {
+if (token.empty()) {
+count += 8 - count;
+

Re: [PR] [improvement](insert) Modify insert into table_id(xxx) to doris_inter… [doris]

2023-10-20 Thread via GitHub


mymeiyi commented on PR #25662:
URL: https://github.com/apache/doris/pull/25662#issuecomment-1772334732

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](load) patch a quick fix of loading failure [doris]

2023-10-20 Thread via GitHub


zclllyybb commented on PR #25690:
URL: https://github.com/apache/doris/pull/25690#issuecomment-1772333219

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](move-memtable) inverted index filter is reversed [doris]

2023-10-20 Thread via GitHub


doris-robot commented on PR #25684:
URL: https://github.com/apache/doris/pull/25684#issuecomment-1772337517

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.24 seconds
stream load tsv:  555 seconds loaded 74807831229 Bytes, about 128 
MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  32 seconds loaded 861443392 Bytes, about 25 
MB/s
insert into select:  29.4 seconds inserted 1000 Rows, about 
340K ops/s
storage size: 17162290801 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Feature](inverted index) support range predicate for inverted index [doris]

2023-10-20 Thread via GitHub


airborne12 commented on PR #24511:
URL: https://github.com/apache/doris/pull/24511#issuecomment-1772338902

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



  1   2   3   4   5   >