Re: [PR] [performance](pipelineX) optimize get local state [doris]
github-actions[bot] commented on PR #25675: URL: https://github.com/apache/doris/pull/25675#issuecomment-1772190418 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](samples)fix IndexOutOfBoundsException [doris]
github-actions[bot] commented on PR #25608: URL: https://github.com/apache/doris/pull/25608#issuecomment-1772191924 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](samples)fix IndexOutOfBoundsException [doris]
github-actions[bot] commented on PR #25608: URL: https://github.com/apache/doris/pull/25608#issuecomment-1772192000 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Bug] About the limit query, limit 1, the background will still perform a large number of calculations after the output of 1 result.关于limit查询,limit 1,输出结果1条后,后台仍会进行大量计算的问题。 [doris]
shaotianfeng commented on issue #25604: URL: https://github.com/apache/doris/issues/25604#issuecomment-1772194450 昨天有试着升级2.0.2还是存在这个问题,然后又降级了 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](stats) analyze spec col only [doris]
Kikyou1997 commented on PR #25660: URL: https://github.com/apache/doris/pull/25660#issuecomment-1772194480 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [typo](doc)Modify the default value of Stale rowset cleanup policy [doris]
JNSimba merged PR #25517: URL: https://github.com/apache/doris/pull/25517 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](samples)fix IndexOutOfBoundsException [doris]
JNSimba merged PR #25608: URL: https://github.com/apache/doris/pull/25608 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [typo](doc)Modify the default value of Stale rowset cleanup policy(#25517)
This is an automated email from the ASF dual-hosted git repository. diwu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 68d3c25f263 [typo](doc)Modify the default value of Stale rowset cleanup policy(#25517) 68d3c25f263 is described below commit 68d3c25f263916b2aff569f44de1b0b8de61816f Author: caoliang-web <71004656+caoliang-...@users.noreply.github.com> AuthorDate: Fri Oct 20 15:03:49 2023 +0800 [typo](doc)Modify the default value of Stale rowset cleanup policy(#25517) --- docs/en/docs/admin-manual/config/be-config.md| 2 +- docs/zh-CN/docs/admin-manual/config/be-config.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/docs/admin-manual/config/be-config.md b/docs/en/docs/admin-manual/config/be-config.md index 4fc3106fbe9..05a8a5d16a0 100644 --- a/docs/en/docs/admin-manual/config/be-config.md +++ b/docs/en/docs/admin-manual/config/be-config.md @@ -1266,7 +1266,7 @@ BaseCompaction:546859: * Type: int64 * Description: It is used to control the expiration time of cleaning up the merged rowset version. When the current time now() minus the max created rowset‘s create time in a version path is greater than tablet_rowset_stale_sweep_time_sec, the current path is cleaned up and these merged rowsets are deleted, the unit is second. - When writing is too frequent and the disk time is insufficient, you can configure less tablet_rowset_stale_sweep_time_sec. However, if this time is less than 5 minutes, it may cause fe to query the version that has been merged, causing a query -230 error. -* Default value: 1800 +* Default value: 300 `tablet_writer_open_rpc_timeout_sec` diff --git a/docs/zh-CN/docs/admin-manual/config/be-config.md b/docs/zh-CN/docs/admin-manual/config/be-config.md index ac0263491ae..4a1476a980e 100644 --- a/docs/zh-CN/docs/admin-manual/config/be-config.md +++ b/docs/zh-CN/docs/admin-manual/config/be-config.md @@ -1293,7 +1293,7 @@ BaseCompaction:546859: * 类型:int64 * 描述:用来表示清理合并版本的过期时间,当当前时间 now() 减去一个合并的版本路径中rowset最近创建创建时间大于tablet_rowset_stale_sweep_time_sec时,对当前路径进行清理,删除这些合并过的rowset, 单位为s。 - 当写入过于频繁,磁盘空间不足时,可以配置较少这个时间。不过这个时间过短小于5分钟时,可能会引发fe查询不到已经合并过的版本,引发查询-230错误。 -* 默认值:1800 +* 默认值:300 `tablet_writer_open_rpc_timeout_sec` - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvements](status) catch some error status [doris]
github-actions[bot] commented on PR #25677: URL: https://github.com/apache/doris/pull/25677#issuecomment-1772196114 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Bug](samples)fix IndexOutOfBoundsException (#25608)
This is an automated email from the ASF dual-hosted git repository. diwu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 42e5a3324c2 [Bug](samples)fix IndexOutOfBoundsException (#25608) 42e5a3324c2 is described below commit 42e5a3324c2a9426df9b464c509063e98e53f195 Author: caoliang-web <71004656+caoliang-...@users.noreply.github.com> AuthorDate: Fri Oct 20 15:04:44 2023 +0800 [Bug](samples)fix IndexOutOfBoundsException (#25608) --- .../src/main/java/org/apache/doris/sdk/DorisReaderExample.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/samples/doris-demo/doris-source-demo/src/main/java/org/apache/doris/sdk/DorisReaderExample.java b/samples/doris-demo/doris-source-demo/src/main/java/org/apache/doris/sdk/DorisReaderExample.java index 5c4a5c101b5..03bebef58d8 100644 --- a/samples/doris-demo/doris-source-demo/src/main/java/org/apache/doris/sdk/DorisReaderExample.java +++ b/samples/doris-demo/doris-source-demo/src/main/java/org/apache/doris/sdk/DorisReaderExample.java @@ -166,7 +166,7 @@ public class DorisReaderExample { if(!eos){ int i = convertArrow(next, selectedColumns); offset += i; -readRowCount += offset; +readRowCount = offset; } } //close - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](decimal) support decimal256 [doris]
doris-robot commented on PR #25386: URL: https://github.com/apache/doris/pull/25386#issuecomment-1772197714 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.42 seconds stream load tsv: 552 seconds loaded 74807831229 Bytes, about 129 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.7 seconds inserted 1000 Rows, about 348K ops/s storage size: 17162108934 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [bug] Doris2.0.2 执行固定SQL后,BE报错,甚至导致所有BE宕机。 [doris]
DA1OOO commented on issue #25644: URL: https://github.com/apache/doris/issues/25644#issuecomment-1772211533 Which version is OK to run this SQL? 2.0.1.1 ? Tableau will generate many SQL like this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch dev_join updated (45d1d16f3e9 -> 16a2f7d0907)
This is an automated email from the ASF dual-hosted git repository. panxiaolei pushed a change to branch dev_join in repository https://gitbox.apache.org/repos/asf/doris.git omit 45d1d16f3e9 update dev_join to pre calculate bucket num (#25663) omit dbd3454abe4 support batch size (#25629) omit c3089031635 update rf omit 032555667fb merge block to single block on join/set node add b0e0a0569a3 [Fix](row store) Real default value should be used instead of default… (#25230) add 6f6264693ff [fix](Nereids) can't choosing best plan for join that could only broadcast (#25511) add ef9cbc4c64e [enhancement](priv) Clarify ccr releated FrontendServiceImpl call privs (#25530) add 64aeeb971b5 [Fix](partial-update) Correct the alignment process when the table has sequence column and add cases (#25346) add 6cb947f72b2 [refactor](unused code) delete unused method from field.h (#25554) add d2400d1d7bf [feature](profile) profilev2 distinguish Sink and Operator in pipelineX (#25491) add db16a149704 [fix](backup) fix backup fail on s3 (#25496) add 76abbfc5556 [fix](mvn source) fix audit compile java-cup and cup-maven-plugin not found (#25564) add 0ec537edef4 [fix](column-id) fix null conn ctx in column id flusher and parser for `database` field in corresponding show stmt (#25393) add 8a8e6edba98 [enhancement](stats) Unify sample algorithm between olap table and external table (#25472) add 62d06584f1e [feature](fe) add function 'BitmapAgg' in nereids (#25508) add 0533911275b [Enhance](regression)change external default docker list (#25573) add c77590414ea [fix](pipeline)fix case (#25567) add 80e5e722024 [fix](scanner) coredump caused by 'prune_predicates_by_zone_map' (#2) add e4a83a22d14 [opt](error msg) Make data codec error clearly when load csv data can't display (#25540) add 85e83b9c6c5 [fix](auth)fix not check udf auth in nereids (#25541) add ea97f2d6e51 [refactor](nereids)make enable_stats session var (#25550) add 9e07291e65a [fix](stats) internal stats table tablet replica num not update as expected (#25563) add 2ddd2e50797 [feature](Nereids) add map_agg function (#25246) add 46158a6555a [tpcds-tools](nereids) remove cascades specific control for global config (#25595) add 9c9fc84f399 [feature](merge-cloud) Abstract BaseTablet for CloudTablet (#24929) add 2a442972a82 [Fix](merge-on-write) Fix some bugs about sequence column (#24915) add c21eb315b06 [feature](thrift api) support expr in MemoryScratchSink and make arrow::Schema recalculate with block info (#24603) add 32fc8a17998 [chore](compaction) Do not print the stack trace when the compaction task already exists (#25597) add 11fecafb749 [fix](move-memtable) fallback if target table contains inverted index (#25498) add dbf57876823 [fix](be) Make DorisCallOnce's function exception-safe (#25579) add 5b814e6e653 [cases](regression-test) Add create and drop S3 repository test (#25568) add 45178f5664d [improvement](disk balance) impr disk rebalancer sched with partition rebalancer (#25549) add 63c89df4741 [enhencement](RowsetWriter) Don't delete files when beta rowset writer destructed (#25578) add 8d65a627058 [Fix](show-frontends-disk)Fix NPE and macOS compatibility (#25565) add 4752b800b2d [typo](doc)update config (#25425) add fcf7bdc9e0e [typo](docs) Rename Import Advanced to CN Version (#25374) add b45f501e511 [improvement](nereids) Support aggregate functions without from clause (#25500) add ed713c2e2d6 [FIX](func) fix count distinct do not support arr/map/struct (#25483) add 3d1206d3259 [date](fix) modify push-down predicate for datev1 type (#25571) add e77b98be883 [fix](months_diff) fix wrong result of months_diff (#25577) add 159be51ea6a [bugfix](schema_change) Fix the coredump when doubly write during schema change (#22557) add 23535824934 [enhancement](load) support for broker load, routine load, mysql load and add docs (#25528) add d0d9c4267dd [fix](move-memtable) fix use-after-free in load stream stub (#25618) add 68eaba7220f [DOC](fix) fix hyperlink to create tpch table (#25561) add 4d2e7d7c86b [improvement](statistics)Set min max to NULL when collect stats with sample (#25593) add 54780c62e0d [improvement](executor)Using cgroup to implement cpu hard limit (#25489) add 7385602b190 [bug](rf) fix only min/max rf return error when has remote target (#25588) add b0e70bd02bc [fix](pipelineX) fix error set delta_urls (#25631) add fa5dd61563f [feature](pipelineX) operator id use negative numbers (#25529) add b964ab76b33 [refactor](shuffle) Simplify hash partitioning strategy (#25596) add 8b5fd616321 [pipelineX](profile) Add pending finish dependency timer (#25635) add a42536efc11 [feature](pipelineX) release dependency when task close (#25633) add 9ea8efe5fae [cover
[doris] 02/04: update rf
This is an automated email from the ASF dual-hosted git repository. panxiaolei pushed a commit to branch dev_join in repository https://gitbox.apache.org/repos/asf/doris.git commit d15c3f064a26199c7c66c5605e2f638daed1b622 Author: BiteThet AuthorDate: Wed Oct 18 14:22:02 2023 +0800 update rf --- be/src/exprs/runtime_filter_slots.h| 35 - be/src/pipeline/exec/hashjoin_build_sink.h | 2 +- be/src/vec/exec/join/vhash_join_node.h | 61 -- 3 files changed, 35 insertions(+), 63 deletions(-) diff --git a/be/src/exprs/runtime_filter_slots.h b/be/src/exprs/runtime_filter_slots.h index e0ff2cb0067..307253f430c 100644 --- a/be/src/exprs/runtime_filter_slots.h +++ b/be/src/exprs/runtime_filter_slots.h @@ -161,7 +161,7 @@ public: return Status::OK(); } -void insert(std::unordered_map>& datas) { +void insert(const std::unordered_set& datas) { for (int i = 0; i < _build_expr_context.size(); ++i) { auto iter = _runtime_filters.find(i); if (iter == _runtime_filters.end()) { @@ -169,30 +169,31 @@ public: } int result_column_id = _build_expr_context[i]->get_last_result_column_id(); -for (auto it : datas) { -auto& column = it.first->get_by_position(result_column_id).column; +for (const auto* it : datas) { +auto column = it->get_by_position(result_column_id).column; -if (auto* nullable = +std::vector indexs; +if (const auto* nullable = vectorized::check_and_get_column(*column)) { -auto& column_nested = nullable->get_nested_column_ptr(); -auto& column_nullmap = nullable->get_null_map_column_ptr(); -std::vector indexs; -for (int row_num : it.second) { -if (assert_cast(column_nullmap.get()) -->get_bool(row_num)) { +column = nullable->get_nested_column_ptr(); +const uint8_t* null_map = assert_cast( + nullable->get_null_map_column_ptr().get()) + ->get_data() + .data(); +for (int i = 0; i < column->size(); i++) { +if (null_map[i]) { continue; } -indexs.push_back(row_num); +indexs.push_back(i); } -for (auto filter : iter->second) { -filter->insert_batch(column_nested, indexs); -} - } else { -for (auto filter : iter->second) { -filter->insert_batch(column, it.second); +for (int i = 0; i < column->size(); i++) { +indexs.push_back(i); } } +for (auto* filter : iter->second) { +filter->insert_batch(column, indexs); +} } } } diff --git a/be/src/pipeline/exec/hashjoin_build_sink.h b/be/src/pipeline/exec/hashjoin_build_sink.h index 9cf559588cc..49c1a459b70 100644 --- a/be/src/pipeline/exec/hashjoin_build_sink.h +++ b/be/src/pipeline/exec/hashjoin_build_sink.h @@ -101,7 +101,7 @@ protected: bool _has_set_need_null_map_for_build = false; bool _build_side_ignore_null = false; size_t _build_rf_cardinality = 0; -std::unordered_map> _inserted_rows; +std::unordered_set _inserted_blocks; std::shared_ptr _shared_hash_table_dependency; RuntimeProfile::Counter* _build_table_timer; diff --git a/be/src/vec/exec/join/vhash_join_node.h b/be/src/vec/exec/join/vhash_join_node.h index ef5a61eae17..c0d964fd66c 100644 --- a/be/src/vec/exec/join/vhash_join_node.h +++ b/be/src/vec/exec/join/vhash_join_node.h @@ -86,10 +86,10 @@ struct ProcessRuntimeFilterBuild { RETURN_IF_ERROR(parent->_runtime_filter_slots->init( state, hash_table_ctx.hash_table->size(), parent->_build_rf_cardinality)); -if (!parent->_runtime_filter_slots->empty() && !parent->_inserted_rows.empty()) { +if (!parent->_runtime_filter_slots->empty() && !parent->_inserted_blocks.empty()) { { SCOPED_TIMER(parent->_push_compute_timer); -parent->_runtime_filter_slots->insert(parent->_inserted_rows); + parent->_runtime_filter_slots->insert(parent->_inserted_blocks); } } { @@ -117,54 +117,25 @@ struct ProcessHashTableBuild { template Status run(HashTableContext& hash_table_ctx, ConstNullMapPtr null_map, bool* has_null_key) { -using KeyGetter
[doris] 03/04: support batch size (#25629)
This is an automated email from the ASF dual-hosted git repository. panxiaolei pushed a commit to branch dev_join in repository https://gitbox.apache.org/repos/asf/doris.git commit a0d727cccac81ec4761e4de9d63913839979a7b2 Author: HappenLee AuthorDate: Fri Oct 20 10:33:31 2023 +0800 support batch size (#25629) * support batch size * support left anti/semi/outer join --- be/src/olap/delta_writer.cpp | 2 +- be/src/olap/delta_writer.h | 2 +- be/src/olap/delta_writer_v2.cpp| 2 +- be/src/olap/delta_writer_v2.h | 2 +- be/src/olap/memtable.cpp | 4 +- be/src/olap/memtable.h | 2 +- be/src/olap/memtable_writer.cpp| 2 +- be/src/olap/memtable_writer.h | 2 +- be/src/olap/tablet.cpp | 2 +- be/src/pipeline/exec/exchange_sink_operator.cpp| 2 +- .../exec/nested_loop_join_probe_operator.cpp | 2 +- be/src/runtime/tablets_channel.cpp | 7 +- be/src/vec/columns/column.h| 6 +- be/src/vec/columns/column_array.cpp| 10 +-- be/src/vec/columns/column_array.h | 4 +- be/src/vec/columns/column_complex.h| 10 +-- be/src/vec/columns/column_const.h | 4 +- be/src/vec/columns/column_decimal.h| 6 +- be/src/vec/columns/column_dictionary.h | 4 +- be/src/vec/columns/column_fixed_length_object.h| 12 +-- be/src/vec/columns/column_map.cpp | 10 +-- be/src/vec/columns/column_map.h| 4 +- be/src/vec/columns/column_nullable.cpp | 5 +- be/src/vec/columns/column_nullable.h | 4 +- be/src/vec/columns/column_object.cpp | 15 ++-- be/src/vec/columns/column_object.h | 8 +- be/src/vec/columns/column_string.cpp | 28 +++ be/src/vec/columns/column_string.h | 4 +- be/src/vec/columns/column_struct.cpp | 4 +- be/src/vec/columns/column_struct.h | 4 +- be/src/vec/columns/column_vector.cpp | 13 ++-- be/src/vec/columns/column_vector.h | 4 +- be/src/vec/columns/predicate_column.h | 4 +- be/src/vec/common/hash_table/hash_map.h| 91 -- be/src/vec/core/block.cpp | 3 +- be/src/vec/core/block.h| 2 +- be/src/vec/exec/join/process_hash_table_probe.h| 2 +- .../vec/exec/join/process_hash_table_probe_impl.h | 9 ++- be/src/vec/exec/join/vhash_join_node.h | 2 +- be/src/vec/exec/join/vnested_loop_join_node.cpp| 2 +- be/src/vec/exec/scan/pip_scanner_context.h | 6 +- be/src/vec/sink/vdata_stream_sender.cpp| 6 +- be/src/vec/sink/vdata_stream_sender.h | 8 +- be/src/vec/sink/vtablet_sink_v2.h | 2 +- 44 files changed, 197 insertions(+), 130 deletions(-) diff --git a/be/src/olap/delta_writer.cpp b/be/src/olap/delta_writer.cpp index 23e1718cb7d..4098f512120 100644 --- a/be/src/olap/delta_writer.cpp +++ b/be/src/olap/delta_writer.cpp @@ -108,7 +108,7 @@ Status DeltaWriter::append(const vectorized::Block* block) { return write(block, {}, true); } -Status DeltaWriter::write(const vectorized::Block* block, const std::vector& row_idxs, +Status DeltaWriter::write(const vectorized::Block* block, const std::vector& row_idxs, bool is_append) { if (UNLIKELY(row_idxs.empty() && !is_append)) { return Status::OK(); diff --git a/be/src/olap/delta_writer.h b/be/src/olap/delta_writer.h index 303f17f14f8..4b2db0c9cc3 100644 --- a/be/src/olap/delta_writer.h +++ b/be/src/olap/delta_writer.h @@ -67,7 +67,7 @@ public: Status init(); -Status write(const vectorized::Block* block, const std::vector& row_idxs, +Status write(const vectorized::Block* block, const std::vector& row_idxs, bool is_append = false); Status append(const vectorized::Block* block); diff --git a/be/src/olap/delta_writer_v2.cpp b/be/src/olap/delta_writer_v2.cpp index ef3ff23f9d8..47723f26d72 100644 --- a/be/src/olap/delta_writer_v2.cpp +++ b/be/src/olap/delta_writer_v2.cpp @@ -138,7 +138,7 @@ Status DeltaWriterV2::append(const vectorized::Block* block) { return write(block, {}, true); } -Status DeltaWriterV2::write(const vectorized::Block* block, const std::vector& row_idxs, +Status DeltaWriterV2::write(const vectorized::Block* block, const std::vector& row_idxs, bool is_append) { if (UNLIKELY(row_idxs.empty() && !is_append)) { return Status::OK(); diff --git a/be/src/olap/delta_writer_v2.h b/be/src/olap/delta_writer_v2.h index b2b1f5f1c19..8f92ec74273 100644 --- a/be/src/olap/delta_writer_v2.h +++
[doris] 04/04: update dev_join to pre calculate bucket num (#25663)
This is an automated email from the ASF dual-hosted git repository. panxiaolei pushed a commit to branch dev_join in repository https://gitbox.apache.org/repos/asf/doris.git commit 16a2f7d090731d7fd6ebf0d695fe933c153d41d5 Author: Pxl AuthorDate: Fri Oct 20 11:50:08 2023 +0800 update dev_join to pre calculate bucket num (#25663) --- be/src/vec/common/hash_table/hash_map.h| 51 +++-- be/src/vec/common/hash_table/hash_map_context.h| 8 be/src/vec/exec/join/process_hash_table_probe.h| 6 --- .../vec/exec/join/process_hash_table_probe_impl.h | 53 ++ be/src/vec/exec/join/vhash_join_node.h | 2 + 5 files changed, 39 insertions(+), 81 deletions(-) diff --git a/be/src/vec/common/hash_table/hash_map.h b/be/src/vec/common/hash_table/hash_map.h index 53cb01dbfaa..89dfe7f8aac 100644 --- a/be/src/vec/common/hash_table/hash_map.h +++ b/be/src/vec/common/hash_table/hash_map.h @@ -216,18 +216,23 @@ public: return phmap::priv::NormalizeCapacity(expect_bucket_size) + 1; } +void reserve(int num_elem) { +bucket_size = calc_bucket_size(num_elem + 1); +first.resize(bucket_size, 0); +next.resize(num_elem); +} + void build(const Key* __restrict keys, const size_t* __restrict hash_values, size_t num_elem, int batch_size) { -max_batch_size = batch_size; -bucket_size = calc_bucket_size(num_elem + 1); +_batch_size = batch_size; +bucket_size = calc_bucket_size(num_elem); first.resize(bucket_size, 0); next.resize(num_elem); build_keys = keys; for (size_t i = 1; i < num_elem; i++) { -uint32_t bucket_num = hash_values[i] & (bucket_size - 1); -next[i] = first[bucket_num]; -first[bucket_num] = i; +next[i] = first[hash_values[i]]; +first[hash_values[i]] = i; } } @@ -248,18 +253,16 @@ public: return std::pair {0, 0}; } +size_t get_bucket_mask() { return bucket_size - 1; } + private: template auto _find_batch_left_semi_anti(const Key* __restrict keys, const size_t* __restrict hash_values, int probe_idx, int probe_rows, std::vector& probe_idxs) { -auto matched_cnt = 0; -const auto batch_size = max_batch_size; - -while (LIKELY(probe_idx < probe_rows && matched_cnt < batch_size)) { -uint32_t bucket_num = hash_values[probe_idx] & (bucket_size - 1); -auto build_idx = first[bucket_num]; - +int matched_cnt = 0; +while (LIKELY(probe_idx < probe_rows && matched_cnt < _batch_size)) { +uint32_t build_idx = first[hash_values[probe_idx]]; while (build_idx) { if (keys[probe_idx] == build_keys[build_idx]) { break; @@ -279,12 +282,11 @@ private: const size_t* __restrict hash_values, int probe_idx, int probe_rows, std::vector& probe_idxs, std::vector& build_idxs) { -auto matched_cnt = 0; -const auto batch_size = max_batch_size; +int matched_cnt = 0; uint32_t build_idx = 0; auto do_the_probe = [&]() { -while (build_idx && LIKELY(matched_cnt < batch_size)) { +while (build_idx && LIKELY(matched_cnt < _batch_size)) { if (keys[probe_idx] == build_keys[build_idx]) { probe_idxs[matched_cnt] = probe_idx; build_idxs[matched_cnt] = build_idx; @@ -302,12 +304,7 @@ private: } } -if (matched_cnt == max_batch_size && build_idx) { -current_probe_idx = probe_idx; -current_build_idx = build_idx; -} else { -probe_idx++; -} +probe_idx++; }; // some row over the batch_size, need dispose first @@ -317,17 +314,21 @@ private: current_build_idx = 0; do_the_probe(); } -while (LIKELY(probe_idx < probe_rows && matched_cnt < batch_size)) { -uint32_t bucket_num = hash_values[probe_idx] & (bucket_size - 1); -build_idx = first[bucket_num]; +while (LIKELY(probe_idx < probe_rows && matched_cnt < _batch_size)) { +build_idx = first[hash_values[probe_idx]]; do_the_probe(); } + +if (matched_cnt == _batch_size && build_idx) { +current_probe_idx = probe_idx - 1; +current_build_idx = build_idx; +} return std::pair {probe_idx, matched_cnt}; } const Key* __restrict build_keys; uint32_t bucket_size = 0; -int max_batch_size = 0; +int _batch_size = 0; int current_probe_idx = -1; uint32_t current_b
[doris] 01/04: merge block to single block on join/set node
This is an automated email from the ASF dual-hosted git repository. panxiaolei pushed a commit to branch dev_join in repository https://gitbox.apache.org/repos/asf/doris.git commit 2e0778a3d87f7a0af33dd11ed3e914ddfd922ed6 Author: BiteThet AuthorDate: Mon Oct 16 18:43:26 2023 +0800 merge block to single block on join/set node update update update --- be/src/pipeline/exec/hashjoin_build_sink.cpp | 101 ++-- be/src/pipeline/exec/hashjoin_build_sink.h | 5 +- be/src/pipeline/exec/hashjoin_probe_operator.h | 4 +- be/src/pipeline/exec/set_sink_operator.cpp | 13 +- be/src/pipeline/exec/set_sink_operator.h | 2 +- be/src/pipeline/exec/set_source_operator.cpp | 6 +- be/src/pipeline/pipeline_x/dependency.h| 7 +- be/src/vec/common/hash_table/hash_map.h| 67 ++ be/src/vec/common/hash_table/hash_map_context.h| 6 +- .../vec/common/hash_table/hash_table_set_build.h | 9 +- be/src/vec/exec/join/join_op.h | 72 +++--- be/src/vec/exec/join/process_hash_table_probe.h| 9 +- .../vec/exec/join/process_hash_table_probe_impl.h | 257 +++-- be/src/vec/exec/join/vhash_join_node.cpp | 72 ++ be/src/vec/exec/join/vhash_join_node.h | 100 ++-- be/src/vec/exec/vset_operation_node.cpp| 69 +++--- be/src/vec/exec/vset_operation_node.h | 5 +- be/src/vec/runtime/shared_hash_table_controller.h | 11 +- 18 files changed, 256 insertions(+), 559 deletions(-) diff --git a/be/src/pipeline/exec/hashjoin_build_sink.cpp b/be/src/pipeline/exec/hashjoin_build_sink.cpp index 153882075b6..e7da32b0340 100644 --- a/be/src/pipeline/exec/hashjoin_build_sink.cpp +++ b/be/src/pipeline/exec/hashjoin_build_sink.cpp @@ -39,10 +39,7 @@ Overload(Callables&&... callables) -> Overload; HashJoinBuildSinkLocalState::HashJoinBuildSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state) -: JoinBuildSinkLocalState(parent, state), - _build_block_idx(0), - _build_side_mem_used(0), - _build_side_last_mem_used(0) {} +: JoinBuildSinkLocalState(parent, state) {} Status HashJoinBuildSinkLocalState::init(RuntimeState* state, LocalSinkStateInfo& info) { RETURN_IF_ERROR(JoinBuildSinkLocalState::init(state, info)); @@ -52,13 +49,7 @@ Status HashJoinBuildSinkLocalState::init(RuntimeState* state, LocalSinkStateInfo auto& p = _parent->cast(); _shared_state->join_op_variants = p._join_op_variants; if (p._is_broadcast_join && state->enable_share_hash_table_for_broadcast_join()) { -_shared_state->build_blocks = p._shared_hash_table_context->blocks; -} else { -_shared_state->build_blocks.reset(new std::vector()); -// avoid vector expand change block address. -// one block can store 4g data, _build_blocks can store 128*4g data. -// if probe data bigger than 512g, runtime filter maybe will core dump when insert data. - _shared_state->build_blocks->reserve(vectorized::HASH_JOIN_MAX_BUILD_BLOCK_COUNT); +_shared_state->build_block = p._shared_hash_table_context->block; } _shared_state->is_null_safe_eq_join = p._is_null_safe_eq_join; _shared_state->store_null_in_hash_table = p._store_null_in_hash_table; @@ -82,11 +73,6 @@ Status HashJoinBuildSinkLocalState::init(RuntimeState* state, LocalSinkStateInfo if (!_should_build_hash_table) { _shared_hash_table_dependency->block_writing(); p._shared_hashtable_controller->append_dependency(p.id(), _shared_hash_table_dependency); -} else if (p._is_broadcast_join) { -// avoid vector expand change block address. -// one block can store 4g data, _build_blocks can store 128*4g data. -// if probe data bigger than 512g, runtime filter maybe will core dump when insert data. - _shared_state->build_blocks->reserve(vectorized::HASH_JOIN_MAX_BUILD_BLOCK_COUNT); } _memory_usage_counter = ADD_LABEL_COUNTER(profile(), "MemoryUsage"); @@ -156,25 +142,24 @@ void HashJoinBuildSinkLocalState::init_short_circuit_for_probe() { _shared_state->short_circuit_for_probe = (_shared_state->_has_null_in_build_side && p._join_op == TJoinOp::NULL_AWARE_LEFT_ANTI_JOIN && !p._is_mark_join) || -(_shared_state->build_blocks->empty() && p._join_op == TJoinOp::INNER_JOIN && +(!_shared_state->build_block && p._join_op == TJoinOp::INNER_JOIN && !p._is_mark_join) || -(_shared_state->build_blocks->empty() && p._join_op == TJoinOp::LEFT_SEMI_JOIN && +(!_shared_state->build_block && p._join_op == TJoinOp::LEFT_SEMI_JOIN && !p._is_mark_join) || -(_shared_state->build_blocks->empty() && p._join_op == TJoinOp::RIGHT_OUTER_JOIN) || -
Re: [PR] [improvement](insert) Modify insert into table_id(xxx) to doris_inter… [doris]
doris-robot commented on PR #25662: URL: https://github.com/apache/doris/pull/25662#issuecomment-1772217857 TeamCity be ut coverage result: Function Coverage: 37.04% (8299/22408) Line Coverage: 29.16% (9/228594) Region Coverage: 27.81% (34596/124407) Branch Coverage: 24.41% (17577/72018) Coverage Report: http://coverage.selectdb-in.cc/coverage/32f2b9e86808aa9e455b82d700e24186c3c33e35_32f2b9e86808aa9e455b82d700e24186c3c33e35/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [FIX](resize) fix array and map offsets resize with default value [doris]
doris-robot commented on PR #25669: URL: https://github.com/apache/doris/pull/25669#issuecomment-1772217514 TeamCity be ut coverage result: Function Coverage: 37.08% (8309/22409) Line Coverage: 29.20% (66744/228598) Region Coverage: 27.84% (34639/124407) Branch Coverage: 24.43% (17591/72018) Coverage Report: http://coverage.selectdb-in.cc/coverage/e4709fd4bff8637b4422595677e2b236a8ae9d05_e4709fd4bff8637b4422595677e2b236a8ae9d05/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](function) support ip function ipv4stringtonum(ordefault, ornull), inet_aton [doris]
superdiaodiao commented on code in PR #25510: URL: https://github.com/apache/doris/pull/25510#discussion_r1365776957 ## gensrc/script/doris_builtins_functions.py: ## @@ -2004,6 +2004,12 @@ [['ipv4numtostring','inet_ntoa'], 'VARCHAR', ['SMALLINT'], 'ALWAYS_NULLABLE'], [['ipv4numtostring','inet_ntoa'], 'VARCHAR', ['INT'], 'ALWAYS_NULLABLE'], [['ipv4numtostring','inet_ntoa'], 'VARCHAR', ['BIGINT'], 'ALWAYS_NULLABLE'], +[['ipv4stringtonum','inet_aton'], 'BIGINT', ['VARCHAR'], 'ALWAYS_NOT_NULLABLE'], Review Comment: Yes, but this BE part is running so much time, so I have a plan to make the Nereids in another PR. Is it alright? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](function) support ip function ipv4stringtonum(ordefault, ornull), inet_aton [doris]
superdiaodiao commented on code in PR #25510: URL: https://github.com/apache/doris/pull/25510#discussion_r1365776957 ## gensrc/script/doris_builtins_functions.py: ## @@ -2004,6 +2004,12 @@ [['ipv4numtostring','inet_ntoa'], 'VARCHAR', ['SMALLINT'], 'ALWAYS_NULLABLE'], [['ipv4numtostring','inet_ntoa'], 'VARCHAR', ['INT'], 'ALWAYS_NULLABLE'], [['ipv4numtostring','inet_ntoa'], 'VARCHAR', ['BIGINT'], 'ALWAYS_NULLABLE'], +[['ipv4stringtonum','inet_aton'], 'BIGINT', ['VARCHAR'], 'ALWAYS_NOT_NULLABLE'], Review Comment: Yes, but this BE part runs so much time, so I have a plan to make the Nereids in another PR. Is it alright? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ut) fix some uts [doris]
hello-stephen commented on PR #25674: URL: https://github.com/apache/doris/pull/25674#issuecomment-1772224944 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 48.68 seconds stream load tsv: 576 seconds loaded 74807831229 Bytes, about 123 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.6 seconds inserted 1000 Rows, about 337K ops/s storage size: 17162848744 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](auth)Fix no priv to insert to mysql catalog [doris]
zddr opened a new pull request, #25678: URL: https://github.com/apache/doris/pull/25678 ## Proposed changes Issue Number: close #xxx fix user can not insert to mysql catalog even if he has load permission ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids)do not generate min-max runtime filter for A<=>B and A!=B [doris]
hello-stephen commented on PR #25676: URL: https://github.com/apache/doris/pull/25676#issuecomment-1772226533 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvements](status) catch some error status [doris]
doris-robot commented on PR #25677: URL: https://github.com/apache/doris/pull/25677#issuecomment-1772228576 TeamCity be ut coverage result: Function Coverage: 37.04% (8301/22409) Line Coverage: 29.17% (66693/228605) Region Coverage: 27.80% (34640/124591) Branch Coverage: 24.41% (17585/72028) Coverage Report: http://coverage.selectdb-in.cc/coverage/4926afbe7c75ccf2c71540669d37d6c353e80032_4926afbe7c75ccf2c71540669d37d6c353e80032/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](regression-test) fix bad case in s3 tvf load test [doris]
TangSiyang2001 opened a new pull request, #25679: URL: https://github.com/apache/doris/pull/25679 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression-test) fix bad case in s3 tvf load test [doris]
TangSiyang2001 commented on PR #25679: URL: https://github.com/apache/doris/pull/25679#issuecomment-1772228802 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [FIX](resize) fix array and map offsets resize with default value [doris]
xiaokang commented on code in PR #25669: URL: https://github.com/apache/doris/pull/25669#discussion_r1366585254 ## be/src/vec/columns/column_array.cpp: ## @@ -422,7 +422,8 @@ void ColumnArray::reserve(size_t n) { //please check you real need size in data column, because it's maybe need greater size when data is string column void ColumnArray::resize(size_t n) { -get_offsets().resize(n); +auto last_off = get_offsets().back(); +get_offsets().resize_fill(n, last_off); Review Comment: We need to check if resize_fill will modify old offsets value. ## be/src/vec/columns/column_array.cpp: ## @@ -422,7 +422,8 @@ void ColumnArray::reserve(size_t n) { //please check you real need size in data column, because it's maybe need greater size when data is string column void ColumnArray::resize(size_t n) { Review Comment: Is there the same problem for ColumnStruct? ## be/src/vec/columns/column_map.cpp: ## @@ -450,7 +450,8 @@ void ColumnMap::reserve(size_t n) { } void ColumnMap::resize(size_t n) { -get_offsets().resize(n); +auto last_off = get_offsets().back(); +get_offsets().resize_fill(n, last_off); keys_column->resize(n); Review Comment: should not call resize on keys and values column ## be/src/vec/columns/column_array.cpp: ## @@ -422,7 +422,8 @@ void ColumnArray::reserve(size_t n) { //please check you real need size in data column, because it's maybe need greater size when data is string column void ColumnArray::resize(size_t n) { -get_offsets().resize(n); +auto last_off = get_offsets().back(); +get_offsets().resize_fill(n, last_off); get_data().resize(n); Review Comment: I think it should not call get_data().resize(n). Just append offsets with the same value. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] test [doris]
github-actions[bot] commented on code in PR #25493: URL: https://github.com/apache/doris/pull/25493#discussion_r1366588162 ## be/src/vec/columns/column_struct.cpp: ## @@ -225,8 +225,8 @@ void ColumnStruct::update_crcs_with_value(uint32_t* __restrict hash, PrimitiveTy } } -void ColumnStruct::insert_indices_from(const IColumn& src, const int* indices_begin, - const int* indices_end) { +void ColumnStruct::insert_indices_from(const IColumn& src, const uint32_t* __restrict indices_begin, Review Comment: warning: method 'insert_indices_from' can be made static [readability-convert-member-functions-to-static] be/src/vec/columns/column_struct.h:123: ```diff - void insert_indices_from(const IColumn& src, const uint32* __restrict indices_begin, + static void insert_indices_from(const IColumn& src, const uint32* __restrict indices_begin, ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix][branch-2.0](regression-test) fix bad case in s3 tvf load test [doris]
TangSiyang2001 commented on PR #25680: URL: https://github.com/apache/doris/pull/25680#issuecomment-1772233211 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-thirdparty] branch clucene updated: [Fix](PFOR) fix PFOR 'illegal operand' error for none SSE4.2/AVX cpu (#129)
This is an automated email from the ASF dual-hosted git repository. kxiao pushed a commit to branch clucene in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git The following commit(s) were added to refs/heads/clucene by this push: new dd2a9c92 [Fix](PFOR) fix PFOR 'illegal operand' error for none SSE4.2/AVX cpu (#129) dd2a9c92 is described below commit dd2a9c929275a0091a59dd7acd2cdf8ad7419da6 Author: airborne12 AuthorDate: Fri Oct 20 15:35:10 2023 +0800 [Fix](PFOR) fix PFOR 'illegal operand' error for none SSE4.2/AVX cpu (#129) --- src/core/CLucene/util/PFORUtil.cpp | 64 +++--- 1 file changed, 60 insertions(+), 4 deletions(-) diff --git a/src/core/CLucene/util/PFORUtil.cpp b/src/core/CLucene/util/PFORUtil.cpp index a9a32dfb..e290ea5b 100644 --- a/src/core/CLucene/util/PFORUtil.cpp +++ b/src/core/CLucene/util/PFORUtil.cpp @@ -29,6 +29,61 @@ ENC_FUNC g_p4nd1enc; ENC_FUNC g_p4nzenc; } // anonymous namespace +size_t DefaultDEC(unsigned char *__restrict in, size_t n, uint32_t *__restrict out) { +size_t bufferSize = 0; +for (uint32_t i = 0; i < n; i++) { +uint8_t b = in[bufferSize++]; +int32_t docCode = b & 0x7F; +for (int32_t shift = 7; (b & 0x80) != 0; shift += 7) { +b = in[bufferSize++]; +docCode |= (b & 0x7F) << shift; +} +out[i] = docCode; +} +} + +size_t DefaultDDEC(unsigned char *__restrict in, size_t n, uint32_t *__restrict out) { +uint32_t docDelta = 0; +size_t bufferSize = 0; +for (uint32_t i = 0; i < n; i++) { +uint8_t b = in[bufferSize++]; +int32_t docCode = b & 0x7F; +for (int32_t shift = 7; (b & 0x80) != 0; shift += 7) { +b = in[bufferSize++]; +docCode |= (b & 0x7F) << shift; +} +docDelta += (docCode >> 1); +out[i] = docDelta; +} +} + +size_t DefaultDENC(uint32_t *__restrict in, size_t n, unsigned char *__restrict out) { +int outIndex = 0; +uint32_t lastDoc = 0; +for (int32_t i = 0; i < n; i++) { +uint32_t curDoc = in[i]; +uint32_t delta = curDoc - lastDoc; +while ((delta & ~0x7F) != 0) { +out[outIndex++] = (uint8_t)((delta & 0x7f) | 0x80); +delta >>= 7; //doing unsigned shift +} +out[outIndex++] = (uint8_t)delta; +lastDoc = curDoc; +} +} + +size_t DefaultENC(uint32_t *__restrict in, size_t n, unsigned char *__restrict out) { +int outIndex = 0; +for (int32_t i = 0; i < n; i++) { +uint32_t curDoc = in[i]; +while ((curDoc & ~0x7F) != 0) { +out[outIndex++] = (uint8_t)((curDoc & 0x7f) | 0x80); +curDoc >>= 7; //doing unsigned shift +} +out[outIndex++] = (uint8_t)curDoc; +} +} + __attribute__((constructor)) void SelectPFORFunctions() { #if (defined(__i386) || defined(__x86_64__)) uint32_t eax, ebx, ecx, edx; @@ -48,10 +103,10 @@ __attribute__((constructor)) void SelectPFORFunctions() { g_p4nd1enc = p4nd1enc128v32; g_p4nzenc = p4nzenc128v32; } else { -g_p4nd1dec = p4nd1dec32; -g_p4nzdec = p4nzdec32; -g_p4nd1enc = p4nd1enc32; -g_p4nzenc = p4nzenc32; +g_p4nd1dec = DefaultDDEC; +g_p4nzdec = DefaultDEC; +g_p4nd1enc = DefaultDENC; +g_p4nzenc = DefaultENC; } #endif #else @@ -65,6 +120,7 @@ __attribute__((constructor)) void SelectPFORFunctions() { size_t P4DEC(unsigned char *__restrict in, size_t n, uint32_t *__restrict out) { return g_p4nd1dec(in, n, out); } + size_t P4NZDEC(unsigned char *__restrict in, size_t n, uint32_t *__restrict out) { return g_p4nzdec(in, n, out); } - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix][branch-2.0](regression-test) fix bad case in s3 tvf load test [doris]
TangSiyang2001 opened a new pull request, #25680: URL: https://github.com/apache/doris/pull/25680 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](PFOR) fix PFOR 'illegal operand' error for none SSE4.2/AVX cpu [doris-thirdparty]
xiaokang merged PR #129: URL: https://github.com/apache/doris-thirdparty/pull/129 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Bug] can not compile on macos [doris]
felixwluo commented on issue #24825: URL: https://github.com/apache/doris/issues/24825#issuecomment-1772233643 To update xcode, run the "softwareupdate --install -a" command -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids)fix bug of duplicate name of inline view [doris]
starocean999 merged PR #25627: URL: https://github.com/apache/doris/pull/25627 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](nereids)fix bug of duplicate name of inline view (#25627)
This is an automated email from the ASF dual-hosted git repository. starocean999 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new a11cde7beec [fix](nereids)fix bug of duplicate name of inline view (#25627) a11cde7beec is described below commit a11cde7beec413b9760cdf9130dd93ff25e36a5a Author: starocean999 <40539150+starocean...@users.noreply.github.com> AuthorDate: Fri Oct 20 15:37:44 2023 +0800 [fix](nereids)fix bug of duplicate name of inline view (#25627) --- .../apache/doris/nereids/analyzer/UnboundSlot.java | 5 + .../nereids/rules/analysis/BindExpression.java | 4 +- .../doris/nereids/trees/expressions/Alias.java | 4 +- .../trees/expressions/ArrayItemReference.java | 3 +- .../doris/nereids/trees/expressions/Slot.java | 4 + .../nereids/trees/expressions/SlotReference.java | 41 +--- .../subquery/test_duplicate_name_in_view.groovy| 108 - 7 files changed, 153 insertions(+), 16 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundSlot.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundSlot.java index 2c8b03e647e..8fa85966dc0 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundSlot.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundSlot.java @@ -63,6 +63,11 @@ public class UnboundSlot extends Slot implements Unbound, PropagateNullable { return nameParts.subList(0, nameParts.size() - 1); } +@Override +public String getInternalName() { +return getName(); +} + @Override public String toSql() { return nameParts.stream().map(Utils::quoteIfNeeded).reduce((left, right) -> left + "." + right).orElse(""); diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java index d09432029c7..4c51578ef92 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java @@ -709,11 +709,11 @@ public class BindExpression implements AnalysisRuleFactory { private void checkSameNameSlot(List childOutputs, String subQueryAlias) { Set nameSlots = new HashSet<>(); for (Slot s : childOutputs) { -if (nameSlots.contains(s.getName())) { +if (nameSlots.contains(s.getInternalName())) { throw new AnalysisException("Duplicated inline view column alias: '" + s.getName() + "'" + " in inline view: '" + subQueryAlias + "'"); } else { -nameSlots.add(s.getName()); +nameSlots.add(s.getInternalName()); } } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Alias.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Alias.java index 0e91dec26c3..b02c968baba 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Alias.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Alias.java @@ -27,6 +27,7 @@ import com.google.common.collect.ImmutableList; import java.util.List; import java.util.Objects; +import java.util.Optional; /** * Expression for alias, such as col1 as c1. @@ -73,7 +74,8 @@ public class Alias extends NamedExpression implements UnaryExpression { return new SlotReference(exprId, name, child().getDataType(), child().nullable(), qualifier, child() instanceof SlotReference ? ((SlotReference) child()).getColumn().orElse(null) -: null); +: null, +nameFromChild ? Optional.of(child().toString()) : Optional.of(name)); } @Override diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/ArrayItemReference.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/ArrayItemReference.java index afebaa3016d..95c29eac87a 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/ArrayItemReference.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/ArrayItemReference.java @@ -28,6 +28,7 @@ import com.google.common.collect.ImmutableList; import java.util.List; import java.util.Objects; +import java.util.Optional; /** * it is item from array, which used in lambda function @@ -138,7 +139,7 @@ public class ArrayItemReference extends NamedExpression implements ExpectsInputT * @param nullable true if nullable */ public ArrayItemSlot(ExprId exprId, String name, DataType dataType, boolean nullable) { -
Re: [PR] [improvement](insert) Modify insert into table_id(xxx) to doris_inter… [doris]
doris-robot commented on PR #25662: URL: https://github.com/apache/doris/pull/25662#issuecomment-1772244041 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.27 seconds stream load tsv: 551 seconds loaded 74807831229 Bytes, about 129 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 29.1 seconds inserted 1000 Rows, about 343K ops/s storage size: 17162064231 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](outfile)support underscore prefix when select outfile [doris]
doris-robot commented on PR #25395: URL: https://github.com/apache/doris/pull/25395#issuecomment-1772244323 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.27 seconds stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.9 seconds inserted 1000 Rows, about 346K ops/s storage size: 17162544885 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pipeline](load) disable pipeline load [doris]
doris-robot commented on PR #25667: URL: https://github.com/apache/doris/pull/25667#issuecomment-1772244629 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.34 seconds stream load tsv: 573 seconds loaded 74807831229 Bytes, about 124 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.9 seconds inserted 1000 Rows, about 346K ops/s storage size: 17162364730 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [FIX](resize) fix array and map offsets resize with default value [doris]
doris-robot commented on PR #25669: URL: https://github.com/apache/doris/pull/25669#issuecomment-1772244414 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.54 seconds stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 28.6 seconds inserted 1000 Rows, about 349K ops/s storage size: 17162350704 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](create-table-like)Fix create table like error, the converted table field COMMENT contains extra characters [doris]
doris-robot commented on PR #25587: URL: https://github.com/apache/doris/pull/25587#issuecomment-1772245047 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.54 seconds stream load tsv: 556 seconds loaded 74807831229 Bytes, about 128 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 28.7 seconds inserted 1000 Rows, about 348K ops/s storage size: 17162250921 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](paimon)paimon catalog supports complex types [doris]
doris-robot commented on PR #25364: URL: https://github.com/apache/doris/pull/25364#issuecomment-1772245437 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.65 seconds stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.1 seconds inserted 1000 Rows, about 343K ops/s storage size: 17162353741 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](restore tablet)fix restore_tablet() null_ptr [doris]
xuwei0912 opened a new pull request, #25682: URL: https://github.com/apache/doris/pull/25682 use StorageEngine::instance() ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](auth)Fix no priv to insert to mysql catalog [doris]
zddr commented on PR #25678: URL: https://github.com/apache/doris/pull/25678#issuecomment-1772251178 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](restore tablet)fix restore_tablet() null_ptr [doris]
xuwei0912 commented on PR #25682: URL: https://github.com/apache/doris/pull/25682#issuecomment-1772251484 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](stats) analyze spec col only [doris]
doris-robot commented on PR #25660: URL: https://github.com/apache/doris/pull/25660#issuecomment-1772253022 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 47.22 seconds stream load tsv: 554 seconds loaded 74807831229 Bytes, about 128 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.7 seconds inserted 1000 Rows, about 348K ops/s storage size: 17162307643 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](restore tablet)fix restore_tablet() null_ptr [doris]
github-actions[bot] commented on PR #25682: URL: https://github.com/apache/doris/pull/25682#issuecomment-1772260460 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](decimal) support decimal256 [doris]
jacktengg commented on PR #25386: URL: https://github.com/apache/doris/pull/25386#issuecomment-1772265653 run external -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](planner) collect_list and collect_set should always not null [doris]
morrySnow opened a new pull request, #25683: URL: https://github.com/apache/doris/pull/25683 cherry-pick from master PR: #25592 commit id: 26f8c7e35260010da5132848e1fc80f6ec430736 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](planner) collect_list and collect_set should always not null [doris]
morrySnow commented on PR #25683: URL: https://github.com/apache/doris/pull/25683#issuecomment-1772266281 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids)do not generate min-max runtime filter for A<=>B and A!=B [doris]
doris-robot commented on PR #25676: URL: https://github.com/apache/doris/pull/25676#issuecomment-1772268087 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.38 seconds stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.3 seconds inserted 1000 Rows, about 341K ops/s storage size: 17162106271 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](restore tablet)fix restore_tablet() null_ptr [doris]
xuwei0912 closed pull request #25682: [fix](restore tablet)fix restore_tablet() null_ptr URL: https://github.com/apache/doris/pull/25682 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](jni) unified jni framework for java udaf [doris]
github-actions[bot] commented on PR #25591: URL: https://github.com/apache/doris/pull/25591#issuecomment-1772271467 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvements](status) catch some error status [doris]
doris-robot commented on PR #25677: URL: https://github.com/apache/doris/pull/25677#issuecomment-1772270094 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.17 seconds stream load tsv: 560 seconds loaded 74807831229 Bytes, about 127 MB/s stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.7 seconds inserted 1000 Rows, about 348K ops/s storage size: 17162134711 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](jni) unified jni framework for java udaf [doris]
github-actions[bot] commented on PR #25591: URL: https://github.com/apache/doris/pull/25591#issuecomment-1772271547 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](insert) Modify insert into table_id(xxx) to doris_inter… [doris]
morrySnow commented on code in PR #25662: URL: https://github.com/apache/doris/pull/25662#discussion_r1366622347 ## fe/fe-core/src/main/cup/sql_parser.cup: ## @@ -347,6 +347,7 @@ terminal String KW_DISTRIBUTION, KW_DIV, KW_DO, +KW_DORIS_INTERNAL_TABLE_ID, Review Comment: should add into non-reserved keyword list, and add same keyword for Nereids in `src/main/antlr4/org/apache/doris/nereids/DorisParser.g4` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](planner) collect_list and collect_set should always not null [doris]
github-actions[bot] commented on PR #25683: URL: https://github.com/apache/doris/pull/25683#issuecomment-1772276480 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [performance](pipelineX) optimize get local state [doris]
doris-robot commented on PR #25675: URL: https://github.com/apache/doris/pull/25675#issuecomment-1772278716 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.66 seconds stream load tsv: 562 seconds loaded 74807831229 Bytes, about 126 MB/s stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 29.0 seconds inserted 1000 Rows, about 344K ops/s storage size: 17162120518 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](move-memtable) fix inverted index condition [doris]
kaijchen opened a new pull request, #25684: URL: https://github.com/apache/doris/pull/25684 ## Proposed changes Fix reversed inverted index check condition. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](jni) unified jni framework for java udaf [doris]
AshinGau merged PR #25591: URL: https://github.com/apache/doris/pull/25591 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](move-memtable) inverted index condition is reversed [doris]
github-actions[bot] commented on PR #25684: URL: https://github.com/apache/doris/pull/25684#issuecomment-1772285767 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](move-memtable) inverted index condition is reversed [doris]
dataroaring commented on PR #25684: URL: https://github.com/apache/doris/pull/25684#issuecomment-1772285120 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](move-memtable) inverted index condition is reversed [doris]
github-actions[bot] commented on PR #25684: URL: https://github.com/apache/doris/pull/25684#issuecomment-1772285715 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) Set to use nereids planner in select aggregation expr without from scene when regression-test [doris]
github-actions[bot] commented on PR #25665: URL: https://github.com/apache/doris/pull/25665#issuecomment-1772287597 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) Set to use nereids planner in select aggregation expr without from scene when regression-test [doris]
github-actions[bot] commented on PR #25665: URL: https://github.com/apache/doris/pull/25665#issuecomment-1772287548 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [nereids](array_agg)support array_agg in nereids [doris]
morrySnow commented on code in PR #25630: URL: https://github.com/apache/doris/pull/25630#discussion_r1366635484 ## fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/ArrayAgg.java: ## @@ -0,0 +1,61 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.trees.expressions.functions.agg; + +import org.apache.doris.catalog.FunctionSignature; +import org.apache.doris.nereids.trees.expressions.Expression; +import org.apache.doris.nereids.trees.expressions.functions.AlwaysNotNullable; +import org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature; +import org.apache.doris.nereids.trees.expressions.shape.UnaryExpression; +import org.apache.doris.nereids.types.ArrayType; +import org.apache.doris.nereids.types.coercion.AnyDataType; +import org.apache.doris.nereids.types.coercion.FollowToAnyDataType; + +import com.google.common.base.Preconditions; +import com.google.common.collect.ImmutableList; + +import java.util.List; + +/** + * AggregateFunction 'array_agg'. + */ +public class ArrayAgg extends AggregateFunction +implements UnaryExpression, ExplicitlyCastableSignature, AlwaysNotNullable { +public static final List SIGNATURES = ImmutableList.of( +FunctionSignature.ret(ArrayType.of(new FollowToAnyDataType(0))).args(new AnyDataType(0)) +); + +public ArrayAgg(Expression arg0) { +super("array_agg", arg0); +} + +public ArrayAgg(boolean distinct, Expression arg0) { +super("array_agg", distinct, arg0); +} + +@Override +public List getSignatures() { +return SIGNATURES; +} + +@Override +public AggregateFunction withDistinctAndChildren(boolean distinct, List children) { +Preconditions.checkArgument(children.size() == 1); +return new ArrayAgg(distinct, children.get(0)); +} Review Comment: need override accept function -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Fix](segment) need to rebuild col_id_to_predicates when true predicates encountered [doris]
airborne12 opened a new pull request, #25685: URL: https://github.com/apache/doris/pull/25685 ## Proposed changes Issue Number: close #xxx relate pr: #25366 ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](segment) need to rebuild col_id_to_predicates when true predicates encountered [doris]
airborne12 commented on PR #25685: URL: https://github.com/apache/doris/pull/25685#issuecomment-1772292082 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](auth)Fix no priv to insert to mysql catalog [doris]
doris-robot commented on PR #25678: URL: https://github.com/apache/doris/pull/25678#issuecomment-1772293001 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.06 seconds stream load tsv: 552 seconds loaded 74807831229 Bytes, about 129 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 29.0 seconds inserted 1000 Rows, about 344K ops/s storage size: 17161990921 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-2.0 updated: [fix](regression-test) fix bad case in s3 tvf load test in 2.0 (#25680)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-2.0 by this push: new fe4a06a0096 [fix](regression-test) fix bad case in s3 tvf load test in 2.0 (#25680) fe4a06a0096 is described below commit fe4a06a00967780ab51372007f05a18e6a1c5cf2 Author: Siyang Tang <82279870+tangsiyang2...@users.noreply.github.com> AuthorDate: Fri Oct 20 16:21:11 2023 +0800 [fix](regression-test) fix bad case in s3 tvf load test in 2.0 (#25680) --- regression-test/suites/load_p0/tvf/test_s3_tvf.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/regression-test/suites/load_p0/tvf/test_s3_tvf.groovy b/regression-test/suites/load_p0/tvf/test_s3_tvf.groovy index 097c551ea2b..391b76d37eb 100644 --- a/regression-test/suites/load_p0/tvf/test_s3_tvf.groovy +++ b/regression-test/suites/load_p0/tvf/test_s3_tvf.groovy @@ -150,7 +150,7 @@ suite("test_s3_tvf", "load_p0") { .addProperty("column_separator", "|")) for(String table : arrayTables) { -attributeList.add(new TvfAttribute(table, ["k00", "k01", "k02", "k03", "k04", "k05", "k06", "k07", "k08", "k09", "k10", "k11", "k12", "k13", "k14", "k15", "k16", "k17"], "", "", true) +attributeList.add(new TvfAttribute(table, ["k00", "k01", "k02", "k03", "k04", "k05", "k06", "k07", "k08", "k09", "k10", "k11", "k12", "k13", "k14", "k15", "k16", "k17", "kd01", "kd02", "kd03", "kd04", "kd05", "kd06", "kd07", "kd08", "kd09", "kd10", "kd11", "kd12", "kd13", "kd14", "kd15", "kd16"], "", "", true) .addProperty("uri", "s3://doris-build-1308700295.cos.ap-beijing.myqcloud.com/regression/load/data/basic_array_data_with_errors.csv") .addProperty("format", "csv") .addProperty("column_separator", "|")) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](move-memtable) inverted index filter is reversed [doris]
github-actions[bot] commented on PR #25684: URL: https://github.com/apache/doris/pull/25684#issuecomment-1772295311 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression-test) fix bad case in s3 tvf load test [doris]
github-actions[bot] commented on PR #25679: URL: https://github.com/apache/doris/pull/25679#issuecomment-1772295688 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression-test) fix bad case in s3 tvf load test [doris]
github-actions[bot] commented on PR #25679: URL: https://github.com/apache/doris/pull/25679#issuecomment-1772295764 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression-test) fix bad case in s3 tvf load test in 2.0 [doris]
dataroaring merged PR #25680: URL: https://github.com/apache/doris/pull/25680 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [opt](profile) set datalake profile level as 1 [doris]
AshinGau opened a new pull request, #25686: URL: https://github.com/apache/doris/pull/25686 ## Proposed changes Follow https://github.com/apache/doris/pull/25491, only the profile marked as 1 will be shown in simplified profile. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](profile) set datalake profile level as 1 [doris]
AshinGau commented on PR #25686: URL: https://github.com/apache/doris/pull/25686#issuecomment-1772298043 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[I] [Bug] doris版本 1.2.7.1 版本 全新安装fe 可以正常启动,但是be启动不了 [doris]
AndyYanQS opened a new issue, #25687: URL: https://github.com/apache/doris/issues/25687 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version [fe.log](https://github.com/apache/doris/files/13051889/fe.log) [be.INFO.log](https://github.com/apache/doris/files/13051907/be.INFO.log) 配置如下 [be.conf.log](https://github.com/apache/doris/files/13051931/be.conf.log) [fe.conf.log](https://github.com/apache/doris/files/13051932/fe.conf.log) ### What's Wrong? doris版本 1.2.7.1 版本 全新安装fe 可以正常启动,但是be启动不了,已经手动添加了ALTER SYSTEM ADD BACKEND 三个节点了,还是不行 ### What You Expected? 辛苦帮忙看看哈 ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] (selectdb-cloud) Rebuild failed channel to avoid connection refused [doris]
w41ter opened a new pull request, #25688: URL: https://github.com/apache/doris/pull/25688 We cannot judge the progress of reconnecting the underlying channel of grpc. In the worst case, it may take two minutes. But we couldn't stand the connection refused for two minutes, so we rebuilt the channel directly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] (selectdb-cloud) Rebuild failed channel to avoid connection refused [doris]
w41ter commented on PR #25688: URL: https://github.com/apache/doris/pull/25688#issuecomment-1772301163 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](auth)Fix no priv to insert to mysql catalog [doris]
zddr commented on PR #25678: URL: https://github.com/apache/doris/pull/25678#issuecomment-1772301589 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](segment) need to rebuild col_id_to_predicates when true predicates encountered [doris]
github-actions[bot] commented on PR #25685: URL: https://github.com/apache/doris/pull/25685#issuecomment-1772303037 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](profile) set datalake profile level as 1 [doris]
github-actions[bot] commented on PR #25686: URL: https://github.com/apache/doris/pull/25686#issuecomment-1772308965 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] (selectdb-cloud) Rebuild failed channel to avoid connection refused [doris]
github-actions[bot] commented on PR #25688: URL: https://github.com/apache/doris/pull/25688#issuecomment-1772310221 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](planner) collect_list and collect_set should always not null [doris]
hello-stephen commented on PR #25683: URL: https://github.com/apache/doris/pull/25683#issuecomment-1772310727 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 48.54 seconds stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.7 seconds inserted 1000 Rows, about 336K ops/s storage size: 17163060685 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](restore tablet)fix restore_tablet() null_ptr [doris]
hello-stephen commented on PR #25682: URL: https://github.com/apache/doris/pull/25682#issuecomment-1772310964 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 48.34 seconds stream load tsv: 583 seconds loaded 74807831229 Bytes, about 122 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 29.6 seconds inserted 1000 Rows, about 337K ops/s storage size: 17162582060 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [typo](doc)Adjust tablet_rowset_stale_sweep_time_sec parameter defaul… [doris]
DongLiang-0 commented on PR #25584: URL: https://github.com/apache/doris/pull/25584#issuecomment-1772312332 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [regression-test](modify) enable and disable suitable test cases [doris]
shuke987 commented on PR #25658: URL: https://github.com/apache/doris/pull/25658#issuecomment-1772313266 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pipeline](load) disable pipeline load [doris]
Gabriel39 closed pull request #25667: [pipeline](load) disable pipeline load URL: https://github.com/apache/doris/pull/25667 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [nereids](array_agg)support array_agg in nereids [doris]
LemonLiTree commented on code in PR #25630: URL: https://github.com/apache/doris/pull/25630#discussion_r1366658973 ## fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/ArrayAgg.java: ## @@ -0,0 +1,61 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.trees.expressions.functions.agg; + +import org.apache.doris.catalog.FunctionSignature; +import org.apache.doris.nereids.trees.expressions.Expression; +import org.apache.doris.nereids.trees.expressions.functions.AlwaysNotNullable; +import org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature; +import org.apache.doris.nereids.trees.expressions.shape.UnaryExpression; +import org.apache.doris.nereids.types.ArrayType; +import org.apache.doris.nereids.types.coercion.AnyDataType; +import org.apache.doris.nereids.types.coercion.FollowToAnyDataType; + +import com.google.common.base.Preconditions; +import com.google.common.collect.ImmutableList; + +import java.util.List; + +/** + * AggregateFunction 'array_agg'. + */ +public class ArrayAgg extends AggregateFunction +implements UnaryExpression, ExplicitlyCastableSignature, AlwaysNotNullable { +public static final List SIGNATURES = ImmutableList.of( +FunctionSignature.ret(ArrayType.of(new FollowToAnyDataType(0))).args(new AnyDataType(0)) +); + +public ArrayAgg(Expression arg0) { +super("array_agg", arg0); +} + +public ArrayAgg(boolean distinct, Expression arg0) { +super("array_agg", distinct, arg0); +} + +@Override +public List getSignatures() { +return SIGNATURES; +} + +@Override +public AggregateFunction withDistinctAndChildren(boolean distinct, List children) { +Preconditions.checkArgument(children.size() == 1); +return new ArrayAgg(distinct, children.get(0)); +} Review Comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature](datatype) Add IPv4/v6 data type for doris [doris]
sjyango commented on PR #24965: URL: https://github.com/apache/doris/pull/24965#issuecomment-1772317959 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](move-memtable) inverted index filter is reversed [doris]
doris-robot commented on PR #25684: URL: https://github.com/apache/doris/pull/25684#issuecomment-1772320936 TeamCity be ut coverage result: Function Coverage: 37.03% (8299/22410) Line Coverage: 29.19% (66675/228388) Region Coverage: 27.82% (34601/124360) Branch Coverage: 24.41% (17572/71996) Coverage Report: http://coverage.selectdb-in.cc/coverage/b2ecf5788b549af4bbc75da1c7f37f5e58945cba_b2ecf5788b549af4bbc75da1c7f37f5e58945cba/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] set default docker list [doris]
zhangguoqiang666 opened a new pull request, #25689: URL: https://github.com/apache/doris/pull/25689 ## Proposed changes set default docker list ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature](datatype) Add IPv4/v6 data type for doris [doris]
sjyango commented on PR #24965: URL: https://github.com/apache/doris/pull/24965#issuecomment-1772323345 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](segment) need to rebuild col_id_to_predicates when true predicates encountered [doris]
doris-robot commented on PR #25685: URL: https://github.com/apache/doris/pull/25685#issuecomment-1772326707 TeamCity be ut coverage result: Function Coverage: 37.03% (8299/22410) Line Coverage: 29.19% (66672/228399) Region Coverage: 27.82% (34594/124363) Branch Coverage: 24.41% (17576/72000) Coverage Report: http://coverage.selectdb-in.cc/coverage/56e60ccfb471136aae64843365672c6df471672c_56e60ccfb471136aae64843365672c6df471672c/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](load) patch a quick fix of loading failure [doris]
zclllyybb opened a new pull request, #25690: URL: https://github.com/apache/doris/pull/25690 ## Proposed changes Issue Number: close #xxx This will decrease the load fail probability ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature](datatype) Add IPv4/v6 data type for doris [doris]
github-actions[bot] commented on code in PR #24965: URL: https://github.com/apache/doris/pull/24965#discussion_r1366675497 ## be/src/olap/types.h: ## @@ -957,6 +969,103 @@ struct FieldTypeTraits } }; +template <> +struct FieldTypeTraits +: public BaseFieldtypeTraits { +static Status from_string(void* buf, const std::string& scan_key, const int precision, + const int scale) { +StringParser::ParseResult result = StringParser::PARSE_SUCCESS; +uint32_t value = StringParser::string_to_unsigned_int(scan_key.c_str(), scan_key.size(), &result); + +if (result == StringParser::PARSE_FAILURE) { +return Status::Error( +"FieldTypeTraits::from_string meet PARSE_FAILURE"); +} +*reinterpret_cast(buf) = value; +return Status::OK(); +} + +static std::string to_string(const void* src) { +uint32_t value = *reinterpret_cast(src); +std::stringstream ss; +ss << ((value >> 24) & 0xFF) << '.' + << ((value >> 16) & 0xFF) << '.' + << ((value >> 8) & 0xFF) << '.' + << (value & 0xFF); +return ss.str(); +} +}; + + +template <> +struct FieldTypeTraits +: public BaseFieldtypeTraits { +static Status from_string(void* buf, const std::string& scan_key, const int precision, + const int scale) { +std::istringstream iss(scan_key); +std::string token; +uint128_t result = 0; +int count = 0; + +while (std::getline(iss, token, ':')) { +if (token.empty()) { +count += 8 - count; +break; +} + +if (count > 8) { +return Status::Error( +"FieldTypeTraits::from_string meet PARSE_FAILURE"); +} + +uint16_t value = 0; +std::istringstream ss(token); +if (!(ss >> std::hex >> value)) { +return Status::Error( +"FieldTypeTraits::from_string meet PARSE_FAILURE"); +} + +result = (result << 16) | value; +count++; +} + +if (count < 8) { +return Status::Error( +"FieldTypeTraits::from_string meet PARSE_FAILURE"); +} + +*reinterpret_cast(buf) = result; +return Status::OK(); +} + +static std::string to_string(const void* src) { +std::stringstream result; +uint128_t ipv6 = *reinterpret_cast(src); + +for (int i = 0; i < 8; i++) { +uint16_t part = static_cast((ipv6 >> (112 - i * 16)) & 0x); +result << std::to_string(part); +if (i != 7) { +result << ":"; +} +} + +return result.str(); +} + +static void set_to_max(void* buf) { +*reinterpret_cast(buf) = +static_cast(99ll) * 10ll * 1000ll + +static_cast(9ll) * 1000ll + 999ll; Review Comment: warning: 1000ll is a magic number; consider replacing it with a named constant [readability-magic-numbers] ```cpp static_cast(9ll) * 1000ll + 999ll; ^ ``` ## be/src/olap/types.h: ## @@ -957,6 +969,103 @@ struct FieldTypeTraits } }; +template <> +struct FieldTypeTraits +: public BaseFieldtypeTraits { +static Status from_string(void* buf, const std::string& scan_key, const int precision, + const int scale) { +StringParser::ParseResult result = StringParser::PARSE_SUCCESS; +uint32_t value = StringParser::string_to_unsigned_int(scan_key.c_str(), scan_key.size(), &result); + +if (result == StringParser::PARSE_FAILURE) { +return Status::Error( +"FieldTypeTraits::from_string meet PARSE_FAILURE"); +} +*reinterpret_cast(buf) = value; +return Status::OK(); +} + +static std::string to_string(const void* src) { +uint32_t value = *reinterpret_cast(src); +std::stringstream ss; +ss << ((value >> 24) & 0xFF) << '.' + << ((value >> 16) & 0xFF) << '.' + << ((value >> 8) & 0xFF) << '.' + << (value & 0xFF); +return ss.str(); +} +}; + + +template <> +struct FieldTypeTraits +: public BaseFieldtypeTraits { +static Status from_string(void* buf, const std::string& scan_key, const int precision, + const int scale) { +std::istringstream iss(scan_key); +std::string token; +uint128_t result = 0; +int count = 0; + +while (std::getline(iss, token, ':')) { +if (token.empty()) { +count += 8 - count; +
Re: [PR] [improvement](insert) Modify insert into table_id(xxx) to doris_inter… [doris]
mymeiyi commented on PR #25662: URL: https://github.com/apache/doris/pull/25662#issuecomment-1772334732 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](load) patch a quick fix of loading failure [doris]
zclllyybb commented on PR #25690: URL: https://github.com/apache/doris/pull/25690#issuecomment-1772333219 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](move-memtable) inverted index filter is reversed [doris]
doris-robot commented on PR #25684: URL: https://github.com/apache/doris/pull/25684#issuecomment-1772337517 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.24 seconds stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.4 seconds inserted 1000 Rows, about 340K ops/s storage size: 17162290801 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature](inverted index) support range predicate for inverted index [doris]
airborne12 commented on PR #24511: URL: https://github.com/apache/doris/pull/24511#issuecomment-1772338902 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org