[GitHub] [doris] morningman commented on a diff in pull request #16602: [feature-wip](iceberg) add dlf catalog impl for iceberg catalog
morningman commented on code in PR #16602:
URL: https://github.com/apache/doris/pull/16602#discussion_r1108127786
##
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/dlf/DLFCatalog.java:
##
@@ -0,0 +1,224 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.iceberg.dlf;
+
+import org.apache.doris.datasource.iceberg.dlf.client.DLFCachedClientPool;
+
+import org.apache.hadoop.conf.Configurable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.iceberg.BaseMetastoreCatalog;
+import org.apache.iceberg.BaseMetastoreTableOperations;
+import org.apache.iceberg.TableOperations;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NamespaceNotEmptyException;
+import org.apache.iceberg.exceptions.NoSuchIcebergTableException;
+import org.apache.iceberg.exceptions.NoSuchNamespaceException;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.hive.MetastoreUtil;
+import org.apache.iceberg.io.FileIO;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+public class DLFCatalog extends BaseMetastoreCatalog implements
SupportsNamespaces, Configurable {
+
+private Configuration conf;
+private DLFCachedClientPool clients;
+private FileIO fileIO;
+private String uid;
+
+@Override
+protected TableOperations newTableOps(TableIdentifier tableIdentifier) {
+String dbName = tableIdentifier.namespace().level(0);
+String tableName = tableIdentifier.name();
+return new DLFTableOperations(this.conf, this.clients, this.fileIO,
this.uid, dbName, tableName);
+}
+
+@Override
+protected String defaultWarehouseLocation(TableIdentifier tableIdentifier)
{
+return null;
+}
+
+@Override
+public void initialize(String name, Map properties) {
+this.uid = name;
+this.fileIO = new HadoopFileIO(conf);
+this.clients = new DLFCachedClientPool(this.conf, properties);
+}
+
+@Override
+protected boolean isValidIdentifier(TableIdentifier tableIdentifier) {
+return tableIdentifier.namespace().levels().length == 1;
+}
+
+private boolean isValidNamespace(Namespace namespace) {
+return namespace.levels().length == 1;
+}
+
+@Override
+public List listTables(Namespace namespace) {
+if (isValidNamespace(namespace)) {
+throw new NoSuchTableException("Invalid namespace: %s", namespace);
+}
+String dbName = namespace.level(0);
+try {
+return clients.run(client -> client.getAllTables(dbName))
+.stream()
+.map(tbl -> TableIdentifier.of(dbName, tbl))
+.collect(Collectors.toList());
+} catch (Exception e) {
+throw new RuntimeException(e);
+}
+}
+
+@Override
+public boolean dropTable(TableIdentifier tableIdentifier, boolean purge) {
Review Comment:
Do we need to implement these drop/rename/create method?
Because it currently not supported.
##
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/dlf/client/DLFCachedClientPool.java:
##
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the
[GitHub] [doris] morningman merged pull request #16767: [improvement](filecache) split file cache into sharding directories
morningman merged PR #16767: URL: https://github.com/apache/doris/pull/16767 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improvement](filecache) split file cache into sharding directories (#16767)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e2245cbdd3 [improvement](filecache) split file cache into sharding
directories (#16767)
e2245cbdd3 is described below
commit e2245cbdd38920941c60950d8143ccc049e5467e
Author: Ashin Gau
AuthorDate: Thu Feb 16 16:04:29 2023 +0800
[improvement](filecache) split file cache into sharding directories (#16767)
Save cached file segment into path like `cache_path /
hash(filepath).substr(0, 3) / hash(filepath) / offset`
to prevent too many directories in `cache_path`.
---
be/src/io/cache/block/block_file_cache.cpp | 12 +-
be/src/io/cache/block/block_file_cache.h | 5 +
be/src/io/cache/block/block_lru_file_cache.cpp | 176 ++---
be/src/io/cache/block/block_lru_file_cache.h | 4 +
be/test/io/cache/file_block_cache_test.cpp | 4 +-
5 files changed, 150 insertions(+), 51 deletions(-)
diff --git a/be/src/io/cache/block/block_file_cache.cpp
b/be/src/io/cache/block/block_file_cache.cpp
index 147a0a3339..292903c812 100644
--- a/be/src/io/cache/block/block_file_cache.cpp
+++ b/be/src/io/cache/block/block_file_cache.cpp
@@ -32,6 +32,9 @@ namespace fs = std::filesystem;
namespace doris {
namespace io {
+const std::string IFileCache::FILE_CACHE_VERSION = "2.0";
+const int IFileCache::KEY_PREFIX_LENGTH = 3;
+
IFileCache::IFileCache(const std::string& cache_base_path, const
FileCacheSettings& cache_settings)
: _cache_base_path(cache_base_path),
_max_size(cache_settings.max_size),
@@ -55,12 +58,17 @@ std::string IFileCache::get_path_in_local_cache(const Key&
key, size_t offset,
bool is_persistent) const {
auto key_str = key.to_string();
std::string suffix = is_persistent ? "_persistent" : "";
-return fs::path(_cache_base_path) / key_str / (std::to_string(offset) +
suffix);
+return fs::path(_cache_base_path) / key_str.substr(0, KEY_PREFIX_LENGTH) /
key_str /
+ (std::to_string(offset) + suffix);
}
std::string IFileCache::get_path_in_local_cache(const Key& key) const {
auto key_str = key.to_string();
-return fs::path(_cache_base_path) / key_str;
+return fs::path(_cache_base_path) / key_str.substr(0, KEY_PREFIX_LENGTH) /
key_str;
+}
+
+std::string IFileCache::get_version_path() const {
+return fs::path(_cache_base_path) / "version";
}
IFileCache::QueryFileCacheContextHolderPtr
IFileCache::get_query_context_holder(
diff --git a/be/src/io/cache/block/block_file_cache.h
b/be/src/io/cache/block/block_file_cache.h
index 466a9938d8..3e3113eb3c 100644
--- a/be/src/io/cache/block/block_file_cache.h
+++ b/be/src/io/cache/block/block_file_cache.h
@@ -43,6 +43,9 @@ class IFileCache {
friend struct FileBlocksHolder;
public:
+static const std::string FILE_CACHE_VERSION;
+static const int KEY_PREFIX_LENGTH;
+
struct Key {
uint128_t key;
std::string to_string() const;
@@ -73,6 +76,8 @@ public:
std::string get_path_in_local_cache(const Key& key) const;
+std::string get_version_path() const;
+
const std::string& get_base_path() const { return _cache_base_path; }
virtual std::vector try_get_cache_paths(const Key& key, bool
is_persistent) = 0;
diff --git a/be/src/io/cache/block/block_lru_file_cache.cpp
b/be/src/io/cache/block/block_lru_file_cache.cpp
index db3418870e..446c662b88 100644
--- a/be/src/io/cache/block/block_lru_file_cache.cpp
+++ b/be/src/io/cache/block/block_lru_file_cache.cpp
@@ -28,6 +28,8 @@
#include "common/status.h"
#include "io/cache/block/block_file_cache.h"
#include "io/cache/block/block_file_cache_settings.h"
+#include "io/fs/local_file_system.h"
+#include "olap/iterators.h"
#include "util/time.h"
#include "vec/common/hex.h"
#include "vec/common/sip_hash.h"
@@ -53,6 +55,7 @@ Status LRUFileCache::initialize() {
return Status::IOError("cannot create {}: {}",
_cache_base_path,
std::strerror(ec.value()));
}
+RETURN_IF_ERROR(write_file_cache_version());
}
}
_is_initialized = true;
@@ -634,63 +637,113 @@ void LRUFileCache::remove(const Key& key, bool
is_persistent, size_t offset,
}
void LRUFileCache::load_cache_info_into_memory(std::lock_guard&
cache_lock) {
+/// version 1.0: cache_base_path / key / offset
+/// version 2.0: cache_base_path / key_prefix / key / offset
+if (read_file_cache_version() != FILE_CACHE_VERSION) {
+// move directories format as version 2.0
+fs::directory_iterator key_it {_cache_base_path};
+for (; key_it != fs::directory_iterator(); ++key_it) {
+if (key_it->is_directory()) {
+std::string cache_key = key_it->path().filename().n
[GitHub] [doris] github-actions[bot] commented on pull request #16734: [Opt](exec) opt aggreate function performance in nullable column
github-actions[bot] commented on PR #16734: URL: https://github.com/apache/doris/pull/16734#issuecomment-1432676098 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16818: [Improvement](parquet-reader) Optimize and refactor parquet reader to improve performance.
github-actions[bot] commented on PR #16818: URL: https://github.com/apache/doris/pull/16818#issuecomment-1432676271 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16785: [Opt](exec) Refactor the code and logical functions to SIMD the code
github-actions[bot] commented on PR #16785: URL: https://github.com/apache/doris/pull/16785#issuecomment-1432681201 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16785: [Opt](exec) Refactor the code and logical functions to SIMD the code
github-actions[bot] commented on PR #16785: URL: https://github.com/apache/doris/pull/16785#issuecomment-1432681163 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #16791: [fix](docker)Fix Dockerfile logic
morningman merged PR #16791: URL: https://github.com/apache/doris/pull/16791 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (e2245cbdd3 -> ce7791c362)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from e2245cbdd3 [improvement](filecache) split file cache into sharding directories (#16767) add ce7791c362 [fix](docker)Fix Dockerfile logic (#16791) No new revisions were added by this update. Summary of changes: docker/runtime/be/resource/init_be.sh | 21 ++--- docker/runtime/broker/resource/init_broker.sh | 1 - docker/runtime/fe/resource/init_fe.sh | 13 ++--- 3 files changed, 16 insertions(+), 19 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee merged pull request #16814: [Bug](Datetime) Fix date time function mem use after free
HappenLee merged PR #16814: URL: https://github.com/apache/doris/pull/16814 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Bug](Datetime) Fix date time function mem use after free (#16814)
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new de1337511c [Bug](Datetime) Fix date time function mem use after free
(#16814)
de1337511c is described below
commit de1337511ce11bdb284ff7aa99174696963e0263
Author: HappenLee
AuthorDate: Thu Feb 16 16:15:58 2023 +0800
[Bug](Datetime) Fix date time function mem use after free (#16814)
---
.../vec/functions/function_date_or_datetime_computation.h | 15 ---
.../datetime_functions/test_date_function.out | 3 +++
.../datetime_functions/test_date_function.groovy | 2 ++
3 files changed, 13 insertions(+), 7 deletions(-)
diff --git a/be/src/vec/functions/function_date_or_datetime_computation.h
b/be/src/vec/functions/function_date_or_datetime_computation.h
index b45333b89d..dd25e0fca9 100644
--- a/be/src/vec/functions/function_date_or_datetime_computation.h
+++ b/be/src/vec/functions/function_date_or_datetime_computation.h
@@ -452,8 +452,8 @@ struct DateTimeAddIntervalImpl {
const auto is_nullable =
block.get_by_position(result).type->is_nullable();
if (const auto* sources =
check_and_get_column>(source_col.get())) {
auto col_to = ColumnVector::create();
-const IColumn& delta_column =
-
*remove_nullable(block.get_by_position(arguments[1]).column);
+auto delta_column_ptr =
remove_nullable(block.get_by_position(arguments[1]).column);
+const IColumn& delta_column = *delta_column_ptr;
if (is_nullable) {
auto null_map = ColumnUInt8::create(input_rows_count, 0);
@@ -545,16 +545,17 @@ struct DateTimeAddIntervalImpl {
auto col_to = ColumnVector::create();
if (is_nullable) {
auto null_map = ColumnUInt8::create(input_rows_count, 0);
+auto not_nullable_column_ptr_arg1 =
+
remove_nullable(block.get_by_position(arguments[1]).column);
if (const auto* delta_vec_column =
check_and_get_column>(
-
*remove_nullable(block.get_by_position(arguments[1]).column))) {
+*not_nullable_column_ptr_arg1)) {
Op::constant_vector(sources_const->template
get_value(),
col_to->get_data(),
null_map->get_data(),
delta_vec_column->get_data());
} else {
-Op::constant_vector(
-sources_const->template get_value(),
col_to->get_data(),
-null_map->get_data(),
-
*remove_nullable(block.get_by_position(arguments[1]).column));
+Op::constant_vector(sources_const->template
get_value(),
+col_to->get_data(),
null_map->get_data(),
+*not_nullable_column_ptr_arg1);
}
if (const auto* nullable_col =
check_and_get_column(
block.get_by_position(arguments[0]).column.get()))
{
diff --git
a/regression-test/data/query_p0/sql_functions/datetime_functions/test_date_function.out
b/regression-test/data/query_p0/sql_functions/datetime_functions/test_date_function.out
index a08ba6294f..fe55f5432e 100644
---
a/regression-test/data/query_p0/sql_functions/datetime_functions/test_date_function.out
+++
b/regression-test/data/query_p0/sql_functions/datetime_functions/test_date_function.out
@@ -590,6 +590,9 @@ true
-- !sql --
2019-08-01T13:21:02.11
+-- !sql --
+-1096
+
-- !sql --
\N \N \N \N
2000-02-29 2000-02-29 2000-02-29 2000-02-29
diff --git
a/regression-test/suites/query_p0/sql_functions/datetime_functions/test_date_function.groovy
b/regression-test/suites/query_p0/sql_functions/datetime_functions/test_date_function.groovy
index 83f1a8..c88de59acc 100644
---
a/regression-test/suites/query_p0/sql_functions/datetime_functions/test_date_function.groovy
+++
b/regression-test/suites/query_p0/sql_functions/datetime_functions/test_date_function.groovy
@@ -548,6 +548,8 @@ suite("test_date_function") {
qt_sql """ select minutes_sub(test_time2,1) result from ${tableName}; """
//seconds_sub
qt_sql """ select seconds_sub(test_time2,1) result from ${tableName}; """
+//datediff
+qt_sql """ select datediff(test_time2, STR_TO_DATE('2022-08-01
00:00:00','%Y-%m-%d')) from ${tableName}; """
// test last_day for vec
sql """ DROP TABLE IF EXISTS ${tableName}; """
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #16799: [enhancement](merge-on-write) do compaction with merge on read
zhannngchen commented on code in PR #16799:
URL: https://github.com/apache/doris/pull/16799#discussion_r1108144748
##
be/src/olap/compaction.cpp:
##
@@ -413,19 +413,36 @@ Status Compaction::construct_input_rowset_readers() {
Status Compaction::modify_rowsets() {
std::vector output_rowsets;
output_rowsets.push_back(_output_rowset);
-{
-std::lock_guard wrlock_(_tablet->get_rowset_update_lock());
-std::lock_guard wrlock(_tablet->get_header_lock());
-// update dst rowset delete bitmap
-if (_tablet->keys_type() == KeysType::UNIQUE_KEYS &&
-_tablet->enable_unique_key_merge_on_write()) {
-_tablet->tablet_meta()->update_delete_bitmap(
-_input_rowsets, _output_rs_writer->version(),
_rowid_conversion);
+if (_tablet->keys_type() == KeysType::UNIQUE_KEYS &&
+_tablet->enable_unique_key_merge_on_write()) {
+Version version = _tablet->max_version();
+DeleteBitmap output_rowset_delete_bitmap(_tablet->tablet_id());
+// Convert the delete bitmap of the input rowsets to output rowset.
+// New loads are not blocked, so some keys of input rowsets might
+// be deleted during the time. We need to deal with delete bitmap
+// of incremental data later.
+_tablet->calc_compaction_output_rowset_delete_bitmap(_input_rowsets,
_rowid_conversion, 0,
Review Comment:
Do we have performance test on this operation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16799: [enhancement](merge-on-write) do compaction with merge on read
github-actions[bot] commented on PR #16799: URL: https://github.com/apache/doris/pull/16799#issuecomment-1432691373 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16799: [enhancement](merge-on-write) do compaction with merge on read
github-actions[bot] commented on PR #16799: URL: https://github.com/apache/doris/pull/16799#issuecomment-1432691428 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zddr commented on a diff in pull request #16789: [Enhance](ComputeNode) K8sDeployManager support computeNode
zddr commented on code in PR #16789:
URL: https://github.com/apache/doris/pull/16789#discussion_r1108148176
##
fe/fe-core/src/main/java/org/apache/doris/deploy/DeployManager.java:
##
@@ -148,38 +146,51 @@ public DeployManager(Env env, long intervalMs) {
// Derived Class can override this method to init more private env
variables,
// but must class the parent's init at first.
protected void initEnvVariables(String envElectableFeServiceGroup, String
envObserverFeServiceGroup,
-String envBackendServiceGroup, String envBrokerServiceGroup) {
+String envBackendServiceGroup, String envBrokerServiceGroup,
String envCnServiceGroup) {
this.electableFeServiceGroup =
Strings.nullToEmpty(System.getenv(envElectableFeServiceGroup));
this.observerFeServiceGroup =
Strings.nullToEmpty(System.getenv(envObserverFeServiceGroup));
this.backendServiceGroup =
Strings.nullToEmpty(System.getenv(envBackendServiceGroup));
this.brokerServiceGroup =
Strings.nullToEmpty(System.getenv(envBrokerServiceGroup));
+this.cnServiceGroup =
Strings.nullToEmpty(System.getenv(envCnServiceGroup));
-LOG.info("get deploy env: {}, {}, {}, {}", envElectableFeServiceGroup,
envObserverFeServiceGroup,
- envBackendServiceGroup, envBrokerServiceGroup);
+LOG.info("get deploy env: {}, {}, {}, {}, {}",
envElectableFeServiceGroup, envObserverFeServiceGroup,
+envBackendServiceGroup, envBrokerServiceGroup,
envCnServiceGroup);
-// electableFeServiceGroup and backendServiceGroup must exist
-if (Strings.isNullOrEmpty(electableFeServiceGroup) ||
Strings.isNullOrEmpty(backendServiceGroup)) {
-LOG.warn("failed to init service group name."
-+ " electableFeServiceGroup: {}, backendServiceGroup: {}",
- electableFeServiceGroup, backendServiceGroup);
-System.exit(-1);
Review Comment:
This is to be compatible with only cn running on k8s
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wsjz commented on a diff in pull request #16602: [feature-wip](iceberg) add dlf catalog impl for iceberg catalog
wsjz commented on code in PR #16602:
URL: https://github.com/apache/doris/pull/16602#discussion_r1108148667
##
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/dlf/DLFCatalog.java:
##
@@ -0,0 +1,224 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.iceberg.dlf;
+
+import org.apache.doris.datasource.iceberg.dlf.client.DLFCachedClientPool;
+
+import org.apache.hadoop.conf.Configurable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.iceberg.BaseMetastoreCatalog;
+import org.apache.iceberg.BaseMetastoreTableOperations;
+import org.apache.iceberg.TableOperations;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NamespaceNotEmptyException;
+import org.apache.iceberg.exceptions.NoSuchIcebergTableException;
+import org.apache.iceberg.exceptions.NoSuchNamespaceException;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.hive.MetastoreUtil;
+import org.apache.iceberg.io.FileIO;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+public class DLFCatalog extends BaseMetastoreCatalog implements
SupportsNamespaces, Configurable {
+
+private Configuration conf;
+private DLFCachedClientPool clients;
+private FileIO fileIO;
+private String uid;
+
+@Override
+protected TableOperations newTableOps(TableIdentifier tableIdentifier) {
+String dbName = tableIdentifier.namespace().level(0);
+String tableName = tableIdentifier.name();
+return new DLFTableOperations(this.conf, this.clients, this.fileIO,
this.uid, dbName, tableName);
+}
+
+@Override
+protected String defaultWarehouseLocation(TableIdentifier tableIdentifier)
{
+return null;
+}
+
+@Override
+public void initialize(String name, Map properties) {
+this.uid = name;
+this.fileIO = new HadoopFileIO(conf);
Review Comment:
Oh, I try to use S3FileIO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16772: [pipeline](profile) Support real-time profile report in pipeline
github-actions[bot] commented on PR #16772: URL: https://github.com/apache/doris/pull/16772#issuecomment-1432697177 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wsjz commented on a diff in pull request #16602: [feature-wip](iceberg) add dlf catalog impl for iceberg catalog
wsjz commented on code in PR #16602:
URL: https://github.com/apache/doris/pull/16602#discussion_r1108150641
##
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/dlf/DLFCatalog.java:
##
@@ -0,0 +1,224 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.iceberg.dlf;
+
+import org.apache.doris.datasource.iceberg.dlf.client.DLFCachedClientPool;
+
+import org.apache.hadoop.conf.Configurable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.iceberg.BaseMetastoreCatalog;
+import org.apache.iceberg.BaseMetastoreTableOperations;
+import org.apache.iceberg.TableOperations;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NamespaceNotEmptyException;
+import org.apache.iceberg.exceptions.NoSuchIcebergTableException;
+import org.apache.iceberg.exceptions.NoSuchNamespaceException;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.hive.MetastoreUtil;
+import org.apache.iceberg.io.FileIO;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+public class DLFCatalog extends BaseMetastoreCatalog implements
SupportsNamespaces, Configurable {
+
+private Configuration conf;
+private DLFCachedClientPool clients;
+private FileIO fileIO;
+private String uid;
+
+@Override
+protected TableOperations newTableOps(TableIdentifier tableIdentifier) {
+String dbName = tableIdentifier.namespace().level(0);
+String tableName = tableIdentifier.name();
+return new DLFTableOperations(this.conf, this.clients, this.fileIO,
this.uid, dbName, tableName);
+}
+
+@Override
+protected String defaultWarehouseLocation(TableIdentifier tableIdentifier)
{
+return null;
+}
+
+@Override
+public void initialize(String name, Map properties) {
+this.uid = name;
+this.fileIO = new HadoopFileIO(conf);
+this.clients = new DLFCachedClientPool(this.conf, properties);
+}
+
+@Override
+protected boolean isValidIdentifier(TableIdentifier tableIdentifier) {
+return tableIdentifier.namespace().levels().length == 1;
+}
+
+private boolean isValidNamespace(Namespace namespace) {
+return namespace.levels().length == 1;
+}
+
+@Override
+public List listTables(Namespace namespace) {
+if (isValidNamespace(namespace)) {
+throw new NoSuchTableException("Invalid namespace: %s", namespace);
+}
+String dbName = namespace.level(0);
+try {
+return clients.run(client -> client.getAllTables(dbName))
+.stream()
+.map(tbl -> TableIdentifier.of(dbName, tbl))
+.collect(Collectors.toList());
+} catch (Exception e) {
+throw new RuntimeException(e);
+}
+}
+
+@Override
+public boolean dropTable(TableIdentifier tableIdentifier, boolean purge) {
Review Comment:
Now we can remove these
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] eldenmoon opened a new pull request, #16827: [chore](be-config) set disable_storage_row_cache default true to def…
eldenmoon opened a new pull request, #16827: URL: https://github.com/apache/doris/pull/16827 …ault disable row cache # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yixiutt opened a new pull request, #16828: [pick](bugfix)pick 16793
yixiutt opened a new pull request, #16828: URL: https://github.com/apache/doris/pull/16828 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16827: [chore](be-config) set disable_storage_row_cache default true to def…
github-actions[bot] commented on PR #16827: URL: https://github.com/apache/doris/pull/16827#issuecomment-1432704955 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16827: [chore](be-config) set disable_storage_row_cache default true to def…
github-actions[bot] commented on PR #16827: URL: https://github.com/apache/doris/pull/16827#issuecomment-1432705479 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16815: [regression pipeline]add git action to trigger teamcity pipeline 0216
github-actions[bot] commented on PR #16815: URL: https://github.com/apache/doris/pull/16815#issuecomment-1432710749 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16815: [regression pipeline]add git action to trigger teamcity pipeline 0216
github-actions[bot] commented on PR #16815: URL: https://github.com/apache/doris/pull/16815#issuecomment-1432710791 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] qidaye commented on a diff in pull request #16735: [doc](point query) modify and refine docs
qidaye commented on code in PR #16735: URL: https://github.com/apache/doris/pull/16735#discussion_r1108167002 ## docs/en/docs/advanced/hight-concurrent-point-query.md: ## @@ -0,0 +1,60 @@ +# High-concurrency point query based on primary key Review Comment: Please add header and license. ## docs/zh-CN/docs/advanced/hight-concurrent-point-query.md: ## @@ -0,0 +1,56 @@ +# 基于主键的高并发点查询 Review Comment: Please add header and license. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #16815: [regression pipeline]add git action to trigger teamcity pipeline 0216
dataroaring merged PR #16815: URL: https://github.com/apache/doris/pull/16815 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improvement][regression] add git action to trigger teamcity pipeline 0216 (#16815)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a56f1ca6b6 [improvement][regression] add git action to trigger
teamcity pipeline 0216 (#16815)
a56f1ca6b6 is described below
commit a56f1ca6b69cf320173db86d94e6e64200aa7944
Author: zhangguoqiang <18372634...@163.com>
AuthorDate: Thu Feb 16 16:45:29 2023 +0800
[improvement][regression] add git action to trigger teamcity pipeline 0216
(#16815)
---
.github/workflows/auto_trigger_teamcity.yml | 162
1 file changed, 162 insertions(+)
diff --git a/.github/workflows/auto_trigger_teamcity.yml
b/.github/workflows/auto_trigger_teamcity.yml
new file mode 100644
index 00..e569798def
--- /dev/null
+++ b/.github/workflows/auto_trigger_teamcity.yml
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Auto trigger teamcity
+
+on:
+ pull_request:
+branches:
+ - master
+ issue_comment:
+types: [created, edited]
+ pull_request_review_comment:
+types: [created, edited]
+
+env:
+ TEAMCITY_URL: 'X POST -H "Content-Type:text/plain" -u
OneMoreChance:OneMoreChance http://43.132.222.7:8111/httpAuth/action.html'
+
+
+jobs:
+ run_compile_pipeline:
+if: contains(github.event.comment.body, 'buildall')
+
+runs-on: ubuntu-latest
+
+env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+steps:
+ - name: Run pipeline by restful
+run: |
+ echo "trigger compile pipeline"
+
+ if [ "_xx""${{ github.event.issue.pull_request.url }}" != "_xx" ];
then
+echo "Comment was made on pull request: $(echo ${{
github.event.issue.pull_request.url }} | awk -F/ '{print $NF}')"
+ else
+echo "Comment was made on an issue, not a pull request."
+ fi
+ pull_request_num=$(echo "${{ github.event.issue.pull_request.url }}"
| awk -F/ '{print $NF}')
+ comment="${{ github.event.comment.body }}"
+ encoded_string=$(perl -MURI::Escape -e 'print uri_escape($ARGV[0]);'
"${comment}")
+ echo ${encoded_string}
+ pipelines_arr="Doris_Doris_FeUt Doris_DorisBeUt_BeUt
Doris_DorisCompile_Compile"
+ for pipeline in ${pipelines_arr}
+ do
+execute_command="curl ${{ env.TEAMCITY_URL
}}\?add2Queue\=${pipeline}\&branchName\=pull/${pull_request_num}\&name=env.latest_pr_comment\&value=${encoded_string}"
+echo "{$execute_command}"
+eval ${execute_command}
+ done
+
+
+ run_p0_external_pipeline:
+if: contains(github.event.comment.body, 'p0') || ||
contains(github.event.comment.body, 'external')
+
+runs-on: ubuntu-latest
+
+env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+steps:
+ - name: Run p0 and external pipeline by restful
+run: |
+ echo "trigger compile pipeline"
+
+ if [ "_xx""${{ github.event.issue.pull_request.url }}" != "_xx" ];
then
+echo "Comment was made on pull request: $(echo ${{
github.event.issue.pull_request.url }} | awk -F/ '{print $NF}')"
+ else
+echo "Comment was made on an issue, not a pull request."
+ fi
+ pull_request_num=$(echo "${{ github.event.issue.pull_request.url }}"
| awk -F/ '{print $NF}')
+ comment="${{ github.event.comment.body }}"
+ encoded_string=$(perl -MURI::Escape -e 'print uri_escape($ARGV[0]);'
"${comment}")
+ echo ${encoded_string}
+ execute_command="curl ${{ env.TEAMCITY_URL
}}\?add2Queue\=Doris_DorisRegression_ExternalRegression\&branchName\=pull/${pull_request_num}\&name=env.latest_pr_comment\&value=${encoded_string}"
+ echo "{$execute_command}"
+ eval ${execute_command}
+
+ run_p1_pipeline:
+if: contains(github.event.comment.body, 'p1')
+
+runs-on: ubuntu-latest
+
+env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+steps:
+ - name: Run fe ut pipeline by restful
+run: |
+ echo "trigger fe ut pipeline"
+
+ if [ "_xx""$
[GitHub] [doris] qidaye commented on a diff in pull request #16735: [doc](point query) modify and refine docs
qidaye commented on code in PR #16735:
URL: https://github.com/apache/doris/pull/16735#discussion_r1108173773
##
docs/en/docs/data-table/best-practice.md:
##
@@ -180,62 +180,4 @@ Users can modify the Schema of an existing table through
the Schema Change opera
- Adding or modifying Bloom Filter
- Adding or removing bitmap index
-For details, please refer to [Schema Change](
-
-## Row Store format
-We support row format for olap table to reduce point lookup io cost, but to
enable this format you need to spend more disk space for row format
store.Currently we store row in an extra column called `row column` for
simplicity.Row store is disabled by default, users can enable it by adding the
following property when create table
-```
-"store_row_column" = "true"
-```
-
-## Accelerate point query for merge-on-write model
-As we provided row store format , we could use such store format to speed up
point query performance for merge-on-write model.For point query on primary
keys when `enable_unique_key_merge_on_write` enabled, planner will optimize
such query and execute in a short path in a light weight RPC interface.Bellow
is an example of point query with row store on merge-on-write model:
-```
-CREATE TABLE `tbl_point_query` (
- `key` int(11) NULL,
- `v1` decimal(27, 9) NULL,
- `v2` varchar(30) NULL,
- `v3` varchar(30) NULL,
- `v4` date NULL,
- `v5` datetime NULL,
- `v6` float NULL,
- `v7` datev2 NULL
-) ENGINE=OLAP
-UNIQUE KEY(key)
-COMMENT 'OLAP'
-DISTRIBUTED BY HASH(key) BUCKETS 1
-PROPERTIES (
-"replication_allocation" = "tag.location.default: 1",
-"enable_unique_key_merge_on_write" = "true",
-"light_schema_change" = "true",
-"store_row_column" = "true"
-);
-```
-[NOTE]
-1. `enable_unique_key_merge_on_write` should be enabled, since we need primary
key for quick point lookup in storage engine
-2. when condition only contains primary key like `select * from
tbl_point_query where key = 123`, such query will go through the short fast path
-3. `light_schema_change` should also been enabled since we rely `column unique
id` of each columns when doing point query.
-
-### Using `PreparedStatement`
-In order to reduce CPU cost for parsing query SQL and SQL expressions, we
provide `PreparedStatement` feature in FE fully compatible with mysql protocol
(currently only support point queries like above mentioned).Enable it will pre
caculate PreparedStatement SQL and expresions and caches it in a session level
memory buffer and will be reused later on.We could improve 4x+ performance by
using `PreparedStatement` when CPU became hotspot doing such queries.Bellow is
an JDBC example of using `PreparedStatement`.
-
-1. Setup JDBC url and enable server side prepared statement
-```
-url = jdbc:mysql://127.0.0.1:9137/ycsb?useServerPrepStmts=true
-``
-
-2. Using `PreparedStatement`
-```java
-// use `?` for placement holders, readStatement should be reused
-PreparedStatement readStatement = conn.prepareStatement("select * from
tbl_point_query where key = ?");
-...
-readStatement.setInt(1234);
-ResultSet resultSet = readStatement.executeQuery();
-...
-readStatement.setInt(1235);
-resultSet = readStatement.executeQuery();
-...
-```
-
-
-
+For details, please refer to [Schema Change](
Review Comment:
Please add relative path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #16828: [bugfix](reader) make segment_overlapping meta correct
morningman merged PR #16828: URL: https://github.com/apache/doris/pull/16828 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated: [bugfix](reader) make segment_overlapping meta correct (#16828)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new 306e004ece [bugfix](reader) make segment_overlapping meta correct
(#16828)
306e004ece is described below
commit 306e004ece81487bbdf789de548537b9a64debcc
Author: yixiutt <102007456+yixi...@users.noreply.github.com>
AuthorDate: Thu Feb 16 16:47:19 2023 +0800
[bugfix](reader) make segment_overlapping meta correct (#16828)
cherry-pick #16793
---
be/src/olap/rowset/beta_rowset_writer.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/olap/rowset/beta_rowset_writer.cpp
b/be/src/olap/rowset/beta_rowset_writer.cpp
index 081df80633..a18d8c5f67 100644
--- a/be/src/olap/rowset/beta_rowset_writer.cpp
+++ b/be/src/olap/rowset/beta_rowset_writer.cpp
@@ -799,7 +799,7 @@ bool BetaRowsetWriter::_is_segment_overlapping(
for (auto segment_encode_key : segments_encoded_key_bounds) {
auto cur_min = segment_encode_key.min_key();
auto cur_max = segment_encode_key.max_key();
-if (cur_min < last) {
+if (cur_min <= last) {
return true;
}
last = cur_max;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangGuoqiang666 commented on pull request #16815: [regression pipeline]add git action to trigger teamcity pipeline 0216
ZhangGuoqiang666 commented on PR #16815: URL: https://github.com/apache/doris/pull/16815#issuecomment-1432726741 buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16772: [pipeline](profile) Support real-time profile report in pipeline
github-actions[bot] commented on PR #16772: URL: https://github.com/apache/doris/pull/16772#issuecomment-1432728081 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hf200012 merged pull request #16768: [fix](docs) Fix broken graph in en/broker-load-manual.md
hf200012 merged PR #16768: URL: https://github.com/apache/doris/pull/16768 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: fix broken graph in broker-load-manual.md (#16768)
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 2f3ce39857 fix broken graph in broker-load-manual.md (#16768) 2f3ce39857 is described below commit 2f3ce39857a5603d52e51cdac71e2c75c9940678 Author: Bowen Liang AuthorDate: Thu Feb 16 16:54:55 2023 +0800 fix broken graph in broker-load-manual.md (#16768) --- .../import/import-way/broker-load-manual.md| 50 +++--- 1 file changed, 25 insertions(+), 25 deletions(-) diff --git a/docs/en/docs/data-operate/import/import-way/broker-load-manual.md b/docs/en/docs/data-operate/import/import-way/broker-load-manual.md index c4d8502745..f4e481c55a 100644 --- a/docs/en/docs/data-operate/import/import-way/broker-load-manual.md +++ b/docs/en/docs/data-operate/import/import-way/broker-load-manual.md @@ -43,38 +43,38 @@ After the user submits the import task, FE will generate the corresponding Plan BE pulls data from the broker during execution, and imports the data into the system after transforming the data. All BEs are imported, and FE ultimately decides whether the import is successful. - +``` + | 1. user create broker load v +++ -| | -| FE | -| | +| | +| FE| +| | +++ | | 2. BE etl and load the data -+--+ -| | | -+---v---+ +--v+ +---v---+ -| | | | | | -| BE | | BE | | BE | -| | | | | | -+---+-^-+ +---+-^-+ +--+-^--+ -| | | | | | -| | | | | | 3. pull data from broker -+---v-+-+ +---v-+-+ +--v-+--+ -| | | | | | -|Broker | |Broker | |Broker | -| | | | | | -+---+-^-+ +---+-^-+ +---+-^-+ -| | | | | | -+---v-+-v-+v-+-+ -| HDFS/BOS/AFS cluster | -| | -+---+ - - ++--+ +|| | ++---v---+ +--v++---v---+ +| | | || | +| BE | | BE || BE | +| | | || | ++---+-^-+ +---+-^-++--+-^--+ +| | | | | | +| | | | | | 3. pull data from broker ++---v-+-+ +---v-+-++--v-+--+ +| | | || | +|Broker | |Broker ||Broker | +| | | || | ++---+-^-+ +---+-^-++---+-^-+ +| | | | | | ++---v-+---v-+--v-+-+ +| HDFS/BOS/AFS cluster | +| | ++--+ + +``` ## start import - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee merged pull request #16785: [Opt](exec) Refactor the code and logical functions to SIMD the code
HappenLee merged PR #16785: URL: https://github.com/apache/doris/pull/16785 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (2f3ce39857 -> f08c1222cc)
This is an automated email from the ASF dual-hosted git repository. lihaopeng pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 2f3ce39857 fix broken graph in broker-load-manual.md (#16768) add f08c1222cc [Opt](exec) Refactor the code and logical functions to SIMD the code (#16785) No new revisions were added by this update. Summary of changes: be/src/vec/columns/column.h| 1 - be/src/vec/functions/functions_logical.cpp | 444 ++--- be/src/vec/functions/functions_logical.h | 42 +-- 3 files changed, 102 insertions(+), 385 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yunaisheng opened a new issue, #16829: [Feature] Partial column updates on aggregate model table include null value
yunaisheng opened a new issue, #16829: URL: https://github.com/apache/doris/issues/16829 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description We require partial updates of certain columns , we are using the Aggregate Model with setting the aggregate type of the non-primary key columns to REPLACE_IF_NOT_NULL now. But the null value is also a meaningful value in our business, and we also need to update the certain column with the null value. So,we cannot update null values in a Aggregate Model with setting the non-primary key columns to REPLACE_IF_NOT_NULL. ### Use case User label update in user portrait analysis scenario ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dinggege1024 opened a new issue, #16830: [Bug] release not link 404
dinggege1024 opened a new issue, #16830: URL: https://github.com/apache/doris/issues/16830 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version all ### What's Wrong? https://doris.apache.org/docs/releasenotes/release-1.2.2 release note 404 ### What You Expected? not 404 ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt opened a new pull request, #16831: [Chore](build) remove memory_copy and remove some wno cbuild check
BiteThet opened a new pull request, #16831: URL: https://github.com/apache/doris/pull/16831 # Proposed changes 1. remove memory_copy, memory_copy is not faster than memcpy now. https://user-images.githubusercontent.com/7939630/219318104-60d46380-8695-4c4f-ba59-fced0f375598.png";> https://quick-bench.com/q/Px_yEBPlLRcQCePRpFcmcZzcp-8 2. remove some wno cbuild check ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16816: [improve](dynamic table) refine SegmentWriter columns writer generate
github-actions[bot] commented on PR #16816: URL: https://github.com/apache/doris/pull/16816#issuecomment-1432747613 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16831: [Chore](build) remove memory_copy and remove some wno cbuild check
github-actions[bot] commented on PR #16831: URL: https://github.com/apache/doris/pull/16831#issuecomment-1432751616 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16758: [improve](inverted index) Add element count limit for inverted index searcher cache
github-actions[bot] commented on PR #16758: URL: https://github.com/apache/doris/pull/16758#issuecomment-1432752005 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16831: [Chore](build) remove memory_copy and remove some wno cbuild check
github-actions[bot] commented on PR #16831: URL: https://github.com/apache/doris/pull/16831#issuecomment-1432754179 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16735: [doc](point query) modify and refine docs
github-actions[bot] commented on PR #16735: URL: https://github.com/apache/doris/pull/16735#issuecomment-1432765351 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16735: [doc](point query) modify and refine docs
github-actions[bot] commented on PR #16735: URL: https://github.com/apache/doris/pull/16735#issuecomment-1432765413 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16772: [pipeline](profile) Support real-time profile report in pipeline
github-actions[bot] commented on PR #16772: URL: https://github.com/apache/doris/pull/16772#issuecomment-1432775821 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16772: [pipeline](profile) Support real-time profile report in pipeline
github-actions[bot] commented on PR #16772: URL: https://github.com/apache/doris/pull/16772#issuecomment-1432775925 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16812: [fix](cooldown)Fix bug for single cooldown compaction, add remote meta
github-actions[bot] commented on PR #16812: URL: https://github.com/apache/doris/pull/16812#issuecomment-1432777114 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16812: [fix](cooldown)Fix bug for single cooldown compaction, add remote meta
github-actions[bot] commented on PR #16812: URL: https://github.com/apache/doris/pull/16812#issuecomment-1432778631 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] qidaye merged pull request #16735: [doc](point query) modify and refine docs
qidaye merged PR #16735: URL: https://github.com/apache/doris/pull/16735 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [doc](point query) modify and refine docs (#16735)
This is an automated email from the ASF dual-hosted git repository.
jianliangqi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 262a2ea10d [doc](point query) modify and refine docs (#16735)
262a2ea10d is described below
commit 262a2ea10d38d2831ef9bb9eb9600bd10a691f37
Author: lihangyu <15605149...@163.com>
AuthorDate: Thu Feb 16 17:36:32 2023 +0800
[doc](point query) modify and refine docs (#16735)
---
.../docs/advanced/hight-concurrent-point-query.md | 91 ++
docs/en/docs/data-table/best-practice.md | 60 +-
docs/sidebars.json | 3 +-
.../docs/advanced/hight-concurrent-point-query.md | 87 +
docs/zh-CN/docs/data-table/best-practice.md| 56 +
5 files changed, 182 insertions(+), 115 deletions(-)
diff --git a/docs/en/docs/advanced/hight-concurrent-point-query.md
b/docs/en/docs/advanced/hight-concurrent-point-query.md
new file mode 100644
index 00..62c35cf820
--- /dev/null
+++ b/docs/en/docs/advanced/hight-concurrent-point-query.md
@@ -0,0 +1,91 @@
+---
+{
+"title": "High-concurrency point query",
+"language": "en"
+}
+---
+
+
+
+# High-concurrency point query based on primary key
+
+
+
+
+## Background
+Doris is built on a columnar storage format engine. In high-concurrency
service scenarios, users always want to retrieve entire rows of data from the
system. However, when tables are wide, the columnar format greatly amplifies
random read IO. Doris query engine and planner are too heavy for some simple
queries, such as point queries. A short path needs to be planned in the FE's
query plan to handle such queries. FE is the access layer service for SQL
queries, written in Java. Parsing and [...]
+
+## Row Store format
+We support row format for olap table to reduce point lookup io cost, but to
enable this format you need to spend more disk space for row format
store.Currently we store row in an extra column called `row column` for
simplicity.Row store is disabled by default, users can enable it by adding the
following property when create table
+```
+"store_row_column" = "true"
+```
+
+## Accelerate point query for merge-on-write model
+As we provided row store format , we could use such store format to speed up
point query performance for merge-on-write model.For point query on primary
keys when `enable_unique_key_merge_on_write` enabled, planner will optimize
such query and execute in a short path in a light weight RPC interface.Bellow
is an example of point query with row store on merge-on-write model:
+```
+CREATE TABLE `tbl_point_query` (
+ `key` int(11) NULL,
+ `v1` decimal(27, 9) NULL,
+ `v2` varchar(30) NULL,
+ `v3` varchar(30) NULL,
+ `v4` date NULL,
+ `v5` datetime NULL,
+ `v6` float NULL,
+ `v7` datev2 NULL
+) ENGINE=OLAP
+UNIQUE KEY(key)
+COMMENT 'OLAP'
+DISTRIBUTED BY HASH(key) BUCKETS 1
+PROPERTIES (
+"replication_allocation" = "tag.location.default: 1",
+"enable_unique_key_merge_on_write" = "true",
+"light_schema_change" = "true",
+"store_row_column" = "true"
+);
+```
+[NOTE]
+1. `enable_unique_key_merge_on_write` should be enabled, since we need primary
key for quick point lookup in storage engine
+2. when condition only contains primary key like `select * from
tbl_point_query where key = 123`, such query will go through the short fast path
+3. `light_schema_change` should also been enabled since we rely `column unique
id` of each columns when doing point query.
+
+## Using `PreparedStatement`
+In order to reduce CPU cost for parsing query SQL and SQL expressions, we
provide `PreparedStatement` feature in FE fully compatible with mysql protocol
(currently only support point queries like above mentioned).Enable it will pre
caculate PreparedStatement SQL and expresions and caches it in a session level
memory buffer and will be reused later on.We could improve 4x+ performance by
using `PreparedStatement` when CPU became hotspot doing such queries.Bellow is
an JDBC example of using [...]
+
+1. Setup JDBC url and enable server side prepared statement
+```
+url = jdbc:mysql://127.0.0.1:9137/ycsb?useServerPrepStmts=true
+``
+
+2. Using `PreparedStatement`
+```java
+// use `?` for placement holders, readStatement should be reused
+PreparedStatement readStatement = conn.prepareStatement("select * from
tbl_point_query where key = ?");
+...
+readStatement.setInt(1234);
+ResultSet resultSet = readStatement.executeQuery();
+...
+readStatement.setInt(1235);
+resultSet = readStatement.executeQuery();
+...
+```
+
+
+
diff --git a/docs/en/docs/data-table/best-practice.md
b/docs/en/docs/data-table/best-practice.md
index b0711bc15f..5f531cfb0b 100644
--- a/docs/en/docs/data-table/best-practice.md
+++ b/docs/en/docs/data-table/best-practice.md
@@ -180,62 +180,4 @@ Users can modify the Schema of an existing table th
[GitHub] [doris] github-actions[bot] commented on pull request #16831: [Chore](build) remove memory_copy and remove some wno build check
github-actions[bot] commented on PR #16831: URL: https://github.com/apache/doris/pull/16831#issuecomment-1432792339 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16831: [Chore](build) remove memory_copy and remove some wno build check
github-actions[bot] commented on PR #16831: URL: https://github.com/apache/doris/pull/16831#issuecomment-1432792401 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] chioWong commented on issue #16403: [Bug] StreamLoad sink.enable-2pc设置不生效
chioWong commented on issue #16403: URL: https://github.com/apache/doris/issues/16403#issuecomment-1432801239 > 据的可见性依赖于checkpoint，将sink.enable-2pc设置为false时，数据最终的submit依然依赖于checkpoint > > ### What You Expected? > StreamLoad可以由用户控制数据的提交间隔 > > ### How to Reproduce? Doris有没有办法由用户自己控制数据的submit间隔时长 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14531: [improvement](meta) make database,table,column names to support unicode (replace PR #13467 with this)
github-actions[bot] commented on PR #14531: URL: https://github.com/apache/doris/pull/14531#issuecomment-1432801489 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] JackDan9 commented on issue #6227: Should a docker image be provided to facilitate deployment?
JackDan9 commented on issue #6227: URL: https://github.com/apache/doris/issues/6227#issuecomment-1432802615 确实应该是，现在做镜像进去部署太费劲了 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16812: [fix](cooldown)Fix bug for single cooldown compaction, add remote meta
github-actions[bot] commented on code in PR #16812:
URL: https://github.com/apache/doris/pull/16812#discussion_r1108241611
##
be/src/olap/tablet.cpp:
##
@@ -1807,13 +1798,32 @@ Status
Tablet::_read_cooldown_meta(io::RemoteFileSystem* fs, int64_t cooldown_re
return Status::OK();
}
-Status Tablet::_write_cooldown_meta(io::RemoteFileSystem* fs, UniqueId
cooldown_meta_id,
-RowsetMeta* new_rs_meta) {
+Status Tablet::write_cooldown_meta(const
std::shared_ptr& fs,
+ UniqueId cooldown_meta_id, const
RowsetMetaSharedPtr& new_rs_meta,
+ const std::vector&
to_deletes) {
std::vector cooldowned_rs_metas;
{
std::shared_lock meta_rlock(_meta_lock);
+std::set rs_meta_set;
for (auto& rs_meta : _tablet_meta->all_rs_metas()) {
if (!rs_meta->is_local()) {
+rs_meta_set.emplace(rs_meta->version().to_string());
+}
Review Comment:
warning: no member named 'version' in 'std::shared_ptr';
did you mean to use '->' instead of '.'? [clang-diagnostic-error]
```suggestion
rs_meta->version().to_string());
```
##
be/src/olap/tablet.cpp:
##
@@ -1807,13 +1798,32 @@
return Status::OK();
}
-Status Tablet::_write_cooldown_meta(io::RemoteFileSystem* fs, UniqueId
cooldown_meta_id,
-RowsetMeta* new_rs_meta) {
+Status Tablet::write_cooldown_meta(const
std::shared_ptr& fs,
+ UniqueId cooldown_meta_id, const
RowsetMetaSharedPtr& new_rs_meta,
+ const std::vector&
to_deletes) {
std::vector cooldowned_rs_metas;
{
std::shared_lock meta_rlock(_meta_lock);
+std::set rs_meta_set;
for (auto& rs_meta : _tablet_meta->all_rs_metas()) {
if (!rs_meta->is_local()) {
+rs_meta_set.emplace(rs_meta->version().to_string());
+}
+}
+
+std::set to_delete_set;
+for (auto& rs_meta : to_deletes) {
+if (rs_meta_set.find(rs_meta->version().to_string()) ==
rs_meta_set.end()) {
Review Comment:
warning: no member named 'version' in 'std::shared_ptr';
did you mean to use '->' instead of '.'? [clang-diagnostic-error]
```suggestion
(auto& rs_meta : to_deletes) {->
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] jeffreys-cat opened a new pull request, #191: [PDF] add doris log on the cover
jeffreys-cat opened a new pull request, #191: URL: https://github.com/apache/doris-website/pull/191 - [PDF] add doris log on the cover -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #16797: [feature](Nereids): add rule split limit into two phase
morrySnow commented on code in PR #16797:
URL: https://github.com/apache/doris/pull/16797#discussion_r1108242007
##
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##
@@ -1338,38 +1338,15 @@ public PlanFragment visitPhysicalLimit(PhysicalLimit physicalLim
if (inputFragment == null) {
return inputFragment;
}
-
+// For case globalLimit(l, o) -> LocalLimit(l+o, 0), that is the
LocalLimit has already gathered
+// The globalLimit can overwrite the limit and offset, so it's still
correct
PlanNode child = inputFragment.getPlanRoot();
-
-// physical plan: limit --> sort
-// after translate, it could be:
-// 1. limit->sort => set (limit and offset) on sort
-// 2. limit->exchange->sort => set (limit and offset) on exchange, set
sort.limit = limit+offset
-if (child instanceof SortNode) {
-SortNode sort = (SortNode) child;
-sort.setLimit(physicalLimit.getLimit());
-sort.setOffset(physicalLimit.getOffset());
-return inputFragment;
-}
-if (child instanceof ExchangeNode) {
-ExchangeNode exchangeNode = (ExchangeNode) child;
-exchangeNode.setLimit(physicalLimit.getLimit());
-// we do not check if this is a merging exchange here,
-// since this guaranteed by translating logic plan to physical plan
-exchangeNode.setOffset(physicalLimit.getOffset());
-if (exchangeNode.getChild(0) instanceof SortNode) {
-SortNode sort = (SortNode) exchangeNode.getChild(0);
-sort.setLimit(physicalLimit.getLimit() +
physicalLimit.getOffset());
-sort.setOffset(0);
-}
-return inputFragment;
-}
-// for other PlanNode, just set limit as limit+offset
-child.setLimit(physicalLimit.getLimit() + physicalLimit.getOffset());
-PlanFragment planFragment = exchangeToMergeFragment(inputFragment,
context);
-planFragment.getPlanRoot().setLimit(physicalLimit.getLimit());
-
planFragment.getPlanRoot().setOffSetDirectly(physicalLimit.getOffset());
-return planFragment;
+child.setLimit(physicalLimit.getLimit());
+child.setOffset(physicalLimit.getOffset());
+// TODO: add a rule in cascades
+// before: GlobalLimit -> LocalLimit -> GlobalSort -> Gather ->
LocalSort
+// after: GlobalLimit -> GlobalSort -> Gather ->LocalLimit ->
LocalSort
+return inputFragment;
Review Comment:
if we have not rule to rewrite Limit(Sort) to TopN, and do not set limit and
offset on ExchangeNode's child, the performance of `order by x limit n` is not
acceptable.
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalLimit.java:
##
@@ -44,19 +45,28 @@
* offset 100
*/
public class LogicalLimit extends
LogicalUnary implements Limit {
-
+private final LimitPhase phase;
private final long limit;
private final long offset;
-public LogicalLimit(long limit, long offset, CHILD_TYPE child) {
-this(limit, offset, Optional.empty(), Optional.empty(), child);
+public LogicalLimit(long limit, long offset, CHILD_TYPE child, LimitPhase
phase) {
+this(limit, offset, Optional.empty(), Optional.empty(), child, phase);
Review Comment:
children should be the last parameter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16812: [fix](cooldown)Fix bug for single cooldown compaction, add remote meta
github-actions[bot] commented on PR #16812: URL: https://github.com/apache/doris/pull/16812#issuecomment-1432808796 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] morningman merged pull request #191: [PDF] add doris logo on the cover
morningman merged PR #191: URL: https://github.com/apache/doris-website/pull/191 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: [PDF] add doris logo on the cover (#191)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new 9ac69cc64e [PDF] add doris logo on the cover (#191) 9ac69cc64e is described below commit 9ac69cc64e90e516fb0ddc1bfe969b7e2689dd5c Author: Jeffrey AuthorDate: Thu Feb 16 17:51:06 2023 +0800 [PDF] add doris logo on the cover (#191) [PDF] add doris logo on the cover --- .github/workflows/cron-generate-pdf.yml | 4 ++-- .github/workflows/generate-pdf.yml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/cron-generate-pdf.yml b/.github/workflows/cron-generate-pdf.yml index 57b898e738..40d4861035 100644 --- a/.github/workflows/cron-generate-pdf.yml +++ b/.github/workflows/cron-generate-pdf.yml @@ -36,8 +36,8 @@ jobs: - name: Generate PDF run: | -npx vitpress-generate-pdf --initialDocURLs="https://doris.apache.org/docs/dev/get-starting/"; --paginationSelector=".pagination-nav__link--next" --contentSelector="article" --coverImage="https://cdn.selectdb.com/images/doris-logo-512.png"; --coverTitle="Apache Doris Docs (English)" --outputPDFFilename="Apache Doris Docs (English).pdf" --tocOnlyH1=true -npx vitpress-generate-pdf --initialDocURLs="https://doris.apache.org/zh-CN/docs/dev/get-starting/"; --paginationSelector=".pagination-nav__link--next" --contentSelector="article" --coverImage="https://cdn.selectdb.com/images/doris-logo-512.png"; --coverTitle="Apache Doris Docs (中文)" --outputPDFFilename="Apache Doris Docs (中文).pdf" --tocOnlyH1=true +npx vitpress-generate-pdf --initialDocURLs="https://doris.apache.org/docs/dev/get-starting/"; --paginationSelector=".pagination-nav__link--next" --contentSelector="article" --coverImage="https://cdn.selectdb.com/static/doris_logo_512_4903556647.png"; --coverTitle="Apache Doris Docs (English)" --outputPDFFilename="Apache Doris Docs (English).pdf" --tocOnlyH1=true +npx vitpress-generate-pdf --initialDocURLs="https://doris.apache.org/zh-CN/docs/dev/get-starting/"; --paginationSelector=".pagination-nav__link--next" --contentSelector="article" --coverImage="https://cdn.selectdb.com/static/doris_logo_512_4903556647.png"; --coverTitle="Apache Doris Docs (中文)" --outputPDFFilename="Apache Doris Docs (中文).pdf" --tocOnlyH1=true ls rm -rf ./build/** mkdir -p ./build/assets/files/ diff --git a/.github/workflows/generate-pdf.yml b/.github/workflows/generate-pdf.yml index 7321108bc2..43158190f1 100644 --- a/.github/workflows/generate-pdf.yml +++ b/.github/workflows/generate-pdf.yml @@ -40,8 +40,8 @@ jobs: - name: Generate PDF run: | -npx vitpress-generate-pdf --initialDocURLs="https://doris.apache.org/docs/dev/get-starting/"; --paginationSelector=".pagination-nav__link--next" --contentSelector="article" --coverImage="https://cdn.selectdb.com/images/doris-logo-512.png"; --coverTitle="Apache Doris Docs (English)" --outputPDFFilename="Apache Doris Docs (English).pdf" --tocOnlyH1=true -npx vitpress-generate-pdf --initialDocURLs="https://doris.apache.org/zh-CN/docs/dev/get-starting/"; --paginationSelector=".pagination-nav__link--next" --contentSelector="article" --coverImage="https://cdn.selectdb.com/images/doris-logo-512.png"; --coverTitle="Apache Doris Docs (中文)" --outputPDFFilename="Apache Doris Docs (中文).pdf" --tocOnlyH1=true +npx vitpress-generate-pdf --initialDocURLs="https://doris.apache.org/docs/dev/get-starting/"; --paginationSelector=".pagination-nav__link--next" --contentSelector="article" --coverImage="https://cdn.selectdb.com/static/doris_logo_512_4903556647.png"; --coverTitle="Apache Doris Docs (English)" --outputPDFFilename="Apache Doris Docs (English).pdf" --tocOnlyH1=true +npx vitpress-generate-pdf --initialDocURLs="https://doris.apache.org/zh-CN/docs/dev/get-starting/"; --paginationSelector=".pagination-nav__link--next" --contentSelector="article" --coverImage="https://cdn.selectdb.com/static/doris_logo_512_4903556647.png"; --coverTitle="Apache Doris Docs (中文)" --outputPDFFilename="Apache Doris Docs (中文).pdf" --tocOnlyH1=true ls rm -rf ./build/** mkdir -p ./build/assets/files/ - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16812: [fix](cooldown)Fix bug for single cooldown compaction, add remote meta
github-actions[bot] commented on PR #16812: URL: https://github.com/apache/doris/pull/16812#issuecomment-1432812507 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] liaoxin01 opened a new pull request, #16832: [fix](merge-on-write) fix that be may coredump when sequence column is null
liaoxin01 opened a new pull request, #16832: URL: https://github.com/apache/doris/pull/16832 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16776: [Feature](map-type) Support stream load and fix some bugs for map type
github-actions[bot] commented on PR #16776: URL: https://github.com/apache/doris/pull/16776#issuecomment-1432824123 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] liaoxin01 opened a new issue, #16833: [Bug] (merge-on-write)be may coredump when sequence column is null
liaoxin01 opened a new issue, #16833: URL: https://github.com/apache/doris/issues/16833 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? Query id: 0-0 *** Aborted at 1675069046 (unix time) try "date -d @1675069046" if you are using GNU date *** Current BE git commitID: aaa19e9b4d *** SIGABRT unkown detail explain (@0x1cd9aa) received by PID 1890730 (TID 0x7f68d2a1f700) from PID 1890730; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/code/doris/be/src/common/signal_handler.h:420 1# 0x7F6AC60A3400 in /lib64/libc.so.6 2# gsignal in /lib64/libc.so.6 3# abort in /lib64/libc.so.6 4# 0x55E5F9D54CD9 in /root/code/doris/output/be/lib/doris_be 5# 0x55E5F9D4A2ED in /root/code/doris/output/be/lib/doris_be 6# google::LogMessage::SendToLog() in /root/code/doris/output/be/lib/doris_be 7# google::LogMessage::Flush() in /root/code/doris/output/be/lib/doris_be 8# google::LogMessageFatal::~LogMessageFatal() in /root/code/doris/output/be/lib/doris_be 9# doris::MemPool::allocate(long, bool) at /root/code/doris/be/src/runtime/mem_pool.h:108 10# doris::FieldTypeTraits<(doris::FieldType)13>::deep_copy(void*, void const*, doris::MemPool*) at /root/code/doris/be/src/olap/types.h:1426 11# doris::ScalarTypeInfo::deep_copy(void*, void const*, doris::MemPool*) const at /root/code/doris/be/src/olap/types.h:109 12# doris::segment_v2::(anonymous namespace)::BloomFilterIndexWriterImpl<(doris::FieldType)17>::add_values(void const*, unsigned long) at /root/code/doris/be/src/olap/rowset/segment_v2/bloom_filter_index_writer.cpp:84 13# doris::PrimaryKeyIndexBuilder::add_item(doris::Slice const&) at /root/code/doris/be/src/olap/primary_key_index.cpp:47 14# doris::segment_v2::SegmentWriter::append_block(doris::vectorized::Block const*, unsigned long, unsigned long) at /root/code/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:188 15# doris::BetaRowsetWriter::_add_block(doris::vectorized::Block const*, std::unique_ptr >*) at /root/code/doris/be/src/olap/rowset/beta_rowset_writer.cpp:574 16# doris::BetaRowsetWriter::flush_single_memtable(doris::vectorized::Block const*) at /root/code/doris/be/src/olap/rowset/beta_rowset_writer.cpp:699 17# doris::MemTable::_do_flush(long&) at /root/code/doris/be/src/olap/memtable.cpp:488 18# doris::MemTable::flush() at /root/code/doris/be/src/olap/memtable.cpp:455 19# doris::FlushToken::_flush_memtable(doris::MemTable*, long) at /root/code/doris/be/src/olap/memtable_flush_executor.cpp:90 20# doris::MemtableFlushTask::run() at /root/code/doris/be/src/olap/memtable_flush_executor.cpp:40 21# doris::ThreadPool::dispatch_thread() at /root/code/doris/be/src/util/threadpool.cpp:534 22# void std::_invoke_impl(std::_invoke_memfun_deref, void (doris::ThreadPool::&)(), doris::ThreadPool&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74 23# std::_invoke_result::type std::_invoke(void (doris::ThreadPool::&)(), doris::ThreadPool&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96 24# void std::Bind::_call(std::tuple<>&&, std::_Index_tuple<0ul>) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420 25# void std::_Bind::operator()<, void>() at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503 26# void std::_invoke_impl&>(std::_invoke_other, std::_Bind&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61 27# std::enable_if&>, void>::type std::_invoke_r&>(std::_Bind&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:117 28# std::_Function_handler >::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 29# std::function::operator()() const at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560 30# doris::Thread::supervise_thread(void*) at /root/code/doris/be/src/util/thread.cpp:454 31# start_thread in /lib64/libpthread.so.0 ### What You Expected? fix it ### How to Reproduce? load data into unique with merge-on-write table when sequence column is null. ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
[GitHub] [doris] github-actions[bot] commented on pull request #16832: [fix](merge-on-write) fix that be may coredump when sequence column is null
github-actions[bot] commented on PR #16832: URL: https://github.com/apache/doris/pull/16832#issuecomment-1432828022 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary commented on a diff in pull request #16806: [Enhancement](ES): Support mapping es date format
stalary commented on code in PR #16806:
URL: https://github.com/apache/doris/pull/16806#discussion_r1108262190
##
fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsUtil.java:
##
@@ -419,6 +421,69 @@ public static List genColumnsFromEs(EsRestClient
client, String indexNam
return columns;
}
+
+/**
+ * Generate columns from ES Cluster.
+ * Add mappingEsId config in es external catalog.
+ **/
+public static List genColumnsFromEs(EsRestClient client, String
indexName, String mappingType,
+boolean mappingEsId) {
+String mapping = client.getMapping(indexName);
+JSONObject mappingProps = getMappingProps(indexName, mapping,
mappingType);
+List arrayFields = getArrayFields(mapping);
+return genColumnsFromEs(mappingProps, arrayFields, mappingEsId);
+}
+
+private static final List ALLOW_DATE_FORMATS =
Lists.newArrayList("-MM-dd HH:mm:ss", "-MM-dd",
+"epoch_millis");
+
+/**
+ * Parse es date to doris type by format
+ **/
+private static Type parseEsDateType(Column column, JSONObject field) {
+if (!field.containsKey("format")) {
+// default format
+column.setComment("Elasticsearch type is date, no format");
+return ScalarType.createDatetimeV2Type(0);
+} else {
+String originFormat = field.get("format").toString();
+String[] formats = originFormat.split("\\|\\|");
+boolean dateTimeFlag = false;
+boolean dateFlag = false;
+boolean bigIntFlag = false;
+for (String format : formats) {
+// pre-check format
+if (!ALLOW_DATE_FORMATS.contains(format)) {
+column.setComment(
+"Elasticsearch type is date, format is " + format
+ " not support, use String type");
+return ScalarType.createStringType();
+}
+switch (format) {
+case "-MM-dd HH:mm:ss":
+dateTimeFlag = true;
+break;
+case "-MM-dd":
+dateFlag = true;
+break;
+case "epoch_millis":
+default:
+bigIntFlag = true;
+}
+}
+column.setComment("Elasticsearch type is date, format is " +
originFormat);
+if (dateTimeFlag) {
+return ScalarType.createDatetimeV2Type(0);
+}
+if (dateFlag) {
+return ScalarType.createDateV2Type();
+}
+if (bigIntFlag) {
+return Type.BIGINT;
Review Comment:
in es epoch_millis format use long
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary commented on pull request #16806: [Enhancement](ES): Support mapping es date format
stalary commented on PR #16806: URL: https://github.com/apache/doris/pull/16806#issuecomment-1432830341 > should handle this issue: > > ```sql > // if es data fomat is default or long > select * from es_table where date_col = "2023-02-16 12:12:12" > ``` Ok, let me handle it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] JNSimba commented on issue #100: [Bug] (concurrency) Thread safety problem of loading data to Doris
JNSimba commented on issue #100: URL: https://github.com/apache/doris-flink-connector/issues/100#issuecomment-1432848516 > When doing checkpoint, method dorisStreamLoad.startLoad() will be invoked and changes pendingLoadFuture, but in checkDone() dorisStreamLoad.getPendingLoadFuture() may get old pendingLoadFuture. Sorry, I didn't understand what you meant. Under normal circumstances, the Interrupt exception in the `checkDone()` method is due to the end of streamload, but it has not reached `dorisStreamLoad.stopLoad()`. When checkpointing, the `snapshotState` method will open a new pendingLoadFuture through `dorisStreamLoad.startLoad`. However, at this time, the `dorisStreamLoad.handlePreCommitResponse` step in the `checkDone` method should not be possible, right? Because `dorisStreamLoad.getPendingLoadFuture().isDone()` is flase? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] pengxiangyu commented on a diff in pull request #16803: [fix](cooldown) Use `pending_remote_rowsets` to avoid deleting rowset files being uploaded
pengxiangyu commented on code in PR #16803:
URL: https://github.com/apache/doris/pull/16803#discussion_r1108292224
##
be/src/olap/tablet.h:
##
@@ -328,7 +328,11 @@ class Tablet : public BaseTablet {
static void remove_unused_remote_files();
-std::shared_mutex& get_remote_files_lock() { return _remote_files_lock; }
+// If a rowset is to be written to remote filesystem, MUST add it to
`pending_remote_rowsets` before uploading,
+// and then erase it from `pending_remote_rowsets` after it has been
insert to the Tablet.
+// `remove_unused_remote_files` MUST NOT delete files of these pending
rowsets.
+static void add_pending_remote_rowset(std::string rowset_id);
Review Comment:
Why static？s_pending_remote_rowsets will be too large, and take too long
time when searching key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yongjinhou opened a new pull request, #16834: [Enhancement](HttpServer) Support https interface
yongjinhou opened a new pull request, #16834: URL: https://github.com/apache/doris/pull/16834 # Proposed changes Issue Number: close #15658 ## Problem summary Support https interface. ## Checklist(Required) * [x] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [x] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] pengxiangyu commented on a diff in pull request #16803: [fix](cooldown) Use `pending_remote_rowsets` to avoid deleting rowset files being uploaded
pengxiangyu commented on code in PR #16803:
URL: https://github.com/apache/doris/pull/16803#discussion_r1108302426
##
be/src/olap/tablet.cpp:
##
@@ -1768,7 +1765,10 @@ Status Tablet::_cooldown_data(const
std::shared_ptr& dest_
UniqueId cooldown_meta_id = UniqueId::gen_uid();
// upload cooldowned rowset meta to remote fs
-RETURN_IF_ERROR(_write_cooldown_meta(dest_fs.get(), cooldown_meta_id,
new_rowset_meta.get()));
+st = _write_cooldown_meta(dest_fs.get(), cooldown_meta_id,
new_rowset_meta.get());
+if (!st.ok()) {
Review Comment:
no erase_pending_remote_rowset
##
be/src/olap/tablet.cpp:
##
@@ -1736,20 +1736,17 @@ Status Tablet::_cooldown_data(const
std::shared_ptr& dest_
return Status::InternalError("cannot pick cooldown rowset in tablet
{}", tablet_id());
}
RowsetId new_rowset_id = StorageEngine::instance()->next_rowset_id();
-
-auto start = std::chrono::steady_clock::now();
-
+add_pending_remote_rowset(new_rowset_id.to_string());
Status st;
-{
-std::shared_lock slock(_remote_files_lock, std::try_to_lock);
-if (!slock.owns_lock()) {
-return Status::Status::Error("try
remote_files_lock failed");
+Defer defer {[&] {
+if (!st.ok()) {
+erase_pending_remote_rowset(new_rowset_id.to_string());
+// reclaim the incomplete rowset data in remote storage
+record_unused_remote_rowset(new_rowset_id, dest_fs->id(),
old_rowset->num_segments());
}
-st = old_rowset->upload_to(dest_fs.get(), new_rowset_id);
-}
-if (!st.ok()) {
-// reclaim the incomplete rowset data in remote storage
-record_unused_remote_rowset(new_rowset_id, dest_fs->id(),
old_rowset->num_segments());
+}};
+auto start = std::chrono::steady_clock::now();
+if (st = old_rowset->upload_to(dest_fs.get(), new_rowset_id); !st.ok()) {
Review Comment:
no erase_pending_remote_rowset
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] pengxiangyu commented on a diff in pull request #16803: [fix](cooldown) Use `pending_remote_rowsets` to avoid deleting rowset files being uploaded
pengxiangyu commented on code in PR #16803: URL: https://github.com/apache/doris/pull/16803#discussion_r1108308600 ## be/src/olap/tablet.cpp: ## @@ -1768,7 +1765,10 @@ Status Tablet::_cooldown_data(const std::shared_ptr& dest_ UniqueId cooldown_meta_id = UniqueId::gen_uid(); // upload cooldowned rowset meta to remote fs -RETURN_IF_ERROR(_write_cooldown_meta(dest_fs.get(), cooldown_meta_id, new_rowset_meta.get())); +st = _write_cooldown_meta(dest_fs.get(), cooldown_meta_id, new_rowset_meta.get()); Review Comment: Why remove RETURN_IF_ERROR？ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] XieJiann commented on a diff in pull request #16797: [feature](Nereids): add rule split limit into two phase
XieJiann commented on code in PR #16797:
URL: https://github.com/apache/doris/pull/16797#discussion_r1108309568
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalLimit.java:
##
@@ -44,19 +45,28 @@
* offset 100
*/
public class LogicalLimit extends
LogicalUnary implements Limit {
-
+private final LimitPhase phase;
private final long limit;
private final long offset;
-public LogicalLimit(long limit, long offset, CHILD_TYPE child) {
-this(limit, offset, Optional.empty(), Optional.empty(), child);
+public LogicalLimit(long limit, long offset, CHILD_TYPE child, LimitPhase
phase) {
+this(limit, offset, Optional.empty(), Optional.empty(), child, phase);
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] XieJiann commented on a diff in pull request #16797: [feature](Nereids): add rule split limit into two phase
XieJiann commented on code in PR #16797:
URL: https://github.com/apache/doris/pull/16797#discussion_r1108310075
##
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##
@@ -1338,38 +1338,15 @@ public PlanFragment visitPhysicalLimit(PhysicalLimit physicalLim
if (inputFragment == null) {
return inputFragment;
}
-
+// For case globalLimit(l, o) -> LocalLimit(l+o, 0), that is the
LocalLimit has already gathered
+// The globalLimit can overwrite the limit and offset, so it's still
correct
PlanNode child = inputFragment.getPlanRoot();
-
-// physical plan: limit --> sort
-// after translate, it could be:
-// 1. limit->sort => set (limit and offset) on sort
-// 2. limit->exchange->sort => set (limit and offset) on exchange, set
sort.limit = limit+offset
-if (child instanceof SortNode) {
-SortNode sort = (SortNode) child;
-sort.setLimit(physicalLimit.getLimit());
-sort.setOffset(physicalLimit.getOffset());
-return inputFragment;
-}
-if (child instanceof ExchangeNode) {
-ExchangeNode exchangeNode = (ExchangeNode) child;
-exchangeNode.setLimit(physicalLimit.getLimit());
-// we do not check if this is a merging exchange here,
-// since this guaranteed by translating logic plan to physical plan
-exchangeNode.setOffset(physicalLimit.getOffset());
-if (exchangeNode.getChild(0) instanceof SortNode) {
-SortNode sort = (SortNode) exchangeNode.getChild(0);
-sort.setLimit(physicalLimit.getLimit() +
physicalLimit.getOffset());
-sort.setOffset(0);
-}
-return inputFragment;
-}
-// for other PlanNode, just set limit as limit+offset
-child.setLimit(physicalLimit.getLimit() + physicalLimit.getOffset());
-PlanFragment planFragment = exchangeToMergeFragment(inputFragment,
context);
-planFragment.getPlanRoot().setLimit(physicalLimit.getLimit());
-
planFragment.getPlanRoot().setOffSetDirectly(physicalLimit.getOffset());
-return planFragment;
+child.setLimit(physicalLimit.getLimit());
+child.setOffset(physicalLimit.getOffset());
+// TODO: add a rule in cascades
+// before: GlobalLimit -> LocalLimit -> GlobalSort -> Gather ->
LocalSort
+// after: GlobalLimit -> GlobalSort -> Gather ->LocalLimit ->
LocalSort
+return inputFragment;
Review Comment:
add a rule limit->sort => topn in LimitPushDown Rule
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangGuoqiang666 opened a new issue, #16835: [Enhancement] change the teamcity pipeline trigger : triggered by github pull request comment
ZhangGuoqiang666 opened a new issue, #16835: URL: https://github.com/apache/doris/issues/16835 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description change the teamcity pipeline trigger : triggered by github pull request comment ### Solution change the teamcity pipeline trigger : triggered by github pull request comment Now we'll update the pipeline trigger on teamcity to include the following special keyword comments in the PULL REQUEST COMMENT section. Key words: buildall: This keyword will trigger other pipelinings on teamcity other than external compile: This keyword will trigger the code compilation pipeline for the corresponding branch feut: This keyword will trigger the FE unit test pipeline beut: This keyword will trigger the BE unit test pipeline p0: This keyword will trigger the test pipeline for the P0 use case p1: This keyword will trigger the test pipeline for the P1 use case external: This keyword will trigger tests for the external component Note: p0,p1, and external pipelining all depend on compile-phase output, so when you want to retry one of these pipelining separately, you need to make sure that the corresponding branch has compiled output. If not, please first run the compile phase by adding the keyword with compile in the comment area of the corresponding pull request. Build the dependency pipeline after compiling. For example: When developers add a comment with a buildall keyword to the Conversation area of the PULL REQUEST, for example: "run buildall cases", git action will detect this behavior and trigger FE single test, BE single test, and code compilation pipeline. when the code compilation pipeline. p0 and p1 pipeline. if you want to execute data lake related use cases simultaneously, You can add external to buildall, and the external pipeline will automatically start the data lake environment to perform the relevant tests. When developers find a pipeline running error but want to retry it, they can add a comment with the pipeline keyword in the Conversation area of the PULL REQUEST, such as rerun p1. If p1 fails the test due to lack of compilation output, So we need to add a comment with the compile keyword first, and after successful compilation, p0,p1 pipeline will be automatically triggered. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] XieJiann commented on a diff in pull request #16797: [feature](Nereids): add rule split limit into two phase
XieJiann commented on code in PR #16797:
URL: https://github.com/apache/doris/pull/16797#discussion_r1108310075
##
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##
@@ -1338,38 +1338,15 @@ public PlanFragment visitPhysicalLimit(PhysicalLimit physicalLim
if (inputFragment == null) {
return inputFragment;
}
-
+// For case globalLimit(l, o) -> LocalLimit(l+o, 0), that is the
LocalLimit has already gathered
+// The globalLimit can overwrite the limit and offset, so it's still
correct
PlanNode child = inputFragment.getPlanRoot();
-
-// physical plan: limit --> sort
-// after translate, it could be:
-// 1. limit->sort => set (limit and offset) on sort
-// 2. limit->exchange->sort => set (limit and offset) on exchange, set
sort.limit = limit+offset
-if (child instanceof SortNode) {
-SortNode sort = (SortNode) child;
-sort.setLimit(physicalLimit.getLimit());
-sort.setOffset(physicalLimit.getOffset());
-return inputFragment;
-}
-if (child instanceof ExchangeNode) {
-ExchangeNode exchangeNode = (ExchangeNode) child;
-exchangeNode.setLimit(physicalLimit.getLimit());
-// we do not check if this is a merging exchange here,
-// since this guaranteed by translating logic plan to physical plan
-exchangeNode.setOffset(physicalLimit.getOffset());
-if (exchangeNode.getChild(0) instanceof SortNode) {
-SortNode sort = (SortNode) exchangeNode.getChild(0);
-sort.setLimit(physicalLimit.getLimit() +
physicalLimit.getOffset());
-sort.setOffset(0);
-}
-return inputFragment;
-}
-// for other PlanNode, just set limit as limit+offset
-child.setLimit(physicalLimit.getLimit() + physicalLimit.getOffset());
-PlanFragment planFragment = exchangeToMergeFragment(inputFragment,
context);
-planFragment.getPlanRoot().setLimit(physicalLimit.getLimit());
-
planFragment.getPlanRoot().setOffSetDirectly(physicalLimit.getOffset());
-return planFragment;
+child.setLimit(physicalLimit.getLimit());
+child.setOffset(physicalLimit.getOffset());
+// TODO: add a rule in cascades
+// before: GlobalLimit -> LocalLimit -> GlobalSort -> Gather ->
LocalSort
+// after: GlobalLimit -> GlobalSort -> Gather ->LocalLimit ->
LocalSort
+return inputFragment;
Review Comment:
fixed, add a rule limit->sort => topn in LimitPushDown Rule
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] XieJiann commented on a diff in pull request #16797: [feature](Nereids): add rule split limit into two phase
XieJiann commented on code in PR #16797:
URL: https://github.com/apache/doris/pull/16797#discussion_r1108310075
##
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##
@@ -1338,38 +1338,15 @@ public PlanFragment visitPhysicalLimit(PhysicalLimit physicalLim
if (inputFragment == null) {
return inputFragment;
}
-
+// For case globalLimit(l, o) -> LocalLimit(l+o, 0), that is the
LocalLimit has already gathered
+// The globalLimit can overwrite the limit and offset, so it's still
correct
PlanNode child = inputFragment.getPlanRoot();
-
-// physical plan: limit --> sort
-// after translate, it could be:
-// 1. limit->sort => set (limit and offset) on sort
-// 2. limit->exchange->sort => set (limit and offset) on exchange, set
sort.limit = limit+offset
-if (child instanceof SortNode) {
-SortNode sort = (SortNode) child;
-sort.setLimit(physicalLimit.getLimit());
-sort.setOffset(physicalLimit.getOffset());
-return inputFragment;
-}
-if (child instanceof ExchangeNode) {
-ExchangeNode exchangeNode = (ExchangeNode) child;
-exchangeNode.setLimit(physicalLimit.getLimit());
-// we do not check if this is a merging exchange here,
-// since this guaranteed by translating logic plan to physical plan
-exchangeNode.setOffset(physicalLimit.getOffset());
-if (exchangeNode.getChild(0) instanceof SortNode) {
-SortNode sort = (SortNode) exchangeNode.getChild(0);
-sort.setLimit(physicalLimit.getLimit() +
physicalLimit.getOffset());
-sort.setOffset(0);
-}
-return inputFragment;
-}
-// for other PlanNode, just set limit as limit+offset
-child.setLimit(physicalLimit.getLimit() + physicalLimit.getOffset());
-PlanFragment planFragment = exchangeToMergeFragment(inputFragment,
context);
-planFragment.getPlanRoot().setLimit(physicalLimit.getLimit());
-
planFragment.getPlanRoot().setOffSetDirectly(physicalLimit.getOffset());
-return planFragment;
+child.setLimit(physicalLimit.getLimit());
+child.setOffset(physicalLimit.getOffset());
+// TODO: add a rule in cascades
+// before: GlobalLimit -> LocalLimit -> GlobalSort -> Gather ->
LocalSort
+// after: GlobalLimit -> GlobalSort -> Gather ->LocalLimit ->
LocalSort
+return inputFragment;
Review Comment:
fixed, add a rule `limit->sort` => `topn` in LimitPushDown Rule
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16803: [fix](cooldown) Use `pending_remote_rowsets` to avoid deleting rowset files being uploaded
github-actions[bot] commented on PR #16803: URL: https://github.com/apache/doris/pull/16803#issuecomment-1432905995 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16803: [fix](cooldown) Use `pending_remote_rowsets` to avoid deleting rowset files being uploaded
github-actions[bot] commented on PR #16803: URL: https://github.com/apache/doris/pull/16803#issuecomment-1432906074 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16812: [fix](cooldown)Fix bug for single cooldown compaction, add remote meta
github-actions[bot] commented on PR #16812: URL: https://github.com/apache/doris/pull/16812#issuecomment-1432913774 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangGuoqiang666 opened a new pull request, #16836: [enhancement] change the teamcity pipeline trigger : triggered by github pull request comment
ZhangGuoqiang666 opened a new pull request, #16836: URL: https://github.com/apache/doris/pull/16836 # Proposed changes Issue Number: close #16835 ## Problem summary Describe your changes. Optimized some code and Reduce invalid code,fix syntax error ## Checklist(Required) * [ no] Does it affect the original behavior * [ no] Has unit tests been added * [ no] Has document been added or modified * [ no] Does it need to update dependencies * [yes ] Is this PR support rollback (If NO, please explain WHY) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16831: [Chore](build) remove memory_copy and remove some wno build check
hello-stephen commented on PR #16831: URL: https://github.com/apache/doris/pull/16831#issuecomment-1432925543 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.83 seconds stream load tsv: 478 seconds loaded 74807831229 Bytes, about 149 MB/s stream load json: 36 seconds loaded 2358488459 Bytes, about 62 MB/s stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230216111428_clickbench_pr_98035.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16799: [enhancement](merge-on-write) do compaction with merge on read
yiguolei merged PR #16799: URL: https://github.com/apache/doris/pull/16799 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhancement](merge-on-write) do compaction with merge on read (#16799)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 2a9e748073 [enhancement](merge-on-write) do compaction with merge on
read (#16799)
2a9e748073 is described below
commit 2a9e7480733d8e908c8045eb73f9a99a677a0ea9
Author: Xin Liao
AuthorDate: Thu Feb 16 19:20:15 2023 +0800
[enhancement](merge-on-write) do compaction with merge on read (#16799)
To avoid data irrecoverable due to delete bitmap calculation error，do
compaction with merge on read. Through this way ，even if the delete bitmap
calculation is wrong, the data can be recovered by full compaction.
---
be/src/olap/compaction.cpp | 35 ++-
be/src/olap/merger.cpp | 7 ---
be/src/olap/tablet.cpp | 43 +++
be/src/olap/tablet.h| 5 +
be/src/olap/tablet_meta.cpp | 38 --
be/src/olap/tablet_meta.h | 3 ---
6 files changed, 74 insertions(+), 57 deletions(-)
diff --git a/be/src/olap/compaction.cpp b/be/src/olap/compaction.cpp
index b7fedd2203..49526b6382 100644
--- a/be/src/olap/compaction.cpp
+++ b/be/src/olap/compaction.cpp
@@ -413,19 +413,36 @@ Status Compaction::construct_input_rowset_readers() {
Status Compaction::modify_rowsets() {
std::vector output_rowsets;
output_rowsets.push_back(_output_rowset);
-{
-std::lock_guard wrlock_(_tablet->get_rowset_update_lock());
-std::lock_guard wrlock(_tablet->get_header_lock());
-// update dst rowset delete bitmap
-if (_tablet->keys_type() == KeysType::UNIQUE_KEYS &&
-_tablet->enable_unique_key_merge_on_write()) {
-_tablet->tablet_meta()->update_delete_bitmap(
-_input_rowsets, _output_rs_writer->version(),
_rowid_conversion);
+if (_tablet->keys_type() == KeysType::UNIQUE_KEYS &&
+_tablet->enable_unique_key_merge_on_write()) {
+Version version = _tablet->max_version();
+DeleteBitmap output_rowset_delete_bitmap(_tablet->tablet_id());
+// Convert the delete bitmap of the input rowsets to output rowset.
+// New loads are not blocked, so some keys of input rowsets might
+// be deleted during the time. We need to deal with delete bitmap
+// of incremental data later.
+_tablet->calc_compaction_output_rowset_delete_bitmap(_input_rowsets,
_rowid_conversion, 0,
+ version.second +
1,
+
&output_rowset_delete_bitmap);
+{
+std::lock_guard
wrlock_(_tablet->get_rowset_update_lock());
+std::lock_guard
wrlock(_tablet->get_header_lock());
+
+// Convert the delete bitmap of the input rowsets to output rowset
for
+// incremental data.
+
_tablet->calc_compaction_output_rowset_delete_bitmap(_input_rowsets,
_rowid_conversion,
+
version.second, UINT64_MAX,
+
&output_rowset_delete_bitmap);
+
+_tablet->merge_delete_bitmap(output_rowset_delete_bitmap);
+RETURN_NOT_OK(_tablet->modify_rowsets(output_rowsets,
_input_rowsets, true));
}
-
+} else {
+std::lock_guard wrlock(_tablet->get_header_lock());
RETURN_NOT_OK(_tablet->modify_rowsets(output_rowsets, _input_rowsets,
true));
}
+
{
std::shared_lock rlock(_tablet->get_header_lock());
_tablet->save_meta();
diff --git a/be/src/olap/merger.cpp b/be/src/olap/merger.cpp
index b350882e4b..7e2a620d95 100644
--- a/be/src/olap/merger.cpp
+++ b/be/src/olap/merger.cpp
@@ -57,9 +57,6 @@ Status Merger::vmerge_rowsets(TabletSharedPtr tablet,
ReaderType reader_type,
merge_tablet_schema->merge_dropped_columns(tablet->tablet_schema(del_pred_rs->version()));
}
reader_params.tablet_schema = merge_tablet_schema;
-if (tablet->enable_unique_key_merge_on_write()) {
-reader_params.delete_bitmap = &tablet->tablet_meta()->delete_bitmap();
-}
if (stats_output && stats_output->rowid_conversion) {
reader_params.record_rowids = true;
@@ -193,10 +190,6 @@ Status Merger::vertical_compact_one_group(
}
reader_params.tablet_schema = merge_tablet_schema;
-if (tablet->enable_unique_key_merge_on_write()) {
-reader_params.delete_bitmap = &tablet->tablet_meta()->delete_bitmap();
-}
-
reader_params.return_columns = column_group;
reader_params.origin_return_columns = &reader_params.return_columns;
RETURN_NOT_OK(reader.init(reader_params));
diff --git a/be/src/olap/tablet.cpp b/be/src/olap/tablet.cpp
index f7b36a2d40..f0ea13 100644
[GitHub] [doris] yiguolei commented on a diff in pull request #16822: [improvement](memory) fix possible memory leak of vcollect iterator
yiguolei commented on code in PR #16822:
URL: https://github.com/apache/doris/pull/16822#discussion_r1108342805
##
be/src/vec/olap/vcollect_iterator.cpp:
##
@@ -166,15 +158,17 @@ Status
VCollectIterator::build_heap(std::vector& rs_reade
++iter;
}
}
-_inner_iter.reset(new Level1Iterator(_children, _reader, _merge,
_is_reverse, _skip_same));
+_inner_iter.reset(
+new Level1Iterator(std::move(_children), _reader, _merge,
_is_reverse, _skip_same));
}
RETURN_IF_NOT_EOF_AND_OK(_inner_iter->init());
// Clear _children earlier to release any related references
_children.clear();
return Status::OK();
}
-bool VCollectIterator::LevelIteratorComparator::operator()(LevelIterator* lhs,
LevelIterator* rhs) {
+bool VCollectIterator::LevelIteratorComparator::operator()(LevelIteratorSPtr
lhs,
Review Comment:
use shared ptr here may be very slow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] catpineapple opened a new pull request, #16837: [fix](dbt)fix dbt incremental
catpineapple opened a new pull request, #16837: URL: https://github.com/apache/doris/pull/16837 # Proposed changes Issue Number: close #xxx ## Problem summary 1. fix dbt incremental :new ideas for no rollback and support incremental data rerun . 2. add snapshot 3. use 'mysql-connector-python' mysql driver to replace 'MysqlDb' driver ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16568: [Improve](regression) Add timeout check in schema change regression
yiguolei merged PR #16568: URL: https://github.com/apache/doris/pull/16568 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Improve](regression) Add timeout check in schema change regression (#16568)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 3c641d4465 [Improve](regression) Add timeout check in schema change
regression (#16568)
3c641d4465 is described below
commit 3c641d44659e9775a3032936063b26ae4806bba1
Author: Lightman <31928846+lchangli...@users.noreply.github.com>
AuthorDate: Thu Feb 16 19:27:22 2023 +0800
[Improve](regression) Add timeout check in schema change regression (#16568)
When the PRs run CI, the BE crash. The case will be always running. It
wastes the resource. So modify it and add timeout check.
---
.../test_agg_keys_schema_change_datev2.groovy | 12 +++---
.../test_dup_keys_schema_change_datev2.groovy | 6 +--
.../test_agg_keys_schema_change_decimalv3.groovy | 10 ++---
.../test_agg_keys_schema_change.groovy | 43 -
.../test_agg_mv_schema_change.groovy | 45 ++
.../test_agg_rollup_schema_change.groovy | 45 ++
.../test_alter_table_column.groovy | 2 +-
.../test_dup_keys_schema_change.groovy | 23 ++-
.../test_dup_mv_schema_change.groovy | 45 ++
.../test_dup_rollup_schema_change.groovy | 45 ++
.../schema_change_p0/test_rename_column.groovy | 42 +---
.../schema_change_p0/test_schema_change.groovy | 2 +-
.../test_uniq_mv_schema_change.groovy | 22 +++
.../test_uniq_rollup_schema_change.groovy | 45 ++
14 files changed, 237 insertions(+), 150 deletions(-)
diff --git
a/regression-test/suites/schema_change_p0/datev2/test_agg_keys_schema_change_datev2.groovy
b/regression-test/suites/schema_change_p0/datev2/test_agg_keys_schema_change_datev2.groovy
index f96067a8a2..05dbb36d7f 100644
---
a/regression-test/suites/schema_change_p0/datev2/test_agg_keys_schema_change_datev2.groovy
+++
b/regression-test/suites/schema_change_p0/datev2/test_agg_keys_schema_change_datev2.groovy
@@ -133,7 +133,7 @@ suite("test_agg_keys_schema_change_datev2") {
sql """ alter table ${tbName} add column `datev3` datev2 DEFAULT
'2022-01-01' """
int max_try_time = 1000
-while(max_try_time--){
+while (max_try_time--){
String result = getJobState(tbName)
if (result == "FINISHED") {
break
@@ -154,7 +154,7 @@ suite("test_agg_keys_schema_change_datev2") {
qt_sql """select * from ${tbName} ORDER BY `datek1`;"""
sql """ alter table ${tbName} drop column `datev3` """
max_try_time = 1000
-while(max_try_time--){
+while (max_try_time--){
String result = getJobState(tbName)
if (result == "FINISHED") {
break
@@ -182,7 +182,7 @@ suite("test_agg_keys_schema_change_datev2") {
qt_sql """select * from ${tbName} ORDER BY `datek1`;"""
sql """ alter table ${tbName} add column `datev3` datetimev2 DEFAULT
'2022-01-01 11:11:11' """
max_try_time = 1000
-while(max_try_time--){
+while (max_try_time--){
String result = getJobState(tbName)
if (result == "FINISHED") {
break
@@ -203,7 +203,7 @@ suite("test_agg_keys_schema_change_datev2") {
qt_sql """select * from ${tbName} ORDER BY `datek1`;"""
sql """ alter table ${tbName} drop column `datev3` """
max_try_time = 1000
-while(max_try_time--){
+while (max_try_time--){
String result = getJobState(tbName)
if (result == "FINISHED") {
break
@@ -231,7 +231,7 @@ suite("test_agg_keys_schema_change_datev2") {
qt_sql """select * from ${tbName} ORDER BY `datek1`;"""
sql """ alter table ${tbName} add column `datev3` datetimev2(3) DEFAULT
'2022-01-01 11:11:11.111' """
max_try_time = 1000
-while(max_try_time--){
+while (max_try_time--){
String result = getJobState(tbName)
if (result == "FINISHED") {
break
@@ -260,7 +260,7 @@ suite("test_agg_keys_schema_change_datev2") {
qt_sql """select * from ${tbName} ORDER BY `datek1`;"""
sql """ alter table ${tbName} drop column `datev3` """
max_try_time = 1000
-while(max_try_time--){
+while (max_try_time--){
String result = getJobState(tbName)
if (result == "FINISHED") {
break
diff --git
a/regression-test/suites/schema_change_p0/datev2/test_dup_keys_schema_change_datev2.groovy
b/regression-test/suites/schema_change_p0/datev2/test_dup_keys_schema_change_datev2.groovy
index 8e17145d39..966eb2b811 100644
---
a/regression-test/suites/schema_change_p0/datev2/test_dup_keys_schema_change_datev2.groovy
+++
b/regression-test/suites/schema_change_p0/datev2/test_dup_keys_schema_change_datev2.groovy
@@ -128,7 +128,7 @@ suite("test_dup_keys_schema_change_d
[GitHub] [doris] catpineapple closed pull request #14767: [fix](dbt)fix dbt incremental
catpineapple closed pull request #14767: [fix](dbt)fix dbt incremental URL: https://github.com/apache/doris/pull/14767 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] catpineapple commented on pull request #14767: [fix](dbt)fix dbt incremental
catpineapple commented on PR #14767: URL: https://github.com/apache/doris/pull/14767#issuecomment-1432944005 new pull request https://github.com/apache/doris/pull/16837 replace this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] airborne12 opened a new issue, #16838: [Bug](inverted index) searching inverted index of numeric array column returns error result
airborne12 opened a new issue, #16838:
URL: https://github.com/apache/doris/issues/16838
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no
similar issues.
### Version
master
### What's Wrong?
`SELECT children FROM hackernews_1m WHERE children element_eq 12148 LIMIT
10; `
return empty result
### What You Expected?
`SELECT children FROM hackernews_1m WHERE children element_eq 12148 LIMIT
10;`
return one result
### How to Reproduce?
- How to reproduce problem
1. create table
`
CREATE DATABASE test_inverted_index;
USE test_inverted_index;
-- define inverted index idx_comment for comment column on table creation
-- USING INVERTED specify using inverted index
-- PROPERTIES("parser" = "english") specify english word parser
CREATE TABLE hackernews_1m
(
`id` BIGINT,
`deleted` TINYINT,
`type` String,
`author` String,
`timestamp` DateTimeV2,
`comment` String,
`dead` TINYINT,
`parent` BIGINT,
`poll` BIGINT,
`children` Array,
`url` String,
`score` INT,
`title` String,
`parts` Array,
`descendants` INT,
INDEX idx_comment (`comment`) USING INVERTED PROPERTIES("parser" =
"english") COMMENT 'inverted index for comment'
)
DUPLICATE KEY(`id`)
DISTRIBUTED BY HASH(`id`) BUCKETS 10
PROPERTIES ("replication_num" = "1");
`
2. load data
`wget
https://doris-build-1308700295.cos.ap-beijing.myqcloud.com/regression/index/hacknernews_1m.csv.gz
curl --location-trusted -u root: -H "compress_type:gz" -T
hacknernews_1m.csv.gz
http://127.0.0.1:8030/api/test_inverted_index/hackernews_1m/_stream_load
{
"TxnId": 2,
"Label": "a8a3e802-2329-49e8-912b-04c800a461a6",
"TwoPhaseCommit": "false",
"Status": "Success",
"Message": "OK",
"NumberTotalRows": 100,
"NumberLoadedRows": 100,
"NumberFilteredRows": 0,
"NumberUnselectedRows": 0,
"LoadBytes": 130618406,
"LoadTimeMs": 8988,
"BeginTxnTimeMs": 23,
"StreamLoadPutTimeMs": 113,
"ReadDataTimeMs": 4788,
"WriteDataTimeMs": 8811,
"CommitAndPublishTimeMs": 38
}`
3. add index for array(bigint)
`CREATE INDEX idx_children ON hackernews_1m(children) USING INVERTED`
4. query data
`SELECT children FROM hackernews_1m WHERE children element_eq 12148 LIMIT
10;`
### Anything Else?
none
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] airborne12 opened a new pull request, #16839: [Fix](inverted index) fix array inverted index error match result when doing schema change add index
airborne12 opened a new pull request, #16839: URL: https://github.com/apache/doris/pull/16839 # Proposed changes Issue Number: close #16838 ## Problem summary There is a bug in inverted_index_writer when adding multiple lines array values' index. This problem can cause error result when doing schema change adding index. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16771: [enhancement](exception safe) make function state exception safe
github-actions[bot] commented on PR #16771: URL: https://github.com/apache/doris/pull/16771#issuecomment-1432950645 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16839: [Fix](inverted index) fix array inverted index error match result when doing schema change add index
github-actions[bot] commented on PR #16839: URL: https://github.com/apache/doris/pull/16839#issuecomment-1432953497 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16836: [enhancement] change the teamcity pipeline trigger : triggered by github pull request comment
github-actions[bot] commented on PR #16836: URL: https://github.com/apache/doris/pull/16836#issuecomment-1432965933 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16836: [enhancement] change the teamcity pipeline trigger : triggered by github pull request comment
github-actions[bot] commented on PR #16836: URL: https://github.com/apache/doris/pull/16836#issuecomment-1432965986 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman closed issue #16835: [Enhancement] change the teamcity pipeline trigger : triggered by github pull request comment
morningman closed issue #16835: [Enhancement] change the teamcity pipeline trigger : triggered by github pull request comment URL: https://github.com/apache/doris/issues/16835 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (3c641d4465 -> f86e8ec395)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 3c641d4465 [Improve](regression) Add timeout check in schema change regression (#16568) add f86e8ec395 [enhancement] change the teamcity pipeline trigger : triggered by github pull request comment (#16836) No new revisions were added by this update. Summary of changes: .github/workflows/auto_trigger_teamcity.yml | 29 +++-- 1 file changed, 27 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
