[GitHub] [doris] github-actions[bot] commented on pull request #15830: [enhancement](load) change transaction limit from global level to db level
github-actions[bot] commented on PR #15830: URL: https://github.com/apache/doris/pull/15830#issuecomment-1420348984 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15830: [enhancement](load) change transaction limit from global level to db level
github-actions[bot] commented on PR #15830: URL: https://github.com/apache/doris/pull/15830#issuecomment-1420349043 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zddr commented on a diff in pull request #16091: [feature](auth)Implementing privilege management with rbac model
zddr commented on code in PR #16091:
URL: https://github.com/apache/doris/pull/16091#discussion_r1098304753
##
fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/PrivTable.java:
##
@@ -72,37 +61,29 @@ public boolean hasPriv(String host, String user,
PrivPredicate wanted) {
* NOTICE, this method does not set password for the newly added entry if
this is a user priv table, the caller
* need to set password later.
*/
-public PrivEntry addEntry(PrivEntry newEntry, boolean errOnExist, boolean
errOnNonExist) throws DdlException {
+public PrivEntry addEntry(PrivEntry newEntry,
+boolean errOnExist, boolean errOnNonExist) throws DdlException {
PrivEntry existingEntry = getExistingEntry(newEntry);
if (existingEntry == null) {
if (errOnNonExist) {
-throw new DdlException("User " + newEntry.getUserIdent() + "
does not exist");
+throw new DdlException("entry does not exist");
}
entries.add(newEntry);
Collections.sort(entries);
LOG.info("add priv entry: {}", newEntry);
return newEntry;
} else {
if (errOnExist) {
-throw new DdlException("User already exist");
+throw new DdlException("entry already exist");
} else {
-checkOperationAllowed(existingEntry, newEntry, "ADD ENTRY");
-// if existing entry is set by domain resolver, just replace
it with the new entry.
-// if existing entry is not set by domain resolver, merge the
2 entries.
-if (existingEntry.isSetByDomainResolver()) {
-existingEntry.setPrivSet(newEntry.getPrivSet());
-
existingEntry.setSetByDomainResolver(newEntry.isSetByDomainResolver());
-LOG.debug("reset priv entry: {}", existingEntry);
-} else if (!newEntry.isSetByDomainResolver()) {
-mergePriv(existingEntry, newEntry);
-existingEntry.setSetByDomainResolver(false);
-LOG.debug("merge priv entry: {}", existingEntry);
-}
+mergePriv(existingEntry, newEntry);
+LOG.debug("merge priv entry: {}", existingEntry);
Review Comment:
"checkOperationAllowed" ===>now,PrivTable only related with 'role',do not
exist this logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zddr commented on a diff in pull request #16091: [feature](auth)Implementing privilege management with rbac model
zddr commented on code in PR #16091:
URL: https://github.com/apache/doris/pull/16091#discussion_r1098306625
##
fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/PrivTable.java:
##
@@ -72,37 +61,29 @@ public boolean hasPriv(String host, String user,
PrivPredicate wanted) {
* NOTICE, this method does not set password for the newly added entry if
this is a user priv table, the caller
* need to set password later.
*/
-public PrivEntry addEntry(PrivEntry newEntry, boolean errOnExist, boolean
errOnNonExist) throws DdlException {
+public PrivEntry addEntry(PrivEntry newEntry,
+boolean errOnExist, boolean errOnNonExist) throws DdlException {
PrivEntry existingEntry = getExistingEntry(newEntry);
if (existingEntry == null) {
if (errOnNonExist) {
-throw new DdlException("User " + newEntry.getUserIdent() + "
does not exist");
+throw new DdlException("entry does not exist");
}
entries.add(newEntry);
Collections.sort(entries);
LOG.info("add priv entry: {}", newEntry);
return newEntry;
} else {
if (errOnExist) {
-throw new DdlException("User already exist");
+throw new DdlException("entry already exist");
} else {
-checkOperationAllowed(existingEntry, newEntry, "ADD ENTRY");
-// if existing entry is set by domain resolver, just replace
it with the new entry.
-// if existing entry is not set by domain resolver, merge the
2 entries.
-if (existingEntry.isSetByDomainResolver()) {
-existingEntry.setPrivSet(newEntry.getPrivSet());
-
existingEntry.setSetByDomainResolver(newEntry.isSetByDomainResolver());
-LOG.debug("reset priv entry: {}", existingEntry);
-} else if (!newEntry.isSetByDomainResolver()) {
-mergePriv(existingEntry, newEntry);
-existingEntry.setSetByDomainResolver(false);
-LOG.debug("merge priv entry: {}", existingEntry);
-}
+mergePriv(existingEntry, newEntry);
+LOG.debug("merge priv entry: {}", existingEntry);
Review Comment:
create user user1@domain -> domain(ip1, ip2)
create user user1@ip1
How to handle it?
===>
update user1@ip1.setByResolver to false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zddr commented on a diff in pull request #16091: [feature](auth)Implementing privilege management with rbac model
zddr commented on code in PR #16091:
URL: https://github.com/apache/doris/pull/16091#discussion_r1098308500
##
fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/PrivTable.java:
##
@@ -72,37 +61,29 @@ public boolean hasPriv(String host, String user,
PrivPredicate wanted) {
* NOTICE, this method does not set password for the newly added entry if
this is a user priv table, the caller
* need to set password later.
*/
-public PrivEntry addEntry(PrivEntry newEntry, boolean errOnExist, boolean
errOnNonExist) throws DdlException {
+public PrivEntry addEntry(PrivEntry newEntry,
+boolean errOnExist, boolean errOnNonExist) throws DdlException {
PrivEntry existingEntry = getExistingEntry(newEntry);
if (existingEntry == null) {
if (errOnNonExist) {
-throw new DdlException("User " + newEntry.getUserIdent() + "
does not exist");
+throw new DdlException("entry does not exist");
}
entries.add(newEntry);
Collections.sort(entries);
LOG.info("add priv entry: {}", newEntry);
return newEntry;
} else {
if (errOnExist) {
-throw new DdlException("User already exist");
+throw new DdlException("entry already exist");
} else {
-checkOperationAllowed(existingEntry, newEntry, "ADD ENTRY");
-// if existing entry is set by domain resolver, just replace
it with the new entry.
-// if existing entry is not set by domain resolver, merge the
2 entries.
-if (existingEntry.isSetByDomainResolver()) {
-existingEntry.setPrivSet(newEntry.getPrivSet());
-
existingEntry.setSetByDomainResolver(newEntry.isSetByDomainResolver());
-LOG.debug("reset priv entry: {}", existingEntry);
-} else if (!newEntry.isSetByDomainResolver()) {
-mergePriv(existingEntry, newEntry);
-existingEntry.setSetByDomainResolver(false);
-LOG.debug("merge priv entry: {}", existingEntry);
-}
+mergePriv(existingEntry, newEntry);
+LOG.debug("merge priv entry: {}", existingEntry);
Review Comment:
How to handle domain now?
=>
now,domainUser,ipUser,and resolveUserByDomain all manage By userManager,The
logic of scheduled tasks is the same as before
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15830: [enhancement](load) change transaction limit from global level to db level
github-actions[bot] commented on PR #15830: URL: https://github.com/apache/doris/pull/15830#issuecomment-1420361757 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yixiutt opened a new pull request, #16469: [bugfix](vertical_compaction) fix base_compaction delete_sign handler
yixiutt opened a new pull request, #16469: URL: https://github.com/apache/doris/pull/16469 In vertical base compaction, same rows will be filtered in verticam_merge_iterator, we should skip these filtered rows when set agg flag of delete sign. For example, schema is a,b,delete_sign, and data is 1,1,1 1,1,0 1,1,0 2,2,1 2,2 and Block we get in VerticalBlockReader is 1,1,1 2,2,1 and we should set agg flag idex 0,4 to true when handle delete sign, so we add a function continuous_agg_count to skip same rows filtered in VerticalMergeIterator. # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16469: [bugfix](vertical_compaction) fix base_compaction delete_sign handler
github-actions[bot] commented on PR #16469: URL: https://github.com/apache/doris/pull/16469#issuecomment-1420367742 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #16371: [Feature-WIP](inverted index) Implementation for alter inverted index.
Tanya-W commented on code in PR #16371:
URL: https://github.com/apache/doris/pull/16371#discussion_r1098318886
##
be/src/olap/schema_change.cpp:
##
@@ -576,6 +583,248 @@ Status
VSchemaChangeWithSorting::_external_sorting(vector& src_
return Status::OK();
}
+SchemaChangeForInvertedIndex::SchemaChangeForInvertedIndex(
+const std::vector& alter_inverted_indexs,
+const TabletSchemaSPtr& tablet_schema)
+: SchemaChange(),
+ _alter_inverted_indexs(alter_inverted_indexs),
+ _tablet_schema(tablet_schema) {}
+
+SchemaChangeForInvertedIndex::~SchemaChangeForInvertedIndex() {
+VLOG_NOTICE << "~SchemaChangeForInvertedIndex()";
+_inverted_index_builders.clear();
+_index_metas.clear();
+}
+
+Status SchemaChangeForInvertedIndex::process(RowsetReaderSharedPtr
rowset_reader,
+ RowsetWriter* rowset_writer,
+ TabletSharedPtr new_tablet,
+ TabletSharedPtr base_tablet,
+ TabletSchemaSPtr
base_tablet_schema) {
+Status res = Status::OK();
+if (rowset_reader->rowset()->empty() ||
rowset_reader->rowset()->num_rows() == 0) {
+return Status::OK();
+}
+
+std::vector return_columns;
+for (auto& inverted_index : _alter_inverted_indexs) {
+DCHECK_EQ(inverted_index.columns.size(), 1);
+auto column_name = inverted_index.columns[0];
+auto idx = _tablet_schema->field_index(column_name);
+return_columns.emplace_back(idx);
+}
+
+// create inverted index writer
+auto rowset_meta = rowset_reader->rowset()->rowset_meta();
+std::string segment_dir = base_tablet->tablet_path();
+auto fs = rowset_meta->fs();
+for (auto i = 0; i < rowset_meta->num_segments(); ++i) {
+std::string segment_filename =
+fmt::format("{}_{}.dat", rowset_meta->rowset_id().to_string(),
i);
+for (auto& inverted_index : _alter_inverted_indexs) {
+DCHECK_EQ(inverted_index.columns.size(), 1);
+auto column_name = inverted_index.columns[0];
+auto column = _tablet_schema->column(column_name);
+auto index_id = inverted_index.index_id;
+
+std::unique_ptr field(FieldFactory::create(column));
+_index_metas.emplace_back(new TabletIndex());
+_index_metas.back()->init_from_thrift(inverted_index,
*_tablet_schema);
+std::unique_ptr
inverted_index_builder;
+try {
+RETURN_IF_ERROR(segment_v2::InvertedIndexColumnWriter::create(
+field.get(), &inverted_index_builder,
segment_filename, segment_dir,
+_index_metas.back().get(), fs));
+} catch (const std::exception& e) {
+LOG(WARNING) << "CLuceneError occured: " << e.what();
+return Status::Error();
+}
+
+if (inverted_index_builder) {
+std::string writer_sign = fmt::format("{}_{}", i, index_id);
+_inverted_index_builders.insert(
+std::make_pair(writer_sign,
std::move(inverted_index_builder)));
+}
+}
+}
+
+SegmentCacheHandle segment_cache_handle;
+// load segments
+RETURN_NOT_OK(SegmentLoader::instance()->load_segments(
+std::static_pointer_cast(rowset_reader->rowset()),
&segment_cache_handle,
+false));
+
+// create iterator for each segment
+StorageReadOptions read_options;
+OlapReaderStatistics stats;
+read_options.stats = &stats;
+read_options.tablet_schema = _tablet_schema;
+std::unique_ptr schema =
+std::make_unique(_tablet_schema->columns(),
return_columns);
+for (auto& seg_ptr : segment_cache_handle.get_segments()) {
+std::unique_ptr iter;
+res = seg_ptr->new_iterator(*schema, read_options, &iter);
+if (!res.ok()) {
+LOG(WARNING) << "failed to create iterator[" << seg_ptr->id()
+ << "]: " << res.to_string();
+return Status::Error();
+}
+
+std::shared_ptr block =
+
std::make_shared(_tablet_schema->create_block(return_columns));
+do {
+block->clear_column_data();
+res = iter->next_batch(block.get());
+if (!res.ok()) {
+if (res.is()) {
+break;
+}
+RETURN_NOT_OK_LOG(
+res, "failed to read next block when schema change for
inverted index.");
+}
+
+// copy block
+auto ref_block = *block;
+
+// write inverted index
+if (_write_inverted_index(iter->data_id(), &ref_block) !=
Status::OK()) {
+res = Status::Error();
+LOG(WARNING) << "failed to write block.";
+
[GitHub] [doris-flink-connector] DarvenDuan opened a new issue, #100: [Bug] (concurrency) Thread safety problem of loading data to Doris
DarvenDuan opened a new issue, #100: URL: https://github.com/apache/doris-flink-connector/issues/100 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version 1.1.1 ### What's Wrong? When I used flink-doris-connector_1.14_2.12(1.1.1) to loading data to Doris, Checkpoint failed due to `java.lang.InterruptedException` which was thrown by `exception checker`. But Stream load job in Doris is normal. In the source code, exception checker will invoke method `DorisStreamLoad.handlePreCommitResponse()` if variable `loading = true`, in method `DorisWriter.prepareCommit()`, variable `loading` will be set to `false` to disable exception checker, and then invokes method `DorisStreamLoad.handlePreCommitResponse()`. But method `DorisStreamLoad.handlePreCommitResponse()` may be invoked by exception checker thread before `loading` is set to `true` by main thread. So exception checker thread and main thread will invokes `DorisStreamLoad.handlePreCommitResponse()` at the same time. So this may cause concurrent access problems. ### What You Expected? Exception checker thread and main thread Invokes method `handlePreCommitResponse()` synchronously ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yixiutt opened a new pull request, #16470: [improvement](compaction) enable compaction if state is TABLET_NOTREADY
yixiutt opened a new pull request, #16470: URL: https://github.com/apache/doris/pull/16470 If alter task in queue, compaction is not enabled and may cause too much version. Keep last 10 version in new tablet so that base tablet's max version will not be merged and than we can copy data from base tablet to new tablet. # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16470: [improvement](compaction) enable compaction if state is TABLET_NOTREADY
github-actions[bot] commented on PR #16470: URL: https://github.com/apache/doris/pull/16470#issuecomment-1420407169 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caoliang-web opened a new pull request, #16471: [fix](message)return result error
caoliang-web opened a new pull request, #16471: URL: https://github.com/apache/doris/pull/16471 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16471: [fix](message)return result error
github-actions[bot] commented on PR #16471: URL: https://github.com/apache/doris/pull/16471#issuecomment-1420419774 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] liutang123 opened a new pull request, #16472: [Enhancement] Enhance Doris's fault tolerance to disk error: sense io error
liutang123 opened a new pull request, #16472: URL: https://github.com/apache/doris/pull/16472 # Proposed changes Issue Number: #16356 ## Problem summary see issue ## Checklist(Required) 1. Does it affect the original behavior: - [x] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [x] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15249: [pipeline](test) test regression
github-actions[bot] commented on PR #15249: URL: https://github.com/apache/doris/pull/15249#issuecomment-1420437552 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16472: [Enhancement] Enhance Doris's fault tolerance to disk error: sense io error
github-actions[bot] commented on code in PR #16472:
URL: https://github.com/apache/doris/pull/16472#discussion_r1098381056
##
be/src/olap/tablet.h:
##
@@ -360,6 +360,11 @@ class Tablet : public BaseTablet {
RowsetSharedPtr get_rowset(const RowsetId& rowset_id);
+inline void increase_io_error_times();
Review Comment:
warning: inline function 'doris::Tablet::increase_io_error_times' is not
defined [clang-diagnostic-undefined-inline]
```cpp
inline void increase_io_error_times();
^
```
**be/src/olap/reader.cpp:102:** used here
```cpp
_tablet->increase_io_error_times();
^
```
##
be/src/olap/tablet.cpp:
##
@@ -1374,7 +1374,10 @@ void Tablet::build_tablet_report_info(TTabletInfo*
tablet_info,
// and perform state modification operations.
}
-if (has_version_cross && tablet_state() == TABLET_RUNNING) {
+auto is_io_error_too_times = is_io_error_too_times();
+if ((has_version_cross || is_io_error_too_times) && tablet_state() ==
TABLET_RUNNING) {
Review Comment:
warning: variable 'is_io_error_too_times' declared with deduced type 'auto'
cannot appear in its own initializer [clang-diagnostic-error]
```cpp
imes = is_io_error_too_times();
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #16470: [improvement](compaction) enable compaction if state is TABLET_NOTREADY
morningman commented on code in PR #16470:
URL: https://github.com/apache/doris/pull/16470#discussion_r1098381452
##
be/src/olap/tablet.cpp:
##
@@ -826,10 +826,9 @@ bool Tablet::can_do_compaction(size_t path_hash,
CompactionType compaction_type)
}
if (tablet_state() == TABLET_NOTREADY) {
-// Before doing schema change, tablet's rowsets that versions smaller
than max converting version will be
-// removed. So, we only need to do the compaction when it is being
converted.
-// After being converted, tablet's state will be changed to
TABLET_RUNNING.
-return SchemaChangeHandler::tablet_in_converting(tablet_id());
+// In TABLET_NOTREADY, we keep last 10 versions in new tablet so basa
tablet max_version
Review Comment:
```suggestion
// In TABLET_NOTREADY, we keep last 10 versions in new tablet so
base tablet max_version
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei opened a new pull request, #16473: [bugfix](scannode) 1. make rows_read correct 2. use single scanner if has limit clause
yiguolei opened a new pull request, #16473: URL: https://github.com/apache/doris/pull/16473 # Proposed changes 1. make rows_read correct so that the scheduler could using this correctly. 3. use single scanner if has limit clause. Move it from fragment context to scannode. ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 opened a new pull request, #16474: [Bug](share hash table) Support shared hash table on Nereids
Gabriel39 opened a new pull request, #16474: URL: https://github.com/apache/doris/pull/16474 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch struct-map-type updated (98577d6ce1 -> c33740066c)
This is an automated email from the ASF dual-hosted git repository. xuyang pushed a change to branch struct-map-type in repository https://gitbox.apache.org/repos/asf/doris.git from 98577d6ce1 [bug](regression-test) fix array regression test add c33740066c [fix](struct-type) remove-empty-line No new revisions were added by this update. Summary of changes: README.md | 1 - be/src/olap/page_cache.cpp | 1 + 2 files changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16472: [Enhancement] Enhance Doris's fault tolerance to disk error
github-actions[bot] commented on PR #16472: URL: https://github.com/apache/doris/pull/16472#issuecomment-1420447227 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16470: [improvement](compaction) enable compaction if state is TABLET_NOTREADY
github-actions[bot] commented on PR #16470: URL: https://github.com/apache/doris/pull/16470#issuecomment-1420448732 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16473: [bugfix](scannode) 1. make rows_read correct 2. use single scanner if has limit clause
github-actions[bot] commented on code in PR #16473:
URL: https://github.com/apache/doris/pull/16473#discussion_r1098390322
##
be/src/vec/exec/scan/vscan_node.h:
##
@@ -94,6 +101,8 @@ class VScanNode : public ExecNode {
Status try_close();
+bool should_run_serial() { return _should_run_serial; }
Review Comment:
warning: method 'should_run_serial' can be made const
[readability-make-member-function-const]
```suggestion
bool should_run_serial() const { return _should_run_serial; }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #16371: [Feature-WIP](inverted index) Implementation for alter inverted index.
Tanya-W commented on code in PR #16371:
URL: https://github.com/apache/doris/pull/16371#discussion_r1098397282
##
be/src/olap/schema_change.cpp:
##
@@ -958,6 +1232,270 @@ Status
SchemaChangeHandler::_do_process_alter_tablet_v2(const TAlterTabletReqV2&
return res;
}
+Status SchemaChangeHandler::_do_process_alter_inverted_index(
+TabletSharedPtr tablet, const TAlterInvertedIndexReq& request) {
+Status res = Status::OK();
+// TODO(wy): check whether the tablet's max continuous version ==
request.version
+if (tablet->tablet_state() == TABLET_TOMBSTONED || tablet->tablet_state()
== TABLET_STOPPED ||
+tablet->tablet_state() == TABLET_SHUTDOWN) {
+LOG(WARNING) << "tablet's state=" << tablet->tablet_state()
+ << " cannot alter inverted index";
+return Status::Error();
+}
+
+std::shared_lock base_migration_rlock(tablet->get_migration_lock(),
std::try_to_lock);
+if (!base_migration_rlock.owns_lock()) {
+return Status::Error();
+}
+
+TabletSchemaSPtr tablet_schema = std::make_shared();
+tablet_schema->copy_from(*tablet->tablet_schema());
+if (!request.columns.empty() && request.columns[0].col_unique_id >= 0) {
+tablet_schema->clear_columns();
+for (const auto& column : request.columns) {
+tablet_schema->append_column(TabletColumn(column));
+}
+}
+
+// get rowset reader
+std::vector rs_readers;
+DeleteHandler delete_handler;
+RETURN_IF_ERROR(
+_get_rowset_readers(tablet, tablet_schema, request, &rs_readers,
&delete_handler));
+if (request.__isset.is_drop_op && request.is_drop_op) {
+// drop index
+res = _drop_inverted_index(rs_readers, tablet_schema, tablet, request);
+} else {
+// add index
+res = _add_inverted_index(rs_readers, &delete_handler, tablet_schema,
tablet, request);
+}
+
+if (!res.ok()) {
+LOG(WARNING) << "failed to alter tablet. tablet=" <<
tablet->full_name();
+return res;
+}
+
+return Status::OK();
+}
+
+Status SchemaChangeHandler::_get_rowset_readers(TabletSharedPtr tablet,
+const TabletSchemaSPtr&
tablet_schema,
+const TAlterInvertedIndexReq&
request,
+
std::vector* rs_readers,
+DeleteHandler* delete_handler)
{
+Status res = Status::OK();
+std::vector versions_to_be_changed;
+std::vector return_columns;
+std::vector alter_inverted_indexs;
+
+if (request.__isset.alter_inverted_indexes) {
+alter_inverted_indexs = request.alter_inverted_indexes;
+}
+
+for (auto& inverted_index : alter_inverted_indexs) {
+DCHECK_EQ(inverted_index.columns.size(), 1);
+auto column_name = inverted_index.columns[0];
+auto idx = tablet_schema->field_index(column_name);
+return_columns.emplace_back(idx);
+}
+
+// obtain base tablet's push lock and header write lock to prevent loading
data
+{
+std::lock_guard tablet_lock(tablet->get_push_lock());
+std::lock_guard
tablet_wlock(tablet->get_header_lock());
+
+do {
+RowsetSharedPtr max_rowset;
+// get history data to rebuild inverted index and it will check if
there is hold in base tablet
+res = _get_versions_to_be_changed(tablet, &versions_to_be_changed,
&max_rowset);
+if (!res.ok()) {
+LOG(WARNING) << "fail to get version to be rebuild inverted
index. res=" << res;
+break;
+}
+
+// should check the max_version >= request.alter_version, if not
the rebuild index is useless
+if (max_rowset == nullptr || max_rowset->end_version() <
request.alter_version) {
+LOG(WARNING) << "base tablet's max version="
+ << (max_rowset == nullptr ? 0 :
max_rowset->end_version())
+ << " is less than request version=" <<
request.alter_version;
+res = Status::InternalError(
+"base tablet's max version={} is less than request
version={}",
+(max_rowset == nullptr ? 0 :
max_rowset->end_version()),
+request.alter_version);
+break;
+}
+
+// init one delete handler
+int64_t end_version = -1;
+for (auto& version : versions_to_be_changed) {
+end_version = std::max(end_version, version.second);
+}
+
+auto& all_del_preds = tablet->delete_predicates();
+for (auto& delete_pred : all_del_preds) {
+if (delete_pred->version().first > end_version) {
+continue;
+}
+
tablet_schema->merge_dropped_c
[GitHub] [doris] github-actions[bot] commented on pull request #16472: [improvement](meta) Enhance Doris's fault tolerance to disk error
github-actions[bot] commented on PR #16472: URL: https://github.com/apache/doris/pull/16472#issuecomment-1420459453 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16469: [bugfix](vertical_compaction) fix base_compaction delete_sign handler
hello-stephen commented on PR #16469: URL: https://github.com/apache/doris/pull/16469#issuecomment-1420461582 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.51 seconds load time: 490 seconds storage size: 17171089449 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230207093049_clickbench_pr_91300.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16444: [Feature](Complex-type) Add struct and map type to Doris
github-actions[bot] commented on code in PR #16444:
URL: https://github.com/apache/doris/pull/16444#discussion_r1098406115
##
be/src/olap/types.h:
##
@@ -322,6 +324,240 @@ class ArrayTypeInfo : public TypeInfo {
TypeInfoPtr _item_type_info;
const size_t _item_size;
};
+///== MapType Info ==///
+class MapTypeInfo : public TypeInfo {
+public:
+explicit MapTypeInfo(TypeInfoPtr key_type_info, TypeInfoPtr
value_type_info)
+: _key_type_info(std::move(key_type_info)),
+ _value_type_info(std::move(value_type_info)) {}
+~MapTypeInfo() override = default;
+
+int cmp(const void* left, const void* right) const override {
+auto l_value = reinterpret_cast(left);
+auto r_value = reinterpret_cast(right);
+uint32_t l_size = l_value->size();
+uint32_t r_size = r_value->size();
+if (l_size < r_size) {
+return -1;
+} else if (l_size > r_size) {
+return 1;
+} else {
+return 0;
+}
+}
+
+void deep_copy(void* dest, const void* src, MemPool* mem_pool) const
override { DCHECK(false); }
+
+void direct_copy(void* dest, const void* src) const override {
CHECK(false); }
+
+void direct_copy(uint8_t** base, void* dest, const void* src) const {
CHECK(false); }
+
+void direct_copy_may_cut(void* dest, const void* src) const override {
direct_copy(dest, src); }
+
+Status from_string(void* buf, const std::string& scan_key, const int
precision = 0,
+ const int scale = 0) const override {
+return Status::Error();
+}
+
+std::string to_string(const void* src) const override { return "{}"; }
+
+void set_to_max(void* buf) const override {
+DCHECK(false) << "set_to_max of list is not implemented.";
+}
+
+void set_to_min(void* buf) const override {
+DCHECK(false) << "set_to_min of list is not implemented.";
+}
+
+// todo . is here only to need return 16 for two ptr?
+size_t size() const override { return sizeof(MapValue); }
+
+FieldType type() const override { return OLAP_FIELD_TYPE_MAP; }
+
+inline const TypeInfo* get_key_type_info() const { return
_key_type_info.get(); }
+inline const TypeInfo* get_value_type_info() const { return
_value_type_info.get(); }
+
+private:
+TypeInfoPtr _key_type_info;
+TypeInfoPtr _value_type_info;
+};
+
+class StructTypeInfo : public TypeInfo {
+public:
+explicit StructTypeInfo(std::vector& type_infos) {
+for (TypeInfoPtr& type_info : type_infos) {
+_type_infos.push_back(std::move(type_info));
+}
+}
+~StructTypeInfo() override = default;
+
+int cmp(const void* left, const void* right) const override {
+auto l_value = reinterpret_cast(left);
+auto r_value = reinterpret_cast(right);
+uint32_t l_size = l_value->size();
+uint32_t r_size = r_value->size();
+size_t cur = 0;
+
+if (!l_value->has_null() && !r_value->has_null()) {
+while (cur < l_size && cur < r_size) {
+int result =
+_type_infos[cur]->cmp(l_value->child_value(cur),
r_value->child_value(cur));
+if (result != 0) {
+return result;
+}
+++cur;
+}
+} else {
+while (cur < l_size && cur < r_size) {
+if (l_value->is_null_at(cur)) {
+if (!r_value->is_null_at(cur)) { // left is null & right
is not null
+return -1;
+}
+} else if (r_value->is_null_at(cur)) { // left is not null &
right is null
+return 1;
+} else { // both are not null
+int result =
_type_infos[cur]->cmp(l_value->child_value(cur),
+
r_value->child_value(cur));
+if (result != 0) {
+return result;
+}
+}
+++cur;
+}
+}
+
+if (l_size < r_size) {
+return -1;
+} else if (l_size > r_size) {
+return 1;
+} else {
+return 0;
+}
+}
+
+void deep_copy(void* dest, const void* src, MemPool* mem_pool) const
override {
+auto dest_value = reinterpret_cast(dest);
+auto src_value = reinterpret_cast(src);
+
+if (src_value->size() == 0) {
+new (dest_value) StructValue(src_value->size());
+return;
+}
+
+dest_value->set_size(src_value->size());
+dest_value->set_has_null(src_value->has_null());
+
+size_t allocate_size = src_value->size() *
sizeof(*src_value->values());
+// allocate memory for children value
+for (size_t i = 0; i < src_value->size(); ++i) {
+
[GitHub] [doris] zhangguichuan opened a new issue, #16475: [Feature] please support scala to write UDF function
zhangguichuan opened a new issue, #16475: URL: https://github.com/apache/doris/issues/16475 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description I am using doris 1.2.1, short commit is 45a8cee, currently we are trying to use scala develop UDF/UDAF in doris, and found underlying doris UDF/UDAF framework only support java, please support scala either ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] mrhhsg opened a new pull request, #16476: [fix](pipeline) incorrect result when disabling sharing hash table
mrhhsg opened a new pull request, #16476: URL: https://github.com/apache/doris/pull/16476 # Proposed changes Incorrect result of regression-test case `ssb_unique_sql_zstd_p0/sql/q3.1.sql`: with session variables: 1. enable_share_hash_table_for_broadcast_join = false 2. parallel_fragment_exec_instance_num = 2 3. enable_local_exchange = 0 4. enable_pipeline_engine = 1 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 6. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 7. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 8. Does it need to update dependencies: - [ ] Yes - [ ] No 9. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16476: [fix](pipeline) incorrect result when disabling sharing hash table
github-actions[bot] commented on PR #16476: URL: https://github.com/apache/doris/pull/16476#issuecomment-1420499427 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener opened a new pull request, #16477: [enhance](Nereids): refactor JoinReorder code.
jackwener opened a new pull request, #16477: URL: https://github.com/apache/doris/pull/16477 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow opened a new pull request, #16478: [fix](Nereids) back off on tpch p1
morrySnow opened a new pull request, #16478: URL: https://github.com/apache/doris/pull/16478 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #16474: [Bug](share hash table) Support shared hash table on Nereids
HappenLee commented on code in PR #16474:
URL: https://github.com/apache/doris/pull/16474#discussion_r1098449919
##
fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java:
##
@@ -1130,22 +1130,20 @@ private void computeFragmentExecParams() throws
Exception {
// here choose the first instance to build hash table.
Map destHosts = new
HashMap<>();
destParams.instanceExecParams.forEach(param -> {
-if (destHosts.containsKey(param.host)) {
-
destHosts.get(param.host).instancesSharingHashTable.add(param.instanceId);
-return;
-}
-destHosts.put(param.host, param);
-
-param.buildHashTableForBroadcastJoin = true;
-TPlanFragmentDestination dest = new
TPlanFragmentDestination();
-dest.fragment_instance_id = param.instanceId;
-try {
-dest.server = toRpcHost(param.host);
-dest.setBrpcServer(toBrpcHost(param.host));
-} catch (Exception e) {
-throw new RuntimeException(e);
+if (!destHosts.containsKey(param.host)) {
Review Comment:
forget call `
destHosts.get(param.host).instancesSharingHashTable.add(param.instanceId);`
##
fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java:
##
@@ -1130,22 +1130,20 @@ private void computeFragmentExecParams() throws
Exception {
// here choose the first instance to build hash table.
Map destHosts = new
HashMap<>();
destParams.instanceExecParams.forEach(param -> {
-if (destHosts.containsKey(param.host)) {
-
destHosts.get(param.host).instancesSharingHashTable.add(param.instanceId);
-return;
-}
-destHosts.put(param.host, param);
-
-param.buildHashTableForBroadcastJoin = true;
-TPlanFragmentDestination dest = new
TPlanFragmentDestination();
-dest.fragment_instance_id = param.instanceId;
-try {
-dest.server = toRpcHost(param.host);
-dest.setBrpcServer(toBrpcHost(param.host));
-} catch (Exception e) {
-throw new RuntimeException(e);
+if (!destHosts.containsKey(param.host)) {
Review Comment:
forget call `
destHosts.get(param.host).instancesSharingHashTable.add(param.instanceId);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] eldenmoon commented on a diff in pull request #16335: [WIP](dynamic-table) support dynamic schema table
eldenmoon commented on code in PR #16335:
URL: https://github.com/apache/doris/pull/16335#discussion_r1098399271
##
fe/fe-core/src/main/cup/sql_parser.cup:
##
@@ -1815,7 +1828,20 @@ create_stmt ::=
opt_ext_properties:extProperties
{:
RESULT = new CreateTableStmt(ifNotExists, isExternal, name, columns,
indexes, engineName, keys, partition,
-distribution, tblProperties, extProperties, tableComment, index);
+distribution, tblProperties, extProperties, tableComment, index,
false);
+:}
+| KW_CREATE opt_external:isExternal KW_TABLE opt_if_not_exists:ifNotExists
table_name:name
+LPAREN column_definition_list:columns COMMA
index_definition_list:indexes COMMA DOTDOTDOT RPAREN opt_engine:engineName
Review Comment:
but it's a little bit difficult to get the bool value from
column_definition_list, it's simple to just add a DOTDOTDOT token after
column_definition_list
##
gensrc/proto/types.proto:
##
@@ -104,6 +104,7 @@ message PGenericType {
FIXEDLENGTHOBJECT = 30;
JSONB = 31;
DECIMAL128I = 32;
+ VARIANT = 33;
Review Comment:
indent
##
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##
@@ -678,7 +678,7 @@ public class SessionVariable implements Serializable,
Writable {
// 1. read related rowids along with necessary column data
// 2. spawn fetch RPC to other nodes to get related data by sorted rowids
@VariableMgr.VarAttr(name = ENABLE_TWO_PHASE_READ_OPT)
-public boolean enableTwoPhaseReadOpt = true;
+public boolean enableTwoPhaseReadOpt = false;
Review Comment:
?
##
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java:
##
@@ -214,6 +228,161 @@ public TGetDbsResult getDbNames(TGetDbsParams params)
throws TException {
return result;
}
+private static ColumnDef initColumnfromThrift(TColumnDesc tColumnDesc,
String comment) {
+TypeDef typeDef = TypeDef.createTypeDef(tColumnDesc);
+boolean isAllowNull = tColumnDesc.isIsAllowNull();
+ColumnDef.DefaultValue defaultVal = ColumnDef.DefaultValue.NOT_SET;
+// Dynamic table's Array default value should be '[]'
+if (typeDef.getType().isArrayType()) {
+defaultVal = ColumnDef.DefaultValue.ARRAY_EMPTY_DEFAULT_VALUE;
+}
+return new ColumnDef(tColumnDesc.getColumnName(), typeDef, false,
null, isAllowNull, defaultVal,
+comment, true);
+}
+
+@Override
+public TAddColumnsResult addColumns(TAddColumnsRequest request) throws
TException {
+String clientAddr = getClientAddrAsString();
+LOG.debug("schema change clientAddr: {}, request: {}", clientAddr,
request);
+
+TStatus status = new TStatus(TStatusCode.OK);
+List allColumns = new ArrayList();
+
+Env env = Env.getCurrentEnv();
+InternalCatalog catalog = env.getInternalCatalog();
+int schemaVersion = 0;
+try {
+if (!env.isMaster()) {
+status.setStatusCode(TStatusCode.ILLEGAL_STATE);
+status.addToErrorMsgs("retry rpc request to master.");
Review Comment:
let the backend retry could be ok
##
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java:
##
@@ -214,6 +228,161 @@ public TGetDbsResult getDbNames(TGetDbsParams params)
throws TException {
return result;
}
+private static ColumnDef initColumnfromThrift(TColumnDesc tColumnDesc,
String comment) {
+TypeDef typeDef = TypeDef.createTypeDef(tColumnDesc);
+boolean isAllowNull = tColumnDesc.isIsAllowNull();
+ColumnDef.DefaultValue defaultVal = ColumnDef.DefaultValue.NOT_SET;
+// Dynamic table's Array default value should be '[]'
+if (typeDef.getType().isArrayType()) {
+defaultVal = ColumnDef.DefaultValue.ARRAY_EMPTY_DEFAULT_VALUE;
+}
+return new ColumnDef(tColumnDesc.getColumnName(), typeDef, false,
null, isAllowNull, defaultVal,
+comment, true);
+}
+
+@Override
+public TAddColumnsResult addColumns(TAddColumnsRequest request) throws
TException {
+String clientAddr = getClientAddrAsString();
+LOG.debug("schema change clientAddr: {}, request: {}", clientAddr,
request);
+
+TStatus status = new TStatus(TStatusCode.OK);
+List allColumns = new ArrayList();
+
+Env env = Env.getCurrentEnv();
+InternalCatalog catalog = env.getInternalCatalog();
+int schemaVersion = 0;
+try {
+if (!env.isMaster()) {
+status.setStatusCode(TStatusCode.ILLEGAL_STATE);
+status.addToErrorMsgs("retry rpc request to master.");
+TAddColumnsResult result = new TAddColumnsResult(status,
request.getTableId(), allColumns, 0);
+LOG.debug("result: {}", re
[GitHub] [doris] Tanya-W commented on a diff in pull request #16371: [Feature-WIP](inverted index) Implementation for alter inverted index.
Tanya-W commented on code in PR #16371:
URL: https://github.com/apache/doris/pull/16371#discussion_r1098490345
##
be/src/olap/schema_change.cpp:
##
@@ -958,6 +1232,270 @@ Status
SchemaChangeHandler::_do_process_alter_tablet_v2(const TAlterTabletReqV2&
return res;
}
+Status SchemaChangeHandler::_do_process_alter_inverted_index(
+TabletSharedPtr tablet, const TAlterInvertedIndexReq& request) {
+Status res = Status::OK();
+// TODO(wy): check whether the tablet's max continuous version ==
request.version
+if (tablet->tablet_state() == TABLET_TOMBSTONED || tablet->tablet_state()
== TABLET_STOPPED ||
+tablet->tablet_state() == TABLET_SHUTDOWN) {
+LOG(WARNING) << "tablet's state=" << tablet->tablet_state()
+ << " cannot alter inverted index";
+return Status::Error();
+}
+
+std::shared_lock base_migration_rlock(tablet->get_migration_lock(),
std::try_to_lock);
+if (!base_migration_rlock.owns_lock()) {
+return Status::Error();
+}
+
+TabletSchemaSPtr tablet_schema = std::make_shared();
+tablet_schema->copy_from(*tablet->tablet_schema());
+if (!request.columns.empty() && request.columns[0].col_unique_id >= 0) {
+tablet_schema->clear_columns();
+for (const auto& column : request.columns) {
+tablet_schema->append_column(TabletColumn(column));
+}
+}
+
+// get rowset reader
+std::vector rs_readers;
+DeleteHandler delete_handler;
+RETURN_IF_ERROR(
+_get_rowset_readers(tablet, tablet_schema, request, &rs_readers,
&delete_handler));
+if (request.__isset.is_drop_op && request.is_drop_op) {
+// drop index
+res = _drop_inverted_index(rs_readers, tablet_schema, tablet, request);
+} else {
+// add index
+res = _add_inverted_index(rs_readers, &delete_handler, tablet_schema,
tablet, request);
+}
+
+if (!res.ok()) {
+LOG(WARNING) << "failed to alter tablet. tablet=" <<
tablet->full_name();
+return res;
+}
+
+return Status::OK();
+}
+
+Status SchemaChangeHandler::_get_rowset_readers(TabletSharedPtr tablet,
+const TabletSchemaSPtr&
tablet_schema,
+const TAlterInvertedIndexReq&
request,
+
std::vector* rs_readers,
+DeleteHandler* delete_handler)
{
+Status res = Status::OK();
+std::vector versions_to_be_changed;
+std::vector return_columns;
+std::vector alter_inverted_indexs;
+
+if (request.__isset.alter_inverted_indexes) {
+alter_inverted_indexs = request.alter_inverted_indexes;
+}
+
+for (auto& inverted_index : alter_inverted_indexs) {
+DCHECK_EQ(inverted_index.columns.size(), 1);
+auto column_name = inverted_index.columns[0];
+auto idx = tablet_schema->field_index(column_name);
+return_columns.emplace_back(idx);
+}
+
+// obtain base tablet's push lock and header write lock to prevent loading
data
+{
+std::lock_guard tablet_lock(tablet->get_push_lock());
+std::lock_guard
tablet_wlock(tablet->get_header_lock());
+
+do {
+RowsetSharedPtr max_rowset;
+// get history data to rebuild inverted index and it will check if
there is hold in base tablet
+res = _get_versions_to_be_changed(tablet, &versions_to_be_changed,
&max_rowset);
+if (!res.ok()) {
+LOG(WARNING) << "fail to get version to be rebuild inverted
index. res=" << res;
+break;
+}
+
+// should check the max_version >= request.alter_version, if not
the rebuild index is useless
+if (max_rowset == nullptr || max_rowset->end_version() <
request.alter_version) {
+LOG(WARNING) << "base tablet's max version="
+ << (max_rowset == nullptr ? 0 :
max_rowset->end_version())
+ << " is less than request version=" <<
request.alter_version;
+res = Status::InternalError(
+"base tablet's max version={} is less than request
version={}",
+(max_rowset == nullptr ? 0 :
max_rowset->end_version()),
+request.alter_version);
+break;
+}
+
+// init one delete handler
+int64_t end_version = -1;
+for (auto& version : versions_to_be_changed) {
+end_version = std::max(end_version, version.second);
+}
+
+auto& all_del_preds = tablet->delete_predicates();
+for (auto& delete_pred : all_del_preds) {
+if (delete_pred->version().first > end_version) {
+continue;
+}
+
tablet_schema->merge_dropped_c
[GitHub] [doris-spark-connector] smallhibiscus opened a new pull request, #68: [improvement](spark-connector)Support each batch interval time paramter setting.
smallhibiscus opened a new pull request, #68: URL: https://github.com/apache/doris-spark-connector/pull/68 # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju commented on a diff in pull request #16335: [WIP](dynamic-table) support dynamic schema table
cambyzju commented on code in PR #16335:
URL: https://github.com/apache/doris/pull/16335#discussion_r1098500173
##
fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java:
##
@@ -271,8 +273,10 @@ public void analyze(boolean isOlap) throws
AnalysisException {
throw new AnalysisException("Array can only be used in the
non-key column of"
+ " the duplicate table at present.");
}
-if (defaultValue.isSet && defaultValue !=
DefaultValue.NULL_DEFAULT_VALUE) {
-throw new AnalysisException("Array type column default value
only support null");
+if (defaultValue.isSet && defaultValue !=
DefaultValue.NULL_DEFAULT_VALUE
Review Comment:
it's ok for array type. null for NULL(Array), [] for NOT NULL(Array).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #16371: [Feature-WIP](inverted index) Implementation for alter inverted index.
Tanya-W commented on code in PR #16371:
URL: https://github.com/apache/doris/pull/16371#discussion_r1098508480
##
be/src/olap/schema_change.cpp:
##
@@ -576,6 +583,248 @@ Status
VSchemaChangeWithSorting::_external_sorting(vector& src_
return Status::OK();
}
+SchemaChangeForInvertedIndex::SchemaChangeForInvertedIndex(
+const std::vector& alter_inverted_indexs,
+const TabletSchemaSPtr& tablet_schema)
+: SchemaChange(),
+ _alter_inverted_indexs(alter_inverted_indexs),
+ _tablet_schema(tablet_schema) {}
+
+SchemaChangeForInvertedIndex::~SchemaChangeForInvertedIndex() {
+VLOG_NOTICE << "~SchemaChangeForInvertedIndex()";
+_inverted_index_builders.clear();
+_index_metas.clear();
+}
+
+Status SchemaChangeForInvertedIndex::process(RowsetReaderSharedPtr
rowset_reader,
+ RowsetWriter* rowset_writer,
+ TabletSharedPtr new_tablet,
+ TabletSharedPtr base_tablet,
+ TabletSchemaSPtr
base_tablet_schema) {
+Status res = Status::OK();
+if (rowset_reader->rowset()->empty() ||
rowset_reader->rowset()->num_rows() == 0) {
+return Status::OK();
+}
+
+std::vector return_columns;
+for (auto& inverted_index : _alter_inverted_indexs) {
+DCHECK_EQ(inverted_index.columns.size(), 1);
+auto column_name = inverted_index.columns[0];
+auto idx = _tablet_schema->field_index(column_name);
+return_columns.emplace_back(idx);
+}
+
+// create inverted index writer
+auto rowset_meta = rowset_reader->rowset()->rowset_meta();
+std::string segment_dir = base_tablet->tablet_path();
+auto fs = rowset_meta->fs();
+for (auto i = 0; i < rowset_meta->num_segments(); ++i) {
+std::string segment_filename =
+fmt::format("{}_{}.dat", rowset_meta->rowset_id().to_string(),
i);
+for (auto& inverted_index : _alter_inverted_indexs) {
+DCHECK_EQ(inverted_index.columns.size(), 1);
+auto column_name = inverted_index.columns[0];
+auto column = _tablet_schema->column(column_name);
+auto index_id = inverted_index.index_id;
+
+std::unique_ptr field(FieldFactory::create(column));
+_index_metas.emplace_back(new TabletIndex());
+_index_metas.back()->init_from_thrift(inverted_index,
*_tablet_schema);
+std::unique_ptr
inverted_index_builder;
+try {
+RETURN_IF_ERROR(segment_v2::InvertedIndexColumnWriter::create(
+field.get(), &inverted_index_builder,
segment_filename, segment_dir,
+_index_metas.back().get(), fs));
+} catch (const std::exception& e) {
+LOG(WARNING) << "CLuceneError occured: " << e.what();
+return Status::Error();
+}
+
+if (inverted_index_builder) {
+std::string writer_sign = fmt::format("{}_{}", i, index_id);
Review Comment:
you are right, use pair as key is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16455: [function](round) Use correct decimal scale for function round
github-actions[bot] commented on PR #16455: URL: https://github.com/apache/doris/pull/16455#issuecomment-1420594764 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16371: [Feature-WIP](inverted index) Implementation for alter inverted index.
github-actions[bot] commented on code in PR #16371:
URL: https://github.com/apache/doris/pull/16371#discussion_r1098514695
##
be/src/olap/schema_change.cpp:
##
@@ -576,6 +582,236 @@ Status
VSchemaChangeWithSorting::_external_sorting(vector& src_
return Status::OK();
}
+SchemaChangeForInvertedIndex::SchemaChangeForInvertedIndex(
+const std::vector& alter_inverted_indexs,
+const TabletSchemaSPtr& tablet_schema)
+: SchemaChange(),
+ _alter_inverted_indexs(alter_inverted_indexs),
+ _tablet_schema(tablet_schema) {
+_olap_data_convertor =
std::make_unique();
+}
+
+SchemaChangeForInvertedIndex::~SchemaChangeForInvertedIndex() {
+VLOG_NOTICE << "~SchemaChangeForInvertedIndex()";
+_inverted_index_builders.clear();
+_index_metas.clear();
+_olap_data_convertor.reset();
+}
+
+Status SchemaChangeForInvertedIndex::process(RowsetReaderSharedPtr
rowset_reader,
+ RowsetWriter* rowset_writer,
+ TabletSharedPtr new_tablet,
+ TabletSharedPtr base_tablet,
+ TabletSchemaSPtr
base_tablet_schema) {
+Status res = Status::OK();
+if (rowset_reader->rowset()->empty() ||
rowset_reader->rowset()->num_rows() == 0) {
+return Status::OK();
+}
+
+std::vector return_columns;
+// create inverted index writer
+auto rowset_meta = rowset_reader->rowset()->rowset_meta();
+std::string segment_dir = base_tablet->tablet_path();
+auto fs = rowset_meta->fs();
+_olap_data_convertor->reserve(_alter_inverted_indexs.size());
+for (auto i = 0; i < rowset_meta->num_segments(); ++i) {
+std::string segment_filename =
+fmt::format("{}_{}.dat", rowset_meta->rowset_id().to_string(),
i);
+for (auto& inverted_index : _alter_inverted_indexs) {
+DCHECK_EQ(inverted_index.columns.size(), 1);
+auto index_id = inverted_index.index_id;
+auto column_name = inverted_index.columns[0];
+auto column = _tablet_schema->column(column_name);
+auto column_idx = _tablet_schema->field_index(column_name);
+return_columns.emplace_back(column_idx);
+_olap_data_convertor->add_column_data_convertor(column);
+
+std::unique_ptr field(FieldFactory::create(column));
+_index_metas.emplace_back(new TabletIndex());
+_index_metas.back()->init_from_thrift(inverted_index,
*_tablet_schema);
+std::unique_ptr
inverted_index_builder;
+try {
+RETURN_IF_ERROR(segment_v2::InvertedIndexColumnWriter::create(
+field.get(), &inverted_index_builder,
segment_filename, segment_dir,
+_index_metas.back().get(), fs));
+} catch (const std::exception& e) {
+LOG(WARNING) << "CLuceneError occured: " << e.what();
+return Status::Error();
+}
+
+if (inverted_index_builder) {
+auto writer_sign = std::make_pair(i, index_id);
+_inverted_index_builders.insert(
+std::make_pair(writer_sign,
std::move(inverted_index_builder)));
+}
+}
+}
+
+// load segments
+SegmentCacheHandle segment_cache_handle;
+RETURN_NOT_OK(SegmentLoader::instance()->load_segments(
+std::static_pointer_cast(rowset_reader->rowset()),
&segment_cache_handle,
+false));
+
+// create iterator for each segment
+StorageReadOptions read_options;
+OlapReaderStatistics stats;
+read_options.stats = &stats;
+read_options.tablet_schema = _tablet_schema;
+std::unique_ptr schema =
+std::make_unique(_tablet_schema->columns(),
return_columns);
+for (auto& seg_ptr : segment_cache_handle.get_segments()) {
+std::unique_ptr iter;
+res = seg_ptr->new_iterator(*schema, read_options, &iter);
+if (!res.ok()) {
+LOG(WARNING) << "failed to create iterator[" << seg_ptr->id()
+ << "]: " << res.to_string();
+return Status::Error();
+}
+
+std::shared_ptr block =
+
std::make_shared(_tablet_schema->create_block(return_columns));
+do {
+res = iter->next_batch(block.get());
+if (!res.ok()) {
+if (res.is()) {
+break;
+}
+RETURN_NOT_OK_LOG(
+res, "failed to read next block when schema change for
inverted index.");
+}
+
+// write inverted index
+if (_write_inverted_index(iter->data_id(), block.get()) !=
Status::OK()) {
+res = Status::Error();
+LOG(WARNING) << "failed to write block.";
+return res;
+
[GitHub] [doris] Tanya-W commented on a diff in pull request #16371: [Feature-WIP](inverted index) Implementation for alter inverted index.
Tanya-W commented on code in PR #16371:
URL: https://github.com/apache/doris/pull/16371#discussion_r1098516172
##
be/src/olap/schema_change.cpp:
##
@@ -958,6 +1232,270 @@ Status
SchemaChangeHandler::_do_process_alter_tablet_v2(const TAlterTabletReqV2&
return res;
}
+Status SchemaChangeHandler::_do_process_alter_inverted_index(
+TabletSharedPtr tablet, const TAlterInvertedIndexReq& request) {
+Status res = Status::OK();
+// TODO(wy): check whether the tablet's max continuous version ==
request.version
+if (tablet->tablet_state() == TABLET_TOMBSTONED || tablet->tablet_state()
== TABLET_STOPPED ||
+tablet->tablet_state() == TABLET_SHUTDOWN) {
+LOG(WARNING) << "tablet's state=" << tablet->tablet_state()
+ << " cannot alter inverted index";
+return Status::Error();
+}
+
+std::shared_lock base_migration_rlock(tablet->get_migration_lock(),
std::try_to_lock);
+if (!base_migration_rlock.owns_lock()) {
+return Status::Error();
+}
+
+TabletSchemaSPtr tablet_schema = std::make_shared();
+tablet_schema->copy_from(*tablet->tablet_schema());
+if (!request.columns.empty() && request.columns[0].col_unique_id >= 0) {
+tablet_schema->clear_columns();
+for (const auto& column : request.columns) {
+tablet_schema->append_column(TabletColumn(column));
+}
+}
+
+// get rowset reader
+std::vector rs_readers;
+DeleteHandler delete_handler;
+RETURN_IF_ERROR(
+_get_rowset_readers(tablet, tablet_schema, request, &rs_readers,
&delete_handler));
+if (request.__isset.is_drop_op && request.is_drop_op) {
+// drop index
+res = _drop_inverted_index(rs_readers, tablet_schema, tablet, request);
+} else {
+// add index
+res = _add_inverted_index(rs_readers, &delete_handler, tablet_schema,
tablet, request);
+}
+
+if (!res.ok()) {
+LOG(WARNING) << "failed to alter tablet. tablet=" <<
tablet->full_name();
+return res;
+}
+
+return Status::OK();
+}
+
+Status SchemaChangeHandler::_get_rowset_readers(TabletSharedPtr tablet,
+const TabletSchemaSPtr&
tablet_schema,
+const TAlterInvertedIndexReq&
request,
+
std::vector* rs_readers,
+DeleteHandler* delete_handler)
{
+Status res = Status::OK();
+std::vector versions_to_be_changed;
+std::vector return_columns;
+std::vector alter_inverted_indexs;
+
+if (request.__isset.alter_inverted_indexes) {
+alter_inverted_indexs = request.alter_inverted_indexes;
+}
+
+for (auto& inverted_index : alter_inverted_indexs) {
+DCHECK_EQ(inverted_index.columns.size(), 1);
+auto column_name = inverted_index.columns[0];
+auto idx = tablet_schema->field_index(column_name);
+return_columns.emplace_back(idx);
+}
+
+// obtain base tablet's push lock and header write lock to prevent loading
data
+{
+std::lock_guard tablet_lock(tablet->get_push_lock());
+std::lock_guard
tablet_wlock(tablet->get_header_lock());
+
+do {
Review Comment:
The main function used here is `capture_rs_readers`, other logic is to
prepare for rs_reader's init param which maybe set differently according to
different use scenarios.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16258: [feature](cooldown)Add cooldown delete
github-actions[bot] commented on PR #16258: URL: https://github.com/apache/doris/pull/16258#issuecomment-1420598431 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16371: [Feature-WIP](inverted index) Implementation for alter inverted index.
github-actions[bot] commented on code in PR #16371:
URL: https://github.com/apache/doris/pull/16371#discussion_r1098518535
##
be/src/olap/schema_change.cpp:
##
@@ -576,6 +582,236 @@ Status
VSchemaChangeWithSorting::_external_sorting(vector& src_
return Status::OK();
}
+SchemaChangeForInvertedIndex::SchemaChangeForInvertedIndex(
+const std::vector& alter_inverted_indexs,
+const TabletSchemaSPtr& tablet_schema)
+: SchemaChange(),
+ _alter_inverted_indexs(alter_inverted_indexs),
+ _tablet_schema(tablet_schema) {
+_olap_data_convertor =
std::make_unique();
+}
+
+SchemaChangeForInvertedIndex::~SchemaChangeForInvertedIndex() {
+VLOG_NOTICE << "~SchemaChangeForInvertedIndex()";
+_inverted_index_builders.clear();
+_index_metas.clear();
+_olap_data_convertor.reset();
+}
+
+Status SchemaChangeForInvertedIndex::process(RowsetReaderSharedPtr
rowset_reader,
+ RowsetWriter* rowset_writer,
+ TabletSharedPtr new_tablet,
+ TabletSharedPtr base_tablet,
+ TabletSchemaSPtr
base_tablet_schema) {
+Status res = Status::OK();
+if (rowset_reader->rowset()->empty() ||
rowset_reader->rowset()->num_rows() == 0) {
+return Status::OK();
+}
+
+std::vector return_columns;
+// create inverted index writer
+auto rowset_meta = rowset_reader->rowset()->rowset_meta();
+std::string segment_dir = base_tablet->tablet_path();
+auto fs = rowset_meta->fs();
+_olap_data_convertor->reserve(_alter_inverted_indexs.size());
+for (auto i = 0; i < rowset_meta->num_segments(); ++i) {
+std::string segment_filename =
+fmt::format("{}_{}.dat", rowset_meta->rowset_id().to_string(),
i);
+for (auto& inverted_index : _alter_inverted_indexs) {
+DCHECK_EQ(inverted_index.columns.size(), 1);
+auto index_id = inverted_index.index_id;
+auto column_name = inverted_index.columns[0];
+auto column = _tablet_schema->column(column_name);
+auto column_idx = _tablet_schema->field_index(column_name);
+return_columns.emplace_back(column_idx);
+_olap_data_convertor->add_column_data_convertor(column);
+
+std::unique_ptr field(FieldFactory::create(column));
+_index_metas.emplace_back(new TabletIndex());
+_index_metas.back()->init_from_thrift(inverted_index,
*_tablet_schema);
+std::unique_ptr
inverted_index_builder;
+try {
+RETURN_IF_ERROR(segment_v2::InvertedIndexColumnWriter::create(
+field.get(), &inverted_index_builder,
segment_filename, segment_dir,
+_index_metas.back().get(), fs));
+} catch (const std::exception& e) {
+LOG(WARNING) << "CLuceneError occured: " << e.what();
+return Status::Error();
+}
+
+if (inverted_index_builder) {
+auto writer_sign = std::make_pair(i, index_id);
+_inverted_index_builders.insert(
+std::make_pair(writer_sign,
std::move(inverted_index_builder)));
+}
+}
+}
+
+// load segments
+SegmentCacheHandle segment_cache_handle;
+RETURN_NOT_OK(SegmentLoader::instance()->load_segments(
+std::static_pointer_cast(rowset_reader->rowset()),
&segment_cache_handle,
+false));
+
+// create iterator for each segment
+StorageReadOptions read_options;
+OlapReaderStatistics stats;
+read_options.stats = &stats;
+read_options.tablet_schema = _tablet_schema;
+std::unique_ptr schema =
+std::make_unique(_tablet_schema->columns(),
return_columns);
+for (auto& seg_ptr : segment_cache_handle.get_segments()) {
+std::unique_ptr iter;
+res = seg_ptr->new_iterator(*schema, read_options, &iter);
+if (!res.ok()) {
+LOG(WARNING) << "failed to create iterator[" << seg_ptr->id()
+ << "]: " << res.to_string();
+return Status::Error();
+}
+
+std::shared_ptr block =
+
std::make_shared(_tablet_schema->create_block(return_columns));
+do {
+res = iter->next_batch(block.get());
+if (!res.ok()) {
+if (res.is()) {
+break;
+}
+RETURN_NOT_OK_LOG(
+res, "failed to read next block when schema change for
inverted index.");
+}
+
+// write inverted index
+if (_write_inverted_index(iter->data_id(), block.get()) !=
Status::OK()) {
+res = Status::Error();
+LOG(WARNING) << "failed to write block.";
+return res;
+
[GitHub] [doris] eldenmoon commented on a diff in pull request #16335: [WIP](dynamic-table) support dynamic schema table
eldenmoon commented on code in PR #16335:
URL: https://github.com/apache/doris/pull/16335#discussion_r1093097861
##
gensrc/thrift/Types.thrift:
##
@@ -92,13 +92,16 @@ enum TPrimitiveType {
DECIMAL128I,
JSONB,
UNSUPPORTED
+ DECIMAL128,
+ VARIANT,
}
enum TTypeNodeType {
SCALAR,
ARRAY,
MAP,
-STRUCT
+STRUCT,
+VARIANT = 100,
Review Comment:
strange
##
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##
@@ -678,7 +678,7 @@ public class SessionVariable implements Serializable,
Writable {
// 1. read related rowids along with necessary column data
// 2. spawn fetch RPC to other nodes to get related data by sorted rowids
@VariableMgr.VarAttr(name = ENABLE_TWO_PHASE_READ_OPT)
-public boolean enableTwoPhaseReadOpt = true;
+public boolean enableTwoPhaseReadOpt = false;
Review Comment:
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15249: [pipeline](test) test regression
github-actions[bot] commented on PR #15249: URL: https://github.com/apache/doris/pull/15249#issuecomment-1420606598 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 opened a new pull request, #16479: [fix](fe)group_concat lost order by info in second phase merge agg
starocean999 opened a new pull request, #16479: URL: https://github.com/apache/doris/pull/16479 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zenoyang opened a new pull request, #16480: [fix](partition cache) fix can not hit partition cache
zenoyang opened a new pull request, #16480:
URL: https://github.com/apache/doris/pull/16480
# Proposed changes
Issue Number: close #xxx
When the number of selected partition queried is greater than 1 and there is
a group by statement, the PartitionCache will not be hit.
Even the examples provided on the official website and `PartitionCacheTest`
UT cannot be hit, and PartitionCache is basically unavailable.
## Problem summary
Reproduce:
```sql
create table appevent (
`eventdate` date NULL,
`bu_name` varchar(50) NULL,
`client_type` varchar(50) NULL,
`userid` varchar(50) NULL
) ENGINE=OLAP
DUPLICATE KEY(`eventdate`)
PARTITION BY RANGE(`eventdate`)
(PARTITION p20230101 VALUES [('2023-01-01'), ('2023-01-02')),
PARTITION p20230102 VALUES [('2023-01-02'), ('2023-01-03')),
PARTITION p20230103 VALUES [('2023-01-03'), ('2023-01-04')),
PARTITION p20230104 VALUES [('2023-01-04'), ('2023-01-05')),
PARTITION p20230105 VALUES [('2023-01-05'), ('2023-01-06')))
DISTRIBUTED BY HASH(`bu_name`) BUCKETS 3
PROPERTIES (
"replication_allocation" = "tag.location.default: 3",
"in_memory" = "false",
"storage_format" = "V2"
)
insert into appevent values('2023-01-01', 'bu1', 'ios', 'user01');
insert into appevent values('2023-01-02', 'bu1', 'ios', 'user01');
insert into appevent values('2023-01-03', 'bu1', 'ios', 'user01');
insert into appevent values('2023-01-04', 'bu1', 'ios', 'user01');
insert into appevent values('2023-01-05', 'bu1', 'ios', 'user01');
SELECT
eventdate,count(userid)
FROM
testdb.appevent
WHERE
eventdate>="2023-01-01" AND eventdate<="2023-01-05"
GROUP BY
eventdate
ORDER BY
eventdate;
```
The following logic is triggered when querying:
```java
if (enablePartitionCache() && ((OlapScanNode)
node).getSelectedPartitionNum() > 1
&& selectStmt.hasGroupByClause()) {
LOG.debug("more than one partition scanned when qeury has agg, partition
cache cannot use, queryid {}",
DebugUtil.printId(queryId));
return CacheMode.None;
}
```
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16123: [enhancement-wip](BE http)Support BE http service with brpc
github-actions[bot] commented on PR #16123: URL: https://github.com/apache/doris/pull/16123#issuecomment-1420625985 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16123: [enhancement-wip](BE http)Support BE http service with brpc
github-actions[bot] commented on PR #16123: URL: https://github.com/apache/doris/pull/16123#issuecomment-1420631491 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 merged pull request #16460: [Bug](sort) disable 2phase read for sort by expressions exclude slotref
Gabriel39 merged PR #16460: URL: https://github.com/apache/doris/pull/16460 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Bug](sort) disable 2phase read for sort by expressions exclude slotref (#16460)
This is an automated email from the ASF dual-hosted git repository.
gabriellee pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 1d0fdff98a [Bug](sort) disable 2phase read for sort by expressions
exclude slotref (#16460)
1d0fdff98a is described below
commit 1d0fdff98a2a31e113059ae98b056b5add036cd5
Author: lihangyu <15605149...@163.com>
AuthorDate: Tue Feb 7 19:42:54 2023 +0800
[Bug](sort) disable 2phase read for sort by expressions exclude slotref
(#16460)
```
create table tbl1 (k1 varchar(100), k2 string) distributed by hash(k1)
buckets 1 properties("replication_num" = "1");
insert into tbl1 values(1, "alice");
select cast(k1 as INT) as id from tbl1 order by id limit 2;
```
The above query could pass `checkEnableTwoPhaseRead` since the order by
element is SlotRef but actually it's an function call expr
---
.../java/org/apache/doris/analysis/SelectStmt.java | 18 ++--
.../org/apache/doris/planner/OriginalPlanner.java | 4 +--
regression-test/data/query_p0/sort/sort.out| 32 ++
regression-test/suites/query_p0/sort/sort.groovy | 13 +
4 files changed, 63 insertions(+), 4 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
index 034717f886..bfb6286b43 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
@@ -128,6 +128,8 @@ public class SelectStmt extends QueryStmt {
// For quick get condition for point query
private Map eqPredicates;
+boolean isTwoPhaseOptEnabled = false;
+
public SelectStmt(ValueList valueList, ArrayList
orderByElement, LimitElement limitElement) {
super(orderByElement, limitElement);
this.valueList = valueList;
@@ -603,6 +605,7 @@ public class SelectStmt extends QueryStmt {
// rest of resultExprs will be marked as `INVALID`, such columns
will
// be prevent from reading from ScanNode.Those columns will be
finally
// read by the second fetch phase
+isTwoPhaseOptEnabled = true;
LOG.debug("two phase read optimize enabled");
// Expr.analyze(resultExprs, analyzer);
Set resultSlots = Sets.newHashSet();
@@ -620,10 +623,12 @@ public class SelectStmt extends QueryStmt {
// invalid slots will be pruned from reading from ScanNode
slot.setInvalid();
}
+
LOG.debug("resultsSlots {}", resultSlots);
LOG.debug("orderingSlots {}", orderingSlots);
LOG.debug("conjuntSlots {}", conjuntSlots);
}
+checkAndSetPointQuery();
if (evaluateOrderBy) {
createSortTupleInfo(analyzer);
}
@@ -648,6 +653,10 @@ public class SelectStmt extends QueryStmt {
}
}
+public boolean isTwoPhaseReadOptEnabled() {
+return isTwoPhaseOptEnabled;
+}
+
// Check whether enable two phase read optimize, if enabled query will be
devieded into two phase read:
// 1. read conjuncts columns and order by columns along with an extra
RowId column from ScanNode
// 2. sort and filter data, and get final RowId column, spawn RPC to other
BE to fetch final data
@@ -707,11 +716,12 @@ public class SelectStmt extends QueryStmt {
// Rethink? implement more generic to support all exprs
LOG.debug("getOrderingExprs {}", sortInfo.getOrderingExprs());
LOG.debug("getOrderByElements {}", getOrderByElements());
-for (OrderByElement orderby : getOrderByElements()) {
-if (!(orderby.getExpr() instanceof SlotRef)) {
+for (Expr sortExpr : sortInfo.getOrderingExprs()) {
+if (!(sortExpr instanceof SlotRef)) {
return false;
}
}
+isTwoPhaseOptEnabled = true;
return true;
}
@@ -2376,6 +2386,10 @@ public class SelectStmt extends QueryStmt {
return eqPredicates;
}
+public boolean isPointQueryShortCircuit() {
+return isPointQuery;
+}
+
// Check if it is a point query and set EQUAL predicates
public boolean checkAndSetPointQuery() {
if (isPointQuery) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/OriginalPlanner.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/OriginalPlanner.java
index 21d5b18dbe..5f0abd125d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/OriginalPlanner.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/OriginalPlanner.java
@@ -263,7 +263,7 @@ public class OriginalPlanner extends Planner {
LOG.debug("this isn't block query");
}
// Check SelectStatement
[GitHub] [doris] yuxuan-luo opened a new issue, #16481: [Bug] 开启向量化后每次执行查询时 be 都会挂
yuxuan-luo opened a new issue, #16481: URL: https://github.com/apache/doris/issues/16481 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 1.1.5，1.2.1 ### What's Wrong? 1.1.5: 在开启向量化后，查询表后会出错，并且必定会有1个 be 会挂掉 关闭向量化后，查询能正常 1.2.1 是否开启向量化，都显示错误结果，查询结果为 0 ### What You Expected? 可以实现正常查询 ### How to Reproduce? CREATE TABLE `A` ( `user_id` int NULL ) ENGINE=OLAP DUPLICATE KEY(`user_id`) DISTRIBUTED BY HASH(`user_id`) BUCKETS 3 PROPERTIES ( "replication_allocation" = "tag.location.default: 3", "in_memory" = "false", "storage_format" = "V2" ); CREATE TABLE `B` ( `user_id` int NULL ) ENGINE=OLAP DUPLICATE KEY(`user_id`) DISTRIBUTED BY HASH(`user_id`) BUCKETS 3 PROPERTIES ( "replication_allocation" = "tag.location.default: 3", "in_memory" = "false", "storage_format" = "V2" ); CREATE TABLE `C` ( `user_id` int NULL ) ENGINE=OLAP DUPLICATE KEY(`user_id`) DISTRIBUTED BY HASH(`user_id`) BUCKETS 3 PROPERTIES ( "replication_allocation" = "tag.location.default: 3", "in_memory" = "false", "storage_format" = "V2" ); CREATE TABLE `D` ( `user_id` int NULL ) ENGINE=OLAP DUPLICATE KEY(`user_id`) DISTRIBUTED BY HASH(`user_id`) BUCKETS 3 PROPERTIES ( "replication_allocation" = "tag.location.default: 3", "in_memory" = "false", "storage_format" = "V2" ); WITH `W1` AS ( SELECT t.user_id FROM A t JOIN B u ON u.user_id = t.user_id ), `W2` AS ( SELECT t.user_id FROM `C` t JOIN B u ON u.user_id = t.user_id GROUP BY t.user_id ), `W3` AS ( SELECT t.user_id FROM D t JOIN B u ON u.user_id = t.user_id GROUP BY t.user_id ) SELECT COUNT(`dataset_30`.`reg_user_cnt`) AS `u_18efaea722a8747c_2` FROM ( SELECT t1.reg_user_cnt, t2.commit_case_user_cnt, t3.watch_live_course_and_commit_case_user_cnt FROM ( SELECT COUNT(user_id) AS reg_user_cnt, '1' AS flag FROM W1 ) t1 JOIN ( SELECT COUNT(user_id) AS commit_case_user_cnt, '1' AS flag FROM W2 ) t2 ON t2.flag = t1.flag JOIN ( SELECT COUNT(t1.user_id) AS watch_live_course_and_commit_case_user_cnt, '1' AS flag FROM W2 t1 JOIN W3 t2 ON t2.user_id = t1.user_id ) t3 ON t3.flag = t1.flag ) `dataset_30` LIMIT 1000; ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16470: [improvement](compaction) enable compaction if state is TABLET_NOTREADY
hello-stephen commented on PR #16470: URL: https://github.com/apache/doris/pull/16470#issuecomment-1420647515 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.37 seconds load time: 480 seconds storage size: 17170680920 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230207115049_clickbench_pr_91368.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16459: [fix](planner)Fix colocate query failed
github-actions[bot] commented on PR #16459: URL: https://github.com/apache/doris/pull/16459#issuecomment-1420651593 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16459: [fix](planner)Fix colocate query failed
github-actions[bot] commented on PR #16459: URL: https://github.com/apache/doris/pull/16459#issuecomment-1420651649 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #16470: [improvement](compaction) enable compaction if state is TABLET_NOTREADY
morningman merged PR #16470: URL: https://github.com/apache/doris/pull/16470 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improvement](compaction) enable compaction in TABLET_NOTREADY (#16470)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f90d844a53 [improvement](compaction) enable compaction in
TABLET_NOTREADY (#16470)
f90d844a53 is described below
commit f90d844a530bb384c6a31e60c62647b3a4e7fe75
Author: yixiutt <102007456+yixi...@users.noreply.github.com>
AuthorDate: Tue Feb 7 19:58:23 2023 +0800
[improvement](compaction) enable compaction in TABLET_NOTREADY (#16470)
If alter task in queue, compaction is not enabled and may cause too much
version.
Keep last 10 version in new tablet so that base tablet's max version will
not be merged and than we can copy data from base tablet to new tablet.
---
be/src/olap/cumulative_compaction_policy.cpp | 8
be/src/olap/tablet.cpp | 7 +++
2 files changed, 11 insertions(+), 4 deletions(-)
diff --git a/be/src/olap/cumulative_compaction_policy.cpp
b/be/src/olap/cumulative_compaction_policy.cpp
index 76775485b5..c2735def03 100644
--- a/be/src/olap/cumulative_compaction_policy.cpp
+++ b/be/src/olap/cumulative_compaction_policy.cpp
@@ -232,6 +232,7 @@ int SizeBasedCumulativeCompactionPolicy::pick_input_rowsets(
std::vector* input_rowsets, Version*
last_delete_version,
size_t* compaction_score) {
size_t promotion_size = tablet->cumulative_promotion_size();
+auto max_version = tablet->max_version().first;
int transient_size = 0;
*compaction_score = 0;
int64_t total_size = 0;
@@ -251,6 +252,13 @@ int
SizeBasedCumulativeCompactionPolicy::pick_input_rowsets(
continue;
}
}
+if (tablet->tablet_state() == TABLET_NOTREADY) {
+// If tablet under alter, keep latest 10 version so that base
tablet max version
+// not merged in new tablet, and then we can copy data from base
tablet
+if (rowset->version().second < max_version - 10) {
+continue;
+}
+}
if (*compaction_score >= max_compaction_score) {
// got enough segments
break;
diff --git a/be/src/olap/tablet.cpp b/be/src/olap/tablet.cpp
index 3cb4079159..009513748f 100644
--- a/be/src/olap/tablet.cpp
+++ b/be/src/olap/tablet.cpp
@@ -828,10 +828,9 @@ bool Tablet::can_do_compaction(size_t path_hash,
CompactionType compaction_type)
}
if (tablet_state() == TABLET_NOTREADY) {
-// Before doing schema change, tablet's rowsets that versions smaller
than max converting version will be
-// removed. So, we only need to do the compaction when it is being
converted.
-// After being converted, tablet's state will be changed to
TABLET_RUNNING.
-return SchemaChangeHandler::tablet_in_converting(tablet_id());
+// In TABLET_NOTREADY, we keep last 10 versions in new tablet so base
tablet max_version
+// not merged in new tablet and then we can do compaction
+return true;
}
return true;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #16459: [fix](planner)Fix colocate query failed
morrySnow commented on code in PR #16459:
URL: https://github.com/apache/doris/pull/16459#discussion_r1098567216
##
fe/fe-core/src/main/java/org/apache/doris/planner/DistributedPlanner.java:
##
@@ -557,6 +557,12 @@ private boolean
dataDistributionMatchEqPredicate(List eqJoinPre
cannotReason.add(DistributedPlanColocateRule.COLOCATE_GROUP_IS_NOT_STABLE);
return false;
}
+} else {
+// query same table but hit different rollup, colocate join should
be disable
+if (leftRoot.getSelectedIndexId() !=
rightRoot.getSelectedIndexId()) {
+
cannotReason.add(DistributedPlanColocateRule.QUERY_SAME_TABLE_BUT_HIT_DIFF_ROLLUP);
+return false;
Review Comment:
if table is a colocate table, it could do colocate join with itself when
group is stable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on a diff in pull request #16371: [Feature-WIP](inverted index) Implementation for alter inverted index.
Tanya-W commented on code in PR #16371:
URL: https://github.com/apache/doris/pull/16371#discussion_r1098574523
##
fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java:
##
@@ -499,6 +590,13 @@ protected void runRunningJob() throws AlterCancelException
{
LOG.info("schema change tasks not finished. job: {}", jobId);
List tasks =
schemaChangeBatchTask.getUnfinishedTasks(2000);
for (AgentTask task : tasks) {
+if (invertedIndexChange) {
+// TODO(wy): more elegant implementation
Review Comment:
yes, need to do, but not in this pr, "more elegant implementation" need
implement specify partition to rebuild index data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16258: [feature](cooldown)Add cooldown delete
github-actions[bot] commented on code in PR #16258:
URL: https://github.com/apache/doris/pull/16258#discussion_r1098575376
##
be/src/agent/task_worker_pool.cpp:
##
@@ -268,7 +272,8 @@ void TaskWorkerPool::notify_thread() {
}
bool TaskWorkerPool::_register_task_info(const TTaskType::type task_type,
int64_t signature) {
-if (task_type == TTaskType::type::PUSH_STORAGE_POLICY) {
+if (task_type == TTaskType::type::PUSH_STORAGE_POLICY
+|| TTaskType::type::COOLDOWN_DELETE_FILE) {
Review Comment:
warning: converting the enum constant to a boolean
[clang-diagnostic-int-in-bool-context]
```cpp
|| TTaskType::type::COOLDOWN_DELETE_FILE) {
^
```
##
be/src/agent/task_worker_pool.cpp:
##
@@ -1701,6 +1706,37 @@
}
}
+void TaskWorkerPool::_cooldown_delete_file_worker_thread_callback() {
+while (_is_work) {
+TAgentTaskRequest agent_task_req;
+{
+std::unique_lock
worker_thread_lock(_worker_thread_lock);
+while (_is_work && _tasks.empty()) {
+_worker_thread_condition_variable.wait(worker_thread_lock);
+}
+if (!_is_work) {
+return;
+}
+
+agent_task_req = _tasks.front();
+_tasks.pop_front();
+}
+
+TCooldownDeleteFileReq& cooldown_delete_file_req =
agent_task_req.cooldown_delete_file;
+for (auto& delete_file :
cooldown_delete_file_req.cooldown_delete_files) {
+int64_t tablet_id = delete_file.tablet_id;
+TabletSharedPtr tablet =
+
StorageEngine::instance()->tablet_manager()->get_tablet(tablet_id);
+if (tablet == nullptr) {
+LOG(WARNING) << "failed to get tablet. tablet_id=" <<
tablet_id;
+continue;
+}
+tablet->enable_cooldown_flag(delete_file.delete_id);
Review Comment:
warning: 'enable_cooldown_flag' is a private member of 'doris::Tablet'
[clang-diagnostic-error]
```cpp
tablet->enable_cooldown_flag(delete_file.delete_id);
^
```
**be/src/olap/tablet.h:404:** declared private here
```cpp
void enable_cooldown_flag(const TUniqueId& cooldown_delete_id);
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yixiutt opened a new pull request, #16482: [pick](compaction)pick 16470
yixiutt opened a new pull request, #16482: URL: https://github.com/apache/doris/pull/16482 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16120: [Fix](Nereids)fix runtime filter cannot generate when expression is cast.
github-actions[bot] commented on PR #16120: URL: https://github.com/apache/doris/pull/16120#issuecomment-1420679956 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16120: [Fix](Nereids)fix runtime filter cannot generate when expression is cast.
github-actions[bot] commented on PR #16120: URL: https://github.com/apache/doris/pull/16120#issuecomment-1420680014 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] SaintBacchus opened a new pull request, #16483: [Doc](Load)Add mysql load document
SaintBacchus opened a new pull request, #16483: URL: https://github.com/apache/doris/pull/16483 # Proposed changes Issue Number: close #xxx ## Problem summary Add mysql load document ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16482: [pick](compaction)pick 16470
github-actions[bot] commented on PR #16482: URL: https://github.com/apache/doris/pull/16482#issuecomment-1420684008 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16482: [pick](compaction)pick 16470
github-actions[bot] commented on PR #16482: URL: https://github.com/apache/doris/pull/16482#issuecomment-1420683949 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16469: [bugfix](vertical_compaction) fix base_compaction delete_sign handler
github-actions[bot] commented on PR #16469: URL: https://github.com/apache/doris/pull/16469#issuecomment-1420689548 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #16484: [improvement](fuzzy) print fuzzy session variable in FE audit log
morningman opened a new pull request, #16484: URL: https://github.com/apache/doris/pull/16484 # Proposed changes Issue Number: close #xxx ## Problem summary When enable `use_fuzzy_session_variable` in fe.conf, some of session variables will be set randomly to test corner cases. This PR changes: 1. Use Annotation callback to define the fuzzy method of the variable. 2. print fuzzy variables' value in fe.audit.log only if `use_fuzzy_session_variable` is true eg: ``` FuzzyVariables=disable_streaming_preaggregations=false,parallel_fragment_exec_instance_num=7,enable_pipeline_engine=false,enable_fold_constant_by_be=false,enable_local_exchange=true,partitioned_hash_join_rows_threshold=8,partitioned_hash_agg_rows_threshold=1048576,enable_share_hash_table_for_broadcast_join=true,external_sort_bytes_threshold=1048576,two_phase_read_opt_limit_threshold=512 ``` ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 4. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 5. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 6. Does it need to update dependencies: - [ ] Yes - [ ] No 7. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] AshinGau opened a new pull request, #16485: [improvement](filecache) use dynamic segment size to cache remote file block
AshinGau opened a new pull request, #16485: URL: https://github.com/apache/doris/pull/16485 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [x] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #16120: [Fix](Nereids)fix runtime filter cannot generate when expression is cast.
morrySnow merged PR #16120: URL: https://github.com/apache/doris/pull/16120 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (f90d844a53 -> a4c28e6efa)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from f90d844a53 [improvement](compaction) enable compaction in TABLET_NOTREADY (#16470) add a4c28e6efa [Fix](Nereids) runtime filter cannot generate when expression is cast. (#16120) No new revisions were added by this update. Summary of changes: .../glue/translator/RuntimeFilterTranslator.java | 17 -- .../processor/post/RuntimeFilterContext.java | 4 +- .../processor/post/RuntimeFilterGenerator.java | 61 ++ .../trees/plans/physical/RuntimeFilter.java| 7 +-- .../apache/doris/nereids/util/ExpressionUtils.java | 7 +++ .../nereids/postprocess/RuntimeFilterTest.java | 30 +-- .../suites/nereids_syntax_p0/runtime_filter.groovy | 2 +- 7 files changed, 82 insertions(+), 46 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16471: [fix](message)return result error
hello-stephen commented on PR #16471: URL: https://github.com/apache/doris/pull/16471#issuecomment-1420695082 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.45 seconds load time: 471 seconds storage size: 17171139269 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230207122918_clickbench_pr_91331.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16485: [improvement](filecache) use dynamic segment size to cache remote file block
github-actions[bot] commented on PR #16485: URL: https://github.com/apache/doris/pull/16485#issuecomment-1420697401 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16258: [feature](cooldown)Add cooldown delete
github-actions[bot] commented on PR #16258: URL: https://github.com/apache/doris/pull/16258#issuecomment-1420698451 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16472: [improvement](meta) Enhance Doris's fault tolerance to disk error
github-actions[bot] commented on PR #16472: URL: https://github.com/apache/doris/pull/16472#issuecomment-1420710534 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16472: [improvement](meta) Enhance Doris's fault tolerance to disk error
hello-stephen commented on PR #16472: URL: https://github.com/apache/doris/pull/16472#issuecomment-1420716080 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.38 seconds load time: 477 seconds storage size: 17171082229 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230207124654_clickbench_pr_91377.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Jibing-Li opened a new pull request, #16486: [Fix](multi catalog)(planner) Fix external table statistic collection bug.
Jibing-Li opened a new pull request, #16486: URL: https://github.com/apache/doris/pull/16486 # Proposed changes Issue Number: close #xxx ## Problem summary Add index id to column statistic id. Refresh statistic cache after analyze. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16335: [WIP](dynamic-table) support dynamic schema table
github-actions[bot] commented on PR #16335: URL: https://github.com/apache/doris/pull/16335#issuecomment-1420738241 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16473: [bugfix](scannode) 1. make rows_read correct 2. use single scanner if has limit clause
hello-stephen commented on PR #16473: URL: https://github.com/apache/doris/pull/16473#issuecomment-1420745256 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.05 seconds load time: 481 seconds storage size: 17170850918 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230207130722_clickbench_pr_91348.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16258: [feature](cooldown)Add cooldown delete
github-actions[bot] commented on PR #16258: URL: https://github.com/apache/doris/pull/16258#issuecomment-1420747359 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 commented on pull request #16468: [fix](func) fix truncate float type result error
Gabriel39 commented on PR #16468: URL: https://github.com/apache/doris/pull/16468#issuecomment-1420750818 Thanks. It's a mistake. I will resolve this losing precision problem later -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16468: [fix](func) fix truncate float type result error
github-actions[bot] commented on PR #16468: URL: https://github.com/apache/doris/pull/16468#issuecomment-1420752256 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16468: [fix](func) fix truncate float type result error
github-actions[bot] commented on PR #16468: URL: https://github.com/apache/doris/pull/16468#issuecomment-1420752316 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16258: [feature](cooldown)Add cooldown delete
github-actions[bot] commented on PR #16258: URL: https://github.com/apache/doris/pull/16258#issuecomment-1420765881 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 merged pull request #16476: [fix](pipeline) incorrect result when disabling sharing hash table
Gabriel39 merged PR #16476: URL: https://github.com/apache/doris/pull/16476 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](pipeline) incorrect result when disabling sharing hash table (#16476)
This is an automated email from the ASF dual-hosted git repository.
gabriellee pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 91325e5ca3 [fix](pipeline) incorrect result when disabling sharing
hash table (#16476)
91325e5ca3 is described below
commit 91325e5ca30ca9ef4329236490c906b09e98b4f3
Author: Jerry Hu
AuthorDate: Tue Feb 7 21:25:32 2023 +0800
[fix](pipeline) incorrect result when disabling sharing hash table (#16476)
---
be/src/vec/sink/vdata_stream_sender.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/vec/sink/vdata_stream_sender.h
b/be/src/vec/sink/vdata_stream_sender.h
index 8b2c1e0847..e644aaf847 100644
--- a/be/src/vec/sink/vdata_stream_sender.h
+++ b/be/src/vec/sink/vdata_stream_sender.h
@@ -370,7 +370,7 @@ public:
}
if (eos || block->column_metas_size()) {
RETURN_IF_ERROR(_buffer->add_block(
-{this, block ? std::make_unique(std::move(*block))
: nullptr, eos}));
+{this, block ? std::make_unique(*block) : nullptr,
eos}));
}
return Status::OK();
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 commented on pull request #16476: [fix](pipeline) incorrect result when disabling sharing hash table
Gabriel39 commented on PR #16476: URL: https://github.com/apache/doris/pull/16476#issuecomment-1420774040 This PR doesn't resolve the root problem. I will fix it in next PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16477: [enhance](Nereids): refactor JoinReorder code.
hello-stephen commented on PR #16477: URL: https://github.com/apache/doris/pull/16477#issuecomment-1420775744 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.58 seconds load time: 485 seconds storage size: 17171439536 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230207132719_clickbench_pr_91413.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16258: [feature](cooldown)Add cooldown delete
github-actions[bot] commented on PR #16258: URL: https://github.com/apache/doris/pull/16258#issuecomment-1420776903 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] platoneko opened a new pull request, #16488: [fix](cooldown) Rewrite update cooldown conf
platoneko opened a new pull request, #16488: URL: https://github.com/apache/doris/pull/16488 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16410: [fix](udf) fix create udf function with uppercase database name can't recognize
github-actions[bot] commented on PR #16410: URL: https://github.com/apache/doris/pull/16410#issuecomment-1420779907 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16488: [fix](cooldown) Rewrite update cooldown conf
github-actions[bot] commented on PR #16488: URL: https://github.com/apache/doris/pull/16488#issuecomment-1420782925 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] eldenmoon opened a new pull request, #16489: [Bug](point query) Reusable in PointQueryExecutor should call init b…
eldenmoon opened a new pull request, #16489: URL: https://github.com/apache/doris/pull/16489 …efore add to LookupCache Otherwise in high concurrent query, _block_pool maybe used before Reusable::init done in other threads # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen opened a new pull request, #16490: [regression](fuzzy) set random value for some session variables
hello-stephen opened a new pull request, #16490: URL: https://github.com/apache/doris/pull/16490 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16489: [Bug](point query) Reusable in PointQueryExecutor should call init b…
github-actions[bot] commented on PR #16489: URL: https://github.com/apache/doris/pull/16489#issuecomment-1420797974 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen closed pull request #16289: [fuzzy](test) fuzzy some session variables stably according to pull_r…
hello-stephen closed pull request #16289: [fuzzy](test) fuzzy some session variables stably according to pull_r… URL: https://github.com/apache/doris/pull/16289 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #16482: [pick](compaction)pick 16470
morningman merged PR #16482: URL: https://github.com/apache/doris/pull/16482 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated: [improvement](compaction) enable compaction if state is TABLET_NOTREADY (#16482)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new df909238eb [improvement](compaction) enable compaction if state is
TABLET_NOTREADY (#16482)
df909238eb is described below
commit df909238ebf13dcf990fac48c7a7970c209f889a
Author: yixiutt <102007456+yixi...@users.noreply.github.com>
AuthorDate: Tue Feb 7 21:46:21 2023 +0800
[improvement](compaction) enable compaction if state is TABLET_NOTREADY
(#16482)
cherry-pick 16470
---
be/src/olap/cumulative_compaction_policy.cpp | 8
be/src/olap/tablet.cpp | 7 +++
2 files changed, 11 insertions(+), 4 deletions(-)
diff --git a/be/src/olap/cumulative_compaction_policy.cpp
b/be/src/olap/cumulative_compaction_policy.cpp
index 5b30e3aa07..821c4a43b3 100644
--- a/be/src/olap/cumulative_compaction_policy.cpp
+++ b/be/src/olap/cumulative_compaction_policy.cpp
@@ -242,6 +242,7 @@ int SizeBasedCumulativeCompactionPolicy::pick_input_rowsets(
std::vector* input_rowsets, Version*
last_delete_version,
size_t* compaction_score) {
size_t promotion_size = tablet->cumulative_promotion_size();
+auto max_version = tablet->max_version().first;
int transient_size = 0;
*compaction_score = 0;
int64_t total_size = 0;
@@ -262,6 +263,13 @@ int
SizeBasedCumulativeCompactionPolicy::pick_input_rowsets(
continue;
}
}
+if (tablet->tablet_state() == TABLET_NOTREADY) {
+// If tablet under alter, keep latest 10 version so that base
tablet max version
+// not merged in new tablet, and then we can copy data from base
tablet
+if (rowset->version().second < max_version - 10) {
+continue;
+}
+}
if (*compaction_score >= max_compaction_score) {
// got enough segments
break;
diff --git a/be/src/olap/tablet.cpp b/be/src/olap/tablet.cpp
index f039bf96f7..d1b3eca443 100644
--- a/be/src/olap/tablet.cpp
+++ b/be/src/olap/tablet.cpp
@@ -794,10 +794,9 @@ bool Tablet::can_do_compaction(size_t path_hash,
CompactionType compaction_type)
}
if (tablet_state() == TABLET_NOTREADY) {
-// Before doing schema change, tablet's rowsets that versions smaller
than max converting version will be
-// removed. So, we only need to do the compaction when it is being
converted.
-// After being converted, tablet's state will be changed to
TABLET_RUNNING.
-return SchemaChangeHandler::tablet_in_converting(tablet_id());
+// In TABLET_NOTREADY, we keep last 10 versions in new tablet so base
tablet max_version
+// not merged in new tablet and then we can do compaction
+return true;
}
return true;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16485: [improvement](filecache) use dynamic segment size to cache remote file block
github-actions[bot] commented on PR #16485: URL: https://github.com/apache/doris/pull/16485#issuecomment-1420829438 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
