[GitHub] [doris] github-actions[bot] commented on pull request #11921: [refractor](bitmap) bitmap serialize and deserialize refractor

2022-08-20 Thread GitBox 


github-actions[bot] commented on PR #11921:
URL: https://github.com/apache/doris/pull/11921#issuecomment-1221252832

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11921: [refractor](bitmap) bitmap serialize and deserialize refractor

2022-08-20 Thread GitBox 


github-actions[bot] commented on PR #11921:
URL: https://github.com/apache/doris/pull/11921#issuecomment-1221252844

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] platoneko opened a new pull request, #11940: [refactor](status) Refactor status handling in agent task

2022-08-20 Thread GitBox 


platoneko opened a new pull request, #11940:
URL: https://github.com/apache/doris/pull/11940

   # Proposed changes
   
   Issue Number: #11874
   
   ## Problem summary
   
   1. Refactor `TaggableLogger`
   2. Refactor status handling in agent task:
   1. Unify log format in TaskWorkerPool
   2. Pass `Status` to the top caller, and replace some `OLAPInternalError` 
with more detailed error message `Status`
   3. Premature return with the opposite condition to reduce indention
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   4. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   5. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   6. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   7. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] nextdreamblue opened a new issue, #11941: [Enhancement] (restore) restore table or partition with the same replication_num as before the backup

2022-08-20 Thread GitBox 


nextdreamblue opened a new issue, #11941:
URL: https://github.com/apache/doris/issues/11941

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Description
   
   right now, if i restore a table, i must set "replication_num"="1" or other 
replication num or default 3 replication, and i will get a table with all 
parititions which have same replication_num.
   
   if a table have different replication num, if backup and restore it, i can 
not get a table which every partition with the same replication num as before 
the backup.
   
   for example:
   every partition of a table do not have same  replication_num:
   MySQL [tmp]> show partitions from part_p;
   
+-+---++-++--++-+-++---+-+-+--+--++-+
   | PartitionId | PartitionName | VisibleVersion | VisibleVersionTime  | State 
 | PartitionKey | Range 
 | DistributionKey | Buckets | ReplicationNum | StorageMedium | 
CooldownTime| RemoteStoragePolicy | LastConsistencyCheckTime | DataSize 
| IsInMemory | ReplicaAllocation   |
   
+-+---++-++--++-+-++---+-+-+--+--++-+
   | 16004   | p1| 2  | 2022-08-20 17:14:51 | 
NORMAL | p_partkey| [types: [INT]; keys: [-2147483648]; ..types: [INT]; 
keys: [1013157]; ) | p_partkey   | 3   | 1  | HDD   
| -12-31 23:59:59 | | NULL | 8.634 
MB | false  | tag.location.default: 1 |
   | 16005   | p2| 2  | 2022-08-20 17:14:51 | 
NORMAL | p_partkey| [types: [INT]; keys: [1013157]; ..types: [INT]; keys: 
[1113157]; ) | p_partkey   | 3   | 2  | HDD   | 
-12-31 23:59:59 | | NULL | 2.175 MB 
| false  | tag.location.default: 2 |
   | 16006   | p3| 2  | 2022-08-20 17:14:51 | 
NORMAL | p_partkey| [types: [INT]; keys: [1113157]; ..types: [INT]; keys: 
[1213157]; ) | p_partkey   | 3   | 3  | HDD   | 
-12-31 23:59:59 | | NULL | 3.265 MB 
| false  | tag.location.default: 3 |
   | 16007   | p4| 2  | 2022-08-20 17:14:51 | 
NORMAL | p_partkey| [types: [INT]; keys: [1213157]; ..types: [INT]; keys: 
[2147483647]; )  | p_partkey   | 3   | 1  | HDD   | 
-12-31 23:59:59 | | NULL | 1.810 MB 
| false  | tag.location.default: 1 |
   
+-+---++-++--++-+-++---+-+-+--+--++-+
   4 rows in set (0.00 sec)
   
   i backup this table and restore it, i can not get same table with same 
partitions with the same replication num as before the backup.
   
   
   ### Solution
   
   add restore new property 'reserve_replica', which means you can get a table 
with same partitions with the same replication num as before the backup.
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11886: [Bugfix](inpredicate) fix in predicate in group by clause may cause NPE

2022-08-20 Thread GitBox 


github-actions[bot] commented on PR #11886:
URL: https://github.com/apache/doris/pull/11886#issuecomment-1221271794

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11886: [Bugfix](inpredicate) fix in predicate in group by clause may cause NPE

2022-08-20 Thread GitBox 


github-actions[bot] commented on PR #11886:
URL: https://github.com/apache/doris/pull/11886#issuecomment-1221271802

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11906: [bugfix](load) fix cancel load stmt cannot recognize key words in upper case

2022-08-20 Thread GitBox 


github-actions[bot] commented on PR #11906:
URL: https://github.com/apache/doris/pull/11906#issuecomment-1221271957

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11906: [bugfix](load) fix cancel load stmt cannot recognize key words in upper case

2022-08-20 Thread GitBox 


github-actions[bot] commented on PR #11906:
URL: https://github.com/apache/doris/pull/11906#issuecomment-1221271961

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] cambyzju commented on pull request #11921: [refractor](bitmap) bitmap serialize and deserialize refractor

2022-08-20 Thread GitBox 


cambyzju commented on PR #11921:
URL: https://github.com/apache/doris/pull/11921#issuecomment-1221272719

   rebase to trigger P0 again


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman commented on a diff in pull request #11450: [feature](cold_on_s3) Show remote data usage via SHOW BACKENDS and SHOW TABLETS statements

2022-08-20 Thread GitBox 


morningman commented on code in PR #11450:
URL: https://github.com/apache/doris/pull/11450#discussion_r950678147


##
be/src/io/fs/s3_file_system.cpp:
##
@@ -141,7 +141,7 @@ Status S3FileSystem::open_file(const Path& path, 
FileReaderSPtr* reader) {
 RETURN_IF_ERROR(file_size(path, &fsize));
 auto key = get_key(path);
 auto fs_path = Path(_s3_conf.endpoint) / _s3_conf.bucket / key;
-*reader = std::make_unique(std::move(fs_path), fsize, 
std::move(key),
+*reader = std::make_shared(std::move(fs_path), fsize, 
std::move(key),

Review Comment:
   Why change to shared ?



##
fe/fe-core/src/main/java/org/apache/doris/common/proc/ReplicasProcNode.java:
##
@@ -33,12 +33,10 @@
  * show replicas' detail info within a tablet
  */
 public class ReplicasProcNode implements ProcNodeInterface {
-public static final ImmutableList TITLE_NAMES = new 
ImmutableList.Builder()
-.add("ReplicaId").add("BackendId").add("Version")
-.add("LstSuccessVersion").add("LstFailedVersion")
-
.add("LstFailedTime").add("SchemaHash").add("DataSize").add("RowCount").add("State")
-
.add("IsBad").add("VersionCount").add("PathHash").add("MetaUrl").add("CompactionStatus")
-.build();
+public static final ImmutableList TITLE_NAMES = new 
ImmutableList.Builder().add("ReplicaId")
+
.add("BackendId").add("Version").add("LstSuccessVersion").add("LstFailedVersion").add("LstFailedTime")
+
.add("SchemaHash").add("LocalDataSize").add("RowCount").add("State").add("IsBad")
+
.add("VersionCount").add("PathHash").add("MetaUrl").add("CompactionStatus").add("RemoteDataSize").build();

Review Comment:
   Add `RemoteDataSize` right after `LocalDataSize`.



##
fe/fe-core/src/main/java/org/apache/doris/common/proc/TabletsProcDir.java:
##
@@ -45,9 +45,9 @@ public class TabletsProcDir implements ProcDirInterface {
 public static final ImmutableList TITLE_NAMES = new 
ImmutableList.Builder()
 
.add("TabletId").add("ReplicaId").add("BackendId").add("SchemaHash").add("Version")
 
.add("LstSuccessVersion").add("LstFailedVersion").add("LstFailedTime")
-.add("DataSize").add("RowCount").add("State")
+.add("LocalDataSize").add("RowCount").add("State")
 .add("LstConsistencyCheckTime").add("CheckVersion")
-
.add("VersionCount").add("PathHash").add("MetaUrl").add("CompactionStatus")
+
.add("VersionCount").add("PathHash").add("MetaUrl").add("CompactionStatus").add("RemoteDataSize")

Review Comment:
   after `LocalDataSize`



##
fe/fe-core/src/main/java/org/apache/doris/common/proc/BackendsProcDir.java:
##
@@ -51,7 +51,7 @@ public class BackendsProcDir implements ProcDirInterface {
 
.add("BePort").add("HttpPort").add("BrpcPort").add("LastStartTime").add("LastHeartbeat").add("Alive")
 
.add("SystemDecommissioned").add("ClusterDecommissioned").add("TabletNum")
 
.add("DataUsedCapacity").add("AvailCapacity").add("TotalCapacity").add("UsedPct")
-
.add("MaxDiskUsedPct").add("Tag").add("ErrMsg").add("Version").add("Status")
+
.add("MaxDiskUsedPct").add("Tag").add("ErrMsg").add("Version").add("Status").add("RemoteUsedCapacity")

Review Comment:
   Better to add `RemoteUsedCapacity` right after `DataUsedCapacity`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11938: [fix](build) Resolve the conflicts when building be with java-udf

2022-08-20 Thread GitBox 


github-actions[bot] commented on PR #11938:
URL: https://github.com/apache/doris/pull/11938#issuecomment-1221273821

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman merged pull request #11938: [fix](build) Resolve the conflicts when building be with java-udf

2022-08-20 Thread GitBox 


morningman merged PR #11938:
URL: https://github.com/apache/doris/pull/11938


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated (28dba65d74 -> 982c5f06b5)

2022-08-20 Thread morningman 
This is an automated email from the ASF dual-hosted git repository.

morningman pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


from 28dba65d74 Update basic-summary.md (#11889)
 add 982c5f06b5 [fix](build) Resolve the conflicts when building be with 
java-udf (#11938)

No new revisions were added by this update.

Summary of changes:
 be/src/util/jni-util.h| 4 +---
 be/src/vec/aggregate_functions/aggregate_function_java_udaf.h | 8 ++--
 be/src/vec/functions/function_java_udf.cpp| 1 +
 3 files changed, 4 insertions(+), 9 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman closed issue #11937: [Bug](build) Failed to build be with java-udf

2022-08-20 Thread GitBox 


morningman closed issue #11937: [Bug](build) Failed to build be with java-udf
URL: https://github.com/apache/doris/issues/11937


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman commented on a diff in pull request #11927: [feature](spark-load)Spark load supports string type data import

2022-08-20 Thread GitBox 


morningman commented on code in PR #11927:
URL: https://github.com/apache/doris/pull/11927#discussion_r950679701


##
fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/SparkDpp.java:
##
@@ -405,6 +405,18 @@ public boolean validateData(Object srcValue, 
EtlJobConfig.EtlColumn etlColumn, C
 return false;
 }
 break;
+case "STRING":
+case "TEXT":
+// TODO(zjf) padding string type
+int strDataSize = 0;
+if (srcValue != null && (strDataSize = 
srcValue.toString().getBytes(StandardCharsets.UTF_8).length)
+> 1048576) {

Review Comment:
   define `1048576`



##
fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/ColumnParser.java:
##
@@ -208,6 +211,25 @@ public boolean parse(String value) {
 }
 }
 
+class StringTypeParser extends ColumnParser {
+
+private EtlJobConfig.EtlColumn etlColumn;
+
+public StringTypeParser(EtlJobConfig.EtlColumn etlColumn) {
+this.etlColumn = etlColumn;
+}
+
+@Override
+public boolean parse(String value) {
+try {
+return value.getBytes("UTF-8").length <= 1048576;

Review Comment:
   Better define this number somewhere.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman merged pull request #11692: [extension](mysql-to-doris) add odbc conf and some fix

2022-08-20 Thread GitBox 


morningman merged PR #11692:
URL: https://github.com/apache/doris/pull/11692


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated (982c5f06b5 -> d4749c2652)

2022-08-20 Thread morningman 
This is an automated email from the ASF dual-hosted git repository.

morningman pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


from 982c5f06b5 [fix](build) Resolve the conflicts when building be with 
java-udf (#11938)
 add d4749c2652 [extension](mysql-to-doris) add odbc conf and  some fix 
(#11692)

No new revisions were added by this update.

Summary of changes:
 extension/mysql_to_doris/all_tables.sh | 12 --
 extension/mysql_to_doris/conf/doris.conf   |  1 +
 extension/mysql_to_doris/user_define_tables.sh | 60 +++---
 3 files changed, 44 insertions(+), 29 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11886: [Bugfix](inpredicate) fix in predicate in group by clause may cause NPE

2022-08-20 Thread GitBox 


github-actions[bot] commented on PR #11886:
URL: https://github.com/apache/doris/pull/11886#issuecomment-1221274522

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman closed pull request #11103: Bump fastjson from 1.2.62 to 1.2.83 in /samples/doris-demo/flink-demo-v1.1

2022-08-20 Thread GitBox 


morningman closed pull request #11103: Bump fastjson from 1.2.62 to 1.2.83 in 
/samples/doris-demo/flink-demo-v1.1
URL: https://github.com/apache/doris/pull/11103


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] dependabot[bot] commented on pull request #11103: Bump fastjson from 1.2.62 to 1.2.83 in /samples/doris-demo/flink-demo-v1.1

2022-08-20 Thread GitBox 


dependabot[bot] commented on PR #11103:
URL: https://github.com/apache/doris/pull/11103#issuecomment-1221274945

   OK, I won't notify you again about this release, but will get in touch when 
a new version is available. If you'd rather skip all updates until the next 
major or minor version, let me know by commenting `@dependabot ignore this 
major version` or `@dependabot ignore this minor version`.
   
   If you change your mind, just re-open this PR and I'll resolve any conflicts 
on it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] hf200012 commented on a diff in pull request #11927: [feature](spark-load)Spark load supports string type data import

2022-08-20 Thread GitBox 


hf200012 commented on code in PR #11927:
URL: https://github.com/apache/doris/pull/11927#discussion_r950680498


##
fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/SparkDpp.java:
##
@@ -405,6 +405,18 @@ public boolean validateData(Object srcValue, 
EtlJobConfig.EtlColumn etlColumn, C
 return false;
 }
 break;
+case "STRING":
+case "TEXT":
+// TODO(zjf) padding string type
+int strDataSize = 0;
+if (srcValue != null && (strDataSize = 
srcValue.toString().getBytes(StandardCharsets.UTF_8).length)
+> 1048576) {

Review Comment:
   Resolved



##
fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/ColumnParser.java:
##
@@ -208,6 +211,25 @@ public boolean parse(String value) {
 }
 }
 
+class StringTypeParser extends ColumnParser {
+
+private EtlJobConfig.EtlColumn etlColumn;
+
+public StringTypeParser(EtlJobConfig.EtlColumn etlColumn) {
+this.etlColumn = etlColumn;
+}
+
+@Override
+public boolean parse(String value) {
+try {
+return value.getBytes("UTF-8").length <= 1048576;

Review Comment:
   Resolved



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] platoneko commented on a diff in pull request #11450: [feature](cold_on_s3) Show remote data usage via SHOW BACKENDS and SHOW TABLETS statements

2022-08-20 Thread GitBox 


platoneko commented on code in PR #11450:
URL: https://github.com/apache/doris/pull/11450#discussion_r950681322


##
be/src/io/fs/s3_file_system.cpp:
##
@@ -141,7 +141,7 @@ Status S3FileSystem::open_file(const Path& path, 
FileReaderSPtr* reader) {
 RETURN_IF_ERROR(file_size(path, &fsize));
 auto key = get_key(path);
 auto fs_path = Path(_s3_conf.endpoint) / _s3_conf.bucket / key;
-*reader = std::make_unique(std::move(fs_path), fsize, 
std::move(key),
+*reader = std::make_shared(std::move(fs_path), fsize, 
std::move(key),

Review Comment:
   This is a typo(not bug) fix, as function signature is 
`S3FileSystem::open_file(const Path& path, FileReaderSPtr* reader)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] platoneko commented on a diff in pull request #11450: [feature](cold_on_s3) Show remote data usage via SHOW BACKENDS and SHOW TABLETS statements

2022-08-20 Thread GitBox 


platoneko commented on code in PR #11450:
URL: https://github.com/apache/doris/pull/11450#discussion_r950681634


##
fe/fe-core/src/main/java/org/apache/doris/common/proc/BackendsProcDir.java:
##
@@ -51,7 +51,7 @@ public class BackendsProcDir implements ProcDirInterface {
 
.add("BePort").add("HttpPort").add("BrpcPort").add("LastStartTime").add("LastHeartbeat").add("Alive")
 
.add("SystemDecommissioned").add("ClusterDecommissioned").add("TabletNum")
 
.add("DataUsedCapacity").add("AvailCapacity").add("TotalCapacity").add("UsedPct")
-
.add("MaxDiskUsedPct").add("Tag").add("ErrMsg").add("Version").add("Status")
+
.add("MaxDiskUsedPct").add("Tag").add("ErrMsg").add("Version").add("Status").add("RemoteUsedCapacity")

Review Comment:
   Some unit tests and regression tests rely on the subscript of these data. If 
`RemoteUsedCapacity` is inserted into the middle, the tests will fail.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] nextdreamblue opened a new pull request, #11942: [Enhancement] (restore) restore table or partition with the same replication_num as before the backup

2022-08-20 Thread GitBox 


nextdreamblue opened a new pull request, #11942:
URL: https://github.com/apache/doris/pull/11942

   add restore new property 'reserve_replica', which means you can
   get a table with same partitions with the same replication num
   as before the backup.
   
   # Proposed changes
   
   Issue Number: close #11941
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   2. Has unit tests been added:
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   4. Does it need to update dependencies:
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11927: [feature](spark-load)Spark load supports string type data import

2022-08-20 Thread GitBox 


github-actions[bot] commented on PR #11927:
URL: https://github.com/apache/doris/pull/11927#issuecomment-1221308024

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11927: [feature](spark-load)Spark load supports string type data import

2022-08-20 Thread GitBox 


github-actions[bot] commented on PR #11927:
URL: https://github.com/apache/doris/pull/11927#issuecomment-1221308037

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] platoneko commented on a diff in pull request #11450: [feature](cold_on_s3) Show remote data usage via SHOW BACKENDS and SHOW TABLETS statements

2022-08-20 Thread GitBox 


platoneko commented on code in PR #11450:
URL: https://github.com/apache/doris/pull/11450#discussion_r950703910


##
fe/fe-core/src/main/java/org/apache/doris/common/proc/ReplicasProcNode.java:
##
@@ -33,12 +33,10 @@
  * show replicas' detail info within a tablet
  */
 public class ReplicasProcNode implements ProcNodeInterface {
-public static final ImmutableList TITLE_NAMES = new 
ImmutableList.Builder()
-.add("ReplicaId").add("BackendId").add("Version")
-.add("LstSuccessVersion").add("LstFailedVersion")
-
.add("LstFailedTime").add("SchemaHash").add("DataSize").add("RowCount").add("State")
-
.add("IsBad").add("VersionCount").add("PathHash").add("MetaUrl").add("CompactionStatus")
-.build();
+public static final ImmutableList TITLE_NAMES = new 
ImmutableList.Builder().add("ReplicaId")
+
.add("BackendId").add("Version").add("LstSuccessVersion").add("LstFailedVersion").add("LstFailedTime")
+
.add("SchemaHash").add("LocalDataSize").add("RowCount").add("State").add("IsBad")
+
.add("VersionCount").add("PathHash").add("MetaUrl").add("CompactionStatus").add("RemoteDataSize").build();

Review Comment:
   done



##
fe/fe-core/src/main/java/org/apache/doris/common/proc/TabletsProcDir.java:
##
@@ -45,9 +45,9 @@ public class TabletsProcDir implements ProcDirInterface {
 public static final ImmutableList TITLE_NAMES = new 
ImmutableList.Builder()
 
.add("TabletId").add("ReplicaId").add("BackendId").add("SchemaHash").add("Version")
 
.add("LstSuccessVersion").add("LstFailedVersion").add("LstFailedTime")
-.add("DataSize").add("RowCount").add("State")
+.add("LocalDataSize").add("RowCount").add("State")
 .add("LstConsistencyCheckTime").add("CheckVersion")
-
.add("VersionCount").add("PathHash").add("MetaUrl").add("CompactionStatus")
+
.add("VersionCount").add("PathHash").add("MetaUrl").add("CompactionStatus").add("RemoteDataSize")

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] platoneko commented on a diff in pull request #11450: [feature](cold_on_s3) Show remote data usage via SHOW BACKENDS and SHOW TABLETS statements

2022-08-20 Thread GitBox 


platoneko commented on code in PR #11450:
URL: https://github.com/apache/doris/pull/11450#discussion_r950703922


##
fe/fe-core/src/main/java/org/apache/doris/common/proc/BackendsProcDir.java:
##
@@ -51,7 +51,7 @@ public class BackendsProcDir implements ProcDirInterface {
 
.add("BePort").add("HttpPort").add("BrpcPort").add("LastStartTime").add("LastHeartbeat").add("Alive")
 
.add("SystemDecommissioned").add("ClusterDecommissioned").add("TabletNum")
 
.add("DataUsedCapacity").add("AvailCapacity").add("TotalCapacity").add("UsedPct")
-
.add("MaxDiskUsedPct").add("Tag").add("ErrMsg").add("Version").add("Status")
+
.add("MaxDiskUsedPct").add("Tag").add("ErrMsg").add("Version").add("Status").add("RemoteUsedCapacity")

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11450: [feature](cold_on_s3) Show remote data usage via SHOW BACKENDS and SHOW TABLETS statements

2022-08-20 Thread GitBox 


github-actions[bot] commented on PR #11450:
URL: https://github.com/apache/doris/pull/11450#issuecomment-1221337920

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11450: [feature](cold_on_s3) Show remote data usage via SHOW BACKENDS and SHOW TABLETS statements

2022-08-20 Thread GitBox 


github-actions[bot] commented on PR #11450:
URL: https://github.com/apache/doris/pull/11450#issuecomment-1221338016

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yiguolei merged pull request #11903: [refactor](planner): refactor equals code in Catalog dir.

2022-08-20 Thread GitBox 


yiguolei merged PR #11903:
URL: https://github.com/apache/doris/pull/11903


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [refactor](planner): refactor equals code in Catalog dir. (#11903)

2022-08-20 Thread yiguolei 
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new c2efa9c3b5 [refactor](planner): refactor equals code in Catalog dir. 
(#11903)
c2efa9c3b5 is described below

commit c2efa9c3b526e7cca1c88a4a445223e1cfb8d47a
Author: jakevin 
AuthorDate: Sun Aug 21 10:01:57 2022 +0800

[refactor](planner): refactor equals code in Catalog dir. (#11903)
---
 .../org/apache/doris/analysis/DataSortInfo.java| 18 +++--
 .../main/java/org/apache/doris/catalog/Column.java | 80 +-
 .../java/org/apache/doris/catalog/Database.java| 25 ++-
 .../org/apache/doris/catalog/DistributionInfo.java | 21 --
 .../java/org/apache/doris/catalog/FsBroker.java|  9 +--
 .../apache/doris/catalog/HashDistributionInfo.java | 23 ---
 .../apache/doris/catalog/ListPartitionItem.java| 18 ++---
 .../apache/doris/catalog/MaterializedIndex.java| 27 ++--
 .../doris/catalog/MaterializedIndexMeta.java   | 37 +++---
 .../java/org/apache/doris/catalog/OlapTable.java   | 46 ++---
 .../java/org/apache/doris/catalog/Partition.java   | 30 +++-
 .../org/apache/doris/catalog/PartitionInfo.java| 23 +++
 .../doris/catalog/RandomDistributionInfo.java  | 22 +++---
 .../main/java/org/apache/doris/catalog/Table.java  |  4 --
 .../org/apache/doris/catalog/TempPartitions.java   | 19 +
 .../org/apache/doris/qe/cache/PartitionRange.java  |  4 --
 16 files changed, 192 insertions(+), 214 deletions(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/analysis/DataSortInfo.java 
b/fe/fe-core/src/main/java/org/apache/doris/analysis/DataSortInfo.java
index 4393dc162c..651bde7fc3 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/DataSortInfo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/DataSortInfo.java
@@ -28,6 +28,7 @@ import java.io.DataInput;
 import java.io.DataOutput;
 import java.io.IOException;
 import java.util.Map;
+import java.util.Objects;
 
 public class DataSortInfo implements Writable {
 public static final String DATA_SORT_PROPERTY_PREFIX = "data_sort";
@@ -86,14 +87,21 @@ public class DataSortInfo implements Writable {
 return GsonUtils.GSON.fromJson(json, DataSortInfo.class);
 }
 
-public boolean equals(DataSortInfo dataSortInfo) {
-if (this.sortType != dataSortInfo.sortType) {
-return false;
+@Override
+public boolean equals(Object o) {
+if (this == o) {
+return true;
 }
-if (this.colNum != dataSortInfo.colNum) {
+if (o == null || getClass() != o.getClass()) {
 return false;
 }
-return true;
+DataSortInfo that = (DataSortInfo) o;
+return colNum == that.colNum && sortType == that.sortType;
+}
+
+@Override
+public int hashCode() {
+return Objects.hash(sortType, colNum);
 }
 
 public String toSql() {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java 
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
index 92887cba36..a7ca238e60 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
@@ -565,8 +565,8 @@ public class Column implements Writable {
 
 @Override
 public int hashCode() {
-return Objects.hash(name, getDataType(), aggregationType, 
isAggregationTypeImplicit, isKey, isAllowNull,
-getDefaultValue(), getStrLen(), getPrecision(), getScale(), 
comment, visible, children);
+return Objects.hash(name, getDataType(), getStrLen(), getPrecision(), 
getScale(), aggregationType,
+isAggregationTypeImplicit, isKey, isAllowNull, defaultValue, 
comment, children, visible);
 }
 
 @Override
@@ -580,62 +580,20 @@ public class Column implements Writable {
 
 Column other = (Column) obj;
 
-if (!this.name.equalsIgnoreCase(other.getName())) {
-return false;
-}
-if (this.getDataType() != other.getDataType()) {
-return false;
-}
-if (this.aggregationType != other.getAggregationType()) {
-return false;
-}
-if (this.isAggregationTypeImplicit != 
other.isAggregationTypeImplicit()) {
-return false;
-}
-if (this.isKey != other.isKey()) {
-return false;
-}
-if (this.isAllowNull != other.isAllowNull) {
-return false;
-}
-if (this.getDefaultValue() == null) {
-if (other.getDefaultValue() != null) {
-return false;
-}
-} else {
-if (!this.getDefaultValue().equals(other.getDefaultValue())) {
-return false;
-}
-}
-
-if (this.getStrLen() !=

[GitHub] [doris] yiguolei merged pull request #11906: [bugfix](load) fix cancel load stmt cannot recognize key words in upper case

2022-08-20 Thread GitBox 


yiguolei merged PR #11906:
URL: https://github.com/apache/doris/pull/11906


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [bugfix](load) fix cancel load stmt cannot recognize key words in upper case (#11906)

2022-08-20 Thread yiguolei 
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 161d134270 [bugfix](load) fix cancel load stmt cannot recognize key 
words in upper case (#11906)
161d134270 is described below

commit 161d1342707d932f14630c5bcb5df42e82e32a67
Author: Zhengguo Yang 
AuthorDate: Sun Aug 21 10:03:10 2022 +0800

[bugfix](load) fix cancel load stmt cannot recognize key words in upper 
case (#11906)
---
 .../src/main/java/org/apache/doris/analysis/CancelLoadStmt.java   | 8 +---
 .../test/java/org/apache/doris/analysis/CancelLoadStmtTest.java   | 8 
 2 files changed, 13 insertions(+), 3 deletions(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/analysis/CancelLoadStmt.java 
b/fe/fe-core/src/main/java/org/apache/doris/analysis/CancelLoadStmt.java
index 7af21948b5..263b66cf31 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CancelLoadStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CancelLoadStmt.java
@@ -23,10 +23,10 @@ import org.apache.doris.common.AnalysisException;
 import org.apache.doris.common.UserException;
 
 import com.google.common.base.Strings;
-import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
 import lombok.Getter;
 
-import java.util.List;
+import java.util.Set;
 
 
 /**
@@ -36,7 +36,7 @@ import java.util.List;
  **/
 public class CancelLoadStmt extends DdlStmt {
 
-private static final List SUPPORT_COLUMNS = 
Lists.newArrayList("label", "state");
+private static final Set SUPPORT_COLUMNS = 
Sets.newTreeSet(String.CASE_INSENSITIVE_ORDER);
 
 @Getter
 private String dbName;
@@ -55,6 +55,8 @@ public class CancelLoadStmt extends DdlStmt {
 public CancelLoadStmt(String dbName, Expr whereClause) {
 this.dbName = dbName;
 this.whereClause = whereClause;
+this.SUPPORT_COLUMNS.add("label");
+this.SUPPORT_COLUMNS.add("state");
 }
 
 private void checkColumn(Expr expr, boolean like) throws AnalysisException 
{
diff --git 
a/fe/fe-core/src/test/java/org/apache/doris/analysis/CancelLoadStmtTest.java 
b/fe/fe-core/src/test/java/org/apache/doris/analysis/CancelLoadStmtTest.java
index 3521a88f44..f2e9a39a63 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/analysis/CancelLoadStmtTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/CancelLoadStmtTest.java
@@ -67,6 +67,14 @@ public class CancelLoadStmtTest extends TestWithFeService {
 Assertions.assertEquals("CANCEL LOAD FROM default_cluster:testDb WHERE 
`label` = 'doris_test_label'",
 stmt.toString());
 
+SlotRef labelSlotRefUpper = new SlotRef(null, "LABEL");
+BinaryPredicate labelBinaryPredicateUpper = new 
BinaryPredicate(BinaryPredicate.Operator.EQ, labelSlotRefUpper,
+labelStringLiteral);
+CancelLoadStmt stmtUpper = new CancelLoadStmt(null, 
labelBinaryPredicateUpper);
+stmtUpper.analyze(analyzer);
+Assertions.assertEquals("CANCEL LOAD FROM default_cluster:testDb WHERE 
`LABEL` = 'doris_test_label'",
+stmtUpper.toString());
+
 BinaryPredicate stateBinaryPredicate = new 
BinaryPredicate(BinaryPredicate.Operator.EQ, stateSlotRef,
 stateStringLiteral);
 stmt = new CancelLoadStmt(null, stateBinaryPredicate);


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yiguolei merged pull request #11886: [Bugfix](inpredicate) fix in predicate in group by clause may cause NPE

2022-08-20 Thread GitBox 


yiguolei merged PR #11886:
URL: https://github.com/apache/doris/pull/11886


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yiguolei closed issue #11881: [Bug] NPE when inpredicate in group by clause

2022-08-20 Thread GitBox 


yiguolei closed issue #11881: [Bug] NPE when inpredicate in group by clause
URL: https://github.com/apache/doris/issues/11881


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [Bugfix](inpredicate) fix in predicate in group by clause may cause NPE (#11886)

2022-08-20 Thread yiguolei 
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 25b427d0c6 [Bugfix](inpredicate) fix in predicate in group by clause 
may cause NPE (#11886)
25b427d0c6 is described below

commit 25b427d0c659144f86d1417036393fe6b8887903
Author: Zhengguo Yang 
AuthorDate: Sun Aug 21 10:03:30 2022 +0800

[Bugfix](inpredicate) fix in predicate in group by clause may cause NPE 
(#11886)

* [bug](inpredicate) fix in predicate in group by clause may cause NPE
---
 .../doris/rewrite/RewriteInPredicateRule.java  |  8 +--
 .../data/query_p0/aggregate/aggregate.out  | 13 +++
 .../suites/query_p0/aggregate/aggregate.groovy | 25 ++
 3 files changed, 44 insertions(+), 2 deletions(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/rewrite/RewriteInPredicateRule.java 
b/fe/fe-core/src/main/java/org/apache/doris/rewrite/RewriteInPredicateRule.java
index f6f6b9bc3f..6a1a3a059b 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/rewrite/RewriteInPredicateRule.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/rewrite/RewriteInPredicateRule.java
@@ -55,8 +55,12 @@ public class RewriteInPredicateRule implements 
ExprRewriteRule {
 }
 InPredicate inPredicate = (InPredicate) expr;
 SlotRef slotRef;
-if (inPredicate.contains(Subquery.class) || 
!inPredicate.isLiteralChildren() || inPredicate.isNotIn()
-|| !(inPredicate.getChild(0).unwrapExpr(false) instanceof 
SlotRef)
+// When the select stmt contains group by, we use oriGroupingExprs to 
store the original group by statement
+// and reset it with the rewritten groupingExpr. Therefore, 
origroupingexprs cannot be analyzed.
+// However, in #4197, oriGroupingExprs is rewritten to fix the problem 
of constant fold.
+// The newly added InPredicteRewriteRule requires that expr must be 
analyzed before being rewritten
+if (!inPredicate.isAnalyzed() || inPredicate.contains(Subquery.class) 
|| !inPredicate.isLiteralChildren()
+|| inPredicate.isNotIn() || 
!(inPredicate.getChild(0).unwrapExpr(false) instanceof SlotRef)
 || (slotRef = inPredicate.getChild(0).getSrcSlotRef()) == null 
|| slotRef.getColumn() == null) {
 return expr;
 }
diff --git a/regression-test/data/query_p0/aggregate/aggregate.out 
b/regression-test/data/query_p0/aggregate/aggregate.out
index 8116f6ae15..c23317e7e9 100644
--- a/regression-test/data/query_p0/aggregate/aggregate.out
+++ b/regression-test/data/query_p0/aggregate/aggregate.out
@@ -110,6 +110,19 @@ TESTINGAGAIN
 18000
 26988
 
+-- !aggregate --
+\N other   98.52
+12 other   12.25
+25 other   55.52
+100again   99.8777
+100other   11.6688334
+964again   0.245
+5000   other   67.87
+5252   sample  12.25
+5748   sample  67.87
+6000   other   \N
+8996   other   98.8777
+
 -- !aggregate1 --
 \N \N
 -32767 2147484649
diff --git a/regression-test/suites/query_p0/aggregate/aggregate.groovy 
b/regression-test/suites/query_p0/aggregate/aggregate.groovy
index 61bcd3f4bd..8053479c1c 100644
--- a/regression-test/suites/query_p0/aggregate/aggregate.groovy
+++ b/regression-test/suites/query_p0/aggregate/aggregate.groovy
@@ -110,6 +110,31 @@ suite("aggregate") {
 qt_aggregate """ select variance(c_bigint), variance(distinct c_double) 
from ${tableName}  """
 qt_aggregate """ select 1 k1, 2 k2, c_bigint k3, sum(c_double) from 
${tableName} group by 1, k2, k3 order by k1, k2, k3 """
 qt_aggregate """ select (k1 + k2) * k3 k4 from (select 1 k1, 2 k2, 
c_bigint k3, sum(c_double) from ${tableName} group by 1, k2, k3) t order by k4 
"""
+qt_aggregate """
+SELECT c_bigint,  
+CASE
+WHEN c_string IN ('sample', 'Sample') THEN
+'sample'
+WHEN c_string IN ('Again', 'AGAIN') THEN
+'again'
+ELSE 'other' end, avg(c_double)
+FROM ${tableName}
+GROUP BY  c_bigint,
+CASE
+WHEN c_string IN ('sample', 'Sample') THEN
+'sample'
+WHEN c_string IN ('Again', 'AGAIN') THEN
+'again'
+ELSE 'other'
+END
+ORDER BY  c_bigint,
+CASE
+WHEN c_string IN ('sample', 'Sample') THEN
+'sample'
+WHEN c_string IN ('Again', 'AGAIN') THEN
+'again'
+ELSE 'other' end
+ """
 
 sql "use test_query_db"
 List fields = ["k1", "k2", "k3", "k4", "k5", "k6", "k10", "k11", 
"k7", "k8", "k9"]

[GitHub] [doris] yiguolei merged pull request #11796: [Bug](cast) change binary predicate finally cast to varchar

2022-08-20 Thread GitBox 


yiguolei merged PR #11796:
URL: https://github.com/apache/doris/pull/11796


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [Bug](cast) change binary predicate finally cast to varchar (#11796)

2022-08-20 Thread yiguolei 
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 192cdd4d76 [Bug](cast) change binary predicate finally cast to varchar 
(#11796)
192cdd4d76 is described below

commit 192cdd4d760d9bef7669fd7811ff2153f979ee4f
Author: Pxl 
AuthorDate: Sun Aug 21 10:13:47 2022 +0800

[Bug](cast) change binary predicate finally cast to varchar (#11796)
---
 .../org/apache/doris/analysis/BinaryPredicate.java | 16 +++--
 .../java/org/apache/doris/analysis/CastExpr.java   |  1 -
 .../main/java/org/apache/doris/catalog/Type.java   | 40 ++
 .../type_inference/test_binary_predicate_cast.out  |  6 
 .../test_binary_predicate_cast.groovy  | 36 +++
 5 files changed, 80 insertions(+), 19 deletions(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/analysis/BinaryPredicate.java 
b/fe/fe-core/src/main/java/org/apache/doris/analysis/BinaryPredicate.java
index 154e724f4f..83d4e3cdc4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/BinaryPredicate.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/BinaryPredicate.java
@@ -376,18 +376,12 @@ public class BinaryPredicate extends Predicate implements 
Writable {
 // When int column compares with string, Mysql will convert string to 
int.
 // So it is also compatible with Mysql.
 
-if (t1 == PrimitiveType.BIGINT && (t2 == PrimitiveType.VARCHAR || t2 
== PrimitiveType.STRING)) {
-Expr rightChild = getChild(1);
-Long parsedLong = Type.tryParseToLong(rightChild);
-if (parsedLong != null) {
-return Type.BIGINT;
+if (t1.isStringType() || t2.isStringType()) {
+if ((t1 == PrimitiveType.BIGINT || t1 == PrimitiveType.LARGEINT) 
&& Type.canParseTo(getChild(1), t1)) {
+return Type.fromPrimitiveType(t1);
 }
-}
-if ((t1 == PrimitiveType.VARCHAR || t1 == PrimitiveType.STRING) && t2 
== PrimitiveType.BIGINT) {
-Expr leftChild = getChild(0);
-Long parsedLong = Type.tryParseToLong(leftChild);
-if (parsedLong != null) {
-return Type.BIGINT;
+if ((t2 == PrimitiveType.BIGINT || t2 == PrimitiveType.LARGEINT) 
&& Type.canParseTo(getChild(0), t2)) {
+return Type.fromPrimitiveType(t2);
 }
 }
 
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/CastExpr.java 
b/fe/fe-core/src/main/java/org/apache/doris/analysis/CastExpr.java
index 8b63d461ff..c07a8d8029 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CastExpr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CastExpr.java
@@ -414,7 +414,6 @@ public class CastExpr extends Expr {
 } else if (type.isDecimalV2() || type.isDecimalV3()) {
 return new DecimalLiteral(value.getStringValue());
 } else if (type.isFloatingPointType()) {
-
 return new FloatLiteral(value.getDoubleValue(), type);
 } else if (type.isStringType()) {
 return new StringLiteral(value.getStringValue());
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Type.java 
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Type.java
index 2520c9850c..164f125da4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Type.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Type.java
@@ -18,7 +18,9 @@
 package org.apache.doris.catalog;
 
 import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.LargeIntLiteral;
 import org.apache.doris.analysis.StringLiteral;
+import org.apache.doris.common.AnalysisException;
 import org.apache.doris.common.DdlException;
 import org.apache.doris.common.Pair;
 import org.apache.doris.thrift.TColumnType;
@@ -545,15 +547,39 @@ public abstract class Type {
 }
 
 /**
- * Returns null if this expr is not instance of StringLiteral or 
StringLiteral
- * inner value could not parse to long. otherwise return parsed Long 
result.
+ * Returns true if expr is StringLiteral and can parse to valid type, false
+ * otherwise.
+ * This function only support LargeInt and BigInt now.
  */
-public static Long tryParseToLong(Expr expectStringExpr) {
-if (expectStringExpr instanceof StringLiteral) {
-String value = ((StringLiteral) expectStringExpr).getValue();
-return Longs.tryParse(value);
+public static boolean canParseTo(Expr expr, PrimitiveType type) {
+if (expr instanceof StringLiteral) {
+if (type == PrimitiveType.BIGINT) {
+return canParseToBigInt((StringLiteral) expr);
+} else if (type == PrimitiveType.LARGEINT) {
+return canParseToLargeInt((StringLiteral) expr);
+}
 }
-

[GitHub] [doris] github-actions[bot] commented on pull request #11796: [Bug](cast) change binary predicate finally cast to varchar

2022-08-20 Thread GitBox 


github-actions[bot] commented on PR #11796:
URL: https://github.com/apache/doris/pull/11796#issuecomment-1221451418

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] xinyiZzz opened a new pull request, #11943: [dev-1.1.2](cherry-pick) Optimize readability of mem exceed limit error message

2022-08-20 Thread GitBox 


xinyiZzz opened a new pull request, #11943:
URL: https://github.com/apache/doris/pull/11943

   # Proposed changes
   
   Issue Number: close #11876
   
   ## Problem summary
   
   cherry-pick #11877
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Lchangliang commented on pull request #11869: [bugfix](light weight schema change) support delete condition in schema change

2022-08-20 Thread GitBox 


Lchangliang commented on PR #11869:
URL: https://github.com/apache/doris/pull/11869#issuecomment-1221461684

   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] xinyiZzz opened a new pull request, #11944: [dev-1.1.2](memory) Default `STRICT_MEMORY_USE` reduce hash table memory

2022-08-20 Thread GitBox 


xinyiZzz opened a new pull request, #11944:
URL: https://github.com/apache/doris/pull/11944

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Vectorized query ssb 3.1, the memory is mainly composed of 4 parts,
   
   1. The memory ratio of each part in the first query: scanner_thread (57%), 
hash_table_build_thread (32%), open + get_next (8.6%), prepare (5.4%)
   
   2. The memory ratio of each part in the second query: scanner_thread (12%), 
hash_table_build_thread (61.9%), open + get_next (15.3%), prepare (10.8%)
   
   3. Change the hash table to 75% fill, which can reduce memory usage by 30%.
   
     | Growing physical memory peak | query tracker peak | scanner_threadpeak | 
_hash_table_build_thread peak | _exec_actual peak | 
PlanFragmentExecutor::prepare peak
   -- | -- | -- | -- | -- | -- | --
   first run | 10702M | 2406M | 1368M | 770M | 207M | 132M
   second run | 1291M | 1218M(少70?) | 147M | 754M | 187M | 132M
     |   |   |   |   |   |  
   75% max_fillfirst run | 10721M | 2046M | 1364M | 406M | 207M | 132M
   75% max_fillsecond run | 924M | 856M(少70?) | 141M | 405M | 190M | 132M
   
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   5. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   6. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] jackwener opened a new pull request, #11945: [improve](*): use `Pair.of` to replace `new Pair<>()`

2022-08-20 Thread GitBox 


jackwener opened a new pull request, #11945:
URL: https://github.com/apache/doris/pull/11945

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   use `Pair.of` to replace `new Pair<>()`
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [x] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [x] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [x] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [x] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [x] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] jackwener opened a new pull request, #11946: [improve](nereids): remove FakeJoin.java

2022-08-20 Thread GitBox 


jackwener opened a new pull request, #11946:
URL: https://github.com/apache/doris/pull/11946

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   remove FakeJoin.java
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [x] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [x] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [x] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [x] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [x] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] xinyiZzz opened a new pull request, #11947: [fix](memtracker) Improve memory tracking accuracy for exec nodes

2022-08-20 Thread GitBox 


xinyiZzz opened a new pull request, #11947:
URL: https://github.com/apache/doris/pull/11947

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   1. fix missing exec node mem tracker consumer.
   2. print real process physical memory when exceeding mem limit
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   3. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   5. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   6. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org