[incubator-doris] branch master updated: [Optimize] Add an unordered_map for TabletSchema to speed up column name lookup (#4779)

2020-11-03 Thread morningman
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git The following commit(s) were added to refs/heads/master by this push: new d1c2b3e [Optimize] Add an unordere

[incubator-doris] branch master updated: [LoadBalance] make BeLoadRebalancer extends from base class Rebalancer (#4771)

2020-11-03 Thread kangkaisen
This is an automated email from the ASF dual-hosted git repository. kangkaisen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git The following commit(s) were added to refs/heads/master by this push: new 80d5f6e [LoadBalance] make BeLoadR

[GitHub] [incubator-doris] Karl-WangSK opened a new issue #4838: How to change line break when stream load

2020-11-03 Thread GitBox
Karl-WangSK opened a new issue #4838: URL: https://github.com/apache/incubator-doris/issues/4838 The default line break is \n. if my data contains \n . it will separate as 2 records. and end up columns number doesn't match error . -

[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #3025: Restructure storage type to support complex types expending

2020-11-03 Thread GitBox
kangkaisen commented on a change in pull request #3025: URL: https://github.com/apache/incubator-doris/pull/3025#discussion_r516385798 ## File path: be/src/olap/collection.h ## @@ -0,0 +1,50 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contribut

[GitHub] [incubator-doris] morningman opened a new issue #4828: [Bug] Compaction cause BE crash

2020-11-03 Thread GitBox
morningman opened a new issue #4828: URL: https://github.com/apache/incubator-doris/issues/4828 **Describe the bug** ![image](https://user-images.githubusercontent.com/2899462/97833978-a54f0180-1d11-11eb-8b07-2edda8459833.png) ---

[GitHub] [incubator-doris] imay commented on issue #4838: How to change line break when stream load

2020-11-03 Thread GitBox
imay commented on issue #4838: URL: https://github.com/apache/incubator-doris/issues/4838#issuecomment-720992443 Not supported now This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [incubator-doris] ccoffline opened a new issue #4833: [Bug] Unexpected exception: org.apache.doris.analysis.CastExpr cannot be cast to org.apache.doris.analysis.LiteralExpr

2020-11-03 Thread GitBox
ccoffline opened a new issue #4833: URL: https://github.com/apache/incubator-doris/issues/4833 **Describe the bug** ``` ERROR 1064 (HY000): errCode = 2, detailMessage = Unexpected exception: org.apache.doris.analysis.CastExpr cannot be cast to org.apache.doris.analysis.LiteralExpr

[GitHub] [incubator-doris] weizuo93 opened a new issue #4834: [Proposal] Take 'tablet scan frequency' into consideration when selecting a tablet for compaction

2020-11-03 Thread GitBox
weizuo93 opened a new issue #4834: URL: https://github.com/apache/incubator-doris/issues/4834 A large number of small segment files will lead to low efficiency for scan operations. Multiple small files can be merged into a large file by compaction operation. So we could take the tablet sc

[GitHub] [incubator-doris] ccoffline opened a new pull request #4836: [Bug] Fix ClassCastException (#4833)

2020-11-03 Thread GitBox
ccoffline opened a new pull request #4836: URL: https://github.com/apache/incubator-doris/pull/4836 ## Proposed changes fix to #4833 ## Types of changes - [x] Bugfix (non-breaking change which fixes an issue) - [] New feature (non-breaking change which adds functiona

[GitHub] [incubator-doris] EmmyMiao87 commented on issue #4795: [BUG] retry submit exception is not catched

2020-11-03 Thread GitBox
EmmyMiao87 commented on issue #4795: URL: https://github.com/apache/incubator-doris/issues/4795#issuecomment-721088608 Another problem is that I think loading should not have so much concurrency control. In fact, db itself already has a control of the number of transactions. So the

[GitHub] [incubator-doris] xinyiZzz edited a comment on pull request #4825: [Feature] Running Profile OLAP_SCAN_NODE layering and enhance readability

2020-11-03 Thread GitBox
xinyiZzz edited a comment on pull request #4825: URL: https://github.com/apache/incubator-doris/pull/4825#issuecomment-720394209 @HappenLee Good question, I have moved `_segment_profile` and `scanner_profile` to `OlapScanNode` and modified the document. ---

[GitHub] [incubator-doris] kangkaisen closed issue #4763: [Proposal] refactor the LoadBalancer to support more rebalance strategies

2020-11-03 Thread GitBox
kangkaisen closed issue #4763: URL: https://github.com/apache/incubator-doris/issues/4763 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [incubator-doris] HappenLee commented on issue #4835: [Proposal] Disable v1 storage format when creating a new table

2020-11-03 Thread GitBox
HappenLee commented on issue #4835: URL: https://github.com/apache/incubator-doris/issues/4835#issuecomment-720952500 good idea~ This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [incubator-doris] morningman commented on a change in pull request #4837: [Optimize] Take 'tablet scan frequency' into consideration when selecting a tablet for compaction

2020-11-03 Thread GitBox
morningman commented on a change in pull request #4837: URL: https://github.com/apache/incubator-doris/pull/4837#discussion_r516629748 ## File path: be/src/olap/tablet.h ## @@ -301,6 +303,10 @@ class Tablet : public BaseTablet { // cumulative compaction policy std::un

[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #4771: [LoadBalance] make BeLoadRebalancer extends from base class Rebalancer

2020-11-03 Thread GitBox
kangkaisen commented on a change in pull request #4771: URL: https://github.com/apache/incubator-doris/pull/4771#discussion_r515754212 ## File path: fe/fe-core/src/main/java/org/apache/doris/clone/Rebalancer.java ## @@ -0,0 +1,96 @@ +// Licensed to the Apache Software Foundatio

[GitHub] [incubator-doris] imay commented on issue #4832: ErrorReason{code=errCode = 104, msg='be 10003 abort task with reason: Broker: Offset out of range'}

2020-11-03 Thread GitBox
imay commented on issue #4832: URL: https://github.com/apache/incubator-doris/issues/4832#issuecomment-720845365 是不是Kafka中已经没有这么多的数据了? This is an automated message from the Apache Git Service. To respond to the message, plea

[GitHub] [incubator-doris] demon-gu opened a new issue #4832: ErrorReason{code=errCode = 104, msg='be 10003 abort task with reason: Broker: Offset out of range'}

2020-11-03 Thread GitBox
demon-gu opened a new issue #4832: URL: https://github.com/apache/incubator-doris/issues/4832 从Kafka中拉取数据到Doris出现这个问题ErrorReason{code=errCode = 104, msg='be 10003 abort task with reason: Broker: Offset out of range'} ,建表语句定义的offset是"kafka_offsets"="0,0,0,0,0,0,0,0,0",请问这是什么原因呀? -

[GitHub] [incubator-doris] vagetablechicken commented on a change in pull request #4771: [LoadBalance] make BeLoadRebalancer extends from base class Rebalancer

2020-11-03 Thread GitBox
vagetablechicken commented on a change in pull request #4771: URL: https://github.com/apache/incubator-doris/pull/4771#discussion_r515766101 ## File path: fe/fe-core/src/main/java/org/apache/doris/clone/Rebalancer.java ## @@ -0,0 +1,96 @@ +// Licensed to the Apache Software Fou

[GitHub] [incubator-doris] px-l opened a new pull request #4830: [FE UI] Fix some bugs about new FE UI

2020-11-03 Thread GitBox
px-l opened a new pull request #4830: URL: https://github.com/apache/incubator-doris/pull/4830 1. Add a search boxer in the left tree view of Playground. 2. Fix some visual bugs of UI. 3. Fix bugs that link failed in QueryProfile view. 4. Fix bugs that cookie is always invalid. 5

[GitHub] [incubator-doris] EmmyMiao87 opened a new pull request #4831: Add other license declare in NOTICE

2020-11-03 Thread GitBox
EmmyMiao87 opened a new pull request #4831: URL: https://github.com/apache/incubator-doris/pull/4831 ## Proposed changes The copyright of some files is not apache, these files will be declared in the NOTICE file Some files that forgot to bring the apache copyright will also be ad

[GitHub] [incubator-doris] morningman opened a new pull request #4840: [Bug] Fix bug that failed to create view with complex select stmt

2020-11-03 Thread GitBox
morningman opened a new pull request #4840: URL: https://github.com/apache/incubator-doris/pull/4840 ## Proposed changes Fix bug that failed to create view with complex select stmt ## Types of changes - [x] Bugfix (non-breaking change which fixes an issue) ## Chec

[GitHub] [incubator-doris] kangkaisen merged pull request #4771: [LoadBalance] make BeLoadRebalancer extends from base class Rebalancer

2020-11-03 Thread GitBox
kangkaisen merged pull request #4771: URL: https://github.com/apache/incubator-doris/pull/4771 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[GitHub] [incubator-doris] HappenLee commented on a change in pull request #4825: [Feature] Running Profile OLAP_SCAN_NODE layering and enhance readability

2020-11-03 Thread GitBox
HappenLee commented on a change in pull request #4825: URL: https://github.com/apache/incubator-doris/pull/4825#discussion_r515754966 ## File path: be/src/exec/scan_node.h ## @@ -96,49 +96,29 @@ class ScanNode : public ExecNode { RuntimeProfile::Counter* rows_read_counter(

[GitHub] [incubator-doris] EmmyMiao87 removed a comment on issue #4795: [BUG] retry submit exception is not catched

2020-11-03 Thread GitBox
EmmyMiao87 removed a comment on issue #4795: URL: https://github.com/apache/incubator-doris/issues/4795#issuecomment-721088608 Another problem is that I think loading should not have so much concurrency control. In fact, db itself already has a control of the number of transactions.

[GitHub] [incubator-doris] morningman opened a new issue #4839: [Bug] Create view failed

2020-11-03 Thread GitBox
morningman opened a new issue #4839: URL: https://github.com/apache/incubator-doris/issues/4839 **Describe the bug** Create view failed with error: "failed to init view stmt" **To Reproduce** create a view with some complex select stmt such as union and join and group by.

[GitHub] [incubator-doris] morningman commented on pull request #4781: [Optimize] Improve LRU cache's performance

2020-11-03 Thread GitBox
morningman commented on pull request #4781: URL: https://github.com/apache/incubator-doris/pull/4781#issuecomment-721076428 Could you rebase the master code to see if unit test can run normally? This is an automated message f

[GitHub] [incubator-doris] acelyc111 commented on pull request #4781: [Optimize] Improve LRU cache's performance

2020-11-03 Thread GitBox
acelyc111 commented on pull request #4781: URL: https://github.com/apache/incubator-doris/pull/4781#issuecomment-721092805 > Could you rebase the master code to see if unit test can run normally? Done This is an automa

[GitHub] [incubator-doris] yangzhg opened a new issue #4835: [Proposal] Disable v1 storage format when creating a new table

2020-11-03 Thread GitBox
yangzhg opened a new issue #4835: URL: https://github.com/apache/incubator-doris/issues/4835 At present, our V2 storage format has been developed for a long time, and V2 has many advantages that V1 does not have. The coexistence of V1 and V2 brings a lot of code and cluster maintenance cos

[GitHub] [incubator-doris] weizuo93 opened a new pull request #4837: [Optimize] Take 'tablet scan frequency' into consideration when selecting a tablet for compaction

2020-11-03 Thread GitBox
weizuo93 opened a new pull request #4837: URL: https://github.com/apache/incubator-doris/pull/4837 ## Proposed changes A large number of small segment files will lead to low efficiency for scan operations. Multiple small files can be merged into a large file by compaction operation

[GitHub] [incubator-doris] hf200012 commented on issue #4835: [Proposal] Disable v1 storage format when creating a new table

2020-11-03 Thread GitBox
hf200012 commented on issue #4835: URL: https://github.com/apache/incubator-doris/issues/4835#issuecomment-720955226 good idea This is an automated message from the Apache Git Service. To respond to the message, please log o

[GitHub] [incubator-doris] morningman merged pull request #4779: [Optimize] Add an unordered_map for TabletSchema to speed up column name lookup

2020-11-03 Thread GitBox
morningman merged pull request #4779: URL: https://github.com/apache/incubator-doris/pull/4779 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[GitHub] [incubator-doris] xinyiZzz commented on pull request #4825: [Feature] Running Profile OLAP_SCAN_NODE layering and enhance readability

2020-11-03 Thread GitBox
xinyiZzz commented on pull request #4825: URL: https://github.com/apache/incubator-doris/pull/4825#issuecomment-720394209 @HappenLee Good question, I have moved `_segment_profile` and `scanner_profile` to ʻOlapScanNode` and modified the document. --

[GitHub] [incubator-doris] morningman opened a new pull request #4829: [Compaction][Buf] Fix bug that meta lock need to be held when calculating compaction score

2020-11-03 Thread GitBox
morningman opened a new pull request #4829: URL: https://github.com/apache/incubator-doris/pull/4829 ## Proposed changes We need to hold the tablet's meta lock when calculating the compaction score ## Types of changes - [x] Bugfix (non-breaking change which fixes an issu

[GitHub] [incubator-doris] acelyc111 commented on issue #4499: BE coredump in StorageEngine::_unused_rowset_monitor_thread_callback

2020-11-03 Thread GitBox
acelyc111 commented on issue #4499: URL: https://github.com/apache/incubator-doris/issues/4499#issuecomment-721071655 A similar coredump: ``` (gdb) f 0 #0 0x0133d395 in Delete (arena=0x0, value=) at /var/local/thirdparty/installed/include/google/protobuf/repeated_field.h:

[GitHub] [incubator-doris] xinghuayu007 closed pull request #4809: 【BUG】Catch runtime exception

2020-11-03 Thread GitBox
xinghuayu007 closed pull request #4809: URL: https://github.com/apache/incubator-doris/pull/4809 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

[GitHub] [incubator-doris] liutang123 commented on a change in pull request #3025: Restructure storage type to support complex types expending

2020-11-03 Thread GitBox
liutang123 commented on a change in pull request #3025: URL: https://github.com/apache/incubator-doris/pull/3025#discussion_r516041649 ## File path: be/src/olap/collection.h ## @@ -0,0 +1,50 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contribut

[GitHub] [incubator-doris] yangzhg closed issue #4817: [Bug] Sequence column is not visible when show_hidden_columns = true

2020-11-03 Thread GitBox
yangzhg closed issue #4817: URL: https://github.com/apache/incubator-doris/issues/4817 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [incubator-doris] zhaojintaozhao commented on issue #4835: [Proposal] Disable v1 storage format when creating a new table

2020-11-03 Thread GitBox
zhaojintaozhao commented on issue #4835: URL: https://github.com/apache/incubator-doris/issues/4835#issuecomment-721121745 Currently, many companies are using the v1 storage format. The v2 format is not stable yet. The new Doris version must be compatible with earlier versions. Therefore,

[GitHub] [incubator-doris] zhaojintaozhao edited a comment on issue #4835: [Proposal] Disable v1 storage format when creating a new table

2020-11-03 Thread GitBox
zhaojintaozhao edited a comment on issue #4835: URL: https://github.com/apache/incubator-doris/issues/4835#issuecomment-721121745 Currently, there are many companies also using the v1 storage format. The v2 format is not stable yet. The new Doris version must be compatible with earlier ve

[GitHub] [incubator-doris] morningman opened a new issue #4841: [Bug][SQL] query failed with Union and Colocate Join

2020-11-03 Thread GitBox
morningman opened a new issue #4841: URL: https://github.com/apache/incubator-doris/issues/4841 **Describe the bug** Query failed with error like: `failed to get tablet. tablet_id=26846, with schema_hash=398972982, reason=tablet does not exist` SQL looks like: `select a

[GitHub] [incubator-doris] morningman opened a new pull request #4842: [Bug][SQL] Fix bug that query failed when SQL contains Union and Colocate join

2020-11-03 Thread GitBox
morningman opened a new pull request #4842: URL: https://github.com/apache/incubator-doris/pull/4842 ## Proposed changes SQL like: `select a join b union select c join d`; if both a/b and c/d are colocate join, the query may failed with error like: `failed to get tabl

[GitHub] [incubator-doris] blueChild commented on issue #4835: [Proposal] Disable v1 storage format when creating a new table

2020-11-03 Thread GitBox
blueChild commented on issue #4835: URL: https://github.com/apache/incubator-doris/issues/4835#issuecomment-721461833 Would it be better if we should allow users to choose which version to use instead of disabling it? T

[GitHub] [incubator-doris] zhaojintaozhao commented on issue #4835: [Proposal] Disable v1 storage format when creating a new table

2020-11-03 Thread GitBox
zhaojintaozhao commented on issue #4835: URL: https://github.com/apache/incubator-doris/issues/4835#issuecomment-721465184 The right to choose the storage format v1 or v2 should be delegated to the user. This is an automat

[GitHub] [incubator-doris] morningman commented on issue #4835: [Proposal] Disable v1 storage format when creating a new table

2020-11-03 Thread GitBox
morningman commented on issue #4835: URL: https://github.com/apache/incubator-doris/issues/4835#issuecomment-721467250 Hi @zhaojintaozhao @blueChild SegmentV2 is implemented since v0.12, and has been online for a long time. In v0.13, the default storage format has already been change

[GitHub] [incubator-doris] HappenLee commented on a change in pull request #4842: [Bug][SQL] Fix bug that query failed when SQL contains Union and Colocate join

2020-11-03 Thread GitBox
HappenLee commented on a change in pull request #4842: URL: https://github.com/apache/incubator-doris/pull/4842#discussion_r517127793 ## File path: fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java ## @@ -1140,17 +1141,26 @@ private long getScanRangeLength(final TSc

[GitHub] [incubator-doris] demon-gu commented on issue #4832: ErrorReason{code=errCode = 104, msg='be 10003 abort task with reason: Broker: Offset out of range'}

2020-11-03 Thread GitBox
demon-gu commented on issue #4832: URL: https://github.com/apache/incubator-doris/issues/4832#issuecomment-721552450 > 是不是Kafka中已经没有这么多的数据了? 我这边解决了,设置一下property.kafka_default_offsets为OFFSET_END或者OFFSET_BEGING就可以了 This

[GitHub] [incubator-doris] EmmyMiao87 closed issue #4832: ErrorReason{code=errCode = 104, msg='be 10003 abort task with reason: Broker: Offset out of range'}

2020-11-03 Thread GitBox
EmmyMiao87 closed issue #4832: URL: https://github.com/apache/incubator-doris/issues/4832 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [incubator-doris] acelyc111 commented on a change in pull request #4837: [Optimize] Take 'tablet scan frequency' into consideration when selecting a tablet for compaction

2020-11-03 Thread GitBox
acelyc111 commented on a change in pull request #4837: URL: https://github.com/apache/incubator-doris/pull/4837#discussion_r517147556 ## File path: be/src/common/config.h ## @@ -329,6 +329,13 @@ namespace config { CONF_mInt32(base_compaction_trace_threshold, "10"); CO

[GitHub] [incubator-doris] acelyc111 commented on a change in pull request #4837: [Optimize] Take 'tablet scan frequency' into consideration when selecting a tablet for compaction

2020-11-03 Thread GitBox
acelyc111 commented on a change in pull request #4837: URL: https://github.com/apache/incubator-doris/pull/4837#discussion_r517147556 ## File path: be/src/common/config.h ## @@ -329,6 +329,13 @@ namespace config { CONF_mInt32(base_compaction_trace_threshold, "10"); CO

[GitHub] [incubator-doris] acelyc111 commented on a change in pull request #4837: [Optimize] Take 'tablet scan frequency' into consideration when selecting a tablet for compaction

2020-11-03 Thread GitBox
acelyc111 commented on a change in pull request #4837: URL: https://github.com/apache/incubator-doris/pull/4837#discussion_r517147556 ## File path: be/src/common/config.h ## @@ -329,6 +329,13 @@ namespace config { CONF_mInt32(base_compaction_trace_threshold, "10"); CO