Re:Re:[Discuss][DSIP] Support single replica load for stream load

2022-05-29 Thread weizuo
I have the write permission of DSIP WIKI, and I will create a new DSIP for this feature. Zuo Wei At 2022-05-30 12:46:55, "陈明雨" wrote: >Good to see that! >As you already have the DSIP WIKI write permission, you can create a new DSIP >for this and show us some detail design about it. >

Re: Re:Re: Re: [Discuss][DSIP] Support Multi Catalog

2022-05-29 Thread 41108453
nice -- Original -- From: 陈明雨 https://cwiki.apache.org/confluence/display/DORIS/DSIP-014%3A+Multi+Catalog+Support >> >> >> >> [2] https://github.com/apache/incubator-doris/pull/9812 >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> 此致!Best Regards >> >> 陈明雨 Mingyu Ch

Re:Re: Re: [Discuss][DSIP] Support Multi Catalog

2022-05-29 Thread 陈明雨
Actually, in my design, after user creating an external datasource, he can access the external databases and tables directly, and no need to create the metadata mapping manuelly. -- 此致!Best Regards 陈明雨 Mingyu Chen Email: chenmin...@apache.org 在 2022-05-30 12:52:49,"张家峰" 写道: >how to cr

Re: Re: [Discuss][DSIP] Support Multi Catalog

2022-05-29 Thread 张家峰
how to create an exterior: When using external tables in this management method, it is best not to manually create the mapping of each table, so that users can use it without perception, especially the field mapping of the external table, which can be simplified to the point where the user o

Re:[Discuss][DSIP] Support single replica load for stream load

2022-05-29 Thread 陈明雨
Good to see that! As you already have the DSIP WIKI write permission, you can create a new DSIP for this and show us some detail design about it. -- 此致!Best Regards 陈明雨 Mingyu Chen Email: chenmin...@apache.org At 2022-05-30 12:37:01, "weizuo" wrote: >Hi all, >During load process, the s

Re:[Discuss][DSIP] Support JSON data type

2022-05-29 Thread 陈明雨
Hi Kang, JSON type is indeed a significant feature for semi-structured data analysis. Do you have any detail design about this feature, If you are willing to implement this feature, I would like to create a DSIP for you, and you can show us the details about it. -- 此致!Best Regards 陈明雨 Mingyu

Re:Re: [Discuss][DSIP] Support Multi Catalog

2022-05-29 Thread 陈明雨
1. Permission In my design, the permission system of the external data source is decoupled from Doris' own permission system. First, when creating an external data source, the user will specify an account of the external data source to connect to (named ”proxy account”), and the permissions of

[Discuss][DSIP] Support single replica load for stream load

2022-05-29 Thread weizuo
Hi all, During load process, the same operation are performed on all replicas such as sort and aggregation which are resource-intensive. Concurrent data load would consume much CPU and memory resources. I plan to support the capacity of single replica load to reduce the usage of CPU and Memory f

Re: [Discuss][DSIP] Support Multi Catalog

2022-05-29 Thread ling miao
Regarding permissions, there is currently a difference between Doris's permission system and the external permission system. For external data sources, still use Doris's permissions, or use the permissions of external data sources? As far as the current architecture is concerned, Doris is still a

[Discuss][DSIP] Support JSON data type

2022-05-29 Thread Kang Xiao
Hi All, I'd like to propose the support for JSON data type. Semi structured data is very common in data analysis and the most popular format is JSON. Several database systems support JSON data type, usually binary format, for efficient storage and query processing. JSON binary data type will be

??????Flink mysql cdc to doris??schema join ????????doris????????

2022-05-29 Thread di wu
Hi, thanks for your question. 1. I tested it, and flink cdc imports the doris aggregation model. In the case of join and groupby, data can be written and updated normally. 2. The flink-doris-connector version 1.0.3 is based on batch-based streamload, which is controlled by parameters such as sink