Here is the design draft, and I has created dsip in doris wiki
https://cwiki.apache.org/confluence/display/DORIS/DSIP-005%3A+Support+Random+Sink
feel free to discuss with it
and some releated implementation work PR see #8041 #8259
At 2022-02-22 13:38:23, "陈明雨" wrote:
>If you want to implement ver
Congrats!
-- --
??:
"dev"
https://github.com/
This does lower the threshold for users, but at the same time will this bring
additional load to doris and what impact will it have
If this is the case are we supporting the delete operation in one piece
-- Original --
From: Xu,Yang(INF) https://cwiki.apache.
Yes. That is my thought too. The best case is to integrate the existing binlog
load into the routine load.
But we need to find a way to parse binlog in be.
在 2022/3/30 上午11:49，“蔡聪辉” 写入:
I think one of the elegant implementations may be something like routine
load
在 2022-03-30 10:47:39
Congrats!
Guolei helps me a lot to participate in Doris community and thanks so much!
Best,
Gabriel
On Wed, 30 Mar 2022 at 13:06, 陈明雨 wrote:
> Hi all,
> After being nominated and voted on by the Doris PPMC, the Doris community
> is honored to invite 4 new Committers and 1 new PPMC member:
>
>
>
Welcome ~
陈明雨 于2022年3月30日周三 13:06写道：
> Hi all,
> After being nominated and voted on by the Doris PPMC, the Doris community
> is honored to invite 4 new Committers and 1 new PPMC member:
>
>
> - New PPMC member: Zuo Wei (https://github.com/weizuo93)
> - New Committer: Guolei Yi(https://github.com
Hi all,
After being nominated and voted on by the Doris PPMC, the Doris community is
honored to invite 4 new Committers and 1 new PPMC member:
- New PPMC member: Zuo Wei (https://github.com/weizuo93)
- New Committer: Guolei Yi(https://github.com/yiguolei)
- New Committer: Huajian Lan(https://git
+1
Looking forward Teacher Guolei's dsip.
GuoLei Yi 于2022年3月29日周二 14:17写道：
> Currently, there are various interfaces for file IO operations in Doris:
>
>- There are FileReader and FileWriter in the query layer. There are
>corresponding implementations for HDFS, S3, Broker, and Local.
>
I think one of the elegant implementations may be something like routine load
在 2022-03-30 10:47:39，"ling miao" 写道：
>> Do you mean 'canal' component?
>Yes.
>
>It sounds like you mean a new way about binlog load?
>You can show the architecture diagram and general design on the wiki, so
>that I can
> Do you mean 'canal' component?
Yes.
It sounds like you mean a new way about binlog load?
You can show the architecture diagram and general design on the wiki, so
that I can better understand your intentions.[1]
Ling Miao
[
https://cwiki.apache.org/confluence/display/DORIS/DSIP-003%3A+Directly+b
Thanks for your advice. I will follow your instructions, and replace the
usage step by step.
陈明雨 于2022年3月29日周二 22:55写道：
> Indeed, we need to refactor the IO layer to make it more clear and
> extensible.
> The basic purpose is that when a new kind of file system is introduced, we
> only need to i
Hi hui,
I created DSIP-003 for you:
https://cwiki.apache.org/confluence/display/DORIS/DSIP-003%3A+Directly+binlog+consumption
And I have some questions:
1.
Using debezium can indeed simplify the analysis difficulty of binglog, but the
disadvantage is that it occupies the computing resources of
Indeed, we need to refactor the IO layer to make it more clear and extensible.
The basic purpose is that when a new kind of file system is introduced, we only
need to implement a new derived class
for it and no need to modify any other interface in upper layer.
BTW, for now, if we change the IO
Hi Yang,
I have not found any library we can use. Maybe it's a lot of work to put it on
BE.
I think it's not necessary to put it in BE, Because we can distribute the sync
job into
other FE instead of Master, and user can use Flink Connector if there is still
too much data.
Hi Ling,
Do you mean 'canal' component? I have not cancelled any component,
only extended current binlog load with a new type of 'debezium'.
Because the Canal service is not provided in my company's product environment,
users can't use the current binlog load. I think the new type simplifies th
Hi hui,
Thanks for the pr.
Is there any way to integrate the binlog aquire and parse function into Be,
instead of Fe?
在 2022/3/29 下午8:13，“ling miao” 写入:
Hi hui,
I want to know what is the reason for canceling the cancel component? Is
there any problem in use? What are the benefits
Hi All,
Since the last discussion, we have done the following based on suggestions.
1. Split flink-connector, spark-connector, and doris-mananger from the Doris
repo into three separate repo.
- https://github.com/apache/incubator-doris-manager
- https://github.com/apache/incubator-dor
Hi hui,
I want to know what is the reason for canceling the cancel component? Is
there any problem in use? What are the benefits ?
Parsing binlog is not actually the function of Doris itself. Or is it
better to appear as a connector? For example, like Flink connector.
Ling Miao
陈明雨 于2022年3月29日
Hi,
I have registered my account, and my cwiki account is `tianhui`. Thanks!
--
Hui Tian
-- --
??:
"dev"
Hi Hui,
Thanks for it. Could you register a cwiki acount and tell me your account id?
So that I can create DSIP and grant write priv for you.
https://cwiki.apache.org/confluence/signup.action
--
此致！Best Regards
陈明雨 Mingyu Chen
Email:
chenmin...@apache.org
At 2022-03-29 18:17:02, "晖" <82
As issue
#8025(https://github.com/apache/incubator-doris/issues/8025) mentioned,
the current binlog load is relying on Canal service. We can integrate the
binlog aquire and parse function into Doris, so that user can be more
convinience to use it.Please determine the final DSIP number for it.
21 matches
Mail list logo
