Re: [Discuss][DSIP] Support load binlog from MySQL directly without relying other service

41108453 Tue, 29 Mar 2022 22:32:26 -0700

This does lower the threshold for users, but at the same time will this bring 
additional load to doris and what impact will it have



If this is the case are we supporting the delete operation in one piece





------------------ Original ------------------
From: Xu,Yang(INF) <xuyan...@baidu.com&gt;
Date: Wed,Mar 30,2022 1:28 PM
To: dev@doris.apache.org <dev@doris.apache.org&gt;
Subject: Re: [Discuss][DSIP] Support load binlog from MySQL directly without 
relying other service



Yes. That is my thought too. The best case is to integrate the existing binlog 
load into the routine load.
But we need to find a way to parse binlog in be.

在 2022/3/30 上午11:49，“蔡聪辉”<caiconghui2...@163.com&gt; 写入:

&nbsp;&nbsp;&nbsp; I think one of the elegant implementations may be something 
like routine load

&nbsp;&nbsp;&nbsp; 在 2022-03-30 10:47:39，"ling miao" <lingm...@apache.org&gt; 
写道：
&nbsp;&nbsp;&nbsp; &gt;&gt; Do you mean 'canal' component?
&nbsp;&nbsp;&nbsp; &gt;Yes.
&nbsp;&nbsp;&nbsp; &gt;
&nbsp;&nbsp;&nbsp; &gt;It sounds like you mean a new way about binlog load?
&nbsp;&nbsp;&nbsp; &gt;You can show the architecture diagram and general design 
on the wiki, so
&nbsp;&nbsp;&nbsp; &gt;that I can better understand your intentions.[1]
&nbsp;&nbsp;&nbsp; &gt;
&nbsp;&nbsp;&nbsp; &gt;Ling Miao
&nbsp;&nbsp;&nbsp; &gt;[
&nbsp;&nbsp;&nbsp; 
&gt;https://cwiki.apache.org/confluence/display/DORIS/DSIP-003%3A+Directly+binlog+consumption
&nbsp;&nbsp;&nbsp; &gt;]
&nbsp;&nbsp;&nbsp; &gt;
&nbsp;&nbsp;&nbsp; &gt;晖 <827677...@qq.com.invalid&gt; 于2022年3月29日周二 20:56写道：
&nbsp;&nbsp;&nbsp; &gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; Hi Ling,&amp;nbsp;
&nbsp;&nbsp;&nbsp; &gt;&gt; Do you mean 'canal' component? I have not cancelled 
any component,&amp;nbsp;
&nbsp;&nbsp;&nbsp; &gt;&gt; only extended current binlog load with a new type 
of 'debezium'.
&nbsp;&nbsp;&nbsp; &gt;&gt;
&nbsp;&nbsp;&nbsp; &gt;&gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; Because the Canal service is not provided in my 
company's product
&nbsp;&nbsp;&nbsp; &gt;&gt; environment,
&nbsp;&nbsp;&nbsp; &gt;&gt; users can't use the current binlog load. I think 
the new type simplifies
&nbsp;&nbsp;&nbsp; &gt;&gt; the usage
&nbsp;&nbsp;&nbsp; &gt;&gt; of binlog load, and shortens the data flow.
&nbsp;&nbsp;&nbsp; &gt;&gt;
&nbsp;&nbsp;&nbsp; &gt;&gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; The similiar function is provided by ClickHouse. I 
think it's a brilliant
&nbsp;&nbsp;&nbsp; &gt;&gt; experience to
&nbsp;&nbsp;&nbsp; &gt;&gt; use a MPP database as a MySQL's slave. It's not the 
same background with
&nbsp;&nbsp;&nbsp; &gt;&gt; Flink&amp;nbsp;
&nbsp;&nbsp;&nbsp; &gt;&gt; Connector.
&nbsp;&nbsp;&nbsp; &gt;&gt;
&nbsp;&nbsp;&nbsp; &gt;&gt;
&nbsp;&nbsp;&nbsp; &gt;&gt;
&nbsp;&nbsp;&nbsp; &gt;&gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; 
------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
&nbsp;&nbsp;&nbsp; &gt;&gt; 发件人:
&nbsp;&nbsp;&nbsp; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 "dev"
&nbsp;&nbsp;&nbsp; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 <
&nbsp;&nbsp;&nbsp; &gt;&gt; lingm...@apache.org&amp;gt;;
&nbsp;&nbsp;&nbsp; &gt;&gt; 发送时间:&amp;nbsp;2022年3月29日(星期二) 晚上8:13
&nbsp;&nbsp;&nbsp; &gt;&gt; 收件人:&amp;nbsp;"dev"<dev@doris.apache.org&amp;gt;;
&nbsp;&nbsp;&nbsp; &gt;&gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; 主题:&amp;nbsp;Re: [Discuss][DSIP] Support load 
binlog from MySQL directly
&nbsp;&nbsp;&nbsp; &gt;&gt; without relying other service
&nbsp;&nbsp;&nbsp; &gt;&gt;
&nbsp;&nbsp;&nbsp; &gt;&gt;
&nbsp;&nbsp;&nbsp; &gt;&gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; Hi hui,
&nbsp;&nbsp;&nbsp; &gt;&gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; I want to know what is the reason for canceling the 
cancel component? Is
&nbsp;&nbsp;&nbsp; &gt;&gt; there any problem in use? What are the benefits ?
&nbsp;&nbsp;&nbsp; &gt;&gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; Parsing binlog is not actually the function of 
Doris itself. Or is it
&nbsp;&nbsp;&nbsp; &gt;&gt; better to appear as a connector? For example, like 
Flink connector.
&nbsp;&nbsp;&nbsp; &gt;&gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; Ling Miao
&nbsp;&nbsp;&nbsp; &gt;&gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; 陈明雨 <morning...@163.com&amp;gt; 于2022年3月29日周二 
19:46写道：
&nbsp;&nbsp;&nbsp; &gt;&gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt; Hi Hui,
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt; Thanks for it. Could you register a cwiki 
acount and tell me your
&nbsp;&nbsp;&nbsp; &gt;&gt; account
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt; id? So that I can create DSIP and grant 
write priv for you.
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt; 
https://cwiki.apache.org/confluence/signup.action
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt; --
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt; 此致！Best Regards
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt; 陈明雨 Mingyu Chen
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt; Email:
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt; chenmin...@apache.org
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt; At 2022-03-29 18:17:02, "晖" 
<827677...@qq.com.INVALID&amp;gt; wrote:
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt; &amp;gt;As issue #8025(
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt;
&nbsp;&nbsp;&nbsp; &gt;&gt; 
https://github.com/apache/incubator-doris/issues/8025)&amp;amp;nbsp;mentioned,
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt; the current binlog load is relying on 
Canal service. We can integrate
&nbsp;&nbsp;&nbsp; &gt;&gt; the
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt; binlog aquire and parse function into 
Doris, so that user can be more
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt; convinience to use it.Please determine the 
final DSIP number for it.
&nbsp;&nbsp;&nbsp; &gt;&gt; &amp;gt;


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org
For additional commands, e-mail: dev-h...@doris.apache.org

Re: [Discuss][DSIP] Support load binlog from MySQL directly without relying other service

Reply via email to