[GitHub] [doris] tempestLXC opened a new issue, #16393: [Bug] Binlog同步 对应表变动后无法投递到Doris中对应的channel中,导致同步失败

[via GitHub]

tempestLXC opened a new issue, #16393:
URL: https://github.com/apache/doris/issues/16393

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no 
similar issues.
   
   
   ### Version
   
   1.2.1
   
   ### What's Wrong?
   
   错误表现:  部分表通过Canal 订阅Binlog, Doris中已经接收到Binlog变动,但是对应表的数据没有变化.
   
   深入分析: 经过对FE的Binlog Debug日志进行分析, 发现对Binlog解析正确(包含表明内容), 但是没有执行后续操作, 
缺少Channel匹配成功日志和执行sql日志.  
   
   通过对日志和 show sync job命令结果检查发现, 通过binlog日志解析的表名中有大写字母, 而初始化渠道时channel的名称为纯小写字母
   
   总结: 因为mysql 开启了大小写敏感, 所以部分表名存在大写字母. 而Doris是纯小写字母进行初始化channel名称. 
而解析binlog后的表名为源表名, 导致该条binlog数据无法投递到对应的channel中.导致数据无法变更.
   
   ### What You Expected?
   
   建议在create sync job中新增一个参数是否大小写敏感, 可以解决该问题.
   
   ### How to Reproduce?
   
   复现环境: 1、mysql 并设置表名大小写敏感
   
   浮现步骤: 1、分别在doris和mysql创建表一:table_test_01(纯小写表名作为测试对照)
                    2、分别在doris和mysql创建表二:Table_test_02(大小写混合的表)
                    3、doris 通过canal 订阅两张表的binlog
                    4、修改表一(新增、修改或者删除)数据---->doris同步数据成功 说明binlog订阅基本功能没问题
                    5、修改表二数据---->doris表数据无变化
   
   ### Anything Else?
   
   目前我的解决方案:
   在创建job时增加参数 "canal.ignoreCase"="true"
   <img width="320" alt="image" 
src="https://user-images.githubusercontent.com/8910690/216529273-904c7a19-50dc-40bd-ab50-6cdb5aa014a0.png";>
   
   修改fe/fe-core/src/main/java/org/apache/doris/load/sync/canal/ 包下相关类和测试类
   
   核心是修改下图方法, 根据 canal.ignoreCase设置的值进行判断是否要对binlog解析完成的表名进行小写转换
   <img width="1463" alt="image" 
src="https://user-images.githubusercontent.com/8910690/216528267-b3f93d22-c143-4da0-8df7-c4aa84f06bf0.png";>
   
   
   上述方案基本可以解决当前问题,但是无法根本解决
   
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org