Re: 回复:flink connector 在strict_mode下过滤数据的问题

2022-07-31 Thread Xie Josh
Hi, 我可以理解为在flinksql中这样配置吗? ... with( 'connector' = 'doris', ... 'sink.properties.max_filter_ratio'='0.1' ); 在 2022/8/1 13:20,“王磊” 写入: strict_mode Stream Load 导入可以开启 strict mode 模式。开启方式为在 HEADER 中声明 strict_mode=true 。默认的 strict mode 为关闭。strict mode 模式的意思是:对于导入过程中的列类型

[GitHub] [doris-website] hf200012 merged pull request #26: fix-document

2022-07-31 Thread GitBox
hf200012 merged PR #26: URL: https://github.com/apache/doris-website/pull/26 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@doris.apach

回复:flink connector 在strict_mode下过滤数据的问题

2022-07-31 Thread 王磊
strict_mode Stream Load 导入可以开启 strict mode 模式。开启方式为在 HEADER 中声明 strict_mode=true 。默认的 strict mode 为关闭。strict mode 模式的意思是:对于导入过程中的列类型转换进行严格过滤。严 max_filter_ratio 导入任务的最大容忍率,默认为0容忍,取值范围是0~1。当导入的错误率超过该值,则导入失败。 如果用户希望忽略错误的行,可以通过设置这个参数大于 0,来保证导入可以成功。 计算公式为: (dpp.abnorm.ALL / (dpp.abnorm.ALL + dpp.n

flink connector 在strict_mode下过滤数据的问题

2022-07-31 Thread Xie Josh
HI, 大家好 我在使用flink_connector 导入数据时, 会时长出现too many filter rows的问题, 我了解到是因为FE配置的strict=true, 并且我的source数据字符串字段中可能存在换行\n, 制表符\t等导致的该问题, 目前我是在sql中replace替换来解决此问题, 但是strict模式下如果一条数据出现此问题就会导致整个任务的失败… 我想咨询开发者在使用中是否也存在此种情况? 如何避免? doris后续有没有考虑对此种场景进行自动转换? 或者说考虑strict的阈值问题? 祝好!

[GitHub] [doris-website] hf200012 merged pull request #24: [blogs/docs] update 1.1.1 release note and flink connector 2.0 blogs

2022-07-31 Thread GitBox
hf200012 merged PR #24: URL: https://github.com/apache/doris-website/pull/24 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@doris.apach

[GitHub] [doris-website] hf200012 merged pull request #25: [typo](doc)fix be config buffer_pool_limit default value

2022-07-31 Thread GitBox
hf200012 merged PR #25: URL: https://github.com/apache/doris-website/pull/25 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@doris.apach

[GitHub] [doris-website] hf200012 commented on pull request #25: [typo](doc)fix be config buffer_pool_limit default value

2022-07-31 Thread GitBox
hf200012 commented on PR #25: URL: https://github.com/apache/doris-website/pull/25#issuecomment-1200425054 Please update the Chinese documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [doris-website] dataalive opened a new pull request, #25: [typo](doc)fix be config buffer_pool_limit default value

2022-07-31 Thread GitBox
dataalive opened a new pull request, #25: URL: https://github.com/apache/doris-website/pull/25 fix default value and type error -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen