Ah yes, missed the kafka part and just saw the array part. FLINK-19771 definitely was solely in the postgres-specific code.
Dylan From: Jark Wu <[email protected]> Date: Thursday, November 19, 2020 at 9:12 AM To: Dylan Forciea <[email protected]> Cc: Danny Chan <[email protected]>, Rex Fenley <[email protected]>, Flink ML <[email protected]> Subject: Re: Filter Null in Array in SQL Connector Hi Dylan, I think Rex encountered another issue, because he is using Kafka with Debezium format. Hi Rex, If you can share the json data and the exception stack, that would be helpful! Besides, you can try to enable 'debezium-json.ignore-parse-errors' option [1] to skip the dirty data. Best, Jark [1]: https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/connectors/formats/debezium.html#debezium-json-ignore-parse-errors On Thu, 19 Nov 2020 at 21:13, Dylan Forciea <[email protected]<mailto:[email protected]>> wrote: Do you mean that the array contains values that are null, or that the entire array itself is null? If it’s the latter, I have an issue written, along with a PR to fix it that has been pending review [1]. Regards, Dylan Forciea [1] https://issues.apache.org/jira/browse/FLINK-19771 From: Danny Chan <[email protected]<mailto:[email protected]>> Date: Thursday, November 19, 2020 at 2:24 AM To: Rex Fenley <[email protected]<mailto:[email protected]>> Cc: Flink ML <[email protected]<mailto:[email protected]>> Subject: Re: Filter Null in Array in SQL Connector Hi, Fenley ~ You are right, parsing nulls of ARRAY field is not supported now, i have logged an issue [1] and would fix it soon ~ [1] https://issues.apache.org/jira/browse/FLINK-20234 Rex Fenley <[email protected]<mailto:[email protected]>> 于2020年11月19日周四 下午2:51写道： Hi, I recently discovered some of our data has NULL values arriving in an ARRAY<STRING> column. This column is being consumed by Flink via the Kafka connector Debezium format. We seem to be receiving NullPointerExceptions for when these NULL values in the arrays arrive which restarts the source operator in a loop. Is there any way to not throw or to possibly filter out NULLs in an Array of Strings in Flink? We're somewhat stuck on how to solve this problem, we'd like to be defensive about this on Flink's side. Thanks! (P.S. The exception was not that informative, there may be room for improvement in terms of a richer error message when this happens.) -- Rex Fenley | Software Engineer - Mobile and Backend Remind.com<https://www.remind.com/> | BLOG<http://blog.remind.com/> | FOLLOW US<https://twitter.com/remindhq> | LIKE US<https://www.facebook.com/remindhq>
