Re: [DISCUSS] Migrate or deprecate the Spark Kinesis connector

2024-10-17 Thread Johnson Chen
Hi Juntaek, Thanks for the information. Given that, are we good to close this Jira https://issues.apache.org/jira/browse/SPARK-45720 ? -Junyu On Wed, Oct 16, 2024 at 8:36 PM Jungtaek Lim wrote: > DStream is deprecated in Spark 3.4.0, hence Kinesis connector for DStream > is inheriting the same

Re: [DISCUSS] Migrate or deprecate the Spark Kinesis connector

2024-10-16 Thread Jungtaek Lim
DStream is deprecated in Spark 3.4.0, hence Kinesis connector for DStream is inheriting the same fate. We just didn't make the whole class of DStream to produce warning messages, as we made the entry class to produce warning messages and thought it's sufficient. On Mon, Oct 14, 2024 at 5:03 PM Joh

[DISCUSS] Migrate or deprecate the Spark Kinesis connector

2024-10-14 Thread Johnson Chen
Hi Spark community, A couple months ago, I raised a PR to upgrade the AWS SDK to v2 for the Spark Kinesis connector: https://github.com/apache/spark/pull/44211. Given that the 4.0 feature freeze is coming, I am following up to check whether we still want to have this change in the upcoming 4.0 re