Thank you, David! Fan Out Reads are certainly on my mind, though probably nothing I’ll be able to tackle soon. I agree, that would be a great feature to get attention for v2. Best, Moritz
From: David Hollands <david.holla...@bbc.co.uk> Date: Monday, 21. March 2022 at 18:53 To: dev@beam.apache.org <dev@beam.apache.org> Subject: Re: [DISCUSS] Deprecation of AWS SDK v1 IO connectors Hi Moritz - I just wanted to say a big THANK YOU for the huge improvements you have contributed to the AWS IOs in recent releases! Such great use of Testcontainers and localstack! 🙂 +1 to deprecating v1 gracefully. Adding ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. Exercise caution when opening attachments or clicking any links. ZjQcmQRYFpfptBannerEnd Hi Moritz - I just wanted to say a big THANK YOU for the huge improvements you have contributed to the AWS IOs in recent releases! Such great use of Testcontainers and localstack! 🙂 +1 to deprecating v1 gracefully. Adding KinesisIO Fan Out Read in v2 and not v1 would be a nice carrot to entice some users I know away from v1... 😜 Best, David Want to work with us? https://datalab.rocks<https://urldefense.com/v3/__https:/datalab.rocks__;!!CiXD_PY!ACyIRENWkKWwuoym5YuZOWykDMIjKyIlEcvHMy7YEv-Pw25sbCau0HADz4SV$> _______________________________________________________ David Hollands Principal Data Engineer, Datalab BBC Product Group [signature_347418782] London: Broadcast Centre, 201 Wood Lane, London, W12 7TP ________________________________ From: Moritz Mack <mm...@talend.com> Sent: 21 March 2022 12:58 To: dev@beam.apache.org <dev@beam.apache.org> Subject: Re: [DISCUSS] Deprecation of AWS SDK v1 IO connectors Thank you both! Absolutely agree on reaching out to users! The release of 2.38 seems to be a very good time to do so to also announce feature parity between the two versions. I’ll get back to the two of you for some help reaching out to users on twitters then! Thanks, Moritz From: Alexey Romanenko <aromanenko....@gmail.com> Date: Sunday, 20. March 2022 at 00:23 To: dev@beam.apache.org <dev@beam.apache.org> Subject: Re: [DISCUSS] Deprecation of AWS SDK v1 IO connectors Yes, good point about asking on Twitter, Ahmet! On 19 Mar 2022, at 04:14, Ahmet Altay <al...@google.com> wrote: Moritz, thank you for your thoughtful approach. +1 to Alexey's suggestion. One minor additional suggestion, in addition to ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. Exercise caution when opening attachments or clicking any links. ZjQcmQRYFpfptBannerEnd Yes, good point about asking on Twitter, Ahmet! On 19 Mar 2022, at 04:14, Ahmet Altay <al...@google.com<mailto:al...@google.com>> wrote: Moritz, thank you for your thoughtful approach. +1 to Alexey's suggestion. One minor additional suggestion, in addition to asking on user@ you can also ask on beam's twitter. Historically we were able to reach more users on Twitter compared to the user list. On Fri, Mar 18, 2022 at 10:01 AM Alexey Romanenko <aromanenko....@gmail.com<mailto:aromanenko....@gmail.com>> wrote: First of all - many thanks for your work to make "amazon-web-services2” stable and up-to-date with already supported features of v1 version. +1 to deprecate all "amazon-web-services” (AWS SDK v1) IOs and recommend to use only v2 API, but before I’d suggest to ask people on user@ if they have some specific reasons to delay it or any other feedback on this that we can miss. Regarding the major releases and removing deprecated code, IIUC, we agreed that deprecated code should stay (but not required to be supported), at least, 3 minor releases (x.y) and it can be finally deleted if we won’t discover any regressions or user complains till then. On the other hand, Beam 3.0 should be the next major release but I’m not sure it’s even on distant horizon for now since this is topic that we didn’t discuss for a long time (maybe it’s a good time to come back to this). — Alexey On 18 Mar 2022, at 12:19, Moritz Mack <mm...@talend.com<mailto:mm...@talend.com>> wrote: Dear all, I’d like to bring up an old discussion again [1]. Currently we have two different versions of AWS IO connectors in Beam for the Java SDK: * amazon-web-services [2] and kinesis [3] for the AWS Java SDK v1 * amazon-web-services2 (including kinesis) [4] for the AWS Java SDK v2 Maintaining two different versions is obviously painful, so working towards sunsetting the earlier v1 is important. Though, historically v1 had (and likely still has) the broader adoption due to a lack of awareness, but also certainly a lack of features in v2. I’ve recently focused a lot on preparing the deprecation of v1, specifically: * implementing all outstanding features in v2 (above all write support for KinesisIO) [5] * full integration test coverage for all IOs in v2 (using localstack), but also improved general test coverage & quality * more consistent configuration and tons of bug fixes Where I’m looking for general feedback is how to proceed next: * Deprecate (IO classes of) v1 immediately to force awareness of v2 with the next release? * Just recommend v2 IOs over v1 IOs on the website [6] to hopefully push adoption and gather more feedback before deprecating v1? Any thoughts or recommendations? Also, looking further ahead … is there a major release planned for Beam that would allow to remove such deprecations? Thanks so much, Moritz [1] https://lists.apache.org/thread/b4opwr3x1rr63732b5j7f47r2bobq6h0<https://urldefense.com/v3/__https:/lists.apache.org/thread/b4opwr3x1rr63732b5j7f47r2bobq6h0__;!!CiXD_PY!EZwzvsmqANBIBbrmVOt-exWI92agUNKLywoXehjD2rLKOk2_xTmAUFP1Ucm2$> [2] https://github.com/apache/beam/tree/master/sdks/java/io/amazon-web-services<https://urldefense.com/v3/__https:/github.com/apache/beam/tree/master/sdks/java/io/amazon-web-services__;!!CiXD_PY!EZwzvsmqANBIBbrmVOt-exWI92agUNKLywoXehjD2rLKOk2_xTmAUFc98SRB$> [3] https://github.com/apache/beam/tree/master/sdks/java/io/kinesis<https://urldefense.com/v3/__https:/github.com/apache/beam/tree/master/sdks/java/io/kinesis__;!!CiXD_PY!EZwzvsmqANBIBbrmVOt-exWI92agUNKLywoXehjD2rLKOk2_xTmAUCA6oerl$> [4] https://github.com/apache/beam/tree/master/sdks/java/io/amazon-web-services2<https://urldefense.com/v3/__https:/github.com/apache/beam/tree/master/sdks/java/io/amazon-web-services2__;!!CiXD_PY!EZwzvsmqANBIBbrmVOt-exWI92agUNKLywoXehjD2rLKOk2_xTmAUMc_5deZ$> [5] https://issues.apache.org/jira/browse/BEAM-13174<https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/BEAM-13174__;!!CiXD_PY!EZwzvsmqANBIBbrmVOt-exWI92agUNKLywoXehjD2rLKOk2_xTmAUC0YFfnS$> [6] https://issues.apache.org/jira/browse/BEAM-14125<https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/BEAM-14125__;!!CiXD_PY!EZwzvsmqANBIBbrmVOt-exWI92agUNKLywoXehjD2rLKOk2_xTmAUP_6TYEh$> As a recipient of an email from Talend, your contact personal data will be on our systems. Please see our privacy notice.<https://www.talend.com/privacy/> As a recipient of an email from Talend, your contact personal data will be on our systems. Please see our privacy notice. <https://www.talend.com/privacy/> As a recipient of an email from Talend, your contact personal data will be on our systems. Please see our privacy notice. <https://www.talend.com/privacy/>
