Hi Samrat, Thanks for the review and feedback. We have evaluated all the three points. Please find the answers below:
1. AWS has announced JSON protocol support in SQS [1]. Can you shed some light on how different protocols will be supported? - We will utilize the AWS client library to connect with the AWS SQS Service. Versions beyond 2.21.19 now support JSON, so simply upgrading the client library will suffice for the protocol switch. However, from the connector's perspective, we do not anticipate any changes in our communication process, as it is handled by the client library. [4] 2. AWS SQS has two types of queues [2]. What are the implementation detail differences for the source connector? - SQS Connector is indifferent to the customer's choice of Queue type. If out-of-order messages are a concern, it will be the responsibility of the application code or main job logic to manage this. 3. Will the SQS source connector implement any kind of callbacks [3] on success to offer any kind of guarantee? - We have proposed deleting SQS messages using the notification provided by the checkpoint framework on checkpoint completion. Thus providing exactly once guarantee.[5] Additionally, when deleting messages, we will monitor the API call responses and log any failures, along with providing observability through appropriate metrics. [1] https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-sqs-support-json-protocol/ [2] https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-queue-types.html [3] https://docs.aws.amazon.com/step-functions/latest/dg/callback-task-sample-sqs.html [4] https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-json-faqs.html#json-protocol-getting-started [5] https://docs.google.com/document/d/1lreo27jNh0LkRs1Mj9B3wj3itrzMa38D4_XGryOIFks/edit#heading=h.jdcikzojx5d9 [Main Proposal doc] - https://docs.google.com/document/d/1lreo27jNh0LkRs1Mj9B3wj3itrzMa38D4_XGryOIFks/edit#heading=h.ci1rrcgbsvkl Please feel free to reach out if you have more feedback. Regards Saurabh & Abhi On Wed, Jul 24, 2024 at 8:52 AM Samrat Deb <decordea...@gmail.com> wrote: > Hi Saurabh, > > Thank you for sharing the FLIP for the SQS source connector. An SQS source > connector will be a great addition to the Flink ecosystem, as there is a > growing demand for SQS source/sink integration. > > I have a few queries: > > 1. AWS has announced JSON protocol support in SQS [1]. Can you shed some > light on how different protocols will be supported? > 2. AWS SQS has two types of queues [2]. What are the implementation detail > differences for the source connector? > 3. Will the SQS source connector implement any kind of callbacks [3] on > success to offer any kind of guarantee? > > > > [1] > > https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-sqs-support-json-protocol/ > [2] > > https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-queue-types.html > [3] > > https://docs.aws.amazon.com/step-functions/latest/dg/callback-task-sample-sqs.html > > Bests, > Samrat > > > On Fri, 19 Jul 2024 at 9:53 PM, Saurabh Singh <saurabhsingh9...@gmail.com> > wrote: > > > Hi Fink Devs, > > > > Our team has been working on migrating various data pipelines to Flink to > > leverage the benefits of exactly-once processing, checkpointing, and > > stateful computing. We have several use cases built around the AWS SQS > > Service. For this migration, we have developed an SQS Source Connector, > > which enables us to run both stateless and stateful Flink-based jobs. > > > > We believe that this SQS Source Connector would be a valuable addition to > > the existing connector set. Therefore, we propose a FLIP to include it. > > > > For more information, please refer to the FLIP document. > > > > > > > https://docs.google.com/document/d/1lreo27jNh0LkRs1Mj9B3wj3itrzMa38D4_XGryOIFks/edit?usp=sharing > > > > Thanks > > Saurabh & Abhi > > >
