Thanks Hong for sharing this - very helpful. ~ Karthik On Thu, Jun 15, 2023 at 2:55 AM Teoh, Hong <lian...@amazon.co.uk> wrote:
> Hi Karthik, > > As an additional update, since Flink 1.15 we have introduced the > asynchronous sink base, which allows easy writing of an asynchronous sink > (you simply provide the client, request and response handling) and the sink > base will handle the state management, retries and batching. > > See the blogpost [1] for more details! > > Hope the above helps. > > Regards, > Hong > > [1] > https://flink.apache.org/2022/03/16/the-generic-asynchronous-base-sink/ > > > On 13 Jun 2023, at 05:29, Karthik Deivasigamani <karthi...@gmail.com> > wrote: > > *CAUTION*: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > Thanks Martijn, the documentation for Async IO was also indicating the > same and that's what prompted me to post this question here. > ~ > Karthik > > On Mon, Jun 12, 2023 at 7:45 PM Martijn Visser <martijnvis...@apache.org> > wrote: > >> Hi Karthik, >> >> In my opinion, it makes more sense to use a sink to leverage Scylla over >> using Async IO. The primary use case for Async IO is enrichment, not for >> writing to a sync. >> >> Best regards, >> >> Martijn >> >> On Mon, Jun 12, 2023 at 4:10 PM Karthik Deivasigamani < >> karthi...@gmail.com> wrote: >> >>> Thanks Martijn for your response. >>> One thing I did not mention was that we are in the process of moving >>> away from Cassandra to Scylla and would like to use the Scylla Java Driver >>> for the following reason : >>> >>>> The Scylla Java driver is shard aware and contains extensions for a >>>> tokenAwareHostPolicy. Using this policy, the driver can select a >>>> connection to a particular shard based on the shard’s token. As a result, >>>> latency is significantly reduced because there is no need to pass data >>>> between the shards. >>>> >>> We were considering writing our own Sink to leverage Scylla Java Driver >>> once the migration is done. >>> ~ >>> Karthik >>> >>> >>> On Mon, Jun 12, 2023 at 4:56 PM Martijn Visser <martijnvis...@apache.org> >>> wrote: >>> >>>> Hi, >>>> >>>> Why wouldn't you just use the Flink Kafka connector and the Flink >>>> Cassandra connector for your use case? >>>> >>>> Best regards, >>>> >>>> Martijn >>>> >>>> On Mon, Jun 12, 2023 at 12:03 PM Karthik Deivasigamani < >>>> karthi...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> I have a use case where I need to read messages from a Kafka topic, >>>>> parse it and write it to a database (Cassandra). Since Cassandra supports >>>>> async APIs I was considering using Async IO operator for my writes. I do >>>>> not need exactly-once semantics for my use-case. >>>>> Is it okay to leverage the Async IO operator as a Sink (writing data >>>>> into a DB)? >>>>> ~ >>>>> Karthik >>>>> >>>> >
