Hi, With parallelism I mean number for connectors in parallel.
Even if group id is same, it will work like any standalone application. If node get down due to some reason, connector running on node will be down as well. In distributed mode, JVMs are shared across connect nodes which makes it more available. Thanks and Regards, Shubham On Mon, Sep 13, 2021, 1:23 AM Saikat Maitra <saikat.mai...@gmail.com> wrote: > Are you looking for a parallel Ignite sink connector that can connect to > kafka topic? > > My thoughts are you should be able to do it as long as your group_id is > same like mentioned here > https://medium.com/@jhansireddy007/how-to-parallelise-kafka-consumers-59c8b0bbc37a > > I may have misunderstood the question as well. > > Regards, > Saikat > > > > > > On Sun, Sep 12, 2021 at 1:32 PM Shubham Shirur <shirurshub...@gmail.com> > wrote: > >> Yes, I agree with you. Basically kafka connect works in two different >> ways i.e standalone and distributed mode. >> >> In standard mode ignite sink connectors run very smoothly. So if I want >> to create multiple topic cache data transmission, I need to setup same >> number of workers and connectors which is not recommended in production >> because its standalone, if connect node goes down connector will stop. >> >> But in distributed mode single worker is running across nodes and >> rebalancing happens for connectors tasks if any node goes down. In this >> case I am able to create just 1 ignite connector at a time. No parallelism. >> >> Hope this gives better clarity. >> >> Thanks and regards, >> Shubham >> >> On Sun, Sep 12, 2021, 11:50 PM Saikat Maitra <saikat.mai...@gmail.com> >> wrote: >> >>> Can you please elaborate on distributed mode? >>> >>> You can always connect multiple client to different kafka topic and >>> write data to cluster of Ignite nodes. >>> >>> Regards, >>> Saikat >>> >>> On Sun, Sep 12, 2021 at 10:47 AM Shubham Shirur <shirurshub...@gmail.com> >>> wrote: >>> >>>> Hey, >>>> >>>> Thanks for replying. I have gone through the documentation and could >>>> setup and run connectors in standalone mode as described there in docs. But >>>> I want to run it in distributed mode, can you help me with that? >>>> >>>> Thanks & Regards, >>>> Shubham >>>> >>>> On Sun, Sep 12, 2021, 7:58 PM Saikat Maitra <saikat.mai...@gmail.com> >>>> wrote: >>>> >>>>> Hi Shubham, >>>>> >>>>> Here are the documents for Apache Ignite Sink Connector using Kafka >>>>> https://github.com/apache/ignite-extensions/tree/master/modules/kafka-ext >>>>> >>>>> Please let us know if you have any questions. >>>>> >>>>> Regards, >>>>> Saikat >>>>> >>>>> On Wed, Aug 18, 2021 at 11:40 PM Shubham Shirur < >>>>> shirurshub...@gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I did not find any specific documentation on Apache Ignite Sink >>>>>> Connector. I am using a kafka ignite sink connector and want to push some >>>>>> kafka topic data in ignite where kafka and ignite are on remote nodes. My >>>>>> connector should ideally run on a remote node from the ignite node. >>>>>> >>>>>> How can I achieve this? >>>>>> What configurations I need to pass in the spring xml file which I >>>>>> pass as a parameter in connector properties? >>>>>> What configurations I need to pass in the spring xml file of the >>>>>> ignite server node? >>>>>> >>>>>> Thanks, >>>>>> Shubham >>>>>> >>>>>
