[
https://issues.apache.org/jira/browse/IGNITE-19458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vedran Ljubovic updated IGNITE-19458:
-------------------------------------
Attachment: no_static_final.patch
> Kafka Connect IgniteSinkConnector multiple topics & caches do not work
> ----------------------------------------------------------------------
>
> Key: IGNITE-19458
> URL: https://issues.apache.org/jira/browse/IGNITE-19458
> Project: Ignite
> Issue Type: Bug
> Components: extensions
> Affects Versions: 2.15
> Reporter: Vedran Ljubovic
> Priority: Major
> Attachments: IgniteSinkTask.java, no_static_final.patch
>
>
> We are using Kafka Connect to stream 4 topics into 4 separate caches. We are
> presently forced to use 4 separate instances of Kafka Connect because when 4
> connectors are passed to one instance all messages will be streamed into the
> same (first) Ignite cache. Multiple Connect instances work, but in the future
> we plan to add more topics/caches and this will affect memory usage as well
> as our ability to distribute workload in a cluster.
> I think that the reason for this is in IgniteSinkTask.java where
> IgniteDataStreamer is declared as private static final attribute of inner
> class, which means that it will be initialized with the first passed cache
> and cannot be changed later.
> I'm not sure which is the best approach performance wise, but possible
> solutions would be:
> * It should be non-static attribute of IgniteSinkTask (different streamer
> for each task)
> * It should be created in IgniteSinkConnector and passed to IgniteSinkTask
> (one streamer per connector)
> * Have a Map of cacheNames to Streamers which ensures that only as many
> Streamers will be created as there are different caches used.
> Also I notice that cacheName, SingleTupleExtractor, stopped boolean, are all
> declared as static which cause various problems when using multiple
> connectors (messages being processed with wrong SimpleTupleExtractor etc.)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
