[
https://issues.apache.org/jira/browse/IGNITE-529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14998831#comment-14998831
]
Anton Vinogradov commented on IGNITE-529:
-----------------------------------------
Roman,
Thanks for comments.
Seems that you was absolutelly right saying "the sink is central here, and
probably in future this integration has to be a part of Flume, not Ignite.".
But I hope we still want to have it as a part of Ignite.
So, it means we absolutelly need no FlumeStreamer, because everything will
start and work at Flume (Sink) side.
I propose following (pretty simmilar to you current solution):
1) IgniteSink starts Ignite node from configuration, but uses getCache instead
of getOrCreateCache. Make sure you cannot start Sink twice.
2) IgniteSink has own transformer, Interface should be provided.
Implementation of transformer should be specified at configuration.
3) Sink.process() writes directly to Ignite.cache, using transformer. Put,
putAll or DataSrtreamer can be used to update cache.
> Implement IgniteFlumeStreamer to stream data from Apache Flume
> --------------------------------------------------------------
>
> Key: IGNITE-529
> URL: https://issues.apache.org/jira/browse/IGNITE-529
> Project: Ignite
> Issue Type: Sub-task
> Components: streaming
> Reporter: Dmitriy Setrakyan
> Assignee: Roman Shtykh
>
> We have {{IgniteDataStreamer}} which is used to load data into Ignite under
> high load. It was previously named {{IgniteDataLoader}}, see ticket
> IGNITE-394.
> See [Apache Flume|http://flume.apache.org/] for more information.
> We should create {{IgniteFlumeStreamer}} which will consume messages from
> Apache Flume and stream them into Ignite caches.
> More details to follow, but to the least we should be able to:
> * Convert Flume data to Ignite data using an optional pluggable converter.
> * Specify the cache name for the Ignite cache to load data into.
> * Specify other flags available on {{IgniteDataStreamer}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
