[
https://issues.apache.org/jira/browse/IGNITE-529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14997902#comment-14997902
]
Roman Shtykh commented on IGNITE-529:
-------------------------------------
Anton, thank you for your feedback.
I rewrote my code following your recommendations as closely as I could.
I extended StreamAdapter as you advised (although the docs say it is optional
;)) and used an extractor for converting Flume events. However, Flume is
different from other streamers we have so far. Sink, which is normally
instantiated with Flume deployment, is central here, not the streamer. User
does not create a cache or a streamer and cannot pass it as a parameter. Sink
reads configurations provided in conf file and instantiates the streamer
through those parameters.
And therefore having two extractors does not work here (lines 124-126 of
IgniteSink.java) -- I wish it was one interface, but for now I used
StreamMultipleTupleExtractor here.
As I said, the sink is central here, and probably in future this integration
has to be a part of Flume, not Ignite.
I hope you have a chance at looking how Flume works
https://flume.apache.org/FlumeDeveloperGuide.html
> Implement IgniteFlumeStreamer to stream data from Apache Flume
> --------------------------------------------------------------
>
> Key: IGNITE-529
> URL: https://issues.apache.org/jira/browse/IGNITE-529
> Project: Ignite
> Issue Type: Sub-task
> Components: streaming
> Reporter: Dmitriy Setrakyan
> Assignee: Roman Shtykh
>
> We have {{IgniteDataStreamer}} which is used to load data into Ignite under
> high load. It was previously named {{IgniteDataLoader}}, see ticket
> IGNITE-394.
> See [Apache Flume|http://flume.apache.org/] for more information.
> We should create {{IgniteFlumeStreamer}} which will consume messages from
> Apache Flume and stream them into Ignite caches.
> More details to follow, but to the least we should be able to:
> * Convert Flume data to Ignite data using an optional pluggable converter.
> * Specify the cache name for the Ignite cache to load data into.
> * Specify other flags available on {{IgniteDataStreamer}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
