Hi,
Thanks for the suggestion. I will take a look at Blob Handler.
On Thu, Aug 31, 2017 at 1:28 PM, Archana Ravindran
wrote:
> You can try using blob handler in http source. Though blob is for binary,
> we were able to push relatively large event.
>
> For size limit, in our case we were not ab
Hi,
I have a question regarding Flume suitability for a particular use case.
Task: There is an incoming constant stream of links that point to files.
Those files to be fetched and stored in HDFS.
Desired implementation:
1. Each link to a file is stored in Kafka queue Q1.
2. Flume A1.source moni
Hi GIntas,
Do you happen to have a config file we could examine?
Based on the scenario you have described, I'm thinking of a
"multi-agent flow" (
http://flume.apache.org/FlumeUserGuide.html#setting-multi-agent-flow )
with a Kafka source at the "start" of the first agent and a HDFS sink
at the "en
Hi GIntas,
What is the average (or expected maximum) size of the files you'd like to
process?
In general it is not recommended to transfer large events (i.e. >64MB if
you use file channel, as this is a hard limit of the protobuf
implementation).
If your files fit into this limit then I'd suggest t
Hi,
Thanks for the quick replies, guys.
Donat, sorry, I do not have example configs. At the moment I am just
considering available solutions to tackle the problem at hand. I would very
much prefer Flume for its modular and scalable approach. I would like to
find an elegant solution that would be
