Which was your final configuration? What speed did you get?
2014-09-03 11:18 GMT+02:00 Sebastiano Di Paola :
> I raised batchSize of a 100 factor, added more heap space and speed
> increased...
> still not reached the same speed as using "hdfs dfs -copyFromLocal" but
> I'm pretty sure it's a tun
I raised batchSize of a 100 factor, added more heap space and speed
increased...
still not reached the same speed as using "hdfs dfs -copyFromLocal" but I'm
pretty sure it's a tuning problem.
thanks a lot for your hint.
Regards
Seba
On Wed, Sep 3, 2014 at 9:55 AM, Sandeep Khurana
wrote:
> Sinc
Since you mentioned "average size of 150 bytes each" is your each record, I
will try increasing the batch size to a higher value.
"HDFS batch size determines the number of events to take from the channel and
send in one go."
So in 1 shot you are sending 150 bytes to hdfs.
On Wed, Sep 3, 20
In my experiment, I just want to transfer a single file...just to test what
performances I can achieve...
so rolling file on hdfs at this point is not vital.
Anyway I did some test rolling file every 300 seconds.
What I can't explain to myself is the "slow" output from the sink...the
memory channel
I see that you have below settings set to zero. You dont want rolling to
hdfs to happen based upon any of the size, count or time interval?
test.sinks.s1.hdfs.rollSize = 0
test.sinks.s1.hdfs.rollCount = 0
test.sinks.s1.hdfs.rollInterval = 0
On Wed, Sep 3, 2014 at 1:06 PM, Sebastiano Di Paola <
Hi Paul,
thank for your answer.
As I' m a newbie of Flume How can I attach multiple sinks to the same
channel? (does they read data in a round robin fashon from the memory
channel?)
(does this create multiple files on the hdfs?, because this is not what I'm
expecting to have I have a 500MB data fil
Start adding additional HDFS sinks attached to the same channel. You can also
tune batch sizes when writing to HDFS to increase per sink performance.
On Sep 2, 2014, at 11:54 PM, "Sebastiano Di Paola"
mailto:sebastiano.dipa...@gmail.com>> wrote:
Hi there,
I'm a completely newbie of Flume, so I
