Re: Spark Streaming using Flume body size limitation

2014-05-23 Thread David Lemieux
Created https://issues.apache.org/jira/browse/SPARK-1916 I'll submit a pull request soon. /D On May 23, 2014, at 9:56 AM, David Lemieux wrote: > For some reason the patch did not make it. > > Trying via email: > > > > /D > > On May 23, 2014, at 9:52 AM, lemieud wrote: > > > Hi, > > > > I think

Re: Spark Streaming using Flume body size limitation

2014-05-23 Thread David Lemieux
For some reason the patch did not make it. Trying via email: /D On May 23, 2014, at 9:52 AM, lemieud wrote: > Hi, > > I think I found the problem. > In SparkFlumeEvent the readExternal method use in.read(bodyBuff) which read > the first 1020 bytes, but no more. The code should make sure to r

Re: Spark Streaming using Flume body size limitation

2014-05-23 Thread lemieud
Hi, I think I found the problem. In SparkFlumeEvent the readExternal method use in.read(bodyBuff) which read the first 1020 bytes, but no more. The code should make sure to read everything. The following change will fix the problem: in.read(bodyBuff) to: in.readFully(bodyBuff) I attached a patch.