Hey Jesse,
Could you provide the operators you using?
For the heap dump, it may be not a real memory leak. Since batches started
to queue up, the memory usage should increase.
On Thu, Jan 28, 2016 at 11:54 AM, Ted Yu wrote:
> bq. The total size by class B is 3GB in 1.5.1 and only 60MB in 1.6.0
bq. The total size by class B is 3GB in 1.5.1 and only 60MB in 1.6.0.
>From the information you posted, it seems the above is backwards.
BTW [B is byte[], not class B.
FYI
On Thu, Jan 28, 2016 at 11:49 AM, Jesse F Chen wrote:
> I ran the same streaming application (compiled individually for 1
I ran the same streaming application (compiled individually for 1.5.1 and
1.6.0) that processes 5-second tweet batches.
I noticed two things:
1. 10% regression in 1.6.0 vs 1.5.1
Spark v1.6.0: 1,564 tweets/s
Spark v1.5.1: 1,747 tweets/s
2. 1.6.0 streaming seems to have a memory leak.
