The imbalance was caused by the stuck partition, after 10s of hours the
receiving rate went down. But the second ERR log I mentioned in the first
mail now occur at most of tasks(I did’t count, but keep flushing my
terminal) and jeopardize the job, as every batch takes 2 min(2-15 seconds
before) exe
Also, you could switch to the Direct KAfka API which was first released as
experimental in 1.3. In 1.5 we graduated it from experimental, but its
quite usable in Spark 1.3.1
TD
On Tue, Sep 22, 2015 at 7:45 PM, SLiZn Liu wrote:
> Cool, we are still sticking with 1.3.1, will upgrade to 1.5 ASAP.
Cool, we are still sticking with 1.3.1, will upgrade to 1.5 ASAP. Thanks
for the tips, Tathagata!
On Wed, Sep 23, 2015 at 10:40 AM Tathagata Das wrote:
> A lot of these imbalances were solved in spark 1.5. Could you give that a
> spin?
>
> https://issues.apache.org/jira/browse/SPARK-8882
>
> On
A lot of these imbalances were solved in spark 1.5. Could you give that a
spin?
https://issues.apache.org/jira/browse/SPARK-8882
On Tue, Sep 22, 2015 at 12:17 AM, SLiZn Liu wrote:
> Hi spark users,
>
> In our Spark Streaming app via Kafka integration on Mesos, we initialed 3
> receivers to rece
