bq.  I haven't added one more HDFS node to a hadoop cluster

Does each of three nodes colocate with hdfs data nodes ?
The absence of 4th data node might have something to do with the partition
allocation.

Can you show your code snippet ?

Thanks

On Sat, Mar 5, 2016 at 2:54 PM, Eugene Morozov <[email protected]>
wrote:

> Hi,
>
> My cluster (standalone deployment) consisting of 3 worker nodes was in the
> middle of computations, when I added one more worker node. I can see that
> new worker is registered in master and that my job actually get one more
> executor. I have configured default parallelism as 12 and thus I see that
> each of three nodes holds 4 RDD blocks. I have expected that by adding one
> more node those partitions would be reassigned into 4 nodes instead of 3,
> but I don't see that. Even though all partitions are cached in memory of
> those 3 nodes and most probably have their locations or preferred locations
> known, I'd still expect partitions be reassigned. Are my expectations
> incorrect?
>
> I do also have an HDFS setup on those the same 3 worker nodes, but when I
> added one more node to the spark cluster I haven't added one more HDFS node
> to a hadoop cluster. That in my mind shouldn't be the reason, but who knows?
>
> Thanks.
> --
> Be well!
> Jean Morozov
>

Reply via email to