Hi all,
I'm confused about Executor and BlockManager, why they have different
memory.
14/10/10 08:50:02 INFO AppClient$ClientActor: Executor added:
app-20141010085001-0000/2 on worker-20141010004933-brick6-35657
(brick6:35657) with 6 cores
14/10/10 08:50:02 INFO SparkDeploySchedulerBackend: Granted executor
ID app-20141010085001-0000/2 on hostPort brick6:35657 with 6 cores,
50.0 GB RAM
14/10/10 08:50:07 INFO BlockManagerMasterActor: Registering block
manager brick6:53296 with 26.8 GB RAM
and on the WebUI,
Executor ID Address RDD Blocks Memory Used Disk Used
Active Tasks Failed Tasks Complete Tasks Total Tasks Task
Time Input Shuffle Read Shuffle Write
0 brick3:37607 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6
0 ms 0.0 B 0.0 B 0.0 B
1 brick1:59493 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6
0 ms 0.0 B 0.0 B 0.0 B
2 brick6:53296 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6
0 ms 0.0 B 0.0 B 0.0 B
3 brick5:38543 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6
0 ms 0.0 B 0.0 B 0.0 B
4 brick2:44937 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6
0 ms 0.0 B 0.0 B 0.0 B
5 brick4:46798 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6
0 ms 0.0 B 0.0 B 0.0 B
<driver> brick0:57692 0 0.0 B / 274.6 MB 0.0 B 0 0
0 0 0 ms 0.0 B 0.0 B 0.0 B
As I understand it, a worker consist of a daemon and an executor, and
executor takes charge both execution and storage.
So does it mean that 26.8 GB is saved for storage and the rest is for
execution?
Another question is that, throughout execution, it seems that the
blockmanager is always almost free.
14/10/05 14:33:44 INFO BlockManagerInfo: Added broadcast_21_piece0 in
memory on brick2:57501 (size: 1669.0 B, free: 21.2 GB)
I don't know what I'm missing here.
Best regards,
Larry
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
