Hi Jagadish
No no no no!!! not the problem of containers number.is one container
use 1G momory is ok?
My task use very much memory. I want this abnormal phenomenon.
If data cleaning task create 3 container. How much memory it uses? It
is no data stored in memory.
My task create 3 container each container uses 1G memory!!! Not count
applicationMaster(SamzaMaster).
Is this a normal phenomenon still mine open mode is bad? It strikes me
that this uses 256~512 memory
ample running 1 container. What do you think?
发送自 Windows 10 版邮件<http://go.microsoft.com/fwlink/?LinkId=550986>应用
发件人: Jagadish Venkatraman
发送时间: 2017年3月10日 0:33
收件人: [email protected]
主题: Re: build encode error
I'm not very clear what you mean. Perhaps, you could help me understand a
bit more.
*>>"But this task very very neey memory the one container to eat 1G
memory!!"*
You can certainly increase the number of containers (assuming you're
running on Yarn). Please look at
samza.apache.org/learn/documentation/0.10/jobs/configuration-table.html .
The properties of interest is yarn.container.count.
*>>this task start three container be going to eat 3G memory, add
am.container 768M! OGM!!! I’m just running six or seven task!*
In a Yarn cluster, there are usually many machines. So, 4G memory is not
much. Samza can scale to hundreds of machines and hundreds of containers.
The memory is *per-container process* and not per-task.
*>>It is my way of opening is wrong or it's just like this?*
I'm not sure how I can help here. 4G memory consumption for the entire job
is definitely expected. There's no free lunch in computer science ;-)
On Thu, Mar 9, 2017 at 2:57 AM, wu shaodong <[email protected]> wrote:
> Hi everybody
>
> Here I come again. I’m use samza runtime clean data project. But
> this task very very neey memory the one container to eat 1G memory!!!
>
> this task start three container be going to eat 3G memory, add
> am.container 768M! OGM!!! I’m just running six or seven task!
>
>
>
> It is my way of opening is wrong or it's just like this?
>
>
>
> Thanks very much for your help.
>
>
>
>
>
> 发送自 Windows 10 版邮件<http://go.microsoft.com/fwlink/?LinkId=550986>应用
>
--
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University
