I realize the JC runs in the AM YARN container. The SamzaContainers run in
their own YARN containers, which may or may not be on the same node as the AM
in the YARN grid. My point is simply that when they are not, it does not work.
This is because the AM is telling the SamzaContainers to fetch the config from
127.0.0.1.
-Tommy
On 07/31/2015 02:31 PM, Navina Ramesh wrote:
Hi Tommy,
I am not sure what you mean below:
{quote}
this context by container I meant the SamzaContainer. What we are seeing
is that jobs only start when YARN happens to place the AM and
SamzaContainer(s) on the same node.Which is increasingly unlikely as you
increase container count for your job and/or expand your YARN grid.
{quote}
Any YARN application finds a yarn container to run the AM. In our case, JC
is running in the same container as the AM. So, I don't understand why this
will cause an issue on cluster expansion. I can understand your concern if
JC and AM are running on 2 separate containers.
Please correct me if I have misunderstood your statement.
Thanks!
Navina
On Fri, Jul 31, 2015 at 6:05 AM, Tommy Becker
<tobec...@tivo.com><mailto:tobec...@tivo.com> wrote:
Hey Yan,
-- I do not quite understand this part. AM essentially is running in a
container as well. And the http server is brought up in the same container
Sorry, the term "container" is overloaded. In this context by container I
meant the SamzaContainer. What we are seeing is that jobs only start when
YARN happens to place the AM and SamzaContainer(s) on the same node. Which
is increasingly unlikely as you increase container count for your job
and/or expand your YARN grid.
-Tommy
On 07/30/2015 10:08 PM, Yan Fang wrote:
Hi Thommy,
{quote}
Because I don't see how this is ever going to work in scenarios where the
AM is on a different node than the containers.
{quote}
-- I do not quite understand this part. AM essentially is running in a
container as well. And the http server is brought up in the same container.
{quote}
even if we can't get a better address for the AM from YARN, we could at
least filter the addresses we get back from the JVM to exclude loopbacks.
{quote}
-- You are right. InetAddress.getLocalHost() gives back loopback address
sometimes. We should filter this out. Just googling one possible solution
<http://www.coderanch.com/t/491883/java/java/IP><http://www.coderanch.com/t/491883/java/java/IP><
http://www.coderanch.com/t/491883/java/java/IP><http://www.coderanch.com/t/491883/java/java/IP>
.
+ @Yi, @Navina,
Also, I think this fix should go to the 0.10.0 release.
What do you guys think?
Thanks,
Fang, Yan
yanfang...@gmail.com<mailto:yanfang...@gmail.com><mailto:yanfang...@gmail.com><mailto:yanfang...@gmail.com>
On Thu, Jul 30, 2015 at 6:39 PM, Yan Fang
<yanfang...@gmail.com><mailto:yanfang...@gmail.com><mailto:
yanfang...@gmail.com><mailto:yanfang...@gmail.com> wrote:
Just one point to add:
{quote}
AM gets notified of container status from the RM.
{quote}
I think this is not 100% correct. AM can communicate with NM through
NMClientAsync
<
https://hadoop.apache.org/docs/r2.7.1/api/org/apache/hadoop/yarn/client/api/async/NMClientAsync.html
<
https://hadoop.apache.org/docs/r2.7.1/api/org/apache/hadoop/yarn/client/api/async/NMClientAsync.html>
to
get container status, though Samza does not implement the CallbackHandler.
Thanks,
Fang, Yan
yanfang...@gmail.com<mailto:yanfang...@gmail.com><mailto:yanfang...@gmail.com><mailto:yanfang...@gmail.com>
On Thu, Jul 30, 2015 at 6:06 PM, Navina Ramesh <
nram...@linkedin.com.invalid<mailto:nram...@linkedin.com.invalid><mailto:nram...@linkedin.com.invalid><mailto:nram...@linkedin.com.invalid>>
wrote:
The NM (and hence, by extension the container) heartbeats to the RM, not
the AM. AM gets notified of container status from the RM.
The AM starts / stops /releases a container process by communicating to
the
NM.
Navina
On Thu, Jul 30, 2015 at 5:55 PM, Thomas Becker
<tobec...@tivo.com><mailto:tobec...@tivo.com><mailto:
tobec...@tivo.com><mailto:tobec...@tivo.com> wrote:
Ok, I thought there was some communication from the container to the AM,
it sounds like you're saying it's in the other direction only? Don't
containers heartbeat to the AM? Regardless, even if we can't get a
better
address for the AM from YARN, we could at least filter the addresses we
get
back from the JVM to exclude loopbacks.
-Tommy
________________________________________
From: Navina Ramesh
[nram...@linkedin.com.INVALID<mailto:nram...@linkedin.com.INVALID><mailto:
nram...@linkedin.com.INVALID><mailto:nram...@linkedin.com.INVALID>]
Sent: Thursday, July 30, 2015 8:40 PM
To:
dev@samza.apache.org<mailto:dev@samza.apache.org><mailto:dev@samza.apache.org><mailto:dev@samza.apache.org>
Subject: Re: Coordinator URL always 127.0.0.1
Hi Tommy,
Yi is right. Container start is coordinated by the AppMaster using an
NMClient. Container host name and port is provided by the RM during
allocation.
In Yarn (at least, afaik), when the node joins a cluster, the NM
registers
itself with the RM. So, the NM might still be using
getLocalhost.getAddress().
I don't know of any other way to programmatically fetch the machine's
hostname (apart from some hacky shell commands).
Cheers,
Navina
On Thu, Jul 30, 2015 at 5:23 PM, Yi Pan
<nickpa...@gmail.com><mailto:nickpa...@gmail.com><mailto:
nickpa...@gmail.com><mailto:nickpa...@gmail.com> wrote:
Hi, Tommy,
Yeah, I agree that the current implementation is not bullet-proof to
any
different networking configuration on the host. As for the AM <->
container
communication, if I am not mistaken, it is through the NMClient and
the
node HTTP address is wrapped within the Container object returned from
RM.
I am not very familiar with that part of source code. Navina may be
able
to
help more here.
-Yi
On Thu, Jul 30, 2015 at 4:27 PM, Thomas Becker
<tobec...@tivo.com><mailto:tobec...@tivo.com><mailto:
tobec...@tivo.com><mailto:tobec...@tivo.com>
wrote:
Hi Yi,
Thanks a lot for your reply. I don't doubt we can get it to work by
mucking with the networking configuration, but to me this feels
like a
workaround, not a solution.
InetAddress.getLocalHost().getHostAddress()
is
not a reliable way of obtaining an IP that other machines can
connect
to.
Just today I tested on several Linux distros and it did not work on
any
of
them. Can we do something more robust here? How does the container
communicate status to the AM?
-Tommy
________________________________________
From: Yi Pan
[nickpa...@gmail.com<mailto:nickpa...@gmail.com><mailto:nickpa...@gmail.com><mailto:nickpa...@gmail.com>]
Sent: Thursday, July 30, 2015 6:48 PM
To:
dev@samza.apache.org<mailto:dev@samza.apache.org><mailto:dev@samza.apache.org><mailto:dev@samza.apache.org>
Subject: Re: Coordinator URL always 127.0.0.1
Hi, Tommy,
I think that it might be a commonly asked question regarding to
multiple
IPs on a single host. A common trick w/o changing code is (copied
from
SO:
http://stackoverflow.com/questions/2381316/java-inetaddress-getlocalhost-returns-127-0-0-1-how-to-get-real-ip
)
{code}
1.
Find your host name. Type: hostname. For example, you find your
hostname
is mycomputer.xzy.com
2.
Put your host name in your hosts file. /etc/hosts . Such as
10.50.16.136 mycomputer.xzy.com
{code}
-Yi
On Thu, Jul 30, 2015 at 11:35 AM, Tommy Becker
<tobec...@tivo.com><mailto:tobec...@tivo.com><mailto:
tobec...@tivo.com><mailto:tobec...@tivo.com>
wrote:
We are testing some jobs on a YARN grid and noticed they are often
not
starting up properly due to being unable to connect to the job
coordinator.
After some investigation it seems as if the jobs are always
getting a
coordinator URL of http://127.0.0.1:<port> But my understanding
is
that
the coordinator runs only in the AM, so I'd expect these URLs to
more
often
than not be to some other machine. Looking at the code however,
I'm
not
sure how that would ever happen since the URL for the coordinator
always
comes from InetAddress.getLocalHost().getHostAddress() in
org.apache.samza.coordinator.server.HttpServer#getUrl
Am I off base here? Because I don't see how this is ever going to
work
in
scenarios where the AM is on a different node than the containers.
--
Tommy Becker
Senior Software Engineer
Digitalsmiths
A TiVo Company
www.digitalsmiths.com<http://www.digitalsmiths.com><http://www.digitalsmiths.com><http://www.digitalsmiths.com><
http://www.digitalsmiths.com><http://www.digitalsmiths.com><http://www.digitalsmiths.com><http://www.digitalsmiths.com>
tobec...@tivo.com<mailto:tobec...@tivo.com><mailto:tobec...@tivo.com><mailto:tobec...@tivo.com><mailto:tobec...@tivo.com
<mailto:tobec...@tivo.com><mailto:tobec...@tivo.com>
________________________________
This email and any attachments may contain confidential and
privileged
material for the sole use of the intended recipient. Any review,
copying,
or distribution of this email (or any attachments) by others is
prohibited.
If you are not the intended recipient, please contact the sender
immediately and permanently delete this email and any
attachments. No
employee or agent of TiVo Inc. is authorized to conclude any
binding
agreement on behalf of TiVo Inc. by email. Binding agreements with
TiVo
Inc. may only be made by a signed written agreement.
________________________________
This email and any attachments may contain confidential and
privileged
material for the sole use of the intended recipient. Any review,
copying,
or distribution of this email (or any attachments) by others is
prohibited.
If you are not the intended recipient, please contact the sender
immediately and permanently delete this email and any attachments.
No
employee or agent of TiVo Inc. is authorized to conclude any binding
agreement on behalf of TiVo Inc. by email. Binding agreements with
TiVo
Inc. may only be made by a signed written agreement.
--
Navina R.
________________________________
This email and any attachments may contain confidential and privileged
material for the sole use of the intended recipient. Any review,
copying,
or distribution of this email (or any attachments) by others is
prohibited.
If you are not the intended recipient, please contact the sender
immediately and permanently delete this email and any attachments. No
employee or agent of TiVo Inc. is authorized to conclude any binding
agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
Inc. may only be made by a signed written agreement.
--
Navina R.
--
Tommy Becker
Senior Software Engineer
Digitalsmiths
A TiVo Company
www.digitalsmiths.com<http://www.digitalsmiths.com><http://www.digitalsmiths.com><http://www.digitalsmiths.com>
tobec...@tivo.com<mailto:tobec...@tivo.com><mailto:tobec...@tivo.com><mailto:tobec...@tivo.com>
________________________________
This email and any attachments may contain confidential and privileged
material for the sole use of the intended recipient. Any review, copying,
or distribution of this email (or any attachments) by others is prohibited.
If you are not the intended recipient, please contact the sender
immediately and permanently delete this email and any attachments. No
employee or agent of TiVo Inc. is authorized to conclude any binding
agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
Inc. may only be made by a signed written agreement.
--
Tommy Becker
Senior Software Engineer
Digitalsmiths
A TiVo Company
www.digitalsmiths.com<http://www.digitalsmiths.com>
tobec...@tivo.com<mailto:tobec...@tivo.com>
________________________________
This email and any attachments may contain confidential and privileged material
for the sole use of the intended recipient. Any review, copying, or
distribution of this email (or any attachments) by others is prohibited. If you
are not the intended recipient, please contact the sender immediately and
permanently delete this email and any attachments. No employee or agent of TiVo
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by
email. Binding agreements with TiVo Inc. may only be made by a signed written
agreement.
