Re: Coordinator URL always 127.0.0.1

+1 for the fix! On Jul 30, 2015 9:55 PM, "Navina Ramesh" wrote: > Yes, Yan. But that communication is initiated by the AM. Whether an > application's AM does it or not, the NM always heartbeats the status of > its containers to the RM. > On Jul 30, 2015 6:40 PM, "Yan Fang" wrote: > >> Just one

Re: Coordinator URL always 127.0.0.1

Yes, Yan. But that communication is initiated by the AM. Whether an application's AM does it or not, the NM always heartbeats the status of its containers to the RM. On Jul 30, 2015 6:40 PM, "Yan Fang" wrote: > Just one point to add: > > {quote} > AM gets notified of container status from the RM

Re: Review Request 36471: Autoscaling for samza (work in progress)

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36471/ --- (Updated July 31, 2015, 4:24 a.m.) Review request for samza and Navina Ramesh.

Re: Review Request 36471: Autoscaling for samza (work in progress)

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36471/ --- (Updated July 31, 2015, 4:22 a.m.) Review request for samza and Navina Ramesh.

Review Request 36973: SAMZA-626: tool to read the RocksDb in a running job

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36973/ --- Review request for samza. Bugs: SAMZA-626 https://issues.apache.org/jira/br

Re: Coordinator URL always 127.0.0.1

Created https://issues.apache.org/jira/browse/SAMZA-748 Fang, Yan yanfang...@gmail.com On Thu, Jul 30, 2015 at 7:17 PM, Yi Pan wrote: > +1 on the fix in 0.10.0. It should be an easy one. > > On Thu, Jul 30, 2015 at 7:08 PM, Yan Fang wrote: > > > Hi Thommy, > > > > {quote} > > Because I don't s

Re: Coordinator URL always 127.0.0.1

+1 on the fix in 0.10.0. It should be an easy one. On Thu, Jul 30, 2015 at 7:08 PM, Yan Fang wrote: > Hi Thommy, > > {quote} > Because I don't see how this is ever going to work in scenarios where the > AM is on a different node than the containers. > {quote} > > -- I do not quite understand thi

Re: Coordinator URL always 127.0.0.1

Hi Thommy, {quote} Because I don't see how this is ever going to work in scenarios where the AM is on a different node than the containers. {quote} -- I do not quite understand this part. AM essentially is running in a container as well. And the http server is brought up in the same container. {

Re: [DISCUSS] Release 0.10.0

For SAMZA-747, may ping Naveen or Chris. :) They have the permission to publish to maven. From the discuss , they seem ready for the release. Thanks, Fang, Yan yanfang...@gmail.com On Thu, Jul 30, 2015 at 5:35 PM, Navina Ramesh wrote: > Ok. I th

Re: Coordinator URL always 127.0.0.1

Just one point to add: {quote} AM gets notified of container status from the RM. {quote} I think this is not 100% correct. AM can communicate with NM through NMClientAsync to get container statu

Re: Coordinator URL always 127.0.0.1

The NM (and hence, by extension the container) heartbeats to the RM, not the AM. AM gets notified of container status from the RM. The AM starts / stops /releases a container process by communicating to the NM. Navina On Thu, Jul 30, 2015 at 5:55 PM, Thomas Becker wrote: > Ok, I thought there

RE: Coordinator URL always 127.0.0.1

Ok, I thought there was some communication from the container to the AM, it sounds like you're saying it's in the other direction only? Don't containers heartbeat to the AM? Regardless, even if we can't get a better address for the AM from YARN, we could at least filter the addresses we get ba

Re: Coordinator URL always 127.0.0.1

Hi Tommy, Yi is right. Container start is coordinated by the AppMaster using an NMClient. Container host name and port is provided by the RM during allocation. In Yarn (at least, afaik), when the node joins a cluster, the NM registers itself with the RM. So, the NM might still be using getLocalhost

Re: [DISCUSS] Release 0.10.0

Ok. I think it got confusing because you are talking about tickets NOT to be included in the release :) Got it, now. +1 for this list of exclusions! SAMZA-723 (StreamAppender bug) and SAMZA-747 (rocksdb) should be in 0.10.0. I think we don't have an ETA on the fix for SAMZA-747? Thanks! Navina O

Re: [DISCUSS] Release 0.10.0

Uh... wrong math all the day today... :( Let me re-try: 29/32 tickets are to be moved to later (i.e. excluded from 0.10.0). 2/32 tickets (including SAMZA-723) will be included 1/32 ticket (SAMZA-689) is up for discussion and I am leaning toward mark it as won't fix. -Yi On Thu, Jul 30, 2015 a

Re: Coordinator URL always 127.0.0.1

Hi, Tommy, Yeah, I agree that the current implementation is not bullet-proof to any different networking configuration on the host. As for the AM <-> container communication, if I am not mistaken, it is through the NMClient and the node HTTP address is wrapped within the Container object returned

Re: [DISCUSS] Release 0.10.0

Hi, Navina, The 29/30 tickets are to be excluded from 0.10.0 (i.e. moved to 0.11.0), the three tickets are either to be included in 0.10.0, or won't fix. -Yi On Thu, Jul 30, 2015 at 5:10 PM, Navina Ramesh wrote: > Hi Yi, > Thanks for summarizing. But why are we excluding SAMZA-723 from the cur

Re: [DISCUSS] Release 0.10.0

Hi Yi, Thanks for summarizing. But why are we excluding SAMZA-723 from the current release ? Doesn't this break the existing StreamAppender functionality in 0.9? Thanks! Navina On Thu, Jul 30, 2015 at 4:55 PM, Yi Pan wrote: > Sorry, hit the send button too fast. Let me correct the summary secti

Re: [DISCUSS] Release 0.10.0

Sorry, hit the send button too fast. Let me correct the summary section: 29/32 tickets Status=open/reopen : https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(Open%2C%20Reopened)%20AND%20fixVersion%20%3D%200.10.0%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C

Re: [DISCUSS] Release 0.10.0

Hi, all, Thanks a lot for helping out to select the features planned in 0.10.0. Based on the above discussion, I am proposing to move the following tickets later (i.e. 0.11.0). 30/32 tickets Status=open/reopen (exception for SAMZA-723: stream appender deadlock issue) : https://issues.apache.org/

RE: Coordinator URL always 127.0.0.1

Hi Yi, Thanks a lot for your reply. I don't doubt we can get it to work by mucking with the networking configuration, but to me this feels like a workaround, not a solution. InetAddress.getLocalHost().getHostAddress() is not a reliable way of obtaining an IP that other machines can connect to.

Re: Coordinator URL always 127.0.0.1

Hi, Tommy, I think that it might be a commonly asked question regarding to multiple IPs on a single host. A common trick w/o changing code is (copied from SO: http://stackoverflow.com/questions/2381316/java-inetaddress-getlocalhost-returns-127-0-0-1-how-to-get-real-ip ) {code} 1. Find you

Re: Review Request 35445: SAMZA-693: Very basic HDFS Producer service for Samza

> On July 30, 2015, 6:59 p.m., Yan Fang wrote: > > samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/JobNameDateTimeBucketer.scala, > > line 40 > > > > > > My overall concern here is that, if there are mo

Re: samza environment variable on containers

Hi Chen Song, I do not think there is a way in Samza with which you can specify the ENV for Samza container. And currently Samza does not read the LD_LIBRARY_PATH either. Samza only puts the files in lib/*.[jw]ar into the CLASSPATH. Though -Djava.library.path might work, it will cause hadoop e

Re: Review Request 35445: SAMZA-693: Very basic HDFS Producer service for Samza

> On July 30, 2015, 6:59 p.m., Yan Fang wrote: > > samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/JobNameDateTimeBucketer.scala, > > line 40 > > > > > > My overall concern here is that, if there are mo

Re: Review Request 35445: SAMZA-693: Very basic HDFS Producer service for Samza

> On July 30, 2015, 6:59 p.m., Yan Fang wrote: > > samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/JobNameDateTimeBucketer.scala, > > line 55 > > > > > > this will very possibly never be executed, becau

Re: Review Request 35445: SAMZA-693: Very basic HDFS Producer service for Samza

> On July 30, 2015, 6:59 p.m., Yan Fang wrote: > > samza-hdfs/src/test/scala/org/apache/samza/system/hdfs/TestHdfsSystemProducerTestSuite.scala, > > line 106 > > > > > > do we need to bring up the MiniCluster in eve

Re: Review Request 35445: SAMZA-693: Very basic HDFS Producer service for Samza

> On July 30, 2015, 6:59 p.m., Yan Fang wrote: > > samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/BinarySequenceFileHdfsWriter.scala, > > line 38 > > > > > > can we use the "camel case with an initial

Re: Review Request 35445: SAMZA-693: Very basic HDFS Producer service for Samza

> On July 30, 2015, 6:59 p.m., Yan Fang wrote: > > samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/TextSequenceFileHdfsWriter.scala, > > line 37 > > > > > > I would prefer the param idea because 1) Samz

Re: Review Request 35445: SAMZA-693: Very basic HDFS Producer service for Samza

> On July 29, 2015, 7:41 p.m., Navina Ramesh wrote: > > docs/learn/documentation/versioned/hdfs/producer.md, line 24 > > > > > > Can you please update the list of available configuration for this > > system at > > h

Re: Review Request 35445: SAMZA-693: Very basic HDFS Producer service for Samza

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35445/#review93614 --- docs/learn/documentation/versioned/hdfs/producer.md (line 33)

Coordinator URL always 127.0.0.1

We are testing some jobs on a YARN grid and noticed they are often not starting up properly due to being unable to connect to the job coordinator. After some investigation it seems as if the jobs are always getting a coordinator URL of http://127.0.0.1: But my understanding is that the coordin

Re: Review Request 36545: SAMZA-682 Refactor Coordinator stream messages

> On July 24, 2015, 6:01 p.m., Navina Ramesh wrote: > > Thanks for picking this up! It feels good to look at a refactored code. > > > > One suggestion: Please run all the intergration test (including the zopkio > > tests) before checking in this patch. I don't think we cleanly start and > > s

Re: Review Request 35445: SAMZA-693: Very basic HDFS Producer service for Samza

> On July 29, 2015, 7:41 p.m., Navina Ramesh wrote: > > docs/learn/documentation/versioned/hdfs/producer.md, line 24 > > > > > > Can you please update the list of available configuration for this > > system at > > h

Re: Review Request 35445: SAMZA-693: Very basic HDFS Producer service for Samza

> On July 29, 2015, 7:41 p.m., Navina Ramesh wrote: > > samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/TextSequenceFileHdfsWriter.scala, > > line 37 > > > > > > What is the motivation for having 2 sequ

Re: Review Request 35445: SAMZA-693: Very basic HDFS Producer service for Samza

> On July 29, 2015, 7:41 p.m., Navina Ramesh wrote: > > samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/TextSequenceFileHdfsWriter.scala, > > line 37 > > > > > > What is the motivation for having 2 sequ

Re: Review Request 35445: SAMZA-693: Very basic HDFS Producer service for Samza

> On July 29, 2015, 7:41 p.m., Navina Ramesh wrote: > > docs/learn/documentation/versioned/hdfs/producer.md, line 24 > > > > > > Can you please update the list of available configuration for this > > system at > > h

Re: Review Request 36545: SAMZA-682 Refactor Coordinator stream messages

> On July 24, 2015, 6:01 p.m., Navina Ramesh wrote: > > Thanks for picking this up! It feels good to look at a refactored code. > > > > One suggestion: Please run all the intergration test (including the zopkio > > tests) before checking in this patch. I don't think we cleanly start and > > s

samza environment variable on containers

Maybe a dumb question. Is there a way to set an ENV for samza containers? We want to set LD_LIBRARY_PATH to include hadoop native libs. -- Chen Song

Re: Security on YARN

Thanks Yan. I will take a look shortly. On Sat, Jul 25, 2015 at 1:20 AM, Yan Fang wrote: > Hi Chen Song, > > If you can work on this issue, it will be great. > > 1. the related ticket is https://issues.apache.org/jira/browse/SAMZA-727 > > 2. most of the change will happen in Yarn AM and Yarn cli

Re: Review Request 36545: SAMZA-682 Refactor Coordinator stream messages

> On July 24, 2015, 6:01 p.m., Navina Ramesh wrote: > > Thanks for picking this up! It feels good to look at a refactored code. > > > > One suggestion: Please run all the intergration test (including the zopkio > > tests) before checking in this patch. I don't think we cleanly start and > > s