Re: job versioning

2015-09-19 Thread Gian Merlino
Hey Richard, The ApplicationReport returned by YarnClient.getApplications() or getApplicationReport(appId) includes the AM host and rpc port. https://hadoop.apache.org/docs/r2.7.0/api/org/apache/hadoop/yarn/client/api/YarnClient.html#getApplications() https://hadoop.apache.org/docs/r2.7.0/api/org

Re: How should Samza be run on AWS?

2015-08-05 Thread Gian Merlino
mzaberunonAWS > ? > How should Samza be run on AWS? > > From Gian Merlino: > >- We've been using Samza in production on AWS for a little over a > month. We're >just using the YARN runner on a mostly stock hadoop 2.4.0 cluster (not >EMR). Our experience

Re: Producer performance in 0.9.0

2015-04-07 Thread Gian Merlino
ome tests, and found that the new producer to > be faster than the old producer default (sync), and almost as fast as the > old producer's async producer. > > Could you paste all of your configs? > > Cheers, > Chris > > On Tue, Apr 7, 2015 at 10:40 AM, Gian Merlino &

Producer performance in 0.9.0

2015-04-07 Thread Gian Merlino
Has anyone else seen issues with producer performance in 0.9.0? I updated a few of our jobs recently and ended up rolling one back to 0.8 since it was being really sluggish. I profiled it for a bit and a lot of time was being spent in BufferPool.allocate and the busy-loop in KafkaSystemProducer's

Re: Samza questions

2015-03-26 Thread Gian Merlino
Hi Ori, Maybe an example would be useful. We use Samza to transform data for materialization in Druid, because Druid is built to index and aggregate a single event stream, but our raw data actually exists in a bunch of streams and tables that need joining. So we have Samza handle the joining and t

Re: Your experience running Samza in AWS?

2015-02-14 Thread Gian Merlino
Hi Geoffry, We've been using Samza in production on AWS for a little over a month. We're just using the YARN runner on a mostly stock hadoop 2.4.0 cluster (not EMR). Our experience is that c3s work well for the YARN instances and i2s work well for the Kafka instances. Things have been pretty solid