Review Request 35492: Port docker setup from hadoop-common

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35492/ --- Review request for samza. Repository: samza-hello-samza Description ---

Re: Review Request 35492: SAMZA-701 : Port docker setup from hadoop-common

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35492/ --- (Updated June 16, 2015, 7:35 a.m.) Review request for samza. Summary (updated

Re: Review Request 35492: SAMZA-701 : Hello Samza - Port docker setup from hadoop-common

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35492/ --- (Updated June 16, 2015, 7:36 a.m.) Review request for samza. Summary (updated

Re: Review Request 35067: SAMZA-546: Add config option max.in.flight.requests.per.connection

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35067/ --- (Updated June 16, 2015, 7:47 a.m.) Review request for samza. Repository: samz

Review Request 35515: SAMZA-449 Expose RocksDB statistic

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35515/ --- Review request for samza. Repository: samza Description --- RocksDB stat

Re: 3 processed message per incoming message

This is what I see on Yarn monitoring page: As we can see, there are 9998 apps pending. There is some 10k limit we are hitting. I see only 1 app running. Apps SubmittedApps PendingApps RunningApps CompletedContainers RunningMemory UsedMemory TotalMemory ReservedVCores UsedVCores TotalVCores Rese

[DISCUSS] Samza 0.9.1 release

Hi all, We have been running a couple of our jobs against `0.9.1` branch last week at LinkedIn with some critical bug fixes back-ported, including: SAMZA-608 Deserialization error causes SystemConsumers to hang SAMZA-616 Shutdown hook does not wait for container to finish SAMZA-658 Iterator.rem

Re: 3 processed message per incoming message

Hi Shekar, Ok. If there is only one application is running, if you kill this one, will you still be able to see the processed messages coming? If not, I think the code in your application maybe the cause of the problem. We can have a further look at your code to see where the problem is. Thanks,

Re: [DISCUSS] Samza 0.9.1 release

Agreed on this. Thanks, Fang, Yan yanfang...@gmail.com On Tue, Jun 16, 2015 at 10:14 AM, Guozhang Wang wrote: > Hi all, > > We have been running a couple of our jobs against `0.9.1` branch last week > at LinkedIn with some critical bug fixes back-ported, including: > > SAMZA-608 > Deserializat

Re: Review Request 35397: Fix SAMZA-697

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35397/ --- (Updated June 16, 2015, 5:22 p.m.) Review request for samza. Bugs: SAMZA-697

Re: [DISCUSS] Samza 0.9.1 release

Wang, I have not caught up but can you please highlight if there are any feature additions as well? - Shekar On Tue, Jun 16, 2015 at 10:14 AM, Guozhang Wang wrote: > Hi all, > > We have been running a couple of our jobs against `0.9.1` branch last week > at LinkedIn with some critical bug fixe

improving hello-samza / testing

I'm learning samza by the hello-samza project and notice the lack of tests. Where's a good place to learn how folks are properly testing things written with samza? Thanks, --tim

Re: [DISCUSS] Samza 0.9.1 release

+1 Agreed. Thanks! On Tue, Jun 16, 2015 at 10:15 AM, Yan Fang wrote: > Agreed on this. > > Thanks, > > Fang, Yan > yanfang...@gmail.com > > On Tue, Jun 16, 2015 at 10:14 AM, Guozhang Wang > wrote: > > > Hi all, > > > > We have been running a couple of our jobs against `0.9.1` branch last > wee

Re: [DISCUSS] Samza 0.9.1 release

Hi, Shekar, This 0.9.1 is a bug-fix only release. No features added yet. New features are expected in 0.10.0. Thanks! On Tue, Jun 16, 2015 at 10:59 AM, Shekar Tippur wrote: > Wang, > > I have not caught up but can you please highlight if there are any feature > additions as well? > > - Shekar

Re: [DISCUSS] Samza 0.9.1 release

Hi Shekar, 0.9.1 is a bug-fix release and does not have additional features compared with 0.9.0: https://issues.apache.org/jira/browse/SAMZA/fixforversion/12332405/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel Guozhang On Tue, Jun 16, 2015 at 10:59 AM, Shekar Tippur

Re: [DISCUSS] Samza 0.9.1 release

+1 Thanks. 2015-06-16 14:15 GMT-03:00 Yan Fang : > Agreed on this. > > Thanks, > > Fang, Yan > yanfang...@gmail.com > > On Tue, Jun 16, 2015 at 10:14 AM, Guozhang Wang > wrote: > > > Hi all, > > > > We have been running a couple of our jobs against `0.9.1` branch last > week > > at LinkedIn wit

Re: [DISCUSS] Samza 0.9.1 release

+1 for the release! On 6/16/15, 11:03 AM, "Yi Pan" wrote: >+1 Agreed. > >Thanks! > >On Tue, Jun 16, 2015 at 10:15 AM, Yan Fang wrote: > >> Agreed on this. >> >> Thanks, >> >> Fang, Yan >> yanfang...@gmail.com >> >> On Tue, Jun 16, 2015 at 10:14 AM, Guozhang Wang >> wrote: >> >> > Hi all, >> >

Re: [DISCUSS] Samza 0.9.1 release

+1 On Tue, Jun 16, 2015 at 11:17 AM, Navina Ramesh < nram...@linkedin.com.invalid> wrote: > +1 for the release! > > On 6/16/15, 11:03 AM, "Yi Pan" wrote: > > >+1 Agreed. > > > >Thanks! > > > >On Tue, Jun 16, 2015 at 10:15 AM, Yan Fang wrote: > > > >> Agreed on this. > >> > >> Thanks, > >> > >>

Re: [DISCUSS] Samza 0.9.1 release

Cool. I will start a voting process soon. On Tue, Jun 16, 2015 at 11:55 AM, Chinmay Soman wrote: > +1 > > On Tue, Jun 16, 2015 at 11:17 AM, Navina Ramesh < > nram...@linkedin.com.invalid> wrote: > > > +1 for the release! > > > > On 6/16/15, 11:03 AM, "Yi Pan" wrote: > > > > >+1 Agreed. > > > >

Re: Review Request 34974: SAMZA-676: implement broadcast stream

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34974/ --- (Updated June 16, 2015, 9:06 p.m.) Review request for samza. Changes ---

Re: Review Request 34974: SAMZA-676: implement broadcast stream

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34974/ --- (Updated June 16, 2015, 9:16 p.m.) Review request for samza. Changes ---

Re: Review Request 34974: SAMZA-676: implement broadcast stream

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34974/ --- (Updated June 16, 2015, 9:23 p.m.) Review request for samza. Changes ---

Re: [DISCUSS] Samza 0.9.1 release

+1 Here. On Tue, Jun 16, 2015 at 12:01 PM, Guozhang Wang wrote: > Cool. I will start a voting process soon. > > On Tue, Jun 16, 2015 at 11:55 AM, Chinmay Soman > > wrote: > > > +1 > > > > On Tue, Jun 16, 2015 at 11:17 AM, Navina Ramesh < > > nram...@linkedin.com.invalid> wrote: > > > > > +1 for

Re: improving hello-samza / testing

Hey Tim, This is a really good discussion to have. The testing that I've seen with Samza falls into two categories: 1. Instantiate your StreamTask, and mock all params in the process()/init() methods. 2. A mini-ontegration test that starts ZooKeeper, and Kafka, and feeds messages into a topic, an

Re: Review Request 35241: refactoring the code for coordinator stream writer

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35241/#review88174 --- samza-core/src/main/java/org/apache/samza/coordinator/stream/Coordi

Powered by page update

Hey all, I'm seeing a lot of new faces on the mailing list, which is really awesome. I want to invite you all to add yourselves to our Powered by page: https://cwiki.apache.org/confluence/display/SAMZA/Powered+By The Apache wiki is pretty locked down due to spam. If you'd like to send me a link

Re: improving hello-samza / testing

We've built a driver program which kinda falls along approach (1) listed in your email. The driver program accepts a custom task object and has a way to inject data -> which in turn invokes the process method. For now we're assuming logical time and use the frequency of process() invocations to de

Re: [DISCUSS] Samza 0.9.1 release

Thank you! Sent using CloudMagic On Tue, Jun 16, 2015 at 8:11 PM, Chris Riccomini wrote: +1 Here. On Tue, Jun 16, 2015 at 12:01 PM, Guozhang Wang wrote: > Cool. I will start a voting process soon. > > On Tue, Jun 16, 2015 at 11:55 AM,

Re: [DISCUSS] Samza 0.9.1 release

+1 On Jun 16, 2015 6:39 PM, "Percy Wegmann" wrote: > Thank you! > > Sent using CloudMagic< > https://cloudmagic.com/k/d/mailapp?ct=pi&cv=6.0.64&pv=8.2> > > On Tue, Jun 16, 2015 at 8:11 PM, Chris Riccomini > wrote: > +1 Here. > > On Tue, Jun 16, 2015 at 12:01 PM, Guozhang Wang > wrote: > > > Coo

Re: [DISCUSS] Samza 0.9.1 release

+1 On Tue, Jun 16, 2015 at 7:41 PM, Shekar Tippur wrote: > +1 > On Jun 16, 2015 6:39 PM, "Percy Wegmann" > wrote: > > > Thank you! > > > > Sent using CloudMagic< > > https://cloudmagic.com/k/d/mailapp?ct=pi&cv=6.0.64&pv=8.2> > > > > On Tue, Jun 16, 2015 at 8:11 PM, Chris Riccomini > > wrote: >

Re: [DISCUSS] Samza 0.9.1 release

+1 > On Jun 16, 2015, at 7:41 PM, Shekar Tippur wrote: > > +1 > On Jun 16, 2015 6:39 PM, "Percy Wegmann" wrote: > >> Thank you! >> >> Sent using CloudMagic< >> https://cloudmagic.com/k/d/mailapp?ct=pi&cv=6.0.64&pv=8.2> >> >> On Tue, Jun 16, 2015 at 8:11 PM, Chris Riccomini >> wrote: >> +1

Re: improving hello-samza / testing

https://issues.apache.org/jira/browse/SAMZA-681 tracks the first effort towards the driver program or unit test harness for samza tasks that Chinmay is referring to. ᐧ On Tue, Jun 16, 2015 at 6:11 PM, Chinmay Soman wrote: > We've built a driver program which kinda falls along approach (1) listed

Measuring Samza Job Throughput

Hi Devs, I was looking for a way to measure Samza job throughput and found that its possible to do it via Samza's metrics reporter. But there several types of metrics reported via this method. For example, TaskInstanceMetrics reports number of messages sent. But if I wanted to get a measurement li

Re: Measuring Samza Job Throughput

Hey Milinda, Specifically, for bytes/sec, you might want to look at serde metrics. I believe the serde manager tracks bytes serialized and deserialized per second. The consumers and producers also do this for Kafka, but on a more granular basis. If you want container-level throughput, serde manage

Re: Measuring Samza Job Throughput

Hmm, correction. I think this has to be done at tbhe KafkaSystem level. We allow consumers and producers to return non-byte messages, which means nothing in container can safely assume that a message is a byte array except the serde manager. I took a look there but didn't see any byte throughout me

Re: Measuring Samza Job Throughput

Hi, One metric I could think of related to Samza job throughput is the "process-envelop" metric listed in SamzaContainerMetrics. This counter get incremented whenever the container process meaningful message( https://github.com/apache/samza/blob/master/samza-core/src/main/scala/org/apache/samza/co