How do you monitor Samza?

One of the challenges we've faced with deploying Samza is effectively monitoring it. We currently use the MetricsSnapshotReporter and consume the resulting stream pulling out specific values that we want to monitor, but this approach isn't scalable. We're exploring pumping the metrics into an E

Re: Review Request 47835: SAMZA-914: Initial draft for Java programming APIs on operators supporting DAGs

> On Sept. 1, 2016, 5:58 p.m., Jake Maes wrote: > > samza-operator/src/main/java/org/apache/samza/operators/api/MessageStream.java, > > line 47 > > > > > > Does MessageStream need to have the key type parameter? > >

Re: Review Request 47835: SAMZA-914: Initial draft for Java programming APIs on operators supporting DAGs

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47835/ --- (Updated Sept. 12, 2016, 5:53 p.m.) Review request for samza, Boris Shkolnik, C

Re: Review Request 47835: SAMZA-914: Initial draft for Java programming APIs on operators supporting DAGs

> On Aug. 26, 2016, 7:34 p.m., Chris Pettitt wrote: > > samza-operator/src/main/java/org/apache/samza/operators/api/window/Trigger.java, > > line 28 > > > > > > Definitely not digging the mutable state here. > >

Re: How do you monitor Samza?

Hey Tommy, At Movio, we feed all of our Samza metrics into Prometheus and use Grafana to create visual dashboards. We configure all of our jobs to produce metrics into a single topic and we have a monitoring tool that consumes that topic and exposes the data to Prometheus ( https://github.com/movi

Review Request 51819: SAMZA-1015 Add support for streams, Lambdas and other JDK 8 like features in checkstyle

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51819/ --- Review request for samza, Boris Shkolnik, Chris Pettitt, Jake Maes, Navina Rames

Re: Review Request 51819: SAMZA-1015 Add support for streams, Lambdas and other JDK 8 like features in checkstyle

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51819/#review148585 --- Ship it! Ship It! - Xinyu Liu On Sept. 12, 2016, 7:28 p.m.,

Review Request 51824: SAMZA-1016 Fix support for 'joint compilation' of scala and java sources

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51824/ --- Review request for samza. Repository: samza Description --- The Samza gr

Re: Review Request 51824: SAMZA-1016 Fix support for 'joint compilation' of scala and java sources

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51824/ --- (Updated Sept. 12, 2016, 9:21 p.m.) Review request for samza, Boris Shkolnik, C

Re: Review Request 51824: SAMZA-1016 Fix support for 'joint compilation' of scala and java sources

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51824/ --- (Updated Sept. 12, 2016, 9:23 p.m.) Review request for samza, Boris Shkolnik, C

Re: Review Request 51613: Fix SAMZA-842

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51613/#review148593 --- Ship it! lgtm! - Yi Pan (Data Infrastructure) On Sept. 2, 20

Re: Review Request 51346: SAMZA-974 - Support finite datasources in Samza that have a notion of End-Of-Stream

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51346/ --- (Updated Sept. 12, 2016, 9:25 p.m.) Review request for samza, Boris Shkolnik, C

Re: Review Request 51346: SAMZA-974 - Support finite datasources in Samza that have a notion of End-Of-Stream

> On Sept. 8, 2016, 6:24 p.m., Xinyu Liu wrote: > > samza-core/src/main/scala/org/apache/samza/system/SystemConsumers.scala, > > line 210 > > > > > > nit: no ";" at the end for scala fixed - Jagadish --

Re: Review Request 51346: SAMZA-974 - Support finite datasources in Samza that have a notion of End-Of-Stream

> On Sept. 8, 2016, 5:03 p.m., Xinyu Liu wrote: > > samza-core/src/main/scala/org/apache/samza/system/SystemConsumers.scala, > > line 43 > > > > > > seems this is only needed for unit test. Please move the code to sc

Re: Review Request 51126: SAMZA 998: Documentation updates for refactored Job Coordinator

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51126/#review148601 --- docs/learn/documentation/versioned/jobs/configuration-table.html

Re: How do you monitor Samza?

Thanks for sharing Nicolas, This is super-useful. On Mon, Sep 12, 2016 at 10:56 AM, Nicolas Maquet wrote: > Hey Tommy, > > At Movio, we feed all of our Samza metrics into Prometheus and use Grafana > to create visual dashboards. We configure all of our jobs to produce > metrics into a single top

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51142/#review148612 --- Still in the middle but don't want to lose what I had up to now.

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51142/#review148629 --- I am at HdfsReaderFactory. Will continue tomorrow. samza-hdfs/sr

Question about Samza Metrics

Hi, I found most metrics of Samza are accumulative, is there a way to get metrics in a certain time frame. Thanks ShuQi