Metrics for Elasticsearch System Producer

Hi all, I've started using the new Elasticsearch System Producer (many thanks, Dan!) and decided to add some metrics to it. The JIRA ticket and review request links are here: https://issues.apache.org/jira/browse/SAMZA-733 https://reviews.apache.org/r/36473/ Cheers, Roger

Review Request 36473: SAMZA-733 Add metrics to Elasticsearch System Producer

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36473/ --- Review request for samza. Repository: samza Description --- SAMZA-733 Ad

Re: Review Request 36471: Autoscaling for samza (work in progress)

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36471/ --- (Updated July 14, 2015, 1:25 a.m.) Review request for samza and Navina Ramesh.

Review Request 36471: added stream for auto scaling, consumer to read from the stream in profiler, sliding window metric

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36471/ --- Review request for samza and Navina Ramesh. Repository: samza Description ---

Re: Thoughts and obesrvations on Samza

Hi, Garry, Just want to chime in to state our experience in LinkedIn. In LinkedIn, we have a lot of aggregation/transformation stream processing jobs that falls into the "transformation" category. That's also the motivation for us to develop the SQL layer on top of streams to allow easy programmin

You're invited to an Apache Samza meetup hosted at LinkedIn on July 21 @ 6PM

Hi all - The Samza development team invites you to join us at an Apache Samza meetup hosted in the Unite conference room at LinkedIn's Mountain View campus on Tuesday, July 21 at 6PM. Food/drinks and streaming video will be provided. Please RSVP here

Re: Thoughts and obesrvations on Samza

Hi, Jay, Given all the user concerns, the board disagreement on sub-projects, I am supporting your 5th option as well. As you said, even the end goal is the same, it might help to pave a smooth path forward. One thing I learned over the years is that what we planned for may not be the final produc

Re: Review Request 36224: SAMZA-728: Samza job fails due to null pointer in JobCoordinator refreshJobModel

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36224/ --- (Updated July 13, 2015, 10:22 p.m.) Review request for samza, Yan Fang, Naveen

Re: Review Request 36224: SAMZA-728: Samza job fails due to null pointer in JobCoordinator refreshJobModel

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36224/ --- (Updated July 13, 2015, 10:23 p.m.) Review request for samza, Yan Fang, Naveen

Re: Review Request 36089: SAMZA-670 Allow easier access to JMX port

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36089/#review91533 --- samza-core/src/main/scala/org/apache/samza/container/SamzaContainer

Re: Review Request 36274: SAMZA-401: getCpuTime to truly calculate duty cycle of the event loop

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36274/#review91527 --- samza-core/src/main/scala/org/apache/samza/container/RunLoop.scala

Re: Thoughts and obesrvations on Samza

>From peanut gallery.. I like Yi's proposal in re-scoping the Samza project / code-base as "Stream Processing as a Service" that will potentially include: 1. A service manager with some REST / Web UI to accept stream processing jobs in terms of tgz / configs and schedule them as for: a. partitio

Re: Thoughts and obesrvations on Samza

I am leaning to Jay's fifth approach. It is not radical and gives us some time to see the outcome. In addition, I would suggest: 1) Keep the SystemConsumer/SystemProducer API. Because current SystemConsumer/SystemProducer API satisfies the usage (From Joardan, and even Garry's feedback) and is no

Re: Thoughts and obesrvations on Samza

Jay, I think doing this iteratively in smaller chunks is a better way to go as new issues arise. As Navina said Kafka is a "stream system" and Samza is a "stream processor" and those two ideas should be mutually exclusive. -Jordan On Mon, Jul 13, 2015 at 10:06 AM, Jay Kreps wrote: > Hmm, though

Re: Question about sub-projects and project merging

Hey Mike, Thanks for sharing, it is helpful to hear the experience that leads to these recommendations. -Jay On Mon, Jul 13, 2015 at 11:01 AM, Mike Kienenberger wrote: > A subproject is one of many projects that fall under the same umbrella > project management committee (PMC). It doesn't ha

Re: Question about sub-projects and project merging

A subproject is one of many projects that fall under the same umbrella project management committee (PMC). It doesn't have to be a separate repo, but it generally has a separate community or a subset of the full community. Speaking as a long-time PMC member for MyFaces, our problem with subproje

Re: Thoughts and obesrvations on Samza

Hmm, thought about this more. Maybe this is just too much too quick. Overall I think there is some enthusiasm for the proposal but it's not really unanimous enough to make any kind of change this big cleanly. The board doesn't really like the merging stuff, user's are concerned about compatibility,

copycat status

Howdy devs, Does code exist somewhere for KIP-26? I thought it was just a proposal, but the talk around here seems like there might actually be real code somewhere? I'm interested in having a "sink" to Apache Blur and a kite-dataset and it seems that this new copycat would be a promising approach

Re: Question about sub-projects and project merging

Hi Jay, Looking at your question, I see the Apache Samza and Apache Kafka *communities* have little overlap(*). The Board looks at communities, and their overlap or lack thereof. Smushing two communities under one TLP is what we have historically called an "umbrella" TLP, and discourage. Communiti