Re: Samza not consuming

Strange. I am unable to comment on the behavior because I don't know what your checkpoints looked like in the checkpoint topic. Did you try reading the checkpoint topic log ? If you setting systems.kafka.streams.nogoalids.samza.reset.offset = true, you are essentially ignoring checkpoints for tha

Re: How to partition a topic into multiple and how to create multiple Samza Containers

Hi Mohan, Samza maps Kafka topic partitions to containers. So if your topic has only 1 partition, only 1 container will be spawned even if you configure Samza job to use more than 1 container. So please partition the input topic first. The "Tasks" section of [1] contains more information on this

Re: Review Request 44920: Remove tight coupling of Samza with Yarn. Define APIs for resource manager integration

> On March 16, 2016, 11:10 p.m., Chris Pettitt wrote: > > Some more comments Sorry, did not mean to create issues for all of the below. I think #1 and #2 are the most interesting to look at of the group. - Chris --- This is an automati

Re: Send a msg to all partitions

Broad cast stream is the Samza feature designed for this exact requirement. Your thinking is on the right lines. Please take a look at task.broadcast.inputs from the Samza configuration reference page. On Friday, March 18, 2016, Louisia Famalda wrote: > Hi, > > I'm new to Samza and I'm trying t

Samza not consuming

I'm trying to debug our samza job, which seem to be stuck from consuming from our Kafka stream. Every time I redeploy the job, only the same handful of events get consumed, and then no more events get processed. I manually checked to make sure the input stream is live and flowing. I also tried bot

Re: Samza not consuming

Hi David, Appreciate the feedback. Currently, Samza is single-threaded. The process() callback executes exactly in the context of the same thread that processes events. You can take a process dump (often over a period of time repeatedly) and figure out where the threads are stuck to debug these ki

Re: Review Request 44920: Remove tight coupling of Samza with Yarn. Define APIs for resource manager integration

> On March 16, 2016, 8:48 p.m., Chris Pettitt wrote: > > samza-core/src/main/java/org/apache/samza/clustermanager/SamzaAppState.java, > > line 36 > > > > > > Agreed. This is a bit mix of atomic and mutable state. Eit

Re: Samza not consuming

HI David, This configuration you have tweaked (systems.kafka.consumer.auto.offset.reset) is honored only when one of the following condition holds: * topic doesn't exist * checkpoint is older than the maximum message history retained by the brokers So, my questions are : Are you using checkpointin

Re: Samza not consuming

Hi, I wanna resurface this thread because I'm still facing issues with our samza not receiving events. Our samza job metric "SamzaContainerMetrics.process-calls" dropped to zero today again. So does "SamzaContainerMetrics.process-envelopes" (of course). Current topic offset and task checkpoint rev

Re: Review Request 44820: SAMZA-896 : Improvements to thread-safety in ContainerRequestState

> On March 16, 2016, 12:16 a.m., Navina Ramesh wrote: > > samza-yarn/src/main/java/org/apache/samza/job/yarn/ContainerRequestState.java, > > line 323 > > > > > > Move these comments to the java doc comment. > >

Re: Samza not consuming

Finally seeing events flowing again. Yes, the "systems.kafka.consumer.auto.offset.reset" option is probably not a factor here. And yes, I am using checkpointing (kafka). Not sure if the offsets are messed up. But I was able to use "systems.kafka.streams.nogoalids.samza.reset.offset=true" to reset

Re: Review Request 44772: SAMZA-893 Fix the host affinity expiration logic bug introduced in SAMZA-867

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44772/ --- (Updated March 17, 2016, 7:24 p.m.) Review request for samza, Navina Ramesh and

Re: Review Request 44920: Remove tight coupling of Samza with Yarn. Define APIs for resource manager integration

> On March 16, 2016, 8:48 p.m., Chris Pettitt wrote: > > samza-core/src/main/java/org/apache/samza/clustermanager/AbstractContainerAllocator.java, > > line 46 > > > > > > Discussed with Jagadish offline, but summariz

Re: Review Request 44920: Make Samza independent of Yarn.

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44920/ --- (Updated March 16, 2016, 6:21 p.m.) Review request for samza, Boris Shkolnik, C

Re: Review Request 44405: SAMZA-882 - Detect partition count changes in input streams

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44405/#review124223 --- samza-core/src/main/scala/org/apache/samza/coordinator/StreamPart

Re: Review Request 44920: Remove tight coupling of Samza with Yarn. Define APIs for resource manager integration

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44920/#review123928 --- Some more comments samza-core/src/main/java/org/apache/samza/clu

Re: Review Request 44920: Remove tight coupling of Samza with Yarn. Define APIs for resource manager integration

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44920/#review123904 --- I had a few minutes, so here's some low-hanging fruit. samza-cor

Re: Samza not consuming

Looks like this has nothing to do with checkpointing. Our samza job has an issue communicating an external service, which left the particular process() call waiting indefinitely. And it doesn't look like samza has a way to timeout a processing cycle. On Thu, Mar 17, 2016 at 5:42 PM, David Yu wrot

Re: Samza not consuming

No, instead, I updated the checkpoint topic with the "upcoming" offsets. (I should have done a check before that though). So a related question: if I delete the checkpoint topic from Kafka, that would essentially clear up all the offset info and samza will be able to recreate this topic with the l

Re: Samza not consuming

Strangely, I was not able to get checkpoint value for one particular partition. Could this cause the job to be stuck? On Thu, Mar 17, 2016 at 5:23 PM, David Yu wrote: > Hi, I wanna resurface this thread because I'm still facing issues with our > samza not receiving events. > > Our samza job metr

Re: Review Request 44920: Remove tight coupling of Samza with Yarn. Define APIs for resource manager integration

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44920/#review123902 --- This is a deceptively big RB :) I got to about SamzaTaskManager, b

Re: Review Request 44920: Remove tight coupling of Samza with Yarn. Define APIs for resource manager integration

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44920/ --- (Updated March 16, 2016, 6:23 p.m.) Review request for samza, Boris Shkolnik, C

Review Request 44920: Remove tight coupling of Samza with Yarn. Define APIs for resource manager integration.

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44920/ --- Review request for samza, Boris Shkolnik, Chris Pettitt, Jake Maes, Yi Pan (Data