Review Request 33674: Fix SAMZA-660 - Default serdes do not work for changelog streams.

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33674/ --- Review request for samza. Repository: samza Description --- Fix SAMZA-66

Re: Review Request 33674: Fix SAMZA-660 - Default serdes do not work for changelog streams.

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33674/#review81978 --- samza-core/src/main/scala/org/apache/samza/container/SamzaContainer

Re: Review Request 33674: Fix SAMZA-660 - Default serdes do not work for changelog streams.

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33674/ --- (Updated April 29, 2015, 2:28 p.m.) Review request for samza. Changes ---

Re: Review Request 33674: Fix SAMZA-660 - Default serdes do not work for changelog streams.

> On April 29, 2015, 1:06 p.m., Yan Fang wrote: > > samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala, > > line 119 > > > > > > This "config" is not used anymore. remove it? Oops, I missed tha

Samza serializers jar for 0.9.0 ?

I don't see the serializers jar being published in Maven. Yan - know anything about this ? -- Thanks and regards Chinmay Soman

Re: Samza serializers jar for 0.9.0 ?

Oh never mind - I forgot it being merged into core. Sorry for the noise. On Wed, Apr 29, 2015 at 12:03 PM, Chinmay Soman wrote: > I don't see the serializers jar being published in Maven. Yan - know > anything about this ? > > -- > Thanks and regards > > Chinmay Soman > -- Thanks and regards

Re: Review Request 32147: SAMZA-465

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32147/#review81995 --- samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointMana

Re: Review Request 33453: SAMZA-557 Reuse local state in SamzaContainer on clean shutdown

> On April 28, 2015, 5:43 p.m., Yi Pan (Data Infrastructure) wrote: > > samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala, > > line 457 > > > > > > Question: shouldn't the base dir to be determ

Re: Review Request 32147: SAMZA-465

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32147/#review82006 --- samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointMana

Re: Review Request 32147: SAMZA-465

> On April 29, 2015, 7:30 p.m., Yan Fang wrote: > > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java, > > line 436 > > > > > > typo, SetChangelogMapping? I'll fix this when I

Re: Review Request 32147: SAMZA-465

- Naveen --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32147/#review81995 --- On April 10, 2015, 3:13 a.m., Naveen Somasundaram wrote:

Re: Review Request 32147: SAMZA-465

> On April 29, 2015, 7:30 p.m., Yan Fang wrote: > > samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java, > > line 98 > > > > > > why do we need to bootstrap everytime we call this method? We ne

Re: Review Request 32147: SAMZA-465

> On April 29, 2015, 7:30 p.m., Yan Fang wrote: > > samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala, line 90 > > > > > > move it to the beginning of this class? This is an mistake from copy-paste :),

Re: Review Request 32147: SAMZA-465

> On April 29, 2015, 7:30 p.m., Yan Fang wrote: > > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java, > > line 149 > > > > > > Will this bootstrappedStreamSet overflow

Re: Review Request 32147: SAMZA-465

> On April 29, 2015, 8:38 p.m., Naveen Somasundaram wrote: > > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java, > > line 448 > > > > > > This is consistent with how it was pr

Re: Review Request 32147: SAMZA-465

> On April 29, 2015, 7:30 p.m., Yan Fang wrote: > > samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java, > > lines 60-68 > > > > > > actually, we do not need the consumer after boostrapping

Re: Review Request 32147: SAMZA-465

> On April 29, 2015, 7:30 p.m., Yan Fang wrote: > > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java, > > line 149 > > > > > > Will this bootstrappedStreamSet overflow

Re: Review Request 32147: SAMZA-465

> On April 29, 2015, 7:30 p.m., Yan Fang wrote: > > samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java, > > lines 60-68 > > > > > > actually, we do not need the consumer after boostrapping

Re: Errors and hung job on broker shutdown

Not sure about the Kafka side. From the Samza side, from your description ( "does not exit nor does it make any progress" ), I think the code is stuck in producer.close

Re: What next for streaming SQL?

Can you give an example of the SQL syntax you are using for tumbling windows? Does it use GROUP BY and FLOOR, as in https://github.com/apache/incubator-calcite/blob/master/doc/STREAM.md#tumbling-windows ? W

Re: Errors and hung job on broker shutdown

Roger, I believe Samza 0.9.0 already uses the Java producer. Java producer's close() call will try to flush all buffered data to the brokers before completing the call. However, if some buffered data's destination partition leader is not known, the producer will block on refreshing the metadata a

Re: Errors and hung job on broker shutdown

And just to answer your first question: SIGTERM with controlled.shutdown=true should be OK for bouncing the broker. Guozhang On Wed, Apr 29, 2015 at 7:36 PM, Guozhang Wang wrote: > Roger, > > I believe Samza 0.9.0 already uses the Java producer. > > Java producer's close() call will try to flus

Re: Errors and hung job on broker shutdown

Guozhang and Yan, Thank you both for your responses. I tried a lot of combinations and I think I've determined that it's new producer + snappy that causes the issue. It never happens with the old producer and it never happens with lz4 or no compression. It only happens when a broker gets restar