Re: Exactly once processing

Hi Sab, For stateful processing where you have persistent state stores, you need to maintain the checkpoint which includes the committed offsets as well as the store flushed in sync, but right not these two operations are not done atomically, and hence if you fail in between, you could still get d

Re: KIP-28 kafka processor

Hello folks, The Kafka processor patch has been updated incorporating collected feedbacks so far, along with the updated KIP wiki for the API and architecture design: https://cwiki.apache.org/confluence/display/KAFKA/KIP-28+-+Add+a+processor+client Any further comments are appreciated! You can f

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

Hello Yan, Thanks for the feedback, replies inlined. Guozhang On Wed, Aug 19, 2015 at 8:22 AM, Yan Fang wrote: > Hi Guozhang, > > Thank you for writing the KIP-28 up. (Hope this is the right thread for me > to post some comments. :) > > I still have some confusing about the implementation of t

Re: [DISCUSS] Release 0.10.0

Would SAMZA-697 worth adding to 0.10.0 as well? Guozhang On Thu, Jul 30, 2015 at 6:45 PM, Yan Fang wrote: > For SAMZA-747, may ping Naveen or Chris. :) > > They have the permission to publish to maven. From the discuss > , they seem ready for the

Re: hello-samza git repo moved

you try this? > > Thanks! > Navina > > On 7/14/15, 3:45 PM, "Guozhang Wang" wrote: > > >Hello, > > > >I'm wondering if the hello-samza git repo has been moved? > > > > > > > >guozhang *~/Workspace/apach

hello-samza git repo moved

Hello, I'm wondering if the hello-samza git repo has been moved? guozhang *~/Workspace/apache/samza* git clone git:// git.apache.org/samza-hello-samza.git hello-samza Cloning into 'hello-samza'... fatal: read error: Connection reset by peer guozhang *~/Workspace/apach

Re: Thoughts and obesrvations on Samza

>From peanut gallery.. I like Yi's proposal in re-scoping the Samza project / code-base as "Stream Processing as a Service" that will potentially include: 1. A service manager with some REST / Web UI to accept stream processing jobs in terms of tgz / configs and schedule them as for: a. partitio

Re: Review Request 35397: Fix SAMZA-697

it: https://reviews.apache.org/r/35397/#review90620 --- On July 10, 2015, 4:54 p.m., Guozhang Wang wrote: > > --- > This is an automatically generated e-mail. To rep

Re: Review Request 35397: Fix SAMZA-697

/TestSamzaContainer.scala 9fb1aa98fcd14397e8a4cb00c67537482e95fa53 samza-core/src/test/scala/org/apache/samza/container/TestTaskInstance.scala 7caad28c9298485753ab861da76793cf925953ed Diff: https://reviews.apache.org/r/35397/diff/ Testing --- unit tests Thanks, Guozhang Wang

Re: Thoughts and obesrvations on Samza

1. I am neutral to modifying the consumer rebalance protocol to move the logic pluggable to the client side, but I think if we decide to go this route we'd better do it now than later as the protocol is not officially "released" yet. This may delay the first release of the new consumer. 2. I like

Re: Thoughts and obesrvations on Samza

; >> reduction of duplication and clashing terminologies/abstractions > > that > > > > >> Chris/Jay describe. Samza as a library would likely be a very nice > > > tool > > > > to > > > > >> add to the Kafka ecosystem. I just

Re: Thoughts and obesrvations on Samza

Read through the code example and it looks good to me. A few thoughts regarding deployment: Today Samza deploys as executable runnable like: deploy/samza/bin/run-job.sh --config-factory=... --config-path=file://... And this proposal advocate for deploying Samza more as embedded libraries in user

Re: Review Request 35397: Fix Samza-697

953ed Diff: https://reviews.apache.org/r/35397/diff/ Testing --- unit tests Thanks, Guozhang Wang

Re: Review Request 35397: v1

scala 7caad28c9298485753ab861da76793cf925953ed Diff: https://reviews.apache.org/r/35397/diff/ Testing --- unit tests Thanks, Guozhang Wang

Re: Review Request 35397: Fix SAMZA-697

ault should work. Not sure I understand clearly, could you elaborate how changing to package default can help accessing a private class function? - Guozhang --- This is an automatically generated e-mail. To reply, visit

Re: Review Request 35397: Fix SAMZA-697

is is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35397/#review88455 --- On June 18, 2015, 6:42 p.m., Guozhang Wang wrote: > > ---

Re: Review Request 35397: Fix SAMZA-697

about this. This means we will load the class every > > time a message comes. Is this too much? > > > > My suggestion is to put this code in RunLoop.runs, before the loop > > starts. What do you think? > > Guozhang Wang wrote: > A class will only be lo

Re: [VOTE] Apache Samza 0.9.1 RC0

l.com > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> Yan, > > > > > > > > >> > > > > > > > > >> You rock.

Re: [VOTE] Apache Samza 0.9.1 RC0

er night. > > On Wed, Jun 17, 2015 at 5:54 PM, Guozhang Wang wrote: > > > Hey all, > > > > This is a call for a vote on a release of Apache Samza 0.9.1. This is a > > bug-fix release against 0.9.0. > > > > The release candidate can be downloaded from here: &

Re: Review Request 35397: Fix SAMZA-697

scala 9fb1aa98fcd14397e8a4cb00c67537482e95fa53 samza-core/src/test/scala/org/apache/samza/container/TestTaskInstance.scala 7caad28c9298485753ab861da76793cf925953ed Diff: https://reviews.apache.org/r/35397/diff/ Testing --- unit tests Thanks, Guozhang Wang

[VOTE] Apache Samza 0.9.1 RC0

Hey all, This is a call for a vote on a release of Apache Samza 0.9.1. This is a bug-fix release against 0.9.0. The release candidate can be downloaded from here: http://people.apache.org/~guozhang/samza-0.9.1-rc0/ The release candidate is signed with pgp key 911402D8, which is included in the

Re: Review Request 35397: Fix SAMZA-697

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35397/#review88260 --- On June 16, 2015, 5:22 p.m., Guozhang Wang wrote: > >

Re: Review Request 35577: Fix SAMZA-714

Thanks, Guozhang Wang

Re: Review Request 35577: Fix SAMZA-714

0.9.1 Thanks, Guozhang Wang

Review Request 35577: Fix SAMZA-714

Thanks, Guozhang Wang

Re: [DISCUSS] Samza 0.9.1 release

AM, "Yi Pan" wrote: > > > > >+1 Agreed. > > > > > >Thanks! > > > > > >On Tue, Jun 16, 2015 at 10:15 AM, Yan Fang > wrote: > > > > > >> Agreed on this. > > >> > > >> Thanks, > > >> > &

Re: [DISCUSS] Samza 0.9.1 release

Tippur wrote: > Wang, > > I have not caught up but can you please highlight if there are any feature > additions as well? > > - Shekar > > On Tue, Jun 16, 2015 at 10:14 AM, Guozhang Wang > wrote: > > > Hi all, > > > > We have been running a couple of ou

Re: Review Request 35397: Fix SAMZA-697

9fb1aa98fcd14397e8a4cb00c67537482e95fa53 samza-core/src/test/scala/org/apache/samza/container/TestTaskInstance.scala 7caad28c9298485753ab861da76793cf925953ed Diff: https://reviews.apache.org/r/35397/diff/ Testing (updated) --- unit tests Thanks, Guozhang Wang

[DISCUSS] Samza 0.9.1 release

Hi all, We have been running a couple of our jobs against `0.9.1` branch last week at LinkedIn with some critical bug fixes back-ported, including: SAMZA-608 Deserialization error causes SystemConsumers to hang SAMZA-616 Shutdown hook does not wait for container to finish SAMZA-658 Iterator.rem

Re: Review Request 35397: Fix SAMZA-697

7caad28c9298485753ab861da76793cf925953ed Diff: https://reviews.apache.org/r/35397/diff/ Testing --- This is WIP patch, I need to add some unit tests for this change. But I would like to have people taking a look at it now to make sure I am on the right direction. Thanks, Guozhang Wang

Review Request 35397: Fix SAMZA-697

--- This is WIP patch, I need to add some unit tests for this change. But I would like to have people taking a look at it now to make sure I am on the right direction. Thanks, Guozhang Wang

Re: Review Request 34500: SAMZA-552 Operator API change: builder and simplified operator classes

too much about API generalogy. Some other minor thing: 1. PartitionSpec / StreamSqlTask are empty files: is this intentional? - Guozhang Wang On May 20, 2015, 11:13 p.m., Yi Pan (Data Infrastructure) wrote: > > --- > This is an a

Re: Did 0.9.1 ever get released?

Hi Tommy, I am currently working on 0.9.1 release now, and expecting it to be out in a week or two. Guozhang On Wed, Jun 3, 2015 at 7:16 AM, Tommy Becker wrote: > Did 0.9.1 ever get released? I don't see it in Maven Central, nor has the > site been updated... > > -- > Tommy Becker > Senior So

Re: Nothing occurs from collector.send()

Hi Garret, Regarding the serde issues that Yan mentioned, you can check https://issues.apache.org/jira/browse/SAMZA-608 and see if its description matches what you saw. Guozhang On Tue, Jun 2, 2015 at 4:56 PM, Yan Fang wrote: > Hi Garrett, > > I guess you run into the serde issues as you menti

Re: Samza producer error

>From the logs it seems your producer (not consumer) cannot successfully connect to the brokers. I would suggest you trying the console-producer with some sample data and with the same bootstrap broker list and see if it works. On Wed, May 27, 2015 at 10:43 AM, Yan Fang wrote: > Hi Davide, > > T

Re: Not able to consume produced data

Hello Renato, Could you paste your console consumer command here? Guozhang On Fri, May 22, 2015 at 9:43 AM, Naveen S wrote: > Hey Renato, > Is there any specific reason why you are extending the > blocking envelope class instead of implementing a StreamTask ? > http://samza

Re: Samza job throughput much lower than Kafka throughput

ucer.batch.size=262144 > > producer.linger.ms=5 > > producer.compression.type: lz4 > > > > On Wed, May 20, 2015 at 9:30 PM, Guozhang Wang > wrote: > > > >> Hi George, > >> > >> Is there any reason you need to set the following con

Re: Yarn jobs in accepted state

Hello Shekar, The Apache mailing list blocks most attachments, could you send a link of the screenshot here? Guozhang On Wed, May 20, 2015 at 7:14 PM, Shekar Tippur wrote: > Hello, > > After submitting Samza job to Yarn, I see a lot of jobs in accepted state. > > Please see the attached screen

Re: Samza job throughput much lower than Kafka throughput

Hi George, Is there any reason you need to set the following configs? systems.kafka.consumer.fetch.wait.max.ms= 1 This setting will basically disable long pooling of the consumer which will then busy fetching data from broker, which has a large impact on network latency especially when the consu

Re: tkafka.common.ReplicaNotAvailableException on application logs

Hi Shekar, It seems the incoming / outgoing topics are not the root of the problem here, but the checkpoint topic "__samza_checkpoint_ver_1_for_Argos". From the error logs this topic only has one replica 1018019532, which was down and hence not available. Guozhang On Thu, May 14, 2015 at 5:16 AM

Re: Errors and hung job on broker shutdown

ings so that they will be fixed so this > problem won't happen for us again. I know this scenario is not a expected > case. > > - Dan > > > On Tue, 12 May 2015 at 01:36 Guozhang Wang wrote: > > > Hello Dan, > > > > I think your scenario is differen

Re: Errors and hung job on broker shutdown

: target/snappy-1.1.1/snappy.cc:423: char* > > snappy::internal::CompressFragment(const char*, size_t, char*, > > snappy::uint16*, int): Assertion `0 == memcmp(base, candidate, matched)' > > failed. > > > > I'll try and get some better traces and po

Re: Review Request 34066: Fix SAMZA-662

/SamzaContainer.scala e8e830e99825ac9802552c43ec9579c01570f9ea samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 60ee36ff344200e89c34d9147c81c00650ec535e Diff: https://reviews.apache.org/r/34066/diff/ Testing --- Unit tests Thanks, Guozhang Wang

Review Request 34066: Fix SAMZA-662

/ThreadJobFactory.scala 60ee36ff344200e89c34d9147c81c00650ec535e Diff: https://reviews.apache.org/r/34066/diff/ Testing --- Unit tests Thanks, Guozhang Wang

Re: Review Request 33761: Fix SAMZA-658

://reviews.apache.org/r/33761/diff/ Testing --- unit tests Thanks, Guozhang Wang

Re: Review Request 33488: SAMZA-657

checkstyleMain checkstyleTest Thanks, Guozhang Wang

Re: Review Request 33761: Fix SAMZA-658

://reviews.apache.org/r/33761/diff/ Testing --- unit tests Thanks, Guozhang Wang

Re: Review Request 33761: Fix SAMZA-658

9d9ca Diff: https://reviews.apache.org/r/33761/diff/ Testing (updated) --- unit tests Thanks, Guozhang Wang

Re: Review Request 33761: Fix SAMZA-658

ge/kv/MockKeyValueStore.scala PRE-CREATION samza-kv/src/test/scala/org/apache/samza/storage/kv/TestCachedStore.scala d03ec925b103ccf3c1561de0461fbc39cbe9d9ca Diff: https://reviews.apache.org/r/33761/diff/ Testing --- Thanks, Guozhang Wang

Re: Review Request 33761: Fix SAMZA-658

ge/kv/TestCachedStore.scala d03ec925b103ccf3c1561de0461fbc39cbe9d9ca Diff: https://reviews.apache.org/r/33761/diff/ Testing --- Thanks, Guozhang Wang

Re: Errors and hung job on broker shutdown

a broker has restarted or had its ZK > session time out in periods of very heavy load. But I need do more > troubleshooting to have something less vague to report over there. > > Garry > > -Original Message- > From: Guozhang Wang [mailto:wangg...@gmail.com] > Sen

Re: Errors and hung job on broker shutdown

e request > > This seems most likely to be a bug in the new Kafka producer. I'll > probably file a JIRA for that project. > > Thanks, > > Roger > > On Wed, Apr 29, 2015 at 7:38 PM, Guozhang Wang wrote: > > > And just to answer your first question: SIGTE

Re: Review Request 33761: Fix SAMZA-658

/ Testing --- Thanks, Guozhang Wang

Review Request 33761: Fix SAMZA-658

61bb3f6acb080b653f8b11176538549738255acc samza-kv/src/main/scala/org/apache/samza/storage/kv/KeyValueStorageEngine.scala 3a23daf053f0b8dec3a7ec83a51c9c5527078a3b Diff: https://reviews.apache.org/r/33761/diff/ Testing --- Thanks, Guozhang Wang

Re: Errors and hung job on broker shutdown

And just to answer your first question: SIGTERM with controlled.shutdown=true should be OK for bouncing the broker. Guozhang On Wed, Apr 29, 2015 at 7:36 PM, Guozhang Wang wrote: > Roger, > > I believe Samza 0.9.0 already uses the Java producer. > > Java producer's clos

Re: Errors and hung job on broker shutdown

Roger, I believe Samza 0.9.0 already uses the Java producer. Java producer's close() call will try to flush all buffered data to the brokers before completing the call. However, if some buffered data's destination partition leader is not known, the producer will block on refreshing the metadata a

Re: Edit Permission on Confluence Pages

Verified, thanks Yan! On Mon, Apr 27, 2015 at 3:45 PM, Yan Fang wrote: > Hi Guozhang, > > Checked the Confluence. you should have the edit permission. > > Thanks, > Fang, Yan > yanfang...@gmail.com > > On Mon, Apr 27, 2015 at 2:16 PM, Guozhang Wang wrote: > >

Edit Permission on Confluence Pages

Hi, Can some committer grant me the edit permission on wiki pages? -- Guozhang

Re: Review Request 33488: SAMZA-657

393727d3e6fde19a2706efd6ea1ce09 samza-test/src/main/java/org/apache/samza/test/integration/join/Joiner.java d2c0c7eaf9c389e3f88b63a2eb7668b31d1b2daf samza-test/src/main/java/org/apache/samza/test/integration/join/Watcher.java 7c82e0a7ac6b175cca935fc058a96aaade92fbe0 Diff: https://reviews.apache.org/r/33488/diff/ Testing --- ./gradlew checkstyleMain checkstyleTest Thanks, Guozhang Wang

Re: Review Request 33488: SAMZA-657

za/test/integration/join/Joiner.java d2c0c7eaf9c389e3f88b63a2eb7668b31d1b2daf samza-test/src/main/java/org/apache/samza/test/integration/join/Watcher.java 7c82e0a7ac6b175cca935fc058a96aaade92fbe0 Diff: https://reviews.apache.org/r/33488/diff/ Testing --- ./gradlew checkstyleMain checkstyleTest Thanks, Guozhang Wang

Review Request 33488: SAMZA-657

va/org/apache/samza/test/integration/join/Watcher.java 7c82e0a7ac6b175cca935fc058a96aaade92fbe0 Diff: https://reviews.apache.org/r/33488/diff/ Testing --- ./gradlew checkstyleMain checkstyleTest Thanks, Guozhang Wang

Re: How to deal with bootstrapping

I feel Ben's solution a bit simpler that you just need to restart your current job with both rules on the check pointed offset, and start a new job from offset 0 with only the new rule and it will stop at the checkout pointed offset. But of course it requires the second job to be able to shutdown i

Re: Review Request 29899: Changes for Kafka producer upgrade

/org/apache/samza/test/integration/TestStatefulTask.scala <https://reviews.apache.org/r/29899/#comment114127> This can be removed. - Guozhang Wang On Jan. 23, 2015, 12:42 a.m., Navina Ramesh wrote: > > --- > This is an automat

RE: Testing dev@samza.apache.org

ICMP From: Chris Riccomini [criccom...@apache.org] Sent: Friday, January 23, 2015 9:49 AM To: dev@samza.apache.org Subject: Testing dev@samza.apache.org Hey all, Could you please confirm that you're seeing this? I'm trying to verify the TLP migration for: