Re: [ANNOUNCE] Welcome Daniel Chen as Samza Committer

Congrats!!! On Fri, Sep 17, 2021 at 2:06 PM Sanil Jain wrote: > Congrats Daniel ! > > On Fri, 17 Sept 2021 at 11:40, Jagadish Venkatraman < > jagadish1...@gmail.com> > wrote: > > > Congrats Daniel on this well deserved recognition. > > > > Look forward to more contributions! > > > > > > On F

Apache Samza 1.3.1 is released

Documentation and Blog are published. See the announcement from samza website and apache blog Thanks, Hai

[RESULT] [VOTE] Apache Samza 1.3.1 RC0

2) > > > > We can follow it up with a ticket if we someone else runs into it as > well. > > > > +1 (binding) > > > > Thanks, > > Bharath > > > > On Tue, Feb 18, 2020 at 12:12 AM Yi Pan wrote: > > > > > Ran check-all and integr

[VOTE] Apache Samza 1.3.1 RC0

Hi, This is a call for a vote on a release of Apache Samza 1.3.1 to redress certain issues found in 1.3.0 The release candidate can be downloaded from here: http://home.apache.org/~lhaiesp/samza-1.3.1-rc0/ The release candidate is signed with pgp key 0x256F8FA2, which can be found here: https://

[DISCUSS] Samza 1.3.1 release

Hi all, We're going to make a 1.3.1 release to address some critical issues that were found in 1.3.0 1.3.1 will be based off 1.3.0 but include the following additional commits: SAMZA-2447: Checkpoint dir removal should only search in valid store dirs (#1261) SAMZA-2446: Invoke onCheckpoint only

Apache Samza 1.3.0 is released

Documentation and Blog are published. See the announcement here Thanks, Hai

[RESULT] [VOTE] Apache Samza 1.3.0 RC2

disable this test for the future releases. Create ticket to track: > https://issues.apache.org/jira/browse/SAMZA-2411 > > Thanks, > Xinyu > > > > On Sun, Dec 1, 2019 at 6:20 PM Yi Pan wrote: > > > +1 (binding), verified the signature, built and local integration

Re: [VOTE] Apache Samza 1.3.0 RC2

; > or disable this test for the future releases. Create ticket to track: > > https://issues.apache.org/jira/browse/SAMZA-2411 > > > > Thanks, > > Xinyu > > > > > > > > On Sun, Dec 1, 2019 at 6:20 PM Yi Pan wrote: > > > > > +1 (binding), v

[CANCEL] [VOTE] Apache Samza 1.3.0 RC1

See details below. -- Forwarded message - From: Hai Lu Date: Wed, Nov 27, 2019 at 2:56 PM Subject: Re: [VOTE] Apache Samza 1.3.0 RC1 To: This is canceled because issues have been detected with the integration tests. On Thu, Nov 21, 2019 at 1:47 PM Hai Lu wrote: >

Re: [VOTE] Apache Samza 1.3.0 RC1

This is canceled because issues have been detected with the integration tests. On Thu, Nov 21, 2019 at 1:47 PM Hai Lu wrote: > Hi, > > This is a call for a vote on a release of Apache Samza 1.3.0. Thanks to > everyone who has contributed to this release. > > The releas

[VOTE] Apache Samza 1.3.0 RC2

Hi, This is a call for a vote on a release of Apache Samza 1.3.0. Thanks to everyone who has contributed to this release. The release candidate can be downloaded from here: http://home.apache.org/~lhaiesp/samza-1.3.0-rc2/ The release candidate is signed with pgp key 0x07678C76, which can be foun

[VOTE] Apache Samza 1.3.0 RC1

Hi, This is a call for a vote on a release of Apache Samza 1.3.0. Thanks to everyone who has contributed to this release. The release candidate can be downloaded from here: http://home.apache.org/~lhaiesp/samza-1.3.0-rc1/ The release candidate is signed with pgp key 0x07678C76, which can be foun

Re: [VOTE] Apache Samza 1.3.0 RC0

hanks, > Bharath > > On Tue, Nov 12, 2019 at 3:41 PM Hai Lu wrote: > > > Hi, > > > > This is a call for a vote on a release of Apache Samza 1.3.0. Thanks to > > everyone who has contributed to this release. > > > > The release candidate can be

[VOTE] Apache Samza 1.3.0 RC0

Hi, This is a call for a vote on a release of Apache Samza 1.3.0. Thanks to everyone who has contributed to this release. The release candidate can be downloaded from here: http://home.apache.org/~lhaiesp/samza-1.3.0-rc0/ The release candidate is signed with pgp key 0x07678C76, which can be foun

[DISCUSS] Samza 1.3 release

Hi all, It's been some time since our last release and we have accumulated some good features/improvements which call for a new 1.3 release. Want to kick off the discussion in the open source forum while we have already tested many of these changes internally at LinkedIn. As a quick highlight of

Re: [ANNOUNCE] Please welcome Boris Shkolnik to the Samza PMC

Congratulations, Boris! On Fri, Jun 7, 2019 at 6:13 PM Aditya wrote: > Congrats Boris! > > > On Jun 7, 2019, at 4:58 PM, Weiqing Yang > wrote: > > > > Congrats, Boris! > > > > On Fri, Jun 7, 2019 at 4:50 PM santhosh venkat < > santhoshvenkat1...@gmail.com> > > wrote: > > > >> Congratulations bo

Re: REMINDER. [VOTE] Apache Samza 1.2.0 RC4

+1 (non-binding) Verified build and test on Linux box. On mac the test is failing but seems like flakiness not real failure. Thanks, Hai On Tue, Jun 4, 2019 at 1:55 PM santhosh venkat wrote: > +1(non-binding) > > 1. ./bin/check-all.sh succeeded > 2. ./bin/integration-tests.sh succeeded > 3. Ex

Re: Samza 1.1.0 on AWS EMR (emr - 5.13.0, amazon 2.8.3, zookeeper 3.4.10)

+samza dev ApplicationRunnerMain should be there in the samza-core module. Are you seeing the samza-core jar in your lib folder? Make sure the scala version also match (2.10 vs. 2.11) Are you upgrading from 0.14 to 1.1 or from 1.0 to 1.1? Thanks, Hai On Wed, Apr 17, 2019 at 4:28 PM Majd F. Sak

Re: Review Request 52570: SAMZA-1025: documentation for hdfs system consumer

Thanks, Hai Lu

Re: Review Request 52570: SAMZA-1025: documentation for hdfs system consumer

sorry for such a reckless mistake. - Hai --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52570/#review163234 ------- On Jan. 27, 2017, 5:48 p.m., Hai Lu wrote: > >

Re: Review Request 52570: SAMZA-1025: documentation for hdfs system consumer

Thanks, Hai Lu

Re: Review Request 52570: SAMZA-1025: documentation for hdfs system consumer

Thanks, Hai Lu

Re: Review Request 52570: SAMZA-1025: documentation for hdfs system consumer

literally "avro", or "plain", or "json". Though the last two are not supported now. No, they are not class name. - Hai --- This is an automatically generated e-mail. To reply, v

Re: Review Request 52570: SAMZA-1025: documentation for hdfs system consumer

Thanks, Hai Lu

Re: Review Request 52570: SAMZA-1025: documentation for hdfs system consumer

ged the wording as Jagadish suggested above. Let me know if you have further suggestion on top of that. - Hai ----------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52570/#review163036 --

Re: Review Request 52570: SAMZA-1025: documentation for hdfs system consumer

. Added a bit more details to explain - Hai --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52570/#review163011 --- On Jan. 2

Re: Review Request 52570: SAMZA-1025: documentation for hdfs system consumer

--- N/A Thanks, Hai Lu

Re: How to Use samza-hdfs

t. > > private final SystemStream OUTPUT_STREAM = new SystemStream("hdfs", > * "default"*); > > On Wed, Dec 21, 2016 at 8:23 AM Rui Tang wrote: > >> Thank you, I'll try it out! >> >> On Wed, Dec 21, 2016 at 1:45 AM Hai Lu wrote: >> >&

Re: How to Use samza-hdfs

Hi Rui, I've tried out the HDFS producer, too. In my experience, you won't be able to see changes written into HDFS in realtime. The content of the files become visible only after they get closed. You can probably play with the "producer.hdfs.write.batch.size.bytes" config to force rolling over t

Review Request 52660: SAMZA-1034: Support LATEST path in the input of HDFSSystemConsumer

/samza/system/hdfs/partitioner/TestHdfsFileSystemAdapter.java 0fb461fa0781ed2f74e2984783a66d881c58ce2d Diff: https://reviews.apache.org/r/52660/diff/ Testing --- Unit tested and manually verified. Thanks, Hai Lu

Review Request 52570: SAMZA-1025: documentation for hdfs system consumer

/versioned/jobs/configuration-table.html f60cd50fb197423ac3c84fd364bbe4fb3767883e Diff: https://reviews.apache.org/r/52570/diff/ Testing --- N/A Thanks, Hai Lu

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

ala/org/apache/samza/job/yarn/YarnJobFactory.scala 4e328a5f8c2b496a71e36c106339b7af263c96c7 Diff: https://reviews.apache.org/r/51142/diff/ Testing --- unit tests pass. manually tested by writing a real hdfs samza job and deploying to a yarn cluster. Thanks, Hai Lu

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

for HdfsSystemConsumer, > > just like ZooKeeper connnect string is required for KafkaSystemConsumer. > > > > Also, under which condition we need to clear the partition descriptor > > info in the staging dir? We need to think about the cleanup procedure as >

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

arn/YarnJobFactory.scala 4e328a5f8c2b496a71e36c106339b7af263c96c7 Diff: https://reviews.apache.org/r/51142/diff/ Testing --- unit tests pass. manually tested by writing a real hdfs samza job and deploying to a yarn cluster. Thanks, Hai Lu

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

src/main/scala/org/apache/samza/job/yarn/YarnJobFactory.scala 4e328a5f8c2b496a71e36c106339b7af263c96c7 Diff: https://reviews.apache.org/r/51142/diff/ Testing --- unit tests pass. manually tested by writing a real hdfs samza job and deploying to a yarn cluster. Thanks, Hai Lu

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

arn/YarnJobFactory.scala 4e328a5f8c2b496a71e36c106339b7af263c96c7 Diff: https://reviews.apache.org/r/51142/diff/ Testing --- unit tests pass. manually tested by writing a real hdfs samza job and deploying to a yarn cluster. Thanks, Hai Lu

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

s://reviews.apache.org/r/51142/#review150953 ------- On Sept. 28, 2016, 9:57 p.m., Hai Lu wrote: > > --- > This is an automatically generated e-mail.

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

ly HDFS systems ARE depending on YARN in that sense. Security is one more thing to deal with (aside from staging directory) before we can say HDFS sytems no long depends on YARN. What do you think? I will keep this issue open. - Hai --------

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

-mail. To reply, visit: https://reviews.apache.org/r/51142/#review150883 --- On Sept. 28, 2016, 9:57 p.m., Hai Lu wrote: > > --- > This is an automatically generated e-mail. To

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

arn cluster. Thanks, Hai Lu

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

So far we don't have a cleaner solution. - Hai --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51142/#review150648 ----------

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

e samza-api to make it actually happen for now. - Hai --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51142/#review150637 ----------

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

> > One concern I had w/ this HdfsAvroFileReader/Writer is the version > > conflict issue. LinkedIn's Kafka version still uses avro-1.4 in the serde, > > while hdfs already uses avro-1.7 in 2.6.1. I guess that we need to find a > > solution inside LinkedIn to reso

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51142/#review148612 ----------- On Sept. 20, 2016, 11:22 p.m., Hai Lu wrote: > > -

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

ter. Thanks, Hai Lu

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

gt; > Question: this seems to be highly related to how the HDFS files are > > organized. It is hard to see how a common practice would look like, > > especially in open source. Can we make the groupIdentifier pluggable? > > Hai Lu wrote: > Why is it HDFS specific

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51142/#review148780 ----------- On Sept. 9, 2016, 1:34 a.m., Hai Lu wrote: > >

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

nd read the next one. Unless you throw a runtime exception in the catch block to completely stop the consumption. - Hai --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51142/#review148612 ---

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

resolve it. Let's sync up face-to-face tomorrow. I was well aware of the avro issue. I tried so many different APIs that I finally found the set of APIs that work for both 1.4 and 1.7 - Hai --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

I can maybe to a separte fix for that.). But I will create a HdfsSystemConsumerMetrics. - Hai --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51142/#review148612 --- On Sept. 9, 201

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

ala 4e328a5f8c2b496a71e36c106339b7af263c96c7 Diff: https://reviews.apache.org/r/51142/diff/ Testing --- unit tests pass. manually tested by writing a real hdfs samza job and deploying to a yarn cluster. Thanks, Hai Lu

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

ala 4e328a5f8c2b496a71e36c106339b7af263c96c7 Diff: https://reviews.apache.org/r/51142/diff/ Testing --- unit tests pass. manually tested by writing a real hdfs samza job and deploying to a yarn cluster. Thanks, Hai Lu

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

4e328a5f8c2b496a71e36c106339b7af263c96c7 Diff: https://reviews.apache.org/r/51142/diff/ Testing --- unit tests pass. manually tested by writing a real hdfs samza job and deploying to a yarn cluster. Thanks, Hai Lu

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

/main/scala/org/apache/samza/job/yarn/YarnJobFactory.scala 4e328a5f8c2b496a71e36c106339b7af263c96c7 Diff: https://reviews.apache.org/r/51142/diff/ Testing (updated) --- unit tests pass. manually tested by writing a real hdfs samza job and deploying to a yarn cluster. Thanks, Hai Lu

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

a comma? The number of "0" matches the number of files in the > > group? Yes. - Hai ------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51142/#review147596 ---

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

: https://reviews.apache.org/r/51142/diff/ Testing --- unit tests pass. tested by writing a real hdfs samza job and deploying to hadoop cluster. Thanks, Hai Lu

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

enerated e-mail. To reply, visit: https://reviews.apache.org/r/51142/#review147414 ------- On Aug. 29, 2016, 5:27 p.m., Hai Lu wrote: > > --- > This is an automati

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

/YarnJobFactory.scala 4e328a5f8c2b496a71e36c106339b7af263c96c7 Diff: https://reviews.apache.org/r/51142/diff/ Testing --- unit tests pass. tested by writing a real hdfs samza job and deploying to hadoop cluster. Thanks, Hai Lu

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

/YarnJobFactory.scala 4e328a5f8c2b496a71e36c106339b7af263c96c7 Diff: https://reviews.apache.org/r/51142/diff/ Testing --- unit tests pass. tested by writing a real hdfs samza job and deploying to hadoop cluster. Thanks, Hai Lu

[DISCUSS] A HDFS system consumer for Samza

Hi, I have been recently working on a HDFS system consumer for Samza. The work includes two major parts: 1. properly partitioning a HDFS directory and 2. consuming from HDFS files. I have attached the design doc in the Jira ticket here: https://issues.apache.org/jira/browse/SAMZA-967 It would be

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

4e328a5f8c2b496a71e36c106339b7af263c96c7 Diff: https://reviews.apache.org/r/51142/diff/ Testing --- unit tests pass. tested by writing a real hdfs samza job and deploying to hadoop cluster. Thanks, Hai Lu

Review Request 51142: SAMZA-967: HDFS System Consumer

samza job and deploying to hadoop cluster. Thanks, Hai Lu