date:20140310

Re: command line tools

2014-03-10 Thread Michael G. Noll

Oh, and one more comment:

I haven't checked all the CLI tools of Kafka in that regard, but preferably 
each tool would properly return zero exit codes on success and non-zero on 
failure (and possibly distinct error exit codes).

That would simplify integration with tools like Puppet, Chef, Ansible, etc. 
Also, it allows shell chaining of commands via && and || for manual activities 
as well as scripting (e.g. to automate tasks during upgrades or migration).

If exit codes are already used consistently across the CLI tools, then please 
ignore this message. :-)

--Michael



> On 08.03.2014, at 20:09, "Michael G. Noll"  wrote:
> 
> I just happen to come across that message.  As someone who is a mere
> Kafka user take my feedback with a grain of salt.
> 
>> On 03/05/2014 05:01 AM, Jay Kreps wrote:
>> Personally I don't mind the current approach as it is discoverable and
>> works with tab completion.
> 
> Having typical shell features such as tab completion are indeed nice.
> 
> 
>> I wouldn't be opposed to replacing kafka-run-class.sh with a generic kafka
>> script that handles the java and logging options and maintaining a human
>> friendly mapping for some of the class names so that e.g.
>>  ./kafka topics --list
>>  ./kafka console-producer --broker localhost:9092
>> would work as a short cut for some fully qualified name:
>>  ./kafka kafka.producer.ConsoleProducer
>> and
>>  ./kafka
>> would print a list of known commands. We would probably need a way to
>> customize memory settings for each command as we do now, though.
> 
> If you decide to go for a `kafka  ...` approach, what about
> at least splitting the admin commands (e.g. topic management and such)
> from non-admin commands (e.g. starting console producers/consumers)?
> 
>$ kafka admin topics --create ...
>$ kafka admin topics --list
> 
> (Admittedly listing topics is a pretty safe command but should sitll
> fall under the admin category IMHO.)
> 
> Such a distinction would also give some hints on how dangerous a
> potential commandline could be (say, `kafka admin` commands are likely
> to change the state of the cluster itself, whereas `kafka
> console-producer` would "only" start to read data, which should have a
> lesser impact if things go wrong).
> 
> What would also be nice is a "[-h|--help]" option (or a `kafka help
> ` variant) that would describe each command.  But IIRC there
> may be a discussion thread/JIRA ticket for that already.
> 
>> We would
>> need some way to make this typo resistent (e.g. if you type a command wrong
>> you should get a reasonable error and not some big class not found stack
>> trace).
> 
> I agree that such stack traces are irritating.  At 2 AM in the morning
> an Ops person does not want filter relevant error messages from the
> stacktrack noise.  (See the related thread on "Logging irrelevant
> things" from Mar 05).
> 
> 
> All the above being said, I'm happy to hear you are discussing how to
> improve the current CLI tools!
> 
> --Michael
> 
> 
> 
>

Re: command line tools

2014-03-10 Thread Martin Kleppmann

+1 for using exit status in the command-line tools. The other day I wanted to 
modify a shell script to create a Kafka topic, using bin/kafka-topics.sh 
--create --topic ...

The tool's behaviour is not very conducive to automation:

- If the topic creation was successful, it prints out a message and exits with 
status 0.
- If the topic already exists, it prints out a message and exits with status 0.
- If the Kafka broker is down, it prints out an error message and exits with 
status 0.
- If Zookeeper is down, it keeps retrying.

In this example, an exit status to indicate what happened would be really 
helpful.

Martin

On 10 Mar 2014, at 07:48, Michael G. Noll  wrote:
> Oh, and one more comment:
> 
> I haven't checked all the CLI tools of Kafka in that regard, but preferably 
> each tool would properly return zero exit codes on success and non-zero on 
> failure (and possibly distinct error exit codes).
> 
> That would simplify integration with tools like Puppet, Chef, Ansible, etc. 
> Also, it allows shell chaining of commands via && and || for manual 
> activities as well as scripting (e.g. to automate tasks during upgrades or 
> migration).
> 
> If exit codes are already used consistently across the CLI tools, then please 
> ignore this message. :-)
> 
> --Michael
> 
> 
> 
>> On 08.03.2014, at 20:09, "Michael G. Noll"  wrote:
>> 
>> I just happen to come across that message.  As someone who is a mere
>> Kafka user take my feedback with a grain of salt.
>> 
>>> On 03/05/2014 05:01 AM, Jay Kreps wrote:
>>> Personally I don't mind the current approach as it is discoverable and
>>> works with tab completion.
>> 
>> Having typical shell features such as tab completion are indeed nice.
>> 
>> 
>>> I wouldn't be opposed to replacing kafka-run-class.sh with a generic kafka
>>> script that handles the java and logging options and maintaining a human
>>> friendly mapping for some of the class names so that e.g.
>>> ./kafka topics --list
>>> ./kafka console-producer --broker localhost:9092
>>> would work as a short cut for some fully qualified name:
>>> ./kafka kafka.producer.ConsoleProducer
>>> and
>>> ./kafka
>>> would print a list of known commands. We would probably need a way to
>>> customize memory settings for each command as we do now, though.
>> 
>> If you decide to go for a `kafka  ...` approach, what about
>> at least splitting the admin commands (e.g. topic management and such)
>> from non-admin commands (e.g. starting console producers/consumers)?
>> 
>>   $ kafka admin topics --create ...
>>   $ kafka admin topics --list
>> 
>> (Admittedly listing topics is a pretty safe command but should sitll
>> fall under the admin category IMHO.)
>> 
>> Such a distinction would also give some hints on how dangerous a
>> potential commandline could be (say, `kafka admin` commands are likely
>> to change the state of the cluster itself, whereas `kafka
>> console-producer` would "only" start to read data, which should have a
>> lesser impact if things go wrong).
>> 
>> What would also be nice is a "[-h|--help]" option (or a `kafka help
>> ` variant) that would describe each command.  But IIRC there
>> may be a discussion thread/JIRA ticket for that already.
>> 
>>> We would
>>> need some way to make this typo resistent (e.g. if you type a command wrong
>>> you should get a reasonable error and not some big class not found stack
>>> trace).
>> 
>> I agree that such stack traces are irritating.  At 2 AM in the morning
>> an Ops person does not want filter relevant error messages from the
>> stacktrack noise.  (See the related thread on "Logging irrelevant
>> things" from Mar 05).
>> 
>> 
>> All the above being said, I'm happy to hear you are discussing how to
>> improve the current CLI tools!
>> 
>> --Michael
>> 
>> 
>> 
>>

Re: Review Request 18299: Fix KAFKA-1253

2014-03-10 Thread Jun Rao


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review36658
---


Could we add the new compression property to existing tools like 
consoleProducer and ProducerPerformance that support compressed input?


clients/src/main/java/org/apache/kafka/common/record/Compressor.java


Could the init be part of the constructor?



clients/src/main/java/org/apache/kafka/common/record/Compressor.java


Would it be better to write the code as:

 catch (Exception e) {
  if (e instanceof IOException || e instanceof IOException) {
// do sth
  }

This avoids duplicating the code and we can just say I/O exception in the 
log.



clients/src/main/java/org/apache/kafka/common/record/Compressor.java


Do we need the input to be final?



clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java


If we have a separate init(), every caller has to remember calling this 
after the construction of MemoryRecord. Is this worth doing lazy init for 
compressor? If it's really worth doing, perhaps we can do the lazy init in each 
of the func (e.g. append) that needs the compressor.



clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java


Do we need to wrap it or just let it pass through?



clients/src/main/java/org/apache/kafka/common/record/Record.java


It's still weird to always write the data as uncompressed. In RecordTest, 
the compressed type is not really being exercised since the record is created 
as uncompressed. Perhaps we should just change the unit test. If the 
compression type is not none, we will have to decompress the value to get the 
uncompressed content.



core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala


Could we also verify the content of the fetched messages?



core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala


Do we actually hit the Overflow exception with this partition size?



core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala


Could we also verify the content of the fetched messages?


- Jun Rao


On March 8, 2014, 12:36 a.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> ---
> 
> (Updated March 8, 2014, 12:36 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Incorporate Jun's comments round two
> 
> Refactored compression logic into Compressor
> 
> GZIP/Snappy integration tests
> 
> 
> Diffs
> -
> 
>   build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 1ac69436f117800815b8d50f042e9e2a29364b43 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 32e12ad149f6d70c96a498d0a390976f77bf9e2a 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  616e1006f21c54af8260e84a96928cb8893ceb7c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  038a05a94b795ec0a95b2d40a89222394b5a74c4 
>   
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java
>  PRE-CREATION 
>   
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java
>  PRE-CREATION 
>   clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 
> 906da02d02c03aadd8ab73ed2fc9a1898acb8d72 
>   clients/src/main/java/org/apache/kafka/common/record/Compressor.java 
> PRE-CREATION 
>   clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 
> 9d8935fa3beeb2a78b109a41ed76fd4374239560 
>   clients/src/main/java/org/apache/kafka/common/record/Record.java 
> f1dc9778502cbdfe982254fb6e25947842622239 
>   clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
> 0c6b3656375721a718fb4de10118170aacce0ea9 
>   clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java 
> b0745b528cef929c4273f7e2ac4de1476cfc25ad 
>   clients/src/test/java/org/apache/kafka/common/record/RecordTest.java 
> ae54d67da9907b0a043180c7395a1370b3d0528d 
>   clients/src/test/java/org/apache/kafka/co

[jira] [Created] (KAFKA-1301) system testcase_0206 fails using the new producer

2014-03-10 Thread Jun Rao (JIRA)

Jun Rao created KAFKA-1301:
--

 Summary: system testcase_0206 fails using the new producer
 Key: KAFKA-1301
 URL: https://issues.apache.org/jira/browse/KAFKA-1301
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Jun Rao


The problem is that the producer doesn't drain the unsent data properly on 
close. The problem is in the following code. It's possible for this loop to 
exit with unfinished requests.

// okay we stopped accepting requests but there may still be
// requests in the accumulator or waiting for acknowledgment,
// wait until these are completed.
int unsent = 0;
do {
try {
unsent = run(time.milliseconds());
} catch (Exception e) {
log.error("Uncaught error in kafka producer I/O thread: ", e);
}
} while (unsent > 0 || this.inFlightRequests.totalInFlightRequests() > 
0);

Suppose that all produce requests are being sent, but the sender is waiting for 
responses. Then the broker failed. In handling disconnects, we cleared all 
inflight requests. When we check the condition in the while clause, there is no 
unsent data and no in flight requests. However, failed records have been added 
to RecordAccumulator and are ready to be sent in the next iteration.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1301) system testcase_0206 fails using the new producer

2014-03-10 Thread Jun Rao (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-1301:
---

Description: 
The problem is that the producer doesn't drain the unsent data properly on 
close. The problem is in the following code in Sender.run(). It's possible for 
this loop to exit with unfinished requests.

// okay we stopped accepting requests but there may still be
// requests in the accumulator or waiting for acknowledgment,
// wait until these are completed.
int unsent = 0;
do {
try {
unsent = run(time.milliseconds());
} catch (Exception e) {
log.error("Uncaught error in kafka producer I/O thread: ", e);
}
} while (unsent > 0 || this.inFlightRequests.totalInFlightRequests() > 
0);

Suppose that all produce requests are being sent, but the sender is waiting for 
responses. Then the broker failed. In handling disconnects, we cleared all 
inflight requests. When we check the condition in the while clause, there is no 
unsent data and no in flight requests. However, failed records have been added 
to RecordAccumulator and are ready to be sent in the next iteration.

  was:
The problem is that the producer doesn't drain the unsent data properly on 
close. The problem is in the following code. It's possible for this loop to 
exit with unfinished requests.

// okay we stopped accepting requests but there may still be
// requests in the accumulator or waiting for acknowledgment,
// wait until these are completed.
int unsent = 0;
do {
try {
unsent = run(time.milliseconds());
} catch (Exception e) {
log.error("Uncaught error in kafka producer I/O thread: ", e);
}
} while (unsent > 0 || this.inFlightRequests.totalInFlightRequests() > 
0);

Suppose that all produce requests are being sent, but the sender is waiting for 
responses. Then the broker failed. In handling disconnects, we cleared all 
inflight requests. When we check the condition in the while clause, there is no 
unsent data and no in flight requests. However, failed records have been added 
to RecordAccumulator and are ready to be sent in the next iteration.


> system testcase_0206 fails using the new producer
> -
>
> Key: KAFKA-1301
> URL: https://issues.apache.org/jira/browse/KAFKA-1301
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>
> The problem is that the producer doesn't drain the unsent data properly on 
> close. The problem is in the following code in Sender.run(). It's possible 
> for this loop to exit with unfinished requests.
> // okay we stopped accepting requests but there may still be
> // requests in the accumulator or waiting for acknowledgment,
> // wait until these are completed.
> int unsent = 0;
> do {
> try {
> unsent = run(time.milliseconds());
> } catch (Exception e) {
> log.error("Uncaught error in kafka producer I/O thread: ", e);
> }
> } while (unsent > 0 || this.inFlightRequests.totalInFlightRequests() 
> > 0);
> Suppose that all produce requests are being sent, but the sender is waiting 
> for responses. Then the broker failed. In handling disconnects, we cleared 
> all inflight requests. When we check the condition in the while clause, there 
> is no unsent data and no in flight requests. However, failed records have 
> been added to RecordAccumulator and are ready to be sent in the next 
> iteration.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [VOTE] Apache Kafka Release 0.8.1 - Candidate 2

2014-03-10 Thread Jay Kreps

+1

There are a few rough edges: slf4j error and the spurious error from the
console consumer but I'm okay if we fix these in 0.8.2.

-Jay


On Tue, Mar 4, 2014 at 10:59 PM, Joe Stein  wrote:

> This is the second candidate for release of Apache Kafka 0.8.1.
>
> This release candidate fixes the following two JIRA
> KAFKA-1288and
> KAFKA-1289  and updated
> release steps with the gradle changes
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Process for
> build
> and verification post build.
>
> Release Notes (updated) for the 0.8.1 release
>
> https://people.apache.org/~joestein/kafka-0.8.1-candidate2/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, March 10th, 12pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://svn.apache.org/repos/asf/kafka/KEYS in addition to the md5, sha1
> and
> sha2 (SHA256) checksum.
>
> * Release artifacts to be voted upon (source and binary):
> https://people.apache.org/~joestein/kafka-0.8.1-candidate2/
>
> * Maven artifacts to be voted upon prior to release:
> https://repository.apache.org/content/groups/staging/
>
> * The tag to be voted upon (off the 0.8.1 branch) is the 0.8.1 tag
>
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=62f8aaf74c9d36d1dd49cc7e572a7289206b6414
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
>

Re: [VOTE] Apache Kafka Release 0.8.1 - Candidate 2

2014-03-10 Thread Neha Narkhede

+1, verified unit tests and quickstart.

Thanks
Neha


On Mon, Mar 10, 2014 at 10:47 AM, Jay Kreps  wrote:

> +1
>
> There are a few rough edges: slf4j error and the spurious error from the
> console consumer but I'm okay if we fix these in 0.8.2.
>
> -Jay
>
>
> On Tue, Mar 4, 2014 at 10:59 PM, Joe Stein  wrote:
>
> > This is the second candidate for release of Apache Kafka 0.8.1.
> >
> > This release candidate fixes the following two JIRA
> > KAFKA-1288and
> > KAFKA-1289  and
> updated
> > release steps with the gradle changes
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Process for
> > build
> > and verification post build.
> >
> > Release Notes (updated) for the 0.8.1 release
> >
> >
> https://people.apache.org/~joestein/kafka-0.8.1-candidate2/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Monday, March 10th, 12pm PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > http://svn.apache.org/repos/asf/kafka/KEYS in addition to the md5, sha1
> > and
> > sha2 (SHA256) checksum.
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://people.apache.org/~joestein/kafka-0.8.1-candidate2/
> >
> > * Maven artifacts to be voted upon prior to release:
> > https://repository.apache.org/content/groups/staging/
> >
> > * The tag to be voted upon (off the 0.8.1 branch) is the 0.8.1 tag
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=62f8aaf74c9d36d1dd49cc7e572a7289206b6414
> >
> > /***
> >  Joe Stein
> >  Founder, Principal Consultant
> >  Big Data Open Source Security LLC
> >  http://www.stealth.ly
> >  Twitter: @allthingshadoop 
> > /
> >
>

Re: [VOTE] Apache Kafka Release 0.8.1 - Candidate 2

2014-03-10 Thread Joel Koshy

+1

There is the zero-length md5 for 2.8.0-.8.1.tgz.md5 (zero missing).
Not sure if that is a blocker though since most people are off 2.8.0

Joel

On Mon, Mar 10, 2014 at 10:51:35AM -0700, Neha Narkhede wrote:
> +1, verified unit tests and quickstart.
> 
> Thanks
> Neha
> 
> 
> On Mon, Mar 10, 2014 at 10:47 AM, Jay Kreps  wrote:
> 
> > +1
> >
> > There are a few rough edges: slf4j error and the spurious error from the
> > console consumer but I'm okay if we fix these in 0.8.2.
> >
> > -Jay
> >
> >
> > On Tue, Mar 4, 2014 at 10:59 PM, Joe Stein  wrote:
> >
> > > This is the second candidate for release of Apache Kafka 0.8.1.
> > >
> > > This release candidate fixes the following two JIRA
> > > KAFKA-1288and
> > > KAFKA-1289  and
> > updated
> > > release steps with the gradle changes
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Process for
> > > build
> > > and verification post build.
> > >
> > > Release Notes (updated) for the 0.8.1 release
> > >
> > >
> > https://people.apache.org/~joestein/kafka-0.8.1-candidate2/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Monday, March 10th, 12pm PT
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > http://svn.apache.org/repos/asf/kafka/KEYS in addition to the md5, sha1
> > > and
> > > sha2 (SHA256) checksum.
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://people.apache.org/~joestein/kafka-0.8.1-candidate2/
> > >
> > > * Maven artifacts to be voted upon prior to release:
> > > https://repository.apache.org/content/groups/staging/
> > >
> > > * The tag to be voted upon (off the 0.8.1 branch) is the 0.8.1 tag
> > >
> > >
> > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=62f8aaf74c9d36d1dd49cc7e572a7289206b6414
> > >
> > > /***
> > >  Joe Stein
> > >  Founder, Principal Consultant
> > >  Big Data Open Source Security LLC
> > >  http://www.stealth.ly
> > >  Twitter: @allthingshadoop 
> > > /
> > >
> >

Re: [VOTE] Apache Kafka Release 0.8.1 - Candidate 2

2014-03-10 Thread Sriram Subramanian

+1

Verified unit test and quick start.

On 3/10/14 11:12 AM, "Joel Koshy"  wrote:

>+1
>
>There is the zero-length md5 for 2.8.0-.8.1.tgz.md5 (zero missing).
>Not sure if that is a blocker though since most people are off 2.8.0
>
>Joel
>
>On Mon, Mar 10, 2014 at 10:51:35AM -0700, Neha Narkhede wrote:
>> +1, verified unit tests and quickstart.
>> 
>> Thanks
>> Neha
>> 
>> 
>> On Mon, Mar 10, 2014 at 10:47 AM, Jay Kreps  wrote:
>> 
>> > +1
>> >
>> > There are a few rough edges: slf4j error and the spurious error from
>>the
>> > console consumer but I'm okay if we fix these in 0.8.2.
>> >
>> > -Jay
>> >
>> >
>> > On Tue, Mar 4, 2014 at 10:59 PM, Joe Stein 
>>wrote:
>> >
>> > > This is the second candidate for release of Apache Kafka 0.8.1.
>> > >
>> > > This release candidate fixes the following two JIRA
>> > > KAFKA-1288and
>> > > KAFKA-1289  and
>> > updated
>> > > release steps with the gradle changes
>> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Process
>>for
>> > > build
>> > > and verification post build.
>> > >
>> > > Release Notes (updated) for the 0.8.1 release
>> > >
>> > >
>> > 
>>https://people.apache.org/~joestein/kafka-0.8.1-candidate2/RELEASE_NOTES.
>>html
>> > >
>> > > *** Please download, test and vote by Monday, March 10th, 12pm PT
>> > >
>> > > Kafka's KEYS file containing PGP keys we use to sign the release:
>> > > http://svn.apache.org/repos/asf/kafka/KEYS in addition to the md5,
>>sha1
>> > > and
>> > > sha2 (SHA256) checksum.
>> > >
>> > > * Release artifacts to be voted upon (source and binary):
>> > > https://people.apache.org/~joestein/kafka-0.8.1-candidate2/
>> > >
>> > > * Maven artifacts to be voted upon prior to release:
>> > > https://repository.apache.org/content/groups/staging/
>> > >
>> > > * The tag to be voted upon (off the 0.8.1 branch) is the 0.8.1 tag
>> > >
>> > >
>> > 
>>https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=62f8aaf74c9d3
>>6d1dd49cc7e572a7289206b6414
>> > >
>> > > /***
>> > >  Joe Stein
>> > >  Founder, Principal Consultant
>> > >  Big Data Open Source Security LLC
>> > >  http://www.stealth.ly
>> > >  Twitter: @allthingshadoop 
>> > > /
>> > >
>> >
>

Re: [VOTE] Apache Kafka Release 0.8.1 - Candidate 2

2014-03-10 Thread Joe Stein

The 2.8.0-.8.1.tgz.md5 (zero missing) is from a bad sign operation (and
will not get copied to publish).

There is a 2.8.0-.0.8.1.tgz.md5 that goes with the tgz so that is what
should be used.


On Mon, Mar 10, 2014 at 2:12 PM, Joel Koshy  wrote:

> +1
>
> There is the zero-length md5 for 2.8.0-.8.1.tgz.md5 (zero missing).
> Not sure if that is a blocker though since most people are off 2.8.0
>
> Joel
>
> On Mon, Mar 10, 2014 at 10:51:35AM -0700, Neha Narkhede wrote:
> > +1, verified unit tests and quickstart.
> >
> > Thanks
> > Neha
> >
> >
> > On Mon, Mar 10, 2014 at 10:47 AM, Jay Kreps  wrote:
> >
> > > +1
> > >
> > > There are a few rough edges: slf4j error and the spurious error from
> the
> > > console consumer but I'm okay if we fix these in 0.8.2.
> > >
> > > -Jay
> > >
> > >
> > > On Tue, Mar 4, 2014 at 10:59 PM, Joe Stein 
> wrote:
> > >
> > > > This is the second candidate for release of Apache Kafka 0.8.1.
> > > >
> > > > This release candidate fixes the following two JIRA
> > > > KAFKA-1288and
> > > > KAFKA-1289  and
> > > updated
> > > > release steps with the gradle changes
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Processfor
> > > > build
> > > > and verification post build.
> > > >
> > > > Release Notes (updated) for the 0.8.1 release
> > > >
> > > >
> > >
> https://people.apache.org/~joestein/kafka-0.8.1-candidate2/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by Monday, March 10th, 12pm PT
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > http://svn.apache.org/repos/asf/kafka/KEYS in addition to the md5,
> sha1
> > > > and
> > > > sha2 (SHA256) checksum.
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > https://people.apache.org/~joestein/kafka-0.8.1-candidate2/
> > > >
> > > > * Maven artifacts to be voted upon prior to release:
> > > > https://repository.apache.org/content/groups/staging/
> > > >
> > > > * The tag to be voted upon (off the 0.8.1 branch) is the 0.8.1 tag
> > > >
> > > >
> > >
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=62f8aaf74c9d36d1dd49cc7e572a7289206b6414
> > > >
> > > > /***
> > > >  Joe Stein
> > > >  Founder, Principal Consultant
> > > >  Big Data Open Source Security LLC
> > > >  http://www.stealth.ly
> > > >  Twitter: @allthingshadoop 
> > > > /
> > > >
> > >
>
>

[jira] Subscription: outstanding kafka patches

2014-03-10 Thread jira

Issue Subscription
Filter: outstanding kafka patches (84 issues)
The list of outstanding kafka patches
Subscriber: kafka-mailing-list

Key Summary
KAFKA-1300  Added WaitForReplaction admin tool.
https://issues.apache.org/jira/browse/KAFKA-1300
KAFKA-1264  Make ConsoleProducer compression codec configurable
https://issues.apache.org/jira/browse/KAFKA-1264
KAFKA-1253  Implement compression in new producer
https://issues.apache.org/jira/browse/KAFKA-1253
KAFKA-1241  Cryptic serde error messages in new producer
https://issues.apache.org/jira/browse/KAFKA-1241
KAFKA-1235  Enable server to indefinitely retry on controlled shutdown
https://issues.apache.org/jira/browse/KAFKA-1235
KAFKA-1234  All kafka-run-class.sh to source in user config file (to set env 
vars like KAFKA_OPTS)
https://issues.apache.org/jira/browse/KAFKA-1234
KAFKA-1230  shell script files under bin don't work with cygwin (bash on 
windows)
https://issues.apache.org/jira/browse/KAFKA-1230
KAFKA-1227  Code dump of new producer
https://issues.apache.org/jira/browse/KAFKA-1227
KAFKA-1215  Rack-Aware replica assignment option
https://issues.apache.org/jira/browse/KAFKA-1215
KAFKA-1210  Windows Bat files are not working properly
https://issues.apache.org/jira/browse/KAFKA-1210
KAFKA-1207  Launch Kafka from within Apache Mesos
https://issues.apache.org/jira/browse/KAFKA-1207
KAFKA-1206  allow Kafka to start from a resource negotiator system
https://issues.apache.org/jira/browse/KAFKA-1206
KAFKA-1194  The kafka broker cannot delete the old log files after the 
configured time
https://issues.apache.org/jira/browse/KAFKA-1194
KAFKA-1190  create a draw performance graph script
https://issues.apache.org/jira/browse/KAFKA-1190
KAFKA-1180  WhiteList topic filter gets a NullPointerException on complex Regex
https://issues.apache.org/jira/browse/KAFKA-1180
KAFKA-1173  Using Vagrant to get up and running with Apache Kafka
https://issues.apache.org/jira/browse/KAFKA-1173
KAFKA-1171  Gradle build for Kafka
https://issues.apache.org/jira/browse/KAFKA-1171
KAFKA-1147  Consumer socket timeout should be greater than fetch max wait
https://issues.apache.org/jira/browse/KAFKA-1147
KAFKA-1145  Broker fail to sync after restart
https://issues.apache.org/jira/browse/KAFKA-1145
KAFKA-1144  commitOffsets can be passed the offsets to commit
https://issues.apache.org/jira/browse/KAFKA-1144
KAFKA-1130  "log.dirs" is a confusing property name
https://issues.apache.org/jira/browse/KAFKA-1130
KAFKA-1109  Need to fix GC log configuration code, not able to override 
KAFKA_GC_LOG_OPTS
https://issues.apache.org/jira/browse/KAFKA-1109
KAFKA-1106  HighwaterMarkCheckpoint failure puting broker into a bad state
https://issues.apache.org/jira/browse/KAFKA-1106
KAFKA-1093  Log.getOffsetsBefore(t, …) does not return the last confirmed 
offset before t
https://issues.apache.org/jira/browse/KAFKA-1093
KAFKA-1086  Improve GetOffsetShell to find metadata automatically
https://issues.apache.org/jira/browse/KAFKA-1086
KAFKA-1082  zkclient dies after UnknownHostException in zk reconnect
https://issues.apache.org/jira/browse/KAFKA-1082
KAFKA-1049  Encoder implementations are required to provide an undocumented 
constructor.
https://issues.apache.org/jira/browse/KAFKA-1049
KAFKA-1032  Messages sent to the old leader will be lost on broker GC resulted 
failure
https://issues.apache.org/jira/browse/KAFKA-1032
KAFKA-1025  Producer.send should provide recoverability info on failiure
https://issues.apache.org/jira/browse/KAFKA-1025
KAFKA-1012  Implement an Offset Manager and hook offset requests to it
https://issues.apache.org/jira/browse/KAFKA-1012
KAFKA-1011  Decompression and re-compression on MirrorMaker could result in 
messages being dropped in the pipeline
https://issues.apache.org/jira/browse/KAFKA-1011
KAFKA-1005  kafka.perf.ConsumerPerformance not shutting down consumer
https://issues.apache.org/jira/browse/KAFKA-1005
KAFKA-998   Producer should not retry on non-recoverable error codes
https://issues.apache.org/jira/browse/KAFKA-998
KAFKA-997   Provide a strict verification mode when reading configuration 
properties
https://issues.apache.org/jira/browse/KAFKA-997
KAFKA-996   Capitalize first letter for log entries
https://issues.apache.org/jira/browse/KAFKA-996
KAFKA-984   Avoid a full rebalance in cases when a new topic is discovered but 
container/broker set stay the same
https://issues.apache.org/jira/browse/KAFKA-984
KAFKA-976   Order-Preserving Mirror Maker Testcase
https://issues.apache.org/jira/browse/KAFKA-976
KAFKA-967   Use key range in Prod

Re: Review Request 18299: Fix KAFKA-1253

2014-03-10 Thread Guozhang Wang



> On March 10, 2014, 5:23 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/Compressor.java, lines 
> > 100-101
> > 
> >
> > Would it be better to write the code as:
> > 
> >  catch (Exception e) {
> >   if (e instanceof IOException || e instanceof IOException) {
> > // do sth
> >   }
> > 
> > This avoids duplicating the code and we can just say I/O exception in 
> > the log.

I would personally prefer the current code layout since 1) IOException and 
BufferOverflowException are actually for different cases, the first one is not 
expected while the other is; we handle IOException here only to not expose them 
in MemoryRecords; 2) If we catch Exception, then for "else" we'd better 
re-throw them, which will make the function having to be declared as throws 
Exceptions.


> On March 10, 2014, 5:23 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/Compressor.java, lines 
> > 111-112
> > 
> >
> > Do we need the input to be final?

Not necessarily, I put them here just to make sure they are not altered at any 
cases. Do you think there is a reason we should better remove them?


> On March 10, 2014, 5:23 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java, 
> > lines 50-53
> > 
> >
> > If we have a separate init(), every caller has to remember calling this 
> > after the construction of MemoryRecord. Is this worth doing lazy init for 
> > compressor? If it's really worth doing, perhaps we can do the lazy init in 
> > each of the func (e.g. append) that needs the compressor.

I think it is necessary to do lazy init, but just efficient. Since while doing 
the init for compression the header will be re-written, which we do not want 
while reading a memory records. I can move the init() to Compressor and check 
that every time though.


> On March 10, 2014, 5:23 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java, 
> > lines 178-179
> > 
> >
> > Do we need to wrap it or just let it pass through?

If this is the only place that could throw BufferUnderflowException in 
consumers/mirrormakers, then probably we do not need to wrap it; otherwise I 
would prefer to indicate where this exception is thrown. How about keeping it 
here for now and remove it later when we realize that this is really not 
necessary?


> On March 10, 2014, 5:23 p.m., Jun Rao wrote:
> > core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala, 
> > lines 145-146
> > 
> >
> > Do we actually hit the Overflow exception with this partition size?

I have confirmed this in my tests with trace logging.


> On March 10, 2014, 5:23 p.m., Jun Rao wrote:
> > core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala, 
> > lines 119-121
> > 
> >
> > Could we also verify the content of the fetched messages?

Ack.


> On March 10, 2014, 5:23 p.m., Jun Rao wrote:
> > core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala, 
> > lines 170-171
> > 
> >
> > Could we also verify the content of the fetched messages?

Ack.


> On March 10, 2014, 5:23 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/Record.java, lines 
> > 91-94
> > 
> >
> > It's still weird to always write the data as uncompressed. In 
> > RecordTest, the compressed type is not really being exercised since the 
> > record is created as uncompressed. Perhaps we should just change the unit 
> > test. If the compression type is not none, we will have to decompress the 
> > value to get the uncompressed content.

Added the comments for clearer explanation.


- Guozhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review36658
---


On March 8, 2014, 12:36 a.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> ---
> 
> (Updated March 8, 2014, 12:36 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
> 
> 
> Repository: kafk

Re: Review Request 18299: Fix KAFKA-1253

2014-03-10 Thread Guozhang Wang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
---

(Updated March 10, 2014, 9:36 p.m.)


Review request for kafka.


Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253


Repository: kafka


Description (updated)
---

Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Incorporate Jun's comments one


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy 2


Dummy


KAFKA-1253


Diffs (updated)
-

  build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
1ac69436f117800815b8d50f042e9e2a29364b43 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
32e12ad149f6d70c96a498d0a390976f77bf9e2a 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 616e1006f21c54af8260e84a96928cb8893ceb7c 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 038a05a94b795ec0a95b2d40a89222394b5a74c4 
  clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 
3ebbb804242be6a001b3bae6524afccc85a87602 
  
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java 
PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java
 PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 
906da02d02c03aadd8ab73ed2fc9a1898acb8d72 
  clients/src/main/java/org/apache/kafka/common/record/Compressor.java 
PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 
9d8935fa3beeb2a78b109a41ed76fd4374239560 
  clients/src/main/java/org/apache/kafka/common/record/Record.java 
f1dc9778502cbdfe982254fb6e25947842622239 
  clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
0c6b3656375721a718fb4de10118170aacce0ea9 
  clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java 
b0745b528cef929c4273f7e2ac4de1476cfc25ad 
  clients/src/test/java/org/apache/kafka/common/record/RecordTest.java 
ae54d67da9907b0a043180c7395a1370b3d0528d 
  clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION 
  clients/src/test/java/org/apache/kafka/test/TestUtils.java 
36cfc0fda742eb501af2c2c0330e3f461cf1f40c 
  core/src/main/scala/kafka/producer/ConsoleProducer.scala 
dd39ff22c918fe5b05f04582b748e32349b2055f 
  core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala 
PRE-CREATION 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
c002f5ea38ece66ad559fadb18ffaf40ac2026aa 
  core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 
66ea76b9b6c0f8839f715c845fb9b9671b8f35c1 

Diff: https://reviews.apache.org/r/18299/diff/


Testing
---

integration tests

snappy dynamic load test


Thanks,

Guozhang Wang

Re: Review Request 18299: Fix KAFKA-1253

2014-03-10 Thread Guozhang Wang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
---

(Updated March 10, 2014, 9:36 p.m.)


Review request for kafka.


Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253


Repository: kafka


Description (updated)
---

Incorporate Jun's comments round three

Refactored compression logic into Compressor

GZIP/Snappy integration tests


Diffs
-

  build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
1ac69436f117800815b8d50f042e9e2a29364b43 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
32e12ad149f6d70c96a498d0a390976f77bf9e2a 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 616e1006f21c54af8260e84a96928cb8893ceb7c 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 038a05a94b795ec0a95b2d40a89222394b5a74c4 
  clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 
3ebbb804242be6a001b3bae6524afccc85a87602 
  
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java 
PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java
 PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 
906da02d02c03aadd8ab73ed2fc9a1898acb8d72 
  clients/src/main/java/org/apache/kafka/common/record/Compressor.java 
PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 
9d8935fa3beeb2a78b109a41ed76fd4374239560 
  clients/src/main/java/org/apache/kafka/common/record/Record.java 
f1dc9778502cbdfe982254fb6e25947842622239 
  clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
0c6b3656375721a718fb4de10118170aacce0ea9 
  clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java 
b0745b528cef929c4273f7e2ac4de1476cfc25ad 
  clients/src/test/java/org/apache/kafka/common/record/RecordTest.java 
ae54d67da9907b0a043180c7395a1370b3d0528d 
  clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION 
  clients/src/test/java/org/apache/kafka/test/TestUtils.java 
36cfc0fda742eb501af2c2c0330e3f461cf1f40c 
  core/src/main/scala/kafka/producer/ConsoleProducer.scala 
dd39ff22c918fe5b05f04582b748e32349b2055f 
  core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala 
PRE-CREATION 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
c002f5ea38ece66ad559fadb18ffaf40ac2026aa 
  core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 
66ea76b9b6c0f8839f715c845fb9b9671b8f35c1 

Diff: https://reviews.apache.org/r/18299/diff/


Testing
---

integration tests

snappy dynamic load test


Thanks,

Guozhang Wang

[jira] [Updated] (KAFKA-1253) Implement compression in new producer

2014-03-10 Thread Guozhang Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1253:
-

Attachment: KAFKA-1253_2014-03-10_14:35:56.patch

> Implement compression in new producer
> -
>
> Key: KAFKA-1253
> URL: https://issues.apache.org/jira/browse/KAFKA-1253
> Project: Kafka
>  Issue Type: Sub-task
>  Components: producer 
>Reporter: Jay Kreps
>Assignee: Guozhang Wang
> Attachments: KAFKA-1253.patch, KAFKA-1253_2014-02-21_16:15:21.patch, 
> KAFKA-1253_2014-02-21_17:55:52.patch, KAFKA-1253_2014-02-24_13:31:50.patch, 
> KAFKA-1253_2014-02-26_17:31:30.patch, KAFKA-1253_2014-03-06_17:48:11.patch, 
> KAFKA-1253_2014-03-07_16:34:33.patch, KAFKA-1253_2014-03-10_14:35:56.patch, 
> compression-fix.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 18299: Fix KAFKA-1253

2014-03-10 Thread Guozhang Wang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
---

(Updated March 10, 2014, 9:37 p.m.)


Review request for kafka.


Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253


Repository: kafka


Description
---

Incorporate Jun's comments round three

Refactored compression logic into Compressor

GZIP/Snappy integration tests


Diffs
-

  build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
1ac69436f117800815b8d50f042e9e2a29364b43 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
32e12ad149f6d70c96a498d0a390976f77bf9e2a 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 616e1006f21c54af8260e84a96928cb8893ceb7c 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 038a05a94b795ec0a95b2d40a89222394b5a74c4 
  clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 
3ebbb804242be6a001b3bae6524afccc85a87602 
  
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java 
PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java
 PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 
906da02d02c03aadd8ab73ed2fc9a1898acb8d72 
  clients/src/main/java/org/apache/kafka/common/record/Compressor.java 
PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 
9d8935fa3beeb2a78b109a41ed76fd4374239560 
  clients/src/main/java/org/apache/kafka/common/record/Record.java 
f1dc9778502cbdfe982254fb6e25947842622239 
  clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
0c6b3656375721a718fb4de10118170aacce0ea9 
  clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java 
b0745b528cef929c4273f7e2ac4de1476cfc25ad 
  clients/src/test/java/org/apache/kafka/common/record/RecordTest.java 
ae54d67da9907b0a043180c7395a1370b3d0528d 
  clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION 
  clients/src/test/java/org/apache/kafka/test/TestUtils.java 
36cfc0fda742eb501af2c2c0330e3f461cf1f40c 
  core/src/main/scala/kafka/producer/ConsoleProducer.scala 
dd39ff22c918fe5b05f04582b748e32349b2055f 
  core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala 
PRE-CREATION 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
c002f5ea38ece66ad559fadb18ffaf40ac2026aa 
  core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 
66ea76b9b6c0f8839f715c845fb9b9671b8f35c1 

Diff: https://reviews.apache.org/r/18299/diff/


Testing
---

integration tests

snappy dynamic load test


Thanks,

Guozhang Wang

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-10 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13926257#comment-13926257
 ] 

Guozhang Wang commented on KAFKA-1253:
--

Updated reviewboard https://reviews.apache.org/r/18299/
 against branch origin/trunk

> Implement compression in new producer
> -
>
> Key: KAFKA-1253
> URL: https://issues.apache.org/jira/browse/KAFKA-1253
> Project: Kafka
>  Issue Type: Sub-task
>  Components: producer 
>Reporter: Jay Kreps
>Assignee: Guozhang Wang
> Attachments: KAFKA-1253.patch, KAFKA-1253_2014-02-21_16:15:21.patch, 
> KAFKA-1253_2014-02-21_17:55:52.patch, KAFKA-1253_2014-02-24_13:31:50.patch, 
> KAFKA-1253_2014-02-26_17:31:30.patch, KAFKA-1253_2014-03-06_17:48:11.patch, 
> KAFKA-1253_2014-03-07_16:34:33.patch, KAFKA-1253_2014-03-10_14:35:56.patch, 
> compression-fix.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[VOTE SUCCESS] Apache Kafka Release 0.8.1 - Candidate 2

2014-03-10 Thread Joe Stein

With 4 x +1 binding, 2 x +1 non-binding, 0 x 0 and 0 x -1 The Apache Kafka
Release 0.8.1 - Candidate 2 vote passes.

I will push the artifacts to maven central now.  Once that shows up
(hopefully later this evening/tomorrow) I will push the new artifacts to
repo and update the download page.

We could then switch over the documentation.  Jay, can you do this? Or let
me know what I need to-do to-do while updating the download page?

Once that is all done then I will send a release announce.

I will also go back through and update the release process wiki for
anything changed or new, etc.

Thanks everyone!!! Awesome!!!

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/


On Mon, Mar 10, 2014 at 2:14 PM, Sriram Subramanian <
srsubraman...@linkedin.com> wrote:

> +1
>
> Verified unit test and quick start.
>
> On 3/10/14 11:12 AM, "Joel Koshy"  wrote:
>
> >+1
> >
> >There is the zero-length md5 for 2.8.0-.8.1.tgz.md5 (zero missing).
> >Not sure if that is a blocker though since most people are off 2.8.0
> >
> >Joel
> >
> >On Mon, Mar 10, 2014 at 10:51:35AM -0700, Neha Narkhede wrote:
> >> +1, verified unit tests and quickstart.
> >>
> >> Thanks
> >> Neha
> >>
> >>
> >> On Mon, Mar 10, 2014 at 10:47 AM, Jay Kreps 
> wrote:
> >>
> >> > +1
> >> >
> >> > There are a few rough edges: slf4j error and the spurious error from
> >>the
> >> > console consumer but I'm okay if we fix these in 0.8.2.
> >> >
> >> > -Jay
> >> >
> >> >
> >> > On Tue, Mar 4, 2014 at 10:59 PM, Joe Stein 
> >>wrote:
> >> >
> >> > > This is the second candidate for release of Apache Kafka 0.8.1.
> >> > >
> >> > > This release candidate fixes the following two JIRA
> >> > > KAFKA-1288and
> >> > > KAFKA-1289  and
> >> > updated
> >> > > release steps with the gradle changes
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Process
> >>for
> >> > > build
> >> > > and verification post build.
> >> > >
> >> > > Release Notes (updated) for the 0.8.1 release
> >> > >
> >> > >
> >> >
> >>https://people.apache.org/~joestein/kafka-0.8.1-candidate2/RELEASE_NOTES
> .
> >>html
> >> > >
> >> > > *** Please download, test and vote by Monday, March 10th, 12pm PT
> >> > >
> >> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> >> > > http://svn.apache.org/repos/asf/kafka/KEYS in addition to the md5,
> >>sha1
> >> > > and
> >> > > sha2 (SHA256) checksum.
> >> > >
> >> > > * Release artifacts to be voted upon (source and binary):
> >> > > https://people.apache.org/~joestein/kafka-0.8.1-candidate2/
> >> > >
> >> > > * Maven artifacts to be voted upon prior to release:
> >> > > https://repository.apache.org/content/groups/staging/
> >> > >
> >> > > * The tag to be voted upon (off the 0.8.1 branch) is the 0.8.1 tag
> >> > >
> >> > >
> >> >
> >>
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=62f8aaf74c9d3
> >>6d1dd49cc7e572a7289206b6414
> >> > >
> >> > > /***
> >> > >  Joe Stein
> >> > >  Founder, Principal Consultant
> >> > >  Big Data Open Source Security LLC
> >> > >  http://www.stealth.ly
> >> > >  Twitter: @allthingshadoop 
> >> > > /
> >> > >
> >> >
> >
>
>

Re: Review Request 18299: Fix KAFKA-1253

2014-03-10 Thread Guozhang Wang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
---

(Updated March 10, 2014, 9:40 p.m.)


Review request for kafka.


Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253


Repository: kafka


Description (updated)
---

Small fix on ProducerPerformance


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Incorporate Jun's comments one


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy 2


Dummy


KAFKA-1253


Diffs (updated)
-

  build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
1ac69436f117800815b8d50f042e9e2a29364b43 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
32e12ad149f6d70c96a498d0a390976f77bf9e2a 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 616e1006f21c54af8260e84a96928cb8893ceb7c 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 038a05a94b795ec0a95b2d40a89222394b5a74c4 
  clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 
3ebbb804242be6a001b3bae6524afccc85a87602 
  
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java 
PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java
 PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 
906da02d02c03aadd8ab73ed2fc9a1898acb8d72 
  clients/src/main/java/org/apache/kafka/common/record/Compressor.java 
PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 
9d8935fa3beeb2a78b109a41ed76fd4374239560 
  clients/src/main/java/org/apache/kafka/common/record/Record.java 
f1dc9778502cbdfe982254fb6e25947842622239 
  clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
0c6b3656375721a718fb4de10118170aacce0ea9 
  clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java 
b0745b528cef929c4273f7e2ac4de1476cfc25ad 
  clients/src/test/java/org/apache/kafka/common/record/RecordTest.java 
ae54d67da9907b0a043180c7395a1370b3d0528d 
  clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION 
  clients/src/test/java/org/apache/kafka/test/TestUtils.java 
36cfc0fda742eb501af2c2c0330e3f461cf1f40c 
  core/src/main/scala/kafka/producer/ConsoleProducer.scala 
dd39ff22c918fe5b05f04582b748e32349b2055f 
  core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala 
PRE-CREATION 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
c002f5ea38ece66ad559fadb18ffaf40ac2026aa 
  core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 
66ea76b9b6c0f8839f715c845fb9b9671b8f35c1 

Diff: https://reviews.apache.org/r/18299/diff/


Testing
---

integration tests

snappy dynamic load test


Thanks,

Guozhang Wang

[jira] [Updated] (KAFKA-1253) Implement compression in new producer

2014-03-10 Thread Guozhang Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1253:
-

Attachment: KAFKA-1253_2014-03-10_14:39:58.patch

> Implement compression in new producer
> -
>
> Key: KAFKA-1253
> URL: https://issues.apache.org/jira/browse/KAFKA-1253
> Project: Kafka
>  Issue Type: Sub-task
>  Components: producer 
>Reporter: Jay Kreps
>Assignee: Guozhang Wang
> Attachments: KAFKA-1253.patch, KAFKA-1253_2014-02-21_16:15:21.patch, 
> KAFKA-1253_2014-02-21_17:55:52.patch, KAFKA-1253_2014-02-24_13:31:50.patch, 
> KAFKA-1253_2014-02-26_17:31:30.patch, KAFKA-1253_2014-03-06_17:48:11.patch, 
> KAFKA-1253_2014-03-07_16:34:33.patch, KAFKA-1253_2014-03-10_14:35:56.patch, 
> KAFKA-1253_2014-03-10_14:39:58.patch, compression-fix.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 18299: Fix KAFKA-1253

2014-03-10 Thread Guozhang Wang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
---

(Updated March 10, 2014, 9:40 p.m.)


Review request for kafka.


Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253


Repository: kafka


Description (updated)
---

Incorporate Jun's comments round three

Refactored compression logic into Compressor

GZIP/Snappy integration tests


Diffs
-

  build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
1ac69436f117800815b8d50f042e9e2a29364b43 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
32e12ad149f6d70c96a498d0a390976f77bf9e2a 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 616e1006f21c54af8260e84a96928cb8893ceb7c 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 038a05a94b795ec0a95b2d40a89222394b5a74c4 
  clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 
3ebbb804242be6a001b3bae6524afccc85a87602 
  
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java 
PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java
 PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 
906da02d02c03aadd8ab73ed2fc9a1898acb8d72 
  clients/src/main/java/org/apache/kafka/common/record/Compressor.java 
PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 
9d8935fa3beeb2a78b109a41ed76fd4374239560 
  clients/src/main/java/org/apache/kafka/common/record/Record.java 
f1dc9778502cbdfe982254fb6e25947842622239 
  clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
0c6b3656375721a718fb4de10118170aacce0ea9 
  clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java 
b0745b528cef929c4273f7e2ac4de1476cfc25ad 
  clients/src/test/java/org/apache/kafka/common/record/RecordTest.java 
ae54d67da9907b0a043180c7395a1370b3d0528d 
  clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION 
  clients/src/test/java/org/apache/kafka/test/TestUtils.java 
36cfc0fda742eb501af2c2c0330e3f461cf1f40c 
  core/src/main/scala/kafka/producer/ConsoleProducer.scala 
dd39ff22c918fe5b05f04582b748e32349b2055f 
  core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala 
PRE-CREATION 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
c002f5ea38ece66ad559fadb18ffaf40ac2026aa 
  core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 
66ea76b9b6c0f8839f715c845fb9b9671b8f35c1 

Diff: https://reviews.apache.org/r/18299/diff/


Testing
---

integration tests

snappy dynamic load test


Thanks,

Guozhang Wang

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-10 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13926267#comment-13926267
 ] 

Guozhang Wang commented on KAFKA-1253:
--

Updated reviewboard https://reviews.apache.org/r/18299/
 against branch origin/trunk

> Implement compression in new producer
> -
>
> Key: KAFKA-1253
> URL: https://issues.apache.org/jira/browse/KAFKA-1253
> Project: Kafka
>  Issue Type: Sub-task
>  Components: producer 
>Reporter: Jay Kreps
>Assignee: Guozhang Wang
> Attachments: KAFKA-1253.patch, KAFKA-1253_2014-02-21_16:15:21.patch, 
> KAFKA-1253_2014-02-21_17:55:52.patch, KAFKA-1253_2014-02-24_13:31:50.patch, 
> KAFKA-1253_2014-02-26_17:31:30.patch, KAFKA-1253_2014-03-06_17:48:11.patch, 
> KAFKA-1253_2014-03-07_16:34:33.patch, KAFKA-1253_2014-03-10_14:35:56.patch, 
> KAFKA-1253_2014-03-10_14:39:58.patch, compression-fix.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-10 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13926296#comment-13926296
 ] 

Guozhang Wang commented on KAFKA-1253:
--

Some interesting performance numbers (1000 messages, 1K message size, ack=1):

  new-producer-performance 
old-producer-performance-new-producer  
old-producer-performance-old-producer

none  4329.00   
3246.7532   
2183.4061
gzip   3134.80   
1620.7455   
1754.3860
snappy  3731.34   
2247.1910   
1890.3592




> Implement compression in new producer
> -
>
> Key: KAFKA-1253
> URL: https://issues.apache.org/jira/browse/KAFKA-1253
> Project: Kafka
>  Issue Type: Sub-task
>  Components: producer 
>Reporter: Jay Kreps
>Assignee: Guozhang Wang
> Attachments: KAFKA-1253.patch, KAFKA-1253_2014-02-21_16:15:21.patch, 
> KAFKA-1253_2014-02-21_17:55:52.patch, KAFKA-1253_2014-02-24_13:31:50.patch, 
> KAFKA-1253_2014-02-26_17:31:30.patch, KAFKA-1253_2014-03-06_17:48:11.patch, 
> KAFKA-1253_2014-03-07_16:34:33.patch, KAFKA-1253_2014-03-10_14:35:56.patch, 
> KAFKA-1253_2014-03-10_14:39:58.patch, compression-fix.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Comment Edited] (KAFKA-1253) Implement compression in new producer

2014-03-10 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13926296#comment-13926296
 ] 

Guozhang Wang edited comment on KAFKA-1253 at 3/10/14 10:01 PM:


Some interesting performance numbers (1000 messages, 1K message size, ack=1):

  new-producer-performance 
old-producer-performance-new-producer  
old-producer-performance-old-producer

none  4329.00   
3246.7532   
2183.4061
gzip   3134.80   
1620.7455   
1754.3860
snappy  3731.34   
2247.1910   
1890.3592


The difference between the old and new producer performance is that for the old 
producer performance, each message creates a new byte array, while for the new 
one the same byte array is reused for all messages.



was (Author: guozhang):
Some interesting performance numbers (1000 messages, 1K message size, ack=1):

  new-producer-performance 
old-producer-performance-new-producer  
old-producer-performance-old-producer

none  4329.00   
3246.7532   
2183.4061
gzip   3134.80   
1620.7455   
1754.3860
snappy  3731.34   
2247.1910   
1890.3592




> Implement compression in new producer
> -
>
> Key: KAFKA-1253
> URL: https://issues.apache.org/jira/browse/KAFKA-1253
> Project: Kafka
>  Issue Type: Sub-task
>  Components: producer 
>Reporter: Jay Kreps
>Assignee: Guozhang Wang
> Attachments: KAFKA-1253.patch, KAFKA-1253_2014-02-21_16:15:21.patch, 
> KAFKA-1253_2014-02-21_17:55:52.patch, KAFKA-1253_2014-02-24_13:31:50.patch, 
> KAFKA-1253_2014-02-26_17:31:30.patch, KAFKA-1253_2014-03-06_17:48:11.patch, 
> KAFKA-1253_2014-03-07_16:34:33.patch, KAFKA-1253_2014-03-10_14:35:56.patch, 
> KAFKA-1253_2014-03-10_14:39:58.patch, compression-fix.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 18299: Fix KAFKA-1253

2014-03-10 Thread Guozhang Wang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
---

(Updated March 10, 2014, 10:28 p.m.)


Review request for kafka.


Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253


Repository: kafka


Description (updated)
---

Dummy


Small fix on ProducerPerformance


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Incorporate Jun's comments one


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy


Dummy 2


Dummy


KAFKA-1253


Diffs (updated)
-

  build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
1ac69436f117800815b8d50f042e9e2a29364b43 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
32e12ad149f6d70c96a498d0a390976f77bf9e2a 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 616e1006f21c54af8260e84a96928cb8893ceb7c 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 038a05a94b795ec0a95b2d40a89222394b5a74c4 
  clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 
3ebbb804242be6a001b3bae6524afccc85a87602 
  
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java 
PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java
 PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 
906da02d02c03aadd8ab73ed2fc9a1898acb8d72 
  clients/src/main/java/org/apache/kafka/common/record/Compressor.java 
PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 
9d8935fa3beeb2a78b109a41ed76fd4374239560 
  clients/src/main/java/org/apache/kafka/common/record/Record.java 
f1dc9778502cbdfe982254fb6e25947842622239 
  clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
0c6b3656375721a718fb4de10118170aacce0ea9 
  clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java 
b0745b528cef929c4273f7e2ac4de1476cfc25ad 
  clients/src/test/java/org/apache/kafka/common/record/RecordTest.java 
ae54d67da9907b0a043180c7395a1370b3d0528d 
  clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION 
  clients/src/test/java/org/apache/kafka/test/TestUtils.java 
36cfc0fda742eb501af2c2c0330e3f461cf1f40c 
  core/src/main/scala/kafka/producer/ConsoleProducer.scala 
dd39ff22c918fe5b05f04582b748e32349b2055f 
  core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala 
PRE-CREATION 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
c002f5ea38ece66ad559fadb18ffaf40ac2026aa 
  core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 
66ea76b9b6c0f8839f715c845fb9b9671b8f35c1 
  perf/src/main/scala/kafka/perf/ProducerPerformance.scala 
f061dbabb49044f8fe94b8fd7dc0153c33bedeee 

Diff: https://reviews.apache.org/r/18299/diff/


Testing
---

integration tests

snappy dynamic load test


Thanks,

Guozhang Wang

Re: Review Request 18299: Fix KAFKA-1253

2014-03-10 Thread Guozhang Wang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
---

(Updated March 10, 2014, 10:28 p.m.)


Review request for kafka.


Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253


Repository: kafka


Description (updated)
---

Incorporate Jun's comments round three

Refactored compression logic into Compressor

GZIP/Snappy integration tests


Diffs
-

  build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
1ac69436f117800815b8d50f042e9e2a29364b43 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
32e12ad149f6d70c96a498d0a390976f77bf9e2a 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 616e1006f21c54af8260e84a96928cb8893ceb7c 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 038a05a94b795ec0a95b2d40a89222394b5a74c4 
  clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 
3ebbb804242be6a001b3bae6524afccc85a87602 
  
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java 
PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java
 PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 
906da02d02c03aadd8ab73ed2fc9a1898acb8d72 
  clients/src/main/java/org/apache/kafka/common/record/Compressor.java 
PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 
9d8935fa3beeb2a78b109a41ed76fd4374239560 
  clients/src/main/java/org/apache/kafka/common/record/Record.java 
f1dc9778502cbdfe982254fb6e25947842622239 
  clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
0c6b3656375721a718fb4de10118170aacce0ea9 
  clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java 
b0745b528cef929c4273f7e2ac4de1476cfc25ad 
  clients/src/test/java/org/apache/kafka/common/record/RecordTest.java 
ae54d67da9907b0a043180c7395a1370b3d0528d 
  clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION 
  clients/src/test/java/org/apache/kafka/test/TestUtils.java 
36cfc0fda742eb501af2c2c0330e3f461cf1f40c 
  core/src/main/scala/kafka/producer/ConsoleProducer.scala 
dd39ff22c918fe5b05f04582b748e32349b2055f 
  core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala 
PRE-CREATION 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
c002f5ea38ece66ad559fadb18ffaf40ac2026aa 
  core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 
66ea76b9b6c0f8839f715c845fb9b9671b8f35c1 
  perf/src/main/scala/kafka/perf/ProducerPerformance.scala 
f061dbabb49044f8fe94b8fd7dc0153c33bedeee 

Diff: https://reviews.apache.org/r/18299/diff/


Testing
---

integration tests

snappy dynamic load test


Thanks,

Guozhang Wang

[jira] [Updated] (KAFKA-1253) Implement compression in new producer

2014-03-10 Thread Guozhang Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1253:
-

Attachment: KAFKA-1253_2014-03-10_15:27:47.patch

> Implement compression in new producer
> -
>
> Key: KAFKA-1253
> URL: https://issues.apache.org/jira/browse/KAFKA-1253
> Project: Kafka
>  Issue Type: Sub-task
>  Components: producer 
>Reporter: Jay Kreps
>Assignee: Guozhang Wang
> Attachments: KAFKA-1253.patch, KAFKA-1253_2014-02-21_16:15:21.patch, 
> KAFKA-1253_2014-02-21_17:55:52.patch, KAFKA-1253_2014-02-24_13:31:50.patch, 
> KAFKA-1253_2014-02-26_17:31:30.patch, KAFKA-1253_2014-03-06_17:48:11.patch, 
> KAFKA-1253_2014-03-07_16:34:33.patch, KAFKA-1253_2014-03-10_14:35:56.patch, 
> KAFKA-1253_2014-03-10_14:39:58.patch, KAFKA-1253_2014-03-10_15:27:47.patch, 
> compression-fix.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-10 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13926348#comment-13926348
 ] 

Guozhang Wang commented on KAFKA-1253:
--

Updated reviewboard https://reviews.apache.org/r/18299/
 against branch origin/trunk

> Implement compression in new producer
> -
>
> Key: KAFKA-1253
> URL: https://issues.apache.org/jira/browse/KAFKA-1253
> Project: Kafka
>  Issue Type: Sub-task
>  Components: producer 
>Reporter: Jay Kreps
>Assignee: Guozhang Wang
> Attachments: KAFKA-1253.patch, KAFKA-1253_2014-02-21_16:15:21.patch, 
> KAFKA-1253_2014-02-21_17:55:52.patch, KAFKA-1253_2014-02-24_13:31:50.patch, 
> KAFKA-1253_2014-02-26_17:31:30.patch, KAFKA-1253_2014-03-06_17:48:11.patch, 
> KAFKA-1253_2014-03-07_16:34:33.patch, KAFKA-1253_2014-03-10_14:35:56.patch, 
> KAFKA-1253_2014-03-10_14:39:58.patch, KAFKA-1253_2014-03-10_15:27:47.patch, 
> compression-fix.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 18299: Fix KAFKA-1253

2014-03-10 Thread Jay Kreps


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review36732
---



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java


I don't see why we need this. We always take the max of the batch size and 
the base data size, no?



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java


This logic should happen inside tryAppend, no?



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java


This is a check method, we can't change the state.



clients/src/main/java/org/apache/kafka/common/record/CompressionType.java


This seems like it doesn't belong in the enum.



clients/src/main/java/org/apache/kafka/common/record/CompressionType.java


I'm concerned about wrapping with DataOutputStream as it adds a layer of 
synchronization.



clients/src/main/java/org/apache/kafka/common/record/CompressionType.java


Do we really need to do this? I think just using new should be okay, no?



clients/src/main/java/org/apache/kafka/common/record/CompressionType.java


This also seems like it doesn't belong in the enum.



clients/src/main/java/org/apache/kafka/common/record/Compressor.java


Why is this lazy?



clients/src/main/java/org/apache/kafka/common/record/Compressor.java


These interfaces are pretty gross, is there no other way to do this?



clients/src/main/java/org/apache/kafka/common/record/Compressor.java


If this is expected don't log it as warn log it as debug. WARN==user should 
do something different in their code.



clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java


Code style: please make two lines:

if (written)
  compressor.written()



clients/src/main/java/org/apache/kafka/common/utils/Utils.java


Why isn't this part of the CRC class?


- Jay Kreps


On March 10, 2014, 10:28 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> ---
> 
> (Updated March 10, 2014, 10:28 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Incorporate Jun's comments round three
> 
> Refactored compression logic into Compressor
> 
> GZIP/Snappy integration tests
> 
> 
> Diffs
> -
> 
>   build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 1ac69436f117800815b8d50f042e9e2a29364b43 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 32e12ad149f6d70c96a498d0a390976f77bf9e2a 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  616e1006f21c54af8260e84a96928cb8893ceb7c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  038a05a94b795ec0a95b2d40a89222394b5a74c4 
>   
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 
> 3ebbb804242be6a001b3bae6524afccc85a87602 
>   
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java
>  PRE-CREATION 
>   
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java
>  PRE-CREATION 
>   clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 
> 906da02d02c03aadd8ab73ed2fc9a1898acb8d72 
>   clients/src/main/java/org/apache/kafka/common/record/Compressor.java 
> PRE-CREATION 
>   clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 
> 9d8935fa3beeb2a78b109a41ed76fd4374239560 
>   clients/src/main/java/org/apache/kafka/common/record/Record.java 
> f1dc9778502cbdfe982254fb6e25947842622239 
>   clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
> 0c6b3656375721a718fb4de10118170aacce0ea9 
>   clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java 
> b0745b528cef929c4273f7e2ac4de1476cfc25ad 
>   clients/src/test/java/org/apache/kafka/common/record/RecordTest.java 
> ae54d67da9907b0a043180c7395

Re: [VOTE SUCCESS] Apache Kafka Release 0.8.1 - Candidate 2

2014-03-10 Thread Jun Rao

Joe,

Thanks for driving the release.

Jun


On Mon, Mar 10, 2014 at 2:37 PM, Joe Stein  wrote:

> With 4 x +1 binding, 2 x +1 non-binding, 0 x 0 and 0 x -1 The Apache Kafka
> Release 0.8.1 - Candidate 2 vote passes.
>
> I will push the artifacts to maven central now.  Once that shows up
> (hopefully later this evening/tomorrow) I will push the new artifacts to
> repo and update the download page.
>
> We could then switch over the documentation.  Jay, can you do this? Or let
> me know what I need to-do to-do while updating the download page?
>
> Once that is all done then I will send a release announce.
>
> I will also go back through and update the release process wiki for
> anything changed or new, etc.
>
> Thanks everyone!!! Awesome!!!
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
>
>
> On Mon, Mar 10, 2014 at 2:14 PM, Sriram Subramanian <
> srsubraman...@linkedin.com> wrote:
>
> > +1
> >
> > Verified unit test and quick start.
> >
> > On 3/10/14 11:12 AM, "Joel Koshy"  wrote:
> >
> > >+1
> > >
> > >There is the zero-length md5 for 2.8.0-.8.1.tgz.md5 (zero missing).
> > >Not sure if that is a blocker though since most people are off 2.8.0
> > >
> > >Joel
> > >
> > >On Mon, Mar 10, 2014 at 10:51:35AM -0700, Neha Narkhede wrote:
> > >> +1, verified unit tests and quickstart.
> > >>
> > >> Thanks
> > >> Neha
> > >>
> > >>
> > >> On Mon, Mar 10, 2014 at 10:47 AM, Jay Kreps 
> > wrote:
> > >>
> > >> > +1
> > >> >
> > >> > There are a few rough edges: slf4j error and the spurious error from
> > >>the
> > >> > console consumer but I'm okay if we fix these in 0.8.2.
> > >> >
> > >> > -Jay
> > >> >
> > >> >
> > >> > On Tue, Mar 4, 2014 at 10:59 PM, Joe Stein 
> > >>wrote:
> > >> >
> > >> > > This is the second candidate for release of Apache Kafka 0.8.1.
> > >> > >
> > >> > > This release candidate fixes the following two JIRA
> > >> > > KAFKA-1288and
> > >> > > KAFKA-1289  and
> > >> > updated
> > >> > > release steps with the gradle changes
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Process
> > >>for
> > >> > > build
> > >> > > and verification post build.
> > >> > >
> > >> > > Release Notes (updated) for the 0.8.1 release
> > >> > >
> > >> > >
> > >> >
> > >>
> https://people.apache.org/~joestein/kafka-0.8.1-candidate2/RELEASE_NOTES
> > .
> > >>html
> > >> > >
> > >> > > *** Please download, test and vote by Monday, March 10th, 12pm PT
> > >> > >
> > >> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > >> > > http://svn.apache.org/repos/asf/kafka/KEYS in addition to the
> md5,
> > >>sha1
> > >> > > and
> > >> > > sha2 (SHA256) checksum.
> > >> > >
> > >> > > * Release artifacts to be voted upon (source and binary):
> > >> > > https://people.apache.org/~joestein/kafka-0.8.1-candidate2/
> > >> > >
> > >> > > * Maven artifacts to be voted upon prior to release:
> > >> > > https://repository.apache.org/content/groups/staging/
> > >> > >
> > >> > > * The tag to be voted upon (off the 0.8.1 branch) is the 0.8.1 tag
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=62f8aaf74c9d3
> > >>6d1dd49cc7e572a7289206b6414
> > >> > >
> > >> > > /***
> > >> > >  Joe Stein
> > >> > >  Founder, Principal Consultant
> > >> > >  Big Data Open Source Security LLC
> > >> > >  http://www.stealth.ly
> > >> > >  Twitter: @allthingshadoop <
> http://www.twitter.com/allthingshadoop>
> > >> > > /
> > >> > >
> > >> >
> > >
> >
> >
>

[GitHub] kafka pull request: Update LogManager.scala

2014-03-10 Thread glorage

GitHub user glorage opened a pull request:

https://github.com/apache/kafka/pull/19

Update LogManager.scala

Make sure newer segments would not be deleted

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/glorage/kafka 0.8.1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/19.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19


commit 54508d3e0ff36db30e5be37ccda5f6886ce9a2d7
Author: glorage 
Date:   2014-03-11T03:28:43Z

Update LogManager.scala

Make sure newer segments would not be deleted




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: command line tools

Re: command line tools

Re: Review Request 18299: Fix KAFKA-1253

[jira] [Created] (KAFKA-1301) system testcase_0206 fails using the new producer

[jira] [Updated] (KAFKA-1301) system testcase_0206 fails using the new producer

Re: [VOTE] Apache Kafka Release 0.8.1 - Candidate 2

Re: [VOTE] Apache Kafka Release 0.8.1 - Candidate 2

Re: [VOTE] Apache Kafka Release 0.8.1 - Candidate 2

Re: [VOTE] Apache Kafka Release 0.8.1 - Candidate 2

Re: [VOTE] Apache Kafka Release 0.8.1 - Candidate 2

[jira] Subscription: outstanding kafka patches

Re: Review Request 18299: Fix KAFKA-1253

Re: Review Request 18299: Fix KAFKA-1253

Re: Review Request 18299: Fix KAFKA-1253

[jira] [Updated] (KAFKA-1253) Implement compression in new producer

Re: Review Request 18299: Fix KAFKA-1253

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

[VOTE SUCCESS] Apache Kafka Release 0.8.1 - Candidate 2

Re: Review Request 18299: Fix KAFKA-1253

[jira] [Updated] (KAFKA-1253) Implement compression in new producer

Re: Review Request 18299: Fix KAFKA-1253

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

[jira] [Comment Edited] (KAFKA-1253) Implement compression in new producer

Re: Review Request 18299: Fix KAFKA-1253

Re: Review Request 18299: Fix KAFKA-1253

[jira] [Updated] (KAFKA-1253) Implement compression in new producer

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

Re: Review Request 18299: Fix KAFKA-1253

Re: [VOTE SUCCESS] Apache Kafka Release 0.8.1 - Candidate 2

[GitHub] kafka pull request: Update LogManager.scala

31 matches

Site Navigation

Mail list logo

Footer information