Kafka development subscription

[Prashant Prakash]

Re: Review Request 22874: Fix KAFKA-1498

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22874/#review47075
---


Could we add a record-size-avg metric in Sender?


clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java


Is the comment still valid? I don't see the default select timeout.



clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java


Is the comment still valid? Don't see the default select time value.


- Jun Rao


On June 30, 2014, 10:47 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22874/
> ---
> 
> (Updated June 30, 2014, 10:47 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1498
> https://issues.apache.org/jira/browse/KAFKA-1498
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> 1. Use a size limit on the memory records to guard too-large message cases; 
> 2. Caller thread check partition readiness due to batch size upon append, and 
> only wake up sender when the appended partition is ready; 3. Sender thread 
> select time based on the partition readiness timeout and metadata timeout. 4. 
> Mirror maker to use one blocking queue per producer thread. 5. The select 
> time in NetworkClient is minimum of the partition readiness (if there is no 
> data at all will be Long.MAX_VALUE) and metadata expiry timeout, hence upper 
> bounded by the metadata age; On the other hand when the appended batch is 
> full or a new batch is created, the producer thread will wake up the network 
> thread. 6. Also fixed a transient test bug in SocketServerTest and speed-up 
> the testNoResponse case of ProducerFailureHandling. 7. Other minor fixes.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 522881c972ca42ff4dfb6237a2db15b625334d7e 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 00775abbcac850b0f2bb9a70b6fbc7cdf319bcf6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Metadata.java
>  57bc285c20b5af8957bcc5322cd75c021a5af215 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  1ed3c28b436d28381d9402896e32d16f2586c65e 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> 6fb5b82dedb48d946d1ac1ec7a535bddfdc693fa 
>   clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 
> 759f577eaf0e7d28a84926d4aa30f4ef0cb27bc2 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 6a3cdcc1f2542479f37bc339baca87464c01e84e 
>   clients/src/test/java/org/apache/kafka/clients/producer/MetadataTest.java 
> 8b4ac0f9a59b4f2e67e48e6d9b0d9fe340f77166 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java
>  93b58d02eac0f8ca28440e3e0ebea28ed3a7673c 
>   clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java 
> 5489acac6806b3ae5e6d568d401d5a20c86cac05 
>   core/src/main/scala/kafka/tools/MirrorMaker.scala 
> 763839157d9736f15110072bcae93fc7fdc33f55 
>   core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
> d1464447c65231abeaa52e94d5a48cf62f054881 
>   core/src/test/scala/unit/kafka/network/SocketServerTest.scala 
> 62fb02cf02d3876b9804d756c4bf8514554cc836 
> 
> Diff: https://reviews.apache.org/r/22874/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

Re: Review Request 22874: Fix KAFKA-1498

[Jay Kreps]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22874/#review47100
---

Ship it!


- Jay Kreps


On June 30, 2014, 10:47 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22874/
> ---
> 
> (Updated June 30, 2014, 10:47 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1498
> https://issues.apache.org/jira/browse/KAFKA-1498
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> 1. Use a size limit on the memory records to guard too-large message cases; 
> 2. Caller thread check partition readiness due to batch size upon append, and 
> only wake up sender when the appended partition is ready; 3. Sender thread 
> select time based on the partition readiness timeout and metadata timeout. 4. 
> Mirror maker to use one blocking queue per producer thread. 5. The select 
> time in NetworkClient is minimum of the partition readiness (if there is no 
> data at all will be Long.MAX_VALUE) and metadata expiry timeout, hence upper 
> bounded by the metadata age; On the other hand when the appended batch is 
> full or a new batch is created, the producer thread will wake up the network 
> thread. 6. Also fixed a transient test bug in SocketServerTest and speed-up 
> the testNoResponse case of ProducerFailureHandling. 7. Other minor fixes.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 522881c972ca42ff4dfb6237a2db15b625334d7e 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 00775abbcac850b0f2bb9a70b6fbc7cdf319bcf6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Metadata.java
>  57bc285c20b5af8957bcc5322cd75c021a5af215 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  1ed3c28b436d28381d9402896e32d16f2586c65e 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> 6fb5b82dedb48d946d1ac1ec7a535bddfdc693fa 
>   clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 
> 759f577eaf0e7d28a84926d4aa30f4ef0cb27bc2 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 6a3cdcc1f2542479f37bc339baca87464c01e84e 
>   clients/src/test/java/org/apache/kafka/clients/producer/MetadataTest.java 
> 8b4ac0f9a59b4f2e67e48e6d9b0d9fe340f77166 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java
>  93b58d02eac0f8ca28440e3e0ebea28ed3a7673c 
>   clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java 
> 5489acac6806b3ae5e6d568d401d5a20c86cac05 
>   core/src/main/scala/kafka/tools/MirrorMaker.scala 
> 763839157d9736f15110072bcae93fc7fdc33f55 
>   core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
> d1464447c65231abeaa52e94d5a48cf62f054881 
>   core/src/test/scala/unit/kafka/network/SocketServerTest.scala 
> 62fb02cf02d3876b9804d756c4bf8514554cc836 
> 
> Diff: https://reviews.apache.org/r/22874/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

[jira] [Commented] (KAFKA-1512) Limit the maximum number of connections per ip address

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049117#comment-14049117
 ] 

Jay Kreps commented on KAFKA-1512:
--

A couple things to review:
0. The new config is max.connections.per.ip
1. I am using Socket.getInetAddress() as the key to limit on. I think an 
InetAddress is what we want...a socket address includes the port so is always 
unique, but there is sort of a weird hierarchy of things there. This also 
depends on this address being properly hashable (which it seems to be).
2. I made an unrelated change to how we set the recv buffer. We were weirdly 
setting this over and over again on the server socket every time we accepted a 
connection. I think this was a mistake, so I changed it to set it once. But if 
anyone knows a reason for this odd code that would make me more confident.
3. I don't know of a way to check the source address of a pending connection 
without actually accepting the connection. So as a result this patch accepts 
the connection, and then, if we are over quota, closes it.

> Limit the maximum number of connections per ip address
> --
>
> Key: KAFKA-1512
> URL: https://issues.apache.org/jira/browse/KAFKA-1512
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Jay Kreps
>Assignee: Jay Kreps
> Attachments: KAFKA-1512.patch
>
>
> To protect against client connection leaks add a new configuration
>   max.connections.per.ip
> that causes the SocketServer to enforce a limit on the maximum number of 
> connections from each InetAddress instance. For backwards compatibility this 
> will default to 2 billion.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1204) Add KAFKA_LOG_DIR to allow LOG_DIR to be outside of code dir

[J Potter (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049119#comment-14049119
 ] 

J Potter commented on KAFKA-1204:
-

Hi Guozhang,

It looks like trunk has a slightly different (and perfectly fine) fix for this 
issue, so this ticket can be marked as resolved by whatever version trunk 
currently reflects.

Thanks!
-Jeff

> Add KAFKA_LOG_DIR to allow LOG_DIR to be outside of code dir
> 
>
> Key: KAFKA-1204
> URL: https://issues.apache.org/jira/browse/KAFKA-1204
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Reporter: J Potter
>  Labels: patch
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> The kafka-run-class.sh script currently generates a LOG_DIR variable and 
> expects the path under base_dir/logs to be writeable by the process, without 
> a way of overriding it.
> I think a simple change as below would allow ops folks to fire it up and 
> point the LOG_DIR elsewhere, as desired.
> bin/kafka-run-class.sh
> @@ -23,9 +23,11 @@ fi
>  base_dir=$(dirname $0)/..
>  
>  # create logs directory
> -LOG_DIR=$base_dir/logs
> -if [ ! -d $LOG_DIR ]; then
> -   mkdir $LOG_DIR
> +if [ -z "KAFKA_LOG_DIR" ]; then
> +KAFKA_LOG_DIR=$base_dir/logs
> +fi
> +if [ ! -d $KAFKA_LOG_DIR ]; then
> +   mkdir $KAFKA_LOG_DIR
>  fi
>  
>  if [ -z "$SCALA_VERSION" ]; then
> @@ -98,7 +100,7 @@ if [ "$1" = "daemon" ] && [ -z "$KAFKA_GC_LOG_OPTS"] ; then
>shift
>GC_LOG_FILE_NAME=$1$GC_FILE_SUFFIX
>shift
> -  KAFKA_GC_LOG_OPTS="-Xloggc:$LOG_DIR/$GC_LOG_FILE_NAME -verbose:gc 
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps "
> +  KAFKA_GC_LOG_OPTS="-Xloggc:$KAFKA_LOG_DIR/$GC_LOG_FILE_NAME -verbose:gc 
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps "
>  fi
>  
>  exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS 
> $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$@"



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1308) Publish jar of test utilities to Maven

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049123#comment-14049123
 ] 

Jun Rao commented on KAFKA-1308:


Jakob,

Do you plan to commit the patch to trunk too?

Thanks,

> Publish jar of test utilities to Maven
> --
>
> Key: KAFKA-1308
> URL: https://issues.apache.org/jira/browse/KAFKA-1308
> Project: Kafka
>  Issue Type: Wish
>Affects Versions: 0.8.1
>Reporter: Martin Kleppmann
>Assignee: Jakob Homan
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: KAFKA-1308-2.patch, KAFKA-1308-8.1.1.patch, 
> KAFKA-1308.patch, KAFKA-1308.patch, kafka-1308_v2_0.8.1.patch
>
>
> For projects that use Kafka, and want to write tests that exercise Kafka (in 
> our case, Samza), it's useful to have access to Kafka's test utility classes 
> such as kafka.zk.EmbeddedZookeeper and kafka.utils.TestUtils. We can use 
> {{./gradlew testJar}} to build jar files that contain those classes, but as 
> far as I know, these are currently not made available in a binary release.
> At the moment, we have to check those kafka*-test.jar files into the Samza 
> repository. To avoid that, would it be possible to publish those jars of 
> tests to Maven, so that they fit into the normal dependency management?
> Or perhaps, if publishing the tests themselves is not appropriate, we could 
> move the test utilities into a separate module that is published, and make 
> the tests depend on that module?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 22874: Fix KAFKA-1498

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22874/
---

(Updated July 1, 2014, 6:12 p.m.)


Review request for kafka.


Bugs: KAFKA-1498
https://issues.apache.org/jira/browse/KAFKA-1498


Repository: kafka


Description
---

1. Use a size limit on the memory records to guard too-large message cases; 2. 
Caller thread check partition readiness due to batch size upon append, and only 
wake up sender when the appended partition is ready; 3. Sender thread select 
time based on the partition readiness timeout and metadata timeout. 4. Mirror 
maker to use one blocking queue per producer thread. 5. The select time in 
NetworkClient is minimum of the partition readiness (if there is no data at all 
will be Long.MAX_VALUE) and metadata expiry timeout, hence upper bounded by the 
metadata age; On the other hand when the appended batch is full or a new batch 
is created, the producer thread will wake up the network thread. 6. Also fixed 
a transient test bug in SocketServerTest and speed-up the testNoResponse case 
of ProducerFailureHandling. 7. Other minor fixes.


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
522881c972ca42ff4dfb6237a2db15b625334d7e 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
00775abbcac850b0f2bb9a70b6fbc7cdf319bcf6 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/Metadata.java 
57bc285c20b5af8957bcc5322cd75c021a5af215 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 1ed3c28b436d28381d9402896e32d16f2586c65e 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
6fb5b82dedb48d946d1ac1ec7a535bddfdc693fa 
  clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 
759f577eaf0e7d28a84926d4aa30f4ef0cb27bc2 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
6a3cdcc1f2542479f37bc339baca87464c01e84e 
  clients/src/test/java/org/apache/kafka/clients/producer/MetadataTest.java 
8b4ac0f9a59b4f2e67e48e6d9b0d9fe340f77166 
  
clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java
 93b58d02eac0f8ca28440e3e0ebea28ed3a7673c 
  clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java 
5489acac6806b3ae5e6d568d401d5a20c86cac05 
  core/src/main/scala/kafka/tools/MirrorMaker.scala 
763839157d9736f15110072bcae93fc7fdc33f55 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
d1464447c65231abeaa52e94d5a48cf62f054881 
  core/src/test/scala/unit/kafka/network/SocketServerTest.scala 
62fb02cf02d3876b9804d756c4bf8514554cc836 

Diff: https://reviews.apache.org/r/22874/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Updated] (KAFKA-1498) new producer performance and bug improvements

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1498:
-

Attachment: KAFKA-1498_2014-07-01_11:12:41.patch

> new producer performance and bug improvements
> -
>
> Key: KAFKA-1498
> URL: https://issues.apache.org/jira/browse/KAFKA-1498
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jun Rao
>Assignee: Guozhang Wang
> Attachments: KAFKA-1498.patch, KAFKA-1498_2014-06-25_16:44:51.patch, 
> KAFKA-1498_2014-06-30_10:47:17.patch, KAFKA-1498_2014-06-30_15:47:56.patch, 
> KAFKA-1498_2014-07-01_11:12:41.patch, kafka-1498.patch
>
>
> We have seen the following issues with the new producer.
> 1. The producer request can be significantly larger than the configured batch 
> size.
> 2. The bottleneck in mirrormaker when there are keyed messages and 
> compression is turned on.
> 3. The selector is woken up on every message in the new producer.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1498) new producer performance and bug improvements

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049140#comment-14049140
 ] 

Guozhang Wang commented on KAFKA-1498:
--

Updated reviewboard https://reviews.apache.org/r/22874/
 against branch origin/trunk

> new producer performance and bug improvements
> -
>
> Key: KAFKA-1498
> URL: https://issues.apache.org/jira/browse/KAFKA-1498
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jun Rao
>Assignee: Guozhang Wang
> Attachments: KAFKA-1498.patch, KAFKA-1498_2014-06-25_16:44:51.patch, 
> KAFKA-1498_2014-06-30_10:47:17.patch, KAFKA-1498_2014-06-30_15:47:56.patch, 
> KAFKA-1498_2014-07-01_11:12:41.patch, kafka-1498.patch
>
>
> We have seen the following issues with the new producer.
> 1. The producer request can be significantly larger than the configured batch 
> size.
> 2. The bottleneck in mirrormaker when there are keyed messages and 
> compression is turned on.
> 3. The selector is woken up on every message in the new producer.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1512) Limit the maximum number of connections per ip address

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049176#comment-14049176
 ] 

Guozhang Wang commented on KAFKA-1512:
--

Jay, seems kafka-review-tool fails to create the RB, could you try again?

> Limit the maximum number of connections per ip address
> --
>
> Key: KAFKA-1512
> URL: https://issues.apache.org/jira/browse/KAFKA-1512
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Jay Kreps
>Assignee: Jay Kreps
> Attachments: KAFKA-1512.patch
>
>
> To protect against client connection leaks add a new configuration
>   max.connections.per.ip
> that causes the SocketServer to enforce a limit on the maximum number of 
> connections from each InetAddress instance. For backwards compatibility this 
> will default to 2 billion.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (KAFKA-1174) Empty jar in Maven Central for Scala 2.8.0

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein resolved KAFKA-1174.
--

Resolution: Won't Fix

this was resolved in the 0.8.1 build release , was a build issue

> Empty jar in Maven Central for Scala 2.8.0
> --
>
> Key: KAFKA-1174
> URL: https://issues.apache.org/jira/browse/KAFKA-1174
> Project: Kafka
>  Issue Type: Bug
>  Components: packaging
>Affects Versions: 0.8.0
>Reporter: David Arthur
>Priority: Critical
>
> As reported by wildag on IRC
> In Maven Central, the jar for kafka core only contains the license and notice 
> files, no classes. I checked the other Scala versions and they seem fine.
> See: 
> http://search.maven.org/#artifactdetails%7Corg.apache.kafka%7Ckafka_2.8.0%7C0.8.0%7Cjar
> Unless we can invoke the power of the Sonatype gods, I think we must bump the 
> version number to fix this (or wait until 0.8.1 drops).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (KAFKA-1454) Update for a Scala 2.11 build

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein resolved KAFKA-1454.
--

Resolution: Won't Fix

Scala 2.11 works with 2.10 binaries http://www.scala-lang.org/news/2.11.0


> Update for a Scala 2.11 build
> -
>
> Key: KAFKA-1454
> URL: https://issues.apache.org/jira/browse/KAFKA-1454
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Joe Stein
>Assignee: Joe Stein
>Priority: Critical
>  Labels: newbie
> Fix For: 0.8.2
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1419) cross build for scala 2.11

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049197#comment-14049197
 ] 

Joe Stein commented on KAFKA-1419:
--

Is there a reason we need to-do this? 

Scala 2.11 works with 2.10 binaries http://www.scala-lang.org/news/2.11.0


> cross build for scala 2.11
> --
>
> Key: KAFKA-1419
> URL: https://issues.apache.org/jira/browse/KAFKA-1419
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.8.1
>Reporter: Scott Clasen
>
> Please publish builds for scala 2.11, hopefully just needs a small tweak to 
> the gradle conf?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1419) cross build for scala 2.11

[Kostya Golikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049204#comment-14049204
 ] 

Kostya Golikov commented on KAFKA-1419:
---

[~joestein], Scala 2.11 is only *source* compatible with 2.10, but not binary 
compatible. To quote release notes you've linked: 

> Code that compiled on 2.10.x without deprecation warnings should compile on 
> 2.11.x ... This release is not binary compatible with the 2.10.x series, to 
> allow us to keep improving the Scala standard library.

That means, that although you likely don't have to support two different 
versions of kafka source code, kafka artifact built against 2.10 would not work 
if user will have 2.11 on classpath (and vice versa).

> cross build for scala 2.11
> --
>
> Key: KAFKA-1419
> URL: https://issues.apache.org/jira/browse/KAFKA-1419
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.8.1
>Reporter: Scott Clasen
>
> Please publish builds for scala 2.11, hopefully just needs a small tweak to 
> the gradle conf?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1419) cross build for scala 2.11

[Scott Clasen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049206#comment-14049206
 ] 

Scott Clasen commented on KAFKA-1419:
-

Makes it a sbt gymnastic task to cross build downstream libraries that are 
built on top of kafka.

rather than simply  ` libraryDependencies = "org.apache.kafka" %% "kafka"  
%"0.8.1" `

it takes more sbt knowledge to do something like what I had to do for spray 
(example below) when they didnt cross build.  The above is something most folks 
can type out. The below took googling and cursing and time to figure out.

```
  def spray:Seq[Setting[Seq[ModuleID]]] = Seq(libraryDependencies <+= 
scalaVersion(sprayDependency(_)))

  def sprayDependency(scalaVersion: String) = scalaVersion match {
case "2.10.4" => "io.spray" % "spray-client" % "1.3.1" % "compile"
case "2.11.1" => "io.spray" % "spray-client_2.11" % "1.3.1-20140423" % 
"compile"
  }

```

> cross build for scala 2.11
> --
>
> Key: KAFKA-1419
> URL: https://issues.apache.org/jira/browse/KAFKA-1419
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.8.1
>Reporter: Scott Clasen
>
> Please publish builds for scala 2.11, hopefully just needs a small tweak to 
> the gradle conf?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1419) cross build for scala 2.11

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049210#comment-14049210
 ] 

Joe Stein commented on KAFKA-1419:
--

That makes sense Scott (thanks) if you or someone else wants to upload a patch 
I can review and commit it we can get it into 0.8.2 otherwise I can do it 
before the release and someone else can review and commit, either way that 
makes sense to me (ease of use for the community is a +1 for sure)

> cross build for scala 2.11
> --
>
> Key: KAFKA-1419
> URL: https://issues.apache.org/jira/browse/KAFKA-1419
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.8.1
>Reporter: Scott Clasen
> Fix For: 0.8.2
>
>
> Please publish builds for scala 2.11, hopefully just needs a small tweak to 
> the gradle conf?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1419) cross build for scala 2.11

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1419:
-

 Priority: Blocker  (was: Major)
Fix Version/s: 0.8.2

> cross build for scala 2.11
> --
>
> Key: KAFKA-1419
> URL: https://issues.apache.org/jira/browse/KAFKA-1419
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.8.1
>Reporter: Scott Clasen
>Priority: Blocker
> Fix For: 0.8.2
>
>
> Please publish builds for scala 2.11, hopefully just needs a small tweak to 
> the gradle conf?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1419) cross build for scala 2.11

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049213#comment-14049213
 ] 

Joe Stein commented on KAFKA-1419:
--

We might also be able to build and release for 0.8.1.1 like was done with the 
test jar [~jghoman] what do you think? I think that should be ok?

> cross build for scala 2.11
> --
>
> Key: KAFKA-1419
> URL: https://issues.apache.org/jira/browse/KAFKA-1419
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.8.1
>Reporter: Scott Clasen
>Priority: Blocker
> Fix For: 0.8.2
>
>
> Please publish builds for scala 2.11, hopefully just needs a small tweak to 
> the gradle conf?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1454) Update for a Scala 2.11 build

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049214#comment-14049214
 ] 

Joe Stein commented on KAFKA-1454:
--

discussion moved to KAFKA-1419 will take this up there

> Update for a Scala 2.11 build
> -
>
> Key: KAFKA-1454
> URL: https://issues.apache.org/jira/browse/KAFKA-1454
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Joe Stein
>Assignee: Joe Stein
>Priority: Critical
>  Labels: newbie
> Fix For: 0.8.2
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 23208: Patch for KAFKA-1512

[Jay Kreps]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23208/
---

Review request for kafka.


Bugs: KAFKA-1512
https://issues.apache.org/jira/browse/KAFKA-1512


Repository: kafka


Description
---

KAFKA-1512 Add per-ip connection limits.


Diffs
-

  core/src/main/scala/kafka/network/SocketServer.scala 
4976d9c3a66bc965f5870a0736e21c7b32650bab 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
ef75b67b67676ae5b8931902cbc8c0c2cc72c0d3 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c22e51e0412843ec993721ad3230824c0aadd2ba 
  core/src/test/scala/unit/kafka/network/SocketServerTest.scala 
62fb02cf02d3876b9804d756c4bf8514554cc836 

Diff: https://reviews.apache.org/r/23208/diff/


Testing
---


Thanks,

Jay Kreps

[jira] [Updated] (KAFKA-1512) Limit the maximum number of connections per ip address

[Jay Kreps (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Kreps updated KAFKA-1512:
-

Attachment: KAFKA-1512.patch

> Limit the maximum number of connections per ip address
> --
>
> Key: KAFKA-1512
> URL: https://issues.apache.org/jira/browse/KAFKA-1512
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Jay Kreps
>Assignee: Jay Kreps
> Attachments: KAFKA-1512.patch, KAFKA-1512.patch
>
>
> To protect against client connection leaks add a new configuration
>   max.connections.per.ip
> that causes the SocketServer to enforce a limit on the maximum number of 
> connections from each InetAddress instance. For backwards compatibility this 
> will default to 2 billion.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1512) Limit the maximum number of connections per ip address

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049235#comment-14049235
 ] 

Jay Kreps commented on KAFKA-1512:
--

Created reviewboard https://reviews.apache.org/r/23208/
 against branch trunk

> Limit the maximum number of connections per ip address
> --
>
> Key: KAFKA-1512
> URL: https://issues.apache.org/jira/browse/KAFKA-1512
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Jay Kreps
>Assignee: Jay Kreps
> Attachments: KAFKA-1512.patch, KAFKA-1512.patch
>
>
> To protect against client connection leaks add a new configuration
>   max.connections.per.ip
> that causes the SocketServer to enforce a limit on the maximum number of 
> connections from each InetAddress instance. For backwards compatibility this 
> will default to 2 billion.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 23215: Follow-up KAFKA-1498

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23215/
---

Review request for kafka.


Bugs: KAFKA-1498
https://issues.apache.org/jira/browse/KAFKA-1498


Repository: kafka


Description
---

Add record-size-avg


Diffs
-

  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
52d209b5022ab640f74eb212f115a08a5949e278 

Diff: https://reviews.apache.org/r/23215/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Commented] (KAFKA-1498) new producer performance and bug improvements

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049357#comment-14049357
 ] 

Guozhang Wang commented on KAFKA-1498:
--

Created reviewboard https://reviews.apache.org/r/23215/
 against branch origin/trunk

> new producer performance and bug improvements
> -
>
> Key: KAFKA-1498
> URL: https://issues.apache.org/jira/browse/KAFKA-1498
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jun Rao
>Assignee: Guozhang Wang
> Attachments: KAFKA-1498.patch, KAFKA-1498.patch, 
> KAFKA-1498_2014-06-25_16:44:51.patch, KAFKA-1498_2014-06-30_10:47:17.patch, 
> KAFKA-1498_2014-06-30_15:47:56.patch, KAFKA-1498_2014-07-01_11:12:41.patch, 
> kafka-1498.patch
>
>
> We have seen the following issues with the new producer.
> 1. The producer request can be significantly larger than the configured batch 
> size.
> 2. The bottleneck in mirrormaker when there are keyed messages and 
> compression is turned on.
> 3. The selector is woken up on every message in the new producer.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1498) new producer performance and bug improvements

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1498:
-

Attachment: KAFKA-1498.patch

> new producer performance and bug improvements
> -
>
> Key: KAFKA-1498
> URL: https://issues.apache.org/jira/browse/KAFKA-1498
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jun Rao
>Assignee: Guozhang Wang
> Attachments: KAFKA-1498.patch, KAFKA-1498.patch, 
> KAFKA-1498_2014-06-25_16:44:51.patch, KAFKA-1498_2014-06-30_10:47:17.patch, 
> KAFKA-1498_2014-06-30_15:47:56.patch, KAFKA-1498_2014-07-01_11:12:41.patch, 
> kafka-1498.patch
>
>
> We have seen the following issues with the new producer.
> 1. The producer request can be significantly larger than the configured batch 
> size.
> 2. The bottleneck in mirrormaker when there are keyed messages and 
> compression is turned on.
> 3. The selector is woken up on every message in the new producer.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 23215: Follow-up KAFKA-1498

[Jay Kreps]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23215/#review47150
---

Ship it!


Ship It!

- Jay Kreps


On July 1, 2014, 9:40 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23215/
> ---
> 
> (Updated July 1, 2014, 9:40 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1498
> https://issues.apache.org/jira/browse/KAFKA-1498
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Add record-size-avg
> 
> 
> Diffs
> -
> 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> 52d209b5022ab640f74eb212f115a08a5949e278 
> 
> Diff: https://reviews.apache.org/r/23215/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

Re: Review Request 23208: Patch for KAFKA-1512

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23208/#review47151
---

Ship it!


Looks good to me, although the connection quotas will not yet work if we are 
still using sth. like a hardware load balancer or a VIP.

- Guozhang Wang


On July 1, 2014, 7:42 p.m., Jay Kreps wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23208/
> ---
> 
> (Updated July 1, 2014, 7:42 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1512
> https://issues.apache.org/jira/browse/KAFKA-1512
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1512 Add per-ip connection limits.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/network/SocketServer.scala 
> 4976d9c3a66bc965f5870a0736e21c7b32650bab 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> ef75b67b67676ae5b8931902cbc8c0c2cc72c0d3 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> c22e51e0412843ec993721ad3230824c0aadd2ba 
>   core/src/test/scala/unit/kafka/network/SocketServerTest.scala 
> 62fb02cf02d3876b9804d756c4bf8514554cc836 
> 
> Diff: https://reviews.apache.org/r/23208/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jay Kreps
> 
>

Consistent logIdent

[Верещагин Евгений]

Hi!

I found many styles of logIdent ( 
https://github.com/apache/kafka/blob/c66e408b244de52f1c5c5bbd7627aa1f028f9a87/core/src/main/scala/kafka/utils/Logging.scala#L26
 ):
"[COMP-N]: "
"[COMP-N], "
"COMP-N "

I can make it consistent if you approve.
For me, best solution, is "[COMP-N]: ".

Can I send GitHub pull request or task in Jira with patches required?

-- 
Best regards,
Evgeny Vereshchagin

Re: Consistent logIdent

[Evgeny Vereshchagin]

Sorry, I was wrong.
It's different parts of one log message.

--
Best regards,
Evgeny Vereshchagin

Jenkins build is back to normal : Kafka-trunk #212

[Apache Jenkins Server]

See 

[jira] [Commented] (KAFKA-1419) cross build for scala 2.11

[Jakob Homan (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049517#comment-14049517
 ] 

Jakob Homan commented on KAFKA-1419:


As long as the sources compiled are what was voted on as the release, it should 
be fine.  The jars are just conveniences for the users.

> cross build for scala 2.11
> --
>
> Key: KAFKA-1419
> URL: https://issues.apache.org/jira/browse/KAFKA-1419
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.8.1
>Reporter: Scott Clasen
>Priority: Blocker
> Fix For: 0.8.2
>
>
> Please publish builds for scala 2.11, hopefully just needs a small tweak to 
> the gradle conf?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1308) Publish jar of test utilities to Maven

[Jakob Homan (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049520#comment-14049520
 ] 

Jakob Homan commented on KAFKA-1308:


Yes, had a merge conflict when I went to do so; will resolve tomorrow and 
commit.

> Publish jar of test utilities to Maven
> --
>
> Key: KAFKA-1308
> URL: https://issues.apache.org/jira/browse/KAFKA-1308
> Project: Kafka
>  Issue Type: Wish
>Affects Versions: 0.8.1
>Reporter: Martin Kleppmann
>Assignee: Jakob Homan
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: KAFKA-1308-2.patch, KAFKA-1308-8.1.1.patch, 
> KAFKA-1308.patch, KAFKA-1308.patch, kafka-1308_v2_0.8.1.patch
>
>
> For projects that use Kafka, and want to write tests that exercise Kafka (in 
> our case, Samza), it's useful to have access to Kafka's test utility classes 
> such as kafka.zk.EmbeddedZookeeper and kafka.utils.TestUtils. We can use 
> {{./gradlew testJar}} to build jar files that contain those classes, but as 
> far as I know, these are currently not made available in a binary release.
> At the moment, we have to check those kafka*-test.jar files into the Samza 
> repository. To avoid that, would it be possible to publish those jars of 
> tests to Maven, so that they fit into the normal dependency management?
> Or perhaps, if publishing the tests themselves is not appropriate, we could 
> move the test utilities into a separate module that is published, and make 
> the tests depend on that module?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (KAFKA-1513) Cleanup KafkaServerStartable code

[Evgeny Vereshchagin (JIRA)]

Evgeny Vereshchagin created KAFKA-1513:
--

 Summary: Cleanup KafkaServerStartable code
 Key: KAFKA-1513
 URL: https://issues.apache.org/jira/browse/KAFKA-1513
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Evgeny Vereshchagin
Priority: Minor


[init 
method|https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/server/KafkaServerStartable.scala#L26]
 too hard to understand.
Var initialization would be simpler for readers.
Patch will be soon. It's my first time:)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1513) Cleanup KafkaServerStartable code

[Evgeny Vereshchagin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Evgeny Vereshchagin updated KAFKA-1513:
---

Attachment: cleanup-kafkaserverstartable-code-v1.patch

Submit patch.

> Cleanup KafkaServerStartable code
> -
>
> Key: KAFKA-1513
> URL: https://issues.apache.org/jira/browse/KAFKA-1513
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Reporter: Evgeny Vereshchagin
>Priority: Minor
> Attachments: cleanup-kafkaserverstartable-code-v1.patch
>
>
> [init 
> method|https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/server/KafkaServerStartable.scala#L26]
>  too hard to understand.
> Var initialization would be simpler for readers.
> Patch will be soon. It's my first time:)



--
This message was sent by Atlassian JIRA
(v6.2#6252)