[jira] [Resolved] (KAFKA-5548) SchemaBuilder does not validate input.

2017-07-01 Thread Ewen Cheslack-Postava (JIRA)
 [ 
https://issues.apache.org/jira/browse/KAFKA-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-5548.
--
   Resolution: Fixed
Fix Version/s: 0.11.1.0

Issue resolved by pull request 3474
[https://github.com/apache/kafka/pull/3474]

> SchemaBuilder does not validate input.
> --
>
> Key: KAFKA-5548
> URL: https://issues.apache.org/jira/browse/KAFKA-5548
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Jeremy Custenborder
>Assignee: Jeremy Custenborder
>Priority: Minor
> Fix For: 0.11.1.0
>
>
> SchemaBuilder.map(), SchemaBuilder.array(), and SchemaBuilder.field() do not 
> validate input. This can cause weird NullPointerException exceptions later. 
> For example I mistakenly called field("somefield", null), then later 
> performed an operation against field.schema() which yielded a null. It would 
> be preferable to throw an exception stating the issue. We could throw the a 
> NPE but state what is null. Schema is null in this case for example.
> {code:java}
>   @Test(expected = NullPointerException.class)
>   public void fieldNameNull() {
> Schema schema = SchemaBuilder.struct()
> .field(null, Schema.STRING_SCHEMA)
> .build();
>   }
>   @Test(expected = NullPointerException.class)
>   public void fieldSchemaNull() {
> Schema schema = SchemaBuilder.struct()
> .field("fieldName", null)
> .build();
>   }
>   @Test(expected = NullPointerException.class)
>   public void arraySchemaNull() {
> Schema schema = SchemaBuilder.array(Schema.STRING_SCHEMA)
> .build();
>   }
>   @Test(expected = NullPointerException.class)
>   public void mapKeySchemaNull() {
> Schema schema = SchemaBuilder.map(null, Schema.STRING_SCHEMA)
> .build();
>   }
>   @Test(expected = NullPointerException.class)
>   public void mapValueSchemaNull() {
> Schema schema = SchemaBuilder.map(Schema.STRING_SCHEMA, null)
> .build();
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (KAFKA-5548) SchemaBuilder does not validate input.

2017-07-01 Thread ASF GitHub Bot (JIRA)
[ 
https://issues.apache.org/jira/browse/KAFKA-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071129#comment-16071129
 ] 

ASF GitHub Bot commented on KAFKA-5548:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3474


> SchemaBuilder does not validate input.
> --
>
> Key: KAFKA-5548
> URL: https://issues.apache.org/jira/browse/KAFKA-5548
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Jeremy Custenborder
>Assignee: Jeremy Custenborder
>Priority: Minor
> Fix For: 0.11.1.0
>
>
> SchemaBuilder.map(), SchemaBuilder.array(), and SchemaBuilder.field() do not 
> validate input. This can cause weird NullPointerException exceptions later. 
> For example I mistakenly called field("somefield", null), then later 
> performed an operation against field.schema() which yielded a null. It would 
> be preferable to throw an exception stating the issue. We could throw the a 
> NPE but state what is null. Schema is null in this case for example.
> {code:java}
>   @Test(expected = NullPointerException.class)
>   public void fieldNameNull() {
> Schema schema = SchemaBuilder.struct()
> .field(null, Schema.STRING_SCHEMA)
> .build();
>   }
>   @Test(expected = NullPointerException.class)
>   public void fieldSchemaNull() {
> Schema schema = SchemaBuilder.struct()
> .field("fieldName", null)
> .build();
>   }
>   @Test(expected = NullPointerException.class)
>   public void arraySchemaNull() {
> Schema schema = SchemaBuilder.array(Schema.STRING_SCHEMA)
> .build();
>   }
>   @Test(expected = NullPointerException.class)
>   public void mapKeySchemaNull() {
> Schema schema = SchemaBuilder.map(null, Schema.STRING_SCHEMA)
> .build();
>   }
>   @Test(expected = NullPointerException.class)
>   public void mapValueSchemaNull() {
> Schema schema = SchemaBuilder.map(Schema.STRING_SCHEMA, null)
> .build();
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (KAFKA-5546) Lost data when the leader is disconnected.

2017-07-01 Thread Michal Borowiecki (JIRA)
[ 
https://issues.apache.org/jira/browse/KAFKA-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071187#comment-16071187
 ] 

Michal Borowiecki commented on KAFKA-5546:
--

Does your producer check if sending was successful? That is, if the broker 
acknowledged the message? If the data is never acknowledged by the broker, it 
can be lost, it's not a defect.
As far as I can see you're just piping data into the console producer and the 
logs on producers side indicate the messages simply weren't sent. Please 
correct me if I'm reading it wrong.

Additionally, as far as I can tell, you are using acks=1 and unclean leader 
election enabled (default in 0.10.2.1 but changed to disabled from 0.11.0.0).
This setup allows loss of even acknowledged messages on leader failure.

If you are trying to set up a resilient kafka cluster, please disable unclean 
leader election and set ack >1 ("all" is the reasonable default) and of course, 
check that the messages were sent.

> Lost data when the leader is disconnected.
> --
>
> Key: KAFKA-5546
> URL: https://issues.apache.org/jira/browse/KAFKA-5546
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.10.2.1
>Reporter: Björn Eriksson
> Attachments: kafka-failure-log.txt
>
>
> We've noticed that if the leaders networking is deconfigured (with {{ifconfig 
> eth0 down}}) the producer won't notice this and doesn't immediately connect 
> to the newly elected leader.
> {{docker-compose.yml}} and test runner are at 
> https://github.com/owbear/kafka-network-failure-tests with sample test output 
> at 
> https://github.com/owbear/kafka-network-failure-tests/blob/master/README.md#sample-results
> I was expecting a transparent failover to the new leader.
> The attached log shows that while the producer produced values between 
> {{12:37:33}} and {{12:37:54}}, theres a gap between {{12:37:41}} and 
> {{12:37:50}} where no values was stored in the log after the network was 
> taken down at {{12:37:42}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Comment Edited] (KAFKA-5546) Lost data when the leader is disconnected.

2017-07-01 Thread Michal Borowiecki (JIRA)
[ 
https://issues.apache.org/jira/browse/KAFKA-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071187#comment-16071187
 ] 

Michal Borowiecki edited comment on KAFKA-5546 at 7/1/17 11:37 AM:
---

Does your producer check if sending was successful? That is, if the broker 
acknowledged the message? If the data is never acknowledged by the broker, it 
can be lost, it's not a defect.
As far as I can see you're just piping data into the console producer and the 
logs on producers side indicate the messages simply weren't sent. Please 
correct me if I'm reading it wrong.

Additionally, as far as I can tell, you are using acks=1 and unclean leader 
election enabled (default in 0.10.2.1 but changed to disabled from 0.11.0.0).
This setup allows loss of even acknowledged messages on leader failure.

If you are trying to set up a resilient kafka cluster, please disable unclean 
leader election and set ack >1 ("all" is the reasonable default) and of course, 
check that the sending was successful, i.e. the broker acknowledged them.


was (Author: mihbor):
Does your producer check if sending was successful? That is, if the broker 
acknowledged the message? If the data is never acknowledged by the broker, it 
can be lost, it's not a defect.
As far as I can see you're just piping data into the console producer and the 
logs on producers side indicate the messages simply weren't sent. Please 
correct me if I'm reading it wrong.

Additionally, as far as I can tell, you are using acks=1 and unclean leader 
election enabled (default in 0.10.2.1 but changed to disabled from 0.11.0.0).
This setup allows loss of even acknowledged messages on leader failure.

If you are trying to set up a resilient kafka cluster, please disable unclean 
leader election and set ack >1 ("all" is the reasonable default) and of course, 
check that the messages were sent.

> Lost data when the leader is disconnected.
> --
>
> Key: KAFKA-5546
> URL: https://issues.apache.org/jira/browse/KAFKA-5546
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.10.2.1
>Reporter: Björn Eriksson
> Attachments: kafka-failure-log.txt
>
>
> We've noticed that if the leaders networking is deconfigured (with {{ifconfig 
> eth0 down}}) the producer won't notice this and doesn't immediately connect 
> to the newly elected leader.
> {{docker-compose.yml}} and test runner are at 
> https://github.com/owbear/kafka-network-failure-tests with sample test output 
> at 
> https://github.com/owbear/kafka-network-failure-tests/blob/master/README.md#sample-results
> I was expecting a transparent failover to the new leader.
> The attached log shows that while the producer produced values between 
> {{12:37:33}} and {{12:37:54}}, theres a gap between {{12:37:41}} and 
> {{12:37:50}} where no values was stored in the log after the network was 
> taken down at {{12:37:42}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Comment Edited] (KAFKA-5546) Lost data when the leader is disconnected.

2017-07-01 Thread Michal Borowiecki (JIRA)
[ 
https://issues.apache.org/jira/browse/KAFKA-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071187#comment-16071187
 ] 

Michal Borowiecki edited comment on KAFKA-5546 at 7/1/17 11:38 AM:
---

Does your producer check if sending was successful? That is, if the broker 
acknowledged the message? If the data is never acknowledged by the broker, it 
can be lost, it's not a defect.
As far as I can see you're just piping data into the console producer and the 
logs on producers side indicate the messages simply weren't sent. Please 
correct me if I'm reading it wrong.

Additionally, as far as I can tell, you are using acks=1 and unclean leader 
election enabled (default in 0.10.2.1 but changed to disabled from 0.11.0.0).
This setup allows loss of even acknowledged messages on leader failure.

If you are trying to set up a resilient kafka cluster, please disable unclean 
leader election and set ack >1 ("all" is the reasonable default) and of course, 
check that the sending of messages was successful, i.e. the broker acknowledged 
them.


was (Author: mihbor):
Does your producer check if sending was successful? That is, if the broker 
acknowledged the message? If the data is never acknowledged by the broker, it 
can be lost, it's not a defect.
As far as I can see you're just piping data into the console producer and the 
logs on producers side indicate the messages simply weren't sent. Please 
correct me if I'm reading it wrong.

Additionally, as far as I can tell, you are using acks=1 and unclean leader 
election enabled (default in 0.10.2.1 but changed to disabled from 0.11.0.0).
This setup allows loss of even acknowledged messages on leader failure.

If you are trying to set up a resilient kafka cluster, please disable unclean 
leader election and set ack >1 ("all" is the reasonable default) and of course, 
check that the sending was successful, i.e. the broker acknowledged them.

> Lost data when the leader is disconnected.
> --
>
> Key: KAFKA-5546
> URL: https://issues.apache.org/jira/browse/KAFKA-5546
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.10.2.1
>Reporter: Björn Eriksson
> Attachments: kafka-failure-log.txt
>
>
> We've noticed that if the leaders networking is deconfigured (with {{ifconfig 
> eth0 down}}) the producer won't notice this and doesn't immediately connect 
> to the newly elected leader.
> {{docker-compose.yml}} and test runner are at 
> https://github.com/owbear/kafka-network-failure-tests with sample test output 
> at 
> https://github.com/owbear/kafka-network-failure-tests/blob/master/README.md#sample-results
> I was expecting a transparent failover to the new leader.
> The attached log shows that while the producer produced values between 
> {{12:37:33}} and {{12:37:54}}, theres a gap between {{12:37:41}} and 
> {{12:37:50}} where no values was stored in the log after the network was 
> taken down at {{12:37:42}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (KAFKA-5546) Lost data when the leader is disconnected.

2017-07-01 Thread JIRA
[ 
https://issues.apache.org/jira/browse/KAFKA-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071252#comment-16071252
 ] 

Björn Eriksson commented on KAFKA-5546:
---

Hi [~mihbor], I've updated the test to use {{acks=all}} and no unclean leader 
election but the results are the same: the console consumer doesn't immediately 
switch to the new leader and data is lost.

You're right, we're trying to set up a resilient Kafka cluster but this seems 
difficult to achieve.


> Lost data when the leader is disconnected.
> --
>
> Key: KAFKA-5546
> URL: https://issues.apache.org/jira/browse/KAFKA-5546
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.10.2.1
>Reporter: Björn Eriksson
> Attachments: kafka-failure-log.txt
>
>
> We've noticed that if the leaders networking is deconfigured (with {{ifconfig 
> eth0 down}}) the producer won't notice this and doesn't immediately connect 
> to the newly elected leader.
> {{docker-compose.yml}} and test runner are at 
> https://github.com/owbear/kafka-network-failure-tests with sample test output 
> at 
> https://github.com/owbear/kafka-network-failure-tests/blob/master/README.md#sample-results
> I was expecting a transparent failover to the new leader.
> The attached log shows that while the producer produced values between 
> {{12:37:33}} and {{12:37:54}}, theres a gap between {{12:37:41}} and 
> {{12:37:50}} where no values was stored in the log after the network was 
> taken down at {{12:37:42}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Assigned] (KAFKA-3465) kafka.tools.ConsumerOffsetChecker won't align with kafka New Consumer mode

2017-07-01 Thread Vahid Hashemian (JIRA)
 [ 
https://issues.apache.org/jira/browse/KAFKA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian reassigned KAFKA-3465:
--

Assignee: Vahid Hashemian

> kafka.tools.ConsumerOffsetChecker won't align with kafka New Consumer mode
> --
>
> Key: KAFKA-3465
> URL: https://issues.apache.org/jira/browse/KAFKA-3465
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: BrianLing
>Assignee: Vahid Hashemian
>
> 1. When we enable mirrorMake to migrate Kafka event from one to other with 
> "new.consumer" mode:
> java -Xmx2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 
> -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC 
> -Djava.awt.headless=true -Dcom.sun.management.jmxremote 
> -Dcom.sun.management.jmxremote.authenticate=false 
> -Dcom.sun.management.jmxremote.ssl=false 
> -Dkafka.logs.dir=/kafka/kafka-app-logs 
> -Dlog4j.configuration=file:/kafka/kafka_2.10-0.9.0.0/bin/../config/tools-log4j.properties
>  -cp :/kafka/kafka_2.10-0.9.0.0/bin/../libs/* 
> -Dkafka.logs.filename=lvs-slca-mm.log kafka.tools.MirrorMaker lvs-slca-mm.log 
> --consumer.config ../config/consumer.properties --new.consumer --num.streams 
> 4 --producer.config ../config/producer-slca.properties --whitelist risk.*
> 2. When we use ConsumerOffzsetChecker tool, notice the lag won't changed and 
> the owner is none.
> bin/kafka-run-class.sh  kafka.tools.ConsumerOffsetChecker --broker-info 
> --group lvs.slca.mirrormaker --zookeeper lvsdmetlvm01.lvs.paypal.com:2181 
> --topic 
> Group   Topic  Pid Offset  logSize
>  Lag Owner
> lvs.slca.mirrormaker   0   418578332   418678347   100015 
>  none
> lvs.slca.mirrormaker  1   418598026   418698338   100312  
> none
> [Root Cause]
> I think it's due to 0.9.0 new feature to switch zookeeper dependency to kafka 
> internal to store offset & consumer owner information. 
>   Does it mean we can not use the below command to check new consumer’s 
> lag since current lag formula: lag= logSize – offset 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L80
>   
> https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L174-L182
>  => offSet Fetch from zookeeper instead of from Kafka



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (KAFKA-3465) kafka.tools.ConsumerOffsetChecker won't align with kafka New Consumer mode

2017-07-01 Thread Vahid Hashemian (JIRA)
 [ 
https://issues.apache.org/jira/browse/KAFKA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian updated KAFKA-3465:
---
Affects Version/s: 0.11.0.0
 Priority: Minor  (was: Major)
Fix Version/s: 0.11.1.0

> kafka.tools.ConsumerOffsetChecker won't align with kafka New Consumer mode
> --
>
> Key: KAFKA-3465
> URL: https://issues.apache.org/jira/browse/KAFKA-3465
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.9.0.0, 0.11.0.0
>Reporter: BrianLing
>Assignee: Vahid Hashemian
>Priority: Minor
> Fix For: 0.11.1.0
>
>
> 1. When we enable mirrorMake to migrate Kafka event from one to other with 
> "new.consumer" mode:
> java -Xmx2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 
> -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC 
> -Djava.awt.headless=true -Dcom.sun.management.jmxremote 
> -Dcom.sun.management.jmxremote.authenticate=false 
> -Dcom.sun.management.jmxremote.ssl=false 
> -Dkafka.logs.dir=/kafka/kafka-app-logs 
> -Dlog4j.configuration=file:/kafka/kafka_2.10-0.9.0.0/bin/../config/tools-log4j.properties
>  -cp :/kafka/kafka_2.10-0.9.0.0/bin/../libs/* 
> -Dkafka.logs.filename=lvs-slca-mm.log kafka.tools.MirrorMaker lvs-slca-mm.log 
> --consumer.config ../config/consumer.properties --new.consumer --num.streams 
> 4 --producer.config ../config/producer-slca.properties --whitelist risk.*
> 2. When we use ConsumerOffzsetChecker tool, notice the lag won't changed and 
> the owner is none.
> bin/kafka-run-class.sh  kafka.tools.ConsumerOffsetChecker --broker-info 
> --group lvs.slca.mirrormaker --zookeeper lvsdmetlvm01.lvs.paypal.com:2181 
> --topic 
> Group   Topic  Pid Offset  logSize
>  Lag Owner
> lvs.slca.mirrormaker   0   418578332   418678347   100015 
>  none
> lvs.slca.mirrormaker  1   418598026   418698338   100312  
> none
> [Root Cause]
> I think it's due to 0.9.0 new feature to switch zookeeper dependency to kafka 
> internal to store offset & consumer owner information. 
>   Does it mean we can not use the below command to check new consumer’s 
> lag since current lag formula: lag= logSize – offset 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L80
>   
> https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L174-L182
>  => offSet Fetch from zookeeper instead of from Kafka



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (KAFKA-5551) StreamThread should not expose methods for testing

2017-07-01 Thread Matthias J. Sax (JIRA)
Matthias J. Sax created KAFKA-5551:
--

 Summary: StreamThread should not expose methods for testing
 Key: KAFKA-5551
 URL: https://issues.apache.org/jira/browse/KAFKA-5551
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Matthias J. Sax
Assignee: Matthias J. Sax


{{StreamsThread}} currently exposes {{createStreamTask()}} and 
{{createStandbyTask()}} as {{protected}} in order to inject "test tasks" in 
unit tests. We should rework this and make both methods {{private}}. Maybe we 
can introduce a {{TaskSupplier}} similar to {{KafkaClientSupplier}} (however, 
{{TaskSupplier}} should not be public API and be in package {{internal}}).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)