[jira] [Resolved] (KAFKA-5548) SchemaBuilder does not validate input.
[
https://issues.apache.org/jira/browse/KAFKA-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewen Cheslack-Postava resolved KAFKA-5548.
--
Resolution: Fixed
Fix Version/s: 0.11.1.0
Issue resolved by pull request 3474
[https://github.com/apache/kafka/pull/3474]
> SchemaBuilder does not validate input.
> --
>
> Key: KAFKA-5548
> URL: https://issues.apache.org/jira/browse/KAFKA-5548
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Jeremy Custenborder
>Assignee: Jeremy Custenborder
>Priority: Minor
> Fix For: 0.11.1.0
>
>
> SchemaBuilder.map(), SchemaBuilder.array(), and SchemaBuilder.field() do not
> validate input. This can cause weird NullPointerException exceptions later.
> For example I mistakenly called field("somefield", null), then later
> performed an operation against field.schema() which yielded a null. It would
> be preferable to throw an exception stating the issue. We could throw the a
> NPE but state what is null. Schema is null in this case for example.
> {code:java}
> @Test(expected = NullPointerException.class)
> public void fieldNameNull() {
> Schema schema = SchemaBuilder.struct()
> .field(null, Schema.STRING_SCHEMA)
> .build();
> }
> @Test(expected = NullPointerException.class)
> public void fieldSchemaNull() {
> Schema schema = SchemaBuilder.struct()
> .field("fieldName", null)
> .build();
> }
> @Test(expected = NullPointerException.class)
> public void arraySchemaNull() {
> Schema schema = SchemaBuilder.array(Schema.STRING_SCHEMA)
> .build();
> }
> @Test(expected = NullPointerException.class)
> public void mapKeySchemaNull() {
> Schema schema = SchemaBuilder.map(null, Schema.STRING_SCHEMA)
> .build();
> }
> @Test(expected = NullPointerException.class)
> public void mapValueSchemaNull() {
> Schema schema = SchemaBuilder.map(Schema.STRING_SCHEMA, null)
> .build();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5548) SchemaBuilder does not validate input.
[
https://issues.apache.org/jira/browse/KAFKA-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071129#comment-16071129
]
ASF GitHub Bot commented on KAFKA-5548:
---
Github user asfgit closed the pull request at:
https://github.com/apache/kafka/pull/3474
> SchemaBuilder does not validate input.
> --
>
> Key: KAFKA-5548
> URL: https://issues.apache.org/jira/browse/KAFKA-5548
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Jeremy Custenborder
>Assignee: Jeremy Custenborder
>Priority: Minor
> Fix For: 0.11.1.0
>
>
> SchemaBuilder.map(), SchemaBuilder.array(), and SchemaBuilder.field() do not
> validate input. This can cause weird NullPointerException exceptions later.
> For example I mistakenly called field("somefield", null), then later
> performed an operation against field.schema() which yielded a null. It would
> be preferable to throw an exception stating the issue. We could throw the a
> NPE but state what is null. Schema is null in this case for example.
> {code:java}
> @Test(expected = NullPointerException.class)
> public void fieldNameNull() {
> Schema schema = SchemaBuilder.struct()
> .field(null, Schema.STRING_SCHEMA)
> .build();
> }
> @Test(expected = NullPointerException.class)
> public void fieldSchemaNull() {
> Schema schema = SchemaBuilder.struct()
> .field("fieldName", null)
> .build();
> }
> @Test(expected = NullPointerException.class)
> public void arraySchemaNull() {
> Schema schema = SchemaBuilder.array(Schema.STRING_SCHEMA)
> .build();
> }
> @Test(expected = NullPointerException.class)
> public void mapKeySchemaNull() {
> Schema schema = SchemaBuilder.map(null, Schema.STRING_SCHEMA)
> .build();
> }
> @Test(expected = NullPointerException.class)
> public void mapValueSchemaNull() {
> Schema schema = SchemaBuilder.map(Schema.STRING_SCHEMA, null)
> .build();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5546) Lost data when the leader is disconnected.
[
https://issues.apache.org/jira/browse/KAFKA-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071187#comment-16071187
]
Michal Borowiecki commented on KAFKA-5546:
--
Does your producer check if sending was successful? That is, if the broker
acknowledged the message? If the data is never acknowledged by the broker, it
can be lost, it's not a defect.
As far as I can see you're just piping data into the console producer and the
logs on producers side indicate the messages simply weren't sent. Please
correct me if I'm reading it wrong.
Additionally, as far as I can tell, you are using acks=1 and unclean leader
election enabled (default in 0.10.2.1 but changed to disabled from 0.11.0.0).
This setup allows loss of even acknowledged messages on leader failure.
If you are trying to set up a resilient kafka cluster, please disable unclean
leader election and set ack >1 ("all" is the reasonable default) and of course,
check that the messages were sent.
> Lost data when the leader is disconnected.
> --
>
> Key: KAFKA-5546
> URL: https://issues.apache.org/jira/browse/KAFKA-5546
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Affects Versions: 0.10.2.1
>Reporter: Björn Eriksson
> Attachments: kafka-failure-log.txt
>
>
> We've noticed that if the leaders networking is deconfigured (with {{ifconfig
> eth0 down}}) the producer won't notice this and doesn't immediately connect
> to the newly elected leader.
> {{docker-compose.yml}} and test runner are at
> https://github.com/owbear/kafka-network-failure-tests with sample test output
> at
> https://github.com/owbear/kafka-network-failure-tests/blob/master/README.md#sample-results
> I was expecting a transparent failover to the new leader.
> The attached log shows that while the producer produced values between
> {{12:37:33}} and {{12:37:54}}, theres a gap between {{12:37:41}} and
> {{12:37:50}} where no values was stored in the log after the network was
> taken down at {{12:37:42}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (KAFKA-5546) Lost data when the leader is disconnected.
[
https://issues.apache.org/jira/browse/KAFKA-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071187#comment-16071187
]
Michal Borowiecki edited comment on KAFKA-5546 at 7/1/17 11:37 AM:
---
Does your producer check if sending was successful? That is, if the broker
acknowledged the message? If the data is never acknowledged by the broker, it
can be lost, it's not a defect.
As far as I can see you're just piping data into the console producer and the
logs on producers side indicate the messages simply weren't sent. Please
correct me if I'm reading it wrong.
Additionally, as far as I can tell, you are using acks=1 and unclean leader
election enabled (default in 0.10.2.1 but changed to disabled from 0.11.0.0).
This setup allows loss of even acknowledged messages on leader failure.
If you are trying to set up a resilient kafka cluster, please disable unclean
leader election and set ack >1 ("all" is the reasonable default) and of course,
check that the sending was successful, i.e. the broker acknowledged them.
was (Author: mihbor):
Does your producer check if sending was successful? That is, if the broker
acknowledged the message? If the data is never acknowledged by the broker, it
can be lost, it's not a defect.
As far as I can see you're just piping data into the console producer and the
logs on producers side indicate the messages simply weren't sent. Please
correct me if I'm reading it wrong.
Additionally, as far as I can tell, you are using acks=1 and unclean leader
election enabled (default in 0.10.2.1 but changed to disabled from 0.11.0.0).
This setup allows loss of even acknowledged messages on leader failure.
If you are trying to set up a resilient kafka cluster, please disable unclean
leader election and set ack >1 ("all" is the reasonable default) and of course,
check that the messages were sent.
> Lost data when the leader is disconnected.
> --
>
> Key: KAFKA-5546
> URL: https://issues.apache.org/jira/browse/KAFKA-5546
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Affects Versions: 0.10.2.1
>Reporter: Björn Eriksson
> Attachments: kafka-failure-log.txt
>
>
> We've noticed that if the leaders networking is deconfigured (with {{ifconfig
> eth0 down}}) the producer won't notice this and doesn't immediately connect
> to the newly elected leader.
> {{docker-compose.yml}} and test runner are at
> https://github.com/owbear/kafka-network-failure-tests with sample test output
> at
> https://github.com/owbear/kafka-network-failure-tests/blob/master/README.md#sample-results
> I was expecting a transparent failover to the new leader.
> The attached log shows that while the producer produced values between
> {{12:37:33}} and {{12:37:54}}, theres a gap between {{12:37:41}} and
> {{12:37:50}} where no values was stored in the log after the network was
> taken down at {{12:37:42}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (KAFKA-5546) Lost data when the leader is disconnected.
[
https://issues.apache.org/jira/browse/KAFKA-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071187#comment-16071187
]
Michal Borowiecki edited comment on KAFKA-5546 at 7/1/17 11:38 AM:
---
Does your producer check if sending was successful? That is, if the broker
acknowledged the message? If the data is never acknowledged by the broker, it
can be lost, it's not a defect.
As far as I can see you're just piping data into the console producer and the
logs on producers side indicate the messages simply weren't sent. Please
correct me if I'm reading it wrong.
Additionally, as far as I can tell, you are using acks=1 and unclean leader
election enabled (default in 0.10.2.1 but changed to disabled from 0.11.0.0).
This setup allows loss of even acknowledged messages on leader failure.
If you are trying to set up a resilient kafka cluster, please disable unclean
leader election and set ack >1 ("all" is the reasonable default) and of course,
check that the sending of messages was successful, i.e. the broker acknowledged
them.
was (Author: mihbor):
Does your producer check if sending was successful? That is, if the broker
acknowledged the message? If the data is never acknowledged by the broker, it
can be lost, it's not a defect.
As far as I can see you're just piping data into the console producer and the
logs on producers side indicate the messages simply weren't sent. Please
correct me if I'm reading it wrong.
Additionally, as far as I can tell, you are using acks=1 and unclean leader
election enabled (default in 0.10.2.1 but changed to disabled from 0.11.0.0).
This setup allows loss of even acknowledged messages on leader failure.
If you are trying to set up a resilient kafka cluster, please disable unclean
leader election and set ack >1 ("all" is the reasonable default) and of course,
check that the sending was successful, i.e. the broker acknowledged them.
> Lost data when the leader is disconnected.
> --
>
> Key: KAFKA-5546
> URL: https://issues.apache.org/jira/browse/KAFKA-5546
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Affects Versions: 0.10.2.1
>Reporter: Björn Eriksson
> Attachments: kafka-failure-log.txt
>
>
> We've noticed that if the leaders networking is deconfigured (with {{ifconfig
> eth0 down}}) the producer won't notice this and doesn't immediately connect
> to the newly elected leader.
> {{docker-compose.yml}} and test runner are at
> https://github.com/owbear/kafka-network-failure-tests with sample test output
> at
> https://github.com/owbear/kafka-network-failure-tests/blob/master/README.md#sample-results
> I was expecting a transparent failover to the new leader.
> The attached log shows that while the producer produced values between
> {{12:37:33}} and {{12:37:54}}, theres a gap between {{12:37:41}} and
> {{12:37:50}} where no values was stored in the log after the network was
> taken down at {{12:37:42}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5546) Lost data when the leader is disconnected.
[
https://issues.apache.org/jira/browse/KAFKA-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071252#comment-16071252
]
Björn Eriksson commented on KAFKA-5546:
---
Hi [~mihbor], I've updated the test to use {{acks=all}} and no unclean leader
election but the results are the same: the console consumer doesn't immediately
switch to the new leader and data is lost.
You're right, we're trying to set up a resilient Kafka cluster but this seems
difficult to achieve.
> Lost data when the leader is disconnected.
> --
>
> Key: KAFKA-5546
> URL: https://issues.apache.org/jira/browse/KAFKA-5546
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Affects Versions: 0.10.2.1
>Reporter: Björn Eriksson
> Attachments: kafka-failure-log.txt
>
>
> We've noticed that if the leaders networking is deconfigured (with {{ifconfig
> eth0 down}}) the producer won't notice this and doesn't immediately connect
> to the newly elected leader.
> {{docker-compose.yml}} and test runner are at
> https://github.com/owbear/kafka-network-failure-tests with sample test output
> at
> https://github.com/owbear/kafka-network-failure-tests/blob/master/README.md#sample-results
> I was expecting a transparent failover to the new leader.
> The attached log shows that while the producer produced values between
> {{12:37:33}} and {{12:37:54}}, theres a gap between {{12:37:41}} and
> {{12:37:50}} where no values was stored in the log after the network was
> taken down at {{12:37:42}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (KAFKA-3465) kafka.tools.ConsumerOffsetChecker won't align with kafka New Consumer mode
[ https://issues.apache.org/jira/browse/KAFKA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vahid Hashemian reassigned KAFKA-3465: -- Assignee: Vahid Hashemian > kafka.tools.ConsumerOffsetChecker won't align with kafka New Consumer mode > -- > > Key: KAFKA-3465 > URL: https://issues.apache.org/jira/browse/KAFKA-3465 > Project: Kafka > Issue Type: Improvement > Components: core >Affects Versions: 0.9.0.0 >Reporter: BrianLing >Assignee: Vahid Hashemian > > 1. When we enable mirrorMake to migrate Kafka event from one to other with > "new.consumer" mode: > java -Xmx2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 > -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC > -Djava.awt.headless=true -Dcom.sun.management.jmxremote > -Dcom.sun.management.jmxremote.authenticate=false > -Dcom.sun.management.jmxremote.ssl=false > -Dkafka.logs.dir=/kafka/kafka-app-logs > -Dlog4j.configuration=file:/kafka/kafka_2.10-0.9.0.0/bin/../config/tools-log4j.properties > -cp :/kafka/kafka_2.10-0.9.0.0/bin/../libs/* > -Dkafka.logs.filename=lvs-slca-mm.log kafka.tools.MirrorMaker lvs-slca-mm.log > --consumer.config ../config/consumer.properties --new.consumer --num.streams > 4 --producer.config ../config/producer-slca.properties --whitelist risk.* > 2. When we use ConsumerOffzsetChecker tool, notice the lag won't changed and > the owner is none. > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --broker-info > --group lvs.slca.mirrormaker --zookeeper lvsdmetlvm01.lvs.paypal.com:2181 > --topic > Group Topic Pid Offset logSize > Lag Owner > lvs.slca.mirrormaker 0 418578332 418678347 100015 > none > lvs.slca.mirrormaker 1 418598026 418698338 100312 > none > [Root Cause] > I think it's due to 0.9.0 new feature to switch zookeeper dependency to kafka > internal to store offset & consumer owner information. > Does it mean we can not use the below command to check new consumer’s > lag since current lag formula: lag= logSize – offset > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L80 > > https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L174-L182 > => offSet Fetch from zookeeper instead of from Kafka -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-3465) kafka.tools.ConsumerOffsetChecker won't align with kafka New Consumer mode
[ https://issues.apache.org/jira/browse/KAFKA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vahid Hashemian updated KAFKA-3465: --- Affects Version/s: 0.11.0.0 Priority: Minor (was: Major) Fix Version/s: 0.11.1.0 > kafka.tools.ConsumerOffsetChecker won't align with kafka New Consumer mode > -- > > Key: KAFKA-3465 > URL: https://issues.apache.org/jira/browse/KAFKA-3465 > Project: Kafka > Issue Type: Improvement > Components: core >Affects Versions: 0.9.0.0, 0.11.0.0 >Reporter: BrianLing >Assignee: Vahid Hashemian >Priority: Minor > Fix For: 0.11.1.0 > > > 1. When we enable mirrorMake to migrate Kafka event from one to other with > "new.consumer" mode: > java -Xmx2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 > -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC > -Djava.awt.headless=true -Dcom.sun.management.jmxremote > -Dcom.sun.management.jmxremote.authenticate=false > -Dcom.sun.management.jmxremote.ssl=false > -Dkafka.logs.dir=/kafka/kafka-app-logs > -Dlog4j.configuration=file:/kafka/kafka_2.10-0.9.0.0/bin/../config/tools-log4j.properties > -cp :/kafka/kafka_2.10-0.9.0.0/bin/../libs/* > -Dkafka.logs.filename=lvs-slca-mm.log kafka.tools.MirrorMaker lvs-slca-mm.log > --consumer.config ../config/consumer.properties --new.consumer --num.streams > 4 --producer.config ../config/producer-slca.properties --whitelist risk.* > 2. When we use ConsumerOffzsetChecker tool, notice the lag won't changed and > the owner is none. > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --broker-info > --group lvs.slca.mirrormaker --zookeeper lvsdmetlvm01.lvs.paypal.com:2181 > --topic > Group Topic Pid Offset logSize > Lag Owner > lvs.slca.mirrormaker 0 418578332 418678347 100015 > none > lvs.slca.mirrormaker 1 418598026 418698338 100312 > none > [Root Cause] > I think it's due to 0.9.0 new feature to switch zookeeper dependency to kafka > internal to store offset & consumer owner information. > Does it mean we can not use the below command to check new consumer’s > lag since current lag formula: lag= logSize – offset > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L80 > > https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L174-L182 > => offSet Fetch from zookeeper instead of from Kafka -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5551) StreamThread should not expose methods for testing
Matthias J. Sax created KAFKA-5551:
--
Summary: StreamThread should not expose methods for testing
Key: KAFKA-5551
URL: https://issues.apache.org/jira/browse/KAFKA-5551
Project: Kafka
Issue Type: Bug
Components: streams
Reporter: Matthias J. Sax
Assignee: Matthias J. Sax
{{StreamsThread}} currently exposes {{createStreamTask()}} and
{{createStandbyTask()}} as {{protected}} in order to inject "test tasks" in
unit tests. We should rework this and make both methods {{private}}. Maybe we
can introduce a {{TaskSupplier}} similar to {{KafkaClientSupplier}} (however,
{{TaskSupplier}} should not be public API and be in package {{internal}}).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
