[GitHub] kafka pull request #2199: HOTFIX: Temporary suspension of 2 tests

[enothereska]

Github user enothereska closed the pull request at:

https://github.com/apache/kafka/pull/2199


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk7 #1728

[Apache Jenkins Server]

See 

Changes:

[jason] KAFKA-4483; Fix NPE in `Log` constructor if log level is INFO or finer

--
[...truncated 14212 lines...]
org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldAssociateStateStoreNameWhenStateStoreSupplierIsInternal STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldAssociateStateStoreNameWhenStateStoreSupplierIsInternal PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testAddStateStoreWithSink STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testAddStateStoreWithSink PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > testTopicGroups STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > testTopicGroups PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > testBuild STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > testBuild PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAddNullStateStoreSupplier STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAddNullStateStoreSupplier PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAllowNullNameWhenAddingSource STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAllowNullNameWhenAddingSource PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAllowNullTopicWhenAddingSink STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAllowNullTopicWhenAddingSink PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testAddStateStoreWithSource STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testAddStateStoreWithSource PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldThroughOnUnassignedStateStoreAccess STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldThroughOnUnassignedStateStoreAccess PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAllowNullProcessorNameWhenConnectingProcessorAndStateStores STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAllowNullProcessorNameWhenConnectingProcessorAndStateStores PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testAddSourceWithSameName STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testAddSourceWithSameName PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testAddMoreThanOnePatternSourceNode STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testAddMoreThanOnePatternSourceNode PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testAddProcessorWithSameName STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testAddProcessorWithSameName PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldSetCorrectSourceNodesWithRegexUpdatedTopics STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldSetCorrectSourceNodesWithRegexUpdatedTopics PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testAddSourceWithSameTopic STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testAddSourceWithSameTopic PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testTopicGroupsByStateStore STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testTopicGroupsByStateStore PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testAddStateStoreWithDuplicates STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testAddStateStoreWithDuplicates PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > testPatternSourceTopic 
STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > testPatternSourceTopic 
PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldAssociateStateStoreNameWhenStateStoreSupplierIsExternal STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldAssociateStateStoreNameWhenStateStoreSupplierIsExternal PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testAddSinkWithWrongParent STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testAddSinkWithWrongParent PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testPatternMatchesAlreadyProvidedTopicSource STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testPatternMatchesAlreadyProvidedTopicSource PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testAddSinkConnectedWithMultipleParent STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testAddSinkConnectedWithMultipleParent PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testSubscribeTopicNameAndPattern STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testSubscribeTopicNameAndPattern PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAdd

[GitHub] kafka pull request #2209: KAFKA-4476: Kafka Streams gets stuck if metadata i...

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/2209

KAFKA-4476: Kafka Streams gets stuck if metadata is missing

 - break loop in StreamPartitionAssigner.assign() in case partition 
metadata is missing
 - fit state transition issue (follow up to KAFKA-3637: Add method that 
checks if streams are initialised)
 - some test improvements

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka 
kafka-4476-stuck-on-missing-metadata

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2209.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2209


commit 77598544bc5c2f9d0c967bcdda392c701ebacff7
Author: Matthias J. Sax 
Date:   2016-12-03T09:19:34Z

KAFKA-4476: Kafka Streams gets stuck if metadata is missing
 - break loop in StreamPartitionAssigner.assign() in case partition 
metadata is missing
 - fit state transition issue (follow up to KAFKA-3637: Add method that 
checks if streams are initialised)
 - some test improvements




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4476) Kafka Streams gets stuck if metadata is missing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15717780#comment-15717780
 ] 

ASF GitHub Bot commented on KAFKA-4476:
---

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/2209

KAFKA-4476: Kafka Streams gets stuck if metadata is missing

 - break loop in StreamPartitionAssigner.assign() in case partition 
metadata is missing
 - fit state transition issue (follow up to KAFKA-3637: Add method that 
checks if streams are initialised)
 - some test improvements

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka 
kafka-4476-stuck-on-missing-metadata

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2209.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2209


commit 77598544bc5c2f9d0c967bcdda392c701ebacff7
Author: Matthias J. Sax 
Date:   2016-12-03T09:19:34Z

KAFKA-4476: Kafka Streams gets stuck if metadata is missing
 - break loop in StreamPartitionAssigner.assign() in case partition 
metadata is missing
 - fit state transition issue (follow up to KAFKA-3637: Add method that 
checks if streams are initialised)
 - some test improvements




> Kafka Streams gets stuck if metadata is missing
> ---
>
> Key: KAFKA-4476
> URL: https://issues.apache.org/jira/browse/KAFKA-4476
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Critical
>
> When a Kafka Streams application gets started for the first time, it can 
> happen that some topic metadata is missing when 
> {{StreamPartitionAssigner#assign()}} is called on the group leader instance. 
> This can result in an infinite loop within 
> {{StreamPartitionAssigner#assign()}}. This issue was detected by 
> {{ResetIntegrationTest}} that does have a transient timeout failure (c.f. 
> https://issues.apache.org/jira/browse/KAFKA-4058 -- this issue was re-opened 
> multiple times as the problem was expected to be in the test -- however, that 
> is not the case).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4476) Kafka Streams gets stuck if metadata is missing

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-4476:
---
Status: Patch Available  (was: In Progress)

> Kafka Streams gets stuck if metadata is missing
> ---
>
> Key: KAFKA-4476
> URL: https://issues.apache.org/jira/browse/KAFKA-4476
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Critical
>
> When a Kafka Streams application gets started for the first time, it can 
> happen that some topic metadata is missing when 
> {{StreamPartitionAssigner#assign()}} is called on the group leader instance. 
> This can result in an infinite loop within 
> {{StreamPartitionAssigner#assign()}}. This issue was detected by 
> {{ResetIntegrationTest}} that does have a transient timeout failure (c.f. 
> https://issues.apache.org/jira/browse/KAFKA-4058 -- this issue was re-opened 
> multiple times as the problem was expected to be in the test -- however, that 
> is not the case).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Suppressing redundant KTable forwards

[Mathieu Fenniak]

Hey all,

I'd like to contribute a new KTable API that would allow for the
suppression of redundant KTable forwards, and I'd like to solicit feedback
before I put together a patch.

A typical use-case of this API would be that you're using mapValues to
pluck a subset of data out of a topic, but you'd like changes to the record
value that don't modify the output of mapValues to not cause output that
trigger expensive and redundant recalculations.

For example, topic "user" contains key:1, value:{"firstName": "Jack",
"lastName": "Brown"}.  builder.topic("user").mapValues((user) ->
user.get("lastName"))  will create a KTable that would forward updates from
the user topic even if lastName never changed.

My proposed API would be to add a filterRedundant method to KTable; one
override takes a Comparator to provide a custom comparison for
evaluating whether a change is redundant, and one parameterless override
would use a comparator backed by the object's equals() method.

/**
 * Creates a new instance of {@link KTable} that filters out redundant
updates and prevents "non-updates" from
 * propagating to further operations on the returned table.  A
redundant update onewhere the same value is provided
 * more than once for a given key.  Object.equals is used to compare
whether a subsequent update has the same value.

 * @return a {@link KTable} that contains the same values as this
table, but suppresses redundant updates
 */
KTable filterRedundant();

/**
 * Creates a new instance of {@link KTable} that filters out redundant
updates and prevents "non-updates" from
 * propagating to further operations on the returned table.  A
redundant update onewhere the same value is provided
 * more than once for a given key.  A user-provided comparator is used
to compare whether a subsequent update has
 * the same value.

 * @return a {@link KTable} that contains the same values as this
table, but suppresses redundant updates
 */
KTable filterRedundant(Comparator comparator);

Extending KStream DSL in applications

[Mathieu Fenniak]

Hello devs,

I'd like to be able to create my own KTableProcessorSupplier implementation
in a Kafka Streams application, but it's currently not possible without the
use of reflection due to KTableImpl's valueGetterSupplier
& enableSendingOldValues methods being package-local.

Would there be any opposition to a PR to make these methods public?

I do understand that implementing interfaces & using classes in an
"internals" package are obviously going to be an unstable API.  But my
current alternative is to swap from DSL to low-level API via a topic, and
then swap back, when I can't do something directly with the existing API...
and that makes my extensions quite divergent from anything I could feasibly
contribute back to the library in the future. :-)

Thanks,

Mathieu