[jira] [Commented] (KAFKA-4130) [docs] Link to Varnish architect notes is broken

[Andrea Cosentino (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15481236#comment-15481236
 ] 

Andrea Cosentino commented on KAFKA-4130:
-

PR submitted:

https://github.com/apache/kafka/pull/1835

> [docs] Link to Varnish architect notes is broken
> 
>
> Key: KAFKA-4130
> URL: https://issues.apache.org/jira/browse/KAFKA-4130
> Project: Kafka
>  Issue Type: Bug
>  Components: website
>Affects Versions: 0.9.0.1, 0.10.0.1
>Reporter: Stevo Slavic
>Assignee: Andrea Cosentino
>Priority: Trivial
>
> Paragraph in Kafka documentation
> {quote}
> This style of pagecache-centric design is described in an article on the 
> design of Varnish here (along with a healthy dose of arrogance). 
> {quote}
> contains a broken link.
> Should probably link to http://varnish-cache.org/wiki/ArchitectNotes



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4147) Transient failure in ConsumerCoordinatorTest.testAutoCommitDynamicAssignment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15481296#comment-15481296
 ] 

ASF GitHub Bot commented on KAFKA-4147:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1841


> Transient failure in ConsumerCoordinatorTest.testAutoCommitDynamicAssignment
> 
>
> Key: KAFKA-4147
> URL: https://issues.apache.org/jira/browse/KAFKA-4147
> Project: Kafka
>  Issue Type: Sub-task
>  Components: unit tests
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>
> Seen recently: 
> https://jenkins.confluent.io/job/kafka-trunk/1143/testReport/org.apache.kafka.clients.consumer.internals/ConsumerCoordinatorTest/testAutoCommitDynamicAssignment/.
> {code}
> java.lang.NullPointerException
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinatorTest.testAutoCommitDynamicAssignment(ConsumerCoordinatorTest.java:821)
> {code}
> Looks like it's caused by a race condition with the heartbeat thread.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1841: KAFKA-4147: Fix transient failure in ConsumerCoord...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1841


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4147) Transient failure in ConsumerCoordinatorTest.testAutoCommitDynamicAssignment

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4147:
---
Fix Version/s: 0.10.1.0

> Transient failure in ConsumerCoordinatorTest.testAutoCommitDynamicAssignment
> 
>
> Key: KAFKA-4147
> URL: https://issues.apache.org/jira/browse/KAFKA-4147
> Project: Kafka
>  Issue Type: Sub-task
>  Components: unit tests
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.10.1.0
>
>
> Seen recently: 
> https://jenkins.confluent.io/job/kafka-trunk/1143/testReport/org.apache.kafka.clients.consumer.internals/ConsumerCoordinatorTest/testAutoCommitDynamicAssignment/.
> {code}
> java.lang.NullPointerException
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinatorTest.testAutoCommitDynamicAssignment(ConsumerCoordinatorTest.java:821)
> {code}
> Looks like it's caused by a race condition with the heartbeat thread.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4147) Transient failure in ConsumerCoordinatorTest.testAutoCommitDynamicAssignment

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-4147.

Resolution: Fixed
  Reviewer: Ismael Juma

> Transient failure in ConsumerCoordinatorTest.testAutoCommitDynamicAssignment
> 
>
> Key: KAFKA-4147
> URL: https://issues.apache.org/jira/browse/KAFKA-4147
> Project: Kafka
>  Issue Type: Sub-task
>  Components: unit tests
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>
> Seen recently: 
> https://jenkins.confluent.io/job/kafka-trunk/1143/testReport/org.apache.kafka.clients.consumer.internals/ConsumerCoordinatorTest/testAutoCommitDynamicAssignment/.
> {code}
> java.lang.NullPointerException
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinatorTest.testAutoCommitDynamicAssignment(ConsumerCoordinatorTest.java:821)
> {code}
> Looks like it's caused by a race condition with the heartbeat thread.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-trunk-jdk8 #875

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk7 #1532

[Apache Jenkins Server]

See 

Changes:

[ismael] KAFKA-4147; Fix transient failure in

--
[...truncated 12295 lines...]
org.apache.kafka.streams.processor.internals.StreamTaskTest > 
testMaybePunctuate PASSED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
testEncodeDecode STARTED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
testEncodeDecode PASSED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
shouldDecodePreviousVersion STARTED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
shouldDecodePreviousVersion PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
testEncodeDecode STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
testEncodeDecode PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldEncodeDecodeWithUserEndPoint STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldEncodeDecodeWithUserEndPoint PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldBeBackwardCompatible STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldBeBackwardCompatible PASSED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testStickiness STARTED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testStickiness PASSED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithStandby STARTED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithStandby PASSED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithoutStandby STARTED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithoutStandby PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexingTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexingTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingStatefulTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingStatefulTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleMultiSourceTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleMultiSourceTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testTopologyMetadata STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testTopologyMetadata PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexByNameTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexByNameTopology PASSED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testSpecificPartition STARTED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testSpecificPartition PASSED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testStreamPartitioner STARTED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testStreamPartitioner PASSED

org.apache.kafka.streams.processor.internals.PunctuationQueueTest > 
testPunctuationInterval STARTED

org.apache.kafka.streams.processor.internals.PunctuationQueueTest > 
testPunctuationInterval PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigIsNotHostPortPair STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigIsNotHostPortPair PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldMapUserEndPointToTopicPartitions STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldMapUserEndPointToTopicPartitions PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldAddUserDefinedEndPointToSubscription STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldAddUserDefinedEndPointToSubscription PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testAssignWithStandbyReplicas STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testAssignWithStandbyReplicas PASSED

org.apache.kafka.streams.processo

Re: [DISCUSS] Kafka 0.10.1.0 Release Plan

[Jason Gustafson]

Hey Rajini,

We took a long look at KIP-55 and decided that the time needed to review,
stabilize, and add system testing might not be sufficient. Usually a
somewhat large patch like that takes a couple weeks of iteration before
landing in trunk. For a new security feature, it might be even longer. We
could delay the release for a week, but it's hard to know if that's enough
time and that might just put some other feature on edge (sort of by
induction, we risk never cutting the release). That said, if one of the
committers thinks it has a chance to get in and has the time to push it
through review, then I'm happy to add it.

Thanks,
Jason

On Sat, Sep 10, 2016 at 1:06 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Would it be possible to include KIP-55: Secure Quotas
>  55%3A+Secure+Quotas+for+Authenticated+Users>
> as
> well? The KIP was approved a while ago and the PR was submitted several
> weeks ago. I was hoping it would get reviewed in time for the next release.
> Jun had said he would take a look.
>
>
> Thank you,
>
> Rajini
>
> On Sat, Sep 10, 2016 at 8:26 AM, Ismael Juma  wrote:
>
> > Jason, thanks for putting this together and driving the release. Your
> > proposal sounds good to me. It would be nice to create a wiki page with
> the
> > information in this email. See the following for the one that Gwen put
> > together for 0.10.0:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.0
> >
> > Also, you merged KIP-70 recently so that can be moved to the completed
> > section.
> >
> > Ismael
> >
> > On Fri, Sep 9, 2016 at 11:45 PM, Jason Gustafson 
> > wrote:
> >
> > > Hi All,
> > >
> > > I've volunteered to be release manager for the upcoming 0.10.1 release
> > and
> > > would like to propose the following timeline:
> > >
> > > Feature Freeze (Sep. 19): The 0.10.1 release branch will be created.
> > > Code Freeze (Oct. 3): The first RC will go out.
> > > Final Release (~Oct. 17): Assuming no blocking issues remain, the final
> > > release will be cut.
> > >
> > > The purpose of the time between the feature freeze and code freeze is
> to
> > > stabilize the set of release features. We will continue to accept bug
> > fixes
> > > during this time and new system tests, but no new features will be
> merged
> > > into the release branch (they will continue to be accepted in trunk,
> > > however). After the code freeze, only blocking bug fixes will be
> > accepted.
> > > Features which cannot be completed in time will have to await the next
> > > release cycle.
> > >
> > > This is the first iteration of the time-based release plan:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Time+
> > Based+Release+Plan.
> > > Note
> > > that the final release is scheduled for October 17, so we have a little
> > > more than a month to prepare.
> > >
> > > Features which have already been merged to trunk and will be included
> in
> > > this release include the following:
> > >
> > > KIP-4 (partial): Add request APIs to create and delete topics
> > > KIP-33: Add time-based index
> > > KIP-60: Make Java client classloading more flexible
> > > KIP-62: Allow consumer to send heartbeats from a background thread
> > > KIP-65: Expose timestamps to Connect
> > > KIP-67: Queryable state for Kafka Streams
> > > KIP-71: Enable log compaction and deletion to co-exist
> > > KIP-75 - Add per-connector Converters
> > >
> > > Since this is the first time-based release, we propose to also include
> > the
> > > following KIPs which already have a patch available and have undergone
> > some
> > > review:
> > >
> > > KIP-58: Make log compaction point configurable
> > > KIP-63: Unify store and downstream caching in streams
> > > KIP-70: Revise consumer partition assignment semantics
> > > KIP-73: Replication quotas
> > > KIP-74: Add fetch response size limit in bytes
> > > KIP-78: Add clusterId
> > >
> > > One of the goals of time-based releases is to avoid the rush to get
> > > unstable features in before the release deadline. If a feature is not
> > ready
> > > now, the next release window is never far away. This helps to ensure
> the
> > > overall quality of the release. We've drawn the line for this release
> > based
> > > on feature progress and code review. For features which can't get in
> this
> > > time, don't worry since January will be here soon!
> > >
> > > Please let me know if you have any feedback on this plan.
> > >
> > > Thanks!
> > > Jason
> > >
> >
>
>
>
> --
> Regards,
>
> Rajini
>

Re: [DISCUSS] KIP-72 Allow Sizing Incoming Request Queue in Bytes

[Jun Rao]

Hi, Radai,

Thanks for the reply. I still have a followup question on #2.

My understanding is that in your proposal, selector will now first read the
size of the Receive. If there is not enough memory, it has to turn off the
READ interest bit for the corresponding KafkaChannel. Otherwise, subsequent
selector.poll() call will always return immediately, adding unnecessary
overhead. If you do that, the  Selector will need to know when to turn on
the READ interest bit again. It may not be enough to do this check until
the next poll call since the timeout used by poll() could be arbitrarily
large. So, it seems that some kind of coordination between the Selector and
the bufferpool is needed?

Jun

On Thu, Sep 8, 2016 at 7:02 PM, radai  wrote:

> Hi Jun,
>
> 1. yes, it is my own personal opinion that people use queued.max.requests
> as an indirect way to bound memory consumption. once a more direct memory
> bound mechanism exists (and works) i dont think queued.max.requests woul
> dbe required. having said that I was not planning on making any changes
> w.r.t queued.max.requests support (so I was aiming to get to a situation
> where both configs are supported) to allow gathering enough data/feedback.
>
> 2. Selector.poll() calls into KafkaChannel.read() to maybe get a
> NetworkReceive. multiple such read() calls may be required until a Receive
> is produced already in the current code base. my pool implementation is
> non-blocking so if there's no memory available the read() call will return
> null. poll() would then move on to try and service other selection keys.
> the pool will be checked for available memory again the next time the
> SocketServer.run() loop gets to poll(). and so right now I dont communicate
> memory becoming available to the selector - it will just go on to try and
> make progress elsewhere and come back again. i never block it or send it to
> sleep. I think for efficiency what could maybe be done is if there's not
> enough memory to service a readable selection key we may want to skip all
> other read-ready selection keys for that iteration of pollSelectionKeys().
> that would require rather invasive changes around
> Selector.pollSelectionKeys() that I'd rather avoid. also different
> KafkaChannels may be backed by different memory pool (under some sort of
> future QoS scheme?), which would complicate such an optimization further.
>
> 3. i added the pool interface and implementation under kafka.common.memory,
> and the API is "thin" enough to be generally useful (currently its
> non-blocking only, but a get(long maxWait) is definitely doable). having
> said that, I'm not really familiar enough with the code to say
>
>
>
> On Fri, Sep 2, 2016 at 2:04 PM, Jun Rao  wrote:
>
> > Hi, Radi,
> >
> > Thanks for the update. At the high level, this looks promising. A few
> > comments below.
> >
> > 1. If we can bound the requests by bytes, it seems that we don't need
> > queued.max.requests
> > any more? Could we just deprecate the config and make the queue size
> > unbounded?
> > 2. How do we communicate back to the selector when some memory is freed
> up?
> > We probably need to wake up the selector. For efficiency, perhaps we only
> > need to wake up the selector if the bufferpool is full?
> > 3. We talked about bounding the consumer's memory before. To fully
> support
> > that, we will need to bound the memory used by different fetch responses
> in
> > the consumer. Do you think the changes that you propose here can be
> > leveraged to bound the memory in the consumer as well?
> >
> > Jun
> >
> >
> > On Tue, Aug 30, 2016 at 10:41 AM, radai 
> > wrote:
> >
> > > My apologies for the delay in response.
> > >
> > > I agree with the concerns about OOM reading from the actual sockets and
> > > blocking the network threads - messing with the request queue itself
> > would
> > > not do.
> > >
> > > I propose instead a memory pool approach - the broker would have a non
> > > blocking memory pool. upon reading the first 4 bytes out of a socket an
> > > attempt would be made to acquire enough memory and if that attempt
> fails
> > > the processing thread will move on to try and make progress with other
> > > tasks.
> > >
> > > I think Its simpler than mute/unmute because using mute/unmute would
> > > require differentiating between sockets muted due to a request in
> > progress
> > > (normal current operation) and sockets muted due to lack of memory.
> > sockets
> > > of the 1st kind would be unmuted at the end of request processing (as
> it
> > > happens right now) but the 2nd kind would require some sort of "unmute
> > > watchdog" which is (i claim) more complicated than a memory pool. also
> a
> > > memory pool is a more generic solution.
> > >
> > > I've updated the KIP page (
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 72%3A+Allow+putting+a+bound+on+memory+consumed+by+Incoming+requests)
> > > to reflect the new proposed implementation, and i've also put up an
> > inital
> >

[GitHub] kafka pull request #1794: KAFKA-1981 Make log compaction point configurable

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1794


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-1981) Make log compaction point configurable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15482763#comment-15482763
 ] 

ASF GitHub Bot commented on KAFKA-1981:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1794


> Make log compaction point configurable
> --
>
> Key: KAFKA-1981
> URL: https://issues.apache.org/jira/browse/KAFKA-1981
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>  Labels: newbie++
> Fix For: 0.10.1.0
>
> Attachments: KIP for Kafka Compaction Patch.md
>
>
> Currently if you enable log compaction the compactor will kick in whenever 
> you hit a certain "dirty ratio", i.e. when 50% of your data is uncompacted. 
> Other than this we don't give you fine-grained control over when compaction 
> occurs. In addition we never compact the active segment (since it is still 
> being written to).
> Other than this we don't really give you much control over when compaction 
> will happen. The result is that you can't really guarantee that a consumer 
> will get every update to a compacted topic--if the consumer falls behind a 
> bit it might just get the compacted version.
> This is usually fine, but it would be nice to make this more configurable so 
> you could set either a # messages, size, or time bound for compaction.
> This would let you say, for example, "any consumer that is no more than 1 
> hour behind will get every message."
> This should be relatively easy to implement since it just impacts the 
> end-point the compactor considers available for compaction. I think we 
> already have that concept, so this would just be some other overrides to add 
> in when calculating that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-1981) Make log compaction point configurable

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-1981.

   Resolution: Fixed
Fix Version/s: 0.10.1.0

Issue resolved by pull request 1794
[https://github.com/apache/kafka/pull/1794]

> Make log compaction point configurable
> --
>
> Key: KAFKA-1981
> URL: https://issues.apache.org/jira/browse/KAFKA-1981
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>  Labels: newbie++
> Fix For: 0.10.1.0
>
> Attachments: KIP for Kafka Compaction Patch.md
>
>
> Currently if you enable log compaction the compactor will kick in whenever 
> you hit a certain "dirty ratio", i.e. when 50% of your data is uncompacted. 
> Other than this we don't give you fine-grained control over when compaction 
> occurs. In addition we never compact the active segment (since it is still 
> being written to).
> Other than this we don't really give you much control over when compaction 
> will happen. The result is that you can't really guarantee that a consumer 
> will get every update to a compacted topic--if the consumer falls behind a 
> bit it might just get the compacted version.
> This is usually fine, but it would be nice to make this more configurable so 
> you could set either a # messages, size, or time bound for compaction.
> This would let you say, for example, "any consumer that is no more than 1 
> hour behind will get every message."
> This should be relatively easy to implement since it just impacts the 
> end-point the compactor considers available for compaction. I think we 
> already have that concept, so this would just be some other overrides to add 
> in when calculating that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #1533

[Apache Jenkins Server]

See 

Changes:

[junrao] KAFKA-1981; Make log compaction point configurable

--
[...truncated 1073 lines...]
kafka.server.SaslPlaintextReplicaFetchTest > testReplicaFetcherThread STARTED

kafka.server.SaslPlaintextReplicaFetchTest > testReplicaFetcherThread PASSED

kafka.server.CreateTopicsRequestTest > testValidCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testValidCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testErrorCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testErrorCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testInvalidCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testInvalidCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testNotController STARTED

kafka.server.CreateTopicsRequestTest > testNotController PASSED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK STARTED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK PASSED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot STARTED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot PASSED

kafka.server.ServerStartupTest > testConflictBrokerRegistration STARTED

kafka.server.ServerStartupTest > testConflictBrokerRegistration PASSED

kafka.server.ServerStartupTest > testBrokerSelfAware STARTED

kafka.server.ServerStartupTest > testBrokerSelfAware PASSED

kafka.server.ApiVersionsRequestTest > 
testApiVersionsRequestWithUnsupportedVersion STARTED

kafka.server.ApiVersionsRequestTest > 
testApiVersionsRequestWithUnsupportedVersion PASSED

kafka.server.ApiVersionsRequestTest > testApiVersionsRequest STARTED

kafka.server.ApiVersionsRequestTest > testApiVersionsRequest PASSED

kafka.server.IsrExpirationTest > testIsrExpirationForSlowFollowers STARTED

kafka.server.IsrExpirationTest > testIsrExpirationForSlowFollowers PASSED

kafka.server.IsrExpirationTest > testIsrExpirationForStuckFollowers STARTED

kafka.server.IsrExpirationTest > testIsrExpirationForStuckFollowers PASSED

kafka.server.IsrExpirationTest > testIsrExpirationIfNoFetchRequestMade STARTED

kafka.server.IsrExpirationTest > testIsrExpirationIfNoFetchRequestMade PASSED

kafka.server.AdvertiseBrokerTest > testBrokerAdvertiseToZK STARTED

kafka.server.AdvertiseBrokerTest > testBrokerAdvertiseToZK PASSED

kafka.server.MetadataRequestTest > testReplicaDownResponse STARTED

kafka.server.MetadataRequestTest > testReplicaDownResponse PASSED

kafka.server.MetadataRequestTest > testRack STARTED

kafka.server.MetadataRequestTest > testRack PASSED

kafka.server.MetadataRequestTest > testIsInternal STARTED

kafka.server.MetadataRequestTest > testIsInternal PASSED

kafka.server.MetadataRequestTest > testControllerId STARTED

kafka.server.MetadataRequestTest > testControllerId PASSED

kafka.server.MetadataRequestTest > testAllTopicsRequest STARTED

kafka.server.MetadataRequestTest > testAllTopicsRequest PASSED

kafka.server.MetadataRequestTest > testNoTopicsRequest STARTED

kafka.server.MetadataRequestTest > testNoTopicsRequest PASSED

kafka.server.MetadataCacheTest > 
getTopicMetadataWithNonSupportedSecurityProtocol STARTED

kafka.server.MetadataCacheTest > 
getTopicMetadataWithNonSupportedSecurityProtocol PASSED

kafka.server.MetadataCacheTest > getTopicMetadataIsrNotAvailable STARTED

kafka.server.MetadataCacheTest > getTopicMetadataIsrNotAvailable PASSED

kafka.server.MetadataCacheTest > getTopicMetadata STARTED

kafka.server.MetadataCacheTest > getTopicMetadata PASSED

kafka.server.MetadataCacheTest > getTopicMetadataReplicaNotAvailable STARTED

kafka.server.MetadataCacheTest > getTopicMetadataReplicaNotAvailable PASSED

kafka.server.MetadataCacheTest > getTopicMetadataPartitionLeaderNotAvailable 
STARTED

kafka.server.MetadataCacheTest > getTopicMetadataPartitionLeaderNotAvailable 
PASSED

kafka.server.MetadataCacheTest > getAliveBrokersShouldNotBeMutatedByUpdateCache 
STARTED

kafka.server.MetadataCacheTest > getAliveBrokersShouldNotBeMutatedByUpdateCache 
PASSED

kafka.server.MetadataCacheTest > getTopicMetadataNonExistingTopics STARTED

kafka.server.MetadataCacheTest > getTopicMetadataNonExistingTopics PASSED

kafka.server.SaslSslReplicaFetchTest > testReplicaFetcherThread STARTED

kafka.server.SaslSslReplicaFetchTest > testReplicaFetcherThread PASSED

kafka.server.ProduceRequestTest > testSimpleProduceRequest STARTED

kafka.server.ProduceRequestTest > testSimpleProduceRequest PASSED

kafka.server.ProduceRequestTest > testCorruptLz4ProduceRequest STARTED

kafka.server.ProduceRequestTest > testCorruptLz4ProduceRequest PASSED

kafka.tools.MirrorMakerTest > 
testDefaultMirrorMakerMessageHandlerWithNoTimestampInSourceMessage STARTED

kafka.tools.MirrorMakerTest > 
testDefaultMirrorMakerMessageHandlerWithNoTimestampInSourceMessage PASSED

kafka.tools.MirrorMakerTest > testDefaultMirrorMakerMessageHandler STARTED

kafka.tools.M

Re: Queryable state client read guarantees

[Guozhang Wang]

Hi Mikael,

Just adding to Damian's comment above, that the IllegalStateStoreException
here is thrown to indicate a "transient" state where the state store
hosting this key is being migrated and hence not available, where users
implementing the REST APIs on top of it, for example, can choose to handle
it differently. For example, either return a sentinel value as "key not
available" or return some error codes.

Guozhang


On Fri, Sep 9, 2016 at 9:40 AM, Damian Guy  wrote:

> Hi Mikael,
>
> During rebalance both instances should throw IllegalStateStoreException
> until the rebalance has completed. Once the rebalance has completed if the
> key is not found on the local store, then you would get a null value. You
> can always find the Kafka Streams instance that will have that key
> (assuming it exists) by using:
>
> StreamsMetadata KafkaStreams.metadataForKey(String storeName, K key,
> Serializer keySerializer)
>
> The StreamsMetadata will tell you which instance, via HostInfo, has the
> given key.
>
> HTH,
> Damian
>
>
>
>
> On Fri, 9 Sep 2016 at 16:56 Mikael Högqvist  wrote:
>
> > Hi Damian,
> >
> > thanks for fixing this so quickly, I re-ran the test and it works fine.
> >
> > The next test I tried was to read from two service instances implementing
> > the same string count topology. First, the client is started sending two
> > read requests, one per instance, every second. Next, I start the first
> > instance and let it complete the store init before the next instance is
> > started.
> >
> > Below is the initial part of the trace when going from 0 to 1 instance.
> The
> > trace log has the following columns: request id, instance, response code
> > and value.
> >
> > 3,localhost:2030,503,
> > 3,localhost:2031,503,
> > 4,localhost:2030,503,
> > 4,localhost:2031,503,
> > 5,localhost:2030,200,2
> > 5,localhost:2031,503,
> > 6,localhost:2030,200,2
> > 6,localhost:2031,503,
> >
> > Before the instance is started, both return 503, the status returned by
> the
> > client when it cannot connect to an instance. When the first instance has
> > started it returns the expected value 2 for request pair 5, 6 and so on.
> > The trace below is from when the second instance starts.
> >
> > 18,localhost:2030,200,2
> > 18,localhost:2031,503,
> > 19,localhost:2030,404,
> > 19,localhost:2031,503,
> > 20,localhost:2030,404,
> > 20,localhost:2031,503,
> > 21,localhost:2030,404,
> > 21,localhost:2031,200,2
> > 22,localhost:2030,404,
> > 22,localhost:2031,200,2
> >
> > The new instance takes over responsibility for the partition containing
> the
> > key "hello". During this period the new instance returns 503 as expected
> > until the store is ready. The issue is that the first instance that
> stored
> > the value starts returning 404 from request pair 19. A client doing
> > requests for this key would then have the following sequence:
> >
> > 18 -> 2
> > 19 -> Not found
> > 20 -> Not found
> > 21 -> 2
> >
> > From the client perspective, I think this violates the guarantee of
> always
> > reading the latest value.
> >
> > Am I making the wrong assumptions or is there some way to detect that the
> > local store is not responsible for the key anymore?
> >
> > Best,
> > Mikael
> >
> > On Thu, Sep 8, 2016 at 11:03 AM Damian Guy  wrote:
> >
> > > Hi Mikael,
> > >
> > > A fix for KAFKA-4123  >
> > > (the
> > > issue you found with receiving null values) has now been committed to
> > > trunk. I've tried it with your github repo and it appears to be
> working.
> > > You will have to make a small change to your code as we now throw
> > > InvalidStateStoreException when the Stores are unavailable (previously
> we
> > > returned null).
> > >
> > > We added a test here
> > > <
> > >
> > https://github.com/apache/kafka/blob/trunk/streams/src/
> test/java/org/apache/kafka/streams/integration/
> QueryableStateIntegrationTest.java#L431
> > > >
> > > to
> > > make sure we only get a value once the store has been (re-)initialized.
> > > Please give it a go and thanks for your help in finding this issue.
> > >
> > > Thanks,
> > > Damian
> > >
> > > On Mon, 5 Sep 2016 at 22:07 Mikael Högqvist 
> wrote:
> > >
> > > > Hi Damian,
> > > >
> > > > > > Failed to read key hello, org.mkhq.kafka.Topology$
> StoreUnavailable
> > > > > > > Failed to read key hello, org.mkhq.kafka.Topology$KeyNotFound
> > > > > > > hello -> 10
> > > > > >
> > > > > >
> > > > > The case where you get KeyNotFound looks like a bug to me. This
> > > shouldn't
> > > > > happen. I can see why it might happen and we will create a JIRA and
> > fix
> > > > it
> > > > > right away.
> > > > >
> > > >
> > > > Great, thanks for looking into this. I'll try again once the PR is
> > > merged.
> > > >
> > > >
> > > > >
> > > > > I'm not sure how you end up with (hello -> 10). It could indicate
> > that
> > > > the
> > > > > offsets for the topic you are consuming from weren't committed so
> the
> > > > data
> > > > > gets processed again on t

[GitHub] kafka pull request #1840: MINOR: catch InvalidStateStoreException in Queryab...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1840


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk8 #877

[Apache Jenkins Server]

See 

Changes:

[wangguoz] MINOR: catch InvalidStateStoreException in 
QueryableStateIntegrationTest

--
[...truncated 12374 lines...]
org.apache.kafka.streams.kstream.KStreamBuilderTest > testMerge STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > testMerge PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > testFrom STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > testFrom PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldThrowExceptionWhenTopicNamesAreNull STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldThrowExceptionWhenTopicNamesAreNull PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldThrowExceptionWhenNoTopicPresent STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldThrowExceptionWhenNoTopicPresent PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > testNewName STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > testNewName PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
shouldHaveSaneEqualsAndHashCode STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
shouldHaveSaneEqualsAndHashCode PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowSizeMustNotBeZero 
STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowSizeMustNotBeZero 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > advanceIntervalMustNotBeZero 
STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > advanceIntervalMustNotBeZero 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowSizeMustNotBeNegative 
STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowSizeMustNotBeNegative 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
advanceIntervalMustNotBeNegative STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
advanceIntervalMustNotBeNegative PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
advanceIntervalMustNotBeLargerThanWindowSize STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
advanceIntervalMustNotBeLargerThanWindowSize PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowsForTumblingWindows 
STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowsForTumblingWindows 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowsForHoppingWindows 
STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowsForHoppingWindows 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
windowsForBarelyOverlappingHoppingWindows STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
windowsForBarelyOverlappingHoppingWindows PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
STARTED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportMultipleBootstrapServers STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportMultipleBootstrapServers PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportNonPrefixedProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportNonPrefixedProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetRestoreConsumerConfigs 
STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetRestoreConsumerConfigs 
PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedRestoreConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedRestoreConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedRestoreConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedRestoreConsumerConfigs PASSED

org.apache.kafka.streams.KafkaStreamsTest > shouldNotGetAllTasksWhenNotRunning 
STARTED

org.apache.kafka.streams.KafkaStreamsTest > shouldNotGetAllTasksWhenNotRunning 
PASSED

org.apache.kafka.streams.KafkaStreamsTest > 
shouldNotGetTaskWithKeyAndPartitionerWhenNotRunni

Re: Queryable state client read guarantees

[Mikael Högqvist]

Hi,

this helps, thanks!

Basically, after each read, I'll check if the key is still supposed to be
on the host. Doing the check after the read is necessary to handle the case
when a rebalance happens in between the metadata lookup and the store get.
When checking after the read, it may happen that a valid read becomes
invalid, but that doesn't affect correctness.

During a rebalance the service either responds not available or redirect.
After the rebalance is completed, the store responds with redirect. With a
REST API, this could mean either 404 or a 303, temporary redirect to the
current host.

Best,
Mikael

On Mon, Sep 12, 2016 at 5:42 AM Guozhang Wang  wrote:

> Hi Mikael,
>
> Just adding to Damian's comment above, that the IllegalStateStoreException
> here is thrown to indicate a "transient" state where the state store
> hosting this key is being migrated and hence not available, where users
> implementing the REST APIs on top of it, for example, can choose to handle
> it differently. For example, either return a sentinel value as "key not
> available" or return some error codes.
>
> Guozhang
>
>
> On Fri, Sep 9, 2016 at 9:40 AM, Damian Guy  wrote:
>
> > Hi Mikael,
> >
> > During rebalance both instances should throw IllegalStateStoreException
> > until the rebalance has completed. Once the rebalance has completed if
> the
> > key is not found on the local store, then you would get a null value. You
> > can always find the Kafka Streams instance that will have that key
> > (assuming it exists) by using:
> >
> > StreamsMetadata KafkaStreams.metadataForKey(String storeName, K key,
> > Serializer keySerializer)
> >
> > The StreamsMetadata will tell you which instance, via HostInfo, has the
> > given key.
> >
> > HTH,
> > Damian
> >
> >
> >
> >
> > On Fri, 9 Sep 2016 at 16:56 Mikael Högqvist  wrote:
> >
> > > Hi Damian,
> > >
> > > thanks for fixing this so quickly, I re-ran the test and it works fine.
> > >
> > > The next test I tried was to read from two service instances
> implementing
> > > the same string count topology. First, the client is started sending
> two
> > > read requests, one per instance, every second. Next, I start the first
> > > instance and let it complete the store init before the next instance is
> > > started.
> > >
> > > Below is the initial part of the trace when going from 0 to 1 instance.
> > The
> > > trace log has the following columns: request id, instance, response
> code
> > > and value.
> > >
> > > 3,localhost:2030,503,
> > > 3,localhost:2031,503,
> > > 4,localhost:2030,503,
> > > 4,localhost:2031,503,
> > > 5,localhost:2030,200,2
> > > 5,localhost:2031,503,
> > > 6,localhost:2030,200,2
> > > 6,localhost:2031,503,
> > >
> > > Before the instance is started, both return 503, the status returned by
> > the
> > > client when it cannot connect to an instance. When the first instance
> has
> > > started it returns the expected value 2 for request pair 5, 6 and so
> on.
> > > The trace below is from when the second instance starts.
> > >
> > > 18,localhost:2030,200,2
> > > 18,localhost:2031,503,
> > > 19,localhost:2030,404,
> > > 19,localhost:2031,503,
> > > 20,localhost:2030,404,
> > > 20,localhost:2031,503,
> > > 21,localhost:2030,404,
> > > 21,localhost:2031,200,2
> > > 22,localhost:2030,404,
> > > 22,localhost:2031,200,2
> > >
> > > The new instance takes over responsibility for the partition containing
> > the
> > > key "hello". During this period the new instance returns 503 as
> expected
> > > until the store is ready. The issue is that the first instance that
> > stored
> > > the value starts returning 404 from request pair 19. A client doing
> > > requests for this key would then have the following sequence:
> > >
> > > 18 -> 2
> > > 19 -> Not found
> > > 20 -> Not found
> > > 21 -> 2
> > >
> > > From the client perspective, I think this violates the guarantee of
> > always
> > > reading the latest value.
> > >
> > > Am I making the wrong assumptions or is there some way to detect that
> the
> > > local store is not responsible for the key anymore?
> > >
> > > Best,
> > > Mikael
> > >
> > > On Thu, Sep 8, 2016 at 11:03 AM Damian Guy 
> wrote:
> > >
> > > > Hi Mikael,
> > > >
> > > > A fix for KAFKA-4123 <
> https://issues.apache.org/jira/browse/KAFKA-4123
> > >
> > > > (the
> > > > issue you found with receiving null values) has now been committed to
> > > > trunk. I've tried it with your github repo and it appears to be
> > working.
> > > > You will have to make a small change to your code as we now throw
> > > > InvalidStateStoreException when the Stores are unavailable
> (previously
> > we
> > > > returned null).
> > > >
> > > > We added a test here
> > > > <
> > > >
> > > https://github.com/apache/kafka/blob/trunk/streams/src/
> > test/java/org/apache/kafka/streams/integration/
> > QueryableStateIntegrationTest.java#L431
> > > > >
> > > > to
> > > > make sure we only get a value once the store has been
> (re-)initialized.
> > > > Please give it a go and thanks fo