Re: [VOTE] KIP-280: Enhanced log compaction

[Matthias J. Sax]

+1 (binding)

On 11/5/19 11:44 AM, Senthilnathan Muthusamy wrote:
> Thanks Gouzhang and I have made a note in the JIRA item to update the wiki.
> 
> Till now got 1 +1 binding... waiting for 2 more +1 binding... thnx!
> 
> Regards,
> Senthil
> 
> -Original Message-
> From: Guozhang Wang  
> Sent: Monday, November 4, 2019 11:01 AM
> To: dev 
> Subject: Re: [VOTE] KIP-280: Enhanced log compaction
> 
> I only have one minor comment on the DISCUSS thread, otherwise I'm +1 
> (binding).
> 
> On Mon, Nov 4, 2019 at 9:53 AM Senthilnathan Muthusamy 
>  wrote:
> 
>> Hi all,
>>
>> I would like to start the vote on the updated KIP-280: Enhanced log 
>> compaction. Thanks to Guozhang, Matthias & Tom for the valuable 
>> feedback on the discussion thread...
>>
>> KIP:
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik
>> i.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-280%253A%2BEnhanced%
>> 2Blog%2Bcompaction&data=02%7C01%7Csenthilm%40microsoft.com%7Ca8ca2
>> 5d3f1894d0d271f08d7615966d3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0
>> %7C637085005478393331&sdata=qrttmbYi2Ea4qfcF5qKVbn7CaYwmvRylO85dfj
>> IY6pI%3D&reserved=0
>>
>> JIRA: 
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissu
>> es.apache.org%2Fjira%2Fbrowse%2FKAFKA-7061&data=02%7C01%7Csenthilm
>> %40microsoft.com%7Ca8ca25d3f1894d0d271f08d7615966d3%7C72f988bf86f141af
>> 91ab2d7cd011db47%7C1%7C0%7C637085005478393331&sdata=7c%2BzF3XRRz%2
>> BijyyjBRntP6ZMWqnyzy4BEE8rqnZaF1s%3D&reserved=0
>>
>> Thanks,
>> Senthil
>>
> 
> 
> --
> -- Guozhang
> 



signature.asc
Description: OpenPGP digital signature

[jira] [Created] (KAFKA-9150) DescribeGroup uses member assignment as metadata

[Jason Gustafson (Jira)]

Jason Gustafson created KAFKA-9150:
--

 Summary: DescribeGroup uses member assignment as metadata
 Key: KAFKA-9150
 URL: https://issues.apache.org/jira/browse/KAFKA-9150
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson
 Fix For: 2.4.0


When we converted the DescribeGroup internally to rely on the generated 
protocol in KAFKA-7922, we introduced a regression in the response handling. 
Basically we serialize the member assignment as both the assignment and 
metadata in the response: 
[https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala#L1326].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-2.4-jdk8 #53

[Apache Jenkins Server]

See 


Changes:

[wangguoz] HOTFIX: remove reference to unused Assignment error code (#7645)


--
[...truncated 5.03 MB...]
kafka.api.GroupEndToEndAuthorizationTest > testProduceConsumeWithWildcardAcls 
PASSED

kafka.api.GroupEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe STARTED

kafka.api.GroupEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe PASSED

kafka.api.GroupEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign STARTED

kafka.api.GroupEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign PASSED

kafka.api.GroupEndToEndAuthorizationTest > testNoGroupAcl STARTED

kafka.api.GroupEndToEndAuthorizationTest > testNoGroupAcl PASSED

kafka.api.GroupEndToEndAuthorizationTest > testNoProduceWithDescribeAcl STARTED

kafka.api.GroupEndToEndAuthorizationTest > testNoProduceWithDescribeAcl PASSED

kafka.api.GroupEndToEndAuthorizationTest > 
testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl STARTED

kafka.api.GroupEndToEndAuthorizationTest > 
testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl PASSED

kafka.api.GroupEndToEndAuthorizationTest > testProduceConsumeViaSubscribe 
STARTED

kafka.api.GroupEndToEndAuthorizationTest > testProduceConsumeViaSubscribe PASSED

kafka.api.SaslPlaintextConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslPlaintextConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslPlaintextConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslPlaintextConsumerTest > testSimpleConsumption PASSED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials STARTED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials PASSED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe STARTED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe PASSED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testProduceConsumeWithPrefixedAcls STARTED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testProduceConsumeWithPrefixedAcls PASSED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testProduceConsumeViaAssign STARTED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testProduceConsumeViaAssign PASSED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign STARTED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign PASSED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testProduceConsumeTopicAutoCreateTopicCreateAcl STARTED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testProduceConsumeTopicAutoCreateTopicCreateAcl PASSED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testProduceConsumeWithWildcardAcls STARTED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testProduceConsumeWithWildcardAcls PASSED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe STARTED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe PASSED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign STARTED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign PASSED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > testNoGroupAcl STARTED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > testNoGroupAcl PASSED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testNoProduceWithDescribeAcl STARTED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testNoProduceWithDescribeAcl PASSED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl STARTED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl PASSED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe STARTED

kafka.api.SaslOAuthBearerSslEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe PASSED

kafka.api.AdminClientIntegrationTest > testDescribeReplicaLogDirs STARTED

kafka.api.AdminClientIntegrationTest > testDescribeReplicaLogDirs PASSED

kafka.api.AdminClientIntegrationTest > 
testIncrementalAlterConfigsForLog4jLogLevels STARTED

kafka.api.AdminClientIntegrationTest > 
testIncrementalAlterConfigsForLog4jLogLevels SKIPPED

kafka.api.AdminClientIntegrationTest > 
testIncrementalAlterConfigsForLog4jLogLevelsCannotResetRootLogger STARTED

kafka.api.AdminClientIntegrationTest > 
testIncrementalAlterConfigsForLog4jLogLevelsCannotResetRootLogger SKIPPED

kafka.api.AdminClientIntegrationTest > testInvalidAlterPartitionReassignments 
STARTED

kafka.api.AdminClien

Build failed in Jenkins: kafka-trunk-jdk8 #4019

[Apache Jenkins Server]

See 


Changes:

[cmccabe] MINOR: Rework NewPartitionReassignment public API (#7638)


--
[...truncated 8.29 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldN

[jira] [Created] (KAFKA-9151) KafkaProducer.send should warmup metadata information when application starts

[Tao Chen (Jira)]

Tao Chen created KAFKA-9151:
---

 Summary: KafkaProducer.send should warmup metadata information 
when application starts
 Key: KAFKA-9151
 URL: https://issues.apache.org/jira/browse/KAFKA-9151
 Project: Kafka
  Issue Type: Improvement
  Components: producer 
Reporter: Tao Chen


When application restarts, the performance of KafkaProducer.send is slow due to 
metadata not available. 

We know that it is an old topic that whether should wait for metadata update or 
not. 

"Some user may still want to wait for a configurable amount of time on 
producer.send() if the queue is full instead of dropping messages immedidately. 
Users who want complete non-blocking producer.send() can set max.block.ms to 0."

If max.block.ms is to 0, these messages can not be send successfully due to 
metadata not available. We have to restore these messages and try again when 
metadata is available, which brings much extra effect.  

 

Is it possible to provide a startup hook for users to warmup the metadata when 
application starts. Only after the metadata is available, application starts to 
work accordingly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9152) Improve Sensor Retrieval

[Bruno Cadonna (Jira)]

Bruno Cadonna created KAFKA-9152:


 Summary: Improve Sensor Retrieval 
 Key: KAFKA-9152
 URL: https://issues.apache.org/jira/browse/KAFKA-9152
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Bruno Cadonna


This ticket shall improve two aspects of the retrieval of sensors:

1. Currently, when a sensor is retrieved with {{*Metrics.*Sensor()}} (e.g. 
{{ThreadMetrics.createTaskSensor()}}) after it was created with the same method 
{{*Metrics.*Sensor()}}, the sensor is added again to the corresponding queue in 
{{*Sensors}} (e.g. {{threadLevelSensors}}) in {{StreamsMetricsImpl}}. Those 
queues are used to remove the sensors when {{removeAll*LevelSensors()}} is 
called. Having multiple times the same sensors in this queue is not an issue 
from a correctness point of view. However, it would reduce the footprint to 
only store a sensor once in those queues.

2. When a sensor is retrieved, the current code attempts to create a new sensor 
and to add to it again the corresponding metrics. This could be avoided.
 
Both aspects could be improved by checking whether a sensor already exists by 
calling {{getSensor()}} on the {{Metrics}} object and checking the return value.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-9079) System Test Failure: TransactionsTest

[Manikumar (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-9079.
--
Fix Version/s: (was: 2.5.0)
   2.4.0
   Resolution: Fixed

Issue resolved by pull request 7653
[https://github.com/apache/kafka/pull/7653]

> System Test Failure: TransactionsTest
> -
>
> Key: KAFKA-9079
> URL: https://issues.apache.org/jira/browse/KAFKA-9079
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Manikumar
>Priority: Major
> Fix For: 2.4.0
>
>
> TransactionsTest tests are failing on 2.4 and trunk.
> http://confluent-kafka-2-4-system-test-results.s3-us-west-2.amazonaws.com/2019-10-21--001.1571716233--confluentinc--2.4--cb4944f/report.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-409: Allow creating under-replicated topics and partitions

[Mickael Maison]

Hi,

Thanks Colin for the feedback. Edo and I have updated the KIP
accordingly. Can you take another look?


On Tue, Oct 22, 2019 at 12:20 AM Colin McCabe  wrote:
>
> Hi Mickael,
>
> We don't have any official way for brokers to join the cluster other than 
> showing up and registering themselves in ZK.  Similarly, we don't have any 
> way of removing brokers from the cluster other than simply removing them and 
> removing their znodes from ZooKeeper.
>
> If we wanted to change this, it seems like it would be a really big step.  We 
> would need public, stable APIs for both of these things.  Or at least for the 
> removal thing, which is currently automatic and doesn't require any action on 
> the part of the administrator. Administrators would have to be retrained to 
> do this whenever shrinking the cluster.
>  We cannot tell people to modify ZK directly for this.
>
> To be honest, I don't think reworking broker registration is worth it for 
> this change.  I think we could pretty easily have placeholder values for the 
> missing replicas like -1, -2, -3, etc. and just fill them in whenever a new 
> broker comes online.  This may be slightly more complex to implement, but it 
> greatly simplifies what users have to do.
>
> It is true that filling in -1, -2, -3, etc. will not preserve rack placement 
> information.  But this is kind of a more general problem that we should 
> probably solve separately.  After placement, a lot of placement information 
> disappears and is not accessible to reassignment.  Since reassignment is 
> becoming more and more important, we should make an effort to preserve this 
> information.  Since that would be a big change, it's probably best to do 
> separately, however.
>
> The "rejected alternatives" section says that adding an option to 
> CreateTopicsRequest to allow users to opt-in to the new behavior "felt too 
> complex."  But I think this could use a little clarification.  Adding a new 
> boolean to the createTopics command is actually fairly simple from the 
> perspective of a developer.  But it adds another thing for end-users to think 
> about when using the software.  It's also not clear how many users would take 
> advantage of this.  I think that's the reason people were not in favor of it, 
> not a general feeling of complexity.  Adding more configuration options is 
> often simple to implement, and making things "just work" is often a little 
> more complex.  But we should prefer the latter, most of the time at least.  I 
> think this is what you meant here, but it would be good to clarify.
>
> "Rejected alternatives" also talks about an error code and an error message 
> when the replication is not up to full strength.  But this was removed, 
> right?  We should clarify that no error code is returned in this case, and 
> the CreateTopicsResponse returns the true number of replicas that was 
> created, in case the client is interested in this information.  Returning an 
> error code would certainly cause problems for a lot of users, who use 
> all().get() to verify that all the topics have been successfully created.
>
> best,
> Colin
>
>
> On Mon, Oct 21, 2019, at 09:50, Mickael Maison wrote:
> > Thanks Stanislav and Colin for the feedback.
> >
> > I've updated the KIP to make it simpler.
> > It's not updating the CreateTopics/CreatePartitions RPCs anymore. I've
> > kept the broker setting so admins can keep the current behaviour but
> > simplified it to be either enabled or disabled.
> >
> > I've also kept the observed_brokers nodes in Zookeeper. I can't think
> > of a better alternative to keep track of the expected brokers. The
> > other option would be to perform the extra replica creation
> > asynchronously (driven by the controller when a broker joins the
> > cluster) but that feels a lot more complicated for this specific use
> > case.
> >
> > I've also made it explicit that at least "min.insync.replicas" brokers
> > have to be online to allow topic/partition creation.
> >
> > Thanks
> >
> > On Mon, Mar 25, 2019 at 1:17 PM Mickael Maison  
> > wrote:
> > >
> > > Thanks Colin for the feedback.
> > >
> > > The idea was to allow both users and administrator to decide if they
> > > wanted to opt-in and if so under what conditions.
> > >
> > > Maybe we could do something simpler and just allow the creation if at
> > > least min-in-sync replicas are available? That should not require
> > > changes to the protocol and while this might not cover all possible
> > > use cases, that would still cover the use cases we've listed in the
> > > KIP. That would also tie in with existing semantics/guarantees
> > > (min-in-sync).
> > >
> > > Thanks
> > >
> > > On Tue, Feb 26, 2019 at 5:40 PM Colin McCabe  wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > I don't think adding CREATED_UNDER_REPLICATED as an error code makes 
> > > > sense.  It is not an error condition, as described here.
> > > >
> > > > > Updates to the Decommissioning brokers section in the documentation

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

Hi Jun,

>21. Could you elaborate a bit why the positions in remote segment is
different from the local one? I thought that they are identical copies.

They may not always be the same. Let me take an example here. If
remote storage is enabled with encryption then those local positions
may not be the same as the positions copied to remote storage.

Thanks,
Satish.


On Tue, Nov 5, 2019 at 3:46 AM Jun Rao  wrote:
>
> Hi, Satish,
>
> Thanks for the response.
>
> 21. Could you elaborate a bit why the positions in remote segment is
> different from the local one? I thought that they are identical copies.
>
> Jun
>
>
> On Fri, Nov 1, 2019 at 4:26 AM Satish Duggana 
> wrote:
>
> > Hi Jun,
> > Thanks for looking into the updated KIP and clarifying our earlier queries.
> >
> > >20. It's fine to keep the HDFS binding temporarily in the PR. We just need
> > to remove it before it's merged to trunk. As Victor mentioned, we can
> > provide a reference implementation based on a mocked version of remote
> > storage.
> >
> > Sure, sounds good.
> >
> > >21. I am not sure that I understood the need for RemoteLogIndexEntry and
> > its relationship with RemoteLogSegmentInfo. It seems
> > that RemoteLogIndexEntry are offset index entries pointing to record
> > batches inside a segment. That seems to be the same as the .index file?
> >
> > That is a good point. `RemoteLogManager` does not put a restriction on
> > `RemoteStorageManager(RSM)` for maintaining positions in the remote
> > segment same as the local segments or keeping a correlation between
> > local segment's positions to the remote segment positions. RSM gives
> > back the respective entries for a given log segment, call RSM to fetch
> > the data by giving the respective entry. This allows RSM to have
> > better control in managing the given log segments.
> >
> > Thanks,
> > Satish.
> >
> > On Fri, Nov 1, 2019 at 2:28 AM Jun Rao  wrote:
> > >
> > > Hi, Harsha,
> > >
> > > I am still looking at the KIP and the PR. A couple of quick
> > > comments/questions.
> > >
> > > 20. It's fine to keep the HDFS binding temporarily in the PR. We just
> > need
> > > to remove it before it's merged to trunk. As Victor mentioned, we can
> > > provide a reference implementation based on a mocked version of remote
> > > storage.
> > >
> > > 21. I am not sure that I understood the need for RemoteLogIndexEntry and
> > > its relationship with RemoteLogSegmentInfo. It seems
> > > that RemoteLogIndexEntry are offset index entries pointing to record
> > > batches inside a segment. That seems to be the same as the .index file?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Oct 28, 2019 at 9:11 PM Satish Duggana  > >
> > > wrote:
> > >
> > > > Hi Viktor,
> > > > >1. Can we allow RLM Followers to serve read requests? After all
> > segments
> > > > on
> > > > the cold storage are closed ones, no modification is allowed. Besides
> > > > KIP-392 (
> > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica
> > > > )
> > > > would introduce follower fetching too, so I think it would be nice to
> > > > prepare RLM for this as well.
> > > >
> > > > That is a good point. We plan to support fetching remote storage from
> > > > followers too. Current code in the PR work fine for this scenario
> > > > though there may be some edge cases to be handled. We have not yet
> > > > tested this scenario.
> > > >
> > > > >2. I think the remote.log.storage.enable config is redundant. By
> > > > specifying
> > > > remote.log.storage.manager.class.name one already declares that they
> > want
> > > > to use remote storage. Would it make sense to remove
> > > > the remote.log.storage.enable config?
> > > >
> > > > I do not think it is really needed. `remote.log.storage.enable`
> > > > property can be removed.
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > > >
> > > > On Thu, Oct 24, 2019 at 2:46 PM Viktor Somogyi-Vass
> > > >  wrote:
> > > > >
> > > > > Hi Harsha,
> > > > >
> > > > > A couple more questions:
> > > > > 1. Can we allow RLM Followers to serve read requests? After all
> > segments
> > > > on
> > > > > the cold storage are closed ones, no modification is allowed. Besides
> > > > > KIP-392 (
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica
> > > > )
> > > > > would introduce follower fetching too, so I think it would be nice to
> > > > > prepare RLM for this as well.
> > > > > 2. I think the remote.log.storage.enable config is redundant. By
> > > > specifying
> > > > > remote.log.storage.manager.class.name one already declares that they
> > > > want
> > > > > to use remote storage. Would it make sense to remove
> > > > > the remote.log.storage.enable config?
> > > > >
> > > > > Thanks,
> > > > > Viktor
> > > > >
> > > > >
> > > > > On Thu, Oct 24, 2019 at 10:37 AM Viktor Somogyi-Vass <
> > > > > viktorsomo...@gmail.com> wrote:
> > > > >
> > > > > >

RE: [VOTE] KIP-280: Enhanced log compaction

[Senthilnathan Muthusamy]

Thanks Matthias! 

Received 2 +1 binding... looking for one more +1 binding !

Regards,
Senthil

-Original Message-
From: Matthias J. Sax  
Sent: Wednesday, November 6, 2019 12:10 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-280: Enhanced log compaction

+1 (binding)

On 11/5/19 11:44 AM, Senthilnathan Muthusamy wrote:
> Thanks Gouzhang and I have made a note in the JIRA item to update the wiki.
> 
> Till now got 1 +1 binding... waiting for 2 more +1 binding... thnx!
> 
> Regards,
> Senthil
> 
> -Original Message-
> From: Guozhang Wang 
> Sent: Monday, November 4, 2019 11:01 AM
> To: dev 
> Subject: Re: [VOTE] KIP-280: Enhanced log compaction
> 
> I only have one minor comment on the DISCUSS thread, otherwise I'm +1 
> (binding).
> 
> On Mon, Nov 4, 2019 at 9:53 AM Senthilnathan Muthusamy 
>  wrote:
> 
>> Hi all,
>>
>> I would like to start the vote on the updated KIP-280: Enhanced log 
>> compaction. Thanks to Guozhang, Matthias & Tom for the valuable 
>> feedback on the discussion thread...
>>
>> KIP:
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
>> k 
>> i.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-280%253A%2BEnhanced
>> %
>> 2Blog%2Bcompaction&data=02%7C01%7Csenthilm%40microsoft.com%7Ca8ca
>> 2
>> 5d3f1894d0d271f08d7615966d3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C
>> 0 
>> %7C637085005478393331&sdata=qrttmbYi2Ea4qfcF5qKVbn7CaYwmvRylO85df
>> j
>> IY6pI%3D&reserved=0
>>
>> JIRA: 
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fiss
>> u 
>> es.apache.org%2Fjira%2Fbrowse%2FKAFKA-7061&data=02%7C01%7Csenthil
>> m 
>> %40microsoft.com%7Ca8ca25d3f1894d0d271f08d7615966d3%7C72f988bf86f141a
>> f
>> 91ab2d7cd011db47%7C1%7C0%7C637085005478393331&sdata=7c%2BzF3XRRz%
>> 2
>> BijyyjBRntP6ZMWqnyzy4BEE8rqnZaF1s%3D&reserved=0
>>
>> Thanks,
>> Senthil
>>
> 
> 
> --
> -- Guozhang
> 

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

[Viktor Somogyi-Vass]

Hi Elmahdi,

I've added the JIRA to the KIP (and also below) where you can track the
progress (but more subtask will come as the current ones don't represent
the full work to be done).
https://issues.apache.org/jira/browse/KAFKA-9119

Viktor

On Tue, Nov 5, 2019 at 5:15 PM Elmahdi FRID  wrote:

> Hello Folks any status abbout this kip and it's possible to test this use
> case ?
>
> On 2019/08/01 21:04:46, "Colin McCabe"  wrote:
> > Hi all,
> >
> > I've written a KIP about removing ZooKeeper from Kafka.  Please take a
> look and let me know what you think:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
> >
> > cheers,
> > Colin
> >
>

[jira] [Created] (KAFKA-9153) Kafka brokers randomly crash (SIGSEGV due to kafka errors)

[Tristan (Jira)]

Tristan created KAFKA-9153:
--

 Summary: Kafka brokers randomly crash (SIGSEGV due to kafka errors)
 Key: KAFKA-9153
 URL: https://issues.apache.org/jira/browse/KAFKA-9153
 Project: Kafka
  Issue Type: Bug
Reporter: Tristan


We upgraded our kafka brokers on many clusters to 2.1.1 a few month ago and did 
not encountered any issues until a few weeks ago.

Now we have kafka service stopping with errors time to time, and we couldn't 
establish correlation with any particular events, messages, or cluster 
operations.

Here is the encountered errors :

kafka logs : 
{code:java}
[2019-11-06 11:19:36,177] ERROR [KafkaApi-5] Error while responding to offset 
request (kafka.server.KafkaApis)scala.MatchError: nullat 
kafka.cluster.Partition.$anonfun$fetchOffsetForTimestamp$1(Partition.scala:813) 
   at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)at 
kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:257)at 
kafka.cluster.Partition.fetchOffsetForTimestamp(Partition.scala:809)at 
kafka.server.ReplicaManager.fetchOffsetForTimestamp(ReplicaManager.scala:784)   
 at 
kafka.server.KafkaApis.$anonfun$handleListOffsetRequestV1AndAbove$3(KafkaApis.scala:833)
at 
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
at scala.collection.Iterator.foreach(Iterator.scala:937)at 
scala.collection.Iterator.foreach$(Iterator.scala:937)at 
scala.collection.AbstractIterator.foreach(Iterator.scala:1425)at 
scala.collection.IterableLike.foreach(IterableLike.scala:70)at 
scala.collection.IterableLike.foreach$(IterableLike.scala:69)at 
scala.collection.AbstractIterable.foreach(Iterable.scala:54)at 
scala.collection.TraversableLike.map(TraversableLike.scala:233)at 
scala.collection.TraversableLike.map$(TraversableLike.scala:226)at 
scala.collection.AbstractTraversable.map(Traversable.scala:104)at 
kafka.server.KafkaApis.handleListOffsetRequestV1AndAbove(KafkaApis.scala:813)   
 at kafka.server.KafkaApis.handleListOffsetRequest(KafkaApis.scala:753)at 
kafka.server.KafkaApis.handle(KafkaApis.scala:108)at 
kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:69)at 
java.lang.Thread.run(Thread.java:745)[2019-11-06 11:19:36,178] ERROR 
[KafkaApi-5] Error while responding to offset request 
(kafka.server.KafkaApis)scala.MatchError: nullat 
kafka.cluster.Partition.$anonfun$fetchOffsetForTimestamp$1(Partition.scala:813) 
   at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)at 
kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:257)at 
kafka.cluster.Partition.fetchOffsetForTimestamp(Partition.scala:809)at 
kafka.server.ReplicaManager.fetchOffsetForTimestamp(ReplicaManager.scala:784)   
 at 
kafka.server.KafkaApis.$anonfun$handleListOffsetRequestV1AndAbove$3(KafkaApis.scala:833)
at 
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
at scala.collection.Iterator.foreach(Iterator.scala:937)at 
scala.collection.Iterator.foreach$(Iterator.scala:937)at 
scala.collection.AbstractIterator.foreach(Iterator.scala:1425)at 
scala.collection.IterableLike.foreach(IterableLike.scala:70)at 
scala.collection.IterableLike.foreach$(IterableLike.scala:69)at 
scala.collection.AbstractIterable.foreach(Iterable.scala:54)at 
scala.collection.TraversableLike.map(TraversableLike.scala:233)at 
scala.collection.TraversableLike.map$(TraversableLike.scala:226)at 
scala.collection.AbstractTraversable.map(Traversable.scala:104)at 
kafka.server.KafkaApis.handleListOffsetRequestV1AndAbove(KafkaApis.scala:813)   
 at kafka.server.KafkaApis.handleListOffsetRequest(KafkaApis.scala:753)at 
kafka.server.KafkaApis.handle(KafkaApis.scala:108)at 
kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:69)at 
java.lang.Thread.run(Thread.java:745)[2019-11-06 11:19:36,302] ERROR 
[KafkaApi-5] Error while responding to offset request 
(kafka.server.KafkaApis)scala.MatchError: null{code}
with 106 consecutive occurences of this stacktrace.

and this error showing with journalctl -u kafka, just after latest stacktrace 
in kafka.log :
{code:java}
Oct 15 14:34:32 kafka-5 systemd[1]: Started kafka daemon. Nov 06 11:19:50 
kafka-5 bash[15874]: # Nov 06 11:19:50 kafka-5 bash[15874]: # A fatal error has 
been detected by the Java Runtime Environment: Nov 06 11:19:50 kafka-5 
bash[15874]: # Nov 06 11:19:50 kafka-5 bash[15874]: # SIGSEGV (0xb) at 
pc=0x7f88f31b8df0, pid=15874, tid=140216143816448 Nov 06 11:19:50 kafka-5 
bash[15874]: # Nov 06 11:19:50 kafka-5 bash[15874]: # JRE version: Java(TM) SE 
Runtime Environment (8.0_91-b14) (build 1.8.0_91-b14) Nov 06 11:19:50 kafka-5 
bash[15874]: # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.91-b14 mixed mode 
linux-amd64 compressed oops) Nov 06 11:19:50 kafka-5 bash[15874]: # Problematic 
frame: Nov 06 11:19:50 kafka-5 bash[15874]: # J 1498

[jira] [Created] (KAFKA-9154) ProducerId generation should be managed by the Controller

[Viktor Somogyi-Vass (Jira)]

Viktor Somogyi-Vass created KAFKA-9154:
--

 Summary: ProducerId generation should be managed by the Controller
 Key: KAFKA-9154
 URL: https://issues.apache.org/jira/browse/KAFKA-9154
 Project: Kafka
  Issue Type: Sub-task
  Components: core
Reporter: Viktor Somogyi-Vass


Currently producerIds are maintained in Zookeeper but in the future we'd like 
them to be managed by the controller quorum in an internal topic.
The reason for storing this in Zookeeper was that this must be unique across 
the cluster. In this task it should be refactored such that the 
TransactionManager turns to the Controller for a ProducerId which connects to 
Zookeeper to acquire this ID. Since ZK is the single source of truth and the 
PID won't be cached anywhere it should be safe (just one extra hop added).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-501 Avoid out-of-sync or offline partitions when follower fetch requests not processed in time

[Satish Duggana]

Hi Dhruvil,
Thanks for looking into the KIP.

10. I have an initial sketch of the KIP-500 in commit[a] which
discusses tracking the pending fetch requests. Tracking is not done in
Partition#readRecords because if it takes longer in reading any of the
partitions then we do not want any of the replicas of this fetch
request to go out of sync.

11. I think `Replica` class should be thread-safe to handle the remote
scenario of concurrent requests running for a follower replica. Or I
may be missing something here. This is a separate issue from KIP-500.
I will file a separate JIRA to discuss that issue.

a - 
https://github.com/satishd/kafka/commit/c69b525abe8f6aad5059236076a003cdec4c4eb7

Thanks,
Satish.

On Tue, Oct 29, 2019 at 10:57 AM Dhruvil Shah  wrote:
>
> Hi Satish,
>
> Thanks for the KIP, those seems very useful. Could you elaborate on how
> pending fetch requests are tracked?
>
> Thanks,
> Dhruvil
>
> On Mon, Oct 28, 2019 at 9:43 PM Satish Duggana 
> wrote:
>
> > Hi All,
> > I wrote a short KIP about avoiding out-of-sync or offline partitions
> > when follower fetch requests are not processed in time by the leader
> > replica.
> > KIP-501 is located at https://s.apache.org/jhbpn
> >
> > Please take a look, I would like to hear your feedback and suggestions.
> >
> > JIRA: https://issues.apache.org/jira/browse/KAFKA-8733
> >
> > Thanks,
> > Satish.
> >

Re: [VOTE] KIP-150 - Kafka-Streams Cogroup

[Bill Bejeck]

Thanks for the KIP.
+1 (binding)

-Bill


On Wed, Nov 6, 2019 at 1:22 AM Matthias J. Sax 
wrote:

> +1 (binding)
>
>
> On 10/31/19 10:52 AM, Walker Carlson wrote:
> > Hello all,
> >
> > I'd like to call a vote on the updated KIP-150: Kafka-Streams Cogroup
> > found here
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+Kafka-Streams+Cogroup
> >
> >
> > Thanks,
> > Walker
> >
>
>

Build failed in Jenkins: kafka-2.4-jdk8 #54

[Apache Jenkins Server]

See 


Changes:

[manikumar] KAFKA-9079: Fix reset logic in transactional message copier


--
[...truncated 5.08 MB...]
kafka.server.KafkaConfigTest > testLogRollTimeNoConfigProvided PASSED

kafka.server.KafkaConfigTest > testInvalidInterBrokerSecurityProtocol STARTED

kafka.server.KafkaConfigTest > testInvalidInterBrokerSecurityProtocol PASSED

kafka.server.KafkaConfigTest > testAdvertiseDefaults STARTED

kafka.server.KafkaConfigTest > testAdvertiseDefaults PASSED

kafka.server.KafkaConfigTest > testBadListenerProtocol STARTED

kafka.server.KafkaConfigTest > testBadListenerProtocol PASSED

kafka.server.KafkaConfigTest > testListenerDefaults STARTED

kafka.server.KafkaConfigTest > testListenerDefaults PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndHoursProvided 
STARTED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndHoursProvided 
PASSED

kafka.server.KafkaConfigTest > testUncleanElectionDisabled STARTED

kafka.server.KafkaConfigTest > testUncleanElectionDisabled PASSED

kafka.server.KafkaConfigTest > 
testListenerNameMissingFromListenerSecurityProtocolMap STARTED

kafka.server.KafkaConfigTest > 
testListenerNameMissingFromListenerSecurityProtocolMap PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeNoConfigProvided STARTED

kafka.server.KafkaConfigTest > testLogRetentionTimeNoConfigProvided PASSED

kafka.server.KafkaConfigTest > testCaseInsensitiveListenerProtocol STARTED

kafka.server.KafkaConfigTest > testCaseInsensitiveListenerProtocol PASSED

kafka.server.KafkaConfigTest > testListenerAndAdvertisedListenerNames STARTED

kafka.server.KafkaConfigTest > testListenerAndAdvertisedListenerNames PASSED

kafka.server.KafkaConfigTest > testNonroutableAdvertisedListeners STARTED

kafka.server.KafkaConfigTest > testNonroutableAdvertisedListeners PASSED

kafka.server.KafkaConfigTest > 
testInterBrokerListenerNameAndSecurityProtocolSet STARTED

kafka.server.KafkaConfigTest > 
testInterBrokerListenerNameAndSecurityProtocolSet PASSED

kafka.server.KafkaConfigTest > testFromPropsInvalid STARTED

kafka.server.KafkaConfigTest > testFromPropsInvalid PASSED

kafka.server.KafkaConfigTest > testInvalidCompressionType STARTED

kafka.server.KafkaConfigTest > testInvalidCompressionType PASSED

kafka.server.KafkaConfigTest > testAdvertiseHostNameDefault STARTED

kafka.server.KafkaConfigTest > testAdvertiseHostNameDefault PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeMinutesProvided STARTED

kafka.server.KafkaConfigTest > testLogRetentionTimeMinutesProvided PASSED

kafka.server.KafkaConfigTest > testValidCompressionType STARTED

kafka.server.KafkaConfigTest > testValidCompressionType PASSED

kafka.server.KafkaConfigTest > testUncleanElectionInvalid STARTED

kafka.server.KafkaConfigTest > testUncleanElectionInvalid PASSED

kafka.server.KafkaConfigTest > testListenerNamesWithAdvertisedListenerUnset 
STARTED

kafka.server.KafkaConfigTest > testListenerNamesWithAdvertisedListenerUnset 
PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndMsProvided 
STARTED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndMsProvided 
PASSED

kafka.server.KafkaConfigTest > testLogRollTimeMsProvided STARTED

kafka.server.KafkaConfigTest > testLogRollTimeMsProvided PASSED

kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault STARTED

kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault PASSED

kafka.server.KafkaConfigTest > testInvalidAdvertisedListenersProtocol STARTED

kafka.server.KafkaConfigTest > testInvalidAdvertisedListenersProtocol PASSED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled STARTED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled PASSED

kafka.server.KafkaConfigTest > testInterBrokerVersionMessageFormatCompatibility 
STARTED

kafka.server.KafkaConfigTest > testInterBrokerVersionMessageFormatCompatibility 
PASSED

kafka.server.KafkaConfigTest > testAdvertisePortDefault STARTED

kafka.server.KafkaConfigTest > testAdvertisePortDefault PASSED

kafka.server.KafkaConfigTest > testVersionConfiguration STARTED

kafka.server.KafkaConfigTest > testVersionConfiguration PASSED

kafka.server.KafkaConfigTest > testEqualAdvertisedListenersProtocol STARTED

kafka.server.KafkaConfigTest > testEqualAdvertisedListenersProtocol PASSED

kafka.server.IsrExpirationTest > testIsrExpirationForSlowFollowers STARTED

kafka.server.IsrExpirationTest > testIsrExpirationForSlowFollowers PASSED

kafka.server.IsrExpirationTest > testIsrExpirationForCaughtUpFollowers STARTED

kafka.server.IsrExpirationTest > testIsrExpirationForCaughtUpFollowers PASSED

kafka.server.IsrExpirationTest > testIsrExpirationForStuckFollowers STARTED

kafka.server.IsrExpirationTest > testIsrExpirationForStuckFollowers PASSED

kafka.server.IsrExpirationTest > testIsrExpirationIfNoFetchRequestMade STARTED

kafka.server.IsrExpira

Re: [DISCUSS] KIP-513: Distinguish between Key and Value serdes in scala wrapper library for kafka streams

[John Roesler]

Hey Mykhailo,

I just wanted to let you know that I'm looking at your proposal, but
it'll take me a little while to re-activate the Scala part of my
brain.

For everyone's benefit, here's the link to the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-513%3A+Distinguish+between+Key+and+Value+serdes+in+scala+wrapper+library+for+kafka+streams

Thanks for starting this discussion!
-John

On Sun, Oct 20, 2019 at 7:11 AM Михаил Ерёменко  wrote:
>
> Hi!
>
> I would like to start discussion.
>
> Regards,
> Mykhailo Yeromenko

gradle uploadArchives

[Carl Graving]

I was trying to use this gradle task and it works, but the files are all
having a timestamp appended to the version. Places like maven central don't
have these timestamps. Is there an option to not have the timestamps
appended when running this task?

Thanks,
Carl

Re: [VOTE] KIP-150 - Kafka-Streams Cogroup

[Guozhang Wang]

+1 (binding),

Thanks Walker!

Guozhang

On Wed, Nov 6, 2019 at 8:41 AM Bill Bejeck  wrote:

> Thanks for the KIP.
> +1 (binding)
>
> -Bill
>
>
> On Wed, Nov 6, 2019 at 1:22 AM Matthias J. Sax 
> wrote:
>
> > +1 (binding)
> >
> >
> > On 10/31/19 10:52 AM, Walker Carlson wrote:
> > > Hello all,
> > >
> > > I'd like to call a vote on the updated KIP-150: Kafka-Streams Cogroup
> > > found here
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+Kafka-Streams+Cogroup
> > >
> > >
> > > Thanks,
> > > Walker
> > >
> >
> >
>


-- 
-- Guozhang

[jira] [Resolved] (KAFKA-9140) Consumer gets stuck rejoining the group indefinitely

[Manikumar (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-9140.
--
  Assignee: Guozhang Wang
Resolution: Fixed

> Consumer gets stuck rejoining the group indefinitely
> 
>
> Key: KAFKA-9140
> URL: https://issues.apache.org/jira/browse/KAFKA-9140
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Affects Versions: 2.4.0
>Reporter: Sophie Blee-Goldman
>Assignee: Guozhang Wang
>Priority: Blocker
> Fix For: 2.4.0
>
> Attachments: debug.tgz, info.tgz, kafka-data-logs-1.tgz, 
> kafka-data-logs-2.tgz, server-start-stdout-stderr.log.tgz, streams.log.tgz
>
>
> There seems to be a race condition that is now causing a rejoining member to 
> potentially get stuck infinitely initiating a rejoin. The relevant client 
> logs are attached (streams.log.tgz; all others attachments are broker logs), 
> but basically it repeats this message (and nothing else) continuously until 
> killed/shutdown:
>  
> {code:java}
> [2019-11-05 01:53:54,699] INFO [Consumer 
> clientId=StreamsUpgradeTest-a4c1cff8-7883-49cd-82da-d2cdfc33a2f0-StreamThread-1-consumer,
>  groupId=StreamsUpgradeTest] Generation data was cleared by heartbeat thread. 
> Initiating rejoin. 
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
> {code}
>  
> The message that appears was added as part of the bugfix ([PR 
> 7460|https://github.com/apache/kafka/pull/7460]) for this related race 
> condition: KAFKA-8104.
> This issue was uncovered by the Streams version probing upgrade test, which 
> fails with a varying frequency. Here is the rate of failures for different 
> system test runs so far:
> trunk (cooperative): 1/1 and 2/10 failures
> 2.4 (cooperative) : 0/10 and 1/15 failures
> trunk (eager): 0/10 failures
> I've kicked off some high-repeat runs to complete overnight and hopefully 
> shed more light.
> Note that I have also kicked off runs of both 2.4 and trunk with the PR for 
> KAFKA-8104 reverted. Both of them saw 2/10 failures, due to hitting the bug 
> that was fixed by [PR 7460|https://github.com/apache/kafka/pull/7460]. It is 
> therefore unclear whether [PR 7460|https://github.com/apache/kafka/pull/7460] 
> introduced another or a new race condition/bug, or merely uncovered an 
> existing one that previously would have first failed due to KAFKA-8104.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is back to normal : kafka-trunk-jdk11 #935

[Apache Jenkins Server]

See 

Jenkins build is back to normal : kafka-trunk-jdk8 #4020

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-150 - Kafka-Streams Cogroup

[Walker Carlson]

Thanks everyone for the votes!

The KIP has been accepted, with 3 binding votes from
Matthias, Bill and Guozhang.

Walker

On Wed, Nov 6, 2019 at 9:41 AM Guozhang Wang  wrote:

> +1 (binding),
>
> Thanks Walker!
>
> Guozhang
>
> On Wed, Nov 6, 2019 at 8:41 AM Bill Bejeck  wrote:
>
> > Thanks for the KIP.
> > +1 (binding)
> >
> > -Bill
> >
> >
> > On Wed, Nov 6, 2019 at 1:22 AM Matthias J. Sax 
> > wrote:
> >
> > > +1 (binding)
> > >
> > >
> > > On 10/31/19 10:52 AM, Walker Carlson wrote:
> > > > Hello all,
> > > >
> > > > I'd like to call a vote on the updated KIP-150: Kafka-Streams Cogroup
> > > > found here
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+Kafka-Streams+Cogroup
> > > >
> > > >
> > > > Thanks,
> > > > Walker
> > > >
> > >
> > >
> >
>
>
> --
> -- Guozhang
>

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

[Vinoth Chandar]

+1 to John, suggestion on Duration/Instant and dropping the API to fetch
all store's lags. However, I do think we need to return lags per topic
partition. So not sure if single return value would work? We need some new
class that holds a TopicPartition and Duration/Instant variables together?

10) Because we needed to return the topicPartition the key belongs to, in
order to correlate with the lag information from the other set of APIs.
Otherwise, we don't know which topic partition's lag estimate to use. We
tried to illustrate this on the example code. StreamsMetadata is simply
capturing state of a streams host/instance, where as TopicPartition depends
on the key passed in. This is a side effect of our decision to decouple lag
based filtering on the metadata apis.

20) Goes back to the previous point. We needed to return information that
is key specific, at which point it seemed natural for the KeyQueryMetadata
to contain active, standby, topic partition for that key. If we merely
returned a standbyMetadataForKey() -> Collection standby,
an active metadataForKey() -> StreamsMetadata, and new
getTopicPartition(key) -> topicPartition object back to the caller, then
arguably you could do the same kind of correlation. IMO having a the
KeyQueryMetadata class to encapsulate all this is a friendlier API.
 allStandbyMetadata() and allStandbyMetadataForStore() are just counter
parts for metadataForStore() and allMetadata() that we introduce mostly for
consistent API semantics. (their presence implicitly could help denote
metadataForStore() is for active instances. Happy to drop them if their
utility is not clear)

30) This would assume we refresh all the standby lag information every
time we query for that StreamsMetadata for a specific store? For time based
lag, this will involve fetching the tail kafka record at once from multiple
kafka topic partitions? I would prefer not to couple them like this and
have the ability to make granular store (or even topic partition level)
fetches for lag information.

32) I actually prefer John's suggestion to let the application drive the
lag fetches/updation and not have flags as the KIP current points to. Are
you reexamining that position?

On fetching lag information, +1 we could do this much more efficiently with
a broker changes. Given I don't yet have a burning need for the time based
lag, I think we can sequence the APIs such that the offset based ones are
implemented first, while we have a broker side change?
Given we decoupled the offset and time based lag API, I am willing to drop
the time based lag functionality (since its not needed right away for my
use-case). @navinder . thoughts?


On Tue, Nov 5, 2019 at 11:10 PM Matthias J. Sax 
wrote:

> Navinder,
>
> thanks for updating the KIP. Couple of follow up questions:
>
>
> (10) Why do we need to introduce the class `KeyQueryMetadata`?
>
> (20) Why do we introduce the two methods `allMetadataForKey()`? Would it
> not be simpler to add `Collection
> standbyMetadataForKey(...)`. This would align with new methods
> `#allStandbyMetadata()` and `#allStandbyMetadataForStore()`?
>
> (30) Why do we need the class `StoreLagInfo` -- it seems simpler to just
> extend `StreamMetadata` with the corresponding attributes and methods
> (of active task, the lag would always be reported as zero)
>
> (32) Via (30) we can avoid the two new methods `#allLagInfo()` and
> `#lagInfoForStore()`, too, reducing public API and making it simpler to
> use the feature.
>
> Btw: If we make `StreamMetadata` thread safe, the lag information can be
> updated in the background without the need that the application
> refreshes its metadata. Hence, the user can get active and/or standby
> metadata once, and only needs to refresh it, if a rebalance happened.
>
>
> About point (4) of the previous thread: I was also thinking about
> when/how to update the time-lag information, and I agree that we should
> not update it for each query.
>
> "How": That we need to fetch the last record is a little bit
> unfortunate, but I don't see any other way without a broker change. One
> issue I still see is with "exactly-once" -- if transaction markers are
> in the topic, the last message is not at offset "endOffset - 1" and as
> multiple transaction markers might be after each other, it's unclear how
> to identify the offset of the last record... Thoughts?
>
> Hence, it might be worth to look into a broker change as a potential
> future improvement. It might be possible that the broker caches the
> latest timestamp per partition to serve this data efficiently, similar
> to `#endOffset()`.
>
> "When": We refresh the end-offset information based on the
> `commit.interval.ms` -- doing it more often is not really useful, as
> state store caches will most likely buffer up all writes to changelogs
> anyway and are only flushed on commit (including a flush of the
> producer). Hence, I would suggest to update the time-lag information
> based on the same strategy in the background

Build failed in Jenkins: kafka-2.4-jdk8 #55

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-9140: Also reset join future when generation was reset in 
order to


--
[...truncated 2.70 MB...]

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.ap

Re: [DISCUSS] KIP-513: Distinguish between Key and Value serdes in scala wrapper library for kafka streams

[John Roesler]

Hi Mykhailo,

I've been mulling over your KIP today. I think that what you're
proposing makes sense. I'm having a little trouble wrapping my head
around the exact problem with the current API, though...

It sounds like you're saying that you want to have different key and
value serdes for the same type, which is not possible in conjunction
with implicitly substituted serdes. (Because there's no way to express
at the type level whether the serde is for keys or not). I guess what
I'm struggling with is why you actually want to have different key and
serdes for the same type. Maybe this is not relevant to the proposal
itself, but I think it would help with the cost/benefit analysis,
especially since the proposal would change approx. every method in the
streams-scala DSL.

I do have some concrete questions, but I'd rather hold onto them until
I feel like I understand the situation.

Thanks again!
-John

On Wed, Nov 6, 2019 at 11:25 AM John Roesler  wrote:
>
> Hey Mykhailo,
>
> I just wanted to let you know that I'm looking at your proposal, but
> it'll take me a little while to re-activate the Scala part of my
> brain.
>
> For everyone's benefit, here's the link to the KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-513%3A+Distinguish+between+Key+and+Value+serdes+in+scala+wrapper+library+for+kafka+streams
>
> Thanks for starting this discussion!
> -John
>
> On Sun, Oct 20, 2019 at 7:11 AM Михаил Ерёменко  wrote:
> >
> > Hi!
> >
> > I would like to start discussion.
> >
> > Regards,
> > Mykhailo Yeromenko

Re: [DISCUSS] KIP-544: Make metrics exposed via JMX configurable

[Xavier Léauté]

>
> Since these configs will work with Kafka's own metrics library, will the
> configs be part of the clients' configurations? It would be good to point
> that out explicitly in the KIP.
>

Those configs are currently only at the broker level. If we feel this is
useful on the client as well, we could submit a similar KIP for the client
side.
I am not sure if the number of metrics is as problematic on the client side
as on the server though.


> Would the regex apply to the whole string? i.e would we be able to match
> parts of the string like `type=`, `name=`, `topic=`, or would it only apply
> to the values?
>

Yes, to keep things simple I decided to apply the regex to the entire JMX
mbean string, since that's typically how a user would refer to those
metrics, regardless of whether it came from KafkaMetrics or YammerMetrics

I've updated the KIP to add some example filters

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Ying Zheng]

21. I am not sure that I understood the need for RemoteLogIndexEntry and
its relationship with RemoteLogSegmentInfo. It seems
that RemoteLogIndexEntry are offset index entries pointing to record
batches inside a segment. That seems to be the same as the .index file?

We do not assume the how the data is stored in the remote storage.
Depends on the implementation, the data of one segment may not necessary be
stored in a single file.
There could be a maximum object / chunk / file size restriction on the
remote storage. So, one Kafka
segment could be saved in multiple chunks in remote storage.

The remote log index also have a larger index interval. The default
interval of the local .index file
(log.index.interval.bytes) is 4KB. In the current HDFS RSM implementation,
the default remote
index interval (hdfs.remote.index.interval.bytes) is 256KB. The
coarse-grained remote index saves
some local disk space. The smaller size also makes it more likely to be
cached in physical memory.




On Thu, Oct 31, 2019 at 1:58 PM Jun Rao  wrote:

> Hi, Harsha,
>
> I am still looking at the KIP and the PR. A couple of quick
> comments/questions.
>
> 20. It's fine to keep the HDFS binding temporarily in the PR. We just need
> to remove it before it's merged to trunk. As Victor mentioned, we can
> provide a reference implementation based on a mocked version of remote
> storage.
>
> 21. I am not sure that I understood the need for RemoteLogIndexEntry and
> its relationship with RemoteLogSegmentInfo. It seems
> that RemoteLogIndexEntry are offset index entries pointing to record
> batches inside a segment. That seems to be the same as the .index file?
>
> Thanks,
>
> Jun
>
> On Mon, Oct 28, 2019 at 9:11 PM Satish Duggana 
> wrote:
>
> > Hi Viktor,
> > >1. Can we allow RLM Followers to serve read requests? After all segments
> > on
> > the cold storage are closed ones, no modification is allowed. Besides
> > KIP-392 (
> >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D392-253A-2BAllow-2Bconsumers-2Bto-2Bfetch-2Bfrom-2Bclosest-2Breplica&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=g7ujYPRBvNrON18SBeCt4g&m=CKNMp77DfMghjYo1JqbWr5jl-DRDBGF2owao5zUXDeE&s=HTPACirRO-wVmOHmGEMlTIAov4szGHn38xrbFbMZK_I&e=
> > )
> > would introduce follower fetching too, so I think it would be nice to
> > prepare RLM for this as well.
> >
> > That is a good point. We plan to support fetching remote storage from
> > followers too. Current code in the PR work fine for this scenario
> > though there may be some edge cases to be handled. We have not yet
> > tested this scenario.
> >
> > >2. I think the remote.log.storage.enable config is redundant. By
> > specifying
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__remote.log.storage.manager.class.name&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=g7ujYPRBvNrON18SBeCt4g&m=CKNMp77DfMghjYo1JqbWr5jl-DRDBGF2owao5zUXDeE&s=QsUunkBFX3dne_4caCiEAbp9xKUPrFx1srwznOR_Sfc&e=
> one already declares that they want
> > to use remote storage. Would it make sense to remove
> > the remote.log.storage.enable config?
> >
> > I do not think it is really needed. `remote.log.storage.enable`
> > property can be removed.
> >
> > Thanks,
> > Satish.
> >
> >
> > On Thu, Oct 24, 2019 at 2:46 PM Viktor Somogyi-Vass
> >  wrote:
> > >
> > > Hi Harsha,
> > >
> > > A couple more questions:
> > > 1. Can we allow RLM Followers to serve read requests? After all
> segments
> > on
> > > the cold storage are closed ones, no modification is allowed. Besides
> > > KIP-392 (
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D392-253A-2BAllow-2Bconsumers-2Bto-2Bfetch-2Bfrom-2Bclosest-2Breplica&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=g7ujYPRBvNrON18SBeCt4g&m=CKNMp77DfMghjYo1JqbWr5jl-DRDBGF2owao5zUXDeE&s=HTPACirRO-wVmOHmGEMlTIAov4szGHn38xrbFbMZK_I&e=
> > )
> > > would introduce follower fetching too, so I think it would be nice to
> > > prepare RLM for this as well.
> > > 2. I think the remote.log.storage.enable config is redundant. By
> > specifying
> > >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__remote.log.storage.manager.class.name&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=g7ujYPRBvNrON18SBeCt4g&m=CKNMp77DfMghjYo1JqbWr5jl-DRDBGF2owao5zUXDeE&s=QsUunkBFX3dne_4caCiEAbp9xKUPrFx1srwznOR_Sfc&e=
> one already declares that they
> > want
> > > to use remote storage. Would it make sense to remove
> > > the remote.log.storage.enable config?
> > >
> > > Thanks,
> > > Viktor
> > >
> > >
> > > On Thu, Oct 24, 2019 at 10:37 AM Viktor Somogyi-Vass <
> > > viktorsomo...@gmail.com> wrote:
> > >
> > > > Hi Jun & Harsha,
> > > >
> > > > I think it would be beneficial to at least provide one simple
> reference
> > > > implementation (file system based?) as we do with connect too.
> > > > That would as a simple example and would help plugin developers to
> > better
> > > > understand the concept and the interfaces.
> > > >
> > > > Best,
> > > > Viktor
> > > >
> 

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Ying Zheng]

On Wed, Nov 6, 2019 at 4:33 PM Ying Zheng  wrote:

> 21. I am not sure that I understood the need for RemoteLogIndexEntry and
> its relationship with RemoteLogSegmentInfo. It seems
> that RemoteLogIndexEntry are offset index entries pointing to record
> batches inside a segment. That seems to be the same as the .index file?
>
> We do not assume the how the data is stored in the remote storage.
> Depends on the implementation, the data of one segment may not necessary
> be stored in a single file.
> There could be a maximum object / chunk / file size restriction on the
> remote storage. So, one Kafka
> segment could be saved in multiple chunks in remote storage.
>
> The remote log index also have a larger index interval. The default
> interval of the local .index file
> (log.index.interval.bytes) is 4KB. In the current HDFS RSM implementation,
> the default remote
> index interval (hdfs.remote.index.interval.bytes) is 256KB. The
> coarse-grained remote index saves
> some local disk space. The smaller size also makes it more likely to be
> cached in physical memory.
>

The remote log index file is also very different from the existing .index
file. With the current design,
one .index file correspond to one segment file. But one remote log index
file can correspond to many
remote segments.

Because only inactive segments can be shipped to remote storage, to be able
to ship log data as soon
as possible, we will roll log segment very fast (e.g. every half hour).
This will lead to a large number of
small segments. If we maintain one remote index file for each remote
segment, we can easily hit some
OS limitations, like the maximum # of open files or the maximum # of
mmapped files.

So, instead of creating a new remote index file, we append
the RemoteLogIndexEntries of multiple
remote segments to one local file. We will roll the remote index file at a
configurable size or time interval.

Re: Subject: [VOTE] 2.2.2 RC2

[Eric Lalonde]

Hello,

In an effort to assist in the verification of release candidates, I have 
authored the following quick-and-dirty utility to help people verify release 
candidate artifacts: 
https://github.com/elalonde/kafka/blob/master/bin/verify-kafka-rc.sh 
 . I have 
executed this script for 2.2.2 rc2 and everything looks good:
- all checksums verify
- all executed gradle commands succeed
- all unit and integration tests pass.

Hope this helps in the release of 2.2.2.

- Eric

> On Nov 5, 2019, at 7:55 AM, Randall Hauch  wrote:
> 
> Thanks, Mickael!
> 
> Anyone else get a chance to validate the 2.2.2 RC2 build? It'd be great to
> get this out the door.
> 
> Randall
> 
> On Tue, Nov 5, 2019 at 6:34 AM Mickael Maison 
> wrote:
> 
>> +1 (non binding)
>> I verified signatures, built it from source, ran unit tests and quickstart
>> 
>> 
>> 
>> On Fri, Oct 25, 2019 at 3:10 PM Randall Hauch  wrote:
>>> 
>>> Hello all, we identified around three dozen bug fixes, including an
>> update
>>> of a third party dependency, and wanted to release a patch release for
>> the
>>> Apache Kafka 2.2.0 release.
>>> 
>>> This is the *second* candidate for release of Apache Kafka 2.2.2. (RC1
>> did
>>> not include a fix for https://issues.apache.org/jira/browse/KAFKA-9053,
>> but
>>> the fix appeared before RC1 was announced so it was easier to just create
>>> RC2.)
>>> 
>>> Check out the release notes for a complete list of the changes in this
>>> release candidate:
>>> https://home.apache.org/~rhauch/kafka-2.2.2-rc2/RELEASE_NOTES.html
>>> 
>>> *** Please download, test and vote by Wednesday, October 30, 9am PT>
>>> 
>>> Kafka's KEYS file containing PGP keys we use to sign the release:
>>> https://kafka.apache.org/KEYS
>>> 
>>> * Release artifacts to be voted upon (source and binary):
>>> https://home.apache.org/~rhauch/kafka-2.2.2-rc2/
>>> 
>>> * Maven artifacts to be voted upon:
>>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>> 
>>> * Javadoc:
>>> https://home.apache.org/~rhauch/kafka-2.2.2-rc2/javadoc/
>>> 
>>> * Tag to be voted upon (off 2.2 branch) is the 2.2.2 tag:
>>> https://github.com/apache/kafka/releases/tag/2.2.2-rc2
>>> 
>>> * Documentation:
>>> https://kafka.apache.org/22/documentation.html
>>> 
>>> * Protocol:
>>> https://kafka.apache.org/22/protocol.html
>>> 
>>> * Successful Jenkins builds for the 2.2 branch:
>>> Unit/integration tests: https://builds.apache.org/job/kafka-2.2-jdk8/1/
>>> System tests:
>>> https://jenkins.confluent.io/job/system-test-kafka/job/2.2/216/
>>> 
>>> /**
>>> 
>>> Thanks,
>>> 
>>> Randall Hauch
>> 

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

>Depends on the implementation, the data of one segment may not necessary be
stored in a single file.
There could be a maximum object / chunk / file size restriction on the
remote storage. So, one Kafka
segment could be saved in multiple chunks in remote storage.

Having one local segment can be stored in multiple files and each file
can have a base position as part of the metadata(like name) of file or
object etc.
File/object name can be --. So
any read request for a position with in that segment can be found by
computing relative position viz `fetchPosition-basePosition`.



On Thu, Nov 7, 2019 at 6:04 AM Ying Zheng  wrote:
>
> 21. I am not sure that I understood the need for RemoteLogIndexEntry and
> its relationship with RemoteLogSegmentInfo. It seems
> that RemoteLogIndexEntry are offset index entries pointing to record
> batches inside a segment. That seems to be the same as the .index file?
>
> We do not assume the how the data is stored in the remote storage.
> Depends on the implementation, the data of one segment may not necessary be
> stored in a single file.
> There could be a maximum object / chunk / file size restriction on the
> remote storage. So, one Kafka
> segment could be saved in multiple chunks in remote storage.
>
> The remote log index also have a larger index interval. The default
> interval of the local .index file
> (log.index.interval.bytes) is 4KB. In the current HDFS RSM implementation,
> the default remote
> index interval (hdfs.remote.index.interval.bytes) is 256KB. The
> coarse-grained remote index saves
> some local disk space. The smaller size also makes it more likely to be
> cached in physical memory.
>
>
>
>
> On Thu, Oct 31, 2019 at 1:58 PM Jun Rao  wrote:
>
> > Hi, Harsha,
> >
> > I am still looking at the KIP and the PR. A couple of quick
> > comments/questions.
> >
> > 20. It's fine to keep the HDFS binding temporarily in the PR. We just need
> > to remove it before it's merged to trunk. As Victor mentioned, we can
> > provide a reference implementation based on a mocked version of remote
> > storage.
> >
> > 21. I am not sure that I understood the need for RemoteLogIndexEntry and
> > its relationship with RemoteLogSegmentInfo. It seems
> > that RemoteLogIndexEntry are offset index entries pointing to record
> > batches inside a segment. That seems to be the same as the .index file?
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Oct 28, 2019 at 9:11 PM Satish Duggana 
> > wrote:
> >
> > > Hi Viktor,
> > > >1. Can we allow RLM Followers to serve read requests? After all segments
> > > on
> > > the cold storage are closed ones, no modification is allowed. Besides
> > > KIP-392 (
> > >
> > >
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D392-253A-2BAllow-2Bconsumers-2Bto-2Bfetch-2Bfrom-2Bclosest-2Breplica&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=g7ujYPRBvNrON18SBeCt4g&m=CKNMp77DfMghjYo1JqbWr5jl-DRDBGF2owao5zUXDeE&s=HTPACirRO-wVmOHmGEMlTIAov4szGHn38xrbFbMZK_I&e=
> > > )
> > > would introduce follower fetching too, so I think it would be nice to
> > > prepare RLM for this as well.
> > >
> > > That is a good point. We plan to support fetching remote storage from
> > > followers too. Current code in the PR work fine for this scenario
> > > though there may be some edge cases to be handled. We have not yet
> > > tested this scenario.
> > >
> > > >2. I think the remote.log.storage.enable config is redundant. By
> > > specifying
> > >
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__remote.log.storage.manager.class.name&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=g7ujYPRBvNrON18SBeCt4g&m=CKNMp77DfMghjYo1JqbWr5jl-DRDBGF2owao5zUXDeE&s=QsUunkBFX3dne_4caCiEAbp9xKUPrFx1srwznOR_Sfc&e=
> > one already declares that they want
> > > to use remote storage. Would it make sense to remove
> > > the remote.log.storage.enable config?
> > >
> > > I do not think it is really needed. `remote.log.storage.enable`
> > > property can be removed.
> > >
> > > Thanks,
> > > Satish.
> > >
> > >
> > > On Thu, Oct 24, 2019 at 2:46 PM Viktor Somogyi-Vass
> > >  wrote:
> > > >
> > > > Hi Harsha,
> > > >
> > > > A couple more questions:
> > > > 1. Can we allow RLM Followers to serve read requests? After all
> > segments
> > > on
> > > > the cold storage are closed ones, no modification is allowed. Besides
> > > > KIP-392 (
> > > >
> > >
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D392-253A-2BAllow-2Bconsumers-2Bto-2Bfetch-2Bfrom-2Bclosest-2Breplica&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=g7ujYPRBvNrON18SBeCt4g&m=CKNMp77DfMghjYo1JqbWr5jl-DRDBGF2owao5zUXDeE&s=HTPACirRO-wVmOHmGEMlTIAov4szGHn38xrbFbMZK_I&e=
> > > )
> > > > would introduce follower fetching too, so I think it would be nice to
> > > > prepare RLM for this as well.
> > > > 2. I think the remote.log.storage.enable config is redundant. By
> > > specifying
> > > >
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__remote.log.storage.manager

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Tom Bentley]

Hi Ying,

Because only inactive segments can be shipped to remote storage, to be able
> to ship log data as soon
> as possible, we will roll log segment very fast (e.g. every half hour).
>

So that means a consumer which gets behind by half an hour will find its
reads being served from remote storage. And, if I understand the proposed
algorithm, each such consumer fetch request could result in a separate
fetch request from the remote storage. I.e. there's no mechanism to
amortize the cost of the fetching between multiple consumers fetching
similar ranges?

(Actually the doc for RemoteStorageManager.read() says "It will read at
least one batch, if the 1st batch size is larger than maxBytes.". Does that
mean the broker might have to retry with increased maxBytes if the first
request fails to read a batch? If so, how does it know how much to increase
maxBytes by?)

Thanks,

Tom

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

>So that means a consumer which gets behind by half an hour will find its
reads being served from remote storage. And, if I understand the proposed
algorithm, each such consumer fetch request could result in a separate
fetch request from the remote storage. I.e. there's no mechanism to
amortize the cost of the fetching between multiple consumers fetching
similar ranges?

local log segments are deleted according to the local
log.retention.time/.size settings though they may have been already
copied to remote storage. Consumers would still be able to fetch the
messages from local storage if they are not yet deleted based on the
retention. They will be served from remote storage only when they are
not locally available.

Thanks,
Satish.

On Thu, Nov 7, 2019 at 7:58 AM Tom Bentley  wrote:
>
> Hi Ying,
>
> Because only inactive segments can be shipped to remote storage, to be able
> > to ship log data as soon
> > as possible, we will roll log segment very fast (e.g. every half hour).
> >
>
> So that means a consumer which gets behind by half an hour will find its
> reads being served from remote storage. And, if I understand the proposed
> algorithm, each such consumer fetch request could result in a separate
> fetch request from the remote storage. I.e. there's no mechanism to
> amortize the cost of the fetching between multiple consumers fetching
> similar ranges?
>
> (Actually the doc for RemoteStorageManager.read() says "It will read at
> least one batch, if the 1st batch size is larger than maxBytes.". Does that
> mean the broker might have to retry with increased maxBytes if the first
> request fails to read a batch? If so, how does it know how much to increase
> maxBytes by?)
>
> Thanks,
>
> Tom

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

Hi Tom,'
Sorry, I missed the other question.

>(Actually the doc for RemoteStorageManager.read() says "It will read at
least one batch, if the 1st batch size is larger than maxBytes.". Does that
mean the broker might have to retry with increased maxBytes if the first
request fails to read a batch? If so, how does it know how much to increase
maxBytes by?)

broker or RemoteLogManager does not need to retry here.
RemoteStorageManager can return `Records` which can be more than
maxBytes if the first batch containing startOffset has more than
maxBytes.

Thanks,
Satish.

On Thu, Nov 7, 2019 at 8:33 AM Satish Duggana  wrote:
>
> >So that means a consumer which gets behind by half an hour will find its
> reads being served from remote storage. And, if I understand the proposed
> algorithm, each such consumer fetch request could result in a separate
> fetch request from the remote storage. I.e. there's no mechanism to
> amortize the cost of the fetching between multiple consumers fetching
> similar ranges?
>
> local log segments are deleted according to the local
> log.retention.time/.size settings though they may have been already
> copied to remote storage. Consumers would still be able to fetch the
> messages from local storage if they are not yet deleted based on the
> retention. They will be served from remote storage only when they are
> not locally available.
>
> Thanks,
> Satish.
>
> On Thu, Nov 7, 2019 at 7:58 AM Tom Bentley  wrote:
> >
> > Hi Ying,
> >
> > Because only inactive segments can be shipped to remote storage, to be able
> > > to ship log data as soon
> > > as possible, we will roll log segment very fast (e.g. every half hour).
> > >
> >
> > So that means a consumer which gets behind by half an hour will find its
> > reads being served from remote storage. And, if I understand the proposed
> > algorithm, each such consumer fetch request could result in a separate
> > fetch request from the remote storage. I.e. there's no mechanism to
> > amortize the cost of the fetching between multiple consumers fetching
> > similar ranges?
> >
> > (Actually the doc for RemoteStorageManager.read() says "It will read at
> > least one batch, if the 1st batch size is larger than maxBytes.". Does that
> > mean the broker might have to retry with increased maxBytes if the first
> > request fails to read a batch? If so, how does it know how much to increase
> > maxBytes by?)
> >
> > Thanks,
> >
> > Tom

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Tom Bentley]

Hi Satish,

>So that means a consumer which gets behind by half an hour will find its
> reads being served from remote storage. And, if I understand the proposed
> algorithm, each such consumer fetch request could result in a separate
> fetch request from the remote storage. I.e. there's no mechanism to
> amortize the cost of the fetching between multiple consumers fetching
> similar ranges?
>
> local log segments are deleted according to the local
> log.retention.time/.size settings though they may have been already
> copied to remote storage. Consumers would still be able to fetch the
> messages from local storage if they are not yet deleted based on the
> retention. They will be served from remote storage only when they are
> not locally available.
>

Thanks, I missed that point. However, there's still a point at which the
consumer fetches start getting served from remote storage (even if that
point isn't as soon as the local log retention time/size). This represents
a kind of performance cliff edge and what I'm really interested in is how
easy it is for a consumer which falls off that cliff to catch up and so its
fetches again come from local storage. Obviously this can depend on all
sorts of factors (like production rate, consumption rate), so it's not
guaranteed (just like it's not guaranteed for Kafka today), but this would
represent a new failure mode.

Another aspect I'd like to understand better is the effect of serving fetch
request from remote storage has on the broker's network utilization. If
we're just trimming the amount of data held locally (without increasing the
overall local+remote retention), then we're effectively trading disk
bandwidth for network bandwidth when serving fetch requests from remote
storage (which I understand to be a good thing, since brokers are
often/usually disk bound). But if we're increasing the overall local+remote
retention then it's more likely that network itself becomes the bottleneck.
I appreciate this is all rather hand wavy, I'm just trying to understand
how this would affect broker performance, so I'd be grateful for any
insights you can offer.

Cheers,

Tom

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

>>Depends on the implementation, the data of one segment may not necessary be
stored in a single file.
There could be a maximum object / chunk / file size restriction on the
remote storage. So, one Kafka
segment could be saved in multiple chunks in remote storage.

>Having one local segment can be stored in multiple files and each file
can have a base position as part of the metadata(like name) of file or
object etc.
File/object name can be --. So
any read request for a position with in that segment can be found by
computing relative position viz `fetchPosition-basePosition`.

Let me elaborate further on how to address a single local segment file
being copied to multiple files/blocks in remote storage without the
need to map local segment positions to remote segment positions.
Let us say a local segment file has offsets from 1000-95000. This may
be copied to remote storage in multiple files/blocks. Each file or
block can be created with name or any other metadata containing
--. This does not require
recomputing positions for the remote segments.

local segment file has offsets: 1000 - 95000

remote segment file suffix format can be :
--
remote-segment-file-1: 1000-20200-0
remote-segment-file-2: 20201-45003-942346
remote-segment-file-3: 45004-78008-6001235
remote-segment-file-4: 78009-95000-20024761

If a read comes for 52340 offset and position as 7321236, relative
position in remote segment-3 is: 7321236-6001235 = 1320001

Thanks,
Satish.

On Thu, Nov 7, 2019 at 7:55 AM Satish Duggana  wrote:
>
> >Depends on the implementation, the data of one segment may not necessary be
> stored in a single file.
> There could be a maximum object / chunk / file size restriction on the
> remote storage. So, one Kafka
> segment could be saved in multiple chunks in remote storage.
>
> Having one local segment can be stored in multiple files and each file
> can have a base position as part of the metadata(like name) of file or
> object etc.
> File/object name can be --. So
> any read request for a position with in that segment can be found by
> computing relative position viz `fetchPosition-basePosition`.
>
>
>
> On Thu, Nov 7, 2019 at 6:04 AM Ying Zheng  wrote:
> >
> > 21. I am not sure that I understood the need for RemoteLogIndexEntry and
> > its relationship with RemoteLogSegmentInfo. It seems
> > that RemoteLogIndexEntry are offset index entries pointing to record
> > batches inside a segment. That seems to be the same as the .index file?
> >
> > We do not assume the how the data is stored in the remote storage.
> > Depends on the implementation, the data of one segment may not necessary be
> > stored in a single file.
> > There could be a maximum object / chunk / file size restriction on the
> > remote storage. So, one Kafka
> > segment could be saved in multiple chunks in remote storage.
> >
> > The remote log index also have a larger index interval. The default
> > interval of the local .index file
> > (log.index.interval.bytes) is 4KB. In the current HDFS RSM implementation,
> > the default remote
> > index interval (hdfs.remote.index.interval.bytes) is 256KB. The
> > coarse-grained remote index saves
> > some local disk space. The smaller size also makes it more likely to be
> > cached in physical memory.
> >
> >
> >
> >
> > On Thu, Oct 31, 2019 at 1:58 PM Jun Rao  wrote:
> >
> > > Hi, Harsha,
> > >
> > > I am still looking at the KIP and the PR. A couple of quick
> > > comments/questions.
> > >
> > > 20. It's fine to keep the HDFS binding temporarily in the PR. We just need
> > > to remove it before it's merged to trunk. As Victor mentioned, we can
> > > provide a reference implementation based on a mocked version of remote
> > > storage.
> > >
> > > 21. I am not sure that I understood the need for RemoteLogIndexEntry and
> > > its relationship with RemoteLogSegmentInfo. It seems
> > > that RemoteLogIndexEntry are offset index entries pointing to record
> > > batches inside a segment. That seems to be the same as the .index file?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Oct 28, 2019 at 9:11 PM Satish Duggana 
> > > wrote:
> > >
> > > > Hi Viktor,
> > > > >1. Can we allow RLM Followers to serve read requests? After all 
> > > > >segments
> > > > on
> > > > the cold storage are closed ones, no modification is allowed. Besides
> > > > KIP-392 (
> > > >
> > > >
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D392-253A-2BAllow-2Bconsumers-2Bto-2Bfetch-2Bfrom-2Bclosest-2Breplica&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=g7ujYPRBvNrON18SBeCt4g&m=CKNMp77DfMghjYo1JqbWr5jl-DRDBGF2owao5zUXDeE&s=HTPACirRO-wVmOHmGEMlTIAov4szGHn38xrbFbMZK_I&e=
> > > > )
> > > > would introduce follower fetching too, so I think it would be nice to
> > > > prepare RLM for this as well.
> > > >
> > > > That is a good point. We plan to support fetching remote storage from
> > > > followers too. Current code in the PR work fine for this scenario
> > > > though there may be some edge

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

[Navinder Brar]

+1 on implementing offset based lag for now and push time-based lag to a later 
point in time when broker changes are done. Although time-based lag enhances 
the readability, it would not be a make or break change for implementing this 
KIP. 

Vinoth has explained the role of KeyQueryMetadata, let me in add in my 2 cents 
as well.    
   - There are basically 2 reasons. One is that instead of having two 
functions, one to get StreamsMetadata for active and one for replicas. We are 
fetching both in a single call and we have a way to get only active or only 
replicas from the KeyQueryMetadata object(just like isStandby() and isActive() 
discussion we had earlier)
   - Since even after fetching the metadata now we have a requirement of 
fetching the topicPartition for which the query came:- to fetch lag for that 
specific topicPartition. Instead of having another call to fetch the partition 
from StreamsMetadataState we thought using one single call and fetching 
partition and all metadata would be better.
   - Another option was to change StreamsMetadata object and add topicPartition 
in that for which the query came but it doesn’t make sense in terms of 
semantics as it StreamsMetadata. Also, KeyQueryMetadata represents all the 
metadata for the Key being queried, i.e. the partition it belongs to and the 
list of StreamsMetadata(hosts) active or replica where the key could be found.
   




On Thursday, 7 November, 2019, 01:53:36 am IST, Vinoth Chandar 
 wrote:  
 
 +1 to John, suggestion on Duration/Instant and dropping the API to fetch
all store's lags. However, I do think we need to return lags per topic
partition. So not sure if single return value would work? We need some new
class that holds a TopicPartition and Duration/Instant variables together?

10) Because we needed to return the topicPartition the key belongs to, in
order to correlate with the lag information from the other set of APIs.
Otherwise, we don't know which topic partition's lag estimate to use. We
tried to illustrate this on the example code. StreamsMetadata is simply
capturing state of a streams host/instance, where as TopicPartition depends
on the key passed in. This is a side effect of our decision to decouple lag
based filtering on the metadata apis.

20) Goes back to the previous point. We needed to return information that
is key specific, at which point it seemed natural for the KeyQueryMetadata
to contain active, standby, topic partition for that key. If we merely
returned a standbyMetadataForKey() -> Collection standby,
an active metadataForKey() -> StreamsMetadata, and new
getTopicPartition(key) -> topicPartition object back to the caller, then
arguably you could do the same kind of correlation. IMO having a the
KeyQueryMetadata class to encapsulate all this is a friendlier API.
 allStandbyMetadata() and allStandbyMetadataForStore() are just counter
parts for metadataForStore() and allMetadata() that we introduce mostly for
consistent API semantics. (their presence implicitly could help denote
metadataForStore() is for active instances. Happy to drop them if their
utility is not clear)

30) This would assume we refresh all the standby lag information every
time we query for that StreamsMetadata for a specific store? For time based
lag, this will involve fetching the tail kafka record at once from multiple
kafka topic partitions? I would prefer not to couple them like this and
have the ability to make granular store (or even topic partition level)
fetches for lag information.

32) I actually prefer John's suggestion to let the application drive the
lag fetches/updation and not have flags as the KIP current points to. Are
you reexamining that position?

On fetching lag information, +1 we could do this much more efficiently with
a broker changes. Given I don't yet have a burning need for the time based
lag, I think we can sequence the APIs such that the offset based ones are
implemented first, while we have a broker side change?
Given we decoupled the offset and time based lag API, I am willing to drop
the time based lag functionality (since its not needed right away for my
use-case). @navinder . thoughts?


On Tue, Nov 5, 2019 at 11:10 PM Matthias J. Sax 
wrote:

> Navinder,
>
> thanks for updating the KIP. Couple of follow up questions:
>
>
> (10) Why do we need to introduce the class `KeyQueryMetadata`?
>
> (20) Why do we introduce the two methods `allMetadataForKey()`? Would it
> not be simpler to add `Collection
> standbyMetadataForKey(...)`. This would align with new methods
> `#allStandbyMetadata()` and `#allStandbyMetadataForStore()`?
>
> (30) Why do we need the class `StoreLagInfo` -- it seems simpler to just
> extend `StreamMetadata` with the corresponding attributes and methods
> (of active task, the lag would always be reported as zero)
>
> (32) Via (30) we can avoid the two new methods `#allLagInfo()` and
> `#lagInfoForStore()`, too, reducing public API and making it simpler to
> use the feature.
>
> Btw: If we

[jira] [Resolved] (KAFKA-9150) DescribeGroup uses member assignment as metadata

[Manikumar (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-9150.
--
Resolution: Fixed

> DescribeGroup uses member assignment as metadata
> 
>
> Key: KAFKA-9150
> URL: https://issues.apache.org/jira/browse/KAFKA-9150
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: David Jacot
>Priority: Blocker
> Fix For: 2.4.0
>
>
> When we converted the DescribeGroup internally to rely on the generated 
> protocol in KAFKA-7922, we introduced a regression in the response handling. 
> Basically we serialize the member assignment as both the assignment and 
> metadata in the response: 
> [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala#L1326].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk11 #937

[Apache Jenkins Server]

See 


Changes:

[bbejeck] [MINOR] Clean up PartitionAssignor for KIP-441 (#7649)

[wangguoz] KAFKA-8729: Change `PartitionResponse` to include all troubling 
records


--
[...truncated 2.75 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFa