Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hi Becket, Relying on the cluster metadata doesn't seem like it would work if there are multiple fetcher threads, right? It also doesn't work for the consumer case, which Jay suggested would be good to handle. Ismael On Thu, Nov 23, 2017 at 2:21 AM, Becket Qin wrote: > Thanks for the KIP, Coli

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hi James, There are 2 options being discussed. Option A is similar to the existing approach where the follower informs the leader of offsets it has seen by asking for the next ones. We just skip the partitions where the offset hasn't changed. Option B involves the leader keeping track of the off

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

The comment is valid. It falls exactly into this topic, it has exactly todo with this! Even when we have a statefull operation last. We move it to the very first processor (KtableSource) and therefore cant present a proper RecordContext. Regarding the other Jiras you are referring to. They harm

[GitHub] kafka pull request #4249: MINOR: fix typo in ProducerConfig doc

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4249 ---

[jira] [Created] (KAFKA-6266) Kafka 1.0.0 : Repeated occurrence of WARN Resetting first dirty offset of __consumer_offsets-xx to log start offset 203569 since the checkpointed offset 120955 is invalid

VinayKumar created KAFKA-6266: - Summary: Kafka 1.0.0 : Repeated occurrence of WARN Resetting first dirty offset of __consumer_offsets-xx to log start offset 203569 since the checkpointed offset 120955 is invalid. (kafka.log.LogCleanerManager$)

[GitHub] kafka pull request #4192: MINOR: Remove unnecessary batch iteration in FileR...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4192 ---

[GitHub] kafka pull request #4256: KAFKA-6238: Fix 1.0.0 upgrade instructions relatin...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4256 ---

[jira] [Created] (KAFKA-6267) Kafka Producer - initTransaction forever waiting

Silvio Papa created KAFKA-6267: -- Summary: Kafka Producer - initTransaction forever waiting Key: KAFKA-6267 URL: https://issues.apache.org/jira/browse/KAFKA-6267 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-6268) Tools should now swallow exceptions like resolving network names

Antony Stubbs created KAFKA-6268: Summary: Tools should now swallow exceptions like resolving network names Key: KAFKA-6268 URL: https://issues.apache.org/jira/browse/KAFKA-6268 Project: Kafka

Build failed in Jenkins: kafka-trunk-jdk9 #216

See Changes: [ismael] MINOR: fix typo in ProducerConfig doc -- [...truncated 1.44 MB...] kafka.security.auth.SimpleAclAuthorizerTest > testHighConcurrencyDeletionOfResourc

[jira] [Created] (KAFKA-6269) KTable state restore fails after rebalance

Andreas Schroeder created KAFKA-6269: Summary: KTable state restore fails after rebalance Key: KAFKA-6269 URL: https://issues.apache.org/jira/browse/KAFKA-6269 Project: Kafka Issue Type:

[GitHub] kafka pull request #4231: MINOR: Small cleanups/refactoring in kafka.control...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4231 ---

Build failed in Jenkins: kafka-trunk-jdk9 #217

See Changes: [ismael] MINOR: Remove unnecessary batch iteration in FileRecords.downConvert [ismael] KAFKA-6238; Fix 1.0.0 upgrade instructions relating to the message -- [

Kafka ERROR after upgrade to Kafka 1.0.0 version : java.lang.OutOfMemoryError: Java heap space

Hi, I upgraded Kafka from 0.10.2.1 to 1.0.0 version. And only from then, I'm seeing the kafka service going down because of below issue: ERROR [KafkaApi-1] Error when handling request ; (kafka.server.KafkaApis) {replica_id=-1,max_wait_time=500,min_bytes=1,topics=[{topic= ,partitions=[{partiti

[GitHub] kafka pull request #4260: KAFKA-5647: Use KafkaZkClient in ReassignPartition...

GitHub user omkreddy opened a pull request: https://github.com/apache/kafka/pull/4260 KAFKA-5647: Use KafkaZkClient in ReassignPartitionsCommand and PreferredReplicaLeaderElectionCommand * Use KafkaZkClient in ReassignPartitionsCommand * Use KafkaZkClient in PreferredReplicaLe

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Yes, caching the log segment position after the index lookup may work. One subtle issue is that for a compacted topic, the underlying log segment may have changed between two consecutive fetch requests, and we need to think through the impact of that. Thanks, Jun On Wed, Nov 22, 2017 at 7:54 PM,

Build failed in Jenkins: kafka-trunk-jdk8 #2236

See Changes: [ismael] MINOR: Remove unnecessary batch iteration in FileRecords.downConvert [ismael] KAFKA-6238; Fix 1.0.0 upgrade instructions relating to the message [ismael] MINOR: Small cleanups/refactoring

Build failed in Jenkins: kafka-1.0-jdk7 #92

See Changes: [ismael] MINOR: fix typo in ProducerConfig doc -- [...truncated 196.78 KB...] kafka.api.SaslSslAdminClientIntegrationTest > testAclOperations2 PASSED kafka.api.S

Jenkins build is back to normal : kafka-trunk-jdk9 #218

See

Jenkins build is back to normal : kafka-1.0-jdk7 #93

See

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hi Ismael, Yes, you are right. The metadata may not help for multiple fetch thread or the consumer case. Session based approach is probably better in this case. The optimization of only returning data at the offset index entry boundary may still be worth considering. It also helps improve the ind

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hey Colin, Thanks for the KIP! This is definitely useful when there are many idle partitions in the clusters. Just in case it is useful, I will provide some number here. We observe that for a clsuter that have around 2.5k partitions per broker, the ProduceRequestTotal time average value is around

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

From a DSL point of view, users only see the new value on a KTable#filter anyway. So why should it be an issue that we use pair under the hood? User sees newValue and gets corresponding RecordContext. I can't see any issue here? I cannot follow here: >> Even when we have a statefull operation l