Question about "prevent tight-loop" logic in Selector's madeReadProgressLastPoll

Hello Kafka Developers, I hope this email finds you well. This might be a silly question, but I had a question about the Kafka client code (v3.9.0) that is `madeReadProgressLastPoll` field in the `Selector` class and the mechanism described in its comment: "to prevent tight loops when memo

Re: a question of ISR and ack all

. > > On Sun, Apr 27, 2025, 18:37 XiJun Ye wrote: > >> I have a question about data consistency. Kafka ISR only determines the >> time of lag, not the number of messages. Does ack=all require the replicas >> in the ISR to completely catch up with the number of messages or

Re: a question of ISR and ack all

With acks=all, it is guaranteed that a message is replicated to all ISRs on successful produce response. lag.time.max is used to determine which replica is in sync, not to determine a message is replicated. On Sun, Apr 27, 2025, 18:37 XiJun Ye wrote: > I have a question about data consiste

a question of ISR and ack all

I have a question about data consistency. Kafka ISR only determines the time of lag, not the number of messages. Does ack=all require the replicas in the ISR to completely catch up with the number of messages or to meet the time requirement? For example, if there are 3 replicas in the ISR

Re: Question about ResourcePatternFilter

Hi Claude, As I recall, the intention was that if PatternType is ANY, the name field would simply be ignored. Probably we should have thrown an exception if someone tried to create a ResourcePatternFilter with a non-null name and PatternType ANY. best, Colin On Tue, Aug 20, 2024, at 04:19, Cl

Re: Question about config.action.reload

Hi Ran, The config.action.reload, subscribe, unsubscribe, and ConfigChangeCallback features are partially-implemented. As far as I can tell if you're using only AK connect-runtime and config providers, none of it is functional. If you have a config provider that implements TTLs, the runtime will

Question about config.action.reload

Hi Kafka team, Currently I'm investigating the config.action.reload feature of Kafka Connect. I'm wondering how to enable this feature. E.g. If I use the FileConfigProvider, clients/src/main/java/org/apache/kafka/common/config/provider/FileConfigProvider.java is it enough to set the config.action.

Question about ResourcePatternFilter

Should a ResourcePatternFilter that has a PatternType of ANY and a name of WILDCARD_RESOURCE not match any Acls? I think this is a bug.I am writing a series of tests to ensure that I have implemented everything correctly in the Trie implementation and this has come up. public boolean matches(Res

Re: ACL authorization question

:DOH: Nevermind. Problem between keyboard and seat. On Thu, Aug 15, 2024 at 8:36 AM Claude Warren, Jr wrote: > If there is an authorizer with no ACLs and > authorizeByResourceType(AuthorizableRequestContext > requestContext, AclOperation op, ResourceType resourceType) is called > with op = UN

ACL authorization question

If there is an authorizer with no ACLs and authorizeByResourceType(AuthorizableRequestContext requestContext, AclOperation op, ResourceType resourceType) is called with op = UNKNOWN or ANY, or resourceType = UKNOWN or ANY should an IllegalArgumentException be thrown as it is when there are ACLs? I

RE: [QUESTION] What to do about conflicted KIP numbers?

al Message- From: Matthias J. Sax Sent: Friday, June 14, 2024 3:35 PM To: dev@kafka.apache.org Subject: Re: [QUESTION] What to do about conflicted KIP numbers? I don't think that there is an official guideline. Personally, I would suggest that the corresponding KIP owners agree who is keeping

Re: [QUESTION] What to do about conflicted KIP numbers?

I don't think that there is an official guideline. Personally, I would suggest that the corresponding KIP owners agree who is keeping the conflicting number, and how is changing it. For the ones changing the number, I would propose to restart a new DISCUSS thread using the new number to separ

[QUESTION] What to do about conflicted KIP numbers?

Hi Kafka devs, I submitted a KIP last week and encountered a KIP-process race condition where my KIP number was consumed by another dev without updating the wiki page containing KIPs: Kafka Improvement Proposals - Apache Kafka - Apache Software Foundation

[QUESTION] Move log start offset back in time for data recovery

Hi, I have one question for tier storage, have a 3.6 cluster with some log segments exists in the remote store with metadata, but kafka's earliest offset is more recent, is there a way to move start log offset back in time? I wonder if we can still serve that data to consumer. If yes,

[jira] [Created] (KAFKA-16648) Question: KIP-848 and KafkaTestKit.java

sanghyeok An created KAFKA-16648: Summary: Question: KIP-848 and KafkaTestKit.java Key: KAFKA-16648 URL: https://issues.apache.org/jira/browse/KAFKA-16648 Project: Kafka Issue Type: Bug

Re: [Question] About Kafka producer design decision making

I should also clarify that Chia-Ping took it over from me originally and improved the proposal significantly. I think he also got busy and hasn't been able to spend time on it for a while though. Ismael On Tue, Jan 2, 2024 at 4:08 PM Ismael Juma wrote: > I had written a KIP a while back: > > h

Re: [Question] About Kafka producer design decision making

I had written a KIP a while back: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100829459 Happy for someone to pick it up and drive it forward. Ismael On Tue, Jan 2, 2024 at 2:00 PM Justine Olshan wrote: > Hey folks -- > I think this is a good conversation. Given we plan t

Re: [Question] About Kafka producer design decision making

Hey folks -- I think this is a good conversation. Given we plan to drop support for Java 8 in 4.0 this seems like a good time to consider this change. Perhaps we should file a JIRA ticket and maybe create a KIP to discuss? One thing we should consider however, is if we want some of these operatio

Re: [Question] About Kafka producer design decision making

hey sean - a lot of uses of the Futures are in the public API and therefore take voting/effort to be changed. i don't know any reason for intentionally avoiding the use of CompletableFuture, however, others might know more than I do. thanks, P On Tue, Nov 14, 2023 at 1:27 AM 신수웅(Sean Sin) wrote

[Question] About Kafka producer design decision making

Dear Apache Kakfa Developers, I'm 4-year SWE in South Korea. I have some questions while watching Kafka Producer API. *Why Use "Future" and Not "CompletableFuture"?* In the case of "Future", blocking occurs when calling "*get()*", so I thought "Computable Future" would be better when doing more

Re: question

; I‘m working inKafika development. Now,I have a question. Does Kafka > support Android client? > > > > > 石殿杰 > 技术中心 > 邮箱：shidian...@mxnavi.com > 电话：18341724011 >

Re: [QUESTION] What is the difference between sequence and offset for a Record?

gt; -Matthias > > >>> > > >>> On 8/1/23 2:19 AM, tison wrote: > > >>>> Hi, > > >>>> > > >>>> I'm wringing a Kafka API Rust codec library[1] to understand how > Kafka > > >>>> models its con

Re: [QUESTION] about topic dimension quota

> 发送时间: 2023-08-08 21:10:39 (星期二) > 收件人: dev@kafka.apache.org > 抄送: > 主题: [QUESTION] about topic dimension quota >

[QUESTION] about topic dimension quota

Hi，all. Let me ask a question first, that is, do we plan to support quota in the topic dimension?

Re: [QUESTION] What is the difference between sequence and offset for a Record?

st codec library[1] to understand how Kafka > >>>> models its concepts and how the core business logic works. > >>>> > >>>> During implementing the codec for Records[2], I saw a twins of fields > >>>> "sequence" and "o

Re: [QUESTION] What is the difference between sequence and offset for a Record?

he correct offset for each produced records? And how does Kafka maintain the calculation for offset and sequence during these modifications? I'll appreciate if anyone can answer the question or give some insights :D Best, tison. [1] https://github.com/tisonkun/kafka-api [2] https://kafka.apache.org/documentation/#messageformat

Re: [QUESTION] What is the difference between sequence and offset for a Record?

d on my codec library, and observe that the server may need > to > > > update offset for records produced. How does Kafka set the correct > offset > > > for each produced records? And how does Kafka maintain the calculation > > for > > > offset and sequence during these modifications? > > > > > > I'll appreciate if anyone can answer the question or give some insights > > :D > > > > > > Best, > > > tison. > > > > > > [1] https://github.com/tisonkun/kafka-api > > > [2] https://kafka.apache.org/documentation/#messageformat > > > > > >

Re: [QUESTION] What is the difference between sequence and offset for a Record?

; > server based on my codec library, and observe that the server may need to > > update offset for records produced. How does Kafka set the correct offset > > for each produced records? And how does Kafka maintain the calculation > for > > offset and sequence

Re: [QUESTION] What is the difference between sequence and offset for a Record?

ect offset for each produced records? And how does Kafka maintain the calculation for offset and sequence during these modifications? I'll appreciate if anyone can answer the question or give some insights :D Best, tison. [1] https://github.com/tisonkun/kafka-api [2] https://kafka.apache.org/documentation/#messageformat

[QUESTION] What is the difference between sequence and offset for a Record?

y need to update offset for records produced. How does Kafka set the correct offset for each produced records? And how does Kafka maintain the calculation for offset and sequence during these modifications? I'll appreciate if anyone can answer the question or give some insights :D Best, tison. [1]

Question about Lag Calculations in Apache Kafka Source Code

Hi everyone, I am interested in understanding how the broker performs lag calculations. Specifically, I would like to explore the possibility of improving the calculation method to use the latest stable offset instead of the latest offset. I noticed that there might be differences between the r

Re: Question ❓

start­ed studying kafka and raised a question. Is it possible for each user to make a separate queue? as I understand it, there is a broker with different topics, and each topic had the number of partitions = the number of use­rs. if yes, you can link to an example or explanation. Google didn't

Re: Question ❓

-原始邮件- > 发件人: "влад тасканов" > 发送时间: 2023-05-10 02:11:21 (星期三) > 收件人: dev@kafka.apache.org > 抄送: > 主题: Question ❓ >

Question ❓

Hi. I recently start­ed studying kafka and raised a question. Is it possible for each user to make a separate queue? as I understand it, there is a broker with different topics, and each topic had the number of partitions = the number of use­rs. if yes, you can link to an example or

Re: Question: CI Pipeline in Kafka GitHub Pull Requests & Github Actions Usage

Dear David: I greatly appreciate your welcome and the explanation that you offered regarding the Jenkins system! I will try to familiarize myself with the standards and contribution process of the Kafka community, and look forward to making some contributions in the coming days. Best regards, Aa

Re: Question: CI Pipeline in Kafka GitHub Pull Requests & Github Actions Usage

Hello, Aaron and welcome to the project! If you look towards the bottom of a Pull Request there will be a section reporting the status of the Checks for the latest commit. There is a "Details" link that takes you to the Jenkins job for that PR. The default Jenkins view is the new UI (called Blue

Question: CI Pipeline in Kafka GitHub Pull Requests & Github Actions Usage

Hey folks, I hope this email finds you well. My name is Aaron Ai, and I'm a beginner interested in the Kafka community. I am eager to contribute to Kafka's development and be part of this amazing project. However, I have noticed some confusion surrounding the CI pipeline in Kafka GitHub pull requ

A question about topic limit in produceBytesIn

Another question about using kafka recently: Currently, Kafka's write throttle only supports the user dimension and clientId dimension of the request. In fact, such a situation is often encountered in actual use: a topic entry traffic suddenly increases, and the resource bottleneck is

A question about the reassign task in kafka

One question were found in the process of using kafka recently: Why does Kafka start fetching from the leader's logStartOffset when Kafka is doing the topic reassign task (such as cluster expansion brokers)? If the amount of data stored locally by the partition leader is large, the di

Kafka General Question

Hello Kafka Team I am trying to some POC of the Messaging and it will working fine with the remote host and client . But I have question how I can send message/response from consumer to producer or server using topic . Please Guide me on this and share any doc if possible . Thank You Rohan Kar

i have question to kafka error

hi~ i have question. i installed kafka in azure kubernetes services. but occur is error. -- error -- {"thread_name":"controller-event-thread","msg":"[ReplicaStateMachine controllerId=0] Triggering offline replica state changes","l

Re: question

Hello jincheng, Kafka provides Java Producer/Consumer/Admin public api, so you can access Kafka if you are using Java to develop an Android App even though it's not common. for example, you can develop an Android App to get the Kafka metadata, send bury-point event log to Kafka, however, a better s

Question about plugins / extension API

Hi colleagues, I tried to find the answer in the mail list archive and the Internet, but found nothing. I see that kafka project doesn't introduce general extension/ plugin API for three major releases. I mean the top-level plugin/extension that allow to add some user-specific code at the brok

Re: Question about the Log Compaction

Hi, Liang, Currently, we store the MD5 of the record key in OffsetMap. Since it has a large domain (16 bytes), we assume there is no collision there. Thanks, Jun On Wed, Mar 2, 2022 at 1:20 AM 阮良 wrote: > Hi all > > I am confused about the Log Compaction logic，use OffsetMap > to deduplicating

Question about the Log Compaction

Hi all I am confused about the Log Compaction logic，use OffsetMap to deduplicating the log. in my opinion when there is a hash conflict , data may be lost Eg： Record1(key1,offset1) Record2(key2,offset2) Conditionhash(key1) == hash(key2) && (offset1 < offset2)

Question: Lazy client connections and status check

Hi All, The admin, consumer and producer client creation are all lazy connections, in the sense that one only knows that the connection is “healthy” when an “action” is carried out (e.g., send a message). There is no API call to check if the connection is healthy or if there are any issues wit

Re: poll block question

Hi xusheng, I checked the code stack, and mapped to the kafka code, I can see we either use Selector#selectNow, or Selector#select(timeoutMs), which should never hang forever. If so, I think it might be the `timeoutMS` is too large, and from your code stack, the timeoutMS came from the Consumer#pol

poll block question

hi i find something not sure about poll. i need some help my kafka server version is 1.1.0 my kafka client version is 2.3.0 then 128 partition of topic xxx, start 128 thread in one application to consume message. always run well, but run about a week long , i find there is a consumer hang forever

Re: I have a question about apache kafka.

Hello Yun Han Nam, The users@ list is more typical for this kind of topic, but this list is fine, too. What’s the question? Thanks, John On Thu, Apr 8, 2021, at 06:25, 남윤한[Yun Han Nam] wrote: > Hi. > > I want to ask you a question about Apache Kafka, is this the right one? >

I have a question about apache kafka.

Hi. I want to ask you a question about Apache Kafka, is this the right one? It is a technical question about error log that comes too often.​

Re: Question on kafka connect's incremental rebalance algorithm

Hi Ahmed, I think this bug KAFKA-12495 is the issue you described, which is under code review now. If not, please open another JIRA ticket to track it. Thanks. Luke On Tue, Apr 6, 2021 at 4:18 AM Thouqueer Ahmed < thouqueer.ah...@maplelabs.com>

Question on kafka connect's incremental rebalance algorithm

Hi, What would happen when new worker joins after the synchronization barrier ? As per code -> performTaskAssignment function of IncrementalAssignor -> Boolean canRevoke is false when it is called during the 2nd rebalance. Hence revocation is skipped and only assignment is done. T

Question for KafkaRequestHandler

Hi Folks, I am confused about the code below ，why the IO thread set the daemon ? in my thought , daemon thread is not suitable for some importment work def createHandler(id: Int): Unit = synchronized { runnables += new KafkaRequestHandler(id, brokerId, aggregateIdleMeter, threadPoolSize,

Re: [jira] [Created] (KAFKA-12157) test Upgrade 2.7.0 from 2.0.0 occur a question

157: > > > Summary: test Upgrade 2.7.0 from 2.0.0 occur a question > Key: KAFKA-12157 > URL: https://issues.apache.org/jira/browse/KAFKA-12157 > Project: Kafka > Issue Type: Bug >

[jira] [Created] (KAFKA-12157) test Upgrade 2.7.0 from 2.0.0 occur a question

Wenbing Shen created KAFKA-12157: Summary: test Upgrade 2.7.0 from 2.0.0 occur a question Key: KAFKA-12157 URL: https://issues.apache.org/jira/browse/KAFKA-12157 Project: Kafka Issue Type

question

Hello, I'm working on Kafka development. Now,I have a question. Does Kafka support Android client?

Re: Question Regarding Offset Behavior When Calling Poll()

Thank you. Regards, Zhen Zhang Software Engineer [image: Twilio] <https://www.twilio.com/?utm_source=email_signature> MOBILE (949) 771-6073 EMAIL zzh...@twilio.com On Wed, Oct 7, 2020 at 11:35 PM Matthias J. Sax wrote: > I guess it's a question of the client implementation

Re: Question Regarding Offset Behavior When Calling Poll()

I guess it's a question of the client implementation and you cannot infer the behavior from the Java client. I would _assume_ that the offset is _not_ advanced. But the behavior is not define by Kafka itself, but it's up the the client implementation itself. Thus, only the client

Re: Question Regarding Offset Behavior When Calling Poll()

s but didn't find anything useful I can relate to, I tried to ask for help from the contributors of the library I was using but didn't get an answer, not sure if this question is too obvious or too noobie lolol. Since that library I used is an implementation of the Kafka protocol, I decided to

Re: Question Regarding Offset Behavior When Calling Poll()

I guess it depends where the exception comes from? Can you clarify? -Matthias On 9/23/20 12:53 AM, Zhen Zhang wrote: > Hi there, > > I am new to Kafka and I would like to get some clarifications for a newbie > question, > > Let's say if I have set up my consumer's &

Question Regarding Offset Behavior When Calling Poll()

Hi there, I am new to Kafka and I would like to get some clarifications for a newbie question, Let's say if I have set up my consumer's "enable.auto.commit" to false, and then poll the records one at a time. So when calling poll(), starting from offset 0, if any exception

Question about [KIP-354]: Compaction can be delayed indefinitely in logs without traffic

Hello, It's my understanding that [KIP-354] should ensure eligibility for compaction after the min.compaction.lag.ms is elapsed. The parameter should ensure that a new segment is rolled within the given time. By doing some tests and looking into the code, I think this is not always the case. A

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

Thanks Gwen and Ismael. On Wed, Jun 24, 2020 at 6:52 PM Gwen Shapira wrote: > Kafka follows a "fully compatible model" and new features are always > optional. We have extensive compatibility testing to make sure rolling > upgrades are safe and painless. > This investment in compatibility allows

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

Kafka follows a "fully compatible model" and new features are always optional. We have extensive compatibility testing to make sure rolling upgrades are safe and painless. This investment in compatibility allows us to reduce the effort needed in maintaining old versions (Security CVE is the obvious

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

I agree that is an option, but is there any reason to not have a 2.3.2 and 2.4.2 released? If so, it would be nice to know about these reasons. Appreciate your time on this. On Wed, Jun 24, 2020 at 6:35 PM Ismael Juma wrote: > I think there's some confusion here. You can upgrade to AK 2.5.0 and

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

I think there's some confusion here. You can upgrade to AK 2.5.0 and completely ignore ZK and TLS. It's completely optional. Ismael On Wed, Jun 24, 2020 at 10:20 AM Gokul Ramanan Subramanian < gokul24...@gmail.com> wrote: > Any updates on Kafka 2.3.2 and 2.4.2? Given the complexity of migrating

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

Any updates on Kafka 2.3.2 and 2.4.2? Given the complexity of migrating from non-encrypted TLS to encrypted TLS connection for ZooKeeper, it would be nice to have a bug-free version of 2.3 and 2.4. Is there a technical reason why we hesitate to get these versions out? Or is it that no one has got

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

Thanks Ismael for the response. For our clusters running 2.3.1 and 2.4.1, we saw some issues which had 2.3.2 and 2.4.2 as the fix versions. I looked at 2.5.0 but since it introduces some major changes like support for ZK encryption and a few others, I was wondering if we should choose a smaller up

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

Hi Sankalp, Is there a reason why you cannot upgrade to Apache Kafka 2.5.0 instead? We are working on the 2.5.1 release, which would be the recommended release. Ismael On Wed, Jun 24, 2020 at 6:18 AM Sankalp Bhatia wrote: > Hi All, > > I would like to know if there are any plans to release a 2

Question around release plans for Apache Kafka 2.3.2 and 2.4.2

Hi All, I would like to know if there are any plans to release a 2.3.2 and 2.4.2 versions for Apache Kafka in the near future. I see there are some issues marked as fixed in these two versions (https://tinyurl.com/ycdpz5cb). However, I could not find a branch/tag corresponding to these versions in

Re: Retention policies question

e Guide' by Narkhede and others and I have a > following question. > On what side topic retention policies (delete or compact) are performed? > I have a guess that they work only on brokers that hold leader replica of > partition. > Or am I wrong ? > > With best regards, > Alex. >

Retention policies question

Hi, Dear Kafka Developers, I've read 'Kafka: The Definitive Guide' by Narkhede and others and I have a following question. On what side topic retention policies (delete or compact) are performed? I have a guess that they work only on brokers that hold leader replica of partition.

Re: Question about log flusher real frequency

Hi Alexandre, Thank you for your quick answer. I want to monitor it cause I'm trying to find out the reason why our existing Kafka cluster is configured to flush data every10 milliseconds! (people who configured it are not available anymore to answer). As that value seems really low to me, I was

Re: Question about log flusher real frequency

Hi Fares, On Linux kernels, you can use the property "dirty_writeback_centisecs" [1] to configure the period between executions of kswapd, which does this "sync" job. The period is usually set to 30 seconds. There are few exceptions where Kafka explicitly forces a sync (via the force() method from

Question about log flusher real frequency

Hello, By default, both log.flush.interval.ms and log.flush.interval.messages are set to Long.MAX_VALUE. As I understand, it makes Kafka flush log to disk (fsync) only depends on file system. Is there any simple way to monitor that frequency ? Is there a rule of thumb to estimate that value dep

Re: question about offsetSync

>From my opinion, condition 4 is downstream data loss because of master down without sync in time, so downstream data producer get smaller downstreamOffset. condition 3 meaning upstream broker down without sync in time condition 1 is init state so condition 2 may write like this: `downstreamTargetO

question about offsetSync

Hi: I am engineer from China, I review kafka mirror latest impl, but I do not figure out the reason why PartitionState update like that: ``` // true if we should emit an offset sync boolean update(long upstreamOffset, long downstreamOffset) { boolean shouldSyncOffsets = false;

Re: KIP-382 + Kafka Streams Question

erialize state from two > topics? > > A Streams app can subscribe to multiple topics. A single "stream" can come > from multiple input topics (see: > https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/StreamsBuilder.html#stream-java.util.Collection- > ) > >

Re: KIP-382 + Kafka Streams Question

c/org/apache/kafka/streams/StreamsBuilder.html#stream-java.util.Collection- ) Likewise, a KTable can be materialized from multiple source topics -- in this case, userEntity, primary.userEntity and/or secondary.userEntity. You can think of these as parts of a "virtual topic", as you described.

Re: KIP-382 + Kafka Streams Question

dary Cluster: (Live) (renamed table to userEntity)* Topic: "primary.userEntity" (contains data from T = 0 to T = n) Topic: "userEntity" (contains data from T = n+1 to now, the failed-over producer) *Q2) How does a Kafka Streams application materialize st

Re: KIP-382 + Kafka Streams Question

umers, the Streams app should subscribe to any remote topics, e.g. with >> a regex, s.t. the application state will reflect input from either source >> cluster. >> >> This is essentially what Streams' "standby replicas" are -- extra copies >> of applicati

Re: KIP-382 + Kafka Streams Question

ting the app and rebuilding all the missing state, or you > could have a copy of everything sitting there ready when you need it. The > easiest way to do the latter is to run your app in both clusters. > > Hope that helps. > > Ryanne > > On Mon, Jul 22, 2019 at 3:11 PM Adam Be

Re: KIP-382 + Kafka Streams Question

Mon, Jul 22, 2019 at 3:11 PM Adam Bellemare wrote: > Hi Ryanne > > I have a quick question for you about Active+Active replication and Kafka > Streams. First, does your org /do you use Kafka Streams? If not then I > think this conversation can end here. ;) > > Secondly, and for t

KIP-382 + Kafka Streams Question

Hi Ryanne I have a quick question for you about Active+Active replication and Kafka Streams. First, does your org /do you use Kafka Streams? If not then I think this conversation can end here. ;) Secondly, and for the broader Kafka Dev group - what happens if I want to use Active+Active

Question about integrating kafka broker with a service

Hi, I am one of the Kafka users and I have a question about how to integrate our service with Kafka. Basically, we want to enable Kafka with TLS and we want to enable mutual authentication use SSL context. We’ve already got a service which will sign the cert and manage the key. Our goal is to

Re: Question on performance data for Kafka vs NATS

It appears that Kafka, despite taking the 2nd place, has a very low > > throughput. My question is, where does Kafka win over NATS? is it the > > unique partitioning and delivery semantics? Or, is it something else. > > > > From what I can see, NATS has traditional pub/sub and

Re: Question on performance data for Kafka vs NATS

https://nats.io/about/ > > this shows a general comparison of sender/receiver throughputs for NATS and > other messaging system including our favourite Kafka. > > It appears that Kafka, despite taking the 2nd place, has a very low > throughput. My question is, where does Kafka

Question on performance data for Kafka vs NATS

HI All, https://nats.io/about/ this shows a general comparison of sender/receiver throughputs for NATS and other messaging system including our favourite Kafka. It appears that Kafka, despite taking the 2nd place, has a very low throughput. My question is, where does Kafka win over NATS? is it

Kafka Topic Volume and (possibly ACL) question

Hello, We have a requirement where, based on business requirementes, we need to publish data only for a specific set of clients. For example, an invoice update shouldn't go to all clients, only the specific client. But a company remittance info should be published to all clients. Also, in some cas

Re: [Discuss] Question on KIP-298: Error Handling in Kafka Connect

> Hi, > a quick question on the KIP-298 Dead letter queue, as I read from the KIP > is only available for the Sink connectors. > > While I know the challenges of defining a dead-letter queue for the > incoming messages, I wanted ask/discuss what is the sense in here for this, >

[Discuss] Question on KIP-298: Error Handling in Kafka Connect

Hi, a quick question on the KIP-298 Dead letter queue, as I read from the KIP is only available for the Sink connectors. While I know the challenges of defining a dead-letter queue for the incoming messages, I wanted ask/discuss what is the sense in here for this, do you completely discard the

Re: A question about kafka streams API

d message. > > Because > > > the offset (t+10) is not committed, the next time you start the app it > > will > > > consume from offset t, and those messages will be processed again. This > > is > > > "at-least-once" processing. > >

Re: A question about kafka streams API

- > > 1) Use Kafka Streams with exactly-once semantics (though, as I am not > > familiar with your framework, it may support it as well). > > 2) Use idempotent practices (ie: it doesn't matter if the same messages > get > > processed more than once). > >

Re: A question about kafka streams API

> > > Hope this helps - > > Adam > > > On Wed, Sep 12, 2018 at 7:59 AM, Yui Yoi wrote: > > > Hi Adam, > > Thanks a lot for the rapid response, it did helped! > > > > Let me though ask one more simple question: Can I make a stream > application >

Re: A question about kafka streams API

doesn't matter if the same messages get processed more than once). Hope this helps - Adam On Wed, Sep 12, 2018 at 7:59 AM, Yui Yoi wrote: > Hi Adam, > Thanks a lot for the rapid response, it did helped! > > Let me though ask one more simple question: Can I make a stream applica

Re: A question about kafka streams API

Hi Adam, Thanks a lot for the rapid response, it did helped! Let me though ask one more simple question: Can I make a stream application stuck on an invalid message? and not consuming any further messages? Thanks again On Wed, Sep 12, 2018 at 2:35 PM Adam Bellemare wrote: > Hi Yui

Re: A question about kafka streams API

Hi Yui Yoi Preface: I am not familiar with the spring framework. "Earliest" when it comes to consuming from Kafka means, "Start reading from the first message in the topic, *if there is no offset stored for that consumer group*". It sounds like you are expecting it to re-read each message wheneve

A question about kafka streams API

TL;DR: my streams application skips uncommitted messages Hello, I'm using streams API via spring framework and experiencing a weird behavior which I would like to get an explanation to: First of all: The attached zip is my test project, I used kafka cli to run a localhost broker and zookeeper wha

Re: Question about connector rebalancing

Can you use stand-alone mode in that case? On Wed, Sep 5, 2018 at 7:12 PM, Chen He wrote: > Hi Kafka experts > > I have a question about connector rebalancing issue. Why don't we make it > option, I mean have a parameter that turn on/off it instead of having it as > a mus

Question about connector rebalancing

Hi Kafka experts I have a question about connector rebalancing issue. Why don't we make it option, I mean have a parameter that turn on/off it instead of having it as a must? We can have a parameter like: "connector.rebalancing.enable" parameter and make it as "true" by

  1   2   3   >