Re: Question about ResourcePatternFilter

Hi Claude, As I recall, the intention was that if PatternType is ANY, the name field would simply be ignored. Probably we should have thrown an exception if someone tried to create a ResourcePatternFilter with a non-null name and PatternType ANY. best, Colin On Tue, Aug 20, 2024, at 04:19, Cl

Re: Question about config.action.reload

Hi Ran, The config.action.reload, subscribe, unsubscribe, and ConfigChangeCallback features are partially-implemented. As far as I can tell if you're using only AK connect-runtime and config providers, none of it is functional. If you have a config provider that implements TTLs, the runtime will

RE: [QUESTION] What to do about conflicted KIP numbers?

al Message- From: Matthias J. Sax Sent: Friday, June 14, 2024 3:35 PM To: dev@kafka.apache.org Subject: Re: [QUESTION] What to do about conflicted KIP numbers? I don't think that there is an official guideline. Personally, I would suggest that the corresponding KIP owners agree who is keeping

Re: [QUESTION] What to do about conflicted KIP numbers?

I don't think that there is an official guideline. Personally, I would suggest that the corresponding KIP owners agree who is keeping the conflicting number, and how is changing it. For the ones changing the number, I would propose to restart a new DISCUSS thread using the new number to separ

Re: [Question] About Kafka producer design decision making

I should also clarify that Chia-Ping took it over from me originally and improved the proposal significantly. I think he also got busy and hasn't been able to spend time on it for a while though. Ismael On Tue, Jan 2, 2024 at 4:08 PM Ismael Juma wrote: > I had written a KIP a while back: > > h

Re: [Question] About Kafka producer design decision making

I had written a KIP a while back: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100829459 Happy for someone to pick it up and drive it forward. Ismael On Tue, Jan 2, 2024 at 2:00 PM Justine Olshan wrote: > Hey folks -- > I think this is a good conversation. Given we plan t

Re: [Question] About Kafka producer design decision making

Hey folks -- I think this is a good conversation. Given we plan to drop support for Java 8 in 4.0 this seems like a good time to consider this change. Perhaps we should file a JIRA ticket and maybe create a KIP to discuss? One thing we should consider however, is if we want some of these operatio

Re: [Question] About Kafka producer design decision making

hey sean - a lot of uses of the Futures are in the public API and therefore take voting/effort to be changed. i don't know any reason for intentionally avoiding the use of CompletableFuture, however, others might know more than I do. thanks, P On Tue, Nov 14, 2023 at 1:27 AM 신수웅(Sean Sin) wrote

Re: question

Hi 殿杰 In short, we don't support it now. But welcome to submit a PR to fix the gap. You can check this ticket for more information: https://issues.apache.org/jira/browse/KAFKA-7025 Thanks. Luke On Sat, Sep 23, 2023 at 2:14 AM shidian...@mxnavi.com wrote: > > hello， > > I‘m working inKafika

Re: [QUESTION] What is the difference between sequence and offset for a Record?

Thanks for your reply! I may not use "normalization". What I want to refer to is: appendInfo.setLastOffset(offset.value - 1) which underneath updates the base offset field (in record batch) but not the offset delta of each record. Best, tison. Justine Olshan 于2023年8月8日周二 00:43写道： > The sequ

Re: [QUESTION] about topic dimension quota

In fact, I have implemented the bytesIn/bytesOut limit of the topic dimension. I don't know the community's attitude towards this feature, so I don't know if I need to propose a KIP to contribute. best, hudeqi > -原始邮件- > 发件人: hudeqi <16120...@bjtu.edu.cn> > 发送时间: 2023-08-08 21:10:39 (星

Re: [QUESTION] What is the difference between sequence and offset for a Record?

The sequence summary looks right to me. For log normalization, are you referring to compaction? The segment's first and last offsets might change, but a batch keeps its offsets when compaction occurs. Hope that helps. Justine On Mon, Aug 7, 2023 at 8:59 AM Matthias J. Sax wrote: > > but the bas

Re: [QUESTION] What is the difference between sequence and offset for a Record?

but the base offset may change during log normalizing. Not sure what you mean by "normalization" but offsets are immutable, so they don't change. (To be fair, I am not an expert on brokers, so not sure how this work in detail when log compaction ticks in). This field is given by the produce

Re: [QUESTION] What is the difference between sequence and offset for a Record?

Hi Matthias and Justine, Thanks for your reply! I can summarize the answer as - Record offset = base offset + offset delta. This field is calculated by the broker and the delta won't change but the base offset may change during log normalizing. Record sequence = base sequence + (offset) delta. T

Re: [QUESTION] What is the difference between sequence and offset for a Record?

For what it's worth -- the sequence number is not calculated "baseOffset/baseSequence + offset delta" but rather by monotonically increasing for a given epoch. If the epoch is bumped, we reset back to zero. This may mean that the offset and sequence may match, but do not strictly need to be the sam

Re: [QUESTION] What is the difference between sequence and offset for a Record?

The _offset_ is the position of the record in the partition. The _sequence number_ is a unique ID that allows broker to de-duplicate messages. It requires the producer to implement the idempotency protocol (part of Kafka transactions); thus, sequence numbers are optional and as long as you don

Re: Question ❓

Partitions are not for different users. If you want to isolate users, you would do it at the topic level. You could use ACLs to grant access to different topics: https://kafka.apache.org/documentation/#security_authz -Matthias On 5/9/23 11:11 AM, влад тасканов wrote: Hi. I recently start­

Re: Question ❓

The current request queue is very single. In fact, there will be many performance problems when the business scenario of a single cluster becomes complicated. Not only to divide according to user, but also to isolate according to request category, this is just my idea. best, hudeqi > -原始邮

Re: Question: CI Pipeline in Kafka GitHub Pull Requests & Github Actions Usage

Dear David: I greatly appreciate your welcome and the explanation that you offered regarding the Jenkins system! I will try to familiarize myself with the standards and contribution process of the Kafka community, and look forward to making some contributions in the coming days. Best regards, Aa

Re: Question: CI Pipeline in Kafka GitHub Pull Requests & Github Actions Usage

Hello, Aaron and welcome to the project! If you look towards the bottom of a Pull Request there will be a section reporting the status of the Checks for the latest commit. There is a "Details" link that takes you to the Jenkins job for that PR. The default Jenkins view is the new UI (called Blue

Re: question

Hello jincheng, Kafka provides Java Producer/Consumer/Admin public api, so you can access Kafka if you are using Java to develop an Android App even though it's not common. for example, you can develop an Android App to get the Kafka metadata, send bury-point event log to Kafka, however, a better s

Re: Question about the Log Compaction

Hi, Liang, Currently, we store the MD5 of the record key in OffsetMap. Since it has a large domain (16 bytes), we assume there is no collision there. Thanks, Jun On Wed, Mar 2, 2022 at 1:20 AM 阮良 wrote: > Hi all > > I am confused about the Log Compaction logic，use OffsetMap > to deduplicating

Re: Question on kafka connect's incremental rebalance algorithm

Hi Ahmed, I think this bug KAFKA-12495 is the issue you described, which is under code review now. If not, please open another JIRA ticket to track it. Thanks. Luke On Tue, Apr 6, 2021 at 4:18 AM Thouqueer Ahmed < thouqueer.ah...@maplelabs.com>

Re: Question Regarding Offset Behavior When Calling Poll()

Thank you. Regards, Zhen Zhang Software Engineer [image: Twilio] MOBILE (949) 771-6073 EMAIL zzh...@twilio.com On Wed, Oct 7, 2020 at 11:35 PM Matthias J. Sax wrote: > I guess it's a question of the client implementation and you cannot > inf

Re: Question Regarding Offset Behavior When Calling Poll()

I guess it's a question of the client implementation and you cannot infer the behavior from the Java client. I would _assume_ that the offset is _not_ advanced. But the behavior is not define by Kafka itself, but it's up the the client implementation itself. Thus, only the client's docs or communi

Re: Question Regarding Offset Behavior When Calling Poll()

Hi, Sorry for the late reply, let me clarify on this. I am developing using Golang so I used a library based on librdkafka, and there's one function, ReadMesage(), which is a wrapper on top of the poll() function, except that it will only poll one message(record) at a time and return either one o

Re: Question Regarding Offset Behavior When Calling Poll()

I guess it depends where the exception comes from? Can you clarify? -Matthias On 9/23/20 12:53 AM, Zhen Zhang wrote: > Hi there, > > I am new to Kafka and I would like to get some clarifications for a newbie > question, > > Let's say if I have set up my consumer's "enable.auto.commit" to false,

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

Thanks Gwen and Ismael. On Wed, Jun 24, 2020 at 6:52 PM Gwen Shapira wrote: > Kafka follows a "fully compatible model" and new features are always > optional. We have extensive compatibility testing to make sure rolling > upgrades are safe and painless. > This investment in compatibility allows

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

Kafka follows a "fully compatible model" and new features are always optional. We have extensive compatibility testing to make sure rolling upgrades are safe and painless. This investment in compatibility allows us to reduce the effort needed in maintaining old versions (Security CVE is the obvious

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

I agree that is an option, but is there any reason to not have a 2.3.2 and 2.4.2 released? If so, it would be nice to know about these reasons. Appreciate your time on this. On Wed, Jun 24, 2020 at 6:35 PM Ismael Juma wrote: > I think there's some confusion here. You can upgrade to AK 2.5.0 and

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

I think there's some confusion here. You can upgrade to AK 2.5.0 and completely ignore ZK and TLS. It's completely optional. Ismael On Wed, Jun 24, 2020 at 10:20 AM Gokul Ramanan Subramanian < gokul24...@gmail.com> wrote: > Any updates on Kafka 2.3.2 and 2.4.2? Given the complexity of migrating

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

Any updates on Kafka 2.3.2 and 2.4.2? Given the complexity of migrating from non-encrypted TLS to encrypted TLS connection for ZooKeeper, it would be nice to have a bug-free version of 2.3 and 2.4. Is there a technical reason why we hesitate to get these versions out? Or is it that no one has got

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

Thanks Ismael for the response. For our clusters running 2.3.1 and 2.4.1, we saw some issues which had 2.3.2 and 2.4.2 as the fix versions. I looked at 2.5.0 but since it introduces some major changes like support for ZK encryption and a few others, I was wondering if we should choose a smaller up

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

Hi Sankalp, Is there a reason why you cannot upgrade to Apache Kafka 2.5.0 instead? We are working on the 2.5.1 release, which would be the recommended release. Ismael On Wed, Jun 24, 2020 at 6:18 AM Sankalp Bhatia wrote: > Hi All, > > I would like to know if there are any plans to release a 2

Re: Question about log flusher real frequency

Hi Alexandre, Thank you for your quick answer. I want to monitor it cause I'm trying to find out the reason why our existing Kafka cluster is configured to flush data every10 milliseconds! (people who configured it are not available anymore to answer). As that value seems really low to me, I was

Re: Question about log flusher real frequency

Hi Fares, On Linux kernels, you can use the property "dirty_writeback_centisecs" [1] to configure the period between executions of kswapd, which does this "sync" job. The period is usually set to 30 seconds. There are few exceptions where Kafka explicitly forces a sync (via the force() method from

Re: question about offsetSync

>From my opinion, condition 4 is downstream data loss because of master down without sync in time, so downstream data producer get smaller downstreamOffset. condition 3 meaning upstream broker down without sync in time condition 1 is init state so condition 2 may write like this: `downstreamTargetO

Re: Question on performance data for Kafka vs NATS

One more thing to note: You are looking at regular, base NATS. On its own, it is not a direct 1-1 comparison to Kafka because it lacks things like data retention, clustering and replication. Instead, you would want to compare it to NATS-Streaming, ( https://github.com/nats-io/nats-streaming-server

Re: Question on performance data for Kafka vs NATS

Thats a 4.5 year old benchmark and it was run with a single broker node and only 1 producer and 1 consumer all running on a single MacBookPro. Definitely not the target production environment for Kafka. -hans > On Mar 21, 2019, at 11:43 AM, M. Manna wrote: > > HI All, > > https://nats.io/ab

Re: Question about connector rebalancing

Can you use stand-alone mode in that case? On Wed, Sep 5, 2018 at 7:12 PM, Chen He wrote: > Hi Kafka experts > > I have a question about connector rebalancing issue. Why don't we make it > option, I mean have a parameter that turn on/off it instead of having it as > a must? > > We can have a par

Re: Question Regarding Apache Kafka replication

Hi Shibha, I have a KIP in the works which proposes to do this. Stay tuned. -Jason On Tue, Aug 14, 2018 at 4:43 PM, Shibha Malik wrote: > Hi Team, > > I wanted to know as to why Kafka does not support read by consumers from a > replica of a partition which is not a leader of the partition ( bu

Re: Question about issues of Kafka release version 1.1.1

Seems like you're right Lambdaliu. Rajini/Jason, can you please check and update the JIRAs? Ismael On Mon, Jul 23, 2018 at 7:09 AM lambdaliu(刘少波) wrote: > Hi team, > > I Have downloaded the source release of kafka version 1.1.1 and found the > JIRA > issues KAFKA-6911 and KAFKA-6809 listed in t

Re: Question about developer documentation

The web page is more about general project info and might be of interest to people beyond just developers. But I agree the wiki landing page could use some updating. Even more than just the developer section as we're missing several releases, the oldest ones are listed at the top, etc. -Ewen On F

Re: Question Kafka Reassign partitions tool

Hi Todd, That worked. Basically I used a python library to read the leaders of a topic/partition from the following path on zookeeper: /brokers/topics/[topic]/partitions/[partitionId]/state. Then I was able to get the correct leader and assign the replica set so that the current leader remains th

Re: Question Kafka Reassign partitions tool

Yes, the replicas are stored in Zookeeper, so you can iterate over the information there to build a view of the cluster that you can use. If you want an example for this, take a look at the code for kafka-assigner in https://github.com/linkedin/kafka-tools. Or you can just use that tool to adjust r

Re: Question Kafka Reassign partitions tool

Hi Todd, Thanks for the reply. Problem is I have about 160 topics(5 partitions for each) for which I need to increase the replication factors for. So, I would have to find the current leader for each of the partitions and hand code the json which would become tedious. The partition leader info is

Re: Question Kafka Reassign partitions tool

Preferred replica election is naive. It will always follow the order of the replicas as they are set. So if you want to set the default leader, just make it the first replica in the list for the partition. We build the JASON this way all the time. -Todd On Dec 22, 2017 6:46 AM, "Sagar" wrote:

Re: Question about fromLogical method in connect/api/src/main/java/org/apache/kafka/connect/data/Date.java

CharSyam, As I understood, it seems like Date#fromLogical assumes that the given Data object is created under UTC timezone - its unit test, DateTest, creates all related GregorianCalendar objects with TimeZone.getTimeZone("UTC"). So if all the Date objects given as the parameter are UTC-oriented,

Re: Question regarding Producer and Duplicates

Thanks Guozhang, So increasing batch.size can lead to more duplicates in case of failure. Also when you said : "The broker will accept a batch of records as a whole or reject them ". For example, if a producer request contains two batches, a first one for Topic A / Partition 0 and a second one fo

Re: Question regarding Producer and Duplicates

Hi Florian, The broker will accept a batch of records as a whole or reject them as a whole unless it encounters an IOException while trying to append the messages, which will be treated as a fatal error anyways. Duplicates usually happen when the whole batch is accepted but the ack was not delive

Re: question in regard to KIP-30

Well, from my point of view it at least makes sense to abstract out the implementation. Now any attempt to replace ZK with something else will lead to significant efforts. So, even providing of pluggable interface for the consensus and metadata would be extremely helpful and would enable other peop

Re: question in regard to KIP-30

I don't think it makes sense to change the core default implementation with KIP-30. Too much risk both in stability and in increasing the time to getting it done and available for folks that want to try Kafka without Zookeeper. It would be interesting to see how that implementation would work alon

Re: Question about sub-projects and project merging

Hi Jay, Looking at your question, I see the Apache Samza and Apache Kafka *communities* have little overlap(*). The Board looks at communities, and their overlap or lack thereof. Smushing two communities under one TLP is what we have historically called an "umbrella" TLP, and discourage. Communiti

Re: Question about concurrency during Log config change

Jay, Thanks for quick response. Yes, this might be not that harmful for users, I'm not sure about that. But it definitely looks like data race. Your solution is simple and should work, hard to tell promptly when it's about concurrency. Initially I was looking through this code to understand wheth

Re: Question about concurrency during Log config change

You are correct. Each read will be a valid value but there is no guarantee that subsequent reads will read from the same config. I don't think that is a problem, do you? If we want to strengthen the guarantee we can grab the config once in the method val config = log.config and then do however m

Re: Question about ZookeeperConsumerConnector

Thanks Jun! Jiangjie, could you file a JIRA? Thanks. Guozhang On Tue, Nov 11, 2014 at 9:27 AM, Jun Rao wrote: > Hi, Jiangjie, > > Thanks for the investigation. Yes, this seems like a real issue. > > 1. It doesn't seem that we need to put the shutdownCommand back into the > queue. Once an itera

Re: Question about ZookeeperConsumerConnector

Hi, Jiangjie, Thanks for the investigation. Yes, this seems like a real issue. 1. It doesn't seem that we need to put the shutdownCommand back into the queue. Once an iterator receives a shutdownCommand, it will be in a Done state and will remain in that state forever. 2. Yes, we just need to ge

Re: Question regarding topic creation

In 0.8.x, a topic doesn't necessarily get created on every broker. Each topic can have one or more partitions. Each partition can have one or more replicas. In general, the replicas are spread over the brokers in a cluster. The producer obtains the metadata of a topic to determine which broker to s

Re: Question regarding topic creation

By "created in all 10 brokers" you mean 10 partitions? Or 10 replicas for each partition? You can create a topic with either option, or both. By specifing partition number and number of replicas when creating the topic or by setting a default for all future topics. In my opinion, 10 replicas is

Re: question about message sets and produce requests/responses

Also, to be clear, by ack I mean an error code as described here: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ProduceResponse On Wed, Jul 23, 2014 at 04:33:29PM -0700, Joel Koshy wrote: > Not sure if I followed the question correctly, bu

Re: question about message sets and produce requests/responses

Ok, that's all I need to know. Thanks! On Wed, Jul 23, 2014 at 4:33 PM, Joel Koshy wrote: > Not sure if I followed the question correctly, but S1 and S2 would be > message-sets to different partitions of T - the producer request > object contains a map of topic-partition -> messageset). So eac

Re: question about message sets and produce requests/responses

Not sure if I followed the question correctly, but S1 and S2 would be message-sets to different partitions of T - the producer request object contains a map of topic-partition -> messageset). So each partition would get its own ack. The response object is just a map of partition -> response status

RE: Question about ConsoleConsumer

Hi Joe, Thanks for your answer, seems I know what's its purpose. Thanks a lot. Jerry -Original Message- From: Joe Stein [mailto:joe.st...@stealth.ly] Sent: Thursday, July 17, 2014 10:15 AM To: dev@kafka.apache.org Cc: us...@kafka.apache.org Subject: Re: Question about ConsoleCon

Re: Question about ConsoleConsumer

auto.offset.reset is only when there is not an already established offset to consume from or an offset that no longer exists (and is invalid) in the case of truncated logs or compaction). You can force this behavior by deleting the znode /*** Joe Stein Fo

Re: Question on Kafka .8 / Logstash client

I am not sure what you are trying to achieve but here is this: https://github.com/joekiller/logstash-kafka On Thu, Jun 12, 2014 at 11:06 AM, Sutanu Das wrote: > Hi Kafka Users, > > > 1. Is there a plugin/client for Logstash for Kakfa 0.8 ? > > > 2. Is there any example of using Kafka

Re: question about produce requests and responses

The topic/partition ordering in the producer response may not be the same as that in the request. In each request, there is supposed to be one instance per topic/partition. Thanks, Jun On Wed, Jun 11, 2014 at 5:27 PM, Dave Peterson wrote: > Hello, I have a question about produce requests and

Re: Question about Kafka code contribution guidelines

All committers have to file the following agreement: http://www.apache.org/licenses/icla.txt Contributor submits patches to a jira. The patches will be reviewed and if accepted, they will be checked in by a committer. If a contributor makes significant contribution, he/she will be nominated as a c

Re: Question about offsets

Thanks, Jay. Makes sense +1 for documentation. I think in general the wiki is a bit unorganized, makes it hard to know what is the "current" info vs old proposals. -David On 1/30/13 12:01 AM, Jay Kreps wrote: Yes, offsets are now logical 0,1,2,3... Some details on this change are here: http

Re: Question about offsets

Yes, offsets are now logical 0,1,2,3... Some details on this change are here: https://cwiki.apache.org/confluence/display/KAFKA/Keyed+Messages+Proposal https://issues.apache.org/jira/browse/KAFKA-506 https://issues.apache.org/jira/browse/KAFKA-631 There was some previous threads on this that disc