Re: High CPU Usage on Brokers

Not sure what version your producers/consumers are, or if you upgraded from a previous version that used to work, or what, but maybe you're hitting this? https://kafka.apache.org/23/documentation.html#upgrade_10_performance_impact On Mon, Jan 6, 2020 at 12:48 PM Navneeth Krishnan wrote: > Hi

Re: Kafka per topic retention.bytes and global log.retention.bytes not working

Thanks, yes we upgraded to 1.0.0 and that has indeed fixed the issue. Thanks for the pointer! -Thunder On Tue, Jan 9, 2018 at 9:50 PM Wim Van Leuven < wim.vanleu...@highestpoint.biz> wrote: > Upgrade? > > On Wed, Jan 10, 2018, 00:26 Thunder Stumpges > wrote: > > >

Re: Best practice for publishing byte messages to Kafka

Byte Array is essentially "serialized" already isn't it? I mean the message itself is sent as a byte array, so the default byte array serializer is as "efficient" as it gets, as it's just sending your byte array through as the message... there's no serialization happening. -Thunder On Tue, Jan 9,

Re: Kafka per topic retention.bytes and global log.retention.bytes not working

che.org/jira/browse/KAFKA-6030? > -w > > On Tue, 9 Jan 2018 at 19:02 Thunder Stumpges > wrote: > > > Hello, I posted this on StackOverflow > > < > > > https://stackoverflow.com/questions/47948399/kafka-per-topic-retention-bytes-and-global-log-retention-byt

Kafka per topic retention.bytes and global log.retention.bytes not working

Hello, I posted this on StackOverflow also but haven't gotten any response. thanks in advance, Thunder __ We are running a 6 node cluster of kafka 0.11.0. We have set a

RE: Problem with node after restart no partitions?

+ not recovering properly). Any thoughts jiangjie? Jason On Mon, Apr 6, 2015 at 7:30 PM, Thunder Stumpges wrote: > Hi all, > > I have collected our logs and paired them down. Jiangjie, I sent you > more complete logs directly to your mailto from the list (minus the .INVALID). > P

RE: Problem with node after restart no partitions?

not sure what's up. Any ideas? It makes us more than a little nervous to restart nodes if they will just "decide" to delete segments. Under what conditions would this happen? Thanks! Thunder -Original Message- From: Thunder Stumpges [mailto:tstump...@ntent.com] Sen

RE: Problem with node after restart no partitions?

e uncleanly shutdown). That did not happen > >this time (the rest of the cluster got in an apparent infinite loop > >where it tried repeatedly (e.g. > >500K times a minute) to fetch partitions that were unavailable). > > > >I'm preparing a longer post with more d

RE: Problem with node after restart no partitions?

o any kafka service restarts now that this has happened twice on this machine. Any suggestions? Should we go back and make another attempt to upgrade to 0.8.2.1 ? Thanks, Thunder From: Thunder Stumpges Sent: Wednesday, April 01, 2015 10:30 AM To: kafka-us...@incubator.apache.org Subject: Problem with

Problem with node after restart no partitions?

Hi group, We've been trying to track down a problem this morning for a little while, and thought I'd ask here while we keep looking. We have 3 nodes (rep-3) running 8.1.1. We attempted a rolling upgrade yesterday to 8.2.1, and on the first node, after restarting, a single topic (a samza interm

RE: Alternative to camus

Sorry to go back in time on this thread, but Camus does NOT use YARN. We have been using camus for a while on our CDH4 (no YARN) Hadoop cluster. It really is fairly easy to set up, and seems to be quite good so far. -Thunder -Original Message- From: amiori...@gmail.com [mailto:amiori..

RE: Trying to get kafka data to Hadoop

What branch of camus are you using? We have our own fork that we updated the camus dependency from the avro snapshot of the REST Schema Repository to the new "official" one you mention in github.com/schema-repo. I was not aware of a branch on the main linked-in camus repo that has this. That be

RE: Stream naming conventions?

kafka-us...@incubator.apache.org Subject: Re: Stream naming conventions? Can you provide some examples on your naming patterns described below? Thanks ** julio On 3/3/15, 6:56 AM, "Thunder Stumpges" wrote: >I'm not sure who you were asking the question to, but since Gwen&#x

RE: Stream naming conventions?

like: > etl.onlineshop.searches.validated > > Or if I have my own test dataset that I don't want to share: > users.gshapira.newapp.testing1 > > Makes it relatively easy to share datasets across the organization, and > also makes white-listing and black-listing relatively simple because

RE: Stream naming conventions?

We have a global namespace hierarchy for topics that is exactly our Avro namespace with Class Name. The template is basically: .Core. .. The up side of this for us is that since the topics are named based on the Avro schema namespace and type, we can look up the avro schema in the Avro Schema

RE: Having trouble with the simplest remote kafka config

The producer machine lists 'localhost:9092' for the Kafka connection? They're on two different machines aren't they? -Original Message- From: Gwen Shapira [gshap...@cloudera.com] Received: Tuesday, 17 Feb 2015, 8:57PM To: users@kafka.apache.org [users@kafka.apache.org] Subject: Re: Having

RE: Turning on cleanup.policy=compact for a topic => not starting cleanup ?

al Message----- From: Thunder Stumpges [mailto:tstump...@ntent.com] Sent: Tuesday, February 03, 2015 4:02 PM To: users@kafka.apache.org Subject: RE: Turning on cleanup.policy=compact for a topic => not starting cleanup ? Thanks Joel. The jira issue mentions a workaround to "remove

RE: Turning on cleanup.policy=compact for a topic => not starting cleanup ?

apache.org Subject: Re: Turning on cleanup.policy=compact for a topic => not starting cleanup ? Oh yes I forgot about that possibility. I think this should be fixed by KAFKA-1641 which will be in the next release. On Tue, Feb 03, 2015 at 11:50:54PM +0000, Thunder Stumpges wrote: > AH hah! The

RE: Turning on cleanup.policy=compact for a topic => not starting cleanup ?

52 for log dev_vsw.avrodto.addelivery.machinelearning.adbanditmodel-9." Thanks, Thunder -Original Message- From: Thunder Stumpges [mailto:tstump...@ntent.com] Sent: Tuesday, February 03, 2015 3:37 PM To: users@kafka.apache.org Subject: RE: Turning on cleanup.policy=compact for a topic

RE: Turning on cleanup.policy=compact for a topic => not starting cleanup ?

the log cleaner is still alive? - Also, there is an mbean that you can poke: "max-dirty-percent" under log cleaner. Can you check on its value? Thanks, Joel On Tue, Feb 03, 2015 at 10:59:27PM +, Thunder Stumpges wrote: > Hi guys, I am having a difficult time getting the log c

Turning on cleanup.policy=compact for a topic => not starting cleanup ?

Hi guys, I am having a difficult time getting the log compaction to run on a topic I created initially with cleanup.policy=delete. Here's the details: 3 brokers, 0.8.1.1 all have log.cleaner.enable=true (other than this there are no non-defaults set in our server.properties) Create the topic fi

RE: per-topic configuration names... unit suffix ?

ge("log.retention.minutes", (1, Int.MaxValue)) } else { millisInHour * props.getIntInRange("log.retention.hours", 24*7, (1, Int.MaxValue)) } Cheers, Thunder -Original Message- From: Thunder Stumpges [mailto:tstump...@ntent.com] Sent: Thursday, J

RE: LinkedIn Engineering Blog Post - Current and Future

/kafka/kafka-linkedin-current-and-future Aditya From: Thunder Stumpges [tstump...@ntent.com] Sent: Thursday, January 29, 2015 2:43 PM To: users@kafka.apache.org Subject: RE: LinkedIn Engineering Blog Post - Current and Future Thanks Jon. That link isn't wo

RE: LinkedIn Engineering Blog Post - Current and Future

Thanks Jon. That link isn't working for me though: "Page not found The requested page "/kafka/kafka-linkedin-%E2%80%93-current-and-future" could not be found." -Original Message- From: Jon Bringhurst [mailto:jbringhu...@linkedin.com.INVALID] Sent: Thursday, January 29, 2015 10:23 AM To

per-topic configuration names... unit suffix ?

Hi all, I am reading about the per-topic configs and the unit suffixes on a couple don't seem to match to the global/server ones. Specifically: retention.ms => log.retention.minutes segment.ms => log.roll.hours Can someone clear this up

RE: Kafka in C#

ad to help you out however we can. Cheers, Thunder -Original Message- From: Matti Waarna [mailto:mwaa...@sapient.com] Sent: Friday, January 09, 2015 9:43 AM To: users@kafka.apache.org; Thunder Stumpges Subject: RE: Kafka in C# Thunder, Have you had a chance to go to production with

RE: Kafka 0.8.1.1 eadership changes are happening very often

Did you check the GC logs in the server? We ran into this and the default setting of 1G max heap on the broker process was nowhere near enough. We currently have set to 4G. -T -Original Message- From: Birla, Lokesh [lokesh.bi...@verizon.com] Received: Monday, 22 Dec 2014, 5:27PM To: user

RE: Kafka in C#

@kafka.apache.org [users@kafka.apache.org] Subject: Re: Kafka in C# Thunder, can you add that to https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-.net didn't know it existed but cool that it uses Rx On Mon, Dec 22, 2014 at 12:52 PM, Thunder Stumpges wrote: > Hi there, > > We loo

RE: Kafka in C#

Hi there, We looked at both of these a while back and ended up writing our own (https://github.com/ntent-ad/kafka4net). The first one in the list was completely synchronous, and had no concept of batching. We initially attempted to use the second one (kafka-net) but had some issues with detec

RE: Trying to figure out kafka latency issues

bly with the producer and not the broker. Only restarting the producer made things better. I'll do more extensive measurement on the broker. On Sun, Dec 21, 2014 at 9:08 AM, Thunder Stumpges wrote: > > Did you see my response and have you checked the server logs especially > the GC

RE: Trying to figure out kafka latency issues

Did you see my response and have you checked the server logs especially the GC logs? It still sounds like you are running out of memory on the broker. What is your max heap memory and are you thrashing once you start writing to all those partitions? You have measured very thoroughly from an ext

RE: Trying to figure out kafka latency issues

That's a pretty detailed analysis, I'll be very interested to see what the root cause is. Have you had a look at the broker GC logs? The spike in cpu and the long tail on the latency make me think garbage collection pauses. I suppose the large number of partitions may have increased the memory

RE: Producer can writes to a follower during preferred lead election?

In this case B will return "not leader for partition" error as soon as the leader is re-elected and I imagine the producer will correct itself. -Thunder -Original Message- From: Xiaoyu Wang [xw...@rocketfuel.com] Received: Saturday, 06 Dec 2014, 6:49PM To: users@kafka.apache.org [users@

RE: [DISCUSSION] adding the serializer api back to the new java producer

I'm not sure I agree with this. I feel that the need to have a consistent, well documented, shared serialization approach at the organization level is important no matter what. How you structure the API doesn't change that or make it any easier or "automatic" than before. It is still possible fo

RE: [DISCUSSION] adding the serializer api back to the new java producer

Hello, while we do not currently use the Java API, we are writing a C#/.net client (https://github.com/ntent-ad/kafka4net). FWIW, we also chose to keep the API simpler accepting just byte arrays. We did not want to impose even a simple interface onto users of the library, feeling that users will

Drawbacks to setting log.roll.hours=1 ?

Hello, We have a 5 node cluster, each with a 5 disk JBOD storage running Kafka 0.8.1. We are interested in the possibility of using the OffsetRequest API taking a time to get offsets from a given time. Because that API can only "find" times where the log segments roll, we would need to set log.

RE: Using OffsetRequest to get the Head and Tail of a partition in a single request.

Nov 18, 2014, at 8:52 PM, Thunder Stumpges wrote: > > Hey all, > > We are working on a .net client, and I have a question about the > OffsetRequest api > (https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI) > >

Using OffsetRequest to get the Head and Tail of a partition in a single request.

Hey all, We are working on a .net client, and I have a question about the OffsetRequest api (https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI) It seems to indicate that you would always get the "log end offset" (tail offset) reg