Re: Neverending KafkaStreams rebalance

unsubscribe On Thu, 14 Oct 2021 at 18:04, Murilo Tavares wrote: > Hi Guozhang > Thanks for your response. I'm on KafkaStreams 2.8.1. > Since you asked, is KafkaStreams 3.0.0 compatible with a 2.4.1 broker? > > But I found the issue. > TL;DR: I increased max.poll.interval.ms and decreased max.pol

Re: Delayed processing

ocess anonymised messages? > > > Guozhang > > > On Thu, Mar 8, 2018 at 12:35 PM, Wim Van Leuven < > wim.vanleu...@highestpoint.biz> wrote: > > > Hello, > > > > I'm wondering how to design a KStreams or regular Kafka application that > > can h

Re: Delayed processing

ly > after make your job for each message (or commit several processed messages > at one time and kept in a local memory buffer) > > > I hope, these previous suggestions help you 😊 > > > Best regards, > > Adrien > > > De : Wim Van Leuv

Delayed processing

Hello, I'm wondering how to design a KStreams or regular Kafka application that can hold of processing of messages until a future time. This related to EU's data protection regulation: we can store raw messages for a given time; afterwards we have to store the anonymised message. So, I was thinki

Re: Kafka per topic retention.bytes and global log.retention.bytes not working

awesome! On Thu, 11 Jan 2018 at 23:48 Thunder Stumpges wrote: > Thanks, yes we upgraded to 1.0.0 and that has indeed fixed the issue. > Thanks for the pointer! > -Thunder > > > On Tue, Jan 9, 2018 at 9:50 PM Wim Van Leuven < > wim.vanleu...@highestpoint.biz> wrote: &g

Re: Kafka per topic retention.bytes and global log.retention.bytes not working

Upgrade? On Wed, Jan 10, 2018, 00:26 Thunder Stumpges wrote: > How would I know if we are seeing that issue? We are running 0.11.0.0 so we > would not have this fix. > > On Tue, Jan 9, 2018 at 11:07 AM Wim Van Leuven < > wim.vanleu...@highestpoint.biz> wrote: > > &g

Re: Kafka per topic retention.bytes and global log.retention.bytes not working

What minor version of Kafka are you running? Might you be impacted by https://issues.apache.org/jira/browse/KAFKA-6030? -w On Tue, 9 Jan 2018 at 19:02 Thunder Stumpges wrote: > Hello, I posted this on StackOverflow > < > https://stackoverflow.com/questions/47948399/kafka-per-topic-retention-byte

log.deleted

Hello, When Kafka evicts a segment, it renames the files by appending .deleted. In configuration you can also specify how long to keep these files laying around. What's the actual use of keeping the .delete files around? Can you 'restore' the deleted data from these files after e.g. incrementing

Re: Topic segments being deleted unexpectedly

OK. tnx! On Fri, 15 Dec 2017 at 15:08 Damian Guy wrote: > I believe that just controls when the segment gets deleted from disk. It is > removed from memory before that. So i don't believe that will help. > > On Fri, 15 Dec 2017 at 13:54 Wim Van Leuven < > wim.vanleu...@hi

Re: Topic segments being deleted unexpectedly

data. Could we achieve this result by increasing the log.segment.delete.delay.ms to e.g. 1d? Would this give us a timeframe of a day to process the data on the intermediary topics? Or is this just wishful thinking? Thanks again! -wim On Fri, 15 Dec 2017 at 14:23 Wim Van Leuven wrote: > Is

Re: Topic segments being deleted unexpectedly

gt; Damian > > On Fri, 15 Dec 2017 at 11:49 Wim Van Leuven < > wim.vanleu...@highestpoint.biz> > wrote: > > > Hello all, > > > > We are running some Kafka Streams processing apps over Confluent OS > > (v3.2.0) and I'm seeing unexpected but 'consit

Topic segments being deleted unexpectedly

Hello all, We are running some Kafka Streams processing apps over Confluent OS (v3.2.0) and I'm seeing unexpected but 'consitent' behaviour regarding segment and index deletion. So, we have a topic 'input' that contains about 30M records to ingest. A 1st processor transforms and pipes the data on

Re: Kafka in virtualized environments

We are running kafka on openstack for a testing/staging environment. It runs good and stable, but it obviously is way slower than bare-metal. Simple reason is the distance to the disk (as with any IO batch oriented system on virtualisation) and virtual network. HTH -wim On Thu, 30 Nov 2017 at 1

Re: Kafka & Canary Release

Isn't that simply a matter of changing your key to route the element to the right topic for 1 or the other microservice? -w On Wed, 29 Nov 2017 at 17:09 Yuval Alon wrote: > Hello, > We are using Apache Kafka 0.10.2, which we understand is not supporting > “Canary Release

Re: Plans to extend streams?

What you are actually asking is if Kafka Streams should be reimplemented as Apache Storm? -wim On Wed, 29 Nov 2017 at 15:10 Adrienne Kole wrote: > Hi, > > The purpose of this email is to get overall intuition for the future plans > of streams library. > > The main question is that, will it be a

Re: GDPR appliance

Thanks, Lars, for the most interesting read! On Sun, 26 Nov 2017 at 00:38 Lars Albertsson wrote: > Hi David, > > You might find this presentation useful: > https://www.slideshare.net/lallea/protecting-privacy-in-practice > > It explains privacy building blocks primarily in a batch processing >

Re: GDPR appliance

doesn't try to boil the ocean. > > On Thu, Nov 23, 2017 at 9:31 AM Wim Van Leuven < > wim.vanleu...@highestpoint.biz> wrote: > > > I think the best way to implement this is via envelope encryption: your > > system manages a key encryption key (kek) which is used to e

Re: GDPR appliance

I think the best way to implement this is via envelope encryption: your system manages a key encryption key (kek) which is used to encrypt data encryption keys (dek) per user/customer which are used to encrypt the user's/customer's data. If the user/customer walks away, simply drop the dek. His da

1 topic that keeps growing indefinitely

Hello, we have a 3 node kafka cluster setup with quite a bunch of topics that have a nice life: regular cleans, compact, etc. 1 topic however keeps growing indefinitely. New segments are recreated at regular intervals, but old segments are never deleted. Our setup is based on Confluent 3.2.0 OSS,

1 topic that keeps growing

Hello, we have a 3 node kafka cluster setup, with quite a bunch of topics, that have a nice life, regular cleans, compact, etc. 1 topic however keeps growing, new segments are recreated at regular intervals, but old segments are never deleted. Our setup is based on Confluent 3.2.0 OSS, hence Apac