RE: EXTERNAL: Re: Backup Apache KAFKA

Hi, We love Kafka but I would pretty strongly disagree with that statement. Backups are not really about recovering from local failures and lots of systems that are resilient and redundant have backups as well (e.g. CockroachDB). Uses of backups in a system like Kafka would include: 1) Recover

RE: EXTERNAL: Re: Separate Kafka partitioning from key compaction

fka Streams app that implemented your desired partitioning? Obviously this would require a duplication in topics between those produced to initially, and those partitioned the way you'd like, but it would solve the issue you're having. On Wed, 6 May 2020, 10:25 pm Young, Ben, wrote:

Separate Kafka partitioning from key compaction

Hi, We have a use case where we'd like the partition a key is hashed to, to be a subset of the keys that are used for compaction. It would be really cool if there was a built in hashing strategy that could help us as we're potentially using Kafka from multiple languages and it could be hard to

RE: Copy topic to offline file

ment? Francis On Thu, Jun 7, 2018 at 7:47 PM, Young, Ben wrote: > Hi > > We have a requirement that we need to be able to copy a portion of a Kafka > topic to a file (perhaps between a range of timestamps) which can then be > loaded into a Kafka instance on a different machi

Copy topic to offline file

Hi We have a requirement that we need to be able to copy a portion of a Kafka topic to a file (perhaps between a range of timestamps) which can then be loaded into a Kafka instance on a different machine. This machine may have a different number of partitions etc for the topic. We can't connect

RE: Issue deleting topic on Windows

91c72cdc5374631599708d5902af694%7Ce3ff91d834c84b15a0b418910a6ac575%7C0%7C0%7C636573436557774999&sdata=V0dXffpRMAvBU8vyQH6ZW71WpmuhFGB4uo5vQdMs1cM%3D&reserved=0 Hope this helps! Lawrence On Thu, Mar 22, 2018 at 9:34 AM, Young, Ben wrote: > Hi, > > Any thoughts on the below? > > Than

RE: Issue deleting topic on Windows

Hi, Any thoughts on the below? Thanks! -Original Message- Sorry, I forgot to mention we're using version: kafka_2.12-1.0.0 -Original Message- Hi, We have an issue in our dev environments when running on Windows. To provide a clean environment for testing we tear down the top

RE: Issue deleting topic on Windows

Sorry, I forgot to mention we're using version: kafka_2.12-1.0.0 -Original Message- From: Young, Ben [mailto:ben.yo...@fisglobal.com] Sent: 20 March 2018 09:37 To: users@kafka.apache.org Subject: Issue deleting topic on Windows Hi, We have an issue in our dev environments when ru

Issue deleting topic on Windows

Hi, We have an issue in our dev environments when running on Windows. To provide a clean environment for testing we tear down the topics and recreate them, but deleting the topics is causing the server to crash. We then need to delete the logs directory for it to be able to restart, it doesn't

Kafka control messages

Hi, We have a situation where we are streaming data in a Kafka topic to be processed. The topic has multiple partitions. From time to time we need to add control messages to the stream and this needs to be coherent with the data steam (e.g. "replay from point X to this new location"). Because o

RE: Partition groups

block of partitions around automatically but still have the IO scaleout. I'm sure I can even think of a use for the supplementary topic! -Original Message- From: Young, Ben [mailto:ben.yo...@fisglobal.com] Sent: 17 May 2017 19:09 To: users@kafka.apache.org Subject: Partition groups

Partition groups

Hi I was wondering if something like this was possible. I'd like to be able to use partitions to gain some IO parallelism, but certain sets of partitions should not be distributed across different machines. Let's say I have data that can be processed by time bucket, but I'd like each day's data

RE: Locality question

Locality question > > If you have a kafka partition that is replicated to 3 nodes the partition > varies > (in time) thus making the colocation pointless. You can only produce and > consume to/from the leader. > > /svante > > > > 2015-11-12 9:00 GMT+01:00 You

RE: Locality question

Hi, Any thoughts on this? Perhaps Kafka is not the best way to go for this, but the docs do mention transaction/replication logs as a use case, and I'd have thought locality would have been important for that? Thanks, Ben -Original Message- From: Young, Ben [mailto:b

RE: Locality question

community Thanks, Prabhjot On Fri, Nov 6, 2015 at 1:50 PM, Young, Ben wrote: > Hi, > > I've had a look over the website and searched the archives, but I > can't find any obvious answers to this, so apologies if it's been asked > before. > > I'm inve

Locality question

est all round, and a really like the idea of just being able to set the log offset to zero to reload on startup. Thanks, Ben Young Ben Young . Principal Software Engineer . Adaptiv .