Hey Winstein,
1. You can’t modify events after they are written. They are immutable.
2. Consumers always see messages in order. Note this guarantee is per
partition. That is, the consumer is guaranteed to read messages in order for a
given partition. There isn’t a guarantee across partitions.
; > > Sunil.
> > > >
> > > > On Thu, 8 Jun 2023 at 3:59 PM, Edvard Fagerholm <
> > > > edvard.fagerh...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > I couldn't find an answer in the documentation to the following. If
> > a new
> > > > > machine joins a consumer group and Kafka triggers a rebalance, will
> > it
> > > > > randomly reassign partitions or will it hand over partitions from
> > > > existing
> > > > > consumers to the newly joined one? In other words, will it attempt
> to
> > > > move
> > > > > as few partitions as possible between consumers?
> > > > >
> > > > > The main implications of this is local in-memory caches and scaling
> > up
> > > > the
> > > > > number of machines in a consumer group, since the scaling up
> > operation
> > > > will
> > > > > require nuking any local caches for the partitions that were moved.
> > This
> > > > > would cause a spike on any DBs that are being cached on the
> > consumers.
> > > > >
> > > > > Best,
> > > > > Edvard
> > > > >
> > > >
> >
>
--
Andrew Grant
8054482621
Hey David,
The followers replicate from the leader and when they do that they write to
their own local log. For the ceph cluster, it sounds like the followers
writes to their local log are slower? Seems like that would sense if those
writes are going over the network. This could explain why the le
xample would give him a replication factor of 2 though, and it
> sounds like he wants 3 unless I missed something. So add an additional
> broker id to each of the replicas arrays in your example and you'd have an RF
> of 3.
>
> -alex
>
>> On Sat, Dec 17, 2022 at 7:12 A
22 at 6:54 PM Chris Peart wrote:
> Hi Andrew,
> Would you be able to provide an example of the json with all the
> partitions in please, I tried this on our dev cluster but it didn’t work.
> Many Thanks,
> Chris
>
> > On 16 Dec 2022, at 9:03 pm, Andrew Grant
> wrote:
&g
ew,
>
> Thanks for the speedy reply, so do I just need to do this for partition
> 49? What about partitions 0-48, will these be covered by reassigning
> partition 49.
>
> Do I need to do this for the __transaction_state topics too?
>
> Many thanks,
> Chris
>
>
Hey Chris,
I think you should be able to use the reassignment tool to add replicas.
You should be able to do something similar to migrate the partitions away
from the old brokers and onto the new ones and also increase the
replication factor at the same time. I tested just increasing the
replicatio
nks for the reply. Currently, we are using the same method as you
> > described. Wanted to make sure if there is a better way.
> >
> > It seems there isn't currently. So we will keep using this only.
> >
> > On Tue, Oct 25, 2022 at 7:23 PM Andrew Grant >
&g
Hey Lehar,
I don’t think there’s a way to control this during topic creation. I just
took a look through
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/AdminUtils.scala
and it does appear partition assignment does not account for each broker’s
different log directories
it's no longer the controller? And when it does realize this, does
it shutdown or does it maybe log something and continue on with its other
duties? I suspect the latter but wanted to check.
Links to any resources that might answer the question would also be
helpful!!
Thanks,
Andrew
--
An
opic partitions on kafka. Currently, all
> topics are 6 partitions. I want to reduce it to 3 for some of the topics.
> Is this supported by kafka without deleting the topic?
>
--
Andrew Grant
8054482621
; "attr3" :"value_3" }
>
> Key : 1 Message: { "attr1": "value_x", "attr2": "value2" ,
> "attr3" : "value_y"}
>
> Key : 2 Message: { "attr1": "value_m", "attr2": null , "attr3" :
> "value_o"}
>
>
> Now, I need the resultant JSON as below.
>
> { "attr1": "value_m", "attr2": "value2" , "attr3" : "value_o"}
>
>
> Could someone help me by letting me know the solution? Thank you.
>
> Thanks
>
> C Suresh
>
--
Andrew Grant
8054482621
12 matches
Mail list logo
