Hi,
I'm reading the documentation of 3.8.0 to see if we can migrate our Kafka
cluster running with ZK and JBOD to Kraft.
The documentation is not clear about that:
- it mentions in https://kafka.apache.org/documentation/#upgrade_3_8_0 that
"JBOD in KRaft is no longer in early access."
- it mentio
Hi,
We have a Kafka cluster that's growing a lot year to year and I was
wondering how much it can handle without custom changes.
If you're running a large Kafka cluster or if you're aware of one, please
share some data around it:
- how many brokers is in the cluster and CPUs / RAM used
- how many
Hi,
How can I know if I have enough network threads configured with
num.network.threads?
I see Producers are slower to produce when we move partitions between
brokers, so I'm thinking maybe there isn't enough network threads to
process the requests from both the consumers and from the other broke
Hey,
we had the same issue as you.
I checked the code and it chooses the first live replica from the
assignment list. So if you describe a topic with kafka-topics, you will see
the brokers list that has the replica of each partition. For example:
[1001, 1002, 1003]. If that is the list, Kafka wil
0 version.
> >
> > [image: image.png]
> > Regards.
> >
> >
> > On Fri, Feb 26, 2021 at 2:54 PM Péter Sinóros-Szabó
> > wrote:
> >
> >> Hi,
> >>
> >> No, CPU increase shouldn't be there. Upgrades usually bring lower CPU
> &
Thats what I understood reading the documentation:
>
> https://kafka.apache.org/documentation/#upgrade
>
> Best Regards!
>
>
> On Fri, Feb 26, 2021 at 2:19 PM Péter Sinóros-Szabó
> wrote:
>
> > Hi,
> >
> > I just upgraded from Kafka 2.4.1 to 2.6.1 and I s
h.svg 8983
>
>
> ... where 8983 is the pid of the Kafka process.
>
> ... and then it spat out a beautiful interactive flame chart.
>
>
> On Fri, Feb 26, 2021 at 10:26 AM Péter Sinóros-Szabó
> wrote:
>
> > Hi,
> >
> > I just upgraded from Kafka 2.4.1 t
Hi,
I just upgraded from Kafka 2.4.1 to 2.6.1 and I see huge CPU usage on the
broker after the upgrade. Upgrade in this case means that I only bumped the
broker version on 1 of the brokers out of the 6 and didn't change the
protocol or message format versions. Before the upgrade, it used about 35%
Hey Sophie,
thanks for the link, I was checking that ticket, but I was not sure if it
is relevant for our case.
Eventually we "fixed" our problem with reducing the session.timeout.ms (it
was set to a high value for other reasons).
But today, in another service, we faced the same problem when upgr
Hi,
I have an application running with 6 instances of it on Kubernetes. All 6
instances (pods) are the same, using the same consumer group id.
Recently we see that when the application is restarted (rolling restart on
K8s), the triggered rebalancing sometimes doesn't finish at all and the
Kafka Cl
Hi,
I tried as well to stop all instances of MM2, but it didn't help for me.
I had to stop all MM2 instances, delete the mm2-config and mm2-status
topics on the destination cluster and start up all MM2 instances again.
Peter
from mm2.properties, rather
> than reading the old configuration from mm2-config and getting stuck.
>
> Can someone report the issue in jira?
>
> Ryanne
>
> On Wed, Nov 11, 2020 at 9:35 AM Péter Sinóros-Szabó
> wrote:
>
> > Hi,
> >
> > I have a similar issue.
Hi,
I have a similar issue. I changed the source cluster bootstrap address and
MM2 picked it up only partially. Some parts of it still use the old
address, some the new. The old and the new address list is routed to the
same cluster, same brokers, just on a different network path.
So is there an
restart at the latest commit. If you are
> replicating something like CDC or changelogs, then dupes might be fine
> (downstream state will be eventually consistent). That's a common pattern
> with MM1 as well.
>
> Ryanne
>
> On Mon, Apr 27, 2020 at 4:47 AM Péter Sinóros-S
Hey Ryanne,
Is there any documentation where I can read more about this "slightly
out-of-order records"?
It would help very much to see how we can use MM2 in our systems.
Thanks,
Peter
On Thu, 23 Apr 2020 at 08:56, Péter Sinóros-Szabó <
peter.sinoros-sz...@transferwise.com> wrot
cords are dropped or skipped, even during consumer failover/migration
> etc, but we can still have duplicates and slightly out-of-order records in
> the downstream remote topics, for now.
>
> Ryanne
>
> On Wed, Apr 22, 2020 at 3:39 AM Péter Sinóros-Szabó
> wrote:
>
> > He
Hey,
so KIP-382 mentions that:
"Partitioning and order of records is preserved between source and remote
topics."
is the ordering of messages (I guess only in a partition) something that is
actually implemented in 2.4 (or in 2.5)?
Or do I need to set `max.in.flight.requests.per.connection=1` ?
T
ation?
Thanks,
Peter
On Tue, 24 Mar 2020 at 16:14, Péter Sinóros-Szabó <
peter.sinoros-sz...@transferwise.com> wrote:
> Hey,
>
> so I turned on debug logging and now I see the MM2 does a lot, so it is
> hard to tell exactly what :D
>
> So, what I see the most during that "
to figure out what Connect is
> doing during that time.
>
> Ryanne
>
> On Fri, Mar 20, 2020 at 6:15 AM Péter Sinóros-Szabó
> wrote:
>
> > Hi,
> >
> > I don't have the previous logs, so I restarted MM2, that produces the
> same
> > results. So new
ieve finer-grained workloads and a more even balance.
>
> Ryanne
>
> On Mon, Mar 23, 2020 at 9:58 AM Péter Sinóros-Szabó
> wrote:
>
> > so I made some tests with tasks.max = 4
> >
> > with 2 instances:
> > - instance 1: 4 MirrorSourceConnec
:58, Ryanne Dolan wrote:
> Peter, what happens when you add an additional node? Usually Connect will
> detect it and rebalance tasks accordingly. I'm wondering if that mechanism
> isn't working for you.
>
> Ryanne
>
> On Fri, Mar 20, 2020 at 2:40 PM Péter Sinóros-Szab
e internal topics are misconfigured s.t. the
> Herders aren't functioning correctly?
>
> Ryanne
>
> On Fri, Mar 20, 2020 at 11:17 AM Péter Sinóros-Szabó
> wrote:
>
> > I use tasks.max = 4.
> >
> > I see 4 tasks of MirrorSourceConnectors on MM2 instance
Tasks, which should be divided across nodes. Make
> sure you have tasks.max set to something higher than the default of 1.
>
> Ryanne
>
> On Fri, Mar 20, 2020, 8:53 AM Péter Sinóros-Szabó
> wrote:
>
> > Hey,
> >
> > I am using MM2 to mirror A cluster to B with t
Hey,
I am using MM2 to mirror A cluster to B with tasks.max = 4.
I started two instances of MM2 and noticed that all MirrorSourceConnectors
were running in one instance and the rest of the connectors in the other.
This results in a very uneven resource utilization and also it did not
really spre
task-thread-MirrorCheckpointConnector-1 took 6 ms.
(org.apache.kafka.connect.mirror.MirrorCheckpointTask)
Peter
On Wed, 18 Mar 2020 at 15:12, Ryanne Dolan wrote:
> Peter, can you share any log lines like "x took y ms" or "x took too long
> (y ms)" or "timed out running task x"?
>
> Ryanne
>
&g
Hey,
Running a MM2 cluster to mirror from A->B clusters I noticed that when I
add a new topic to A cluster, MM2 will notice it:
[2020-03-17 13:14:05,477] INFO Found 2719 topic-partitions on main. 1 are
new. 0 were removed. Previously had 2718.
(org.apache.kafka.connect.mirror.MirrorSourceConnector
figuring
> multiple directories, i.e. multiple files being written to concurrently,
> may be necessary to maximize throughput.
>
> Eugen
> ____
> 差出人: Péter Sinóros-Szabó
> 送信日時: 2020年3月13日 21:42
> 宛先: users@kafka.apache.org
> 件名: Re: Sequential write
Hi,
yes, if you write one partition only, it will be sequential. But that's
unlikely, so in practice, it won't be sequential overall.
I used AWS EC2 instances with st1 EBS disks, that is the old HDD type
rotational disk. It struggled to give any kind of performance to support
our 6000+ partitions.
Hi,
it is mirroring but the problem was that on of the task failed with not
being able to produce messages to the destination cluster and the task
stopped there.
Peter
On Mon, 9 Mar 2020 at 11:20, Péter Sinóros-Szabó <
peter.sinoros-sz...@transferwise.com> wrote:
> Hi,
>
Hi,
Should MM2 automatically start to mirror new topics?
I see in the logs that MM2 created the remote topic, but it is not
mirroring the new messages.
In the source cluster, topic was created 3 days ago and there is about 200
million messages in it.
Just to test if the messages are there, I coul
Hi,
I read here and there that Kafka is not CPU intensive, but mostly disk and
network. Seems to be reasonable, but that's not what I see on my monitoring.
Could anyone help me to see if the CPU usage I see is about the expected
usage or there is something how we use Kafka that makes it more CPU
Hi Ryanne,
> I frequently demo this stuff, where I pull the plug on entire DCs and
apps keep running like nothing happened.
Is there any public recording, documentation about these demos?
I would be very useful to see how it works.
Thanks,
Peter
On Thu, 13 Feb 2020 at 00:42, Ryanne Dolan wrote:
It seems that one of the brokers somehow had a high CPU utilization, like 5
of the brokers had 15%, and one had 100% utilization.
After I added more CPUs to that broker with 100% CPUs utilization, the
issue solved itself.
Peter
On Thu, 20 Feb 2020 at 10:54, Péter Sinóros-Szabó <
peter.sinoros
Hi,
we use Kafka 1.1.1, recently I faced with an issue/bug I can't see how to
solve.
We have a service running two instances of it, using the same consumer
group id to access some topics. When the service starts and it starts to
join the consumer group, the join does not succeed.
The application
Hi,
is MM2's semantics documented clearly somewhere?
Like that it is At Least Once.
My question is about that if it does guarantee the same message order in
partitions or not?
Cheers,
Peter
Hi,
Why do MM2 need offset sync messages too? It seems to me that the
checkpoint messages contains everything that's needed for offset
translation. What am I missing here?
Cheers,
Peter
nt for the deprecation of MM1. I was planning to
> implement it but would love the help if you're interested.
>
> Ryanne
>
> On Tue, Jan 21, 2020, 8:25 AM Péter Sinóros-Szabó
> wrote:
>
> > Ryanne,
> >
> > I didn't do much work yet, just checked the I
erested.
>
> Ryanne
>
> On Fri, Jan 17, 2020 at 9:34 AM Péter Sinóros-Szabó
> wrote:
>
> > Hi Sebastian & Ryanne,
> >
> > do you have maybe an implementation of this is just some ideas about how
> to
> > implement the policy that does not rename topics
Hi Sebastian & Ryanne,
do you have maybe an implementation of this is just some ideas about how to
implement the policy that does not rename topics?
I am checking the ReplicationPolicy interface and don't really know what
the impact will be if I implement this:
public String formatRemoteTopic(Str
I figured it out.
The correct config is:
my_cluster_name.offset.flush.timeout.ms = 15000
Peter
On Thu, 16 Jan 2020 at 16:48, Péter Sinóros-Szabó <
peter.sinoros-sz...@transferwise.com> wrote:
> Hey,
>
> running MM2 that tried to process its backlog of 1 week just after it was
&
lly speaking, rolling restarts are a good idea -- just be
> advised that it won't necessarily behave as you expect.
>
> Ryanne
>
> On Thu, Jan 16, 2020, 7:55 AM Péter Sinóros-Szabó
> wrote:
>
> > Hi,
> >
> > I run two instances of MM2 with the command
Hey,
running MM2 that tried to process its backlog of 1 week just after it was
started.
I see these in the logs:
[2020-01-16 13:07:30,985] ERROR
WorkerSourceTask{id=MirrorSourceConnector-0} Failed to flush, timed out
while waiting for producer to flush outstanding 4112 messages
(org.apache.kafka
Hi,
I run two instances of MM2 with the command connect-mirror-maker.sh
Q1., Is there any requirement to cluster MM2? Like a network connection
between the nodes? How do MM2 coordinate the work between nodes?
Q2., Assuming I run two instances and want to update the configuration,
should it work
ter.
> >
> > When running at the source, it is the kafka producer that has to cross
> the
> > network boundary. If the producer can't produce, it will eventually drop
> > messages.
> >
> >
> >
> > On Thu, Jan 9, 2020 at 11:28 AM Péter Sinóros
Hey,
I am thinking about where (well in which AWS region) should I run MM2.
I might be wrong, but as I know it is better to run it close to the
destination cluster.
But for other reasons, it would be much easier for me to run it at the
source.
So is it still advised to run MM2 at the destination?
Hi Ryanne,
Am I right that as far as I implement ReplicationPolicy properly, those
features you just mentioned will work fine?
Asking because we already use dot(.) underscore(_) and even hyphen(-)
characters in not replicated topics :D , so it seems to be that we will
need a more advanced renamin
wrote:
> Peter, have you tried overriding the client ID used by MM2's consumers?
> Otherwise, the client IDs are dynamic, which would make it difficult to
> throttle using quotas.
>
> Ryanne
>
> On Wed, Jan 8, 2020, 10:12 AM Péter Sinóros-Szabó
> wrote:
>
> > Hi
Hi,
I'd like to throttle the mirroring process when I start Mirror Maker 2 at
the first time, so it starts to pull all the messages that exists on the
source cluster. I'd like to do this only to avoid putting too much traffic
on the source cluster that may slow down existing production client on i
it waits there
for minutes. I could not find out why, I hope someone has a clue what may
happen there.
We have about 2800 partitions.
Cheers,
Peter
On Mon, 9 Dec 2019 at 12:28, Péter Sinóros-Szabó <
peter.sinoros-sz...@transferwise.com> wrote:
> Hi,
>
> I am experimenting with M
Hi,
so I am planning to use MM2 and was thinking if it has any impact on the
source cluster when mirroring.
Obviously it impacts the performance of the source cluster, so I plan to
use quotas to solve that, but other than that,
Does MM2 write anything back to the source cluster?
As I understand
eter, Jamie, can one of you file a jira ticket if you are still seeing
> this? Thanks!
>
> Ryanne
>
> On Fri, Dec 27, 2019 at 12:04 PM Ryanne Dolan
> wrote:
>
> > Thanks Peter, I'll take a look.
> >
> > Ryanne
> >
> > On Fri, Dec 27, 2019, 7
#x27;s harmless tho.
>
> > Failed to fetch offsets
>
> I have not encountered this. Is it possible one of the clusters is/was
> unreachable? Are any of the clusters using Kerberos or SSL that may be
> misconfigured?
>
> > Plugin class loader...
>
> I'll
Hello Ryanne,
Are there any plans to implement an easy to use throttling to be a little
more kind with the cluster that we start to replicate?
I guess it is possible to use the existing throttling in the source and
destination clusters, but it is not really easy to use.
Also maybe an option to st
Hi,
I see the same.
I just downloaded the Kafka zip and I run:
~/kafka-2.4.0-rc3$ ./bin/connect-mirror-maker.sh
config/connect-mirror-maker.properties
Peter
On Mon, 16 Dec 2019 at 17:14, Ryanne Dolan wrote:
> Hey Jamie, are you running the MM2 connectors on an existing Connect
> cluster, or w
onths ago to force bootstrapping to
> complete correctly. I can share the patch, and if it helps we can raise a
> PR.
>
> Ryanne
>
> On Mon, Dec 9, 2019 at 5:28 AM Péter Sinóros-Szabó
> wrote:
>
> > Hi,
> >
> > I am experimenting with Mirror Make 2 in 2.4
Hi,
I am experimenting with Mirror Make 2 in 2.4.0-rc3. It seems to start up
fine, connects to both source and destination, creates new topics...
But it does not start to actually mirror the messages until about 12
minutes after MM2 was started. I would expect it to start mirroring in some
seconds
56 matches
Mail list logo
