Hello,
I have kafka cluster with certain topic that had too few partitions, so a
large backlog of messages was collected. After i added additional
partitions, only the newly messages balanced between all the new partitions.
What is the preferred way to balance the "old" backlog of messages inside
Hi Jazz,
I'm afraid the only way of rebalancing old messages is indeed to
rewrite them to the topic - thus creating duplication.
Once a message has been written to a partition by Kafka this
assignment is final, there is no way of moving it to another
partition.
Changing the partition count of top
Thanks, Sönke
Is there any available kafka tool to move messages between topics?
On Sun, Jan 27, 2019 at 2:40 PM Sönke Liebau
wrote:
> Hi Jazz,
>
> I'm afraid the only way of rebalancing old messages is indeed to
> rewrite them to the topic - thus creating duplication.
> Once a message has been
Hello,
In order to be prepared for large volume of messages i want to configure my
topic with large amount of partitions (>1000).
What are the drawbacks of this?
Should I except any pitfalls?
Best regards,
jazz
Hi Harsha,
Thanks for the reply.
Issue is resolved as of now and the root cause was a runaway application
spawning many instances of kafkacat and hammering kafka brokers. I am still
wondering that what could be reason for shrink and expand is a client hammers a
broker .
--Ashish
On Thursd
Jazz, the number of partitions isn't necessarily related to message volume.
The biggest factors for max message volume would be the number of brokers
and their write speed. You should choose the number of partitions based on
the number of brokers and the number of consumers you expect to have.
Rya
You can use MirrorMaker to copy data between topics.
Ryanne
On Sun, Jan 27, 2019, 7:12 AM jaaz jozz Thanks, Sönke
> Is there any available kafka tool to move messages between topics?
>
> On Sun, Jan 27, 2019 at 2:40 PM Sönke Liebau
> wrote:
>
> > Hi Jazz,
> >
> > I'm afraid the only way of reba
Hi Jazz,
you probably already found this, but [1] is a good read and starting
point around this topic!
Best regards,
Sönke
[1]
https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster
On Sun, Jan 27, 2019 at 6:22 PM Ryanne Dolan wrote:
>
> Jazz, the number of partitions
Hi all,
This is the second candidate for release of Apache Kafka 2.1.1. This release
includes many bug fixes for Apache Kafka 2.1.
Compared to rc0, this release includes the following changes:
* MINOR: Upgrade ducktape to 0.7.5 (#6197)
* KAFKA-7837: Ensure offline partitions are picked up as so
Yes but I find this even easier to do with KSQL.
CREATE STREAM OUTPUTTOPIC AS SELECT * FROM INPUTTOPIC;
There are similar examples like this that also filter messages while copying,
or change the message format while copying on the KSQL Recipe page here
https://www.confluent.io/stream-processin
Hi Matthias, thanks for your reply. Let me to explain better what I'm
trying to say, in the meantime I've played with this problem and I think
now I have a more clear view, though I haven't still a solution.
I've an input topic A which is a stream of message where each message
contains just an ID.
Hi Pavel, did you understood why do you have such strange behaviour?
On Tue, Oct 30, 2018 at 12:22 PM Pavel Koroliov
wrote:
> I'm sorry guy's. Aggregation works fine, but i've found new problem with
> *groupByKey()*. After restart application some aggregations starts from
> beginning, although t
12 matches
Mail list logo
