contact the sender and
> > delete this email and its contents from any computer. Any views expressed
> > in this email are those of the individual sender and may not necessarily
> > reflect the views of the company.
> >
> >Please consider the environmet b
sed
> in this email are those of the individual sender and may not necessarily
> reflect the views of the company.
>
>Please consider the environmet before printing this email.
>
>
> -Original Message-
> From: Wes Peng
> Sent: Thursday, December
flect the views of the company.
Please consider the environmet before
printing this email.
-Original Message-
From: Wes Peng
Sent: Thursday, December 23, 2021 10:11 PM
To: users@kafka.apache.org
Subject: Re: Kafka Topics
That depends on your resources
That depends on your resources such as ram, disk etc. General speaking
there is no problem.
Regards
> Dears,
>
> I'm new to using Kafka, and I was wondering up to how many topics can
> Kafka Handle. I'm trying to use Kafka but using it I will be obliged to
> create thousands of topics to keep up
Hi Ashock,
Check this article of mine
Real Time Processing of Trade Data with Kafka, Flume, Spark, Hbase and
MongoDB
https://www.linkedin.com/pulse/real-time-processing-trade-data-kafka-flume-spark-talebzadeh-ph-d-/
under
*Kafka setup*
HTH
Dr Mich Talebzadeh
LinkedIn *
https://www.linked
I can be updated once the Kafka AdminAPI is available and does everything
over the Kafka wire protocol that the current kafka-topics command does by
talking directly with zookeeper. For example create a topic or delete a
topic. Unfortunately is has to remain this way for just a little while
longer.
Since Kafka itself has replication, I'm not sure what HDFS backups would
bring – how would you recover from e.g. all Kafka nodes blowing up if you
only have an HDFS backup? Why not use MirrorMaker to replicate the cluster
to a remote DC, with a process of reversing the direction in case you need
to
> I'm not sure what HDFS backups would bring
I’m not sure how realistic the threat is, but I was thinking a case in which
bug in Kafka corrupts the log files. I would personally sleep better knowing
there is a Kafka-independent backup of all data.
> how would you recover from e.g. all Kafka n
If you use camus to backup you could use something like this?
https://groups.google.com/forum/#!searchin/camus_etl/sumac/camus_etl/hIi1cbOCFvU/grqd7UHLoFUJ
Please note I haven't used this, but remember seeing it ages ago.
On Wed, Mar 16, 2016 at 2:24 PM, Giidox
wrote:
> > I'm not sure what HDF
Good points.
I would backup all partitions to HDFS (etc.), as fast as the data arrives. In
case the Kafka becomes corrupted, the topics can be repopulated from the
backup. In my case, all clients track their own offsets, so they should in
theory be able to continue as if nothing had happened.
A couple of things:
- Compacted topics provide a useful way to retain meaningful datasets inside
the broker, which don’t grow indefinitely. If you have an update-in-place use
case, where the event sourced approach doesn’t buy you much, these will keep
the reload time down when you regenerate ma
This is definitely an interesting use case. However, you need to be aware
that changing the broker topology won't rebalance the preexisting data from
the previous brokers. That is, you risk loosing data.
Cheers,
Jens
On Wed, Mar 9, 2016 at 2:10 PM Daniel Schierbeck
wrote:
> I'm considering an a
You might find what you want when looking how Kafka is used for samza,
http://samza.apache.org/
On Mon, Mar 14, 2016 at 10:34 AM Daniel Schierbeck
wrote:
> Partitions being limited by disk size is no different from e.g. a SQL
> store. This would not be used for extremely high throughput. If,
> e
Partitions being limited by disk size is no different from e.g. a SQL
store. This would not be used for extremely high throughput. If,
eventually, there was a good case for not requiring that an entire
partition must be stored on a single machine, it would be possible to use
the log segments for di
I would like to read an answer to this question as well. This is a similar
architecture as I am planning. Dealing with secondary data store for old
messages would indeed make things complicated.
Clark Haskins wrote that the partition size is limited by machines capacity (I
assume disk space):
Hi,
> 4. but when i close my Ubuntu virtual machine and i start Kafka and
> Zookeeper again. I can see the topics present in the kafka cluster. However
> all are default to 0 offset and data seems to have been lost.
By default kafka stores the data in /tmp/kafka-logs directory.
When you restar
16 matches
Mail list logo
