8, and we've got topics up to 512
>> partitions in some places just taking into account the produce rate alone
>> (not counting those 720-partition topics that aren't that busy). Many of
>> our brokers run with over 10k open file handles for regular files alone,
>> and over 50k ope
20 happens to have a lot of factors, so it was a good number for
>>> them to pick.
>>>
>>> As a note, we have up to 5000 partitions per broker right now on
>>>current
>>> hardware, and we're moving to new hardware (more disk, 256 GB of
>>>memory
partitions
http://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/
On Fri, Sep 4, 2015 at 10:08 PM, Jörg Wagner
wrote:
Hello!
Regarding the recommended amount of partitions I am a bit confused.
Basically I got the impression that it's better to have
, Sep 4, 2015 at 10:08 PM, Jörg Wagner
wrote:
Hello!
Regarding the recommended amount of partitions I am a bit confused.
Basically I got the impression that it's better to have lots of
partitions
(see information from linkedin etc). On the other hand, a lot of
performance benchmarks fl
n Fri, Sep 4, 2015 at 10:08 PM, Jörg Wagner
> wrote:
>
> > Hello!
> >
> > Regarding the recommended amount of partitions I am a bit confused.
> > Basically I got the impression that it's better to have lots of
> partitions
> > (see information from l
Here is a good doc to describe how to choose the right number of partitions
http://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/
On Fri, Sep 4, 2015 at 10:08 PM, Jörg Wagner wrote:
> Hello!
>
> Regarding the recommended amount of partitions I
Hello!
Regarding the recommended amount of partitions I am a bit confused.
Basically I got the impression that it's better to have lots of
partitions (see information from linkedin etc). On the other hand, a lot
of performance benchmarks floating around show only a few partitions are
Kane,
The in-built offset management is already in master branch, and will be
included in 0.8.2. For now you can give the current trunk a spin.
Guozhang
On Fri, Aug 8, 2014 at 1:42 PM, Kane Kane wrote:
> Hello Guozhang,
>
> Is storing offsets in kafka topic already in master branch?
> We woul
Hello Guozhang,
Is storing offsets in kafka topic already in master branch?
We would like to use that feature, when do you plan to release 0.8.2?
Can we use master branch meanwhile (i.e. is it stable enough).
Thanks.
On Fri, Aug 8, 2014 at 1:38 PM, Guozhang Wang wrote:
> Hi Roman,
>
> Current K
Hi Roman,
Current Kafka messaging guarantee is at-least once, and we are working on
transactional messaging features to make it exactly once. We are expecting
it to be used as synchronization/replication layer for some storage systems
as your use case after that.
As for your design, since you wil
The approach may well depend on your deploy horizon. Currently the offset
tracking of each partition is done in Zookeeper, which places an upper limit on
the number of topic/partitions you want to have and operate with any kind of
efficiency.
In 0.8.2 hopefully coming in the next month or two,
Dear all,
I'm new to Kafka, and I'm considering using it for a maybe not very usual
purpose. I want it to be a backend for data synchronization between a
magnitude of devices, which are not always online (mobile and embedded
devices). All the synchronized information belong to some user, and ca
12 matches
Mail list logo
