Yeah, scaling through adding partitions ("sharding") is a basic feature of
Kafka.
We expect topics to have many partitions (at least as many as number of
consumers), and each consumer to get a subset of the messages by getting a
subset of partitions.This design gives Kafka its two biggest advantages: 1. Order guarantee - consumers are guaranteed to get messages in order because they are reading from a subset of partitions in order (rather than getting a mix of messages from different partitions) 2. Scalability - because we just need to track the last message each consumer read from each partition (and we know it consumed everything that came before, due to #1), we can scale to huge number of consumers and partitions without worrying about overhead of worrying about who got which message. This is the biggest different between Kafka and JMS queues. Hope this helps. Gwen On Thu, Sep 3, 2015 at 9:49 AM, Joris Peeters <j.peet...@wintoncapital.com> wrote: > I imagine this has been asked before, but I have googled around quite a > bit and can’t really find a clear answer. Apologies in advance, though .. > > > > I’m interested in Kafka setups that allow for competing customers. I’ll > have one topic where a lot of messages get published to, and I’d like to be > able to (dynamically, eventually) fire up services to take messages of the > queue and process them. Obviously, I’d expect each of the services to see > consume only a subset of the messages. > > > > Do I understand correctly that I would need multiple partitions for this? > I’ve been messing around a bit with a one topic/one partition setup, but > all consumers receive the same (and total amount of) messages. > > > > Do all the clients support this? I’ve currently got the option between C#, > Java and Python, more or less. (I expect the Java one to be most > feature-complete). > > > > Thanks! > > > > *Joris Peeters* > > Developer > > > > *Research and Data Technology* > > T: > > +44 (0) 20 8576 5800 > > > > *Winton* > > Grove House > 27 Hammersmith Grove > London W6 0NE > > > > wintoncapital.com <http://www.wintoncapital.com/> > > > > <http://www.wintoncapital.com/> > > > > > > > Winton Capital Management Limited (“Winton”) is a limited company > registered in England and Wales with its registered offices at 16 Old > Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is > authorised and regulated by the Financial Conduct Authority in the United > Kingdom, registered as an investment adviser with the US Securities and > Exchange Commission, registered with the US Commodity Futures Trading > Commission and a member of the National Futures Association in the United > States. > > This communication, including any attachments, is confidential and may be > privileged. This email is for use by the intended recipient only. If you > receive it in error, please notify the sender and delete it. You should not > copy or disclose all or any part of this email. > > This email does not constitute an offer or solicitation and nothing > contained in this email constitutes, and should not be construed as, > investment advice. Prospective investors should request offering materials > and consult their own advisers with respect to investment decisions and > inform themselves as to applicable legal requirements, exchange control > regulations and taxes in the countries of their citizenship, residence or > domicile. Past performance is not indicative of future results. > > Winton takes reasonable steps to ensure the accuracy and integrity of its > communications, including emails. However Winton accepts no liability for > any materials transmitted. Emails are not secure and cannot be guaranteed > to be error free. >
