Thanks, Vincent! Yes, i totally agree about coupling/decoupling. The need for the producer to send messages to a specific partition and the consumer to consume from a specific partition creates high coupling between them. So in the future changing the topic partitioning scheme will be impossible without reconfiguring/restarting consumers.
пн, 1 апр. 2019 г., 13:48 Vincent Maurin <vincent.maurin...@gmail.com>: > Maybe you can find some litterature about messaging patterns ? > > Usually, a single kafka topic is used to do PubSub pattern, i.e decoupling > producers and consumers. > In your case, it seems that the situation is quite coupled, i.e you need to > generate and send 20k price lists to 20k specific consumers (so it is > looking like more what is describe as PAIR for 0mq > > https://learning-0mq-with-pyzmq.readthedocs.io/en/latest/pyzmq/patterns/pair.html > ) > > I could still see Kafka as an option, but I am not sure about using > partitioning for the split. I would rather go for one topics per point of > sales with like the POS id in the name, with a single partition each. > You guarantee the order, you know where to produce, where to consume from > and you don't have any complicated consumption logic to implement. You just > probably need to take care of the topic management programmatically. > > Best > > > On Mon, Apr 1, 2019 at 11:26 AM Dimitry Lvovsky <dlvov...@gmail.com> > wrote: > > > Going off of what Hans mentioned, I don't see any reason for 200,000 > > partitions...you don't need one partition per POS. You can have all of > your > > pos listening to one partition and each pos agent having a unique group > > id. The POS agent only processes the messages that are relevant to him, > > and simply ignores the rest. If you have bandwidth or processing power > > concerns could also consider subdividing topics ( not partitions ) per > > regions. > > > > Dimitry > > > > On Mon, Apr 1, 2019 at 8:16 AM Hans Jespersen <h...@confluent.io> wrote: > > > > > Yes but you have more than 1 POS terminal per location so you still > don't > > > need 20,000 partitions. Just one per location. How many locations do > you > > > have? > > > > > > In doesn’t matter anyway since you can build a Kafka cluster with up to > > > 200,000 partitions if you use the latest versions of Kafka. > > > > > > > > > https://blogs.apache.org/kafka/entry/apache-kafka-supports-more-partitions > > > > > > “As a rule of thumb, we recommend each broker to have up to 4,000 > > > partitions and each cluster to have up to 200,000 partitions” > > > > > > -hans > > > > > > > On Apr 1, 2019, at 2:02 AM, Alexander Kuterin <akute...@gmail.com> > > > wrote: > > > > > > > > Thanks, Hans! > > > > We use location specific SKU pricing and send specific price lists to > > the > > > > specific POS terminal. > > > > > > > > пн, 1 апр. 2019 г., 3:01 Hans Jespersen <h...@confluent.io>: > > > > > > > >> Doesn’t every one of the 20,000 POS terminals want to get the same > > price > > > >> list messages? If so then there is no need for 20,000 partitions. > > > >> > > > >> -hans > > > >> > > > >>> On Mar 31, 2019, at 7:24 PM, <akute...@gmail.com> < > > akute...@gmail.com> > > > >> wrote: > > > >>> > > > >>> Hello! > > > >>> > > > >>> > > > >>> > > > >>> I ask for your help in connection with the my recent task: > > > >>> > > > >>> - Price lists are delivered to 20,000 points of sale with a > frequency > > > of > > > >> <10 > > > >>> price lists per day. > > > >>> > > > >>> - The order in which the price lists follow is important. It is > also > > > >>> important that the price lists are delivered to the point of sale > > > online. > > > >>> > > > >>> - At each point of sale, an agent application is deployed, which > > > >> processes > > > >>> the received price lists. > > > >>> > > > >>> > > > >>> > > > >>> This task is not particularly difficult. Help in solving the task > is > > > not > > > >>> required. > > > >>> > > > >>> > > > >>> > > > >>> The difficulty is that Kafka in our company is a new "silver > bullet", > > > and > > > >>> the project manager requires me to implement the following > technical > > > >>> decision: > > > >>> > > > >>> deploy 20,000 Kafka consumer instances (one instance for each point > > of > > > >> sale) > > > >>> for one topic partitioned into 20,000 partitions - one partition > per > > > >>> consumer. > > > >>> > > > >>> Technical problems obtained in experiments with this technical > > decision > > > >> do > > > >>> not convince him. > > > >>> > > > >>> > > > >>> > > > >>> Please give me references to the books/documents/blogposts. which > > > clearly > > > >>> shows that Kafka not intended for this way to use (references to > > other > > > >>> anti-patterns/pitfalls will be useful). > > > >>> > > > >>> My own attempts to find such references were unsuccessful. > > > >>> > > > >>> > > > >>> > > > >>> Thank you! > > > >>> > > > >>> > > > >>> > > > >> > > > > > >
