Kafka provides total ordering only within individual partitions. A topic with multiple partitions is considered a "partial order", in which multiple subsets of the topic are considered well-ordered but the topic as a whole is not. The tradeoff between scalability via partitioning and message ordering is fundamental to Kafka's design and unfortunately makes it difficult to built Kafka-based pipelines that rely on strict ordering.
Partition keys are used to ensure that messages sharing a certain characteristic will always go to the same partition. It's possible that you could use this to your advantage given the ordering guarantees provided by individual partitions, but you'd still be limited in your ability to scale beyond a single partition for any given partition key. On Fri, Jul 6, 2018 at 10:15 AM chinchu chinchu <chinchutec...@gmail.com> wrote: > Hello, > We are building a data pipe line with the following semantics. We need to > maintain order till the last unit of work is done in this pipe line .We > cannot have a single partition since that looses our ability to scale . > Looked at using partitioning keys ,but does that guarantee order in the > pipe line as such as opposed to between a topic and consumers? > > produces > consumes enriches and produces >consumes ,does work and produces > > persistent store > > Thanks, > Chinchu > -- Emmett Butler | Senior Software Engineer <http://www.parsely.com/?utm_source=Signature&utm_medium=emmett-butler&utm_campaign=Signature>
