Use cases: prioritise current data
When processing messages sometimes there is a need to re process old data.
It would be nice to be abled to send the old data as messages to a separate
topic and that would only be processed when the current topic doesn’t have any
messages left to process.
This would prevent customers getting delays in current data processing due to
message processors being busy processing old data.
> On 17 Jan 2019, at 7:55 PM, Tim Ward <tim.w...@origamienergy.com> wrote:
>
> Use cases: processing alerts.
>
> High priority alerts ("a large chunk of your system has stopped providing
> service, immediate action essential") should be processed before low priority
> alerts ("some minor component has put out a not-very serious warning,
> somebody should probably have a look at it when they get bored"), of which
> there could be a long queue.
>
> Urgent alerts (a phone call telling someone "you need to do this now") should
> be processed before non-urgent alerts (a phone call telling someone "FYI,
> such and such is going to happen in a couple of hours").
>
> Tim Ward
>
> -----Original Message-----
> From: n...@afshartous.com <n...@afshartous.com>
> Sent: 17 January 2019 02:52
> To: users@kafka.apache.org
> Subject: Prioritized Topics for Kafka
>
>
>
> Hi all,
>
> On the dev list we’ve been discussing a proposed new feature (prioritized
> topics). In a nutshell, when consuming from a set of topics with assigned
> priorities, consumption from lower-priority topics only occurs if there’s no
> data flowing in from a higher-priority topic.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics
>
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-349:+Priorities+for+Source+Topics>
>
> One question is are there use-cases for the proposed API. If you think this
> would be useful and have use-cases in mind please reply with the use-cases.
>
> Its also possible to implement prioritization with the existing API by using
> a combination of pausing, resuming, and local buffering. The question is
> then does it make sense to introduce the proposed higher-level API to make
> this easier ?
>
> The responses will be used as input to determine if we move ahead with the
> proposal. Thanks in advance for input.
>
> Cheers,
> --
> Nick
>
> The contents of this email and any attachment are confidential to the
> intended recipient(s). If you are not an intended recipient: (i) do not use,
> disclose, distribute, copy or publish this email or its contents; (ii) please
> contact the sender immediately; and (iii) delete this email. Our privacy
> policy is available here: https://origamienergy.com/privacy-policy/. Origami
> Energy Limited (company number 8619644); Origami Storage Limited (company
> number 10436515) and OSSPV001 Limited (company number 10933403), each
> registered in England and each with a registered office at: Ashcombe Court,
> Woolsack Way, Godalming, GU7 1LQ.
