Could you deploy a Kafka Streams app that implemented your desired partitioning? Obviously this would require a duplication in topics between those produced to initially, and those partitioned the way you'd like, but it would solve the issue you're having.
On Wed, 6 May 2020, 10:25 pm Young, Ben, <ben.yo...@fisglobal.com.invalid> wrote: > Hi, > > We have a use case where we'd like the partition a key is hashed to, to be > a subset of the keys that are used for compaction. It would be really cool > if there was a built in hashing strategy that could help us as we're > potentially using Kafka from multiple languages and it could be hard to > standardise. > > For instance, large message processing. We have keys like "msg1:1/3", > "msg1:2/3" and "msg:3/3". We'd like all messages to be retained by > compaction, but all of these messages to go to the same partition... > There's lots of similar use cases where we'd like compaction to keep more > than we'd use for partitioning. > > Obviously we could write our own hashing etc but that's hard when our main > producers and consumers are in C# and we want to integration with KSQL etc. > > My desert island solution would be to have the partition key optionally > called out by something like braces in Redis ( > https://redislabs.com/blog/redis-clustering-best-practices-with-keys/) , > but the whole key used for log compaction. This wouldn't be backwards > compatible so I guess it would have to be a new strategy... > > Does anyone else have requirements like this? How have they solved them? > > Thanks, > Ben Young > The information contained in this message is proprietary and/or > confidential. If you are not the intended recipient, please: (i) delete the > message and all copies; (ii) do not disclose, distribute or use the message > in any manner; and (iii) notify the sender immediately. In addition, please > be aware that any message addressed to our domain is subject to archiving > and review by persons other than the intended recipient. FIS is a trading > name of the following companies: Advanced Portfolio Technologies Ltd (No: > 6312142) | Clear2Pay Limited (No: 5792457) | Decalog (UK) Limited (No: > 2567370) | FIS Apex (International) Limited (No: 2999960) | FIS Apex (UK) > Limited (No. 3573008) | FIS Consulting Services (UK) Limited (No: 2486794) > | FIS Derivatives Utility Services (UK) Limited (No: 9398140) | FIS Energy > Solutions Limited (No: 1889028) | FIS Global Execution Services Limited > (No. 3127109) | FIS Global Trading (UK) Limited (No: 2523114) | FIS > Investment Systems (UK) Limited (No: 1366010) | FIS Sherwood Systems Group > Limited (No: 982833) | FIS Systems Limited (No: 1937159) | FIS Treasury > Systems (Europe) Limited (No: 2624209) | FIS Treasury Systems (UK) Limited > (No: 2893376) | GL Settle Limited (No: 2396127) | Integrity Treasury > Solutions Europe Limited (No: 3289271) | Monis Software Limited (No: > 2333925) | Reech Capital Limited (No: 3649490) | Solutions Plus Consulting > Services Limited (No: 3839487) | Valuelink Information Services Limited > (No: 3827424) all registered in England & Wales with their registered > office at 25 Canada Square, London E14 5LQ | FIS Global Execution Services > Limited is authorised and regulated by the Financial Conduct Authority | > Certegy Card Services Limited (No: 3517639) | Certegy France Limited (No: > 2557650) | eFunds International Limited (No: 1930117) | Fidelity > Information Services Limited (No: 2225203) | FIS Payments (UK) Limited (No: > 4215488) | Metavante Technologies Limited (No: 2659326) all registered in > England & Wales with their registered office at 1st Floor Tricorn House, > 51-53 Hagley Road, Edgbaston, Birmingham, West Midlands, B16 8TU, United > Kingdom | FIS Payments (UK) Limited is authorised and regulated by the > Financial Conduct Authority; some services are covered by the Financial > Ombudsman Service (in the UK). Clear2Pay Limited, Registered in Scotland > (No SC157659), Registered Office: Clear2Pay House, Pitreavie Court, > Pitreavie Business Park Queensferry Rd, Dunfermline, Fife, SS, KY11 8UU, > Scotland | FIS eProcess Intelligence LLC (UK Branch), UK Establishment > Registered in England & Wales (No: FC16527/Branch No. BR000355), Registered > Branch Office: 25 Canada Square, London, E14 5LQ; FIS eProcess Intelligence > LLC is a limited liability company formed in the USA registered on file > with the Office of the Delaware Secretary of State, Division of > Corporations (File No. 2032143), Head Office: 601 Riverside Avenue, > Jacksonville Florida, FL32204, USA | FIS Investment Systems LLC, UK > Establishment Registered in England & Wales (No: FC033836/Branch No. > BR018923), Registered Branch Office: 25 Canada Square, London, E14 5LQ; FIS > Investment Systems LLC is a limited liability company formed in the USA > registered on file with the Office of the Delaware Secretary of State, > Division of Corporations (File No. 0881255), Head Office: 377 E. > Butterfield Road, Suite 800, Lombard, IL 60148, USA | Calls to and from the > companies may be recorded for quality purposes. | All of the named > companies are part of FIS (Fidelity National Information Services, Inc.). >
