Hi, We have a use case where we'd like the partition a key is hashed to, to be a subset of the keys that are used for compaction. It would be really cool if there was a built in hashing strategy that could help us as we're potentially using Kafka from multiple languages and it could be hard to standardise.
For instance, large message processing. We have keys like "msg1:1/3", "msg1:2/3" and "msg:3/3". We'd like all messages to be retained by compaction, but all of these messages to go to the same partition... There's lots of similar use cases where we'd like compaction to keep more than we'd use for partitioning. Obviously we could write our own hashing etc but that's hard when our main producers and consumers are in C# and we want to integration with KSQL etc. My desert island solution would be to have the partition key optionally called out by something like braces in Redis (https://redislabs.com/blog/redis-clustering-best-practices-with-keys/) , but the whole key used for log compaction. This wouldn't be backwards compatible so I guess it would have to be a new strategy... Does anyone else have requirements like this? How have they solved them? Thanks, Ben Young The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. FIS is a trading name of the following companies: Advanced Portfolio Technologies Ltd (No: 6312142) | Clear2Pay Limited (No: 5792457) | Decalog (UK) Limited (No: 2567370) | FIS Apex (International) Limited (No: 2999960) | FIS Apex (UK) Limited (No. 3573008) | FIS Consulting Services (UK) Limited (No: 2486794) | FIS Derivatives Utility Services (UK) Limited (No: 9398140) | FIS Energy Solutions Limited (No: 1889028) | FIS Global Execution Services Limited (No. 3127109) | FIS Global Trading (UK) Limited (No: 2523114) | FIS Investment Systems (UK) Limited (No: 1366010) | FIS Sherwood Systems Group Limited (No: 982833) | FIS Systems Limited (No: 1937159) | FIS Treasury Systems (Europe) Limited (No: 2624209) | FIS Treasury Systems (UK) Limited (No: 2893376) | GL Settle Limited (No: 2396127) | Integrity Treasury Solutions Europe Limited (No: 3289271) | Monis Software Limited (No: 2333925) | Reech Capital Limited (No: 3649490) | Solutions Plus Consulting Services Limited (No: 3839487) | Valuelink Information Services Limited (No: 3827424) all registered in England & Wales with their registered office at 25 Canada Square, London E14 5LQ | FIS Global Execution Services Limited is authorised and regulated by the Financial Conduct Authority | Certegy Card Services Limited (No: 3517639) | Certegy France Limited (No: 2557650) | eFunds International Limited (No: 1930117) | Fidelity Information Services Limited (No: 2225203) | FIS Payments (UK) Limited (No: 4215488) | Metavante Technologies Limited (No: 2659326) all registered in England & Wales with their registered office at 1st Floor Tricorn House, 51-53 Hagley Road, Edgbaston, Birmingham, West Midlands, B16 8TU, United Kingdom | FIS Payments (UK) Limited is authorised and regulated by the Financial Conduct Authority; some services are covered by the Financial Ombudsman Service (in the UK). Clear2Pay Limited, Registered in Scotland (No SC157659), Registered Office: Clear2Pay House, Pitreavie Court, Pitreavie Business Park Queensferry Rd, Dunfermline, Fife, SS, KY11 8UU, Scotland | FIS eProcess Intelligence LLC (UK Branch), UK Establishment Registered in England & Wales (No: FC16527/Branch No. BR000355), Registered Branch Office: 25 Canada Square, London, E14 5LQ; FIS eProcess Intelligence LLC is a limited liability company formed in the USA registered on file with the Office of the Delaware Secretary of State, Division of Corporations (File No. 2032143), Head Office: 601 Riverside Avenue, Jacksonville Florida, FL32204, USA | FIS Investment Systems LLC, UK Establishment Registered in England & Wales (No: FC033836/Branch No. BR018923), Registered Branch Office: 25 Canada Square, London, E14 5LQ; FIS Investment Systems LLC is a limited liability company formed in the USA registered on file with the Office of the Delaware Secretary of State, Division of Corporations (File No. 0881255), Head Office: 377 E. Butterfield Road, Suite 800, Lombard, IL 60148, USA | Calls to and from the companies may be recorded for quality purposes. | All of the named companies are part of FIS (Fidelity National Information Services, Inc.).
