Re: Partitioning by composite key, But type and number of keys are dynamic

2018-11-15 Thread Chesnay Schepler
Why don't you calculate the hashCode for each field, combine them and use that as the key? You won't get around calculating /something /for each field and combining the result. On 15.11.2018 11:02, Gaurav Luthra wrote: Hi Chesnay, My End user will be aware about the fields of "input records"

Re: Partitioning by composite key, But type and number of keys are dynamic

2018-11-15 Thread Gaurav Luthra
Hi Chesnay, My End user will be aware about the fields of "input records" (GenericRecord). In configuration my end user only will tell me the name and number of the fields, based on these fields of GenericRecord I will have to partition the DataStream and make Keyed Stream. Currently, I have impl

Re: Partitioning by composite key, But type and number of keys are dynamic

2018-11-15 Thread Chesnay Schepler
How do you determine which fields you want to use if you don't know the names and types beforehand? I would wrap the GenericRecord in my own type, implements the field selection logic in hashCode/equals, and unwrap them again in your functions. On 14.11.2018 10:57, Gaurav Luthra wrote: There