Hi Eno, Thanks for the fast response.
We are doing a deduplication process here, so yes you are correct the keys are normally unique. Sounds like a wrong tool for the job issue on my end. Thanks for your input here. On 10 March 2017 at 10:59, Eno Thereska <eno.there...@gmail.com> wrote: > Hi Ian, > > Sounds like you have a total topic size of ~20GB (96 partitions x 200mb). > If most keys are unique then group and reduce might not be as effective in > grouping/reducing. Can you comment on the key distribution? Are most keys > unique? Or do you expect lots of keys to be the same in the topic? > > Thanks > Eno > > > > On Mar 10, 2017, at 9:05 AM, Ian Duffy <i...@ianduffy.ie> wrote: > > > > Hi All, > > > > I'm doing a groupBy and reduce on a kstream which results in a state > store > > being created. > > > > This state store is growing to be massive, its filled up a 20gb drive. > This > > feels very unexpected. Is there some cleanup or flushing process for the > > state stores that I'm missing or is such a large size expected? > > > > The topic in question has 96 partitions and the state is about ~200mb > > average for each one. > > > > 175M 1_0 > > 266M 1_1 > > 164M 1_10 > > 177M 1_11 > > 142M 1_12 > > 271M 1_13 > > 158M 1_14 > > 280M 1_15 > > 286M 1_16 > > 181M 1_17 > > 185M 1_18 > > 187M 1_19 > > 281M 1_2 > > 278M 1_20 > > 188M 1_21 > > 262M 1_22 > > 166M 1_23 > > 177M 1_24 > > 268M 1_25 > > 264M 1_26 > > 147M 1_27 > > 179M 1_28 > > 276M 1_29 > > 177M 1_3 > > 157M 1_30 > > 137M 1_31 > > 247M 1_32 > > 275M 1_33 > > 169M 1_34 > > 267M 1_35 > > 283M 1_36 > > 171M 1_37 > > 166M 1_38 > > 277M 1_39 > > 160M 1_4 > > 273M 1_40 > > 278M 1_41 > > 279M 1_42 > > 170M 1_43 > > 139M 1_44 > > 272M 1_45 > > 179M 1_46 > > 283M 1_47 > > 263M 1_48 > > 267M 1_49 > > 181M 1_5 > > 282M 1_50 > > 166M 1_51 > > 161M 1_52 > > 176M 1_53 > > 152M 1_54 > > 172M 1_55 > > 148M 1_56 > > 268M 1_57 > > 144M 1_58 > > 177M 1_59 > > 271M 1_6 > > 279M 1_60 > > 266M 1_61 > > 194M 1_62 > > 177M 1_63 > > 267M 1_64 > > 177M 1_65 > > 271M 1_66 > > 175M 1_67 > > 168M 1_68 > > 140M 1_69 > > 175M 1_7 > > 173M 1_70 > > 179M 1_71 > > 178M 1_72 > > 166M 1_73 > > 180M 1_74 > > 177M 1_75 > > 276M 1_76 > > 177M 1_77 > > 162M 1_78 > > 266M 1_79 > > 194M 1_8 > > 158M 1_80 > > 187M 1_81 > > 162M 1_82 > > 163M 1_83 > > 177M 1_84 > > 286M 1_85 > > 165M 1_86 > > 171M 1_87 > > 162M 1_88 > > 179M 1_89 > > 145M 1_9 > > 166M 1_90 > > 190M 1_91 > > 159M 1_92 > > 284M 1_93 > > 172M 1_94 > > 149M 1_95 > >
