So I have my custom coder created for TreeMap and I'm ready to set it... So my Type is "TreeMap<String, ArrayList<Integer>>"
What do I put for ".setCoder(TreeMapCoder.of(???, ???))" On Thu, Jul 11, 2019 at 8:21 PM Rui Wang <ruw...@google.com> wrote: > Hi Shannon, [1] will be a good start on coder in Java SDK. > > > [1] > https://beam.apache.org/documentation/programming-guide/#data-encoding-and-type-safety > > Rui > > On Thu, Jul 11, 2019 at 3:08 PM Shannon Duncan <joseph.dun...@liveramp.com> > wrote: > >> Was able to get it to use ArrayList by doing List<List<Integer>> result = >> new ArrayList<List<Integer>>(); >> >> Then storing my keys in a separate array that I'll pass in as a side >> input to key for the list of lists. >> >> Thanks for the help, lemme know more in the future about how coders work >> and instantiate and I'd love to help contribute by adding some new coders. >> >> - Shannon >> >> On Thu, Jul 11, 2019 at 4:59 PM Shannon Duncan < >> joseph.dun...@liveramp.com> wrote: >> >>> Will do. Thanks. A new coder for deterministic Maps would be great in >>> the future. Thank you! >>> >>> On Thu, Jul 11, 2019 at 4:58 PM Rui Wang <ruw...@google.com> wrote: >>> >>>> I think Mike refers to ListCoder >>>> <https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/ListCoder.java> >>>> which >>>> is deterministic if its element is the same. Maybe you can search the repo >>>> for examples of ListCoder? >>>> >>>> >>>> -Rui >>>> >>>> On Thu, Jul 11, 2019 at 2:55 PM Shannon Duncan < >>>> joseph.dun...@liveramp.com> wrote: >>>> >>>>> So ArrayList doesn't work either, so just a standard List? >>>>> >>>>> On Thu, Jul 11, 2019 at 4:53 PM Rui Wang <ruw...@google.com> wrote: >>>>> >>>>>> Shannon, I agree with Mike on List is a good workaround if your >>>>>> element within list is deterministic and you are eager to make your new >>>>>> pipeline working. >>>>>> >>>>>> >>>>>> Let me send back some pointers to adding new coder later. >>>>>> >>>>>> >>>>>> -Rui >>>>>> >>>>>> On Thu, Jul 11, 2019 at 2:45 PM Shannon Duncan < >>>>>> joseph.dun...@liveramp.com> wrote: >>>>>> >>>>>>> I just started learning Java today to attempt to convert our python >>>>>>> pipelines to Java to take advantage of key features that Java has. I >>>>>>> have >>>>>>> no idea how I would create a new coder and include it in for beam to >>>>>>> recognize. >>>>>>> >>>>>>> If you can point me in the right direction of where it hooks >>>>>>> together I might be able to figure that out. I can duplicate MapCoder >>>>>>> and >>>>>>> try to make changes, but how will beam know to pick up that coder for a >>>>>>> groupByKey? >>>>>>> >>>>>>> Thanks! >>>>>>> Shannon >>>>>>> >>>>>>> On Thu, Jul 11, 2019 at 4:42 PM Rui Wang <ruw...@google.com> wrote: >>>>>>> >>>>>>>> It could be just straightforward to create a SortedMapCoder for >>>>>>>> TreeMap. Just add checks on map instances and then change >>>>>>>> verifyDeterministic. >>>>>>>> >>>>>>>> If this is a common need we could just submit it into Beam repo. >>>>>>>> >>>>>>>> [1]: >>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/MapCoder.java#L146 >>>>>>>> >>>>>>>> On Thu, Jul 11, 2019 at 2:26 PM Mike Pedersen <m...@mikepedersen.dk> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> There isn't a coder for deterministic maps in Beam, so even if >>>>>>>>> your datastructure is deterministic, Beam will assume the serialized >>>>>>>>> bytes >>>>>>>>> aren't deterministic. >>>>>>>>> >>>>>>>>> You could make one using the MapCoder as a guide: >>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/MapCoder.java >>>>>>>>> Just change it such that the exception in VerifyDeterministic is >>>>>>>>> removed and when decoding it instantiates a TreeMap or such instead >>>>>>>>> of a >>>>>>>>> HashMap. >>>>>>>>> >>>>>>>>> Alternatively, you could just represent your key as a sorted list >>>>>>>>> of KV pairs. Lookups could be done using binary search if necessary. >>>>>>>>> >>>>>>>>> Mike >>>>>>>>> >>>>>>>>> Den tor. 11. jul. 2019 kl. 22.41 skrev Shannon Duncan < >>>>>>>>> joseph.dun...@liveramp.com>: >>>>>>>>> >>>>>>>>>> So I'm working on essentially doing a word-count on a complex >>>>>>>>>> data structure. >>>>>>>>>> >>>>>>>>>> I tried just using a HashMap as the Structure, but that didn't >>>>>>>>>> work because it is non-deterministic. >>>>>>>>>> >>>>>>>>>> However when Given a LinkedHashMap or TreeMap which is >>>>>>>>>> deterministic the SDK complains that it's non-deterministic when >>>>>>>>>> trying to >>>>>>>>>> use it as a key for GroupByKey. >>>>>>>>>> >>>>>>>>>> What would be an appropriate Map style data structure that would >>>>>>>>>> be deterministic enough for Apache Beam to accept it as a key? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Shannon >>>>>>>>>> >>>>>>>>>
