Hi Gyorgy,
I guess Your problem might not be directly related to UDFs not UDAFs but
to nested structure of Your data. I did have a problem with processing
nested data and also did not find a way to work this out - especially if
You are not in control of input data structure.
This issues seem
Sorry I was not specific enough. I ment using the SqlTransform registerUdf
and registerUdaf. I use a lot of SQL in my pipeline and I would prefer
using SQL UDFs in many cases over writing beam transforms. I already have
UDFs but I did not find a way to make them work over nested structures.
Thank y
Yes, for sure. This is one of the areas Beam excels vs. more simple tools
like SQL. You can write arbitrary code to iterate over arbitrary structures
in the typical Java/Python/Go/Typescript/Scala/[pick your language] way. In
the Beam nomenclature. UDFs correspond to DoFns and UDAFs correspond to
C
