Explode is called UNNEST in Beam SQL (and I believe this is the more standard name).
FlatMap(arr -> arr) is a simple and efficient implementation for straight Beam. Kenn On Tue, Jan 12, 2021 at 4:58 PM Kyle Weaver <kcwea...@google.com> wrote: > @Reuven Lax <re...@google.com> yes I am aware of that transform, but >> that’s different from the explode operation I was referring to: >> https://spark.apache.org/docs/latest/api/sql/index.html#explode >> > > How is it different? It'd help if you could provide the signature (input > and output PCollection types) of the transform you have in mind. > > On Tue, Jan 12, 2021 at 4:49 PM Tao Li <t...@zillow.com> wrote: > >> @Reuven Lax <re...@google.com> yes I am aware of that transform, but >> that’s different from the explode operation I was referring to: >> https://spark.apache.org/docs/latest/api/sql/index.html#explode >> >> >> >> *From: *Reuven Lax <re...@google.com> >> *Reply-To: *"user@beam.apache.org" <user@beam.apache.org> >> *Date: *Tuesday, January 12, 2021 at 2:04 PM >> *To: *user <user@beam.apache.org> >> *Subject: *Re: Is there an array explode function/transform? >> >> >> >> Have you tried Flatten.iterables >> >> >> >> On Tue, Jan 12, 2021, 2:02 PM Tao Li <t...@zillow.com> wrote: >> >> Hi community, >> >> >> >> Is there a beam function to explode an array (similarly to spark sql’s >> explode())? I did some research but did not find anything. >> >> >> >> BTW I think we can potentially use FlatMap to implement the explode >> functionality, but a Beam provided function would be very handy. >> >> >> >> Thanks a lot! >> >>
