Gotcha. It would certainly look cleaner, I'd be open to such a change as long as it doesn't complicate the code or impact performance, but I don't think we should make any guarantees.
+Reuven Lax <[email protected]> WDYT? Brian On Wed, Jan 20, 2021 at 10:14 AM Tao Li <[email protected]> wrote: > Hi Brian, > > > > Thanks for your quick response. I totally agree that the we should not > rely on any assumption on the field order and we can always specify the > order of the flattened fields as we want. There is no blocker issue for me > with the current behavior, but I am just wondering if may be convenient in > some use cases if we can just keep the order (roughly) consistent with the > order of the parent fields from the original schema. > > > > *From: *Brian Hulette <[email protected]> > *Reply-To: *"[email protected]" <[email protected]> > *Date: *Wednesday, January 20, 2021 at 9:42 AM > *To: *user <[email protected]> > *Subject: *Re: Regarding the field ordering after Select.Flattened > transform > > > > This does seem like an odd choice, I suspect this was just a matter of > convenience of implementation since the javadoc makes no claims about field > order. > > In general schema transforms don't take care to maintain a particular > field order and I'd recommend against relying on it. Instead fields should > be addressed by name, either with Row.getValue(Sring), or by mapping to a > user type. Is there a reason you want to rely on a particular field order? > Maybe when writing to certain IOs field order could be important. > > > > On Tue, Jan 19, 2021 at 1:36 PM Tao Li <[email protected]> wrote: > > Hi community, > > > > I have been experimenting with Select.Flattened transform and noticed that > the field order in the flattened schema is not consistent with the order of > the top level fields from the original schema. For example, in the original > schema, we have field “foo” as the first field and it has a nested field > “bar”. After applying the flattening transform, the new field “foo.bar” > becomes the last field in the flattened schema. Seems like the order of the > new fields are not that deterministic in the flattened schema. Is this an > expected behavior? Don’t we guarantee any ordering of the flattened fields > (e.g. being consistent with the original order)? Thanks! > >
