Hi, I remember creating one for a similar scenario in the past - https://issues.apache.org/jira/browse/SPARK-29372.
Thanks, Shubham On Wed, May 19, 2021 at 5:34 PM Takeshi Yamamuro <linguin....@gmail.com> wrote: > hi, Andrew, > > Welcome any improvement proposal for that. > Could you file an issue in jira first to show us your idea and an example > query > to reproduce the issue you described? > > Bests, > Takeshi > > On Wed, May 19, 2021 at 11:38 AM Andrew Melo <andrew.m...@gmail.com> > wrote: > >> Hello, >> >> When reading a very wide (> 1000 cols) input, WholeStageCodeGen blows >> past the 64kB source limit and fails. Looking at the generated code, a >> big part of the code is simply the DSv2 convention that the codegen'd >> variable names are the same as the columns instead of something more >> compact like 'c1', 'c2', etc.. >> >> Would there be any interest in accepting a patch that shortens these >> variable names to try and stay under the limit? >> >> Thanks >> Andrew >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> > > -- > --- > Takeshi Yamamuro >
