Hi Pat,
I don't understand what "lazy casting" mean here. Why do you think
current Catalyst casting is "eager"? Casting happens at runtime, and
doesn't disable column pruning.
Cheng
On 3/28/15 11:26 PM, Patrick Woody wrote:
Hi all,
In my application, we take input from Parquet files where BigDecimals are
written as Strings to maintain arbitrary precision.
I was hoping to convert these back over to Decimal with Unlimited
precision, but I'd still like to maintain the Parquet column pruning (all
my attempts thus far seem to bring in the whole Row). Is it possible to do
this lazily through catalyst?
Basically I'd want to do Cast(col, DecimalType()) whenever col is actually
referenced. Any tips on how to approach this would be appreciated.
Thanks!
-Pat
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
