Thanks Cheng, Michael - that was super helpful.
On Sun, Dec 21, 2014 at 7:27 AM, Cheng Lian wrote:
> Would like to add that compression schemes built in in-memory columnar
> storage only supports primitive columns (int, string, etc.), complex types
> like array, map and struct are not supported
Would like to add that compression schemes built in in-memory columnar
storage only supports primitive columns (int, string, etc.), complex
types like array, map and struct are not supported.
On 12/20/14 6:17 AM, Sadhan Sood wrote:
Hey Michael,
Thank you for clarifying that. Is tachyon the ri
Thanks Michael, that makes sense.
On Fri, Dec 19, 2014 at 3:13 PM, Michael Armbrust
wrote:
> Yeah, tachyon does sound like a good option here. Especially if you have
> nested data, its likely that parquet in tachyon will always be better
> supported.
>
> On Fri, Dec 19, 2014 at 2:17 PM, Sadhan
Yeah, tachyon does sound like a good option here. Especially if you have
nested data, its likely that parquet in tachyon will always be better
supported.
On Fri, Dec 19, 2014 at 2:17 PM, Sadhan Sood wrote:
>
> Hey Michael,
>
> Thank you for clarifying that. Is tachyon the right way to get compre
Hey Michael,
Thank you for clarifying that. Is tachyon the right way to get compressed
data in memory or should we explore the option of adding compression to
cached data. This is because our uncompressed data set is too big to fit in
memory right now. I see the benefit of tachyon not just with st
There is only column level encoding (run length encoding, delta encoding,
dictionary encoding) and no generic compression.
On Thu, Dec 18, 2014 at 12:07 PM, Sadhan Sood wrote:
>
> Hi All,
>
> Wondering if when caching a table backed by lzo compressed parquet data,
> if spark also compresses it (u
Hi All,
Wondering if when caching a table backed by lzo compressed parquet data, if
spark also compresses it (using lzo/gzip/snappy) along with column level
encoding or just does the column level encoding when
"*spark.sql.inMemoryColumnarStorage.compressed"
*is set to true. This is because when I
