I think some example code would help to understand what you are doing.
Am 18.12.19 um 08:12 schrieb Tzahi File:
no.. there're 100M records both even and odd
On Tue, Dec 17, 2019 at 8:13 PM Russell Spitzer
mailto:russell.spit...@gmail.com>> wrote:
Is there a chance your data is all even o
no.. there're 100M records both even and odd
On Tue, Dec 17, 2019 at 8:13 PM Russell Spitzer
wrote:
> Is there a chance your data is all even or all odd?
>
> On Tue, Dec 17, 2019 at 11:01 AM Tzahi File
> wrote:
>
>> I have in my spark sql query a calculated field that gets the value if
>> field
Is there a chance your data is all even or all odd?
On Tue, Dec 17, 2019 at 11:01 AM Tzahi File wrote:
> I have in my spark sql query a calculated field that gets the value if
> field1 % 3.
>
> I'm using this field as a partition so I expected to get 3 partitions in
> the mentioned case, and I d
I have in my spark sql query a calculated field that gets the value if
field1 % 3.
I'm using this field as a partition so I expected to get 3 partitions in
the mentioned case, and I do get. The issue happened with even numbers
(instead of 3 - 4,2 ... ).
When I tried to use even numbers, for exampl
Our build is complex; it uses a large number of third party jars and
generates an uber jar that is shaded before we pass it to spark submit. We
shade to avoid ClassLoader collisions with Spark platform dependencies
(e.g. protobuf 3).
Managing the dependencies/shade is cumbersome and error prone. T
