Yes, drop malformed does filter out record4. However, record 5 is not.
On Tue, 4 Jul 2023 at 07:41, Vikas Kumar wrote:
> Have you tried dropmalformed option ?
>
> On Mon, Jul 3, 2023, 1:34 PM Shashank Rao wrote:
>
>> Update: Got it working by using the *_corrupt_record *field for the
>> first c
I think you can define schema with column z and filter out records with z
is null.
On Tue, Jul 4, 2023 at 3:24 PM Shashank Rao wrote:
> Yes, drop malformed does filter out record4. However, record 5 is not.
>
> On Tue, 4 Jul 2023 at 07:41, Vikas Kumar wrote:
>
>> Have you tried dropmalformed op
Z is just an example. It could be anything. Basically, anything that's not
in schema should be filtered out.
On Tue, 4 Jul 2023, 13:27 Hill Liu, wrote:
> I think you can define schema with column z and filter out records with z
> is null.
>
> On Tue, Jul 4, 2023 at 3:24 PM Shashank Rao
> wrote:
Dear spark users,
I'm experiencing an unusual issue with Spark 3.4.x.
When creating a new column as the sum of several existing columns, the time
taken almost doubles as the number of columns increases. This operation doesn't
require much resources, so I suspect there might be a problem with the
Hi team,
Any updates on this below issue
On Mon, 3 Jul 2023 at 6:18 PM, elango vaidyanathan
wrote:
>
>
> Hi all,
>
> I am reading a parquet file like this and it gives
> java.lang.IllegalArgumentException.
> However i can work with other parquet files (such as nyc taxi parquet
> files) without
