Re: Join stucks in the last stage step

Just to demonstrate BIG difference between ordinary task (id 450) and last remaining task (id 0) Index ID Attempt Status ▾Locality LevelLaunch Time DurationGC Time Shuffle ReadShuffle Spill (Memory) Shuffle Spill (Disk)Errors 0 24130 RUNNING 2

Join stucks in the last stage step

Hello, I have problem with join of two tables via Spark - I have tried to do it via Spark SQL and API but no progress so far. I have basicaly two tables ACCONTS - 16 mio records and TRANSACTIONS 2,5 billion records. When I try to join the tables (please see code) the job stucks in the last stage