The key is this line from Amit's email (emphasis added):
> Change the join_col to *all possible values* of the sale.
The two tables are treated asymmetrically:
1. The skewed table gets random salts appended to the join key.
2. The other table gets all possible salts appended to the join key (e.g
I think you are looking for the spark.task.cpus configuration parameter.
Op vr 29 jul. 2022 om 07:41 schreef Andrew Melo :
> Hello,
>
> Is there a way to tell Spark that PySpark (arrow) functions use
> multiple cores? If we have an executor with 8 cores, we would like to
> have a single PySpark f
Hi Ivan,
Unlike cache/persist, checkpoint does not operate in-place but requires the
result to be assigned to a new variable. In your case:
val recordsRDD = convertToRecords(anotherRDD).checkpoint()
Best,
Jacob
Op wo 19 aug. 2020 om 14:39 schreef Ivan Petrov :
> Hi!
> Seems like I do smth wron
You are overflowing the integer type, which goes up to a max value
of 2147483647 (2^31 - 1). Change the return type of `sha2Int2` to
`LongType()` and it works as expected.
On Mon, Mar 23, 2020 at 6:15 AM ayan guha wrote:
> Hi
>
> I am trying to implement simple hashing/checksum logic. The key lo
issue:
https://issues.apache.org/jira/browse/SPARK-1239.
On Mon, Nov 11, 2019 at 4:43 PM Vadim Semenov wrote:
> There's an umbrella ticket for various 2GB limitations
> https://issues.apache.org/jira/browse/SPARK-6235
>
> On Fri, Nov 8, 2019 at 4:11 PM Jacob Lynn wrote:
> >
&g
Sorry for the noise, folks! I understand that reducing the number of
partitions works around the issue (at the scale I'm working at, anyway) --
as I mentioned in my initial email -- and I understand the root cause. I'm
not looking for advice on how to resolve my issue. I'm just pointing out
that th
File system is HDFS. Executors are 2 cores, 14GB RAM. But I don't think
either of these relate to the problem -- this is a memory allocation issue
on the driver side, and happens in an intermediate stage that has no HDFS
read/write.
On Fri, Nov 8, 2019 at 10:01 AM Spico Florin wrote:
> Hi!
> Wha
