About how to read spark source code with a good way

Hi everyone ?? I am an engineer, I have been using spark, and I want to try to make spark better. I want to be a commitor. Before that, I want to know spark thoroughly, so who knows how to make it better Read spark source code, or recommend related blogs for me to learn. please somebody can hel

Out of scope RDDs not getting cleaned up

Hi, I am using spark 2.1 and I am leveraging spark streaming for my data pipeline. Now, in my case the batch size is 3 minutes and we persist couple of RDDs while processing a batch and after processing we rely on Spark's ContextCleaner to clean out RDDs which are no longer in scope. So we have s

Re: About how to read spark source code with a good way [Marketing Mail]

Hi, From my experience, I suggest to read both blogs and source code. Blogs will give you the high-level knowledge for the different parts of the source code. Iacovos On 18/8/20 3:53 ??.??., 2400 wrote: Hi everyone ?? I am an engineer, I have been using spark, and I want to try to make sp

Re: About how to read spark source code with a good way [Marketing Mail]

Hi Jack and Spark experts, Further to the question asked in this thread, what are some recommended resources (blog/videos) that have helped you to deep dive into the spark source code. Thanks Regards Joyan On Wed, Aug 19, 2020 at 11:06 AM Jack Kolokasis wrote: > Hi, > > From my experience, I

Re: About how to read spark source code with a good way [Marketing Mail]

Hi Joyan, check this link: https://github.com/jackkolokasis/SparkInternals Thanks Iacovos On 19/8/20 9:09 π.μ., joyan sil wrote: Hi Jack and Spark experts, Further to the question asked in this thread, what are some recommended resources (blog/videos) that have helped you to deep dive into