Re: Adding pause() method to pyspark.sql.streaming.StreamingQuery

Hi Mich. So it sounds like what you're really after is a way to apply new stream options in runtime without downtime? BR, Martin From: Mich Talebzadeh Sent: Tuesday, March 14, 2023 16:39 To: Martin Andersson Cc: Spark dev list Subject: Re: Adding pause() metho

Re: Adding pause() method to pyspark.sql.streaming.StreamingQuery

Hi Martin. Yes, that is the intent. There may be other ways, but I cannot think of. HTH view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility fo

Re: Topics for Spark online classes & webinars

Hi all, Thanks to @Denny Lee to give access to https://www.linkedin.com/company/apachespark/ and contribution from @asma zgolli You will see my post at the bottom. Please add anything else on topics to the list as a comment. We will then put them together in an article perhaps. Comments and

Re: Topics for Spark online classes & webinars

Thanks Mich for tackling this! I encourage everyone to add to the list so we can have a comprehensive list of topics, eh?! On Wed, Mar 15, 2023 at 10:27 Mich Talebzadeh wrote: > Hi all, > > Thanks to @Denny Lee to give access to > > https://www.linkedin.com/company/apachespark/ > > and contri

Re: Topics for Spark online classes & webinars

Great. A case that I hope can be better documented, especially now that we have Pandas API on Spark and many potential new users coming from Pandas. Is how to start Spark with full available memory and CPU. I use this function to do this in a notebook. import multiprocessing import os import sys f

Re: Topics for Spark online classes & webinars

Hi Nitin, Linkedin is more of a professional media. FYI, I am only a member of Linkedin, no facebook, etc.There is no reason for you NOT to create a profile for yourself in linkedin :) https://www.linkedin.com/help/linkedin/answer/a1338223/sign-up-to-join-linkedin?lang=en see you there as well

Re: Topics for Spark online classes & webinars

Understood Nitin It would be wrong to act against one's conviction. I am sure we can find a way around providing the contents Regards Mich Talebzadeh, Lead Solutions Architect/Engineering Lead Palantir Technologies Limited view my Linkedin profile

Re: Topics for Spark online classes & webinars

What we can do is get into the habit of compiling the list on LinkedIn but making sure this list is shared and broadcast here, eh?! As well, when we broadcast the videos, we can do this using zoom/jitsi/ riverside.fm as well as simulcasting this on LinkedIn. This way you can view directly on the f