I don't know why I don't see my last message in the thread here: https://lists.apache.org/thread/5wgdqp746nj4f6ovdl42rt82wc8ltkcn Also don't get messages from Artemis in my mail, I can only see them in the thread web UI, which is very confusing. On top of that when I click on "reply via your own email client" in the web UI, I get: Bad Request Error 400
Anyways to answer to your last comment Artemis: > I guess there are several misconceptions here: There's no confusion on my side, all that makes sense. When I said "worker" in that comment I meant the scheduler worker not Spark worker, which in the Spark realm would be the client. Everything else you said is undoubtedly correct, but unrelated to the issue/problem at hand. Sean, Artemis - I appreciate your feedback about the infra setup, but it's beside the problem behind this issue. Let me describe a simpler setup/example with the same problem, say: 1. I have a jupyter notebook 2. use local/driver spark mode only 3. I start the driver, process some data, store it in pandas dataframe 4. now say I want to add a package to spark driver (or increase the JVM memory etc) There's currently no way to do the step 4 without restarting the notebook process which holds the "reference" to the Spark driver/JVM. If I restart the Jupter notebook I would lose all the data in memory (e.g. pandas data), ofc I can save that data to e.g. disk but that's beside the point. I understand you don't want to provide this functionality in Spark, nor warn users on changes in Spark Configuration that won't actually work - as a user I wish I could get at least a warning in that case, but I respect your decision. It seems like the workaround to shutdown the JVM works in this case, I would much appreciate your feedback about **that specific workaround** please. Any reason not to use it? Cheers - Rafal On Thu, 10 Mar 2022 at 18:50, Rafał Wojdyła <ravwojd...@gmail.com> wrote: > If you have a long running python orchestrator worker (e.g. Luigi worker), > and say it's gets a DAG of A -> B ->C, and say the worker first creates a > spark driver for A (which doesn't need extra jars/packages), then it gets B > which is also a spark job but it needs an extra package, it won't be able > to create a new spark driver with extra packages since it's "not possible" > to create a new driver JVM. I would argue it's the same scenario if you > have multiple spark jobs that need different amounts of memory or anything > that requires JVM restart. Of course I can use the workaround to shut down > the driver/JVM, do you have any feedback about that workaround (see my > previous comment or the issue). > > On Thu, 10 Mar 2022 at 18:12, Sean Owen <sro...@gmail.com> wrote: > >> Wouldn't these be separately submitted jobs for separate workloads? You >> can of course dynamically change each job submitted to have whatever >> packages you like, from whatever is orchestrating. A single job doing >> everything sound right. >> >> On Thu, Mar 10, 2022, 12:05 PM Rafał Wojdyła <ravwojd...@gmail.com> >> wrote: >> >>> Because I can't (and should not) know ahead of time which jobs will be >>> executed, that's the job of the orchestration layer (and can be dynamic). I >>> know I can specify multiple packages. Also not worried about memory. >>> >>> On Thu, 10 Mar 2022 at 13:54, Artemis User <arte...@dtechspace.com> >>> wrote: >>> >>>> If changing packages or jars isn't your concern, why not just specify >>>> ALL packages that you would need for the Spark environment? You know you >>>> can define multiple packages under the packages option. This shouldn't >>>> cause memory issues since JVM uses dynamic class loading... >>>> >>>> On 3/9/22 10:03 PM, Rafał Wojdyła wrote: >>>> >>>> Hi Artemis, >>>> Thanks for your input, to answer your questions: >>>> >>>> > You may want to ask yourself why it is necessary to change the jar >>>> packages during runtime. >>>> >>>> I have a long running orchestrator process, which executes multiple >>>> spark jobs, currently on a single VM/driver, some of those jobs might >>>> require extra packages/jars (please see example in the issue). >>>> >>>> > Changing package doesn't mean to reload the classes. >>>> >>>> AFAIU this is unrelated >>>> >>>> > There is no way to reload the same class unless you customize the >>>> classloader of Spark. >>>> >>>> AFAIU this is an implementation detail. >>>> >>>> > I also don't think it is necessary to implement a warning or error >>>> message when changing the configuration since it doesn't do any harm >>>> >>>> To reiterate right now the API allows to change configuration of the >>>> context, without that configuration taking effect. See example of confused >>>> users here: >>>> * >>>> https://stackoverflow.com/questions/41886346/spark-2-1-0-session-config-settings-pyspark >>>> * >>>> https://stackoverflow.com/questions/53606756/how-to-set-spark-driver-memory-in-client-mode-pyspark-version-2-3-1 >>>> >>>> I'm curious if you have any opinion about the "hard-reset" workaround, >>>> copy-pasting from the issue: >>>> >>>> ``` >>>> s: SparkSession = ... >>>> >>>> # Hard reset: >>>> s.stop() >>>> s._sc._gateway.shutdown() >>>> s._sc._gateway.proc.stdin.close() >>>> SparkContext._gateway = None >>>> SparkContext._jvm = None >>>> ``` >>>> >>>> Cheers - Rafal >>>> >>>> On 2022/03/09 15:39:58 Artemis User wrote: >>>> > This is indeed a JVM issue, not a Spark issue. You may want to ask >>>> > yourself why it is necessary to change the jar packages during >>>> runtime. >>>> > Changing package doesn't mean to reload the classes. There is no way >>>> to >>>> > reload the same class unless you customize the classloader of Spark. >>>> I >>>> > also don't think it is necessary to implement a warning or error >>>> message >>>> > when changing the configuration since it doesn't do any harm. Spark >>>> > uses lazy binding so you can do a lot of such "unharmful" things. >>>> > Developers will have to understand the behaviors of each API before >>>> when >>>> > using them.. >>>> > >>>> > >>>> > On 3/9/22 9:31 AM, Rafał Wojdyła wrote: >>>> > > Sean, >>>> > > I understand you might be sceptical about adding this functionality >>>> > > into (py)spark, I'm curious: >>>> > > * would error/warning on update in configuration that is currently >>>> > > effectively impossible (requires restart of JVM) be reasonable? >>>> > > * what do you think about the workaround in the issue? >>>> > > Cheers - Rafal >>>> > > >>>> > > On Wed, 9 Mar 2022 at 14:24, Sean Owen <sr...@gmail.com> wrote: >>>> > > >>>> > > Unfortunately this opens a lot more questions and problems than >>>> it >>>> > > solves. What if you take something off the classpath, for >>>> example? >>>> > > change a class? >>>> > > >>>> > > On Wed, Mar 9, 2022 at 8:22 AM Rafał Wojdyła >>>> > > <ra...@gmail.com> wrote: >>>> > > >>>> > > Thanks Sean, >>>> > > To be clear, if you prefer to change the label on this issue >>>> > > from bug to sth else, feel free to do so, no strong opinions >>>> > > on my end. What happens to the classpath, whether spark uses >>>> > > some classloader magic, is probably an implementation >>>> detail. >>>> > > That said, it's definitely not intuitive that you can change >>>> > > the configuration and get the context (with the updated >>>> > > config) without any warnings/errors. Also what would you >>>> > > recommend as a workaround or solution to this problem? Any >>>> > > comments about the workaround in the issue? Keep in mind >>>> that >>>> > > I can't restart the long running orchestration process >>>> (python >>>> > > process if that matters). >>>> > > Cheers - Rafal >>>> > > >>>> > > On Wed, 9 Mar 2022 at 13:15, Sean Owen <sr...@gmail.com> >>>> wrote: >>>> > > >>>> > > That isn't a bug - you can't change the classpath once >>>> the >>>> > > JVM is executing. >>>> > > >>>> > > On Wed, Mar 9, 2022 at 7:11 AM Rafał Wojdyła >>>> > > <ra...@gmail.com> wrote: >>>> > > >>>> > > Hi, >>>> > > My use case is that, I have a long running process >>>> > > (orchestrator) with multiple tasks, some tasks might >>>> > > require extra spark dependencies. It seems once the >>>> > > spark context is started it's not possible to update >>>> > > `spark.jars.packages`? I have reported an issue at >>>> > > https://issues.apache.org/jira/browse/SPARK-38438, >>>> > > together with a workaround ("hard reset of the >>>> > > cluster"). I wonder if anyone has a solution for >>>> this? >>>> > > Cheers - Rafal >>>> > > >>>> > >>>> >>>>> >>>>
