Re: FYI: A Hallucination about Spark Connect Stability in Spark 4

Thanks for update and looking into it. Excuse the thumb typos On Tue, 21 Jan 2025 at 4:09 PM, Hyukjin Kwon wrote: > Just a quick note on that: the major reason is 1. OOM we should figure out > and fix the CI environment. 2. structured streaming test failure that is > still in development. > I

Re: FYI: A Hallucination about Spark Connect Stability in Spark 4

I'm passionate about and have lots of experience fixing OOMs. Contact me if you need some help. El mié, 22 ene 2025, 1:10, Hyukjin Kwon escribió: > Just a quick note on that: the major reason is 1. OOM we should figure out > and fix the CI environment. 2. structured streaming test failure that i

Re: FYI: A Hallucination about Spark Connect Stability in Spark 4

Thank you, Hyukjin! Dongjoon On Tue, Jan 21, 2025 at 16:10 Hyukjin Kwon wrote: > Just a quick note on that: the major reason is 1. OOM we should figure out > and fix the CI environment. 2. structured streaming test failure that is > still in development. > I made an umbrella JIRA (https://issue

Re: FYI: A Hallucination about Spark Connect Stability in Spark 4

Just a quick note on that: the major reason is 1. OOM we should figure out and fix the CI environment. 2. structured streaming test failure that is still in development. I made an umbrella JIRA (https://issues.apache.org/jira/browse/SPARK-50907), and I will work there. Should be easier to look at w

Re: FYI: A Hallucination about Spark Connect Stability in Spark 4

Let me take a look. shouldn't be a major issue. On Wed, 22 Jan 2025 at 08:31, Mich Talebzadeh wrote: > As discussed on a thread over the weekend, we agreed among us including > Matei on a shift towards a more stable and version-independent APIs. > Spark Connect IMO is a key enabler of this shi

Re: FYI: A Hallucination about Spark Connect Stability in Spark 4

As discussed on a thread over the weekend, we agreed among us including Matei on a shift towards a more stable and version-independent APIs. Spark Connect IMO is a key enabler of this shift, allowing users and developers to build applications and libraries that are more resilient to changes in Sp

Re: FYI: A Hallucination about Spark Connect Stability in Spark 4

To be clear, (1) is `PySpark 4.0 Client` + `Spark 4.0 Server`, which is more severe. And, your point matches with (2) exactly. Thank you for your reply, Holden. Dongjoon. On 2025/01/21 22:38:20 Holden Karau wrote: > Interesting. So given one of the features of Spark connect should be > simpler

Re: FYI: A Hallucination about Spark Connect Stability in Spark 4

Interesting. So given one of the features of Spark connect should be simpler migrations we should (in my mind) only declare it stable once we’ve gone through two releases where the previous client + its code can talk to the new server. Twitter: https://twitter.com/holdenkarau Fight Health Insuranc

FYI: A Hallucination about Spark Connect Stability in Spark 4

It seems that there is misinformation about the stability of Spark Connect in Spark 4. I would like to reduce the gap in our dev mailing list. Frequently, some people claim `Spark Connect` is stable because it uses Protobuf. Yes, we standardize the interface layer. However, may I ask if it implies

Re: How do I repackage org.spark-project.hive-exec-1.2.1.spark2

Well, I had an old archive. This is on linux. Create an sh file say* procedure.sh *using vi and put the commands below there. Don't forget to do *chmod +x procedure.sh a*t the end after saving the file #!/bin/bash # Exit immediately if a command exits with a non-zero status set -e # Move to the te

Re: [DISCUSS] Support spark.ml on Spark Connect

Given our recent discussion on using spark connect as a stable API, this will be another positive step. HTH Mich Talebzadeh, Architect | Data Science | Financial Crime | Forensic Analysis | GDPR view my Linkedin profile On Tue, 2