unsubscribe

unsubscribe

Re: Dev list policy on posting genAI hallucinations

Thanks to the committers and PMC members who chimed in on this thread. > On Oct 10, 2024, at 3:27 AM, Jungtaek Lim > wrote: > > I'd ask people to quote the part you got from GPT and explicitly call out the > part as "GPT-generated" so that people would consider that there is > hallucination.

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

The Synapse config "spark.yarn.executor.decommission.enabled" is the closest thing to the proposed config "spark.executor.decommission.enabled" we've seen so far, I was only remarking that. On the other hand, seems like the "decomission" config came out in Spark 3.4.0: https://archive.apache.org

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Ángel, https://spark.apache.org/docs/latest/configuration.html search through `decommission` in this page. The config you may have found from Synapse is spark."yarn".executor.decommission.enabled. And the question was even "k8s" and none of the information for the vendor was mentioned from the qu

[DISCUSS?] Adding some empty string check in to_date built-in function + warning in documentation

Hi, I opened a Jira ticket back in August, but it seems to have been overlooked. While it may not be a critical issue, I would appreciate if you could take a moment to consider it before deciding whether to close it. Here is the ticket for reference: SPARK-49288

Re: [DISCUSS] Support spark.ml on Spark Connect

You have my vote (btw, great idea, ML is so sexy nowadays 😉) El jue, 10 oct 2024 a las 3:19, Bobby () escribió: > Hi, > > I'd like to start a discussion about support spark.ml on Connect. With > this feature, Users don't need to change their code to run Spark ML cases > on Connect. > > Please re

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Looks like it actually exists ... but only for the Spark Synapse implementation ... https://learn.microsoft.com/en-us/answers/questions/1496283/purpose-of-spark-yarn-executor-decommission-enable Jay Han was asking for some config on k8s, so we shouldn't bring this config to the table, shoul

Re: Dev list policy on posting genAI hallucinations

minor clarification: content from GPT might be valuable - what I meant was, the value of content from GPT "in the community". The content is either something people in the thread knows, or something they can query to GPT on their own. Though I barely use GPT for the query which matters with experti

Re: Dev list policy on posting genAI hallucinations

+1 on this. I'd ask people to quote the part you got from GPT and explicitly call out the part as "GPT-generated" so that people would consider that there is hallucination. Do not ever expect that people wouldn't know the content GPT wrote. "People know." ASF requires every code contribution to e

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Mich: you can set any key-value pair you want in Spark config. It doesn't mean it is a real flag that code reads. spark.conf.set("ham", "sandwich") print(spark.conf.get("ham")) prints "sandwich" forceKillTimeout is a real config: https://github.com/apache/spark/blob/fed9a8da3d4187794161e0be325aa

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

sorry, a missed line spark = SparkSession.builder \ .appName("Verifying Spark Configurations") \ .config("spark.executor.decommission.enabled", "true") \ *.config("spark.executor.decommission.gracefulShutdown", "true")* \ .config("spark.executor.decommission.forceKillTimeout", "100

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Let us take this for a ride using these so called non-existent configuration settings spark.executor.decommission.enabled=true spark.executor.decommission.gracefulShutdown=true Tested on Spark 3.4 from pyspark.sql import SparkSession # Initialize a Spark session spark = SparkSession.builder \

Re: Dev list policy on posting genAI hallucinations

Agree, I can't really explain this post except as AI hallucination, because: - those configs don't exist and it's not a simple typo away from a real one - they are kind of like unrelated real Spark config names and the kind of thing it seems an AI would 'infer' - no claim it was a typo with plausi

Re: Dev list policy on posting genAI hallucinations

FWIW - Mich - I've often found your responses "gpt" like and can often be a distraction. Now I don't know if that's your actual writing style or you were indeed using genai tools to generate the responses on your behalf. I don't think we should sanction you if that's your writing style. But if you

Dev list policy on posting genAI hallucinations

Mich, You are “helping” someone by giving them configs that don’t exist. And this is the second time that I am aware of that you are doing this. Both times you have done this and I have asked you to explain where you are getti

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Do you have a better recommendation? Or trying to waste time as usual. It is far easier to throw than catch. Do your homework and stop throwing spanners at work. Mich Talebzadeh, Architect | Data Engineer | Data Science | Financial Crime PhD

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Mich, Can you please share with the list where exactly you are citing these configs from? As far as I can tell, these two configs don’t exist and have never existed in the Spark codebase: spark.executor.decommission.enabled=true spark.executor.decommission.gracefulShutdown=true Where exactly

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

spark.executor.decommission.gracefulShutdown=true WHAT? ons. 9. okt. 2024 kl. 12:30 skrev Mich Talebzadeh : > Before responding, what configuration parameters are you using to make > this work? > > spark.executor.decommission.enabled=true > spark.executor.decommission.gracefulShutdown=true > spa

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Before responding, what configuration parameters are you using to make this work? spark.executor.decommission.enabled=true spark.executor.decommission.gracefulShutdown=true spark.executor.decommission.forceKillTimeout=100s HTH Mich Talebzadeh, Architect | Data Engineer | Data Science | Financia

[Question] Why driver doesn't shutdown executors gracefully on k8s?

Hi spark community, I have such a question: Why driver doesn't shutdown executors gracefully on k8s. For instance, kubernetesClient.pods().withGracePeriod(100).delete(). -- Best, Jay