Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

That said, if you guys understand the proposal better and have a preference on one side, could you please participate in the VOTE thread? https://lists.apache.org/thread/nm3p1zjcybdl0p0mc56t2rl92hb9837n Specifically on this topic, I do think the input from users is very important, especially if yo

Re: PR review

May I please get review on the following outstanding PRs: https://github.com/apache/spark/pull/49276 (open on 12/23/2024) https://github.com/apache/spark/pull/49870 Thank you, Vlad On Feb 25, 2025, at 5:31 PM, Rozov, Vlad wrote: Thanks, looking for committers to review/discuss my pending PRs:

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

Thanks for the input. > From a quick glance it seems like the incorrect config would just be ignored from the checkpoint, and the new config would just be applied with the default value going forward. That's not how it works. https://github.com/apache/spark/blob/master/sql/core/src/main/scala/or

Re: [DISCUSS] New Spark Connect Client repository for Swift language

The first link seems to be still invalid, although the proposal itself is sound https://github.com/apache/spark-connect-swift Can someone else please confirm it? Dr Mich Talebzadeh, Architect | Data Science | Financial Crime | Forensic Analysis | GDPR view my Linkedin profile

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

+1 (non-binding) It's a pretty in the weeds issue with how Structured Streaming works under the hood that's kinda hard to understand if you're not familiar with it. The migration logic doesn't mean users can still use the old config, it's purely behind the scenes to fix checkpoint metadata in stre

Re: [DISCUSS] New Spark Connect Client repository for Swift language

+1 On Mon, Mar 10, 2025 at 6:48 AM Yang Jie wrote: > Great! Really happy to see that spark-connect supports more programming > languages. > > > On 2025/03/10 07:00:32 Martin Grund wrote: > > Great work and proposal! > > > > I'm supportive. > > > > On Sun, Mar 9, 2025 at 23:31 Dongjoon Hyun wrote

Re: [DISCUSS] New Spark Connect Client repository for Swift language

+1 Kent 在 2025年3月10日星期一，Max Gekk 写道： > +1 > > On Mon, Mar 10, 2025 at 6:17 PM Wenchen Fan wrote: > > > > +1 > > > > On Mon, Mar 10, 2025 at 8:02 AM L. C. Hsieh wrote: > >> > >> +1 > >> > >> Thanks Dongjoon for contributing to Swift implementation. > >> > >> On Mon, Mar 10, 2025 at 7:18 AM Hyu

Re: [DISCUSS] New Spark Connect Client repository for Swift language

+1 Thanks Dongjoon for contributing to Swift implementation. On Mon, Mar 10, 2025 at 7:18 AM Hyukjin Kwon wrote: > > +1 > On Mon, Mar 10, 2025 at 6:48 AM Yang Jie wrote: >> >> Great! Really happy to see that spark-connect supports more programming >> languages. >> >> >> On 2025/03/10 07:00:32

Re: [DISCUSS] New Spark Connect Client repository for Swift language

+1 (non-binding) On Mon, Mar 10, 2025 at 9:47 AM Peter Toth wrote: > +1 > > On Mon, Mar 10, 2025 at 5:39 PM Kent Yao wrote: > >> +1 >> >> Kent >> >> 在 2025年3月10日星期一，Max Gekk 写道： >> >>> +1 >>> >>> On Mon, Mar 10, 2025 at 6:17 PM Wenchen Fan wrote: >>> > >>> > +1 >>> > >>> > On Mon, Mar 10, 202

Re: [DISCUSS] New Spark Connect Client repository for Swift language

Thank you everyone for your support. New Apache Spark repository is created at the proposed location with ASF license and open for `Spark Connect Client for Swift language` contributions. https://github.com/apache/spark-connect-swift FYI, this repository will be managed in the same way with `

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

-1 because there exists a feasible migration path for Apache Spark 3.5.4 via Apache Spark 3.5.5. It's obvious that this Databricks' mistake already causes a huge communication cost in the Apache Spark community and is suggesting a burden to enforce us to handle at least two more PRs at 4.0.0 a

Re: [VOTE] Release Spark 4.0.0 (RC2)

I ran into an exception issue when playing around spark connect, more details can be found at https://issues.apache.org/jira/browse/SPARK-51451 pyspark.errors.exceptions.connect.AnalysisException: [UNSUPPORTED_GENERATOR.NESTED_IN_EXPRESSIONS] The generator is not supported: nested in expressions "

Re: [DISCUSS] New Spark Connect Client repository for Swift language

+1 (non-binding) On Mon, Mar 10, 2025 at 11:10 AM DB Tsai wrote: > +1 > > It's exciting to see Apple developing a Spark Connect Swift client, > showcasing Spark Connect as a truly language-agnostic protocol between the > client and the Spark driver. > > Swift, known for its power and intuitivene

Re: [DISCUSS] New Spark Connect Client repository for Swift language

+1 On Mon, Mar 10, 2025 at 5:39 PM Kent Yao wrote: > +1 > > Kent > > 在 2025年3月10日星期一，Max Gekk 写道： > >> +1 >> >> On Mon, Mar 10, 2025 at 6:17 PM Wenchen Fan wrote: >> > >> > +1 >> > >> > On Mon, Mar 10, 2025 at 8:02 AM L. C. Hsieh wrote: >> >> >> >> +1 >> >> >> >> Thanks Dongjoon for contribut

Re: [DISCUSS] New Spark Connect Client repository for Swift language

Thank you, Martin and Yang and Hyukjin. Dongjoon On 2025/03/10 14:17:06 Hyukjin Kwon wrote: > +1 > On Mon, Mar 10, 2025 at 6:48 AM Yang Jie wrote: > > > Great! Really happy to see that spark-connect supports more programming > > languages. > > > > > > On 2025/03/10 07:00:32 Martin Grund wrote:

Re: [DISCUSS] New Spark Connect Client repository for Swift language

+1 On Mon, Mar 10, 2025 at 8:02 AM L. C. Hsieh wrote: > +1 > > Thanks Dongjoon for contributing to Swift implementation. > > On Mon, Mar 10, 2025 at 7:18 AM Hyukjin Kwon wrote: > > > > +1 > > On Mon, Mar 10, 2025 at 6:48 AM Yang Jie wrote: > >> > >> Great! Really happy to see that spark-connec

[DISCUSS] Upgrade Hive compile time dependency to 4.0

Hi All, As Apache Hive announced EOL for Hive 2.x [1] and 3.x [2], should Spark be compiled against Hive 4.x and use it as default? Thank you, Vlad [1] https://lists.apache.org/thread/4ctrzfw60jkhc0hq2xoh1jpqxgt2zd93 [2] https://lists.apache.org/thread/99h6wr7nk4684r6tkcbm8ydfytgqy6f3 [3] http

Re: [DISCUSS] New Spark Connect Client repository for Swift language

+1 On Mon, Mar 10, 2025 at 6:17 PM Wenchen Fan wrote: > > +1 > > On Mon, Mar 10, 2025 at 8:02 AM L. C. Hsieh wrote: >> >> +1 >> >> Thanks Dongjoon for contributing to Swift implementation. >> >> On Mon, Mar 10, 2025 at 7:18 AM Hyukjin Kwon wrote: >> > >> > +1 >> > On Mon, Mar 10, 2025 at 6:48 A

Contribution to Spark SQL: new data type TIME

Hi Spark devs, I would like to invite you to develop the new data type TIME in Spark SQL. At the moment, there are > 10 sub-tasks in the umbrella JIRA: https://issues.apache.org/jira/browse/SPARK-51162 including some "starter" tasks. If you have some ideas and proposals where we need to support th

Unsubscribe

-- Thanks & Regards Yogitha R

Re: [DISCUSS] Upgrade Hive compile time dependency to 4.0

Yes I am all for it, as I use Hive with Oracle as its metastore extensively. Case in point, on 6th March A Hive user alluded to it and I quote "I just wanted to highlight that Hive 3.x line is EOL. It has various known security v

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

+1 to retain, to avoid problems for users at ~0 cost. On Mon, Mar 10, 2025 at 7:45 AM Jungtaek Lim wrote: > Hi dev, > > Please vote to retain migration logic of incorrect `spark.databricks.*` > configuration in Spark 4.0.x. > > - DISCUSSION: > https://lists.apache.org/thread/xzk9729lsmo397crdtk1