Re: [VOTE] Release Spark 3.5.7 (RC1)

oday. On Thu, Jul 17, 2025 at 6:30 AM Rozov, Vlad mailto:vro...@amazon.com>> wrote: What will be the next version on 3.5 branch? Will it be 3.5.7 or 3.5.8 (version in the pom files)? Thank you, Vlad From: Hyukjin Kwon mailto:gurwls...@apache.org>> Date: Monday, June 9, 2025 at

Re: [VOTE] Release Spark 3.5.7 (RC1)

What will be the next version on 3.5 branch? Will it be 3.5.7 or 3.5.8 (version in the pom files)? Thank you, Vlad From: Hyukjin Kwon Date: Monday, June 9, 2025 at 9:01 AM To: "dev@spark.apache.org" Subject: RE: [EXTERNAL] [VOTE] Release Spark 3.5.7 (RC1) These RC artifacts were dropped prop

Re: [VOTE] Release Spark 4.1.0-preview1 (RC1)

+1 (non-binding) Thank you, Vlad From: Dongjoon Hyun Date: Tuesday, July 8, 2025 at 8:09 AM To: Hyukjin Kwon Cc: "dev@spark.apache.org" Subject: RE: [EXTERNAL] [VOTE] Release Spark 4.1.0-preview1 (RC1) +1 Dongjoon On Tue, Jul 8, 2025 at 05:41 Hyukjin Kwon mailto:gurwls...@apache.org>> wro

Re: [VOTE] Release Apache Spark Connect Go Client 0.1.0

+1 (non-binding) Thank you, Vlad On Jun 10, 2025, at 10:44 AM, Sakthi wrote: +1 (non-binding) On Mon, Jun 9, 2025 at 8:28 PM bo yang mailto:bobyan...@gmail.com>> wrote: +1 (non-binding), thanks Martin! On Mon, Jun 9, 2025 at 7:47 PM Cheng Pan mailto:pan3...@gmail.com>> wrote: +1 (non-bindi

[DISCUSS] SPIP: Upgrade Apache Hive to 4.x

Hi All, I want to start a discussion thread on the SPIP titled "Upgrade Apache Hive to 4.x” [JIRA][Doc] that I was researching for some time now.

Re: [VOTE] Release Apache Spark Connect Swift Client 0.3.0 (RC1)

+1 (non-binding) Thank you, Vlad On Jun 1, 2025, at 7:21 PM, Wenchen Fan wrote: +1 On Mon, Jun 2, 2025 at 9:55 AM Yuanjian Li mailto:xyliyuanj...@gmail.com>> wrote: +1 On Sun, Jun 1, 2025 at 18:30 DB Tsai mailto:dbt...@dbtsai.com>> wrote: +1 Sent from my iPhone > On Jun 1, 2025, at 2:32 

Re: [VOTE] SPIP: Real-Time Mode in Apache Spark Structured Streaming

+1 (non-binding) Thank you, Vlad On Jun 2, 2025, at 7:08 AM, Wenchen Fan wrote: +1 On Mon, Jun 2, 2025 at 8:55 PM Peter Toth mailto:peter.t...@gmail.com>> wrote: +1 On Mon, Jun 2, 2025 at 2:33 PM xianjin mailto:xian...@apache.org>> wrote: +1. Sent from my iPhone On Jun 2, 2025, at 12:50 P

Re: [VOTE] Release Apache Spark K8s Operator 0.3.0 (RC1)

+1 (non-binding) Thank you, Vlad On Jun 1, 2025, at 7:44 PM, Liu Cao wrote: +1 (non-binding) On Sun, Jun 1, 2025 at 7:22 PM Wenchen Fan mailto:cloud0...@gmail.com>> wrote: +1 On Mon, Jun 2, 2025 at 9:30 AM DB Tsai mailto:dbt...@dbtsai.com>> wrote: +1 Sent from my iPhone > On Jun 1, 2025,

Re: [VOTE] Release Spark 3.5.6 (RC1)

wrote: The key issue was fixed. On Mon, 26 May 2025 at 10:05, Hyukjin Kwon wrote: Probably should avoid backporting it for improvements but If there is a CVE that directly affects Spark, let's upgrade. On Mon, 26 May 2025 at 00:27, Rozov, Vlad wrote: Should parquet version be upgraded t

Re: [VOTE] Release Spark 3.5.6 (RC1)

CVEs actually affects Spark, and create a backport if so. For improvements, it is generally not backported down to old branches On Wed, 28 May 2025 at 01:17, Rozov, Vlad wrote: Hi Dongjoon, > I guess you wanted to propose Apache Parquet 1.5.2 backport instead. Correct, that was my question:

Re: [VOTE] Release Spark 3.5.6 (RC1)

Upgrade Parquet to 1.15.2 > https://github.com/apache/spark/pull/50755 > > Dongjoon. > > On 2025/05/26 03:08:32 "Rozov, Vlad" wrote: >> There is an existing PR that was reverted due to a deadlock. As deadlock is >> now fixed, the revert can now b

Re: [VOTE] Release Spark 3.5.6 (RC1)

, Vlad On May 25, 2025, at 6:05 PM, Hyukjin Kwon wrote: Probably should avoid backporting it for improvements but If there is a CVE that directly affects Spark, let's upgrade. On Mon, 26 May 2025 at 00:27, Rozov, Vlad wrote: Should parquet version be upgraded to 1.15.1 or 1.15.2? There a

Re: [VOTE] Release Spark 3.5.6 (RC1)

Should parquet version be upgraded to 1.15.1 or 1.15.2? There are 10 CVEs in the current 1.13.1 and even though they may not impact Spark there are other improvements (better performance) that will benefit Spark users. Thank you, Vlad On May 24, 2025, at 8:02 PM, Hyukjin Kwon wrote: Oh let m

Re: [VOTE] Release Spark 4.0.0 (RC7)

+1 (non-binding) Vlad On May 19, 2025, at 8:56 PM, Jules Damji wrote: + 1 (non-binding) — Sent from my iPhone Pardon the dumb thumb typos :) On May 19, 2025, at 5:26 PM, Gengliang Wang wrote:  +1 On Mon, May 19, 2025 at 5:21 PM Jungtaek Lim mailto:kabhwan.opensou...@gmail.com>> wrote: +1

Re: [VOTE] Release Apache Spark Connect Swift Client 0.2.0 (RC1)

+1 (non-binding) Vlad On May 17, 2025, at 4:30 PM, Zhou Jiang wrote: +1 (non-binding) On May 17, 2025, at 16:28, Hyukjin Kwon wrote:  +1 On Sun, 18 May 2025 at 07:47, L. C. Hsieh mailto:vii...@gmail.com>> wrote: +1 Thanks Dongjoon. On Sat, May 17, 2025 at 5:40 AM Dongjoon Hyun mailto:d

Re: [VOTE] Release Apache Spark K8s Operator 0.2.0 (RC1)

+1 (non-binding) Vlad On May 18, 2025, at 7:02 AM, Peter Toth wrote: +1 On Sun, May 18, 2025 at 1:29 AM Hyukjin Kwon mailto:gurwls...@apache.org>> wrote: +1 On Sun, 18 May 2025 at 07:37, huaxin gao mailto:huaxin.ga...@gmail.com>> wrote: +1 Thanks Dongjoon! On Sat, May 17, 2025 at 2:44 PM L

Re: [VOTE] Release Spark 4.0.0 (RC6)

+1 to wait for the fix. Thank you, Vlad On May 14, 2025, at 5:30 AM, Weichen Xu wrote: Hi folks, We have a RCE fix https://github.com/apache/spark/pull/50889 pending merging, and it needs to be backported to 4.0.0. Shall we wait for it ? Thanks ! On Wed, May 14, 2025 at 7:19 PM Peter Toth

Re: [VOTE] Release Spark 4.0.0 (RC5)

+1 (non-binding) Thank you, Vlad On May 12, 2025, at 5:44 PM, huaxin gao wrote: +1 On Mon, May 12, 2025 at 5:34 PM Hyukjin Kwon mailto:gurwls...@apache.org>> wrote: +1 On Tue, 13 May 2025 at 03:24, Xinrong Meng mailto:xinr...@apache.org>> wrote: +1 Thank you Wenchen! On Mon, May 12, 2025

Re: [DISCUSS] SPARK-51318: Remove `jar` files from Apache Spark repository and disable affected tests

com/apache/spark/pull/50378, CI passed. Now it has 9 +1s from all different groups. Why do we need to change the way? I don't think we should override the community consensus because you think the approach is hacky. On Wed, 26 Mar 2025 at 11:40, Rozov, Vlad wrote: I think that ther

Re: [VOTE] Release Apache Spark Connect Swift Client 0.1.0 (RC1)

+1 (non-binding) On May 6, 2025, at 6:33 AM, Sakthi wrote: +1 (non-binding) On Tue, May 6, 2025 at 2:00 AM Jungtaek Lim mailto:kabhwan.opensou...@gmail.com>> wrote: +1 (non-binding) Nice addition on Spark Connect! On Tue, May 6, 2025 at 5:47 PM Peter Toth mailto:peter.t...@gmail.com>> wrote

Re: [VOTE] SPIP: Add geospatial types to Spark

+1 (non-binding) On May 6, 2025, at 6:34 AM, Sakthi wrote: +1 (non-binding) On Tue, May 6, 2025 at 1:30 AM Max Gekk mailto:max.g...@gmail.com>> wrote: +1 On Tue, May 6, 2025 at 9:59 AM Yang Jie mailto:yangji...@apache.org>> wrote: +1 On 2025/05/06 03:25:21 "L. C. Hsieh" wrote: > +1 > > On

Re: [VOTE] Release Apache Spark K8s Operator 0.1.0 (RC1)

+1 (not binding) Checked checksum and signatures, build, and confirmed that binary files are not included into the release. Is Apache RAT part of the gradle build? If not, how headers are validated to include correct license? Thank you, Vlad > On May 4, 2025, at 5:38 PM, Dongjoon Hyun wr

Re: [VOTE] Release Spark 4.0.0 (RC4)

0 AM Manu Zhang mailto:owenzhang1...@gmail.com>> wrote: I don't think PARQUET-2432 has any issue itself. It looks to have triggered a deadlock case like https://github.com/apache/spark/pull/50594. I'd suggest that we fix forward if possible. Thanks, Manu On Mon, Apr 21, 2025 at 1

Re: [VOTE] Release Spark 4.0.0 (RC4)

The deadlock is reproducible without Parquet. Please see https://github.com/apache/spark/pull/50594. Thank you, Vlad On Apr 21, 2025, at 1:59 AM, Cheng Pan wrote: The deadlock is introduced by PARQUET-2432(1.14.0), if we decide downgrade, the latest workable version is Parquet 1.13.1. Thank

Re: [VOTE] Release Spark 4.0.0 (RC4)

It may not be the Parquet introduced issue. It looks like a race condition between Spark UninterruptibleThread and Hadoop/HDFS DFSOutputStream. I tried to resolve the deadlock in https://github.com/apache/spark/pull/50594. Can you give it a try? I will see if I can reproduce the deadlock in a un

Re: Revert of [SPARK-51229][BUILD][CONNECT] Fix dependency:analyze goal on connect common

you, Vlad On Mar 26, 2025, at 3:18 PM, Hyukjin Kwon wrote: That only fixes Maven. Both SBT build and Maven build should work in the same or similar wat. Let's make sure both work. On Thu, Mar 27, 2025 at 3:18 AM Rozov, Vlad wrote: Please see https://github.com/vrozov/spark/tree/spark-she

Re: Revert of [SPARK-51229][BUILD][CONNECT] Fix dependency:analyze goal on connect common

, Hyukjin Kwon wrote: Vlad, let's open a PR and discuss it there. We have many other committees to review / help with as well. On Fri, Mar 28, 2025 at 6:28 AM Rozov, Vlad wrote: Hi Hyukjin, I open https://issues.apache.org/jira/browse/SPARK-51643 and https://issues.apache.org/jira/browse/

Re: Revert of [SPARK-51229][BUILD][CONNECT] Fix dependency:analyze goal on connect common

anges usually are shared between SBT and Maven so it is not a tricky part. However, some changes like this have to be made in both places. Both Maven and SBT builds are usually sync, and changes are mode together. Let me know if I am underding your change wrongly. On Thu, 27 Mar 2025 at 09:25, R

Re: Revert of [SPARK-51229][BUILD][CONNECT] Fix dependency:analyze goal on connect common

tricky part. However, some changes like this have to be made in both places. Both Maven and SBT builds are usually sync, and changes are mode together. Let me know if I am underding your change wrongly. On Thu, 27 Mar 2025 at 09:25, Rozov, Vlad wrote: Hi Hyukjin, Can you clarify what is broken

Re: Revert of [SPARK-51229][BUILD][CONNECT] Fix dependency:analyze goal on connect common

not broken ... because we run SBT in PR builders for ASF resource restrictions and faster build. We use Maven for release so it was found out now. CI did not test your change. The part you are fixing is a special path .. On Thu, Mar 27, 2025 at 9:53 AM Rozov, Vlad wrote: If it is not broken, can

Re: Revert of [SPARK-51229][BUILD][CONNECT] Fix dependency:analyze goal on connect common

e is the simple fix? As a new contributor, you should not be coming in guns blazing blaming committers who are trying to keep the master branch sane and clean. On Tue, Mar 25, 2025 at 10:53 PM Rozov, Vlad wrote: There is a simple fix. This is exactly what I outlined in the e-mail. Prior to rever

Re: Revert of [SPARK-51229][BUILD][CONNECT] Fix dependency:analyze goal on connect common

te. Let me know. On Thu, 27 Mar 2025 at 00:13, Rozov, Vlad wrote: Every graduated from incubating Apache project has guards against what you name “chaotic” and what other name breaking best development practices. Such guards include JIRA, unit tests and PR review. Instead of reverting commit, I wo

Re: [DISCUSS] Upgrade Hive compile time dependency to 4.0

-o1TuS4mYXyi2KO6Xmx6ikHPySa9MLaLZ8t2hrA6AUcxSxDgHIwmKE] view my Linkedin profile<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> On Wed, 26 Mar 2025 at 06:02, Rozov, Vlad wrote: I started working on it. See https://github.com/apache/spark/pull/50213. Review and comments on the PR will help a lot. +1 for 4

Re: Revert of [SPARK-51229][BUILD][CONNECT] Fix dependency:analyze goal on connect common

:06 PM, Reynold Xin wrote: Sorry Vlad - I disagree. Where is the simple fix? As a new contributor, you should not be coming in guns blazing blaming committers who are trying to keep the master branch sane and clean. On Tue, Mar 25, 2025 at 10:53 PM Rozov, Vlad wrote: There is a simple fix

Re: [DISCUSS] Upgrade Hive compile time dependency to 4.0

ncial Crime | Forensic Analysis | GDPR [https://ci3.googleusercontent.com/mail-sig/AIorK4zholKucR2Q9yMrKbHNn-o1TuS4mYXyi2KO6Xmx6ikHPySa9MLaLZ8t2hrA6AUcxSxDgHIwmKE] view my Linkedin profile<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> On Tue, 11 Mar 2025 at 19:08, Rozov, Vlad wrote: Hi

Re: Revert of [SPARK-51229][BUILD][CONNECT] Fix dependency:analyze goal on connect common

ry points, Spark shalls, don't work and developers cannot debug and test. The snapshots become uesless. The tests passed because you did not fix SBT. It needs a larger change. Such change cannot be in the source. I can start a vote if you think this is an issue. On Wed, Mar 26, 2025 at

Re: Revert of [SPARK-51229][BUILD][CONNECT] Fix dependency:analyze goal on connect common

Spark shells. Why don't you submit a PR that contains the proper fix? It is easier to have one PR that has no issue, e.g., reverting backporting etc. On Wed, 26 Mar 2025 at 00:17, Rozov, Vlad wrote: Hi All, I kind of understand why https://github.com/apache/spark/pull/49971 was rever

Re: [DISCUSS] SPARK-51318: Remove `jar` files from Apache Spark repository and disable affected tests

at 8:48 AM Rozov, Vlad wrote: Please see inline. Thank you, Vlad On Mar 25, 2025, at 1:42 PM, Hyukjin Kwon mailto:gurwls...@apache.org>> wrote: > - the approach encourages keeping jars files in the Apache Spark repo Yes, and removes it from source releases. I believe this is a minimized

Re: [DISCUSS] SPARK-51318: Remove `jar` files from Apache Spark repository and disable affected tests

red too. I also have an outstanding question on the revert here https://lists.apache.org/thread/o8047n1cp8nc0q8c2ndht82h28p8j9jq. On Wed, 26 Mar 2025 at 04:14, Rozov, Vlad wrote: The policy [1] is quite clear and the fact that other projects do not include compiled jars (including test jars) int

Re: [DISCUSS] SPARK-51318: Remove `jar` files from Apache Spark repository and disable affected tests

lean this up. But I think your position isn't actually solving any problem that this principle is intended to prevent. On Tue, Mar 25, 2025 at 1:25 PM Rozov, Vlad wrote: I already casted my vote. To clarify, having compiled unlicensed jars in the source release is strictly against ASF policy [1

Re: [DISCUSS] SPARK-51318: Remove `jar` files from Apache Spark repository and disable affected tests

ue, I still don't understand why we would block the release for this. On Tue, Mar 25, 2025 at 7:49 AM Rozov, Vlad wrote: The difference is in the way how tests are disabled. - the approach encourages keeping jars files in the Apache Spark repo - it is hard to identify what tests are impacted

Revert of [SPARK-51229][BUILD][CONNECT] Fix dependency:analyze goal on connect common

Hi All, I kind of understand why https://github.com/apache/spark/pull/49971 was reverted on the branch-4.0 to allow testing of 4.0 release. Why was it also reverted on the master branch? I don’t see any JIRA being open for the failure. AFAIK, the proper way to handle the issue in Apache project

Re: [DISCUSS] SPARK-51318: Remove `jar` files from Apache Spark repository and disable affected tests

x27;s the difference between disabling tests for dev and release vs only for release? On Tue, 25 Mar 2025 at 15:36, Rozov, Vlad wrote: Overall I don’t buy the solution where tests are skipped based on the presence of a jar file. It looks too fragile to me. What if there is a bug that does not ad

Re: [DISCUSS] SPARK-51318: Remove `jar` files from Apache Spark repository and disable affected tests

e code to build the golden file. Did we check and confirm these jars are not the case and we lost the source code to build? On Tue, Mar 25, 2025 at 9:35 AM Rozov, Vlad wrote: First of all I don’t think that conclusion on the https://lists.apache.org/thread/xmbgpgt30n7fdd99pnbg7983qzzrx24k is corre

Re: [DISCUSS] SPARK-51318: Remove `jar` files from Apache Spark repository and disable affected tests

s are not the case and we lost the source code to build? On Tue, Mar 25, 2025 at 9:35 AM Rozov, Vlad wrote: First of all I don’t think that conclusion on the https://lists.apache.org/thread/xmbgpgt30n7fdd99pnbg7983qzzrx24k is correct. Jar files included into the source release are comp

Re: [DISCUSS] SPARK-51318: Remove `jar` files from Apache Spark repository and disable affected tests

rst disable the tests, and 2. open an umbrella JIRA to enable individual tests. Since you're driving this, would you mind either making a proper fix in one go, or create an umbrella JIRA to drive this? On Mon, 24 Mar 2025 at 23:46, Rozov, Vlad wrote: Let’s open a formal vote on the subject.

Re: [DISCUSS] SPARK-51318: Remove `jar` files from Apache Spark repository and disable affected tests

are quite old and stable, failures are unlikely. Thanks, Wenchen On Thu, Mar 13, 2025 at 12:15 AM Rozov, Vlad wrote: There is a difference between technical debt and legal issue. ASF may request to pull out release that does not meet ASF policy (and having tests is not ASF policy). IMO, SPARK

Re: [VOTE] Release Spark 4.0.0 (RC3)

-1 (non binding) There are jar files included into the source release and it is prohibited by ASF policy [1] Thank you, Vlad [1] https://www.apache.org/legal/release-policy.html On Mar 20, 2025, at 6:10 AM, Wenchen Fan wrote: Please vote on releasing the following candidate as Apache Spark

Re: [DISCUSS] SPARK-51318: Remove `jar` files from Apache Spark repository and disable affected tests

ld fix, let's make sure we don't just disable the tests - we will create another set of technical debt. On Thu, 27 Feb 2025 at 09:11, Rozov, Vlad wrote: I’ll look into the JIRA. Please assign it to me. Thank you, Vlad > On Feb 26, 2025, at 11:33 PM, Yang Jie > mailto:yangji

[DISCUSS] Upgrade Hive compile time dependency to 4.0

Hi All, As Apache Hive announced EOL for Hive 2.x [1] and 3.x [2], should Spark be compiled against Hive 4.x and use it as default? Thank you, Vlad [1] https://lists.apache.org/thread/4ctrzfw60jkhc0hq2xoh1jpqxgt2zd93 [2] https://lists.apache.org/thread/99h6wr7nk4684r6tkcbm8ydfytgqy6f3 [3] http

Re: PR review

May I please get review on the following outstanding PRs: https://github.com/apache/spark/pull/49276 (open on 12/23/2024) https://github.com/apache/spark/pull/49870 Thank you, Vlad On Feb 25, 2025, at 5:31 PM, Rozov, Vlad wrote: Thanks, looking for committers to review/discuss my pending PRs

Re: [DISCUSS] New Spark Connect Client repository for Swift language

+1 (non-binding) Thank you, Vlad On Mar 9, 2025, at 3:30 PM, Dongjoon Hyun wrote: Hi, All. I'd like to propose to add a new Apache Spark repository for `Spark Connect Client for Swift` in Apache Spark 4.1.0 timeframe. https://github.com/apache/spark-connect-swift To do this, I created an u

Re: [DISCUSS] SPARK-51318: Remove `jar` files from Apache Spark repository and disable affected tests

, I believe we can > definitely find a more reasonable testing approach. > > Thanks, > Jie Yang > > On 2025/02/26 16:57:45 "Rozov, Vlad" wrote: >> +1 on fixing test jars, though the way how it is fixed needs to be >> discussed, IMO. In the short term rem

Re: [DISCUSS] SPARK-51318: Remove `jar` files from Apache Spark repository and disable affected tests

+1 on fixing test jars, though the way how it is fixed needs to be discussed, IMO. In the short term removing jars may still be the best option to satisfy ASF legal policy and avoid release removal. AFAIK, ASF mandates that users and developers have source code that they build from (source rele

Re: [VOTE] Release Spark 3.5.5 (RC1)

-0 (non-binding). IMO, it will be good to address https://issues.apache.org/jira/browse/SPARK-51318 to avoid legal issues and meet ASF source release policy. Thank you, Vlad On Feb 25, 2025, at 2:51 AM, Kent Yao wrote: +1 Kent On 2025/02/25 10:26:38 Max Gekk wrote: +1, since SPARK-51281 i

Re: [VOTE] Release Spark 3.5.5 (RC1)

/junitLargeJar.jar * ./data/artifact-tests/smallJar.jar On Tue, Feb 25, 2025 at 9:22 PM Rozov, Vlad wrote: Right, the issue does not seem to be new for 3.5 and it is not new for 3.5.5. Here is the list of all jars I found in the source release: ./core/src/test/resources/TestHelloV3_2.12.jar

Re: [VOTE] SPIP: Add the TIME data type

+1 (non-binding) Thank you, Vlad On Feb 25, 2025, at 1:33 AM, Sakthi wrote: +1 (non-binding) On Tue, Feb 25, 2025 at 12:50 AM Kent Yao mailto:y...@apache.org>> wrote: +1(binding), Thank you, Max! Kent Armaan Sait mailto:armaansait...@gmail.com>> 于2025年2月25日周二 06:11写道： +1 Thanks & Regard

Re: PR review

//en.wikipedia.org/wiki/Wernher_von_Braun> Von Braun<https://en.wikipedia.org/wiki/Wernher_von_Braun>)". On Mon, 30 Dec 2024 at 21:47, Rozov, Vlad wrote: I have an open PR https://github.com/apache/spark/pull/49276, though my question is more generic. After PR is open, should a contributor

Re: [VOTE] Release Spark 3.5.5 (RC1)

nsed test jar, `TestHelloV2.jar`? >> >> https://issues.apache.org/jira/browse/SPARK-44246 >> https://github.com/apache/spark/pull/41789 >> >> And, it was spread to `TestHelloV3_2.13.jar` via SPARK-44297 in the same way? >> >> https://issues.apache.org/j

Re: [VOTE] Release Spark 3.5.5 (RC1)

I am not sure if this was already discussed and noted, so want to confirm with PMC members: I see several (test) JAR files included into the ASF source release that do not have LICENSE in the MANIFEST or META-INF and do not have the source code. For example core/src/test/resources/TestHelloV3_2

Re: [VOTE] Publish additional Spark distribution with Spark Connect enabled

+1 (non-binding) Thank you, Vlad On Feb 4, 2025, at 11:05 PM, Wenchen Fan wrote: Hi all, Given the positive feedback in the previous DISCUSS email, I'd like to start the vote for the proposal "Publish additional Spark distrib

Re: relative path in DataFrameWriter and DataStreamWriter

each node write to different physical paths and claim it is working. It doesn't and it is NOT a spec. It's not a bug, sorry. On Fri, Jan 17, 2025 at 4:41 AM Rozov, Vlad wrote: > More problematic thing is to use the local filesystem for the path which is > interpreted by dist

Re: relative path in DataFrameWriter and DataStreamWriter

It really depends on the setup of the cluster). But we are not expecting metadata directory and the actual files to be placed in physically different locations; this actually requires people to mostly use absolute paths (including scheme or not). On Thu, Jan 16, 2025 at 3:16 AM Rozov, Vlad wrote:

Re: relative path in DataFrameWriter and DataStreamWriter

Resending... > On Jan 9, 2025, at 1:57 PM, Rozov, Vlad wrote: > > Hi, > > I see a difference in how “path" is handled in DataFrameWriter.save(path) and > DataStreamWriter.start(path) while using relative path (for example > “test.parquet") to write to parquet f

RE: [ANNOUNCE] Apache Spark 3.5.4 released

It looks like new KEYS were not uploaded to https://downloads.apache.org/spark/KEYS. I open https://issues.apache.org/jira/browse/SPARK-50816. Thank you, Vlad On 2024/12/20 17:06:31 杨杰 wrote: > We are happy to announce the availability of Spark 3.5.4! > > Spark 3.5.4 is a maintenance release c

relative path in DataFrameWriter and DataStreamWriter

Hi, I see a difference in how “path" is handled in DataFrameWriter.save(path) and DataStreamWriter.start(path) while using relative path (for example “test.parquet") to write to parquet files (possibly applies to other file formats as well). In case of DataFrameWriter path is relative to the cu

Re: [VOTE] Use plain text logs by default

+1 (non-binding) Vlad On Jan 8, 2025, at 8:28 PM, Wenchen Fan wrote: Hi all, Following the discussion[1], I'd like to start the vote for 'Use plain text logs by default'. Note: This is not to overthrow the previous vote that adds the structured logging framework. The framework is still ther

Re: PR review

the other side I checked few open PRs and I don’t see such requests. Thank you, Vlad On Dec 30, 2024, at 1:09 PM, Herman van Hovell wrote: What do you need to have reviewed? On Mon, Dec 30, 2024 at 3:48 PM Rozov, Vlad wrote: Hi, How can I request PR review? Sorry if this was already

PR review

Hi, How can I request PR review? Sorry if this was already discussed on the list or is available in the archive or spark.apache.org. Thank you, Vlad

Re: [VOTE] Release Spark 3.5.4 (RC3)

+1 (not binding) Thanks Vlad Rozov - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org