[DISCUSS] SPARK-51318: Remove `jar` files from Apache Spark repository and disable affected tests

Hi, All. Unfortunately, the Apache Spark project seems to have a technical debt in the source code releases. It happens to be discussed at least twice on both dev@spark and legal-discuss mailing lists. (Thank you for the head-up, Vlad.) 1. https://lists.apache.org/thread/3sxw9gwp51mrkzlo2xchq1g20

Re: [VOTE] Release Spark 3.5.5 (RC1)

Let me spin-off this as a new thread because this is related to all Apache Spark releases. Thanks, Dongjoon - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] Release Spark 3.5.5 (RC1)

Thank you for sharing more details and suggestions, Vlad and Herman. Additionally, I found one interesting thread where an Apache Spark PMC member asked the specific question to `legal-discuss@` literally for the above `TestSerDe.jar` file. The thread at https://lists.apache.org/list?legal-disc

Re:Re: [VOTE] SPIP: Add the TIME data type

+1 At 2025-02-26 13:48:44, "Yuming Wang" wrote: +1 On Wed, Feb 26, 2025 at 1:42 PM Shubham Patel wrote: +1 On Sun, 23 Feb 2025, 21:21 Max Gekk, wrote: Hi Spark devs, Following the discussion [1], I'd like to start the vote for the SPIP [2]. The SPIP aims to add a new data type T

Re: [VOTE] SPIP: Add the TIME data type

+1 On Wed, Feb 26, 2025 at 1:42 PM Shubham Patel wrote: > +1 > > On Sun, 23 Feb 2025, 21:21 Max Gekk, wrote: > >> Hi Spark devs, >> >> Following the discussion [1], I'd like to start the vote for the SPIP >> [2]. The SPIP aims to add a new data type TIME to Spark SQL types. New type >> should c

Re: [VOTE] SPIP: Add the TIME data type

+1 On Sun, 23 Feb 2025, 21:21 Max Gekk, wrote: > Hi Spark devs, > > Following the discussion [1], I'd like to start the vote for the SPIP [2]. > The SPIP aims to add a new data type TIME to Spark SQL types. New type > should conform to TIME(n) WITHOUT TIME ZONE as defined by the SQL > standard.

Re: [VOTE] Release Spark 3.5.5 (RC1)

AFAIK, there are 3 ways to satisfy ASF source release policy: 1. Use source code to build test jars and remove jars from the source release and they may still be part of the source in the github repo (preferable). 2. Publish jars as artifacts in Maven central and pull them as a dependency. 3. Rep

Re: [VOTE] Release Spark 3.5.5 (RC1)

https://github.com/apache/spark/blob/master/sql/connect/client/jvm/src/test/resources/StubClassDummyUdf.scala (commit 408c3df) is the source code for the following jars: - ./connector/connect/server/src/test/resources/udf_noA.jar - ./connector/connect/client/jvm/src/test/resources/udf2.13.ja

Re: [VOTE] SPIP: Add the TIME data type

+1 (non-binding) Thank you, Vlad On Feb 25, 2025, at 1:33 AM, Sakthi wrote: +1 (non-binding) On Tue, Feb 25, 2025 at 12:50 AM Kent Yao mailto:y...@apache.org>> wrote: +1(binding), Thank you, Max! Kent Armaan Sait mailto:armaansait...@gmail.com>> 于2025年2月25日周二 06:11写道： +1 Thanks & Regard

Re: PR review

Thanks, looking for committers to review/discuss my pending PRs: https://github.com/apache/spark/pulls/vrozov Thank you, Vlad On Dec 31, 2024, at 6:47 AM, Mich Talebzadeh wrote: This is my experience 1) Initial Discussion in this email list Subject Line: Use a clear subject line that indi

Re: [VOTE] Release Spark 3.5.5 (RC1)

Right, the issue does not seem to be new for 3.5 and it is not new for 3.5.5. Here is the list of all jars I found in the source release: ./core/src/test/resources/TestHelloV3_2.12.jar ./core/src/test/resources/TestHelloV2_2.12.jar ./core/src/test/resources/TestHelloV2_2.13.jar ./core/src/test/re

Re: [VOTE] Release Spark 3.5.5 (RC1)

Thank you for reply, Vlad. Do you mean SPARK-44246 violated ASF policy since Apache Spark 3.5.0 by injecting a unlicensed test jar, `TestHelloV2.jar`? https://issues.apache.org/jira/browse/SPARK-44246 https://github.com/apache/spark/pull/41789 And, it was spread to `TestHelloV3_2.13.jar` via SP

Re: [VOTE] Release Spark 3.5.5 (RC1)

Just for the record, `TestUDTF.jar` test resource seems to be one of the oldest examples which Vlad pointed. https://github.com/apache/spark/blob/v2.0.0/core/src/test/resources/TestUDTF.jar (Added 9 years ago) $ tar tvf spark-2.0.0.tgz | grep TestUDTF.jar -rw-r--r-- 0 jenkins jenkins 1328 Jul

Re: [VOTE] Release Spark 3.5.5 (RC1)

I am not sure if this was already discussed and noted, so want to confirm with PMC members: I see several (test) JAR files included into the ASF source release that do not have LICENSE in the MANIFEST or META-INF and do not have the source code. For example core/src/test/resources/TestHelloV3_2

Re: [VOTE] Release Spark 3.5.5 (RC1)

+1 Kent On 2025/02/25 10:26:38 Max Gekk wrote: > +1, since SPARK-51281 is not a release blocker. > > On Mon, Feb 24, 2025 at 7:12 AM Yang Jie wrote: > > > +1 > > > > On 2025/02/24 04:04:22 Dongjoon Hyun wrote: > > > Thank you for your voting. > > > > > > I have been aware of SPARK-51281 since

Re: [VOTE] Release Spark 3.5.5 (RC1)

+1, since SPARK-51281 is not a release blocker. On Mon, Feb 24, 2025 at 7:12 AM Yang Jie wrote: > +1 > > On 2025/02/24 04:04:22 Dongjoon Hyun wrote: > > Thank you for your voting. > > > > I have been aware of SPARK-51281 since Wenchen pinged me three days ago. > > > > I just thought it was not g

Re: [VOTE] Release Spark 4.0.0 (RC1)

Found another issue, Spark 4.0.0 RC1 fails to submit to Kerberized cluster. I have filled SPARK-51311 and opened [1] to fix it. [1] https://github.com/apache/spark/pull/50077 Thanks, Cheng Pan > On Feb 24, 2025, at 17:47, Cheng Pan wrote: > > I found an issue in [1], would be great if some

Re: [VOTE] SPIP: Add the TIME data type

+1 (non-binding) On Tue, Feb 25, 2025 at 12:50 AM Kent Yao wrote: > +1(binding), > > Thank you, Max! > > Kent > > Armaan Sait 于2025年2月25日周二 06:11写道： > >> +1 >> >> *Thanks & Regards,* >> Armaan Sait >> *LinkedIn:* Armaan Sait | LinkedIn >> >> >> >> On S

Re: [VOTE] SPIP: Add the TIME data type

+1(binding), Thank you, Max! Kent Armaan Sait 于2025年2月25日周二 06:11写道： > +1 > > *Thanks & Regards,* > Armaan Sait > *LinkedIn:* Armaan Sait | LinkedIn > > > > On Sun, Feb 23, 2025 at 11:40 PM Hyukjin Kwon > wrote: > >> +1 >> >> On Mon, 24 Feb 2025 at 1