+1， Agree to remove the jar files from the Apache Spark repository and disable
the affected tests.
For the current test scenarios that use jar files, I believe we can definitely
find a more reasonable testing approach.
Thanks,
Jie Yang
On 2025/02/26 16:57:45 "Rozov, Vlad" wrote:
> +1 on fixing test jars, though the way how it is fixed needs to be discussed,
> IMO. In the short term removing jars may still be the best option to satisfy
> ASF legal policy and avoid release removal.
>
> AFAIK, ASF mandates that users and developers have source code that they
> build from (source release), not that they run (binary release).
>
> Thank you,
>
> Vlad
>
> > On Feb 26, 2025, at 8:47 AM, Dongjoon Hyun <dongj...@apache.org> wrote:
> >
> > Thank you for your reply, Sean.
> >
> > I expected that argument exactly so that I started by quoting your sentence
> > in the above.
> >
> > I understood the reasoning in 2018. However, there are two reasons why I
> > brought this again in 2025:
> >
> > First, the open source sprit is technically and literally "no compiled code
> > in a source release" like Apache Hadoop and Hive community does. Justin,
> > Vlad, and Alex shared the same perspective to the Apache Spark PMC.
> >
> > $ tar tvf apache-hive-4.0.1-src.tar.gz | grep 'jar$' | wc -l
> > 0
> > $ tar tvfz hadoop-3.4.1-src.tar.gz | grep 'jar$' | wc -l
> > 0
> >
> > Second, last year, the open source communities were hit by CVE-2024-3094
> > ("XZ Utils Backdoor") in the world-wide manner where the backdoor was
> > hidden in the test object. I believe most of us are aware of that. At that
> > time, the GitHub repository was disabled. As a member of Apache Spark PMC,
> > I'm suggesting to remove that risk from the Apache Spark repository in
> > 2025. I attached the following link to provide the XZ Utils history
> > explicitly.
> >
> >
> > https://www.akamai.com/blog/security-research/critical-linux-backdoor-xz-utils-discovered-what-to-know
> >
> > Although I agree that those test coverages are important, I don't think
> > that's worthy for Apache Spark community to take a risk to be shutdown.
> > That's the lesson which I've learned last year.
> >
> > Sincerely,
> > Dongjoon.
> >
> > On 2025/02/26 13:31:56 Sean Owen wrote:
> >> The gist of the initial 2018 thread was:
> >> These are not source .jar files that users use, but .jar files used to test
> >> loading of from .jar files. These are test resources only.
> >> I don't think this is what the spirit of the rule is speaking to, that the
> >> end-user code should always have source code, which is the right principle.
> >> Checking in the code somewhere is nice to have though and I think that was
> >> the idea here.
> >>
> >> But, removing these and disabling potentially valuable tests seems like a
> >> step too far. There is no actual 'problem' w.r.t. the principle that users
> >> have source to the code they run.
> >>
> >> The 2025 thread just retreads the same ground as the 2018 thread.
> >> But I don't see that we put this argument to the person who raised it
> >> again. Why not that first?
> >> And, if possible, go stick the source to these jars in the source tree,
> >> where available.
> >>
> >>
> >> On Wed, Feb 26, 2025 at 1:08 AM Dongjoon Hyun <dongjoon.h...@gmail.com>
> >> wrote:
> >>
> >>> Hi, All.
> >>>
> >>> Unfortunately, the Apache Spark project seems to have a technical debt in
> >>> the source code releases. It happens to be discussed at least twice on
> >>> both
> >>> dev@spark and legal-discuss mailing lists. (Thank you for the head-up,
> >>> Vlad.)
> >>>
> >>> 1. https://lists.apache.org/thread/3sxw9gwp51mrkzlo2xchq1g20gbgbnz8
> >>> (2018-06-21, dev@spark)
> >>> 2. https://lists.apache.org/thread/xmbgpgt30n7fdd99pnbg7983qzzrx24k
> >>> (2018-06-25, legal-discuss@)
> >>> 3. https://lists.apache.org/thread/z3oq1db80vc8c7r6892hwjnq4h7hnwmd
> >>> (2025-02-25, dev@spark)
> >>>
> >>> To be short, according to the previous conclusion in 2018, the Apache
> >>> Spark community wanted to adhere to the ASF policy by removing those jar
> >>> files from source code releases (although it was not considered as a
> >>> release blocker at that time and until now).
> >>>
> >>>> it's important to be able to recreate these JARs somehow,
> >>>> and I don't think we have the source in the repo for all of them
> >>>> (at least, the ones that originate from Spark).
> >>>> That much seems like a must-do. After that, seems worth figuring out
> >>>> just how hard it is to build these artifacts from source.
> >>>> If it's easy, great. If not, either the test can be removed or
> >>>> we figure out just how hard a requirement this is.
> >>>
> >>> Given the unresolved issue for seven years, I proposed SPARK-51318 as a
> >>> potential solution to comply with ASF policy. After SPARK-51318, we can
> >>> recover the test coverage one by one later by addressing IDed TODO items
> >>> without any legal concerns during the votes.
> >>>
> >>> https://issues.apache.org/jira/browse/SPARK-51318
> >>> (Remove `jar` files from Apache Spark repository and disable affected
> >>> tests)
> >>>
> >>> WDYT?
> >>>
> >>> BTW, please note that I didn't define SPARK-51318 as a blocker for any
> >>> on-going releases yet.
> >>>
> >>> Best regards,
> >>> Dongjoon.
> >>>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
