vicennial commented on PR #49604: URL: https://github.com/apache/spark/pull/49604#issuecomment-2687751626
Thanks for the comment @LuciferYang > I have some doubts about merging this part of the code into the main codebase, as it seems more like an independent project to me. It is independent, but this is due to the independence property of the Spark Connect Client (which is, by design, a thin client with lightweight dependencies). Since this serves to demonstrate extending Spark Connect and displaying its usage of the extension via a Spark Connect Client, I chose to mimic how a practical example would look, and thus, there is no direct connection to the parent Spark tree. > Additionally, I haven't found any process for building and testing it in any of Spark CI (if there is one, please point it out to me). What is the recognized way to ensure its integrity and health after we make changes to it? You are right, there is no automated process in the Spark CI at the moment. In an earlier [comment](https://issues.apache.org/jira/browse/SPARK-51338), I requested to separate out the addition of the automated build step in order to separate out and tackle the somewhat intricate complexity in modifying the CI in a fresh PR. I've opened up this ticket now to track this follow-up https://issues.apache.org/jira/browse/SPARK-51338. I am actively looking into the process to accomplish this step, albeit a bit slow as I haven't modified the CI in the past. > Additionally, in what form will this be released to users? This largely serves as a reference skeleton project structure (accompanied by the documentation in https://dist.apache.org/repos/dist/dev/spark/v4.0.0-rc1-docs/_site/app-dev-spark-connect.html). No intention atm to have users depend on this project but rather use it as a base for their own separate/independent development -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
