+1 on better notebook and other REPL experience On Thu, Mar 23, 2023 at 9:17 AM Dongjoon Hyun <dongjoon.h...@gmail.com> wrote:
> I also support Herman's `SPARK-42884 Add Ammonite REPL integration` PR. > > Thanks, > Dongjoon. > > > On Thu, Mar 23, 2023 at 7:51 AM Mridul Muralidharan <mri...@gmail.com> > wrote: > >> >> Sounds good, thanks for clarifying ! >> >> Regards, >> Mridul >> >> On Thu, Mar 23, 2023 at 9:09 AM Herman van Hovell <her...@databricks.com> >> wrote: >> >>> The goal of adding this, is to make it easy for a user to connect a >>> scala REPL to a Spark Connect server. Just like Spark shell makes it easy >>> to work with a regular Spark environment. >>> >>> It is not meant as a Spark shell replacement. They represent two >>> different modes of working with Spark, and they have very different API >>> surfaces (Connect being a subset of what regular Spark has to offer). I do >>> think we should consider using ammonite for Spark shell at some point, >>> since this has better UX and does not require us to fork a REPL. That >>> discussion is for another day though. >>> >>> I guess you can use it as an example of building an integration. In >>> itself I wouldn't call it that because I think this a key part of getting >>> started with connect, and/or doing debugging. >>> >>> On Thu, Mar 23, 2023 at 4:00 AM Mridul Muralidharan <mri...@gmail.com> >>> wrote: >>> >>>> >>>> What is unclear to me is why we are introducing this integration, how >>>> users will leverage it. >>>> >>>> * Are we replacing spark-shell with it ? >>>> Given the existing gaps, this is not the case. >>>> >>>> * Is it an example to showcase how to build an integration ? >>>> That could be interesting, and we can add it to external/ >>>> >>>> Anything else I am missing ? >>>> >>>> Regards, >>>> Mridul >>>> >>>> >>>> >>>> On Wed, Mar 22, 2023 at 6:58 PM Herman van Hovell < >>>> her...@databricks.com> wrote: >>>> >>>>> Ammonite is maintained externally by Li Haoyi et al. We are including >>>>> it as a 'provided' dependency. The integration bits and pieces (1 file) >>>>> are >>>>> included in Apache Spark. >>>>> >>>>> On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan <mri...@gmail.com> >>>>> wrote: >>>>> >>>>>> >>>>>> Will this be maintained externally or included into Apache Spark ? >>>>>> >>>>>> Regards , >>>>>> Mridul >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell >>>>>> <her...@databricks.com.invalid> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> For Spark Connect Scala Client we are working on making the REPL >>>>>>> experience a bit nicer <https://github.com/apache/spark/pull/40515>. >>>>>>> In a nutshell we want to give users a turn key scala REPL, that works >>>>>>> even >>>>>>> if you don't have a Spark distribution on your machine (through >>>>>>> coursier <https://get-coursier.io/>). We are using Ammonite >>>>>>> <https://ammonite.io/> instead of the standard scala REPL for this, >>>>>>> the main reason for going with Ammonite is that it is easier to >>>>>>> customize, >>>>>>> and IMO has a superior user experience. >>>>>>> >>>>>>> Does anyone object to doing this? >>>>>>> >>>>>>> Kind regards, >>>>>>> Herman >>>>>>> >>>>>>> -- John Zhuge
