Thank you Larry. We are from Shopee, I lead the Spark team here. I have several team members participant in Livy development, but most finished feature are done by me and my colleague Jianzhen, information as follows: Xilang Yan, xilang....@gmail.com, Shopee Jianzhen Wu, myjianz...@gmail.com, Shopee
Thanks! larry mccay <lmc...@apache.org> 于2022年10月10日周一 23:01写道： > Exciting to see interest growing! > I think rather than try and gather all of these things now, we follow up > the proposal with a roadmap discussion. > > Just as we often entertain new committers during the proposal consideration > we can do that then as well. > If you'd like to provide your names and affiliations here, I can add them > to the upcoming proposal. > If you'd like to wait to agree on the proposal first that would work too. > > Again, very good to see traction building here! > > --larry > > On Mon, Oct 10, 2022 at 8:56 AM 严细浪 <xilang....@gmail.com> wrote: > > > So glad to see there is new progress on Livy community. We have a huge > > Livy cluster on our prod environment with nearly one million Spark > > applications submitted everyday, currently we have to maintain Livy > > ourselves. > > Some common features we finished: > > Livy rest cluster, I engaged in the Livy rest cluster design but after > > that there is no progress so we implement it ourselves, although has a > > little difference with design > > Support multi Spark versions > > Implemented a metrics system for Livy > > Support customize batch/interactive session lifecycle event handler, > > default log event with log4j, very helpful for trouble shooting > > Optimize log to track which session id the log message came from, also > > very helpful for trouble shooting > > Support customize Spark config optimization rules, can be used to > optimize > > config for users’ job > > A sets of command line tool which can be used to replace Spark’s > > spark-submit, pyspark, spark-sql but actually submit application in Livy > > > > We are planning to implement a JDBC state store, and allow multi Livy > > Thrift session to share one backend Spark application in next few months. > > > > We can deliver our changes if it is in Livy’s plan. > > > > Thanks, > > Jeffrey > > > > > > On 2022/10/10 09:27:46 Jean-Baptiste Onofré wrote: > > > Hi, > > > > > > If you plan to contribute on Livy, you are welcome to be part of the > > > proposal imho. > > > > > > Regards > > > JB > > > > > > On Mon, Oct 10, 2022 at 7:19 AM Brahma Reddy Battula <br...@apache.org > > > > wrote: > > > > > > > > Hi Larry, > > > > > > > > Nice to hear from you that you are planning propose new features and > > make > > > > active. > > > > > > > > We do use Livy in our prod clusters, any chance to include us in your > > > > proposal so that we can also contribute. > > > > > > > > > > > > > > > > On Mon, 10 Oct 2022 at 3:54 AM, larry mccay <lm...@apache.org> > wrote: > > > > > > > > > Sure. > > > > > Thanks, Craig. > > > > > > > > > > --larry > > > > > > > > > > On Sun, Oct 9, 2022, 5:38 PM Craig Russell <ap...@gmail.com> > wrote: > > > > > > > > > > > Hi Larry, > > > > > > > > > > > > > On Oct 8, 2022, at 10:30, larry mccay <lm...@apache.org> > wrote: > > > > > > > > > > > > > > Good to hear, Jean-Baptiste! > > > > > > > We will send the proposal both here and to the Livy private > > list, I > > > > > > guess. > > > > > > > Once we come to an agreement and direction we can share with > the > > dev@ > > > > > > and > > > > > > > user@ lists. > > > > > > > > > > > > I support the effort to revive the Livy podling, but I'd advise > > against > > > > > > using the private list to discuss it. The incubator general list > is > > > > > public > > > > > > and the livy dev list is public, so these are the lists I'd > > suggest you > > > > > use > > > > > > for further discussion. > > > > > > > > > > > > Again, good luck with the effort, > > > > > > Craig > > > > > > > > > > > > > > Thanks again! > > > > > > > > > > > > > > --larry > > > > > > > > > > > > > > On Sat, Oct 8, 2022 at 12:49 AM Jean-Baptiste Onofré < > > jb...@nanthrax.net> > > > > > > > wrote: > > > > > > > > > > > > > >> Hi Larry, > > > > > > >> > > > > > > >> Agree with Claude: if we can have concrete proposal (in terms > of > > > > > > >> community and roadmap), it's OK. We are not in a rush, if we > > can have > > > > > > >> this in the commit weeks, it's fine. > > > > > > >> I would sent this on the Livy mailing list. > > > > > > >> > > > > > > >> Again, I would be more than happy to help and contribute on > > Livy. > > > > > > >> > > > > > > >> Regards > > > > > > >> JB > > > > > > >> > > > > > > >> On Fri, Oct 7, 2022 at 6:08 PM larry mccay <lm...@apache.org> > > wrote: > > > > > > >>> > > > > > > >>> I was hoping that we could bring a set of contributors as: > > > > > > >>> > > > > > > >>> * mentors - 1 or 2 plus whoever wants to remain > > > > > > >>> * PPMC/Committers - 2 or 3 plus whoever wants to remain > > > > > > >>> * election of a PPMC chair if needed > > > > > > >>> > > > > > > >>> Also open it up for folks to ask to join (as well as go > > emeritus) as > > > > > I > > > > > > >> know > > > > > > >>> there are others that have tried to contribute and have > gotten > > know > > > > > > where > > > > > > >>> or have long since decided it wasn't worth trying anymore. > > > > > > >>> > > > > > > >>> So, my thought was almost a new Proposal and solicitation for > > > > > community > > > > > > >>> bootstrap with a few seeds planted to help it grow. > > > > > > >>> I was also considering a goal of getting it back off the > > ground with > > > > > > some > > > > > > >>> number of releases and proof of health as a project to fast > > track to > > > > > a > > > > > > >>> graduation discussion. > > > > > > >>> A bit early to talk about that now but that was a detail > that I > > > > > thought > > > > > > >>> would possibly go into such a reboot proposal. > > > > > > >>> > > > > > > >>> @Jean-Baptiste - we have compiled a list of > > features/improvements > > > > > that > > > > > > we > > > > > > >>> would be able to bring to the table with the reboot as well. > > > > > > >>> These are improvements that have been made internally due the > > sense > > > > > of > > > > > > >> not > > > > > > >>> being able to get them in. I lack the details around the > > attempts at > > > > > > this > > > > > > >>> time but we would be bringing source as well. > > > > > > >>> We could list these as part of the proposal/concrete plans. > > > > > > >>> > > > > > > >>> If you think this proposal can be done more casually, that's > > fine > > > > > too. > > > > > > I > > > > > > >>> just would like to make it clear that we would open it up to > > new > > > > > folks > > > > > > as > > > > > > >>> well. > > > > > > >>> > > > > > > >>> However you'd like to see it done, we can have concrete > > commitments > > > > > to > > > > > > >> you > > > > > > >>> within a week or so more than likely. > > > > > > >>> > > > > > > >>> BTW, I did note that Ambari was sent to the attic and then > > pulled > > > > > back > > > > > > >> out > > > > > > >>> [1] back in June into July of this year. > > > > > > >>> I would hope that we could avoid the administrative headache > > that > > > > > seems > > > > > > >> to > > > > > > >>> have been involved in that reinstatement. > > > > > > >>> > > > > > > >>> 1. https://issues.apache.org/jira/browse/ATTIC-205 > > > > > > >>> > > > > > > >>> On Fri, Oct 7, 2022 at 11:04 AM Claude Warren, Jr > > > > > > >>> <cl...@aiven.io.invalid> wrote: > > > > > > >>> > > > > > > >>>> Larry et.al. > > > > > > >>>> > > > > > > >>>> I think we need concrete steps. So if Larry produced a list > > of new > > > > > > >>>> committers could we get them added to the list and see if > > > > > development > > > > > > >>>> starts again? > > > > > > >>>> > > > > > > >>>> Mentors / Champion does that work for you? > > > > > > >>>> > > > > > > >>>> Larry, can you produce such a list? > > > > > > >>>> > > > > > > >>>> Is there an objection to proceeding along these lines? > > > > > > >>>> > > > > > > >>>> Claude > > > > > > >>>> > > > > > > >>>> On Fri 7 Oct 2022, 03:07 Jean-Baptiste Onofré, < > > jb...@nanthrax.net> > > > > > > >> wrote: > > > > > > >>>> > > > > > > >>>>> Hi Larry, > > > > > > >>>>> > > > > > > >>>>> I tried maybe 3 times to get Livy back to active, without > > success. > > > > > > >>>>> I was a volunteer to contribute, but not alone, we need a > > community > > > > > > >>>> around > > > > > > >>>>> Livy. > > > > > > >>>>> > > > > > > >>>>> I don't remember having seen new active contributions on > the > > > > > mailing > > > > > > >>>>> list, and very limited on source code. > > > > > > >>>>> > > > > > > >>>>> If you really think we can have a team, that would be > great, > > and I > > > > > > >>>>> still agree to support. But it has to be concrete. > > > > > > >>>>> > > > > > > >>>>> Let's see what the other IPMC members are thinking, but > > IMHO, I > > > > > doubt > > > > > > >>>>> we will have Livy back to life in the short term. > > > > > > >>>>> > > > > > > >>>>> Regards > > > > > > >>>>> JB > > > > > > >>>>> > > > > > > >>>>> On Thu, Oct 6, 2022 at 5:29 PM larry mccay < > lm...@apache.org > > > > > > > > > >> wrote: > > > > > > >>>>>> > > > > > > >>>>>> Folks, > > > > > > >>>>>> > > > > > > >>>>>> As I mentioned in a previous thread [1], I would like to > > help > > > > > avoid > > > > > > >>>>> sending > > > > > > >>>>>> a well regarded, actively consumed and mature product to > the > > > > > attic. > > > > > > >>>> There > > > > > > >>>>>> have been discussions internally at my employer and we > have > > also > > > > > > >>>>> struggled > > > > > > >>>>>> getting our contributions committed. The original group of > > > > > > >> committers > > > > > > >>>>> seem > > > > > > >>>>>> to have all moved on, and this is the core challenge for > > those > > > > > > >> that are > > > > > > >>>>>> trying to contribute, but are not committers. This > doesn’t > > > > > > >> represent a > > > > > > >>>>>> lack of interest in doing so and I think there are others > > in this > > > > > > >>>>> position. > > > > > > >>>>>> > > > > > > >>>>>> We are interested in trying to reboot with additional PPMC > > > > > > >> leadership > > > > > > >>>> and > > > > > > >>>>>> investing in contributors that we would like to install as > > new > > > > > > >>>>> committers. > > > > > > >>>>>> > > > > > > >>>>>> I’d like to hear what the incubator would think about a > > Reproposal > > > > > > >> for > > > > > > >>>>> the > > > > > > >>>>>> Livy project that would be open to existing PMC members > and > > > > > > >> committers > > > > > > >>>> as > > > > > > >>>>>> well as new contributors with a vested interest in this > > project > > > > > > >> and the > > > > > > >>>>>> health of its community. > > > > > > >>>>>> > > > > > > >>>>>> How would we best propose such a thing here? > > > > > > >>>>>> > > > > > > >>>>>> Thanks, > > > > > > >>>>>> > > > > > > >>>>>> –larry > > > > > > >>>>>> > > > > > > >>>>>> > > > > > > >>>>>> 1. > > > > > > >>>>>> > > > > > > >>>>>> > > > > > > >> > > https://lists.apache.org/list?general@incubator.apache.org:2022-7 > > > > > > >>>>>> > > > > > > >>>>>> > > > > > > >>>>>> On Thu, Oct 6, 2022 at 5:21 AM Luciano Resende < > > > > > > >> luckbr1...@gmail.com> > > > > > > >>>>> wrote: > > > > > > >>>>>> > > > > > > >>>>>>> Based on the failed attempts to resurrect a community, +1 > > for > > > > > > >>>>> retirement. > > > > > > >>>>>>> > > > > > > >>>>>>> On Wed, Oct 5, 2022 at 07:26 Jean-Baptiste Onofré < > > > > > > >> j...@nanthrax.net> > > > > > > >>>>> wrote: > > > > > > >>>>>>> > > > > > > >>>>>>>> Hi guys, > > > > > > >>>>>>>> > > > > > > >>>>>>>> After several attempts, we are struggling to execute > Livy > > > > > > >> podling: > > > > > > >>>>> the > > > > > > >>>>>>>> activity is pretty low, and even after a couple of calls > > for > > > > > > >>>> action, > > > > > > >>>>>>>> we don't have a clear sign of community around Livy. > > > > > > >>>>>>>> > > > > > > >>>>>>>> I would like to discuss/propose to retire the Livy > > podling. > > > > > > >>>>>>>> > > > > > > >>>>>>>> Thoughts ? > > > > > > >>>>>>>> > > > > > > >>>>>>>> Regards > > > > > > >>>>>>>> JB > > > > > > >>>>>>>> > > > > > > >>>>>>>> > > > > > > >>>> > > > > > > --------------------------------------------------------------------- > > > > > > >>>>>>>> To unsubscribe, e-mail: > > > > > > >> general-unsubscr...@incubator.apache.org > > > > > > >>>>>>>> For additional commands, e-mail: > > > > > > >> general-h...@incubator.apache.org > > > > > > >>>>>>>> > > > > > > >>>>>>>> -- > > > > > > >>>>>>> Sent from my Mobile device > > > > > > >>>>>>> > > > > > > >>>>> > > > > > > >>>>> > > > > > > --------------------------------------------------------------------- > > > > > > >>>>> To unsubscribe, e-mail: > > general-unsubscr...@incubator.apache.org > > > > > > >>>>> For additional commands, e-mail: > > general-h...@incubator.apache.org > > > > > > >>>>> > > > > > > >>>>> > > > > > > >>>> > > > > > > >> > > > > > > >> > > --------------------------------------------------------------------- > > > > > > >> To unsubscribe, e-mail: > > general-unsubscr...@incubator.apache.org > > > > > > >> For additional commands, e-mail: > > general-h...@incubator.apache.org > > > > > > >> > > > > > > >> > > > > > > > > > > > > Craig L Russell > > > > > > c...@apache.org > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > > > > > For additional commands, e-mail: > general-h...@incubator.apache.org > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > > > >
