Hi Liu, See this link https://community.apache.org/gsoc.html
Xun Liu <neliu...@163.com> 于2019年3月8日周五 下午1:58写道： > Hi, Jongyoul Lee, Морковкин > > I queried the information about GSOS. Is it still necessary to apply for > the zeppelin community first? > I don't know much about GSOS. In addition to helping the project, the > mentor > What other work needs to be done? > > > 在 2019年3月8日，上午10:01，Xun Liu <neliu...@163.com> 写道： > > > > Hi, Морковкин > > > > I am very happy to be your mentor for GSOC. :-) > > I believe that by completing this work, I can also learn a lot. > > > > Please watch to https://issues.apache.org/jira/browse/ZEPPELIN-4018 < > https://issues.apache.org/jira/browse/ZEPPELIN-4018> > > > >> 在 2019年3月8日，上午12:08，Морковкин, Василий Владимирович < > morkovkin...@phystech.edu> 写道： > >> > >> Hi! For fun I've sketched a toy-prototype of workflow manager in Scala. > It makes it easy to impose dependencies on the execution order of tasks. > Check this out: https://scastie.scala-lang.org/aRJGberkQ4CWatyABCOJcQ < > https://scastie.scala-lang.org/aRJGberkQ4CWatyABCOJcQ> . It reproduces > the flow which is shown in the attached picture. > >> Xun Liu, It would be great to clarify whether you agree to be a mentor > exactly within GSOC, or without it? :) > >> > >> ---------------------------------------- > >> Best regards, Basil Morkovkin > >> > >> чт, 7 мар. 2019 г. в 11:32, Jeff Zhang <zjf...@gmail.com <mailto: > zjf...@gmail.com>>: > >> > >> Thanks Liu for taking over this, I will help review the design. > >> > >> Xun Liu <neliu...@163.com <mailto:neliu...@163.com>> 于2019年3月7日周四 > 下午4:05写道： > >> Hi Vasiliy Morkovkin > >> > >> Thank you very much for your willingness to implement this feature of > workflow. > >> I will work with you with the highest priority. > >> I am planning to update the system design documentation for workflow > first at https://issues.apache.org/jira/browse/ZEPPELIN-4018 < > https://issues.apache.org/jira/browse/ZEPPELIN-4018> < > https://issues.apache.org/jira/browse/ZEPPELIN-4018 < > https://issues.apache.org/jira/browse/ZEPPELIN-4018>> . > >> Please set the Watcher in ZEPPELIN-4018. > >> This way you can get notification messages for document updates in a > timely manner. > >> > >> We can communicate all the questions in the ZEPPELIN-4018 JIRA comments. > >> If you need it, you can email me at liuxun...@gmail.com <mailto: > liuxun...@gmail.com> <mailto:liuxun...@gmail.com <mailto: > liuxun...@gmail.com>> , I will reply you the fastest. > >> Do you think this kind of cooperation is OK? > >> > >> > >> @moon, @Jeff, @Jongyoul Lee , If interested, Please help us improve our > system design. Thanks! > >> > >> :-) > >> > >>> 在 2019年3月7日，上午6:04，Морковкин, Василий Владимирович < > morkovkin...@phystech.edu <mailto:morkovkin...@phystech.edu>> 写道： > >>> > >>> Thank you for such a detailed feedback! > >>> I am definitely interested to work on the workflow implementation with > you Xun Liu! Could you become a mentor in GSOC with this task? > >>> Some front-end work is not a problem at all. > >>> I'm ready to work at least 30 hours per week in the summer, while now > I'd like to take some smaller tasks to take a closer look at existing > codebase and to get familiar with your development workflow. Do you have > such tasks on mind? > >>> > >>> ср, 6 мар. 2019 г. в 05:23, Xun Liu <neliu...@163.com <mailto: > neliu...@163.com> <mailto:neliu...@163.com <mailto:neliu...@163.com>>>: > >>> Hi Vasiliy Morkovkin > >>> > >>> I said my thoughts on workflow, > https://issues.apache.org/jira/browse/ZEPPELIN-4018 < > https://issues.apache.org/jira/browse/ZEPPELIN-4018> < > https://issues.apache.org/jira/browse/ZEPPELIN-4018 < > https://issues.apache.org/jira/browse/ZEPPELIN-4018>> > >>> > >>> Because there are more than 20 interpreters in zeppelin, > >>> Data analysts can be used to do a variety of data development, > >>> A lot of data development is interdependent. For example, > >>> the development of machine learning algorithms requires relying on > spark to preprocess data, and so on. > >>> > >>> Now open source workflow software has Azkaban, airflow, > >>> Azkaban is relatively simple and has been used to meet most scenarios, > and our company is using it. > >>> Airflow looks complicated and I have not used it. > >>> In fact, I have previously implemented workflow workflow for notes and > paragraphs in zeppelin via azkaban. > >>> https://youtu.be/2r6q-2Tq7hk?t=33 <https://youtu.be/2r6q-2Tq7hk?t=33> > <https://youtu.be/2r6q-2Tq7hk?t=33 <https://youtu.be/2r6q-2Tq7hk?t=33>> > >>> > >>> However, I think zeppelin should have built-in workflow capabilities. > >>> Instead of relying on external software to schedule notes in zeppelin > for the following reasons: > >>> 1. Now that we have upgraded from the data processing era to the > algorithm era, > >>> After zeppelin has its own workflow, it will form a data loop. > >>> > >>> 2. zeppelin's powerful interactive processing capabilities help > algorithm engineers improve productivity and work. > >>> Zeppelin should give the algorithm engineer more direct control. > >>> Instead of handing the algorithm to other teams(or software) to do the > workflow. > >>> > >>> 3. zeppelin knows more about the processing status of data than > Azkaban and airflow. > >>> So the built-in workflow will have better performance, user experience > and control. > >>> > >>> If you are interested in workflow(ZEPPELIN-4018), > >>> I am willing to work with you to complete all system design and code > development work. > >>> > >>> :-) > >>> > >>>> 在 2019年3月6日，上午9:32，Jeff Zhang <zjf...@gmail.com <mailto: > zjf...@gmail.com> <mailto:zjf...@gmail.com <mailto:zjf...@gmail.com>>> 写道： > >>>> > >>>> https://issues.apache.org/jira/browse/ZEPPELIN-3857Hi < > https://issues.apache.org/jira/browse/ZEPPELIN-3857Hi> < > https://issues.apache.org/jira/browse/ZEPPELIN-3857Hi < > https://issues.apache.org/jira/browse/ZEPPELIN-3857Hi>> Basil, > >>>> > >>>> Thanks for your interest in zeppelin, here's my comments about the > tickets > >>>> you interested. > >>>> > >>>> 1. https://issues.apache.org/jira/browse/ZEPPELIN-3651 < > https://issues.apache.org/jira/browse/ZEPPELIN-3651> < > https://issues.apache.org/jira/browse/ZEPPELIN-3651 < > https://issues.apache.org/jira/browse/ZEPPELIN-3651>> > >>>> This involves 2 sides of work: frontend and backend: > >>>> In frontend, we should use arrow js to handle the table data, > include > >>>> display it and processing it (such as aggregation) > >>>> In backend, we should use arrow for each language, and allow them to > >>>> exchange data in the same process. And use arrow IPC to exchange data > >>>> across processes. > >>>> Overall, this is a pretty large task. If you really want to do, I > would > >>>> suggest you to just take part of it. > >>>> > >>>> 2. https://issues.apache.org/jira/browse/ZEPPELIN-3994 < > https://issues.apache.org/jira/browse/ZEPPELIN-3994> < > https://issues.apache.org/jira/browse/ZEPPELIN-3994 < > https://issues.apache.org/jira/browse/ZEPPELIN-3994>> > >>>> Regarding model serving, I don't have clear picture about this. > Others > >>>> can comment on this. > >>>> > >>>> 3. https://issues.apache.org/jira/browse/ZEPPELIN-4018 < > https://issues.apache.org/jira/browse/ZEPPELIN-4018> < > https://issues.apache.org/jira/browse/ZEPPELIN-4018 < > https://issues.apache.org/jira/browse/ZEPPELIN-4018>> > >>>> Job scheduling is pretty important for zeppelin, I would make this > as > >>>> the highest priority for zeppelin among these tickets. airflow is one > >>>> option, but I am open to other solutions. First we need to figure out > how > >>>> user schedule jobs in zeppelin, then choose the right framework. It > would > >>>> also involves some frontend work > >>>> > >>>> 4. https://issues.apache.org/jira/browse/ZEPPELIN-3857 < > https://issues.apache.org/jira/browse/ZEPPELIN-3857> < > https://issues.apache.org/jira/browse/ZEPPELIN-3857 < > https://issues.apache.org/jira/browse/ZEPPELIN-3857>> > >>>> Spark 2.4.0 supporting is already there, but scala 2.12 is not > >>>> supported yet. It won't be a big project for GSOC IMO. > >>>> > >>>> 5. OLAP. > >>>> Regarding OLAP, as long as the OLAP engine provide Jdbc interface, > >>>> Zeppelin can support it very well. But we could create specific > interpreter > >>>> for OLAP engine if their native api perform better than jdbc. Another > thing > >>>> I can think of improving OLAP is visualization, although Zeppelin > already > >>>> support some built-in visualization, there's still some visualization > >>>> missing. We could provide more. > >>>> > >>>> 6. Auto-completions. > >>>> We have already support ipython[1] in zeppelin which provide almost > the > >>>> same auto-completion like jupyter. But it lacks for accessing python > api > >>>> doc. This is also pretty important for python users IMO. SQL is > another > >>>> popular language in Zeppelin, but it also doesn't provide good > >>>> code-completion experience, we can do better as well. > >>>> > >>>> 7. Notifications. > >>>> I think notification can be integrated into job scheduling. > Notification > >>>> can be sent when job is failed/succeed. > >>>> > >>>> > >>>> Let us know which jira you are more interested, and also please > consider > >>>> how much time you can spent on this. Again, we are very appreciated > your > >>>> interest on zeppelin and look forward your contribution. > >>>> > >>>> > >>>> [1] > >>>> > http://zeppelin.apache.org/docs/0.8.1/interpreter/python.html#ipython-support > < > http://zeppelin.apache.org/docs/0.8.1/interpreter/python.html#ipython-support> > < > http://zeppelin.apache.org/docs/0.8.1/interpreter/python.html#ipython-support > < > http://zeppelin.apache.org/docs/0.8.1/interpreter/python.html#ipython-support > >> > >>>> > >>>> > >>>> > >>>> Морковкин, Василий Владимирович <morkovkin...@phystech.edu <mailto: > morkovkin...@phystech.edu> <mailto:morkovkin...@phystech.edu <mailto: > morkovkin...@phystech.edu>>> 于2019年3月6日周三 > >>>> 上午7:41写道： > >>>> > >>>>> Thank you for your replies! I've checked existing set of issues and > found > >>>>> several curious ones: > >>>>> - https://issues.apache.org/jira/browse/ZEPPELIN-3651 < > https://issues.apache.org/jira/browse/ZEPPELIN-3651> < > https://issues.apache.org/jira/browse/ZEPPELIN-3651 < > https://issues.apache.org/jira/browse/ZEPPELIN-3651>> seems to be very > >>>>> nice > >>>>> way to increase analytical processing performance using Arrow > project; > >>>>> - https://issues.apache.org/jira/browse/ZEPPELIN-3994 < > https://issues.apache.org/jira/browse/ZEPPELIN-3994> < > https://issues.apache.org/jira/browse/ZEPPELIN-3994 < > https://issues.apache.org/jira/browse/ZEPPELIN-3994>> deploying models > >>>>> regardless of ZeppelinServer sounds quite intriguing too. Although > there is > >>>>> much to think about; > >>>>> - https://issues.apache.org/jira/browse/ZEPPELIN-4018 < > https://issues.apache.org/jira/browse/ZEPPELIN-4018> < > https://issues.apache.org/jira/browse/ZEPPELIN-4018 < > https://issues.apache.org/jira/browse/ZEPPELIN-4018>> at first glance > >>>>> https://airflow.apache.org/ <https://airflow.apache.org/> < > https://airflow.apache.org/ <https://airflow.apache.org/>> seems to be > useful in implementing complex > >>>>> execution workflows. > >>>>> Those tasks are global and intriguing, requiring complex > architectural > >>>>> solutions. > >>>>> Also I've probably found the ticket which is suitable for me to get > >>>>> involved into the project: > >>>>> - https://issues.apache.org/jira/browse/ZEPPELIN-3857 < > https://issues.apache.org/jira/browse/ZEPPELIN-3857> < > https://issues.apache.org/jira/browse/ZEPPELIN-3857 < > https://issues.apache.org/jira/browse/ZEPPELIN-3857>>. What do you think? > >>>>> Are there any "low hanging fruits"? > >>>>> > >>>>> And I have several ideas on my own. Some of them might be not > relevant due > >>>>> to the vision of the project or other reasons. Just ideas: > >>>>> - OLAP. As Zeppelin is a tool aimed at analytics, it seems to be > quite > >>>>> logical to add more integrations with existing OLAP solutions like > Pinot, > >>>>> ClickHouse and Druid. Currently I've found integration only with > Kylin; > >>>>> - Better autocompletion. Jupyter offers not only a list of already > >>>>> initialized variables, but also quick access to documentation. It's > >>>>> convenient; > >>>>> - Notifications. Some colleagues would have appreciated the > notifications > >>>>> service, which sends you messages (via mail, Slack bot or something > else) > >>>>> indicating that your long-running paragraphs has completed. > >>>>> > >>>>> Feedback is very appreciated :) > >>>>> > >>>>> It would be wonderful if someone agreed to sacrifice his time and > become a > >>>>> mentor in GSOC program! > >>>>> > >>>>> ---------------------------------------- > >>>>> Best regards, Basil Morkovkin. > >>>>> > >>>>> > >>>>> вт, 5 мар. 2019 г. в 11:48, Jongyoul Lee <jongy...@gmail.com > <mailto:jongy...@gmail.com> <mailto:jongy...@gmail.com <mailto: > jongy...@gmail.com>>>: > >>>>> > >>>>>> Hello, > >>>>>> > >>>>>> I've confirmed I could add more issues for GSOC. Can you explain > what you > >>>>>> would like to contribute to? I can add more issues > >>>>>> > >>>>>> JL > >>>>>> > >>>>>> On Tue, Mar 5, 2019 at 1:03 PM Xun Liu <neliu...@163.com <mailto: > neliu...@163.com> <mailto:neliu...@163.com <mailto:neliu...@163.com>>> > wrote: > >>>>>> > >>>>>>> Hi, Vasiliy Morkovkin > >>>>>>> > >>>>>>> Welcome to the zeppelin community! :-) > >>>>>>> > >>>>>>>> 在 2019年3月5日，上午11:49，Jongyoul Lee <jongy...@gmail.com <mailto: > jongy...@gmail.com> <mailto:jongy...@gmail.com <mailto:jongy...@gmail.com>>> > 写道： > >>>>>>>> > >>>>>>>> Thanks for contacting Zeppelin with your interest. > >>>>>>>> > >>>>>>>> I added FE topics for GSOC because FE is the most urgent issue I > have > >>>>>>>> thought about. We always encourage to contribute Zeppelin with > several > >>>>>>>> topics including your idea. > >>>>>>>> > >>>>>>>> Please describe something more. > >>>>>>>> > >>>>>>>> Thanks. > >>>>>>>> JL > >>>>>>>> > >>>>>>>> On Tue, Mar 5, 2019 at 10:41 AM moon soo Lee <m...@apache.org > <mailto:m...@apache.org> <mailto:m...@apache.org <mailto:m...@apache.org>>> > wrote: > >>>>>>>> > >>>>>>>>> Hi, > >>>>>>>>> > >>>>>>>>> Great to see your interest to project. Thanks! > >>>>>>>>> Looks like we need volunteers for a mentor and some backend > subject > >>>>> for > >>>>>>>>> GSoC2019. > >>>>>>>>> Any ideas? > >>>>>>>>> > >>>>>>>>> Best, > >>>>>>>>> moon > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Mon, Mar 4, 2019 at 3:05 PM Vasiliy Morkovkin < > >>>>>>>>> morkovkin...@phystech.edu <mailto:morkovkin...@phystech.edu> > <mailto:morkovkin...@phystech.edu <mailto:morkovkin...@phystech.edu>>> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Hi everyone, I'm pursuing bachelor degree at Moscow institute of > >>>>>>> physics > >>>>>>>>>> and technology and eager to contribute to Zeppelin in context of > >>>>> GSOC > >>>>>>>>>> 2019. I've become a real fan of Zeppelin over the past couple of > >>>>>>> months, > >>>>>>>>>> using it at my job. But I have found out only one ticket > (front-end > >>>>>>>>>> task) with label of GSOC 2019 on your Jira. Perhaps you may > have any > >>>>>>>>>> ideas for new features or improvements in Zeppelin, but you > don't > >>>>> have > >>>>>>>>>> enough hands on them. It would be wonderful if anyone agreed to > >>>>> mentor > >>>>>>>>>> these ideas within GSOC :) > >>>>>>>>>> Currently I am in a position of Scala developer (back-end) for > 1.5 > >>>>>>> year. > >>>>>>>>>> I also can write in Java or Python without any problems if > >>>>> necessary. > >>>>>>>>>> Really fond of databases and highload. Also I have experience > with > >>>>>>> some > >>>>>>>>>> other great Apache projects like Cassandra, Kafka and Spark. > >>>>>>>>>> > >>>>>>>>>> Best regards, Basil Morkovkin. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>>>> 이종열, Jongyoul Lee, 李宗烈 > >>>>>>>> http://madeng.net <http://madeng.net/> <http://madeng.net/ < > http://madeng.net/>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> -- > >>>>>> 이종열, Jongyoul Lee, 李宗烈 > >>>>>> http://madeng.net <http://madeng.net/> <http://madeng.net/ < > http://madeng.net/>> > >>>>>> > >>>>> > >>>> > >>>> > >>>> -- > >>>> Best Regards > >>>> > >>>> Jeff Zhang > >>> > >> > >> > >> > >> -- > >> Best Regards > >> > >> Jeff Zhang > > > > > -- Best Regards Jeff Zhang
