Sorry – catching up after a few days away. I agree with Jeff’s list. To clarify 2a I would like to be able to see the default interpreter group, and set it, at the top of each notebook. A dropdown would be nice.
I still think the execution output of a paragraph should also identify the fully qualified name of the interpreter that was actually used to produce that output. This is because the interpreter group or interpreter, or both, can be changed by the user editing the paragraph since they last ran it. For example, I see this at the foot of one of my paragraphs: Took 4 sec. Last updated by XXXX at March 12 2018, 1:08:14 PM. It could say something like this instead: Took 4 sec. Last updated by XXXX at March 12 2018, 1:08:14 PM. [%spark.pyspark] Thanks, Lucas. From: Jongyoul Lee <jongy...@gmail.com> Sent: 09 July 2018 06:10 To: Jeff Zhang <zjf...@gmail.com> Cc: Paul Brenner <pbren...@placeiq.com>; users <users@zeppelin.apache.org> Subject: EXT: Re: [DISCUSS] Is interpreter binding necessary ? Thank you Jeff for summarizing it. Please file an issue to discuss how to migrate features. On Mon, Jul 9, 2018 at 11:20 AM, Jeff Zhang <zjf...@gmail.com<mailto:zjf...@gmail.com>> wrote: Yes, it is covered, you can choose default interpreter group (with different configuration of max number of cores, or yarn queue) in note page. But you still need to specify %pyspark at the beginning of paragraph to tell zeppelin that you want to use python otherwise zeppelin will treat it as scala code Because scala interpreter is default interpreter of spark interpreter. Paul Brenner <pbren...@placeiq.com<mailto:pbren...@placeiq.com>>于2018年7月9日周一 上午9:31写道： [https://share.polymail.io/v2/z/a/NWI0MmJhMmUwODgy/pd940T9Ui4qGERrcLz0Pxmz26LP3hSe2o-SmkBL9mH9G2LqnZuk21lFYVkXoWXEGMsoRZCM34la8QU9fTClDzcRVPnflz-MHNet5lY9hw50ohdm2e_xVcfoJz0TAoiZVTQHDb2hVqEgSYfA=.png] For 2a, just making sure I understand and see if it covers our use case: we have multiple spark interpeters that we want to be able to set as default. They could all be pyspark, but differ by the configuration of max number of cores, or yarn queue. Is the plan to still allow for setting these differently configured interpreters as default? Thanks! [https://marketing.placeiq.net/images/placeiq.png]<https://share.polymail.io/v1/z/b/NWI0MmJhMmUwODgy/pd940T9Ui4qGERrcLz0Pxmz26LP3hSe2jeSmkBL9mH9G2LqnZuk21lFYVkXoWXEGMuQRZCM34la8QU9fTClDzcRVPnflz-MHNetXlY9hw50ohdm2e_xVcfoJzwnpjjqTbwRrpqEIEXQBnbLP3EwZvXe3EKKSvWfyKqQeAbxEqcxMaEA=> Paul Brenner [https://marketing.placeiq.net/images/twitter1.png]<https://share.polymail.io/v1/z/b/NWI0MmJhMmUwODgy/pd940T9Ui4qGERrcLz0Pxmz26LP3hSe2jeSmkBL9mH9G2LqnZuk21lFYVkXoWXEGMuQRZCM34la8QU9fTClDzcRVPnflz-MHNetXlY9hw50ohdm2e_xVcfoJzwnpjjqTJhFr_qIIVnAZmaOE1lJa8Wi2XjMjqY8JCAvws0lxPS-ooJG7IfL-> [https://marketing.placeiq.net/images/facebook.png]<https://share.polymail.io/v1/z/b/NWI0MmJhMmUwODgy/pd940T9Ui4qGERrcLz0Pxmz26LP3hSe2jeSmkBL9mH9G2LqnZuk21lFYVkXoWXEGMuQRZCM34la8QU9fTClDzcRVPnflz-MHNetXlY9hw50ohdm2e_xVcfoJzwnpjjqTJhFr_qEISCoLnbLP11JYtTa5UD1pkJJTwrbBLcftkPKM8ys2al3B4pSG-Sw=> [https://marketing.placeiq.net/images/linkedin.png]<https://share.polymail.io/v1/z/b/NWI0MmJhMmUwODgy/pd940T9Ui4qGERrcLz0Pxmz26LP3hSe2jeSmkBL9mH9G2LqnZuk21lFYVkXoWXEGMuQRZCM34la8QU9fTClDzcRVPnflz-MHNetXlY9hw50ohdm2e_xVcfoJzwnpjjqTJhFr_qEISCoBlb_B0FlesDa5UD1po5Ff0bLmBZwrMQhfTauAT4W8atODY1R-LK9N6J87IQ==> SR. DATA SCIENTIST (217) 390-3033<tel:(217)%20390-3033> [PlaceIQ:CES 2018]<https://share.polymail.io/v1/z/b/NWI0MmJhMmUwODgy/pd940T9Ui4qGERrcLz0Pxmz26LP3hSe2jeSmkBL9mH9G2LqnZuk21lFYVkXoWXEGMuQRZCM34la8QU9fTClDzcRVPnflz-MHNetXlY9hw50ohdm2e_xVcfoJzwnpjjqTbwRrobcYWndDjL3L1lherza5UD1p8s4Dlv7BEsc-OhtdXKeVCmBn885huh-7DYTLuqjRMsmrV8ux4kPMynbniqfxF5ws6aGI5NH2IUEsetI-k0tLG-TOQz7F> On Sun, Jul 08, 2018 at 8:52 PM Jeff Zhang <Jeff Zhang <mailto:jeff+zhang+%3czjf...@gmail.com%3E> > wrote: Thanks everyone. Let me summarize it. 1. I think no one object to remove the binding info in zeppelin server side (interpreter.json), this would reduce the maintenance effect in zeppelin server. 2. We need to keep 2 ability if we remove the interpreter binding. a. Set the default interpreter group. Actually in the discussion, we talk about 2 concepts of default interpreter. One means the default interpreter group, such as spark, spark2 or livy Another means the default interpreter of interpreter group. such as making pyspark as the default interpreter of spark. Regarding the later one, I plan to make it configurable in interpreter setting page which is not related for interpreter binding. b. Restart interpreter in note page. If there's no more objection, I will go ahead to create ticket and continue the work. moon soo Lee <m...@apache.org<mailto:m...@apache.org>>于2018年7月7日周六 下午6:46写道： +1 for remove binding. But keep ability to set default interpreter and restart interpreter (in note level) On Sat, Jul 7, 2018 at 01:21 Ruslan Dautkhanov <dautkha...@gmail.com<mailto:dautkha...@gmail.com>> wrote: That interpreting binding UI was also used to restart interpreters that our users were using form time to time. If we remove interpreter binding, there should still be a way to restart them quickly without going deep into menus. And also there should be a way to show interpreter status, like for example proposed in https://github.com/apache/zeppelin/pull/2974 -- Ruslan Dautkhanov On Fri, Jul 6, 2018 at 9:56 AM Ruslan Dautkhanov <dautkha...@gmail.com<mailto:dautkha...@gmail.com>> wrote: +1 to remove it Setting default interpreter is not very useful anyway (for example, we can't make %pyspark default without manually editing xml files in Zeppelin distro). https://issues.apache.org/jira/browse/ZEPPELIN-3282 -- Ruslan Dautkhanov On Fri, Jul 6, 2018 at 7:27 AM Paul Brenner <pbren...@placeiq.com<mailto:pbren...@placeiq.com>> wrote: [https://share.polymail.io/v2/z/a/NWIzZjZkNTc0YWE0/kVB6OMitKz7Pqm5NuaQG4awyekI99c0gvBUdEDv2kriJlBnjhW1b_KV6uy_oCGrO27pi9NpPRXrqb1oYLH5U3-CVKF4hGqzYaRjjEfcRIrXjFevdugvD0d2pmfVXx7uzGRDDgTEtQ-FpZrw2ktzsAl7DmYc=.png] I agree with Partridge. We have different interpreters defined with different queues and settings. So we need a way to quickly change the default interpreter and can’t rely on typing the desired interpreter at the start of each paragraph. [https://marketing.placeiq.net/images/placeiq.png]<https://share.polymail.io/v1/z/b/NWIzZjZkNTc0YWE0/kVB6OMitKz7Pqm5NuaQG4awyekI99c0gkhUdEDv2kriJlBnjhW1b_KV6uy_oCGrO25Ri9NpPRXrqb1oYLH5U3-CVKF4hGqzYaRjNEfcRIrXjFevdugvD0d2pmfVXx7uzGRDDgZjI9stT-RZWW0TiJaTPs_IOYGlF1ZvI7h9Qb2g_eDdzcp6fWnRGMKU=> Paul Brenner [https://marketing.placeiq.net/images/twitter1.png]<https://share.polymail.io/v1/z/b/NWIzZjZkNTc0YWE0/kVB6OMitKz7Pqm5NuaQG4awyekI99c0gkhUdEDv2kriJlBnjhW1b_KV6uy_oCGrO25Ri9NpPRXrqb1oYLH5U3-CVKF4hGqzYaRjNEfcRIrXjFevdugvD0d2pmfVXx7uzGRDDgZjI9stTsANWA0fiYqDXt-NFancGmYTJoNoEym-hk4ZudikiEh22ahQg0LL2> [https://marketing.placeiq.net/images/facebook.png]<https://share.polymail.io/v1/z/b/NWIzZjZkNTc0YWE0/kVB6OMitKz7Pqm5NuaQG4awyekI99c0gkhUdEDv2kriJlBnjhW1b_KV6uy_oCGrO25Ri9NpPRXrqb1oYLH5U3-CVKF4hGqzYaRjNEfcRIrXjFevdugvD0d2pmfVXx7uzGRDDgZjI9stTsANWA0TifPrFs_IOa3cE3drGrtRO83LeA-sOurZBpe40Kl9Vs2ciD7HZiFQ=> [https://marketing.placeiq.net/images/linkedin.png]<https://share.polymail.io/v1/z/b/NWIzZjZkNTc0YWE0/kVB6OMitKz7Pqm5NuaQG4awyekI99c0gkhUdEDv2kriJlBnjhW1b_KV6uy_oCGrO25Ri9NpPRXrqb1oYLH5U3-CVKF4hGqzYaRjNEfcRIrXjFevdugvD0d2pmfVXx7uzGRDDgZjI9stTsANWA0TifPrPu_8AbHwC2NrGrtROwHHSEO8pkpgUNczTb1OqOVPmpWhChtSWVFdvwywOLA==> SR. DATA SCIENTIST (217) 390-3033<tel:(217)%20390-3033> [PlaceIQ:CES 2018]<https://share.polymail.io/v1/z/b/NWIzZjZkNTc0YWE0/kVB6OMitKz7Pqm5NuaQG4awyekI99c0gkhUdEDv2kriJlBnjhW1b_KV6uy_oCGrO25Ri9NpPRXrqb1oYLH5U3-CVKF4hGqzYaRjNEfcRIrXjFevdugvD0d2pmfVXx7uzGRDDgZjI9stT-RZWXFLybqeNov0Kan0Cx9rGrtROkS6OV6MOhcMBPt_Rfl-_H1C83vwzLxF9yYwvak6Ys54FEOBh84xdmt2aAQW1UYN7FyjxR-i4tcfIzqeCHMRwEvO3G63G> On Fri, Jul 06, 2018 at 6:02 AM Jongyoul Lee <Jongyoul Lee <mailto:jongyoul+lee+%3cjongy...@gmail.com%3E> > wrote: It looks like a good idea to show the default interpreter group - and it should be also changeable - and store it to `note.json`. It could remove interpreter-binding information, keep default interpreter when extracting note. Moreover it also helps to reduce our server code as well. What do you guys think of? On Fri, Jul 6, 2018 at 6:16 PM, Sanjay Dasgupta <sanjay.dasgu...@gmail.com<mailto:sanjay.dasgu...@gmail.com>> wrote: If there is no per-interpreter overhead of binding all the interpreters from the beginning, we should definitely do it. This will simplify the GUI somewhat. Regards, - Sanjay On Fri, Jul 6, 2018 at 1:49 PM, Partridge, Lucas (GE Aviation) <lucas.partri...@ge.com<mailto:lucas.partri...@ge.com>> wrote: “So usually we would recommend user to specify the full qualified interpreter name.” - I usually recommend the exact opposite to our users. We frequently change interpreter groups to allow for different Spark cluster settings (number of executors, memory, etc). Users with more demanding requirements are asked to use custom interpreter groups with more allocated resources. If users included the interpreter group name at the start of every paragraph they would then have to manually edit the start of every paragraph before they could run their note using a different interpreter group. Very tedious! But I agree the short names without the interpreter group are often ambiguous and can cause confusion. Maybe somewhere in the execution output of each paragraph there should be some discrete text giving the fully qualified name of the interpreter that was actually used to produce that output. Or a clearly defined ‘default interpreter group’ text in the toolbar at the top of each notebook. Make it a dropdown so it would be easy to change the default. From: Jeff Zhang <zjf...@gmail.com<mailto:zjf...@gmail.com>> Sent: 06 July 2018 08:53 To: users@zeppelin.apache.org<mailto:users@zeppelin.apache.org> Cc: dev <d...@zeppelin.apache.org<mailto:d...@zeppelin.apache.org>> Subject: EXT: Re: [DISCUSS] Is interpreter binding necessary ? We already allow setting default interpreter when creating note. Another way to set default interpreter is to reorder the interpreter setting binding in note page. But personally I don't recommend user to use short interpreter name because of default interpreter. 2 Reaons: 1. It introduce in-accurate info. e.g. In our product, we have 2 spark interpreters (`spark`: for spark 1.x & `spark2` for spark 2.x). Then user often specify `%spark` for spark interpreter. But it could mean both `%spark.spark` and `%spark2.spark`, So usually it is very hard to tell what's wrong when user expect to work spark2 but actually he still use spark 1.x. So usually we would recommend user to specify the full qualified interpreter name. Just type several more characters which just cost 2 seconds but make it more clear and readable. 2. Another issue is that interpreter binding is stored in interpreter.json, that means if they export this note to another zeppelin instance, the default interpreter won't work. So I don't think setting default interpreter via interpreter binding is valuable for users. If user really want to do that, I would suggest to store it in note.json instead of interpreter.json Jongyoul Lee <jongy...@gmail.com<mailto:jongy...@gmail.com>>于2018年7月6日周五 下午3:36写道： There are two purposes of interpreter binding. One is what you mentioned and another one is to manage a default interpreter. If we provide a new way to set default interpreter, I think we can remove them :-) We could set permissions in other ways. Overall, +1 On Fri, Jul 6, 2018 at 4:24 PM, Jeff Zhang <zjf...@gmail.com<mailto:zjf...@gmail.com>> wrote: Hi Folks, I raise this thread to discuss whether we need the interpreter binding. Currently when user create notes, they have to bind interpreters to their notes in note page. Otherwise they will hit interpreter not found issue. Besides that in zeppelin server side, we maintain the interpreter binding info in memory as well as in interpreter.json. IMHO, it is not necessary to do interpreter binding. Because it just add extra burden to maintain the interpreter binding info in zeppelin server side, and doesn't introduce any benefits. The only benefit is that we will check whether user have permission to use this interpreter, but actually zeppelin will check the permission when running paragraph, so I don't think we need to introduce interpreter binding just for this kind of permission check that we will do later. So overall, I would suggest to remove interpreter binding feature. What do you think ? -- 이종열, Jongyoul Lee, 李宗烈 http://madeng.net -- 이종열, Jongyoul Lee, 李宗烈 http://madeng.net -- 이종열, Jongyoul Lee, 李宗烈 http://madeng.net
