@Max: Added all three things and opened a PR yesterday. :) -- Sachin Goel Computer Science, IIT Delhi m. +91-9871457685
On Mon, Nov 9, 2015 at 3:10 PM, Maximilian Michels <m...@apache.org> wrote: > Hi Sachin, > > Thank you for your work. From the screenshots it looks like you replicated > the behavior of the old web client. I would like to take a look at the > actual code. There are at least three things we need to work on next: > > 1. Introduce a config variable to disable the job submission page > 2. Add an upload functionality > 3. Integrate with YARN (respect the correct directories to write into). > > As for the parameter tool, this is an additional feature and is IMHO not > required as a first step. > > Cheers, > Max > > On Sun, Nov 8, 2015 at 2:45 PM, Sachin Goel <sachingoel0...@gmail.com> > wrote: > > > Hi Flavio > > Yes. Multiple classes are reported properly. > > > > And having a Parameter interface sounds nice, but IMO it'll make the UI > too > > cluttered. What can be done is to, say, search for a method named > > `getDescription` and display that to the user on frontend? Right now, we > > only do that for Program interface extending classes. A little bit of > > reflection magic will allow so for Interactive programs too. > > I also looked at the ParameterTool API, but it doesn't appear to be used > > much anywhere. Although I think there's some work going on this. I'm not > > sure if that's been merged yet. > > Also, managing required parameters, and their types etc. is somewhat > > frustrating in JavaScript. Everything has to be rechecked at the Server > > again anyways. > > > > I opened a PR just now: https://github.com/apache/flink/pull/1338. Let > me > > know what you guys think. :) > > > > Cheers! > > Sachin > > > > > > -- Sachin Goel > > Computer Science, IIT Delhi > > m. +91-9871457685 > > > > On Sat, Nov 7, 2015 at 5:11 AM, Flavio Pompermaier <pomperma...@okkam.it > > > > wrote: > > > > > Hi Sachin, > > > great work and very quickly! > > > Just a question: do you already manage a jar contain multiple jobs > > > (multiple main classes or Program interface implementation)? > > > I have also a suggestion for a very nice feature: right now there's the > > > ParameterTool API that centralize and standardize the passing of > > parameters > > > to the program. With a little bit of effort I think it should be > possible > > > to show to the user an automatically computed list of required and > > optional > > > parameters, with their default value (if any) and the possibility to > > check > > > if the passed parameter respect the specified type. > > > What do you think? > > > Update: I've finished most of the work on this. > > > Here are a few screenshots: > > > > > > > > > https://drive.google.com/file/d/0B0Xlv1qTgpjDRlMxRkQ5OG5PRUk/view?usp=sharing > > > > > > > > > https://drive.google.com/file/d/0B0Xlv1qTgpjDczJwLU13ZWM5anM/view?usp=sharing > > > > > > > > > https://drive.google.com/file/d/0B0Xlv1qTgpjDbXlwbWhVelFFRWM/view?usp=sharing > > > > > > > > > https://drive.google.com/file/d/0B0Xlv1qTgpjDX2VlWkl6ZTRTLWM/view?usp=sharing > > > > > > > > > https://drive.google.com/file/d/0B0Xlv1qTgpjDZHNFWFFBVlg1dmM/view?usp=sharing > > > > > > > > > https://drive.google.com/file/d/0B0Xlv1qTgpjDZEZHdE9pS1V0RVE/view?usp=sharing > > > > > > If you want to try it out, here's my branch: > > > https://github.com/sachingoel0101/flink/tree/client > > > Just make sure to copy the jars to "webclient-jobs" in this format: <13 > > > digit timestamp>_<jar_name>.jar. [Uploading functionality is not yet > > > implemented.] > > > > > > I also made the decision to only allow two flink options: parallelism > and > > > entry class. I think this is reasonable because we do not need anything > > > related to yarn here. Web monitor already has the gateway to job > manager. > > > > > > Any comments or review is welcome. :) > > > > > > I'll likely be able to finish up by the end of this week. > > > > > > Cheers! > > > Sachin > > > > > > > > > -- Sachin Goel > > > Computer Science, IIT Delhi > > > m. +91-9871457685 > > > > > > On Thu, Nov 5, 2015 at 5:46 PM, Sachin Goel <sachingoel0...@gmail.com> > > > wrote: > > > > > > > I plan to implement all features of the existing web client. :) I > think > > > > multiple programs will be best displayed as a list in the Entry Class > > > > column. > > > > > > > > -- Sachin Goel > > > > Computer Science, IIT Delhi > > > > m. +91-9871457685 > > > > > > > > On Thu, Nov 5, 2015 at 5:36 PM, Matthias J. Sax <mj...@apache.org> > > > wrote: > > > > > > > >> Looks good. How would you display a jar that contains multiple > > programs > > > >> and entry classes? Right now, WebClient can handle this: > > > >> > > > >> > > > > > > https://ci.apache.org/projects/flink/flink-docs-master/apis/web_client.html > > > >> > > > >> The Manifest entries "Main-Class" and "program-class" can have a > comma > > > >> separated list of entry classes. Furthermore, if interface > > > >> ProgramDescription is implemented by entry class, a tooltip shows > the > > > >> description right now. > > > >> > > > >> And you should go with "Flink Options" instead of "Flink Arguments". > > > >> That was the consensus in WebClient too, in order to distinguish > both > > > >> clearly. > > > >> > > > >> -Matthias > > > >> > > > >> On 11/05/2015 12:04 PM, Sachin Goel wrote: > > > >> > Disabling the access for secure installation: Pretty easy. Writing > > it > > > >> right > > > >> > now. > > > >> > > > > >> > @mxm: Thanks. I haven't implemented any handlers for actually > > running > > > >> the > > > >> > job. I hope to finish it by this weekend. > > > >> > > > > >> > One quick question: The runtime monitor will be running the job > > > itself, > > > >> > i.e., the main method of program's entry class. Will that be a > major > > > >> issue, > > > >> > that one thread is performing this work [although the server is > > > capable > > > >> to > > > >> > handling several requests in parallel]? > > > >> > > > > >> > -- Sachin Goel > > > >> > Computer Science, IIT Delhi > > > >> > m. +91-9871457685 > > > >> > > > > >> > On Thu, Nov 5, 2015 at 4:28 PM, Maximilian Michels < > m...@apache.org> > > > >> wrote: > > > >> > > > > >> >> @Robert: I'm in favor of having a submission page which can be > > > >> deactivated. > > > >> >> My point was simply that we don't have proper security support > > right > > > >> now. I > > > >> >> agree with Stephan that we'll need to support it! > > > >> >> > > > >> >> @Stephan: Good suggestion to disable the submission page for the > > > >> per-job > > > >> >> YARN cluster. > > > >> >> > > > >> >> @Sachin: Looks nice. Probably there should also be an option to > > view > > > >> the > > > >> >> plan like in the web client. Should not be so hard since we > already > > > >> have > > > >> >> the visualization in place on other pages. > > > >> >> > > > >> >> On Thu, Nov 5, 2015 at 11:53 AM, Sachin Goel < > > > sachingoel0...@gmail.com > > > >> > > > > >> >> wrote: > > > >> >> > > > >> >>> Ah. Sharing the links then: > > > >> >>> > > > >> >>> > > > >> >> > > > >> > > > > > > > > > https://drive.google.com/file/d/0B0Xlv1qTgpjDT0ZCd0pSU3VYMUU/view?usp=sharing > > > >> >>> > > > >> >>> > > > >> >> > > > >> > > > > > > > > > https://drive.google.com/file/d/0B0Xlv1qTgpjDeG51azRRc0FJYjQ/view?usp=sharing > > > >> >>> > > > >> >>> > > > >> >> > > > >> > > > > > > > > > https://drive.google.com/file/d/0B0Xlv1qTgpjDSHZXQ0h5S0FBS3c/view?usp=sharing > > > >> >>> > > > >> >>> -- Sachin Goel > > > >> >>> Computer Science, IIT Delhi > > > >> >>> m. +91-9871457685 > > > >> >>> > > > >> >>> On Thu, Nov 5, 2015 at 4:18 PM, Robert Metzger < > > rmetz...@apache.org > > > > > > > >> >>> wrote: > > > >> >>> > > > >> >>>> Hey Sachin, I think you can not upload images to apache mailing > > > >> lists. > > > >> >>>> > > > >> >>>> On Thu, Nov 5, 2015 at 11:46 AM, Sachin Goel < > > > >> sachingoel0...@gmail.com > > > >> >>> > > > >> >>>> wrote: > > > >> >>>> > > > >> >>>>> Okay. I think that does it. I've already designed the basic > > > >> >> interface, > > > >> >>>>> although the major part for uploading jars and actually > running > > > the > > > >> >> job > > > >> >>>> are > > > >> >>>>> left. :') > > > >> >>>>> Here are some screenshots: > > > >> >>>>> [image: Inline image 2][image: Inline image 1][image: Inline > > image > > > >> 3] > > > >> >>>>> > > > >> >>>>> -- Sachin Goel > > > >> >>>>> Computer Science, IIT Delhi > > > >> >>>>> m. +91-9871457685 > > > >> >>>>> > > > >> >>>>> On Thu, Nov 5, 2015 at 4:12 PM, Maximilian Michels < > > > m...@apache.org> > > > >> >>>> wrote: > > > >> >>>>> > > > >> >>>>>> +1 The web client needs to go. > > > >> >>>>>> > > > >> >>>>>> I don't really see how how a submission page on the job > manager > > > >> >> would > > > >> >>>> make > > > >> >>>>>> security worse. At the moment, anyone who sends an Akka > message > > > to > > > >> >> the > > > >> >>>> job > > > >> >>>>>> manager, can execute arbitrary code.. > > > >> >>>>>> > > > >> >>>>>> On Thu, Nov 5, 2015 at 11:16 AM, Flavio Pompermaier < > > > >> >>>> pomperma...@okkam.it > > > >> >>>>>>> > > > >> >>>>>> wrote: > > > >> >>>>>> > > > >> >>>>>>> +1 to remove the old web client in favor of an improved > > > dashboard > > > >> >>> (not > > > >> >>>>>> so > > > >> >>>>>>> urgent however) > > > >> >>>>>>> > > > >> >>>>>>> On Thu, Nov 5, 2015 at 11:11 AM, Robert Metzger < > > > >> >>> rmetz...@apache.org> > > > >> >>>>>>> wrote: > > > >> >>>>>>> > > > >> >>>>>>>> Hi, > > > >> >>>>>>>> I personally like the idea of allowing users to submit a > job > > > >> >> from > > > >> >>>> the > > > >> >>>>>> web > > > >> >>>>>>>> interface a lot. In particular for users new to the system, > > its > > > >> >> a > > > >> >>>> very > > > >> >>>>>>>> convenient feature. > > > >> >>>>>>>> We would also get rid of this ugly, old web client :) > > > >> >>>>>>>> > > > >> >>>>>>>> Regarding security: We can always add a configuration > > parameter > > > >> >>>> which > > > >> >>>>>>>> allows users to disable the feature. > > > >> >>>>>>>> > > > >> >>>>>>>> Please share the screenshot with us :) > > > >> >>>>>>>> > > > >> >>>>>>>> On Thu, Nov 5, 2015 at 10:08 AM, Sachin Goel < > > > >> >>>>>> sachingoel0...@gmail.com> > > > >> >>>>>>>> wrote: > > > >> >>>>>>>> > > > >> >>>>>>>>> The current authentication mechanism is via htaccess, in > the > > > >> >>>>>> webclient. > > > >> >>>>>>>> The > > > >> >>>>>>>>> dashboard however isn't secured at all. > > > >> >>>>>>>>> > > > >> >>>>>>>>> -- Sachin Goel > > > >> >>>>>>>>> Computer Science, IIT Delhi > > > >> >>>>>>>>> m. +91-9871457685 > > > >> >>>>>>>>> > > > >> >>>>>>>>> On Thu, Nov 5, 2015 at 2:33 PM, Aljoscha Krettek < > > > >> >>>>>> aljos...@apache.org> > > > >> >>>>>>>>> wrote: > > > >> >>>>>>>>> > > > >> >>>>>>>>>> The reason why we didn’t want to have it in the > JobManager > > > >> >>>>>> interface > > > >> >>>>>>> by > > > >> >>>>>>>>>> default is that it could be a security concern. I also > see, > > > >> >>>>>> however, > > > >> >>>>>>>> that > > > >> >>>>>>>>>> security in general is not very strong right now. > > > >> >>>>>>>>>> > > > >> >>>>>>>>>>> On 05 Nov 2015, at 09:58, Matthias J. Sax < > > > >> >> mj...@apache.org > > > >> >>>> > > > >> >>>>>>> wrote: > > > >> >>>>>>>>>>> > > > >> >>>>>>>>>>> I like the idea in general! Not sure if integrating it > > > >> >> into > > > >> >>>>>>>> JobMangaer > > > >> >>>>>>>>>>> Webinterface is good or not... What would be the > pros/cons > > > >> >>> for > > > >> >>>>>>>>>>> standalone vs. integrated? > > > >> >>>>>>>>>>> > > > >> >>>>>>>>>>> -Matthias > > > >> >>>>>>>>>>> > > > >> >>>>>>>>>>> On 11/05/2015 05:23 AM, Sachin Goel wrote: > > > >> >>>>>>>>>>>> Hi all > > > >> >>>>>>>>>>>> Are there any plans to upgrade the webclient interface? > > > >> >> It > > > >> >>>>>> could > > > >> >>>>>>>>>> certainly > > > >> >>>>>>>>>>>> use an upgrade compared to the much better dashboard . > > > >> >>>>>>>>>>>> If so, is integrating them into one the best way to go? > > > >> >> We > > > >> >>>> can > > > >> >>>>>>> have > > > >> >>>>>>>> a > > > >> >>>>>>>>>> tab > > > >> >>>>>>>>>>>> under Job Manager on the dashboard. > > > >> >>>>>>>>>>>> > > > >> >>>>>>>>>>>> I can share screenshots of the UI I designed last > night, > > > >> >> if > > > >> >>>>>> it's > > > >> >>>>>>>> not a > > > >> >>>>>>>>>>>> terrible idea to do this. :') > > > >> >>>>>>>>>>>> > > > >> >>>>>>>>>>>> Cheers! > > > >> >>>>>>>>>>>> Sachin > > > >> >>>>>>>>>>>> > > > >> >>>>>>>>>>> > > > >> >>>>>>>>>> > > > >> >>>>>>>>>> > > > >> >>>>>>>>> > > > >> >>>>>>>> > > > >> >>>>>>> > > > >> >>>>>> > > > >> >>>>> > > > >> >>>>> > > > >> >>>> > > > >> >>> > > > >> >> > > > >> > > > > >> > > > >> > > > > > > > > > >
