Hi Sachin, Thank you for your work. From the screenshots it looks like you replicated the behavior of the old web client. I would like to take a look at the actual code. There are at least three things we need to work on next:
1. Introduce a config variable to disable the job submission page 2. Add an upload functionality 3. Integrate with YARN (respect the correct directories to write into). As for the parameter tool, this is an additional feature and is IMHO not required as a first step. Cheers, Max On Sun, Nov 8, 2015 at 2:45 PM, Sachin Goel <sachingoel0...@gmail.com> wrote: > Hi Flavio > Yes. Multiple classes are reported properly. > > And having a Parameter interface sounds nice, but IMO it'll make the UI too > cluttered. What can be done is to, say, search for a method named > `getDescription` and display that to the user on frontend? Right now, we > only do that for Program interface extending classes. A little bit of > reflection magic will allow so for Interactive programs too. > I also looked at the ParameterTool API, but it doesn't appear to be used > much anywhere. Although I think there's some work going on this. I'm not > sure if that's been merged yet. > Also, managing required parameters, and their types etc. is somewhat > frustrating in JavaScript. Everything has to be rechecked at the Server > again anyways. > > I opened a PR just now: https://github.com/apache/flink/pull/1338. Let me > know what you guys think. :) > > Cheers! > Sachin > > > -- Sachin Goel > Computer Science, IIT Delhi > m. +91-9871457685 > > On Sat, Nov 7, 2015 at 5:11 AM, Flavio Pompermaier <pomperma...@okkam.it> > wrote: > > > Hi Sachin, > > great work and very quickly! > > Just a question: do you already manage a jar contain multiple jobs > > (multiple main classes or Program interface implementation)? > > I have also a suggestion for a very nice feature: right now there's the > > ParameterTool API that centralize and standardize the passing of > parameters > > to the program. With a little bit of effort I think it should be possible > > to show to the user an automatically computed list of required and > optional > > parameters, with their default value (if any) and the possibility to > check > > if the passed parameter respect the specified type. > > What do you think? > > Update: I've finished most of the work on this. > > Here are a few screenshots: > > > > > https://drive.google.com/file/d/0B0Xlv1qTgpjDRlMxRkQ5OG5PRUk/view?usp=sharing > > > > > https://drive.google.com/file/d/0B0Xlv1qTgpjDczJwLU13ZWM5anM/view?usp=sharing > > > > > https://drive.google.com/file/d/0B0Xlv1qTgpjDbXlwbWhVelFFRWM/view?usp=sharing > > > > > https://drive.google.com/file/d/0B0Xlv1qTgpjDX2VlWkl6ZTRTLWM/view?usp=sharing > > > > > https://drive.google.com/file/d/0B0Xlv1qTgpjDZHNFWFFBVlg1dmM/view?usp=sharing > > > > > https://drive.google.com/file/d/0B0Xlv1qTgpjDZEZHdE9pS1V0RVE/view?usp=sharing > > > > If you want to try it out, here's my branch: > > https://github.com/sachingoel0101/flink/tree/client > > Just make sure to copy the jars to "webclient-jobs" in this format: <13 > > digit timestamp>_<jar_name>.jar. [Uploading functionality is not yet > > implemented.] > > > > I also made the decision to only allow two flink options: parallelism and > > entry class. I think this is reasonable because we do not need anything > > related to yarn here. Web monitor already has the gateway to job manager. > > > > Any comments or review is welcome. :) > > > > I'll likely be able to finish up by the end of this week. > > > > Cheers! > > Sachin > > > > > > -- Sachin Goel > > Computer Science, IIT Delhi > > m. +91-9871457685 > > > > On Thu, Nov 5, 2015 at 5:46 PM, Sachin Goel <sachingoel0...@gmail.com> > > wrote: > > > > > I plan to implement all features of the existing web client. :) I think > > > multiple programs will be best displayed as a list in the Entry Class > > > column. > > > > > > -- Sachin Goel > > > Computer Science, IIT Delhi > > > m. +91-9871457685 > > > > > > On Thu, Nov 5, 2015 at 5:36 PM, Matthias J. Sax <mj...@apache.org> > > wrote: > > > > > >> Looks good. How would you display a jar that contains multiple > programs > > >> and entry classes? Right now, WebClient can handle this: > > >> > > >> > > > https://ci.apache.org/projects/flink/flink-docs-master/apis/web_client.html > > >> > > >> The Manifest entries "Main-Class" and "program-class" can have a comma > > >> separated list of entry classes. Furthermore, if interface > > >> ProgramDescription is implemented by entry class, a tooltip shows the > > >> description right now. > > >> > > >> And you should go with "Flink Options" instead of "Flink Arguments". > > >> That was the consensus in WebClient too, in order to distinguish both > > >> clearly. > > >> > > >> -Matthias > > >> > > >> On 11/05/2015 12:04 PM, Sachin Goel wrote: > > >> > Disabling the access for secure installation: Pretty easy. Writing > it > > >> right > > >> > now. > > >> > > > >> > @mxm: Thanks. I haven't implemented any handlers for actually > running > > >> the > > >> > job. I hope to finish it by this weekend. > > >> > > > >> > One quick question: The runtime monitor will be running the job > > itself, > > >> > i.e., the main method of program's entry class. Will that be a major > > >> issue, > > >> > that one thread is performing this work [although the server is > > capable > > >> to > > >> > handling several requests in parallel]? > > >> > > > >> > -- Sachin Goel > > >> > Computer Science, IIT Delhi > > >> > m. +91-9871457685 > > >> > > > >> > On Thu, Nov 5, 2015 at 4:28 PM, Maximilian Michels <m...@apache.org> > > >> wrote: > > >> > > > >> >> @Robert: I'm in favor of having a submission page which can be > > >> deactivated. > > >> >> My point was simply that we don't have proper security support > right > > >> now. I > > >> >> agree with Stephan that we'll need to support it! > > >> >> > > >> >> @Stephan: Good suggestion to disable the submission page for the > > >> per-job > > >> >> YARN cluster. > > >> >> > > >> >> @Sachin: Looks nice. Probably there should also be an option to > view > > >> the > > >> >> plan like in the web client. Should not be so hard since we already > > >> have > > >> >> the visualization in place on other pages. > > >> >> > > >> >> On Thu, Nov 5, 2015 at 11:53 AM, Sachin Goel < > > sachingoel0...@gmail.com > > >> > > > >> >> wrote: > > >> >> > > >> >>> Ah. Sharing the links then: > > >> >>> > > >> >>> > > >> >> > > >> > > > > > https://drive.google.com/file/d/0B0Xlv1qTgpjDT0ZCd0pSU3VYMUU/view?usp=sharing > > >> >>> > > >> >>> > > >> >> > > >> > > > > > https://drive.google.com/file/d/0B0Xlv1qTgpjDeG51azRRc0FJYjQ/view?usp=sharing > > >> >>> > > >> >>> > > >> >> > > >> > > > > > https://drive.google.com/file/d/0B0Xlv1qTgpjDSHZXQ0h5S0FBS3c/view?usp=sharing > > >> >>> > > >> >>> -- Sachin Goel > > >> >>> Computer Science, IIT Delhi > > >> >>> m. +91-9871457685 > > >> >>> > > >> >>> On Thu, Nov 5, 2015 at 4:18 PM, Robert Metzger < > rmetz...@apache.org > > > > > >> >>> wrote: > > >> >>> > > >> >>>> Hey Sachin, I think you can not upload images to apache mailing > > >> lists. > > >> >>>> > > >> >>>> On Thu, Nov 5, 2015 at 11:46 AM, Sachin Goel < > > >> sachingoel0...@gmail.com > > >> >>> > > >> >>>> wrote: > > >> >>>> > > >> >>>>> Okay. I think that does it. I've already designed the basic > > >> >> interface, > > >> >>>>> although the major part for uploading jars and actually running > > the > > >> >> job > > >> >>>> are > > >> >>>>> left. :') > > >> >>>>> Here are some screenshots: > > >> >>>>> [image: Inline image 2][image: Inline image 1][image: Inline > image > > >> 3] > > >> >>>>> > > >> >>>>> -- Sachin Goel > > >> >>>>> Computer Science, IIT Delhi > > >> >>>>> m. +91-9871457685 > > >> >>>>> > > >> >>>>> On Thu, Nov 5, 2015 at 4:12 PM, Maximilian Michels < > > m...@apache.org> > > >> >>>> wrote: > > >> >>>>> > > >> >>>>>> +1 The web client needs to go. > > >> >>>>>> > > >> >>>>>> I don't really see how how a submission page on the job manager > > >> >> would > > >> >>>> make > > >> >>>>>> security worse. At the moment, anyone who sends an Akka message > > to > > >> >> the > > >> >>>> job > > >> >>>>>> manager, can execute arbitrary code.. > > >> >>>>>> > > >> >>>>>> On Thu, Nov 5, 2015 at 11:16 AM, Flavio Pompermaier < > > >> >>>> pomperma...@okkam.it > > >> >>>>>>> > > >> >>>>>> wrote: > > >> >>>>>> > > >> >>>>>>> +1 to remove the old web client in favor of an improved > > dashboard > > >> >>> (not > > >> >>>>>> so > > >> >>>>>>> urgent however) > > >> >>>>>>> > > >> >>>>>>> On Thu, Nov 5, 2015 at 11:11 AM, Robert Metzger < > > >> >>> rmetz...@apache.org> > > >> >>>>>>> wrote: > > >> >>>>>>> > > >> >>>>>>>> Hi, > > >> >>>>>>>> I personally like the idea of allowing users to submit a job > > >> >> from > > >> >>>> the > > >> >>>>>> web > > >> >>>>>>>> interface a lot. In particular for users new to the system, > its > > >> >> a > > >> >>>> very > > >> >>>>>>>> convenient feature. > > >> >>>>>>>> We would also get rid of this ugly, old web client :) > > >> >>>>>>>> > > >> >>>>>>>> Regarding security: We can always add a configuration > parameter > > >> >>>> which > > >> >>>>>>>> allows users to disable the feature. > > >> >>>>>>>> > > >> >>>>>>>> Please share the screenshot with us :) > > >> >>>>>>>> > > >> >>>>>>>> On Thu, Nov 5, 2015 at 10:08 AM, Sachin Goel < > > >> >>>>>> sachingoel0...@gmail.com> > > >> >>>>>>>> wrote: > > >> >>>>>>>> > > >> >>>>>>>>> The current authentication mechanism is via htaccess, in the > > >> >>>>>> webclient. > > >> >>>>>>>> The > > >> >>>>>>>>> dashboard however isn't secured at all. > > >> >>>>>>>>> > > >> >>>>>>>>> -- Sachin Goel > > >> >>>>>>>>> Computer Science, IIT Delhi > > >> >>>>>>>>> m. +91-9871457685 > > >> >>>>>>>>> > > >> >>>>>>>>> On Thu, Nov 5, 2015 at 2:33 PM, Aljoscha Krettek < > > >> >>>>>> aljos...@apache.org> > > >> >>>>>>>>> wrote: > > >> >>>>>>>>> > > >> >>>>>>>>>> The reason why we didn’t want to have it in the JobManager > > >> >>>>>> interface > > >> >>>>>>> by > > >> >>>>>>>>>> default is that it could be a security concern. I also see, > > >> >>>>>> however, > > >> >>>>>>>> that > > >> >>>>>>>>>> security in general is not very strong right now. > > >> >>>>>>>>>> > > >> >>>>>>>>>>> On 05 Nov 2015, at 09:58, Matthias J. Sax < > > >> >> mj...@apache.org > > >> >>>> > > >> >>>>>>> wrote: > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> I like the idea in general! Not sure if integrating it > > >> >> into > > >> >>>>>>>> JobMangaer > > >> >>>>>>>>>>> Webinterface is good or not... What would be the pros/cons > > >> >>> for > > >> >>>>>>>>>>> standalone vs. integrated? > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> -Matthias > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> On 11/05/2015 05:23 AM, Sachin Goel wrote: > > >> >>>>>>>>>>>> Hi all > > >> >>>>>>>>>>>> Are there any plans to upgrade the webclient interface? > > >> >> It > > >> >>>>>> could > > >> >>>>>>>>>> certainly > > >> >>>>>>>>>>>> use an upgrade compared to the much better dashboard . > > >> >>>>>>>>>>>> If so, is integrating them into one the best way to go? > > >> >> We > > >> >>>> can > > >> >>>>>>> have > > >> >>>>>>>> a > > >> >>>>>>>>>> tab > > >> >>>>>>>>>>>> under Job Manager on the dashboard. > > >> >>>>>>>>>>>> > > >> >>>>>>>>>>>> I can share screenshots of the UI I designed last night, > > >> >> if > > >> >>>>>> it's > > >> >>>>>>>> not a > > >> >>>>>>>>>>>> terrible idea to do this. :') > > >> >>>>>>>>>>>> > > >> >>>>>>>>>>>> Cheers! > > >> >>>>>>>>>>>> Sachin > > >> >>>>>>>>>>>> > > >> >>>>>>>>>>> > > >> >>>>>>>>>> > > >> >>>>>>>>>> > > >> >>>>>>>>> > > >> >>>>>>>> > > >> >>>>>>> > > >> >>>>>> > > >> >>>>> > > >> >>>>> > > >> >>>> > > >> >>> > > >> >> > > >> > > > >> > > >> > > > > > >
