It seems more appealing to me that the UI should JMX directly, because: * We're support the JMX in the long term for other management software. * The information provided by the JMX API will be mostly identical of the JSON API. Today the Web UI covers most of the information provided by JMX. The Web UI does some trivial work to extract the information and renders it as HTML. * We can compatibility and unit tests for free.
I do agree that the JMX APIs are imperfect and they should be revisited in the 3.0 timeframe. However, this is orthogonal of the discussions of transitioning from JSP-based Web UI to client-side JavaScript Web UI. The architecture of the new Web UI allows easy migration to any JSON-based APIs whenever they land in the trunk. Thanks, Haohui On Mon, Oct 28, 2013 at 5:13 PM, Alejandro Abdelnur <t...@cloudera.com>wrote: > Isn't using JMX to expose JSON for the web UI misusing JMX? > > I would think a more appropriate approach would be having /JMX for > monitoring integration and a /JSON end point for the UI data. > > Thanks. > > > On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai <h...@hortonworks.com> wrote: > > > Alejandro, > > > > If I understand correctly, that is the exact approach that the new web UI > > is taking. The new web UI takes the output from JMX and renders them as > > HTML at the client side. > > > > ~Haohui > > > > > > On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <t...@cloudera.com > > >wrote: > > > > > Haohui, > > > > > > If you have NN and DNs producing JSON instead HTML, then you can build > JS > > > based web UIs. Take for example Oozie, Oozie produces JSON, it has a > > built > > > in JS web ui that consumes JSON and Hue has built an external web UI > that > > > also consumes JSON. In the case of Hue UI, Oozie didn't have to change > > > anything to get that UI and improvements on the Hue UI don't require > > > changes in Oozie unless it is to produce additional information. > > > > > > hope this clarifies. > > > > > > Thx > > > > > > > > > On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <h...@hortonworks.com> > > wrote: > > > > > > > Echo my comments on HDFS-5402: > > > > > > > > bq. If we're going to remove the old web UI, I think the new web UI > has > > > > to have the same level of unit testing. We shouldn't go backwards in > > > > terms of unit testing. > > > > > > > > I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper / > > > > TestClusterJspHelper. It seems to me that we can merge these tests > with > > > the > > > > unit tests on JMX. > > > > > > > > bq. If we are going to > > > > remove this capability, we need to add some other command-line tools > > > > to get the same functionality. These tools could use REST if we have > > > > that, or JMX, but they need to exist before we can consider removing > > > > the old UI. > > > > > > > > This is a good point. Since all information are available through > JMX, > > > the > > > > easiest way to approach it is to write some scripts using Node.js. > The > > > > architecture of the new Web UIs is ready for this. > > > > > > > > > > > > On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur < > t...@cloudera.com > > > > >wrote: > > > > > > > > > Producing JSON would be great. Agree with Colin that we should > leave > > > for > > > > > now the current JSP based web ui. > > > > > > > > > > thx > > > > > > > > > > > > > > > On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe < > > cmcc...@alumni.cmu.edu > > > > > >wrote: > > > > > > > > > > > This is a really interesting project, Haohui. I think it will > make > > > > > > our web UI much nicer. > > > > > > > > > > > > I have a few concerns about removing the old web UI, however: > > > > > > > > > > > > * If we're going to remove the old web UI, I think the new web UI > > has > > > > > > to have the same level of unit testing. We shouldn't go > backwards > > in > > > > > > terms of unit testing. > > > > > > > > > > > > * Most of the deployments of elinks and links out there don't > > support > > > > > > Javascript. This is just a reality of life when using CentOS 5 > or > > 6, > > > > > > which many users are still using. I have used "links" to > diagnose > > > > > > problems through the web UI in the past, in systems where access > to > > > > > > the cluster was available only through telnet. If we are going > to > > > > > > remove this capability, we need to add some other command-line > > tools > > > > > > to get the same functionality. These tools could use REST if we > > have > > > > > > that, or JMX, but they need to exist before we can consider > > removing > > > > > > the old UI. > > > > > > > > > > > > best, > > > > > > Colin > > > > > > > > > > > > On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai < > h...@hortonworks.com> > > > > > wrote: > > > > > > > Thanks for the reply, Luke. Here I just echo my response from > the > > > > jira: > > > > > > > > > > > > > > bq. this client-side js only approach, which is less secure > than > > a > > > > > > > progressively enhanced hybrid approach used by YARN. The recent > > > gmail > > > > > > > XSS fiasco highlights the issue. > > > > > > > > > > > > > > I'm presenting an informal security analysis to compare the > > > security > > > > of > > > > > > the > > > > > > > old and the new web UIs. > > > > > > > > > > > > > > An attacker launches an XSS attack by injecting malicious code > > > which > > > > > are > > > > > > > usually HTML or JavaScript fragments into the web page, so that > > the > > > > > > > malicious code can have the same privileges of the web page. > > > > > > > > > > > > > > First, in the scope of XSS attacks, note that the threat models > > of > > > > > > > launching XSS attacks on Internet sites Gmail/Linkedin and the > > one > > > of > > > > > the > > > > > > > Hadoop UIs are different. They have fundamental different sets > of > > > > > > external > > > > > > > inputs that the attackers have control to. Internet sites have > > > little > > > > > > > control of these inputs. In the case of Gmail / Linkedin, an > > attack > > > > can > > > > > > > send you a crafted e-mail, or put malicious description in his > / > > > > > > > her Linkedin profile. The sets of external inputs are > > *restricted* > > > in > > > > > > > Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs. The > > > > > > > attacker has to launch a XSS attack by: > > > > > > > > > > > > > > * Compromise the jars so that the output of JMX / WebHDFS have > > the > > > > > > > malicious code. > > > > > > > * Replace the web UIs completely to include the malicious code. > > > > > > > > > > > > > > In either case *the attacker has to compromise the hadoop core > or > > > the > > > > > > > namenode*. That means the new web UIs are at least as secure as > > the > > > > > > hadoop > > > > > > > core, and the namenode machine. > > > > > > > > > > > > > > Second, I argue that using client-side templates are more > secure > > > than > > > > > the > > > > > > > current JSP-based server-side templates. To defend against XSS > > > > > > > attacks, both techniques have to filter the external inputs at > > > > *every* > > > > > > > possible execution paths. Several facts much be taken into > > > > > > > plays when evaluating the security of both approaches in > > real-world > > > > > > > environments: > > > > > > > > > > > > > > * The JavaScript libraries used in the new web UIs have > survived > > in > > > > > > > extremely large-scale production tests. jQuery is used by > Google > > > and > > > > > > > Microsoft, bootstrap is used by Twitter, and dust.js is used > by > > > > > > Linkedin. > > > > > > > All libraries survived from hundreds of thousands of > > > > > > > attack attempts on a daily basis. I agree that the libraries > > might > > > > > still > > > > > > > be imperfect, but there's no way that we can test the JSP web > > > > > > > UIs to achieve the same level of assurances given the amount > of > > > > > > resources > > > > > > > the community has. > > > > > > > * Client-side templates consolidate all filtering logic in one > > > > central > > > > > > > place. Recall that the goal is to filter all external inputs at > > > every > > > > > > > execution paths, this is a much more systematic approach > > compared > > > to > > > > > the > > > > > > > server-side templates we have today. It is difficult (if not > > > > > > > impossible) to do it in a JSP/ASP/PHP application, since such > > > > > filtering > > > > > > > can be only achieved via ad-hoc approaches ([1] shows some > > > > > > > empirical data). Also, HDFS-4901 recently describes a XSS > > > > > vulnerability > > > > > > in > > > > > > > browseDirectory.jsp. > > > > > > > > > > > > > > bq. You'd require proper SSL (not self signed) setup to avoid > JS > > > > > > > injection > > > > > > > > > > > > > > Commodity browsers enforce Same-Origin Policy to defend against > > > code > > > > > > > injections. It has nothing to do with what kinds of SSL > > > certificates > > > > > > > you hold. > > > > > > > > > > > > > > bq. I also have concerns that we commit these changes without > > > > matching > > > > > > > unit tests > > > > > > > > > > > > > > The JavaScript code can be automatically tested. The same code > > can > > > be > > > > > run > > > > > > > by node.js and the test can compared with pre-defined > > > > > > > results. It is also possible to write an adapter to use Rhino > to > > > > > > accomplish > > > > > > > the same task. We can discuss how to integrate them into > > > > > > > the maven test routines in a different thread. > > > > > > > > > > > > > > bq. Client side rendering completely breaks the workflows for > ops > > > who > > > > > > rely > > > > > > > on text based terminal/emacs/vim browsers (no js support) to > > > > > > > monitor component UI. > > > > > > > > > > > > > > links / elinks (http://elinks.or.cz/) are text-based web > > browsers > > > > that > > > > > > > support JavaScript. > > > > > > > > > > > > > > bq. The priority/requirements for UI in core Hadoop should be > > > > security > > > > > > and > > > > > > > correctness, which client side templating cannot address > properly > > > > > > > so far. > > > > > > > > > > > > > > I agree that we should focus on security and correctness. The > > > > > paragraphs > > > > > > > above explain that how the architecture of the new UIs > > > > > > > makes the UIs more secure in real-world settings compared to > the > > UI > > > > we > > > > > > have > > > > > > > today. > > > > > > > > > > > > > > References: > > > > > > > > > > > > > > 1. A. Yip et al. Improving Application Security with Data Flow > > > > > > Assertions. > > > > > > > In SOSP'2009. > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <l...@apache.org> > > wrote: > > > > > > > > > > > > > >> Echoing my comments on HDFS-3555: > > > > > > >> > > > > > > >> I have concerns with this client-side js only approach, which > is > > > > less > > > > > > >> secure than a progressively enhanced hybrid approach used by > > YARN. > > > > The > > > > > > >> recent gmail XSS fiasco highlights the issue. I also have > > concerns > > > > > that > > > > > > we > > > > > > >> commit these changes without matching unit tests – the fact > you > > > > cannot > > > > > > >> effectively unit test these changes should tell you something > > > about > > > > > this > > > > > > >> approach. > > > > > > >> > > > > > > >> *Requiring* JS means that an admin cannot turn off js to > > > (partially) > > > > > use > > > > > > >> core Hadoop UI. You'd *require* proper SSL (not self signed) > > setup > > > > to > > > > > > avoid > > > > > > >> JS injection, even if security of js libraries used is > perfect, > > > > which > > > > > I > > > > > > >> doubt (search gmail/linkedin XSS). Client side rendering > > > completely > > > > > > breaks > > > > > > >> the workflows for ops who rely on text based > terminal/emacs/vim > > > > > browsers > > > > > > >> (no js support) to monitor component UI. > > > > > > >> > > > > > > >> IMO, JS-only rendering belongs to social networking sites > and/or > > > > SaaS > > > > > > >> front-ends, where full time UI/security specialists babysits > UI > > > > > > changes. I > > > > > > >> think eventually most users will use a self servicing UI in a > > SaaS > > > > > > >> front-end that uses REST/JMX API to get data from back-end > > > > components, > > > > > > >> besides their own app master/service UI. The > > priority/requirements > > > > for > > > > > > UI > > > > > > >> in core Hadoop should be security and correctness, which > client > > > side > > > > > > >> templating cannot address properly so far. > > > > > > >> > > > > > > >> > > > > > > >> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai < > > h...@hortonworks.com > > > > > > > > > > wrote: > > > > > > >> > > > > > > >> > Hi all, > > > > > > >> > > > > > > > >> > > > > > > > >> > Jing Zhao and I recently have reimplemented the JSP-based > web > > > UIs > > > > in > > > > > > >> HTML 5 > > > > > > >> > applications (HDFS-5333). Based on our prelimanary testing > > > results > > > > > we > > > > > > >> > believe thst the new web UIs of the namenodes and the > datanode > > > are > > > > > > ready > > > > > > >> > for everyday uses. > > > > > > >> > > > > > > > >> > You're more than welcome to try it out on trunk by visiting > > > > http:// > > > > > > >> > <namenode>/dfshealth.html > > > > > > >> > > > > > > > >> > There are a number of benefits from this transition. From a > > > > > > developer's > > > > > > >> > prospective, the most notable one is *maintainability*: > > > > > > >> > > > > > > > >> > (1) The abstractions between the UI and the core server are > > > > > > well-defined, > > > > > > >> > decoupling the UI and the core hadoop servers. > > > > > > >> > > > > > > > >> > (2) It allows us to deprecate the logic in the JSP pages. > The > > > old > > > > > web > > > > > > UIs > > > > > > >> > have to duplicate the logic in the JSPs. The logic is often > > > > > > out-of-dated > > > > > > >> > and not well-tested, which leads to broken pages and > security > > > > > > >> > vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308, > > HDFS-5317 > > > > and > > > > > > >> > HDFS-4901). The architecture of the new UIs prevent these > bugs > > > at > > > > > the > > > > > > >> very > > > > > > >> > beginning. > > > > > > >> > > > > > > > >> > > > > > > > >> > I propose that deprecate the old, JSP-based web UIs in 2.3. > I > > > > opened > > > > > > >> > HDFS-5402 to track the relevant discussions. > > > > > > >> > > > > > > > >> > Your feedbacks are highly appreciated. > > > > > > >> > > > > > > > >> > > > > > > > >> > Sincerely, > > > > > > >> > > > > > > > >> > Haohui > > > > > > >> > > > > > > > >> > -- > > > > > > >> > CONFIDENTIALITY NOTICE > > > > > > >> > NOTICE: This message is intended for the use of the > individual > > > or > > > > > > entity > > > > > > >> to > > > > > > >> > which it is addressed and may contain information that is > > > > > > confidential, > > > > > > >> > privileged and exempt from disclosure under applicable law. > If > > > the > > > > > > reader > > > > > > >> > of this message is not the intended recipient, you are > hereby > > > > > notified > > > > > > >> that > > > > > > >> > any printing, copying, dissemination, distribution, > disclosure > > > or > > > > > > >> > forwarding of this communication is strictly prohibited. If > > you > > > > have > > > > > > >> > received this communication in error, please contact the > > sender > > > > > > >> immediately > > > > > > >> > and delete it from your system. Thank You. > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > -- > > > > > > > CONFIDENTIALITY NOTICE > > > > > > > NOTICE: This message is intended for the use of the individual > or > > > > > entity > > > > > > to > > > > > > > which it is addressed and may contain information that is > > > > confidential, > > > > > > > privileged and exempt from disclosure under applicable law. If > > the > > > > > reader > > > > > > > of this message is not the intended recipient, you are hereby > > > > notified > > > > > > that > > > > > > > any printing, copying, dissemination, distribution, disclosure > or > > > > > > > forwarding of this communication is strictly prohibited. If you > > > have > > > > > > > received this communication in error, please contact the sender > > > > > > immediately > > > > > > > and delete it from your system. Thank You. > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Alejandro > > > > > > > > > > > > > -- > > > > CONFIDENTIALITY NOTICE > > > > NOTICE: This message is intended for the use of the individual or > > entity > > > to > > > > which it is addressed and may contain information that is > confidential, > > > > privileged and exempt from disclosure under applicable law. If the > > reader > > > > of this message is not the intended recipient, you are hereby > notified > > > that > > > > any printing, copying, dissemination, distribution, disclosure or > > > > forwarding of this communication is strictly prohibited. If you have > > > > received this communication in error, please contact the sender > > > immediately > > > > and delete it from your system. Thank You. > > > > > > > > > > > > > > > > -- > > > Alejandro > > > > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > > > > > -- > Alejandro > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
