bq. Besides that obvious problem, how does one simulate HDFS browsing (especially going through the web auth--which is the key part!) via JMX or REST? Will it provide the guarantees that I know that users who are using browsers are functional?
It can be tested in the same way that the UI is tested today. The test needs to make sure JMX / webhdfs have authenticate the connections, which is well covered today. In fact, parts of the JSP UI can take a separate code path for authentication (e.g. HDFS-1017). One motivation of the new UI work is to unify these code paths and to say goodbye to these bugs from an architectural prospective. bq. Are we declaring that UIs are defacto non-stable? UI has been declared non-stable explicitly. Just to quote Hadoop Compatibility Policy ( http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html ): Web UI: Web UI, particularly the content and layout of web pages, changes could potentially interfere with attempts to screen scrape the web pages for information. Policy: Web pages are not meant to be scraped and hence incompatible changes to them are allowed at any time. Users are expected to use REST APIs to get any information. By the way, maybe I've missed some of your points, I'm somewhat confused by your messages. I agree that it might be good to start the discussion of 3.x, the issue of removing the old UI from branch-2, however, is orthogonal. UI is non-stable according to Hadoop Compatibility Policy, and the new UI has been the default UI since 2.3. Even for people that have workflows based upon scraping the Web UIs (I doubt there are any), they should be scraping the new UI instead of the old one by now. I failed to understand the reasoning behind bundling this issue with the 3.x release -- though some work that has dramatic changes / refactoring is not immediately merged to branch-2, these changes are actually quite good and should be merged into branch-2 when it is stabilized. IMO the scripting work you've done is a good example -- personally I really appreciate the work and I think it should be merged into branch-2 to benefit mainstream users when it's appropriate. Merging the work into branch-2 and starting the discussion of 3.x release, should be independent. ~Haohui On Wed, Oct 1, 2014 at 5:25 PM, Allen Wittenauer <a...@altiscale.com> wrote: > > > Webui equivalent data has been exposed as jmx http APIs for a long time > > Not true. HDFS-5342 was allegedly committed to branch-2 less than > a year ago. > > Besides that obvious problem, how does one simulate HDFS browsing > (especially going through the web auth--which is the key part!) via JMX or > REST? Will it provide the guarantees that I know that users who are using > browsers are functional? > > This is also an interesting precedent: Are we declaring that UIs > are defacto non-stable? Does this mean we can break the output of, e.g., > count or ls or whatever because they don't have stabilities associated with > them? What line is to be drawn here? IMO, difficulty in moving code is > NOT a sufficient reason to throw out the compatibility guarantees. It is > only an indicator that we as a community are taking too long getting out > releases. > > If we do this change, then it's pretty much open season on all of > other UI-incompatible changes sitting in trunk (and there are a bunch….). > Are we really ready to open the floodgates? > > > On Oct 1, 2014, at 4:05 PM, Suresh Srinivas <sur...@hortonworks.com> > wrote: > > > Webui equivalent data has been exposed as jmx http APIs for a long time > (I think from 0.20 release, almost 4 years ago). We have made many jsp > changes that we have made that should have broken these applications many > times. I believe this is a frivolous objection for a functionality that as > a community we have never committed to keeping stable or backward > compatible. > > > > Again I think given there is a clear and easier path for such uses to > move to, I am +1 on this change. > > > > Sent from phone > > > >> On Oct 1, 2014, at 3:44 PM, Allen Wittenauer <a...@altiscale.com> wrote: > >> > >> > >> OK, then consider this an official objection. Yes, people do have > workflows based upon scraping the Web UIs, especially workflows that > simulate what interfaces users actually use. > >> > >>> On Oct 1, 2014, at 3:34 PM, Haohui Mai <h...@hortonworks.com> wrote: > >>> > >>> Hi, > >>> > >>> Thanks for the discussions. If there is no objection, I plan to merge > the > >>> related jiras from trunk to branch-2 in the coming days. > >>> > >>> ~Haohui > >>> > >>> On Wed, Oct 1, 2014 at 2:35 PM, Suresh Srinivas < > sur...@hortonworks.com> > >>> wrote: > >>> > >>>> Sorry for responding very late. I am +1 on removing the JSP UI in > branch-2 > >>>> for the following reasons: > >>>> 1. The new web UI added has all the functionality of JSP UI. > >>>> 2. For folks who might have used web UI to scrape the HDFS status, we > have > >>>> build a http JMX interface a long time ago. > >>>> > >>>> Given we do not guarantee the web UI as a stable and backward > compatible > >>>> interface, removing this should be okay. > >>>> > >>>> > >>>> > >>>>> On Thu, Sep 11, 2014 at 3:29 PM, Haohui Mai <h...@hortonworks.com> > wrote: > >>>>> > >>>>> Hi all, > >>>>> > >>>>> In HDFS, the code that relates to webhdfs and UIs has diverged quite > a > >>>> bit > >>>>> between branch-2 and trunk. This complicates the processes of > maintaining > >>>>> webhdfs in several jiras (e.g., HDFS-5946) > >>>>> > >>>>> Most of the divergences come from removing the JSP UI and the related > >>>> clean > >>>>> ups that are committed to trunk. > >>>>> > >>>>> Though I myself have to take the blame of creating these divergences > -- I > >>>>> wonder, is it acceptable to resolve these divergences, by committing > >>>> these > >>>>> patches that remove JSP UI to branch-2? This would save us the pains > of > >>>>> providing two patches when fixing issues in webhdfs. > >>>>> > >>>>> Note that the new HTML5-based UI is fairly mature, as it has been > around > >>>>> since 2.2, and it has been the default UI since 2.3. Compatibility, > >>>>> however, might be a concern -- according to the compatibility guide, > >>>> Hadoop > >>>>> claims no compatibility for web UIs, thus we might be able to get > away > >>>> with > >>>>> that. > >>>>> > >>>>> Thoughts? > >>>>> > >>>>> Regards, > >>>>> Haohui > >>>>> > >>>>> -- > >>>>> CONFIDENTIALITY NOTICE > >>>>> NOTICE: This message is intended for the use of the individual or > entity > >>>> to > >>>>> which it is addressed and may contain information that is > confidential, > >>>>> privileged and exempt from disclosure under applicable law. If the > reader > >>>>> of this message is not the intended recipient, you are hereby > notified > >>>> that > >>>>> any printing, copying, dissemination, distribution, disclosure or > >>>>> forwarding of this communication is strictly prohibited. If you have > >>>>> received this communication in error, please contact the sender > >>>> immediately > >>>>> and delete it from your system. Thank You. > >>>> > >>>> > >>>> > >>>> -- > >>>> http://hortonworks.com/download/ > >>>> > >>>> -- > >>>> CONFIDENTIALITY NOTICE > >>>> NOTICE: This message is intended for the use of the individual or > entity to > >>>> which it is addressed and may contain information that is > confidential, > >>>> privileged and exempt from disclosure under applicable law. If the > reader > >>>> of this message is not the intended recipient, you are hereby > notified that > >>>> any printing, copying, dissemination, distribution, disclosure or > >>>> forwarding of this communication is strictly prohibited. If you have > >>>> received this communication in error, please contact the sender > immediately > >>>> and delete it from your system. Thank You. > >>> > >>> -- > >>> CONFIDENTIALITY NOTICE > >>> NOTICE: This message is intended for the use of the individual or > entity to > >>> which it is addressed and may contain information that is confidential, > >>> privileged and exempt from disclosure under applicable law. If the > reader > >>> of this message is not the intended recipient, you are hereby notified > that > >>> any printing, copying, dissemination, distribution, disclosure or > >>> forwarding of this communication is strictly prohibited. If you have > >>> received this communication in error, please contact the sender > immediately > >>> and delete it from your system. Thank You. > >> > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
