Hi, There's no better way to start a project than...github it :-) Create a new project, clone it and do dzieła! (= go ahead in Polish).
Jacek On 27 Aug 2016 10:42 a.m., "Sivakumaran S" <siva.kuma...@icloud.com> wrote: > I would love to participate in developing a dashboard of some sort in lieu > (or at least complement it) of Spark UI . > > Regards, > > Sivakumaran S > > On 27 Aug 2016 9:34 a.m., Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > > Are we actually looking for a eal time dashboard of some sort for Spark UI > interface? > > After all one can think a real time dashboard can do this! > > HTH > > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 26 August 2016 at 23:38, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > > Thanks Jacek, > > I will have a look. I think it is long overdue. > > I mean we try to micro batch and stream everything below seconds but when > it comes to help monitor basics we are still miles behind :( > > Cheers, > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 26 August 2016 at 23:21, Jacek Laskowski <ja...@japila.pl> wrote: > > Hi Mich, > > I don't think so. There is support for a UI page refresh but I haven't > seen it in use. > > See StreamingPage [1] where it schedules refresh every 5 secs, i.e. > Some(5000). In SparkUIUtils.headerSparkPage [2] there is > refreshInterval but it's not used in any place in Spark. > > Time to fill an JIRA issue? > > What about REST API and httpie updating regularly [3]? Perhaps Metrics > with ConsoleSink [4]? > > [1] https://github.com/apache/spark/blob/master/streaming/src/ma > in/scala/org/apache/spark/streaming/ui/StreamingPage.scala#L158 > [2] https://github.com/apache/spark/blob/master/core/src/main/sc > ala/org/apache/spark/ui/UIUtils.scala#L202 > [3] http://spark.apache.org/docs/latest/monitoring.html#rest-api > [4] http://spark.apache.org/docs/latest/monitoring.html#metrics > > Pozdrawiam, > Jacek Laskowski > ---- > https://medium.com/@jaceklaskowski/ > Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark > Follow me at https://twitter.com/jaceklaskowski > > > On Thu, Aug 25, 2016 at 11:55 AM, Mich Talebzadeh > <mich.talebza...@gmail.com> wrote: > > Hi, > > > > This may be already there. > > > > A spark job opens up a UI on port specified by --conf > "spark.ui.port=${SP}" > > that defaults to 4040. > > > > However, on UI one needs to refresh the page to see the progress. > > > > Can this be polled so it is refreshed automatically > > > > Thanks > > > > > > Dr Mich Talebzadeh > > > > > > > > LinkedIn > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJ > d6zP6AcPCCdOABUrV8Pw > > > > > > > > http://talebzadehmich.wordpress.com > > > > > > Disclaimer: Use it at your own risk. Any and all responsibility for any > > loss, damage or destruction of data or any other property which may arise > > from relying on this email's technical content is explicitly disclaimed. > The > > author will in no case be liable for any monetary damages arising from > such > > loss, damage or destruction. > > > > > > > > >
