Re: Spark UI Storage Memory

any suggestion please. Thanks Amit On Fri, Dec 4, 2020 at 2:27 PM Amit Sharma wrote: > Is there any memory leak in spark 2.3.3 version as mentioned in below > Jira. > https://issues.apache.org/jira/browse/SPARK-29055. > > Please let me know how to solve it. > > Thanks > Amit > > On Fri, Dec 4,

RE: Spark UI Storage Memory

unsubsribe

Re: Spark UI Storage Memory

Is there any memory leak in spark 2.3.3 version as mentioned in below Jira. https://issues.apache.org/jira/browse/SPARK-29055. Please let me know how to solve it. Thanks Amit On Fri, Dec 4, 2020 at 1:55 PM Amit Sharma wrote: > Can someone help me on this please. > > > Thanks > Amit > > On Wed,

Re: Spark UI Storage Memory

Can someone help me on this please. Thanks Amit On Wed, Dec 2, 2020 at 11:52 AM Amit Sharma wrote: > Hi , I have a spark streaming job. When I am checking the Excetors tab , > there is a Storage Memory column. It displays used memory /total memory. > What is used memory. Is it memory in use

Re: Spark UI

Thanks Xiao for the info.  I was looking for this, too.  This page wasn't linked from anywhere on the main doc page (Overview) or any of the pull-down menus.  Someone should remind the doc team to update the table of contents on the Overview page. -- ND On 7/19/20 10:30 PM, Xiao Li wrote: htt

Re: Spark UI

https://www.youtube.com/watch?v=YgQgJceojJY (Xiao's video ) On Mon, Jul 20, 2020 at 8:03 AM Xiao Li wrote: > https://spark.apache.org/docs/3.0.0/web-ui.html is the official doc > for Spark UI. > > Xiao > > On Sun, Jul 19, 2020 at 1:38 PM venkatadevarapu > wrote: > >> Hi, >> >> I'm looking

Re: Spark UI

https://spark.apache.org/docs/3.0.0/web-ui.html is the official doc for Spark UI. Xiao On Sun, Jul 19, 2020 at 1:38 PM venkatadevarapu wrote: > Hi, > > I'm looking for a tutorial/video/material which explains the content of > various tabes in SPARK WEB UI. > Can some one direct me with the rele

Re: Spark UI History server on Kubernetes

t; following properties while submitting the spark job > > spark.eventLog.enabled true > > spark.eventLog.dir > > > > Thanks and Regards, > > Abhishek > > > > *From:* Battini Lakshman > *Sent:* Wednesday, January 23, 2019 1:55 PM > *To:* Rao, Abhishek (Nokia - IN/Bangalo

RE: Spark UI History server on Kubernetes

spark.eventLog.dir Thanks and Regards, Abhishek From: Battini Lakshman Sent: Wednesday, January 23, 2019 1:55 PM To: Rao, Abhishek (Nokia - IN/Bangalore) Subject: Re: Spark UI History server on Kubernetes HI Abhishek, Thank you for your response. Could you please let me know the properties you configured

RE: Spark UI History server on Kubernetes

Hi, We’ve setup spark-history service (based on spark 2.4) on K8S. UI works perfectly fine when running on NodePort. We’re facing some issues when on ingress. Please let us know what kind of inputs do you need? Thanks and Regards, Abhishek From: Battini Lakshman Sent: Tuesday, January 22, 201

Re: [Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs

Done: https://issues.apache.org/jira/browse/SPARK-25837 On Thu, Oct 25, 2018 at 10:21 AM Marcelo Vanzin wrote: > Ah that makes more sense. Could you file a bug with that information > so we don't lose track of this? > > Thanks > On Wed, Oct 24, 2018 at 6:13 PM Patrick Brown > wrote: > > > > On

Re: [Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs

Ah that makes more sense. Could you file a bug with that information so we don't lose track of this? Thanks On Wed, Oct 24, 2018 at 6:13 PM Patrick Brown wrote: > > On my production application I am running ~200 jobs at once, but continue to > submit jobs in this manner for sometimes ~1 hour. >

Re: [Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs

When you say many jobs at once, what ballpark are you talking about? The code in 2.3+ does try to keep data about all running jobs and stages regardless of the limit. If you're running into issues because of that we may have to look again at whether that's the right thing to do. On Tue, Oct 23, 20

Re: [Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs

I believe I may be able to reproduce this now, it seems like it may be something to do with many jobs at once: Spark 2.3.1 > spark-shell --conf spark.ui.retainedJobs=1 scala> import scala.concurrent._ scala> import scala.concurrent.ExecutionContext.Implicits.global scala> for (i <- 0 until 5

Re: [Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs

Just tried on 2.3.2 and worked fine for me. UI had a single job and a single stage (+ the tasks related to that single stage), same thing in memory (checked with jvisualvm). On Sat, Oct 20, 2018 at 6:45 PM Marcelo Vanzin wrote: > > On Tue, Oct 16, 2018 at 9:34 AM Patrick Brown > wrote: > > I rec

Re: [Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs

On Tue, Oct 16, 2018 at 9:34 AM Patrick Brown wrote: > I recently upgraded to spark 2.3.1 I have had these same settings in my spark > submit script, which worked on 2.0.2, and according to the documentation > appear to not have changed: > > spark.ui.retainedTasks=1 > spark.ui.retainedStages=1 >

Re: [Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs

I have the same problem when I upgrade my application from Spark 2.2.1 to Spark 2.3.2 and run in Yarn client mode. Also I noticed that in my Spark driver,  org.apache.spark.status.TaskDataWrapper could take up more than 2G of memory. Shing On Tuesday, 16 October 2018, 17:34:02 GMT+1, Patr

Re: Spark UI Source Code

imestamp? >> >> Only if the REST API has that feature, don't remember off the top of my >> head. >> >> >> -- >> Marcelo >> >> --------- >> To unsubscribe e-mail: [hidden email] >>

Re: Spark UI Source Code

On Mon, May 7, 2018 at 1:44 AM, Anshi Shrivastava wrote: > I've found a KVStore wrapper which stores all the metrics in a LevelDb > store. This KVStore wrapper is available as a spark-dependency but we cannot > access the metrics directly from spark since they are all private. I'm not sure what i

Re: Spark UI crashes on Large Workloads

Hi Riccardo, Thanks for your suggestions. The thing is that my Spark UI is the one thing that is crashing - and not the app. In fact the app does end up completing successfully. That's why I'm a bit confused by this issue? I'll still try out some of your suggestions. Thanks and Regards, Saatvik Sh

Re: Spark UI crashes on Large Workloads

The reason you get connection refused when connecting to the application UI (port 4040) is because you app gets stopped thus the application UI stops as well. To inspect your executors logs after the fact you might find useful the Spark History server

Re: Spark UI crashes on Large Workloads

Hi Riccardo, Yes, Thanks for suggesting I do that. [Stage 1:==> (12750 + 40) / 15000]17/07/18 13:22:28 ERROR org.apache.spark.scheduler.LiveListenerBus: Dropping SparkListenerEvent because no remaining room in event queue. This likely means one of the

Re: Spark UI crashes on Large Workloads

Hi, can you share more details. do you have any exceptions from the driver? or executors? best, On Jul 18, 2017 02:49, "saatvikshah1994" wrote: > Hi, > > I have a pyspark App which when provided a huge amount of data as input > throws the error explained here sometimes: > https://stackoverflow

Re: Spark UI shows Jobs are processing, but the files are already written to S3

Could I be experiencing the same thing? https://www.dropbox.com/s/egtj1056qeudswj/sparkwut.png?dl=0 On Wed, Nov 16, 2016 at 10:37 AM, Shreya Agarwal wrote: > I think that is a bug. I have seen that a lot especially with long running > jobs where Spark skips a lot of stages because it has pre-co

RE: Spark UI not coming up in EMR

Any clue on this. Jobs are running fine , But not able to access Spark UI in EMR -yarn. Where I can see statistics like , No of events /per sec and rows processed for streaming in log files (If UI is not working) -Saurabh From: Saurabh Malviya (samalviy) Sent: Monday, January 09, 2017 10:59

RE: Spark UI shows Jobs are processing, but the files are already written to S3

I think that is a bug. I have seen that a lot especially with long running jobs where Spark skips a lot of stages because it has pre-computed results. And some of these are never marked as completed, even though in reality they are. I figured this out because I was using the interactive shell (s

Re: Spark UI error spark 2.0.1 hadoop 2.6

I'm able to fix.. added servlet 3.0 to classpath -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-UI-error-spark-2-0-1-hadoop-2-6-tp27970p27971.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --

Re: Spark UI shows finished when job had an error

Spark GUI runs by default on 4040 and if a job crashes (assuming you meant there was an issue with spark-submit), then the GUI will disconnect. GUI is not there for diagnostics as it reports on statistics. My inclination would be to look at the YARN log files assuming you are using YARN as your re

Re: Spark UI shows finished when job had an error

Hi, Can you please see the query plan (in case you are using a query)? There is a very high chance that the query was broken into multiple steps and only a subsequent step failed. Regards, Gourav Sengupta On Fri, Jun 17, 2016 at 2:49 PM, Sumona Routh wrote: > Hi there, > Our Spark job had an

Re: Spark UI shows finished when job had an error

Hi, How do you access Cassandra? Could that connector not have sent a SparkListenerEvent to inform about failure? Jacek On 17 Jun 2016 3:50 p.m., "Sumona Routh" wrote: > Hi there, > Our Spark job had an error (specifically the Cassandra table definition > did not match what was in Cassandra), w

Re: Spark UI doesn't give visibility on which stage job actually failed (due to lazy eval nature)

I think it does because user doesn't exactly see their application logic and flow as spark internal does. Off course we follow general guidelines for performance but we shouldn't care really how exactly spark decide to execute DAG. Spark scheduler or core can keep changing over time to optimize it.

Re: Spark UI doesn't give visibility on which stage job actually failed (due to lazy eval nature)

But when you talk about optimizing the DAG, it really doesn't make sense to also talk about transformation steps as separate entities. The DAGScheduler knows about Jobs, Stages, TaskSets and Tasks. The TaskScheduler knows about TaskSets ad Tasks. Neither of them understands the transformation st

Re: Spark UI doesn't give visibility on which stage job actually failed (due to lazy eval nature)

Hi Mark, I might have said stage instead of step in my last statement "UI just says Collect failed but in fact it could be any stage in that lazy chain of evaluation." Anyways even you agree that this visibility of underlaying steps wont't be available. which does pose difficulties in terms of tr

Re: Spark UI doesn't give visibility on which stage job actually failed (due to lazy eval nature)

You appear to be misunderstanding the nature of a Stage. Individual transformation steps such as `map` do not define the boundaries of Stages. Rather, a sequence of transformations in which there is only a NarrowDependency between each of the transformations will be pipelined into a single Stage.

Re: Spark UI Completed Jobs

Thanks Mark and Jeff On Wed, Mar 16, 2016 at 7:11 AM, Mark Hamstra wrote: > Looks to me like the one remaining Stage would execute 19788 Task if all > of those Tasks succeeded on the first try; but because of retries, 19841 > Tasks were actually executed. Meanwhile, there were 41405 Tasks in th

Re: Spark UI Completed Jobs

Looks to me like the one remaining Stage would execute 19788 Task if all of those Tasks succeeded on the first try; but because of retries, 19841 Tasks were actually executed. Meanwhile, there were 41405 Tasks in the the 163 Stages that were skipped. I think -- but the Spark UI's accounting may n

Re: Spark UI Completed Jobs

Okay, so out of 164 stages, is 163 are skipped. And how 41405 tasks are skipped if the total is only 19788. On Wed, Mar 16, 2016 at 6:31 AM, Mark Hamstra wrote: > It's not just if the RDD is explicitly cached, but also if the map outputs > for stages have been materialized into shuffle files and

Re: Spark UI Completed Jobs

It's not just if the RDD is explicitly cached, but also if the map outputs for stages have been materialized into shuffle files and are still accessible through the map output tracker. Because of that, explicitly caching RDD actions often gains you little or nothing, since even without a call to c

Re: Spark UI Completed Jobs

If RDD is cached, this RDD is only computed once and the stages for computing this RDD in the following jobs are skipped. On Wed, Mar 16, 2016 at 8:14 AM, Prabhu Joseph wrote: > Hi All, > > > Spark UI Completed Jobs section shows below information, what is the > skipped value shown for Stages a

Re: Spark UI standalone "crashes" after an application finishes

Hi Teng, I was not asking the question, I was responding in terms of what to expect from SPARK UI in terms of how you start using SPARK application. Thanks and Regards, Gourav On Tue, Mar 1, 2016 at 8:30 PM, Teng Qiu wrote: > as Gourav said, the application UI on port 4040 will no more availab

Re: Spark UI standalone "crashes" after an application finishes

as Gourav said, the application UI on port 4040 will no more available after your spark app finished. you should go to spark master's UI (port 8080), and take a look "completed applications"... refer to doc: http://spark.apache.org/docs/latest/monitoring.html read the first "note that" :) 2016-03

Re: Spark UI standalone "crashes" after an application finishes

Hi, in case you are submitting your SPARK jobs then the UI is only available when the job is running. Else if you are starting a SPARK cluster in standalone mode or HADOOP or etc, then the SPARK UI remains alive. The other way to keep the SPARK UI alive is to use the Jupyter notebook for Python

Re: Spark UI standalone "crashes" after an application finishes

KnHOPJxIX5_n_zXe51k8z9hVuw4svP6dqWF0JrjabAa&wd=&eqid=be50a4160f49000256d50b7b>, > and so you still need to set a big java heap for master. > > > > -- 原始邮件 -- > *发件人:* "Shixiong(Ryan) Zhu";; > *发送时间:* 2016年3月1日(星期二)

RE: Spark UI standalone "crashes" after an application finishes

bruary 29, 2016 4:03 PM To: Sumona Routh Cc: user@spark.apache.org Subject: Re: Spark UI standalone "crashes" after an application finishes Do you mean you cannot access Master UI after your application completes? Could you check the master log? On Mon, Feb 29, 2016 at 3:48

Re: Spark UI standalone "crashes" after an application finishes

Do you mean you cannot access Master UI after your application completes? Could you check the master log? On Mon, Feb 29, 2016 at 3:48 PM, Sumona Routh wrote: > Hi there, > I've been doing some performance tuning of our Spark application, which is > using Spark 1.2.1 standalone. I have been usin

Re: Spark UI documentaton needed

Hi Ajay, Feel free to open a JIRA with the fields that you think are missing and what kind of documentation you wish to see. It would be best to have it in a JIRA to actually track and triage your suggestions. Thank you. - Neelesh S. Salian Cloudera -- View this message in context: http:

Re: spark ui security

I know, but I need only to hide/protect web ui at least with servlet/filter api On Jan 7, 2016, at 4:59 PM, Ted Yu wrote: > Without kerberos you don't have true security. > > Cheers > > On Thu, Jan 7, 2016 at 1:56 PM, Kostiantyn Kudriavtsev > wrote: > can I do it without kerberos and hadoop

Re: spark ui security

Without kerberos you don't have true security. Cheers On Thu, Jan 7, 2016 at 1:56 PM, Kostiantyn Kudriavtsev < kudryavtsev.konstan...@gmail.com> wrote: > can I do it without kerberos and hadoop? > ideally using filters as for job UI > > On Jan 7, 2016, at 1:22 PM, Prem Sure wrote: > > you can r

Re: spark ui security

can I do it without kerberos and hadoop? ideally using filters as for job UI On Jan 7, 2016, at 1:22 PM, Prem Sure wrote: > you can refer more on https://searchcode.com/codesearch/view/97658783/ > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SecurityManager.sc

Re: spark ui security

According to https://spark.apache.org/docs/latest/security.html#web-ui , web UI is covered. FYI On Thu, Jan 7, 2016 at 6:35 AM, Kostiantyn Kudriavtsev < kudryavtsev.konstan...@gmail.com> wrote: > hi community, > > do I understand correctly that spark.ui.filters property sets up filters > only fo

Re: Spark UI - Streaming Tab

The Streaming tab is only supported in the live UI, not in the History Server. On Fri, Dec 4, 2015 at 9:31 AM, patcharee wrote: > I ran streaming jobs, but no streaming tab appeared for those jobs. > > Patcharee > > > > On 04. des. 2015 18:12, PhuDuc Nguyen wrote: > > I believe the "Streaming" t

Re: Spark UI - Streaming Tab

I ran streaming jobs, but no streaming tab appeared for those jobs. Patcharee On 04. des. 2015 18:12, PhuDuc Nguyen wrote: I believe the "Streaming" tab is dynamic - it appears once you have a streaming job running, not when the cluster is simply up. It does not depend on 1.6 and has been in

Re: Spark UI - Streaming Tab

I believe the "Streaming" tab is dynamic - it appears once you have a streaming job running, not when the cluster is simply up. It does not depend on 1.6 and has been in there since at least 1.0. HTH, Duc On Fri, Dec 4, 2015 at 7:28 AM, patcharee wrote: > Hi, > > We tried to get the streaming t

Re: Spark UI consuming lots of memory

Hi Nicholas, I think you are right about the issue relating to Spark-11126, I'm seeing it as well. Did you find any workaround? Looking at the pull request for the fix it doesn't look possible. Best regards, Patrick On 15 October 2015 at 19:40, Nicholas Pritchard < nicholas.pritch...@falkonry.

Re: Spark UI consuming lots of memory

Thanks for your help, most likely this is the memory leak you are fixing in https://issues.apache.org/jira/browse/SPARK-11126. -Nick On Mon, Oct 12, 2015 at 9:00 PM, Shixiong Zhu wrote: > In addition, you cannot turn off JobListener and SQLListener now... > > Best Regards, > Shixiong Zhu > > 201

Re: Spark UI consuming lots of memory

In addition, you cannot turn off JobListener and SQLListener now... Best Regards, Shixiong Zhu 2015-10-13 11:59 GMT+08:00 Shixiong Zhu : > Is your query very complicated? Could you provide the output of `explain` > your query that consumes an excessive amount of memory? If this is a small > quer

Re: Spark UI consuming lots of memory

Is your query very complicated? Could you provide the output of `explain` your query that consumes an excessive amount of memory? If this is a small query, there may be a bug that leaks memory in SQLListener. Best Regards, Shixiong Zhu 2015-10-13 11:44 GMT+08:00 Nicholas Pritchard < nicholas.prit

Re: Spark UI consuming lots of memory

As an update, I did try disabling the ui with "spark.ui.enabled=false", but the JobListener and SQLListener still consume a lot of memory, leading to OOM error. Has anyone encountered this before? Is the only solution just to increase the driver heap size? Thanks, Nick On Mon, Oct 12, 2015 at 8:4

Re: Spark UI consuming lots of memory

I set those configurations by passing to spark-submit script: "bin/spark-submit --conf spark.ui.retainedJobs=20 ...". I have verified that these configurations are being passed correctly because they are listed in the environments tab and also by counting the number of job/stages that are listed. T

Re: Spark UI consuming lots of memory

Could you show how did you set the configurations? You need to set these configurations before creating SparkContext and SQLContext. Moreover, the history sever doesn't support SQL UI. So "spark.eventLog.enabled=true" doesn't work now. Best Regards, Shixiong Zhu 2015-10-13 2:01 GMT+08:00 pnpritc

Re: Spark UI tunneling

Oh in that case you could try adding the hostname in your /etc/hosts under your localhost. Also make sure there is a request going to another host by inspecting the network calls: [image: Inline image 1] Thanks Best Regards On Mon, Mar 23, 2015 at 1:55 PM, Sergey Gerasimov wrote: > Akhil, > >

Re: Spark UI tunneling

Akhil, that's what I did. The problem is that probably web server tried to forward my request to another address accessible locally only. > 23 марта 2015 г., в 11:12, Akhil Das написал(а): > > Did you try ssh -L 4040:127.0.0.1:4040 user@host > > Thanks > Best Regards > >> On Mon, Mar 23,

Re: Spark UI tunneling

Did you try ssh -L 4040:127.0.0.1:4040 user@host Thanks Best Regards On Mon, Mar 23, 2015 at 1:12 PM, sergunok wrote: > Is it a way to tunnel Spark UI? > > I tried to tunnel client-node:4040 but my browser was redirected from > localhost to some cluster locally visible domain name.. > > Maybe

Re: Spark UI and running spark-submit with --master yarn

That does not look like the RM UI. Please check your configuration for the port (see Ted's e-mail). On Mon, Mar 2, 2015 at 4:45 PM, Anupama Joshi wrote: > Hi , > port 8088 does not show me anything .(can not connect) > where as port ec2-52-10-234-111.us-west-2.compute.amazonaws.com:9026 shows >

Re: Spark UI and running spark-submit with --master yarn

Default RM Web UI port is 8088 (configurable through yarn.resourcemanager.webapp.address) Cheers On Mon, Mar 2, 2015 at 4:14 PM, Anupama Joshi wrote: > Hi Marcelo, > Thanks for the quick reply. > I have a EMR cluster and I am running the spark-submit on the master node > in the cluster. > When

Re: Spark UI and running spark-submit with --master yarn

That's the RM's RPC port, not the web UI port. (See Ted's e-mail - normally web UI is on 8088.) On Mon, Mar 2, 2015 at 4:14 PM, Anupama Joshi wrote: > Hi Marcelo, > Thanks for the quick reply. > I have a EMR cluster and I am running the spark-submit on the master node in > the cluster. > When I s

Re: Spark UI and running spark-submit with --master yarn

Hi Marcelo, Thanks for the quick reply. I have a EMR cluster and I am running the spark-submit on the master node in the cluster. When I start the spark-submit , I see 15/03/02 23:48:33 INFO client.RMProxy: Connecting to ResourceManager at / 172.31.43.254:9022 But If I try that URL or the use the e

Re: Spark UI and running spark-submit with --master yarn

What are you calling ""? In yarn-cluster mode, the driver is running somewhere in your cluster, not on the machine where you run spark-submit. The easiest way to get to the Spark UI when using Yarn is to use the Yarn RM's web UI. That will give you a link to the application's UI regardless of whet

Re: Spark UI and Spark Version on Google Compute Engine

Unfortunately we don't have anything to do with Spark on GCE, so I'd suggest asking in the GCE support forum. You could also try to launch a Spark cluster by hand on nodes in there. Sigmoid Analytics published a package for this here: http://spark-packages.org/package/9 Matei > On Jan 17, 2015

Re: Spark UI port issue when deploying Spark driver on YARN in yarn-cluster mode on EMR

On YARN, spark does not manage the cluster, but YARN does. Usually the cluster manager UI is under http://:9026/cluster. I believe that it chooses the port for the spark driver UI randomly, but an easy way of accessing it is by clicking on the "Application Master" link under the "Tracking UI" colum

Re: spark ui redirecting to port 8100

Hi Sadhan, Which port are you specifically trying to redirect? The driver program has a web UI, typically on port 4040... or the Spark Standalone Cluster Master has a UI exposed on port 7077. Which setting did you update in which file to make this change? And finally, which version of Spark are

Re: SPARK UI - Details post job processiong

I am working on a PR that allows one to send the same spark listener event message back to the application in yarn cluster mode. So far I have put this function in our application, our UI will receive and display the same spark job event message such as progress, job start, completed etc Esse

Re: SPARK UI - Details post job processiong

Yes, I’m running Hadoop’s Timeline server that does this for the YARN/Hadoop logs (and works very nicely btw). Are you saying I can do the same for the SparkUI as well? Also, where do I set these Spark configurations since this will be executed inside a YARN container? On the “client” machine

Re: SPARK UI - Details post job processiong

Matt you should be able to set an HDFS path so you'll get logs written to a unified place instead of to local disk on a random box on the cluster. On Thu, Sep 25, 2014 at 1:38 PM, Matt Narrell wrote: > How does this work with a cluster manager like YARN? > > mn > > On Sep 25, 2014, at 2:23 PM, A

Re: SPARK UI - Details post job processiong

How does this work with a cluster manager like YARN? mn On Sep 25, 2014, at 2:23 PM, Andrew Or wrote: > Hi Harsha, > > You can turn on `spark.eventLog.enabled` as documented here: > http://spark.apache.org/docs/latest/monitoring.html. Then, if you are running > standalone mode, you can acces

Re: SPARK UI - Details post job processiong

Hi Harsha, You can turn on `spark.eventLog.enabled` as documented here: http://spark.apache.org/docs/latest/monitoring.html. Then, if you are running standalone mode, you can access the finished SparkUI through the Master UI. Otherwise, you can start a HistoryServer to display finished UIs. -Andr

Re: spark ui on yarn

my yarn environment does have less memory for the executors. i am checking if the RDDs are cached by calling sc.getRDDStorageInfo, which shows an RDD as fully cached in memory, yet it does not show up in the UI On Sun, Jul 13, 2014 at 1:49 AM, Matei Zaharia wrote: > The UI code is the same in

Re: spark ui on yarn

The UI code is the same in both, but one possibility is that your executors were given less memory on YARN. Can you check that? Or otherwise, how do you know that some RDDs were cached? Matei On Jul 12, 2014, at 4:12 PM, Koert Kuipers wrote: > hey shuo, > so far all stage links work fine for

Re: spark ui on yarn

hey shuo, so far all stage links work fine for me. i did some more testing, and it seems kind of random what shows up on the gui and what does not. some partially cached RDDs make it to the GUI, while some fully cached ones do not. I have not been able to detect a pattern. is the codebase for the

Re: spark ui on yarn

Hi Koert, Just curious did you find any information like "CANNOT FIND ADDRESS" after clicking into some stage? I've seen similar problems due to lost of executors. Best, On Fri, Jul 11, 2014 at 4:42 PM, Koert Kuipers wrote: > I just tested a long lived application (that we normally run in s