Github user davies commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-62473806
@pwendell @kayousterhout , Thanks for review this.
Console is a character device, all the graph are implemented as an stream
of characters. It's very hard to show a near realtime progress bar without
discruptting the logging (they don't know each other), for example: 1) if
progress bar is showed after cursor of logging, then future logging will
overwrite the progress bar (part of) 2) if progress bar is showed after the
cursor of logging (such as top line of console), then old loggings (also the
output of results) will be overwritten by progress bar.
So the current approach is that the progress bar is only showed when the
logging level is WARN (or higher). If the logging level is DEBUG or INFO, users
can get the progress info from logging, also it's hard the manage these twos.
The progress bar is showed between call a action API and it returns, so it's
expected that there is no output/logging in this period, the console will not
become mess. If we move to listener based implementation, then it's hard to
cleanup the progress bar before the api `return`, it's also the reason that I
move `sched.dagScheduler.taskEnded` after showProgressBar().
It's did not work properly when a job has multiple concurrent stages, the
concurrent progress bar will overwrite each other randomly. Each bar will begin
with it's stage id, so it's still kind of readable.
I agree that putting the code of progress bar into TaskSetManager is not
good idea, I will move them out after we finalize other stuff (how to deal with
logging, use listener api or not).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
