@Yadong 2. I agree that we can update the task executor ui after flip-56 is done. But I would suggest keep it on discussion to come up with a proper ui design for task executor resources. I don't think the mentioned image from flip-56 is a good choice. That image is a simplified figure with cpu and total memory only, for the purpose of demonstrating dynamic slot allocation. In fact, there are 6 fields to be displayed (cpu, task heap, task off-heap, shuffle, on-heap managed, off-heap managed). If we display cpu and total memory only, then user will be confused when seeing a task executor with enough remaining resources but tasks cannot be deployed onto it (because the desired type of memory might be used up).
4. I've been using blink webui, which already have log pagination. It's quite common that we need do search for some keywords (e.g., exception, error, warning) from a large amount of logs for diagnosing problems. I find it very inconvenient that I have to click into each page searching for the keywords, and I'd rather take the effort to find the original log files from the filesystem to view the log. Personally speaking, if the keyword searching cannot be supported, I would prefer to take some time loading the non-paginated logs over than paginated ones. Or we may at least have a button on the webui for switching between the two alternatives. @Till Thanks for the inputs. Thank you~ Xintong Song On Mon, Sep 30, 2019 at 5:55 PM Till Rohrmann <trohrm...@apache.org> wrote: > For 3. At the moment the log and stdout file serving requires the > TaskExecutor to be running. But in some scenarios when having a NFS, it > should be enough to know where the file is located. However, this > assumption does not hold in the general case. > > Cheers, > Till > > On Mon, Sep 30, 2019 at 11:43 AM Yadong Xie <vthink...@gmail.com> wrote: > > > Hi Xintong Song > > > > Thanks for your comments! > > > > 1. I think it is a good idea that to align CPU and memory usage with > > FLIP-49 if it will release in version 1.10 > > 2. We can update the task executor UI design after FLIP-56 merged into > > master. Actually, the image > > < > > > https://cwiki.apache.org/confluence/download/attachments/125309297/BlinkResourceTM.png?version=1&modificationDate=1566223821000&api=v2 > > > > > in FLIP-56 is a good UI design, we can follow it in the Flink web. > > 3. No idea about it, maybe anyone famailar with the runtime part could > > answer it? but it would be great to add it to the web UI in my opinion. > > 4. I'm not sure will keyword searching across all the pages may cost too > > many resources in job manager, but I think it would be very useful if the > > REST API could support it. > > > > Best, > > Yadong > > > > Xintong Song <tonysong...@gmail.com> 于2019年9月29日周日 下午8:11写道： > > > > > Thanks for drafting the FLIP and starting this discussion, Yadong. > > > > > > > > > I have some comments: > > > > > > > > > - I can see that the proposed memory and cpu usage to be displayed > (in > > > section 1.1) are aligned with the current ResourceProfile fields. > > > However, > > > we are working on changing the memory fields in 1.10 with FLIP-49 > > [1]. I > > > suggest we align the UI design with the new FLIP-49 memory fields. > > > - The task executor overview design (in section 1.2) is based on the > > > current slot model. The coming FLIP-56 [2] which is also planned for > > > 1.10 > > > is changing the model so that task executors no longer have fixed > > > number of > > > slots, but allocated slots (may have different resources) and > > available > > > resources. > > > - I can see that there's discussions in the google doc about > using > > > different color for available resources. However, the resource > > > availability > > > for different fields can be different, and may not be simply > > > displayed by a > > > different color. E.g., a task executor may have two slot, while > > slot > > > 1 > > > takes (20% cpu, 10% heap mem, 50% managed mem, etc.), slot 2 > takes > > > (10% > > > cpu, 35% heap mem, 0% managed mem etc.), and the remaining > > > resources in > > > the task executor are (70% cpu, 55% heap mem, 50% managed mem, > > > etc.). How > > > do you plan to display that? > > > - I would suggest to have multiple bars for each task executor, > > while > > > each bar represents one of the resource fields. In addition, we > > > may have a > > > number (or some other figures) showing how many slots are > allocated > > > from > > > the task executor. > > > - Is there any way we provide access to logs of terminated task > > > executors? It occurs to us a lot that a job failed due to a task > > > executor > > > fail/lost. And we have to find the logs of failed task executors by > > > manually accessing the file system. I think it would be helpful if > we > > > can > > > find the logs of failed task executors directly in flink webui. > > > - Regarding log pagination, is there any way to provide keyword > > > searching across all the pages? > > > > > > > > > Thank you~ > > > > > > Xintong Song > > > > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > [2] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation > > > > > > On Fri, Sep 27, 2019 at 3:57 PM Paul Lam <paullin3...@gmail.com> > wrote: > > > > > > > Filed a jira to track this[1]. Thanks a lot. > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-14242 < > > > > https://issues.apache.org/jira/browse/FLINK-14242> > > > > > > > > Best, > > > > Paul Lam > > > > > > > > > 在 2019年9月27日，14:34，Yadong Xie <vthink...@gmail.com> 写道： > > > > > > > > > > Hi Paul > > > > > Thanks for your suggestion. > > > > > I think it is easy to implement, could you create a JIRA for me? > > > > > > > > > > Paul Lam <paullin3...@gmail.com> 于2019年9月27日周五 上午11:11写道： > > > > > > > > > >> Hi Yadong, > > > > >> > > > > >> Thanks a lot for summing up the Web UI efforts. > > > > >> > > > > >> I have a minor suggestion: can we provide a collapse button for > the > > > task > > > > >> names in job graph visualization? For some complex jobs, > especially > > > SQL > > > > >> jobs, the task names are quite long which makes the job graph hard > > to > > > > read. > > > > >> > > > > >> Best, > > > > >> Paul Lam > > > > >> > > > > >>> 在 2019年9月27日，10:13，Yadong Xie <vthink...@gmail.com> 写道： > > > > >>> > > > > >>> Hi all > > > > >>> > > > > >>> Flink Web UI is the main platform for most users to monitor their > > > jobs > > > > >> and > > > > >>> clusters. We have reconstructed Flink web in 1.9.0 version, but > > there > > > > are > > > > >>> still some shortcomings. > > > > >>> > > > > >>> This discussion thread aims to provide a better experience for > > Flink > > > UI > > > > >>> users. > > > > >>> > > > > >>> Here is the design doc I drafted: > > > > >>> > > > > >>> > > > > >> > > > > > > > > > > https://docs.google.com/document/d/1tIa8yN2prWWKJI_fa1u0t6h1r6RJpp56m48pXEyh6iI/edit?usp=sharing > > > > >>> > > > > >>> > > > > >>> The FLIP can be found at [2]. > > > > >>> > > > > >>> Please keep the discussion here, in the mailing list. > > > > >>> > > > > >>> Looking forward to your opinions, any feedbacks are welcome. > > > > >>> > > > > >>> [1]: > > > > >>> > > > > >> > > > > > > > > > > https://docs.google.com/document/d/1tIa8yN2prWWKJI_fa1u0t6h1r6RJpp56m48pXEyh6iI/edit?usp=sharing > > > > >>> < > > > > >> > > > > > > > > > > https://docs.google.com/document/d/1tIa8yN2prWWKJI_fa1u0t6h1r6RJpp56m48pXEyh6iI/edit# > > > > >>> > > > > >>> [2]: > > > > >>> > > > > >> > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-75%3A+Flink+Web+UI+Improvement+Proposal > > > > >> > > > > >> > > > > > > > > > > > > > >
