Thanks for the FLIP, Yadong. In general I think this work is valuable for users to better understand the Flink's memory usages in different dimensions.
Sorry for not going through every detailed discussions below, and I try to do that later if possible. Firstly I try to answer some Andrey's concerns with mmap. > - I do not know how the mapped memory works. Is it meant for the new spilled > partitions? If the mapped memory also pulls from the direct > memory limit then this is something we do not account in our network buffers > as I understand. In this case, this metric may be useful for tuning to > understand > how much the mapped memory uses from the direct memory limit to set e.g. > framework off-heap limit correctly and avoid direct OOM. > It could be something to discuss with Zhijiang. e.g. is the direct memory > used there to buffer fetched regions of partition files or what for? Yes, the mapped memory is used in bounded blocking partition for batch jobs now, but not the default mode. AIK it is not related and limited to the setting of `MaxDirectMemory`, so we do not need to worry about the current direct memory setting and the potential OOM issue. It is up to the address space to determine the mapped file size, and in 64 bit system we can regard the limitless size in theory. Regarding the size of mapped buffer pool from MXBean, it only indicates how much file size were already mapped before, even it is unchanged to not reflect the real physical memory use. E.g. when the file was mapped 100GB region at the beginning, the mapped buffer pool from MXBean would be 100GB. But how many physical memories are really consumed is up to the specific read or write operations in practice, and also controlled by the operator system. E.g some unused regions might be exchanged into SWAP virtual memory when physical memory is limited. From this point, I guess it is no meaningful to show the size of mapped buffer pool for users who may be more concerned with how many physical memories are really used. Best, Zhijiang ------------------------------------------------------------------ From:Andrey Zagrebin <azagre...@apache.org> Send Time:2020 Mar. 30 (Mon.) 22:56 To:dev <dev@flink.apache.org> Subject:Re: [VOTE] FLIP-102: Add More Metrics to TaskManager Hi All, Thanks for this FLIP, Yadong. This is a very good improvement to the Flink's UI. It looks like there are still couple of things to resolve before the final vote. - I also find the non-heap title in configuration confusing because there are also other non-heap types of memory. The "off-heap" concept is quite broad. What about "JVM specific" meaning that it is not coming directly from Flink? or we could remove the "Non-heap" box at all and show directly JVM Metaspace and Overhead as separate boxes, this would also fit if we decide to keep the Metaspace metric. - Total Process Memory Used: I agree with Xintong, it is hard to say what is used there. Then the size of "Total Process Memory" basically becomes part of configuration. - Non-Heap Used/Max/.. Not sure what committed means here. I also think we should either exclude it or display what is known for sure. In general, the metaspace usage would be nice to have but it should be then exactly metaspace usage without any thing else. - I do not know how the mapped memory works. Is it meant for the new spilled partitions? If the mapped memory also pulls from the direct memory limit then this is something we do not account in our network buffers as I understand. In this case, this metric may be useful for tuning to understand how much the mapped memory uses from the direct memory limit to set e.g. framework off-heap limit correctly and avoid direct OOM. It could be something to discuss with Zhijiang. e.g. is the direct memory used there to buffer fetched regions of partition files or what for? - Not sure, we need an extra wrapping box "other" for the managed memory atm. I could be just "Managed" or "Managed by Flink". Best, Andrey On Fri, Mar 27, 2020 at 6:13 AM Xintong Song <tonysong...@gmail.com> wrote: > Sorry for the late response. > > I have shared my suggestions with Yadong & Lining offline. I think it would > be better to also post them here, for the public record. > > - I'm not sure about displaying Total Process Memory Used. Currently, we > do not have a good way to monitor all memory footprints of the process. > Metrics for some native memory usages (e.g., thread stack) are absent. > Displaying a partial used memory size could be confusing for users. > - I would suggest merge the current Mapped Memory metrics into Direct > Memory. Actually, the metrics are retrieved from MXBeans for direct > buffer > pool and mapped buffer pool. Both of the two pools are accounted for in > -XX:MaxDirectMemorySize. There's no Flink configuration that can modify > the > individual pool sizes. Therefore, I think displaying the total Direct > Memory would be good enough. Moreover, in most use cases the size of > mapped > buffer pool is zero and users do not need to understand what is Mapped > Memory. For expert users who do need the separated metrics for > individual > pools, they can subscribe the metrics on their own. > - I would suggest to not display Non-Heap Memory. Despite the name, the > metrics (also retrieved from MXBeans) actually accounts for metaspace, > code > cache, and compressed class space. It does not account for all JVM > native > memory overheads, e.g., thread stack. That means the metrics of Non-Heap > Memory do not well correspond to any of the FLIP-49 memory components. > They > account for Flink's JVM Metaspace and part of JVM Overhead. I think this > brings more confusion then help to users, especially primary users. > > > Thank you~ > > Xintong Song > > > > On Thu, Mar 26, 2020 at 6:34 PM Till Rohrmann <trohrm...@apache.org> > wrote: > > > Thanks for updating the FLIP Yadong. > > > > What is the difference between managedMemory and managedMemoryTotal > > and networkMemory and networkMemoryTotal in the REST response? If they > are > > duplicates, then we might be able to remove one. > > > > Apart from that, the proposal looks good to me. > > > > Pulling also Andrey in to hear his opinion about the representation of > the > > memory components. > > > > Cheers, > > Till > > > > On Thu, Mar 19, 2020 at 11:37 AM Yadong Xie <vthink...@gmail.com> wrote: > > > >> Hi all > >> > >> I have updated the design of the metric page and FLIP doc, please let me > >> know what you think about it > >> > >> FLIP-102: > >> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager > >> POC web: > >> > >> > http://101.132.122.69:8081/web/#/task-manager/8e1f1beada3859ee8e46d0960bb1da18/metrics > >> > >> Till Rohrmann <trohrm...@apache.org> 于2020年2月27日周四 下午10:27写道： > >> > >> > Thinking a bit more about the problem whether to report the aggregated > >> > memory statistics or the individual slot statistics, I think reporting > >> it > >> > on a per slot basis won't work nicely together with FLIP-56 (dynamic > >> slot > >> > allocation). The problem is that with FLIP-56, we will no longer have > >> > dedicated slots. The number of slots might change over the lifetime > of a > >> > TaskExecutor. Hence, it won't be easy to generate a metric path for > >> every > >> > slot which are furthermore also ephemeral. So maybe, the more general > >> and > >> > easier solution would be to report the overall memory usage of a > >> > TaskExecutor even though it means to do some aggregation on the > >> > TaskExecutor. > >> > > >> > Concerning the JVM limit: Isn't it mainly the code cache? If we > display > >> > this value, then we should explain what exactly it means. I fear that > >> most > >> > users won't understand what JVM limit actually means. > >> > > >> > Cheers, > >> > Till > >> > > >> > On Wed, Feb 26, 2020 at 11:15 AM Yadong Xie <vthink...@gmail.com> > >> wrote: > >> > > >> > > Hi Till > >> > > > >> > > Thanks a lot for your response > >> > > > >> > > > 2. I'm not entirely sure whether I would split the memory ... > >> > > > >> > > Split the memory display comes from the 'ancient' design of the web, > >> it > >> > is > >> > > ok for me to change it following > total/heap/managed/network/direct/jvm > >> > > overhead/mapped sequence > >> > > > >> > > > 3. Displaying the memory configurations... > >> > > > >> > > I agree with you that it is not a very nice way, but the > hierarchical > >> > > relationship of configurations is too complex and hard to display in > >> the > >> > > other ways (I have tried) > >> > > > >> > > if anyone has a better idea, please feels no hesitates to help me > >> > > > >> > > > >> > > > 4. What does JVM limit mean in Non-heap.JVM-Overhead? > >> > > > >> > > JVM limit is "non-heap max metric minus metaspace configuration" as > >> > > @Xintong > >> > > Song <tonysong...@gmail.com> replyed in this mail thread > >> > > > >> > > > >> > > Till Rohrmann <trohrm...@apache.org> 于2020年2月25日周二 下午6:58写道： > >> > > > >> > > > Thanks for creating this FLIP Yadong. I think your proposal makes > it > >> > much > >> > > > easier for the user to understand what's happening on Flink > >> > > TaskManager's. > >> > > > > >> > > > I have some comments: > >> > > > > >> > > > 1. Some of the newly introduced metrics involve computations on > the > >> > > > TaskManager. I would like to avoid additional computations > >> introduced > >> > by > >> > > > metrics as much as possible because metrics should not affect the > >> > system. > >> > > > In particular, total memory sizes which are configured should not > be > >> > > > derived computationally (getManagedMemoryTotal, > getTotalMemorySize). > >> > For > >> > > > the currently available memory sizes (e.g. getManagedMemoryUsed), > >> one > >> > > could > >> > > > think about reporting them on a per slot basis and to do the > >> > aggregation > >> > > on > >> > > > the client side. Of course, this would increase the size of the > >> > response > >> > > > payload. > >> > > > > >> > > > 2. I'm not entirely sure whether I would split the memory display > >> into > >> > > JVM > >> > > > memory and non JVM memory as you've done it int the POC. From a > >> user's > >> > > > perspective, one could start displaying the total process memory. > >> The > >> > > next > >> > > > three most important metrics are the heap, managed memory and > >> network > >> > > > buffer usage, I guess. If one is interested in more details, one > >> could > >> > > then > >> > > > display the remaining direct memory usage, the JVM overhead (I'm > not > >> > sure > >> > > > whether I would call this non-heap though) and the mapped memory. > >> > > > > >> > > > 3. Displaying the memory configurations in three nested boxes does > >> not > >> > > look > >> > > > so nice to me. I'm not sure how else one could display it, though. > >> > > > > >> > > > 4. What does JVM limit mean in Non-heap.JVM-Overhead? > >> > > > > >> > > > Cheers, > >> > > > Till > >> > > > > >> > > > On Tue, Feb 25, 2020 at 8:19 AM Yadong Xie <vthink...@gmail.com> > >> > wrote: > >> > > > > >> > > > > Hi Xintong > >> > > > > thanks for your advice, the POC web and the FLIP doc was updated > >> now > >> > > > > here is the new link: > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > http://101.132.122.69:8081/web/#/task-manager/7e7cf0293645c8537caab915c829aa73/metrics > >> > > > > > >> > > > > > >> > > > > Xintong Song <tonysong...@gmail.com> 于2020年2月21日周五 下午12:00写道： > >> > > > > > >> > > > > > > > >> > > > > > > 1. Should the managed memory be part of direct memory? > >> > > > > > > > >> > > > > > The answer is no. Managed memory is currently allocated by > >> > accessing > >> > > to > >> > > > > > private field of Unsafe. It is not accounted for in JVM's > direct > >> > > memory > >> > > > > > limit and corresponding metrics. To that end, it is equivalent > >> to > >> > > > > > native memory. > >> > > > > > > >> > > > > > > >> > > > > > > 2. Should the shuffle memory also be part of the managed > >> memory? > >> > > > > > > >> > > > > > I don't think so. Shuffle (Network) memory is allocated with > >> direct > >> > > > > > buffers, and accounted for in JVM's direct memory limit and > >> > > > corresponding > >> > > > > > metrics. Moreover, the FLIP-49 memory model expose network > >> memory > >> > and > >> > > > > > managed memory as two independent components of the overall > >> memory > >> > > > > > footprint. > >> > > > > > > >> > > > > > > >> > > > > > Thank you~ > >> > > > > > > >> > > > > > Xintong Song > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > On Fri, Feb 21, 2020 at 11:45 AM Kurt Young <ykt...@gmail.com > > > >> > > wrote: > >> > > > > > > >> > > > > > > Some questions related to "managed memory": > >> > > > > > > > >> > > > > > > 1. Should the managed memory be part of direct memory? > >> > > > > > > 2. Should the shuffle memory also be part of the managed > >> memory? > >> > > > > > > > >> > > > > > > Best, > >> > > > > > > Kurt > >> > > > > > > > >> > > > > > > > >> > > > > > > On Fri, Feb 21, 2020 at 10:41 AM Xintong Song < > >> > > tonysong...@gmail.com > >> > > > > > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > > > Thanks for driving this FLIP, Yadong. > >> > > > > > > > > >> > > > > > > > +1 (non-binding) for the FLIP in general. I think this > >> really > >> > > helps > >> > > > > our > >> > > > > > > > users to understand and use the new FLIP-49 memory > >> > configuration. > >> > > > > > > > > >> > > > > > > > I have a few minor comments. > >> > > > > > > > - There's a frame "Other" in the frame "Non-Heap", besides > >> "JVM > >> > > > > > Overhead" > >> > > > > > > > and "JVM Metaspace". IIUC, the purpose of this is to > explain > >> > the > >> > > > > > > > mismatching between the metric "non-heap maximum" and the > >> sum > >> > of > >> > > > the > >> > > > > > > > configurations "JVM metaspace" & "JVM Overhead". However, > >> from > >> > > the > >> > > > > > > > perspective of FLIP-49, JVM Overhead accounts for all the > >> JVM > >> > > > > non-heap > >> > > > > > > > memory usages except for metaspace. The metrics does not > >> match > >> > > the > >> > > > > > > > configuration because we did not set the a JVM parameter > for > >> > "max > >> > > > > > > non-heap > >> > > > > > > > memory" (actually I'm not sure whether it can be specified > >> in > >> > > java > >> > > > > 8). > >> > > > > > > The > >> > > > > > > > current UI might confuse people making them think there > are > >> > other > >> > > > > > > non-heap > >> > > > > > > > memory usages not accounted by the configurations. > >> Therefore, I > >> > > > would > >> > > > > > > > suggest to remove the "Other" frame, but add another frame > >> > inside > >> > > > > "JVM > >> > > > > > > > Overhead", besides "Configuration", with "JVM limit" as > the > >> > title > >> > > > and > >> > > > > > > > "non-heap max metric minus metaspace configuration" as the > >> > value > >> > > . > >> > > > > > > > > >> > > > > > > > - In the final release, we have changed "shuffle memory" > to > >> > > > "network > >> > > > > > > > memory" because the latter is easier to understand for > >> users. I > >> > > > think > >> > > > > > we > >> > > > > > > > should be updated it in this FLIP as well. > >> > > > > > > > > >> > > > > > > > - There's a typo "Directed" (should be "Direct") at the > >> direct > >> > > > memory > >> > > > > > > > metric. > >> > > > > > > > > >> > > > > > > > Thank you~ > >> > > > > > > > > >> > > > > > > > Xintong Song > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > On Thu, Feb 20, 2020 at 5:52 PM Yadong Xie < > >> > vthink...@gmail.com> > >> > > > > > wrote: > >> > > > > > > > > >> > > > > > > > > Hi all > >> > > > > > > > > > >> > > > > > > > > I want to start the vote for FLIP-102, which proposes to > >> add > >> > > more > >> > > > > > > metrics > >> > > > > > > > > to the task manager in web UI. > >> > > > > > > > > > >> > > > > > > > > To help everyone better understand the proposal, we > spent > >> > some > >> > > > > > efforts > >> > > > > > > on > >> > > > > > > > > making an online POC > >> > > > > > > > > > >> > > > > > > > > previous web: > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > http://101.132.122.69:8081/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/metrics > >> > > > > > > > > POC web: > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > http://101.132.122.69:8081/web/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/metrics > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > The vote will last for at least 72 hours, following the > >> > > consensus > >> > > > > > > voting > >> > > > > > > > > process. > >> > > > > > > > > > >> > > > > > > > > FLIP wiki: > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager > >> > > > > > > > > > >> > > > > > > > > Discussion thread: > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html > >> > > > > > > > > > >> > > > > > > > > Thanks, > >> > > > > > > > > > >> > > > > > > > > Yadong > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > >
