On Fri, Oct 26, 2012 at 1:58 PM, Evan Huus <eapa...@gmail.com> wrote:
> On Fri, Oct 26, 2012 at 11:40 AM, Graham Bloice > <graham.blo...@trihedral.com> wrote: > > > > On 26 October 2012 14:44, Evan Huus <eapa...@gmail.com> wrote: > >> > >> On Fri, Oct 26, 2012 at 9:29 AM, Sébastien Tandel > >> <sebastien.tan...@gmail.com> wrote: > >> > > >> > > >> > On Wed, Oct 24, 2012 at 11:13 AM, Evan Huus <eapa...@gmail.com> > wrote: > >> >> > >> >> On Wed, Oct 24, 2012 at 8:10 AM, Sébastien Tandel > >> >> <sebastien.tan...@gmail.com> wrote: > >> >> > > >> >> > > >> >> > On Wed, Oct 24, 2012 at 1:10 AM, Guy Harris <g...@alum.mit.edu> > wrote: > >> >> >> > >> >> >> > >> >> >> On Oct 18, 2012, at 6:01 PM, Evan Huus <eapa...@gmail.com> wrote: > >> >> >> > >> >> >> > I have linked a tarball [2] containing the following files: > >> >> >> > - wmem_allocator.h - the definition of the allocator interface > >> >> >> > - wmem_allocator_glib.* - a simple implementation of the > allocator > >> >> >> > interface backed by g_malloc and a singly-linked list. > >> >> >> > >> >> >> Presumably an implementation of the allocator could, instead of > >> >> >> calling > >> >> >> a > >> >> >> lower-level memory allocator (malloc(), g_malloc(), etc.) for each > >> >> >> allocation call, allocate larger chunks and parcel out memory from > >> >> >> the > >> >> >> larger chunks (as the current emem allocator does), if that ends > up > >> >> >> saving > >> >> >> enough CPU, by making fewer allocate and free calls to the > >> >> >> underlying > >> >> >> memory > >> >> >> allocator, so as to make it worth whatever wasted memory we have > at > >> >> >> the > >> >> >> ends > >> >> >> of chunks? > >> >> >> > >> >> > > >> >> > One step further, instead of mempools, I think wireshark could have > >> >> > great > >> >> > interest in implementing slabs (slab allocator). Slabs had > initially > >> >> > been > >> >> > designed for kernel with several advantages over traditional > >> >> > allocators > >> >> > in > >> >> > terms of resources needed to allocate (CPU), (external / internal) > >> >> > fragmentation and also cache friendliness (most of the traditional > >> >> > allocators don't care). I've attached some slides about a > high-level > >> >> > description of slab. > >> >> > > >> >> > Since then, another paper has been written showing some > improvements > >> >> > and > >> >> > what it took to write a slab for user-space (libumem). There is > >> >> > another > >> >> > well-known exampel out there, called memcache, that implements its > >> >> > own > >> >> > version (and could be a good intial point for wireshark > >> >> > implementation, > >> >> > who > >> >> > knows? :)) > >> >> > >> >> If I understand correctly, a slab allocator provides the most benefit > >> >> when you have to alloc/free a large number of the same type of > object, > >> > > >> > you're right, that's where slab is the most efficient at. Although, > the > >> > second paper shows it can be efficient for general purpose allocation > >> > based > >> > on size and not specific structure. > >> > > >> >> but I don't know if this is necessarily the case in Wireshark. There > >> >> are probably places where it would be useful, but I can't think of > any > >> >> off the top of my head. TVBs maybe? I know emem is currently used all > >> >> over the place for all sorts of different objects... > >> > > >> > I guess the most obvious would be emem_tree (emem_tree_node) might be > an > >> > example used all over and over while dissecting. :) > >> > There is indeed a bunch of different objects allocated with emem. > Also, > >> > it > >> > might be used to allocate memory for some fragments. > >> > >> Ah, yes, the various emem data structures (tree, stack, etc.) would > >> likely benefit from slab allocators. Converting them to use slabs > >> would be something to do while porting them from emem to wmem. > >> > >> > Since your interface seems to allow it, we could create several slabs > >> > types, > >> > one for each specific structures that are allocated very frequently > >> > (emem_tree_node?), others for packets/fragments with some tuned slabs > >> > sizes > >> > and another with some generic sizes. > >> > >> That seems reasonable, presumably with some shared slab code doing the > >> type-agnostic heavy lifting. I'll have to give a bit of thought to > >> what the interface for that would be like - if you already have an > >> interface in mind, please share :) > >> > > > > Are the slab allocators mentioned "homegrown" or provided by the host > OS. If > > the latter, what platforms are they available on? > > Homegrown on top of malloc/g_malloc/mmap, I believe. A slab allocator > is (or was) used internally in the linux and solaris kernels, but has > never been exposed to userspace to my knowledge. > It's indeed not exposed to users. It's used internally as a "kernel object cache allocator". But, memcached has a user-space implementation that could -probably- be leveraged for wireshark. ___________________________________________________________________________ > Sent via: Wireshark-dev mailing list <wireshark-dev@wireshark.org> > Archives: http://www.wireshark.org/lists/wireshark-dev > Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev > mailto:wireshark-dev-requ...@wireshark.org > ?subject=unsubscribe >
___________________________________________________________________________ Sent via: Wireshark-dev mailing list <wireshark-dev@wireshark.org> Archives: http://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
