Re: [PATCH 1/2] kmemcheck v3

2008-02-09 Thread Ingo Molnar
* Andi Kleen <[EMAIL PROTECTED]> wrote: > > Yes, this is true. Then our task is to make sure that this memory is > > never allocated from tracked caches. We do have some changes in this > > area, for instance we never track task structs. Keep in mind that > > only slab objects are tracked curr

Re: [PATCH 1/2] kmemcheck v3

2008-02-08 Thread Andi Kleen
On Fri, Feb 08, 2008 at 01:59:57PM +0100, Vegard Nossum wrote: > On 2/8/08, Andi Kleen <[EMAIL PROTECTED]> wrote: > > On Fri, Feb 08, 2008 at 01:18:37PM +0100, Vegard Nossum wrote: > > > On 2/8/08, Andi Kleen <[EMAIL PROTECTED]> wrote: > > > > Your assumption that only the string instructions can t

Re: [PATCH 1/2] kmemcheck v3

2008-02-08 Thread Vegard Nossum
On 2/8/08, Andi Kleen <[EMAIL PROTECTED]> wrote: > On Fri, Feb 08, 2008 at 01:18:37PM +0100, Vegard Nossum wrote: > > On 2/8/08, Andi Kleen <[EMAIL PROTECTED]> wrote: > > > Your assumption that only the string instructions can take > > > multiple page faults seems a little dangerous too. > > > > Ye

Re: [PATCH 1/2] kmemcheck v3

2008-02-08 Thread Andi Kleen
On Fri, Feb 08, 2008 at 01:18:37PM +0100, Vegard Nossum wrote: > On 2/8/08, Andi Kleen <[EMAIL PROTECTED]> wrote: > > Your assumption that only the string instructions can take > > multiple page faults seems a little dangerous too. > > Yes, this is true. I cannot guarantee that there are no other

Re: [PATCH 1/2] kmemcheck v3

2008-02-08 Thread Vegard Nossum
On 2/8/08, Andi Kleen <[EMAIL PROTECTED]> wrote: > Your assumption that only the string instructions can take > multiple page faults seems a little dangerous too. Yes, this is true. I cannot guarantee that there are no other instructions that could access more than one memory location but only tak

Re: [PATCH 1/2] kmemcheck v3

2008-02-08 Thread Pekka Enberg
On Feb 8, 2008 2:15 PM, Andi Kleen <[EMAIL PROTECTED]> wrote: > > need to worry about it just yet. In case it's from kmalloc() you can > > pass __GFP_NOTRACK to annotate those call sites where the memory is > > Ok you should add that then to skbuff.c. Indeed. If you look at the second patch, I thi

Re: [PATCH 1/2] kmemcheck v3

2008-02-08 Thread Andi Kleen
On Fri, Feb 08, 2008 at 01:37:11PM +0200, Pekka Enberg wrote: > Hi Andi, > > On Feb 8, 2008 1:55 PM, Andi Kleen <[EMAIL PROTECTED]> wrote: > > Also i'm not sure how you handle initializedness of DMAed data > > (like network buffers). Wouldn't you need hooks into pci_dma_* > > for this? > > If the

Re: [PATCH 1/2] kmemcheck v3

2008-02-08 Thread Pekka Enberg
Hi Andi, On Feb 8, 2008 2:10 PM, Andi Kleen <[EMAIL PROTECTED]> wrote: > It's probably tricky; there are all kinds of hidden page faults > on x86 on data structures allocated as pages (e.g. GDT, LDT [which > is sometimes kmalloc too], stack etc.) Aah, I see. We can annotate those callers to disab

Re: [PATCH 1/2] kmemcheck v3

2008-02-08 Thread Pekka Enberg
Hi Andi, On Feb 8, 2008 1:55 PM, Andi Kleen <[EMAIL PROTECTED]> wrote: > Also i'm not sure how you handle initializedness of DMAed data > (like network buffers). Wouldn't you need hooks into pci_dma_* > for this? If the DMA'd memory is allocated from the page allocator, we don't need to worry abo

Re: [PATCH 1/2] kmemcheck v3

2008-02-08 Thread Andi Kleen
On Fri, Feb 08, 2008 at 01:31:44PM +0200, Pekka Enberg wrote: > Hi Andi, > > On Feb 8, 2008 1:55 PM, Andi Kleen <[EMAIL PROTECTED]> wrote: > > Impressive patch! On the other hand a lot of the interesting > > data isn't it kmalloc anymore, but in slab. Does it really track > > all that much? > > I

Re: [PATCH 1/2] kmemcheck v3

2008-02-08 Thread Pekka Enberg
Hi Andi, On Feb 8, 2008 1:55 PM, Andi Kleen <[EMAIL PROTECTED]> wrote: > Impressive patch! On the other hand a lot of the interesting > data isn't it kmalloc anymore, but in slab. Does it really track > all that much? It tracks all slab caches. What we're not tracking is pages from the page alloc

Re: [PATCH 1/2] kmemcheck v3

2008-02-08 Thread Andi Kleen
On Thu, Feb 07, 2008 at 10:36:20PM +0100, Vegard Nossum wrote: > With a lot of help from Ingo Molnar and Pekka Enberg over the last couple of > weeks, we've been able to produce a new version of kmemcheck! Impressive patch! On the other hand a lot of the interesting data isn't it kmalloc anymore,

Re: [PATCH 1/2] kmemcheck v3

2008-02-08 Thread Ingo Molnar
* Pekka Enberg <[EMAIL PROTECTED]> wrote: > On Feb 8, 2008 1:32 AM, Christoph Lameter <[EMAIL PROTECTED]> wrote: > > But the slab layer allocates pages < PAGE_SIZE. You need to take a > > fault right? So each object would need its own page? > > No. We allocate a shadow page for each data page w

Re: [PATCH 1/2] kmemcheck v3

2008-02-07 Thread Pekka Enberg
Hi Christoph, On Thu, 7 Feb 2008, Vegard Nossum wrote: > > - DMA can be a problem since there's generally no way for kmemcheck to > > determine when/if a chunk of memory is used for DMA. Ideally, DMA should > > be > > allocated with untracked caches, but this requires annotation of the > >

Re: [PATCH 1/2] kmemcheck v3

2008-02-07 Thread Christoph Lameter
On Thu, 7 Feb 2008, Vegard Nossum wrote: > - DMA can be a problem since there's generally no way for kmemcheck to > determine when/if a chunk of memory is used for DMA. Ideally, DMA should be > allocated with untracked caches, but this requires annotation of the > drivers in question. There

Re: [PATCH 1/2] kmemcheck v3

2008-02-07 Thread Pekka Enberg
On Feb 8, 2008 1:32 AM, Christoph Lameter <[EMAIL PROTECTED]> wrote: > But the slab layer allocates pages < PAGE_SIZE. You need to take a fault > right? So each object would need its own page? No. We allocate a shadow page for each data page which we then use as a per-byte "bitmap." For every trac

Re: [PATCH 1/2] kmemcheck v3

2008-02-07 Thread Pekka Enberg
Pekka Enberg wrote: No. We need to not track the whole page to avoid recursive faults. So for kmemcheck we absolutely do need cache_cache but we can, of course, hide that under a alloc_cache() function that only uses the extra cache when CONFIG_KMEMCHECK is enabled? Btw, one option is to have

Re: [PATCH 1/2] kmemcheck v3

2008-02-07 Thread Pekka Enberg
Hi Christoph, Christoph Lameter wrote: SLABs can be excepted from tracking? Yes. __GFP_NOTRACK can be used to suppress tracking of objects (but we still take the page fault for each access). That is required for things like DMA filled pages that are never initialized by the CPU. SLAB_NOTRAC

Re: [PATCH 1/2] kmemcheck v3

2008-02-07 Thread Christoph Lameter
On Fri, 8 Feb 2008, Vegard Nossum wrote: > The tracking that kmemcheck does is actually a byte-for-byte tracking > of whether memory has been initialized or not. Think of it as valgrind > for the kernel. We do this by "hiding" pages (marking them non-present > for for MMU) and taking the page faul

Re: [PATCH 1/2] kmemcheck v3

2008-02-07 Thread Vegard Nossum
On Feb 7, 2008 11:53 PM, Christoph Lameter <[EMAIL PROTECTED]> wrote: > > Are you sure you are not confusing tracking with tracing? It's only > > one letter different in spelling, but makes a huge difference in > > meaning :-) > > No I am quite sure what tracking is since the slab allocators have t

Re: [PATCH 1/2] kmemcheck v3

2008-02-07 Thread Christoph Lameter
On Thu, 7 Feb 2008, Vegard Nossum wrote: > > > */ > > > +#define SLAB_NOTRACK 0x0040UL/* Don't track use of > > > uninitialized memory */ > > > > Ok new exception for tracking. > > New exception? Please explain. SLABs can be excepted from tracking? > > H... You seem to assum

Re: [PATCH 1/2] kmemcheck v3

2008-02-07 Thread Vegard Nossum
Hello, Thank you for taking the time to look at this patch! On Feb 7, 2008 10:53 PM, Christoph Lameter <[EMAIL PROTECTED]> wrote: > On Thu, 7 Feb 2008, Vegard Nossum wrote: > > > --- a/include/linux/slab.h > > +++ b/include/linux/slab.h > > @@ -28,6 +28,7 @@ > > #define SLAB_DESTROY_BY_RCU 0x00

Re: [PATCH 1/2] kmemcheck v3

2008-02-07 Thread Christoph Lameter
On Thu, 7 Feb 2008, Vegard Nossum wrote: > --- a/include/linux/slab.h > +++ b/include/linux/slab.h > @@ -28,6 +28,7 @@ > #define SLAB_DESTROY_BY_RCU 0x0008UL/* Defer freeing slabs to RCU > */ > #define SLAB_MEM_SPREAD 0x0010UL/* Spread some memory > over cpuset */ >

[PATCH 1/2] kmemcheck v3

2008-02-07 Thread Vegard Nossum
Hi, With a lot of help from Ingo Molnar and Pekka Enberg over the last couple of weeks, we've been able to produce a new version of kmemcheck! General description: kmemcheck is a patch to the linux kernel that detects use of uninitialized memory. It does this by trapping every read and write to