likely signedness bug in drm and nvidia drivers

2015-07-21 Thread Daniel Thompson
On 21/07/15 03:44, Ilia Mirkin wrote: > I think you're right. The intent is to mask off the bits above> bits_per_pixel. So if bits_per_pixel is 24, the mask would be> 0xff00. If it's 16, then the mask would be 0x. If it's 32,> then the mask is 0.> > In reality, bits_per_pixel is alm

likely signedness bug in drm and nvidia drivers

2015-07-20 Thread Rasmus Villemoes
Hi, The files drivers/gpu/drm/nouveau/nv50_fbcon.c drivers/gpu/drm/nouveau/nvc0_fbcon.c drivers/video/fbdev/nvidia/nv_accel.c all contain a right-shift of ~0 (aka -1) - just grep for '~0 >>'. gcc always does arithmetic right shift of signed types, which means that the result is always -1 a

likely signedness bug in drm and nvidia drivers

2015-07-20 Thread Ilia Mirkin
I think you're right. The intent is to mask off the bits above bits_per_pixel. So if bits_per_pixel is 24, the mask would be 0xff00. If it's 16, then the mask would be 0x. If it's 32, then the mask is 0. In reality, bits_per_pixel is almost exclusively 32, which will end up with a mask