Re: [Mesa-dev] [PATCH] llvmpipe: fix overflow bug in total texture size computation

2012-09-20 Thread Jose Fonseca
Looks good to me. Thanks. Jose - Original Message - > v2: use uint64_t for the total_size variable, per Jose. > > Also add two earlier checks for exceeding the max texture size. > For example a 1K^3 RGBA volume would overflow the lpr->image_stride > variable. > > Use simple algebra to a

[Mesa-dev] [PATCH] llvmpipe: fix overflow bug in total texture size computation

2012-09-20 Thread Brian Paul
v2: use uint64_t for the total_size variable, per Jose. Also add two earlier checks for exceeding the max texture size. For example a 1K^3 RGBA volume would overflow the lpr->image_stride variable. Use simple algebra to avoid overflow in intermediate values. So instead of "x * y > z" use "x > z /

Re: [Mesa-dev] [PATCH] llvmpipe: fix overflow bug in total texture size computation

2012-09-20 Thread Brian Paul
On 09/20/2012 05:15 AM, Jose Fonseca wrote: Good find. Stride could be bigger than 1024 yet not a 1024 multiple. I explored using a 1024 scale factor, and it works (it wasn't based on stride), but just using uint64_t is much simpler. I think it would be simpler to use uint64_t. Should cove

Re: [Mesa-dev] [PATCH] llvmpipe: fix overflow bug in total texture size computation

2012-09-20 Thread Jose Fonseca
Good find. Stride could be bigger than 1024 yet not a 1024 multiple. I think it would be simpler to use uint64_t. Should cover all cases. Jose - Original Message - > > The particular failing case was a 1024^3 3D texture. > 1024*1024*1024*4 > (RGBA) = 4GB which overflows to zero in a 3

Re: [Mesa-dev] [PATCH] llvmpipe: fix overflow bug in total texture size computation

2012-09-19 Thread Brian Paul
The particular failing case was a 1024^3 3D texture. 1024*1024*1024*4 (RGBA) = 4GB which overflows to zero in a 32-bit uint. The limit used to be 512^3 but Jose bumped it up in commit bc8509b4. I think we could look if the img_stride is > 1024 then do an alternate computation in kilobytes i

Re: [Mesa-dev] [PATCH] llvmpipe: fix overflow bug in total texture size computation

2012-09-19 Thread Roland Scheidegger
Good catch. I'm not sure though using size_t casts (and size_t sized total_value) is good enough since this could also be hit on archs using 32bit size_t? Though i guess using 64bit arithmetic on 32bit would be sort of slow... Roland Am 19.09.2012 21:30, schrieb Brian Paul: > Add size_t casts wh

[Mesa-dev] [PATCH] llvmpipe: fix overflow bug in total texture size computation

2012-09-19 Thread Brian Paul
Add size_t casts when multiplying slice size by number of slices to avoid 32-bit uint overflow. This bug has been here for a long time. But before the recent proxy texture changes, core Mesa was detecting that the texture was too large and we never got this far. Fixes https://bugs.freedesktop.org