[PATCH v3 3/5] drm: Avoid undefined behavior on u16 multiplication in drm_vram_helper_mode_valid_internal()

[Krzysztof Karas]

Fields hdisplay and vdisplay are defined as u16 and their
multiplication causes implicit promotion to signed 32-bit value,
which may overflow and cause undefined behavior.

Prevent possible undefined behavior by explicitly casting one of
the operands to (unsigned int) type.

Fixes: 80f7c3f77697 ("drm/vram: Add helpers to validate a display mode's memory 
requirements")
Cc: Thomas Zimmermann <tzimmerm...@suse.de>
Cc: <sta...@vger.kernel.org> # v5.7+
Signed-off-by: Krzysztof Karas <krzysztof.ka...@intel.com>
---
 drivers/gpu/drm/drm_gem_vram_helper.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/drm_gem_vram_helper.c 
b/drivers/gpu/drm/drm_gem_vram_helper.c
index b04cde4a60e7..4b144e9603b8 100644
--- a/drivers/gpu/drm/drm_gem_vram_helper.c
+++ b/drivers/gpu/drm/drm_gem_vram_helper.c
@@ -967,7 +967,7 @@ drm_vram_helper_mode_valid_internal(struct drm_device *dev,
 
        max_fbpages = (vmm->vram_size / 2) >> PAGE_SHIFT;
 
-       fbsize = mode->hdisplay * mode->vdisplay * max_bpp;
+       fbsize = (unsigned int)mode->hdisplay * mode->vdisplay * max_bpp;
        fbpages = DIV_ROUND_UP(fbsize, PAGE_SIZE);
 
        if (fbpages > max_fbpages)
-- 
2.34.1


-- 
Best Regards,
Krzysztof