From: Markus Armbruster <arm...@redhat.com>
VncTight member uint8_t quality is either (uint8_t)-1 for lossless or
less than 10 for lossy.
tight_detect_smooth_image() first promotes it to int, then compares
with -1. Always unequal, so we always execute the lossy code. Reads
beyond tight_conf[] and returns crap when quality is actually
lossless.
Compare to (uint8_t)-1 instead, like we do elsewhere.
Spotted by Coverity.
Signed-off-by: Markus Armbruster <arm...@redhat.com>
Signed-off-by: Gerd Hoffmann <kra...@redhat.com>
(cherry picked from commit 2e7bcdb99adbd8fc10ad9ddcf93bd2bf3c0f1f2d)
Signed-off-by: Michael Roth <mdr...@linux.vnet.ibm.com>
---
ui/vnc-enc-tight.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/ui/vnc-enc-tight.c b/ui/vnc-enc-tight.c
index e6966ae..59b59c0 100644
--- a/ui/vnc-enc-tight.c
+++ b/ui/vnc-enc-tight.c
@@ -330,7 +330,7 @@ tight_detect_smooth_image(VncState *vs, int w, int h)
} else {
errors = tight_detect_smooth_image16(vs, w, h);
}
- if (quality != -1) {
+ if (quality != (uint8_t)-1) {
return (errors < tight_conf[quality].jpeg_threshold);
}
return (errors < tight_conf[compression].gradient_threshold);
--
1.9.1
