From: Prasad J Pandit <pjp@fedoraproject.org>
Date: Thu, 3 Dec 2015 13:24:17 +0000 (+0530)
Subject: ui: vnc: avoid floating point exception
X-Git-Tag: qemu-xen-4.5.3~13
X-Git-Url: http://xenbits.xensource.com/gitweb?a=commitdiff_plain;h=ca4b8af6798b1e21c3ef4dfa8df7fbb133f55396;p=qemu-xen.git

ui: vnc: avoid floating point exception

While sending 'SetPixelFormat' messages to a VNC server,
the client could set the 'red-max', 'green-max' and 'blue-max'
values to be zero. This leads to a floating point exception in
write_png_palette while doing frame buffer updates.

Reported-by: Lian Yihan <lianyihan@360.cn>
Signed-off-by: Prasad J Pandit <pjp@fedoraproject.org>
Reviewed-by: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---

diff --git a/ui/vnc.c b/ui/vnc.c
index 0c7d75f1e9..647a356cd4 100644
--- a/ui/vnc.c
+++ b/ui/vnc.c
@@ -2020,15 +2020,15 @@ static void set_pixel_format(VncState *vs,
         return;
     }
 
-    vs->client_pf.rmax = red_max;
+    vs->client_pf.rmax = red_max ? red_max : 0xFF;
     vs->client_pf.rbits = hweight_long(red_max);
     vs->client_pf.rshift = red_shift;
     vs->client_pf.rmask = red_max << red_shift;
-    vs->client_pf.gmax = green_max;
+    vs->client_pf.gmax = green_max ? green_max : 0xFF;
     vs->client_pf.gbits = hweight_long(green_max);
     vs->client_pf.gshift = green_shift;
     vs->client_pf.gmask = green_max << green_shift;
-    vs->client_pf.bmax = blue_max;
+    vs->client_pf.bmax = blue_max ? blue_max : 0xFF;
     vs->client_pf.bbits = hweight_long(blue_max);
     vs->client_pf.bshift = blue_shift;
     vs->client_pf.bmask = blue_max << blue_shift;