Hi Anthony, sorry for the delayed response to your delayed response. I actually haven't been working much on the remote display stuff in a while, though it's something I'm hoping to get involved with again time permitting. I'm cc'ing Dustin Byford and Johnson Liu, VMware engineers who handle a lot of our remote display work these days.
Also, I'm wondering if it would make sense to set up a separate list so we don't spam qemu-devel with this discussion, and also so that we can all subscribe to it rather than having to be cc'ed and so forth. If you think that's a good idea let me know. We can host the list @vmware.com, or put it somewhere else (nongnu.org? or elsewhere?) if that would be preferred. > > (1) no support for keyboard scancodes (VNC uses higher > level symbolic keys > > which aren't necessarily 1:1 mappable back to their > scancode equivalents in > > all the scenarios we care about) > > Yes, right now, we do not address this. So how do you handle the keyboard today? Do you just hardcode the reverse mapping back to a pc105 keyboard? It would be nice to get something upstream so that all clients have the ability to send scancode values. We have an extension in place which we can just document. We use 16-bit "virtual scan code values" (they're 16-bit more for convenience of implementation, i.e. directly mapped to a lookup table). I believe the values we actually are equivalent/identical to PC scancodes. Meanwhile, in our newest version of Workstation (6.0) we added server-side support for emulating different keyboard layouts when standard VNC clients connect. Basically you can set a server config option to tell the server "pretend that a de105 keyboard is attached" and it will do the reverse translation of KeySym to Scancode accordingly. This is good enough for some people, but the ideal situation is obviously to do the translation on the client-side where you can actually detect the local keyboard and do the right thing (which is what the VMware remote client does). > > (2) no relative mouse support > > > > We have an extension that addresses this. As do we ;-). > > (3) no alpha cursor support > > > > There is a pseudo-encoding that provides alpha cursors with a 1-bit > alpha channel. An 8 bit alpha channel would be nice of course. More than nice, it's pretty much required if you want to draw some modern Linux guest cursors correctly. The cheap hack of treating < 255 alpha as fully transparent doesn't really work that well, especially for things an I-Bar cursor with shadow. We do support the standard VNC pseudo-cursor encoding as of Workstation 5.5 and ESX 3.0, and if the guest sets an alpha cursor and the client doesn't support our alpha cursor extension, we strip out the alpha channel and send it using the standard VNC encoding. But it doesn't look that great. > > (4) no dynamic pixel depth/bpp change support > > > > This is something I do want to fix. This isn't just a server-side > issue, there is also a race in the protocol when a client uses > SetPixelFormat. A proper notification of when the server > switches pixel > format after a SetPixelFormat would solve this problem. So yeah in the original protocol, depending on how you read it, it's just illegal to ever send a SetPixelFormat after the original request. But at any rate, client->server format requests aren't as interesting as server->client requests. You only do the former if the client decides for some reason that it wants a different pixel format, because e.g. it decides it doesn't have enough bandwidth or something. Server->client changes OTOH, can be guest driven, e.g. there are several mode changes when booting a Windows guest. When a standard VNC client is connected, we hide these from the client by doing pixel format conversion on the server, which is expensive, and potentially wastes bandwidth on the wire as well (e.g. when the guest is in 640x480x256 but the client wants 32bpp). When a client which supports our "VMModeChange" extension is connected, we just send down the mode change at the start of the next UpdateRequest. We do implement the standard VNC desktop size encoding (since otherwise we'd have to just drop the connection every time the guest changed display size). But getting back to your original point, since our client never sends mode change requests upstream, so we haven't addressed the race that you describe. I agree that if you ever want to drive these on the client you need to do something about that. > > (5) No notion of multiplexing displays on a single port. > > > > Right. ATM, I don't really plan on addressing this. So, the old Connectix Virtual Server product (which was in beta when Microsoft snapped them up) had a super slick way of dealing with this, that we actually joked about doing but never seriously thought anyone would do. They rendered an entire "picker" UI using VNC primitives. The UI would show live thumbnails of each VM, and then when you click on one of the tiles, you'd get redirected to that console. You couldn't go back out to the picker, but still, it was pretty neat. I've since looked into doing the same thing, but the closest thing to any VNC UI/widget rendering library that I've found is TclRFB (which, as the name would imply, is written in Tcl). Anyway, what you really want is a way to say "you've connected to a multi-home server, here's a list of VMs available (with arbitrary or perhaps structured metadata attached to each VM tag), which one do you want?". And then have the ability for the client ot pick. But you can't really do this sanely with just extensions, I think this is a good candidate for actually extending the base protocol. > We actually can do this. I've reserved the 255 client > message ID which > I'm using to multiplex various other messages. Yeah, we did this too. We use '127' as the client message ID, but it's otherwise exactly as you describe. It's a nice performance boost, though it makes implementing CopyRect encoding much trickier <g>. > I currently use this to implement a shared memory pseudo-encoding for VNC. > The idea is that on localhost, the server can render directly to a shared > memory segment that the client also maps. If the one is smart about how > this shared memory segment is chosen, it can be used as an XShmImage. Yeah we did the exact same thing, with a slight twist. We get the server to allocate the SHM segment, and the client just attaches to it. If you do it this way, you don't need a new client->server message, the server just sends the client the shm id if the client supports the "SharedFB" extension (that's what we named it). There's some complexity involved with how the server can actually figure out whether the client is on the same machine (e.g. if the client is tunneling over SSH, this is not easy to do). So in fact what we do is just if the client supports SharedFBInfo and SharedFBRect, the server just sends along the messages. The client then tries to map the segment described in SharedFBInfo. If it fails, it just sends a new encoding set with those encodings removed, and the server starts sending real pixel data instead. Yes, this means that we always try and fail sharedfb so there's a couple extra roundtrips before remote clients actually get real pixel data. This was deemed acceptable (by me ;-) for our use cases since most networks are fast enough. But if this is too hackish for your taste we could bake in something better. > I believe http://wiki.multimedia.cx/index.php?title=VMNC > covers some of > your extensions right? Have you gotten the appropriate encodings > reserved in the RFB spec? I don't want to support any encodings that > aren't properly registered so this would be an important first step. So once upon a time, we tried to register encodings, but we didn't get very far. It sounds like perhaps the process has improved since then. I will try to get the ball rolling on this again. > I'm certainly happy to work toward a common set of VNC extensions to > cover virtualization. It's in everyones benefit to ensure that the > largest number of clients Just Work out of the box. Agreed. I look forward to hearing more of your thoughts on this subject. -- Ramesh
