1*4882a593Smuzhiyun<?xml version="1.0" encoding="ISO-8859-1"?> 2*4882a593Smuzhiyun<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.3//EN" 3*4882a593Smuzhiyun "http://www.oasis-open.org/docbook/xml/4.3/docbookx.dtd" [ 4*4882a593Smuzhiyun <!ENTITY % defs SYSTEM "/xserver/doc/xml/xserver.ent"> %defs; 5*4882a593Smuzhiyun]> 6*4882a593Smuzhiyun <article> 7*4882a593Smuzhiyun 8*4882a593Smuzhiyun <articleinfo> 9*4882a593Smuzhiyun <!-- Title information --> 10*4882a593Smuzhiyun <title>Scaled Window Support in DMX</title> 11*4882a593Smuzhiyun <authorgroup> 12*4882a593Smuzhiyun <author><firstname>Kevin E.</firstname><surname>Martin</surname></author> 13*4882a593Smuzhiyun <author><firstname>Rickard E.</firstname><surname>Faith</surname></author> 14*4882a593Smuzhiyun </authorgroup> 15*4882a593Smuzhiyun <pubdate>15 October 2003 (created 19 September 2003)</pubdate> 16*4882a593Smuzhiyun <releaseinfo>X Server Version &xserver.version;</releaseinfo> 17*4882a593Smuzhiyun <abstract> 18*4882a593Smuzhiyun <para> 19*4882a593Smuzhiyun This document investigates the possibility of adding scaled window 20*4882a593Smuzhiyun support to the DMX X server, thereby allowing a window or some 21*4882a593Smuzhiyun selected part of the logical DMX area to be displayed using a 22*4882a593Smuzhiyun scaling factor. For example, this might allow the contents of a 23*4882a593Smuzhiyun window to be magnified for easier viewing. In particular, scaling 24*4882a593Smuzhiyun for the VNC client is explored. <emphasis remap="it">Copyright 2003 25*4882a593Smuzhiyun by Red Hat, Inc., Raleigh, North Carolina</emphasis> 26*4882a593Smuzhiyun </para> 27*4882a593Smuzhiyun </abstract> 28*4882a593Smuzhiyun </articleinfo> 29*4882a593Smuzhiyun 30*4882a593Smuzhiyun <!-- Begin the document --> 31*4882a593Smuzhiyun <sect1><title>Introduction</title> 32*4882a593Smuzhiyun <sect2><title>DMX</title> 33*4882a593Smuzhiyun <para> 34*4882a593Smuzhiyun The DMX X server (Xdmx) is a proxy server that is designed 35*4882a593Smuzhiyun to allow X servers on multiple machines to be combined into 36*4882a593Smuzhiyun a single multi-headed X server. Combined with Xinerama, 37*4882a593Smuzhiyun these heads can appear as a single very high-resolution 38*4882a593Smuzhiyun screen. Typical applications include the creation of a 39*4882a593Smuzhiyun video wall with 16 1280x1024 displays arranged in a 40*4882a593Smuzhiyun rectangle, for a total resolution of of 5120x4096. 41*4882a593Smuzhiyun </para> 42*4882a593Smuzhiyun </sect2> 43*4882a593Smuzhiyun <sect2><title>Problem Statement</title> 44*4882a593Smuzhiyun <para> 45*4882a593Smuzhiyun Applications displayed on a physically large video wall that 46*4882a593Smuzhiyun provides high pixel-resolution may be difficult to see, 47*4882a593Smuzhiyun especially if the application is designed for use on a 48*4882a593Smuzhiyun typical desktop computer with a relatively small display 49*4882a593Smuzhiyun located close to the human operator. The goal of this paper 50*4882a593Smuzhiyun is to describe and discuss solutions to this problem. 51*4882a593Smuzhiyun </para> 52*4882a593Smuzhiyun <para> 53*4882a593Smuzhiyun The original driving problem for this work is to provide 54*4882a593Smuzhiyun scaling for the <command>vncviewer</command> application when 55*4882a593Smuzhiyun displayed using DMX (VNC scaling is currently available only 56*4882a593Smuzhiyun with the Windows client, and there is no plan to extend that 57*4882a593Smuzhiyun capability to other clients). While this specific problem 58*4882a593Smuzhiyun will be addressed in this paper, the general solution space 59*4882a593Smuzhiyun will also be explored, since this may lead to a good 60*4882a593Smuzhiyun solution not only for <command>vncviewer</command> but also for 61*4882a593Smuzhiyun other applications. 62*4882a593Smuzhiyun </para> 63*4882a593Smuzhiyun </sect2> 64*4882a593Smuzhiyun <sect2><title>Task</title> 65*4882a593Smuzhiyun <para> 66*4882a593Smuzhiyun For reference, here is the original description of the task 67*4882a593Smuzhiyun this paper addresses: 68*4882a593Smuzhiyun <itemizedlist> 69*4882a593Smuzhiyun <listitem><para>Scaled window support (for VNC) 70*4882a593Smuzhiyun <itemizedlist> 71*4882a593Smuzhiyun <listitem><para> 72*4882a593Smuzhiyun Investigate possibility of implementing a "scaled 73*4882a593Smuzhiyun window" extension: 74*4882a593Smuzhiyun <itemizedlist> 75*4882a593Smuzhiyun <listitem><para> 76*4882a593Smuzhiyun Add XCreateScaledWindow call that could be used 77*4882a593Smuzhiyun in place of XCreateWindow 78*4882a593Smuzhiyun </para></listitem> 79*4882a593Smuzhiyun <listitem><para> 80*4882a593Smuzhiyun All primitives drawn to scaled window would be 81*4882a593Smuzhiyun scaled by appropriate (integral?) scaling factor 82*4882a593Smuzhiyun </para></listitem> 83*4882a593Smuzhiyun </itemizedlist> 84*4882a593Smuzhiyun </para></listitem> 85*4882a593Smuzhiyun <listitem><para> 86*4882a593Smuzhiyun Alternate approach: special case VNC support 87*4882a593Smuzhiyun </para></listitem> 88*4882a593Smuzhiyun </itemizedlist> 89*4882a593Smuzhiyun </para></listitem> 90*4882a593Smuzhiyun </itemizedlist> 91*4882a593Smuzhiyun </para> 92*4882a593Smuzhiyun </sect2> 93*4882a593Smuzhiyun </sect1> 94*4882a593Smuzhiyun 95*4882a593Smuzhiyun <sect1><title>Previous Work</title> 96*4882a593Smuzhiyun <para> 97*4882a593Smuzhiyun This section reviews relevant previous work. 98*4882a593Smuzhiyun </para> 99*4882a593Smuzhiyun <sect2><title>VNC</title> 100*4882a593Smuzhiyun <sect3><title>Scaling under VNC</title> 101*4882a593Smuzhiyun <para> 102*4882a593Smuzhiyun When using the <command>vncviewer</command> program for Windows, it 103*4882a593Smuzhiyun is possible to specify a scaling factor (as numerator and 104*4882a593Smuzhiyun denominator). When scaling is in effect, the viewer 105*4882a593Smuzhiyun software uses StretchBlt (instead of BitBlt) to display 106*4882a593Smuzhiyun the pixels for the user. When this call is made, the 107*4882a593Smuzhiyun viewer already has received all of the pixel information 108*4882a593Smuzhiyun (at full unscaled resolution). 109*4882a593Smuzhiyun </para> 110*4882a593Smuzhiyun <para> 111*4882a593Smuzhiyun The scaling in VNC is primitive. It does not conserve 112*4882a593Smuzhiyun bandwidth, it does not treat textual information 113*4882a593Smuzhiyun differently (i.e., by using a suitably scaled font), and 114*4882a593Smuzhiyun it does not provide any anti-aliasing other than that 115*4882a593Smuzhiyun provided by the underlying (Windows-only) system library. 116*4882a593Smuzhiyun </para> 117*4882a593Smuzhiyun </sect3> 118*4882a593Smuzhiyun </sect2> 119*4882a593Smuzhiyun <sect2><title>The X Video Extension</title> 120*4882a593Smuzhiyun <para> 121*4882a593Smuzhiyun The X Video Extension is a widely-available extension to the 122*4882a593Smuzhiyun X11 protocol that provides support for streaming video. 123*4882a593Smuzhiyun Integral to this support is the ability to arbitrarily scale 124*4882a593Smuzhiyun the output. In version 2.2 of the X Video specification, 125*4882a593Smuzhiyun support for scaled still images was provided, using both 126*4882a593Smuzhiyun shared memory and traditional transport. The API for this 127*4882a593Smuzhiyun support uses calls that are quite similar to XCreateWindow, 128*4882a593Smuzhiyun XPutImage, and XShmPutImage. Currently, most of the drivers 129*4882a593Smuzhiyun implemented in XFree86 only support data in various YUV 130*4882a593Smuzhiyun formats. However, several modern video adaptors support RGB 131*4882a593Smuzhiyun as well. 132*4882a593Smuzhiyun </para> 133*4882a593Smuzhiyun <para> 134*4882a593Smuzhiyun Note, though, that the target output for this scaling is an 135*4882a593Smuzhiyun overlay plane -- so X Video provides functionality that is 136*4882a593Smuzhiyun fundamentally different from that provided by the Windows 137*4882a593Smuzhiyun StrechBlt call. 138*4882a593Smuzhiyun </para> 139*4882a593Smuzhiyun </sect2> 140*4882a593Smuzhiyun </sect1> 141*4882a593Smuzhiyun 142*4882a593Smuzhiyun <sect1><title>Possible Solutions</title> 143*4882a593Smuzhiyun <para> 144*4882a593Smuzhiyun This section briefly discusses possible solutions, including 145*4882a593Smuzhiyun major advantages and disadvantages from both the 146*4882a593Smuzhiyun implementation and the end-user programmer standpoint. 147*4882a593Smuzhiyun </para> 148*4882a593Smuzhiyun <sect2><title>VNC-like Scaling</title> 149*4882a593Smuzhiyun <sect3><title>Software Scaling</title> 150*4882a593Smuzhiyun <para> 151*4882a593Smuzhiyun The <command>vncviewer</command> application could be modified to 152*4882a593Smuzhiyun provide software scaling. This is not a general solution, 153*4882a593Smuzhiyun but it does solve one of the goals of this work. 154*4882a593Smuzhiyun </para> 155*4882a593Smuzhiyun <para> 156*4882a593Smuzhiyun A prototype of this solution was implemented and a patch 157*4882a593Smuzhiyun against <filename>vnc-3.3.7-unixsrc</filename> is available in the 158*4882a593Smuzhiyun <filename>dmx/external</filename> directory. Because of limited time 159*4882a593Smuzhiyun available for this work, all of the edge cases were not 160*4882a593Smuzhiyun considered and the solution works well mainly for integer 161*4882a593Smuzhiyun scaling. 162*4882a593Smuzhiyun </para> 163*4882a593Smuzhiyun <para> 164*4882a593Smuzhiyun Currently, <command>vncviewer</command> writes to the X display 165*4882a593Smuzhiyun with XPutImage, XCopyArea, and XFillRectangle. All 166*4882a593Smuzhiyun instances of these calls have to be aware of scaling 167*4882a593Smuzhiyun and must round correctly. In the prototype solution, 168*4882a593Smuzhiyun rounding is incorrect and can cause artifacts. 169*4882a593Smuzhiyun </para> 170*4882a593Smuzhiyun <para> 171*4882a593Smuzhiyun A better solution would be to cache all updates to the 172*4882a593Smuzhiyun desktop image in <command>vncviewer</command> and only send the 173*4882a593Smuzhiyun damaged area to the X display with XPutImage. This would 174*4882a593Smuzhiyun allow the damaged area to be computed so that rounding 175*4882a593Smuzhiyun errors do not create artifacts. This method is probably 176*4882a593Smuzhiyun similar to what is used in the Window client. (The whole 177*4882a593Smuzhiyun VNC suite is being re-written in C++ and the forthcoming 178*4882a593Smuzhiyun version 4 has not been evaluated.) 179*4882a593Smuzhiyun </para> 180*4882a593Smuzhiyun </sect3> 181*4882a593Smuzhiyun <sect3><title>Scaling with the X Video Extension</title> 182*4882a593Smuzhiyun <para> 183*4882a593Smuzhiyun The scaling in the Windows <command>vncviewer</command> application 184*4882a593Smuzhiyun makes use of a scaled blit that is supplied by the 185*4882a593Smuzhiyun underlying system library. Several video cards currently 186*4882a593Smuzhiyun provide support for a scaled blit, and some X servers 187*4882a593Smuzhiyun (including XFree86) expose this capability to applications 188*4882a593Smuzhiyun via the XvPutImage interface of the X Video Extension. 189*4882a593Smuzhiyun The capability exposed by XvPutImage results in the scaled 190*4882a593Smuzhiyun image being drawn to an overlay plane. Most video cards 191*4882a593Smuzhiyun also provide support for a scaled blit into the normal 192*4882a593Smuzhiyun output planes, but this is not exposed via XvPutImage. 193*4882a593Smuzhiyun </para> 194*4882a593Smuzhiyun <para> 195*4882a593Smuzhiyun The <command>vncviewer</command> program could be modified to use 196*4882a593Smuzhiyun the X Video Extension to provide scaling under X11 that is 197*4882a593Smuzhiyun similar to the scaling currently provided under Windows. 198*4882a593Smuzhiyun Unfortunately, Xdmx does not currently export the X Video 199*4882a593Smuzhiyun Extension, so this would not provide an immediate solution 200*4882a593Smuzhiyun usable with DMX. 201*4882a593Smuzhiyun </para> 202*4882a593Smuzhiyun <para> 203*4882a593Smuzhiyun A very early-stage proof-of-concept prototype was 204*4882a593Smuzhiyun implemented and a preliminary patch against 205*4882a593Smuzhiyun <filename>vnc-3.3.7-unixsrc</filename> is available in the 206*4882a593Smuzhiyun <filename>dmx/external</filename> directory. This prototype was 207*4882a593Smuzhiyun implemented to better understand the problems that must be 208*4882a593Smuzhiyun solved to make this solution viable: 209*4882a593Smuzhiyun <itemizedlist> 210*4882a593Smuzhiyun <listitem><para> 211*4882a593Smuzhiyun As noted under the software scaling section above, 212*4882a593Smuzhiyun <command>vncviewer</command> writes to the X display with 213*4882a593Smuzhiyun several different calls. These calls write to the 214*4882a593Smuzhiyun normal output planes and are compatible with 215*4882a593Smuzhiyun XvPutImage, which writes to an overlay plane. To 216*4882a593Smuzhiyun eliminate artifacts caused by this problem, 217*4882a593Smuzhiyun <command>vncviewer</command> should be modified so that a cached 218*4882a593Smuzhiyun copy of the desktop is available, either as a 219*4882a593Smuzhiyun client-side image or a server-side off-screen pixmap, 220*4882a593Smuzhiyun so that XvPutImage would be the only method for 221*4882a593Smuzhiyun writing to the X display. 222*4882a593Smuzhiyun </para></listitem> 223*4882a593Smuzhiyun <listitem> 224*4882a593Smuzhiyun <para> 225*4882a593Smuzhiyun Although several modern graphics adaptors support 226*4882a593Smuzhiyun hardware scaling using an RGB format (e.g., ATI 227*4882a593Smuzhiyun Radeon, nVidia, etc.), XFree86 drivers typically 228*4882a593Smuzhiyun only implement YUV formats. YUV generally compress 229*4882a593Smuzhiyun the pixel information in some way. For example, two 230*4882a593Smuzhiyun commonly implemented formats, YUY2 and UYVY provide 231*4882a593Smuzhiyun intensity information for every RGB pixel, but only 232*4882a593Smuzhiyun provide chroma and luminance information for pairs 233*4882a593Smuzhiyun of horizontal pixels. Since VNC uses 234*4882a593Smuzhiyun pixel-resolution for communicating updates on the 235*4882a593Smuzhiyun wire, additional artifacts are introduced (because 236*4882a593Smuzhiyun there may not be enough information from the wire to 237*4882a593Smuzhiyun update a pair of pixels). 238*4882a593Smuzhiyun </para> 239*4882a593Smuzhiyun <para> 240*4882a593Smuzhiyun Further, the well-known problem with YUV encoding 241*4882a593Smuzhiyun is even more evident when the image is a desktop 242*4882a593Smuzhiyun instead of a movie. For example, consider a 243*4882a593Smuzhiyun 1-pixel-wide vertical window border. If the border 244*4882a593Smuzhiyun changes in color but not intensity (e.g., because a 245*4882a593Smuzhiyun window manager uses color to indicate focus), there 246*4882a593Smuzhiyun may or may not be a change in the YUY2 image, 247*4882a593Smuzhiyun depending on the algorithm used for RGB to YUV 248*4882a593Smuzhiyun conversion and on how the border pixel is ordered in 249*4882a593Smuzhiyun the pair of pixels used by the algorithm. 250*4882a593Smuzhiyun </para> 251*4882a593Smuzhiyun <para> 252*4882a593Smuzhiyun Many of these artifacts could be eliminated if 253*4882a593Smuzhiyun <command>vncviewer</command> cached a complete RGB image of 254*4882a593Smuzhiyun the desktop, and only did the conversion to YUV for 255*4882a593Smuzhiyun properly aligned areas of damage. The remaining artifacts 256*4882a593Smuzhiyun could be eliminated if an RGB format was used with X 257*4882a593Smuzhiyun Video (which may require the extension of existing 258*4882a593Smuzhiyun XFree86 drivers to support RGB). 259*4882a593Smuzhiyun </para> 260*4882a593Smuzhiyun </listitem> 261*4882a593Smuzhiyun <listitem><para> 262*4882a593Smuzhiyun Most modern video cards support exactly one overlay 263*4882a593Smuzhiyun plane that is suitable for use with X Video. 264*4882a593Smuzhiyun Therefore, only one application can use X Video at any 265*4882a593Smuzhiyun given time. This is a severe limitation in a desktop 266*4882a593Smuzhiyun environment. 267*4882a593Smuzhiyun </para></listitem> 268*4882a593Smuzhiyun </itemizedlist> 269*4882a593Smuzhiyun </para> 270*4882a593Smuzhiyun <sect4><title>Implementing the X Video Extension for DMX</title> 271*4882a593Smuzhiyun <para> 272*4882a593Smuzhiyun The user-level API for X Video is fairly simple, but the 273*4882a593Smuzhiyun underlying support required for the full specification 274*4882a593Smuzhiyun is large. However, since the API provides a method to 275*4882a593Smuzhiyun query supported capabilities, a usable subset of X 276*4882a593Smuzhiyun Video can be implemented that would support XvPutImage 277*4882a593Smuzhiyun and little else. This would require support for the 278*4882a593Smuzhiyun following: 279*4882a593Smuzhiyun <itemizedlist> 280*4882a593Smuzhiyun <listitem><para> 281*4882a593Smuzhiyun X Video Extension API calls, including the 282*4882a593Smuzhiyun following: 283*4882a593Smuzhiyun <itemizedlist> 284*4882a593Smuzhiyun <listitem><para>XvQueryExtension</para></listitem> 285*4882a593Smuzhiyun <listitem><para>XvQueryAdaptors</para></listitem> 286*4882a593Smuzhiyun <listitem><para>XvQueryPortAttributes</para></listitem> 287*4882a593Smuzhiyun <listitem><para>XvFreeAdaptorInfo</para></listitem> 288*4882a593Smuzhiyun <listitem><para>XvListImageFormats</para></listitem> 289*4882a593Smuzhiyun <listitem><para>XvGrabPort</para></listitem> 290*4882a593Smuzhiyun <listitem><para>XvCreateImage</para></listitem> 291*4882a593Smuzhiyun <listitem><para>XvPutImage</para></listitem> 292*4882a593Smuzhiyun <listitem><para>XvShmCreateImage</para></listitem> 293*4882a593Smuzhiyun <listitem><para>XvShmPutImage</para></listitem> 294*4882a593Smuzhiyun </itemizedlist> 295*4882a593Smuzhiyun </para></listitem> 296*4882a593Smuzhiyun <listitem><para> 297*4882a593Smuzhiyun Support for querying back-end X Video Extension 298*4882a593Smuzhiyun capabilities. 299*4882a593Smuzhiyun </para></listitem> 300*4882a593Smuzhiyun <listitem><para> 301*4882a593Smuzhiyun Support for sending the image to the back-ends. 302*4882a593Smuzhiyun Because X Video requires sending full images, there 303*4882a593Smuzhiyun may be a trade-off between bandwidth limitations and 304*4882a593Smuzhiyun additional complexity to divide the image up such 305*4882a593Smuzhiyun that is scales properly. 306*4882a593Smuzhiyun </para></listitem> 307*4882a593Smuzhiyun <listitem><para> 308*4882a593Smuzhiyun Possible support for a software fall-back. For 309*4882a593Smuzhiyun example, if all of the back-ends do not support the X 310*4882a593Smuzhiyun Video Extension, software scaling can be implemented 311*4882a593Smuzhiyun such that the image is sent to the back-end with 312*4882a593Smuzhiyun XPutImage. This pathway would have poor 313*4882a593Smuzhiyun performance. 314*4882a593Smuzhiyun </para></listitem> 315*4882a593Smuzhiyun </itemizedlist> 316*4882a593Smuzhiyun </para> 317*4882a593Smuzhiyun </sect4> 318*4882a593Smuzhiyun <sect4><title>Supporting RGB formats for the X Video Extension</title> 319*4882a593Smuzhiyun <para> 320*4882a593Smuzhiyun Assuming an XFree86 driver already supports the X Video 321*4882a593Smuzhiyun Extension, and assuming the target hardware supports an 322*4882a593Smuzhiyun RGB format, then adding support for that format is 323*4882a593Smuzhiyun relatively simple and straightforward. 324*4882a593Smuzhiyun </para> 325*4882a593Smuzhiyun </sect4> 326*4882a593Smuzhiyun </sect3> 327*4882a593Smuzhiyun <sect3><title>Scaling with an XPutImageScaled Extension</title> 328*4882a593Smuzhiyun <para> 329*4882a593Smuzhiyun Instead of (or in addition to) implementing the X Video 330*4882a593Smuzhiyun Extension in DMX, one obvious solution would be to 331*4882a593Smuzhiyun implement a new extension that provides access to 332*4882a593Smuzhiyun hardware-assisted scaled blits, similar to the StretchBlt 333*4882a593Smuzhiyun call available under Windows. This call would scale RGB 334*4882a593Smuzhiyun images and would not use the overlay plane (unlike the X 335*4882a593Smuzhiyun Video Extension). 336*4882a593Smuzhiyun </para> 337*4882a593Smuzhiyun <para> 338*4882a593Smuzhiyun This approach has many of the same advantages and 339*4882a593Smuzhiyun disadvantages as the XCopyAreaScaled Extension, discussed 340*4882a593Smuzhiyun in the next section. Discussion of XPutImageScaled is 341*4882a593Smuzhiyun deferred in favor of XCopyAreaScaled for the following 342*4882a593Smuzhiyun reasons: 343*4882a593Smuzhiyun <itemizedlist> 344*4882a593Smuzhiyun <listitem><para> 345*4882a593Smuzhiyun XPutImageScaled can be emulated with XCopyAreaScaled 346*4882a593Smuzhiyun by first using XPutImage to copy the image to an 347*4882a593Smuzhiyun off-screen pixmap, and then calling XCopyAreaScaled 348*4882a593Smuzhiyun between that off-screen pixmap and the target 349*4882a593Smuzhiyun drawable. 350*4882a593Smuzhiyun </para></listitem> 351*4882a593Smuzhiyun <listitem><para> 352*4882a593Smuzhiyun Since XCopyAreaScaled would copy between two areas of 353*4882a593Smuzhiyun on-screen or off-screen memory, it has additional uses 354*4882a593Smuzhiyun and can be viewed as efficiently providing a superset 355*4882a593Smuzhiyun of XPutImageScaled functionality. 356*4882a593Smuzhiyun </para></listitem> 357*4882a593Smuzhiyun </itemizedlist> 358*4882a593Smuzhiyun </para> 359*4882a593Smuzhiyun </sect3> 360*4882a593Smuzhiyun <sect3><title>Scaling with an XCopyAreaScaled Extension</title> 361*4882a593Smuzhiyun <para> 362*4882a593Smuzhiyun As noted in the previous section, because XCopyAreaScaled 363*4882a593Smuzhiyun provides a superset of the functionality provided by 364*4882a593Smuzhiyun XPutImageScaled, we will consider this extension instead. 365*4882a593Smuzhiyun </para> 366*4882a593Smuzhiyun <para> 367*4882a593Smuzhiyun First, XCopyAreaScaled would provide for RGB scaling 368*4882a593Smuzhiyun between pixmaps (i.e., on-screen or off-screen areas of 369*4882a593Smuzhiyun memory that reside on the video card). Unlike the X Video 370*4882a593Smuzhiyun Extension, which writes into an overlay plane, 371*4882a593Smuzhiyun XCopyAreaScaled would write into the non-overlay areas of 372*4882a593Smuzhiyun the screen. Key points to consider are as follows: 373*4882a593Smuzhiyun <itemizedlist> 374*4882a593Smuzhiyun <listitem><para> 375*4882a593Smuzhiyun Because different planes are involved, the two scaling 376*4882a593Smuzhiyun operations are usually implemented in hardware 377*4882a593Smuzhiyun differently, so an XCopyAreaScaled extension could be 378*4882a593Smuzhiyun added in a manner that would neither conflict with nor 379*4882a593Smuzhiyun interact with the X Video extension in any way. 380*4882a593Smuzhiyun </para></listitem> 381*4882a593Smuzhiyun <listitem><para> 382*4882a593Smuzhiyun The XCopyAreaScaled extension provides new 383*4882a593Smuzhiyun functionality that the X Video Extension does not 384*4882a593Smuzhiyun provide. Based on anecdotal feedback, we believe that 385*4882a593Smuzhiyun many people outside the DMX and VNC communities would 386*4882a593Smuzhiyun be excited about this extension. 387*4882a593Smuzhiyun </para></listitem> 388*4882a593Smuzhiyun <listitem><para> 389*4882a593Smuzhiyun The main drawback to this extension is that it is new 390*4882a593Smuzhiyun and needs to be implemented at the driver level in 391*4882a593Smuzhiyun XFree86 for each video card to be supported. At the 392*4882a593Smuzhiyun present time, it is more likely that the X Video 393*4882a593Smuzhiyun Extension will be implemented for a particular piece 394*4882a593Smuzhiyun hardware because the X Video extension has multimedia 395*4882a593Smuzhiyun uses. However, over time, we would expect the 396*4882a593Smuzhiyun XCopyAreaScaled extension to be implemented along with 397*4882a593Smuzhiyun the X Video extension, especially if it becomes 398*4882a593Smuzhiyun popular. 399*4882a593Smuzhiyun </para></listitem> 400*4882a593Smuzhiyun <listitem><para> 401*4882a593Smuzhiyun Another drawback is that not all modern cards provide 402*4882a593Smuzhiyun support for a simple scaled blit operation. However, 403*4882a593Smuzhiyun these cards usually do provide a 3D pipeline which 404*4882a593Smuzhiyun could be used to provide this functionality in a 405*4882a593Smuzhiyun manner that is transparent to the client application 406*4882a593Smuzhiyun that is using the XCopyAreaScaled extension. However, 407*4882a593Smuzhiyun this implementation pathway would make this extension 408*4882a593Smuzhiyun somewhat more difficult to implement on certain cards. 409*4882a593Smuzhiyun </para></listitem> 410*4882a593Smuzhiyun </itemizedlist> 411*4882a593Smuzhiyun </para> 412*4882a593Smuzhiyun </sect3> 413*4882a593Smuzhiyun <sect3><title>Scaling with OpenGL</title> 414*4882a593Smuzhiyun <para> 415*4882a593Smuzhiyun Another general solution to the scaling problem is to use 416*4882a593Smuzhiyun the texture scaling found in all 3D hardware. This 417*4882a593Smuzhiyun ability is already exposed through OpenGL and can be 418*4882a593Smuzhiyun exploited by clients without X server modification (i.e., 419*4882a593Smuzhiyun other than the ability to support OpenGL). An application 420*4882a593Smuzhiyun using OpenGL would transmit the non-scaled image to the X 421*4882a593Smuzhiyun server as a texture, and would then display a single 422*4882a593Smuzhiyun non-transformed rect using that texture. This also works 423*4882a593Smuzhiyun around the single overlay problem with the X Video 424*4882a593Smuzhiyun Extension as well as the need to implement additional 425*4882a593Smuzhiyun scaled primitive extensions. 426*4882a593Smuzhiyun </para> 427*4882a593Smuzhiyun <para> 428*4882a593Smuzhiyun The downside is that most OpenGL implementations require 429*4882a593Smuzhiyun power of 2 texture sizes and this can be very wasteful of 430*4882a593Smuzhiyun memory if, for example, the application needs to scale a 431*4882a593Smuzhiyun 1025x1025 image, which would require a 2048x2048 texture 432*4882a593Smuzhiyun area (even a 640x480 image would require a 1024x512 433*4882a593Smuzhiyun texture). Another downside is that some OpenGL 434*4882a593Smuzhiyun implementations have a limited about of texture memory and 435*4882a593Smuzhiyun cannot handle textures that are very large. For example, 436*4882a593Smuzhiyun they might limit the texture size to 1024x1024. 437*4882a593Smuzhiyun </para> 438*4882a593Smuzhiyun </sect3> 439*4882a593Smuzhiyun </sect2> 440*4882a593Smuzhiyun <sect2><title>Application-transparent Scaling for DMX 441*4882a593Smuzhiyun </title><sect3><title>Back-end Scaling Without Disconnect/Reconnect</title> 442*4882a593Smuzhiyun <para> 443*4882a593Smuzhiyun VNC does scaling on the client side (in the 444*4882a593Smuzhiyun <command>vncviewer</command> application). Implementing a similar 445*4882a593Smuzhiyun solution for DMX would require support in the back-end X 446*4882a593Smuzhiyun servers and, therefore, is not a general solution. 447*4882a593Smuzhiyun </para> 448*4882a593Smuzhiyun <para> 449*4882a593Smuzhiyun XFree86 already implements some support for "scaling" that 450*4882a593Smuzhiyun could be used with DMX: if, in the XF86Config file, 451*4882a593Smuzhiyun multiple Modes are listed in the Display Subsection of the 452*4882a593Smuzhiyun Screen Section, then pressing Ctrl-Alt-Plus and 453*4882a593Smuzhiyun Ctrl-Alt-Minus can be used to iterate through the listed 454*4882a593Smuzhiyun modes. The display dimensions will change to the 455*4882a593Smuzhiyun dimensions in the Modes line, but the logical dimensions 456*4882a593Smuzhiyun of the X server (i.e., the dimensions that Xdmx knows 457*4882a593Smuzhiyun about) will not change. 458*4882a593Smuzhiyun </para> 459*4882a593Smuzhiyun <para> 460*4882a593Smuzhiyun Further, the dimensions of the XFree86 display are under 461*4882a593Smuzhiyun software control (via the XFree86-VidModeExtension), so 462*4882a593Smuzhiyun the Xdmx server could change the screen dimensions on a 463*4882a593Smuzhiyun per-display basis, thereby scaling the information on part 464*4882a593Smuzhiyun of that display. 465*4882a593Smuzhiyun </para> 466*4882a593Smuzhiyun <para> 467*4882a593Smuzhiyun However, this scaling appears to have limited use. For 468*4882a593Smuzhiyun example, assume a 4 by 4 display wall consisting of 16 469*4882a593Smuzhiyun 1280x1024 displays. If all of the back-end servers were 470*4882a593Smuzhiyun simultaneously configured to display 640x480, the left 471*4882a593Smuzhiyun hand corner of each display would be magnified, but the 472*4882a593Smuzhiyun composite result would be unreadable. Magnifying one 473*4882a593Smuzhiyun display at a time could be usable, but could have limited 474*4882a593Smuzhiyun utility, since the result would still be no larger than a 475*4882a593Smuzhiyun single display. 476*4882a593Smuzhiyun </para> 477*4882a593Smuzhiyun </sect3> 478*4882a593Smuzhiyun <sect3><title>Back-end Scaling With Disconnect/Reconnect</title> 479*4882a593Smuzhiyun <para> 480*4882a593Smuzhiyun Disconnect and reconnect features are not currently 481*4882a593Smuzhiyun supported in DMX, but are scheduled to be implemented in 482*4882a593Smuzhiyun the future. These features, combined with the 483*4882a593Smuzhiyun XFree86-VidModeExtension Extension, would allow an 484*4882a593Smuzhiyun application to do the following: 485*4882a593Smuzhiyun <itemizedlist> 486*4882a593Smuzhiyun <listitem><para> 487*4882a593Smuzhiyun Disconnect a specific back-end server (via the DMX 488*4882a593Smuzhiyun Extension), 489*4882a593Smuzhiyun </para></listitem> 490*4882a593Smuzhiyun <listitem><para> 491*4882a593Smuzhiyun reconfigure the XFree86 back-end server resolution, 492*4882a593Smuzhiyun and 493*4882a593Smuzhiyun </para></listitem> 494*4882a593Smuzhiyun <listitem><para> 495*4882a593Smuzhiyun reconnect the back-end server to DMX -- at a new 496*4882a593Smuzhiyun origin with the new screen resolution. 497*4882a593Smuzhiyun </para></listitem> 498*4882a593Smuzhiyun </itemizedlist> 499*4882a593Smuzhiyun </para> 500*4882a593Smuzhiyun <para> 501*4882a593Smuzhiyun For example, consider a display wall consisting of 16 502*4882a593Smuzhiyun 1280x1024 displays with a total resolution of 5120x4096. 503*4882a593Smuzhiyun All of the screens could be disconnected, repositioned, 504*4882a593Smuzhiyun and reconnected each at a resolution of 640x480. The 505*4882a593Smuzhiyun total resolution of the display wall would be 2560x1920, 506*4882a593Smuzhiyun allowing a view of a selected area approximately 507*4882a593Smuzhiyun one-fourth of the size of the DMX display. This change 508*4882a593Smuzhiyun would be completely application independent (except, 509*4882a593Smuzhiyun perhaps, for a DMX-aware window manager). When work at 510*4882a593Smuzhiyun the increased resolution was completed, the back-end 511*4882a593Smuzhiyun servers could be disconnected, reconfigured, and 512*4882a593Smuzhiyun reconnected for the original 5120x4096 view. 513*4882a593Smuzhiyun </para> 514*4882a593Smuzhiyun <para> 515*4882a593Smuzhiyun Support for this type of scaling can be implemented in a 516*4882a593Smuzhiyun DMX-aware X11 client assuming the DMX server support 517*4882a593Smuzhiyun arbitrary disconnect and reconnect semantics. Because 518*4882a593Smuzhiyun this application cannot be written before 519*4882a593Smuzhiyun disconnect/reconnect is implemented, this solution will 520*4882a593Smuzhiyun not be discussed further in this paper. 521*4882a593Smuzhiyun </para> 522*4882a593Smuzhiyun </sect3> 523*4882a593Smuzhiyun <sect3><title>Server-side Scaling</title> 524*4882a593Smuzhiyun <para> 525*4882a593Smuzhiyun In earlier versions of DMX, a frame buffer was maintained 526*4882a593Smuzhiyun on the server side, and XPutImage was used to move the 527*4882a593Smuzhiyun information from the server to the client (similar to some 528*4882a593Smuzhiyun early VNC implementations). The use of a server-side 529*4882a593Smuzhiyun frame buffer would allow the server to do scaling, but is 530*4882a593Smuzhiyun not a recommended solution because of overall performance 531*4882a593Smuzhiyun issues and server-side memory issues (i.e., the frame 532*4882a593Smuzhiyun buffer would be very large for large display walls). 533*4882a593Smuzhiyun </para> 534*4882a593Smuzhiyun <para> 535*4882a593Smuzhiyun Exploration of this path is not recommended. 536*4882a593Smuzhiyun </para> 537*4882a593Smuzhiyun </sect3> 538*4882a593Smuzhiyun </sect2> 539*4882a593Smuzhiyun <sect2><title>XCreateScaledWindow API</title> 540*4882a593Smuzhiyun <para> 541*4882a593Smuzhiyun The implementation of X Video Extension in DMX, and the use 542*4882a593Smuzhiyun of XvPutImage by applications requiring scaling requires 543*4882a593Smuzhiyun significant changes in DMX Further, XvPutImage is, 544*4882a593Smuzhiyun essentially a scaled blit, and it is only useful for 545*4882a593Smuzhiyun applications which are already using (or can be modified to 546*4882a593Smuzhiyun use) XPutImage. Therefore, a more general API will be 547*4882a593Smuzhiyun discussed as another possibility. 548*4882a593Smuzhiyun </para> 549*4882a593Smuzhiyun <para> 550*4882a593Smuzhiyun X applications typically create windows with the 551*4882a593Smuzhiyun XCreateWindow call. A new extension could provide an 552*4882a593Smuzhiyun XCreateScaledWindow call that could be used in place of the 553*4882a593Smuzhiyun XCreateWindow call and be otherwise transparent to the 554*4882a593Smuzhiyun application. This would allow applications, even those that 555*4882a593Smuzhiyun do not depend on XPutImage, to take advantage of window 556*4882a593Smuzhiyun scaling. In this section we describe how the call would 557*4882a593Smuzhiyun work, what transparency it provides, and how to solve the 558*4882a593Smuzhiyun potential problems that transparency creates. 559*4882a593Smuzhiyun </para> 560*4882a593Smuzhiyun <sect3><title>XCreateWindow</title> 561*4882a593Smuzhiyun <para> 562*4882a593Smuzhiyun The XCreateWindow call takes width and height as 563*4882a593Smuzhiyun parameters. An XCreateScaledWindow call could take all 564*4882a593Smuzhiyun the same parameters, with the addition of a scaling factor. 565*4882a593Smuzhiyun </para> 566*4882a593Smuzhiyun </sect3> 567*4882a593Smuzhiyun <sect3><title>XSetWindowAttributes</title> 568*4882a593Smuzhiyun <para> 569*4882a593Smuzhiyun An X11 window has several attributes that would have to be 570*4882a593Smuzhiyun scaled: 571*4882a593Smuzhiyun <itemizedlist> 572*4882a593Smuzhiyun <listitem><para>Background and border pixmaps</para></listitem> 573*4882a593Smuzhiyun <listitem><para>Border width</para></listitem> 574*4882a593Smuzhiyun <listitem><para>Cursor</para></listitem> 575*4882a593Smuzhiyun </itemizedlist> 576*4882a593Smuzhiyun </para> 577*4882a593Smuzhiyun </sect3> 578*4882a593Smuzhiyun <sect3><title>XGetWindowAttributes, XGetGeometry</title> 579*4882a593Smuzhiyun <para> 580*4882a593Smuzhiyun For transparency, calls that query the window attributes 581*4882a593Smuzhiyun should return unscaled information. This suggests that 582*4882a593Smuzhiyun all unscaled pixmaps and window attributes should be 583*4882a593Smuzhiyun cached. 584*4882a593Smuzhiyun </para> 585*4882a593Smuzhiyun <para> 586*4882a593Smuzhiyun Unfortunately, a window manager requires the scaled 587*4882a593Smuzhiyun geometry to properly decorate the window. The X server 588*4882a593Smuzhiyun can probably determine which client is acting as the 589*4882a593Smuzhiyun window manager (e.g., because that client will select 590*4882a593Smuzhiyun events that are used exclusively by the window manager). 591*4882a593Smuzhiyun However, other Scaled Window Extension aware clients may 592*4882a593Smuzhiyun also need to determine the scaled geometry. Therefore, at 593*4882a593Smuzhiyun least two additional extension calls should be 594*4882a593Smuzhiyun implemented: XGetScaledWindowAttributes and 595*4882a593Smuzhiyun XGetScaledGeometry. 596*4882a593Smuzhiyun </para> 597*4882a593Smuzhiyun </sect3> 598*4882a593Smuzhiyun <sect3><title>Popup and Child window positions</title> 599*4882a593Smuzhiyun <para> 600*4882a593Smuzhiyun Some applications may position popup and child windows 601*4882a593Smuzhiyun based on an unscaled notion of the main window geometry. 602*4882a593Smuzhiyun In this case, additional modifications to the client would 603*4882a593Smuzhiyun be required. 604*4882a593Smuzhiyun </para> 605*4882a593Smuzhiyun </sect3> 606*4882a593Smuzhiyun <sect3><title>Events</title> 607*4882a593Smuzhiyun <para> 608*4882a593Smuzhiyun Most events (e.g., for mouse motion) return information 609*4882a593Smuzhiyun about the coordinates at which the even occurred. These 610*4882a593Smuzhiyun coordinates would have to be modified so that unscaled 611*4882a593Smuzhiyun values were presented to the client. 612*4882a593Smuzhiyun </para> 613*4882a593Smuzhiyun </sect3> 614*4882a593Smuzhiyun <sect3><title>Implementation</title> 615*4882a593Smuzhiyun <para> 616*4882a593Smuzhiyun There are many implementation issues, some of which are 617*4882a593Smuzhiyun similar to the issues involved in implementing the X Video 618*4882a593Smuzhiyun Extension for DMX. The window contents must be scaled, 619*4882a593Smuzhiyun either by performing all operations to a frame buffer and 620*4882a593Smuzhiyun then writing the image to the display (perhaps using 621*4882a593Smuzhiyun hardware scaling support), or by modifying all of the 622*4882a593Smuzhiyun various drawing operations to perform scaling. Because of 623*4882a593Smuzhiyun the complexity involved, the frame buffer option is 624*4882a593Smuzhiyun recommended. 625*4882a593Smuzhiyun </para> 626*4882a593Smuzhiyun </sect3> 627*4882a593Smuzhiyun </sect2> 628*4882a593Smuzhiyun </sect1> 629*4882a593Smuzhiyun 630*4882a593Smuzhiyun <sect1><title>Conclusion and Recommendations 631*4882a593Smuzhiyun </title><para> 632*4882a593Smuzhiyun We recommend a three phase implementation strategy, based on 633*4882a593Smuzhiyun how an application could be written to take advantage of 634*4882a593Smuzhiyun scaling: 635*4882a593Smuzhiyun <orderedlist> 636*4882a593Smuzhiyun <listitem> 637*4882a593Smuzhiyun <para> 638*4882a593Smuzhiyun The XCopyAreaScaled extension should be implemented, since 639*4882a593Smuzhiyun this is the ideal solution for applications like VNC, and 640*4882a593Smuzhiyun since making use of this extension will require minimal 641*4882a593Smuzhiyun changes to applications that already use XPutImage or 642*4882a593Smuzhiyun XCopyArea. 643*4882a593Smuzhiyun </para> 644*4882a593Smuzhiyun <para> 645*4882a593Smuzhiyun The initial implementation work would include the design 646*4882a593Smuzhiyun of the X protocol extension, writing this up in the 647*4882a593Smuzhiyun usual format for extension documentation, implementation 648*4882a593Smuzhiyun of the protocol transport pieces in XFree86, 649*4882a593Smuzhiyun implementation of a software fall-back in XFree86 and 650*4882a593Smuzhiyun DMX, one example hardware implementation for XFree86, 651*4882a593Smuzhiyun and implementation of support for this extension in DMX. 652*4882a593Smuzhiyun </para> 653*4882a593Smuzhiyun <para> 654*4882a593Smuzhiyun We suggest implementing the extension first on the ATI 655*4882a593Smuzhiyun Radeon cards. However, since these cards do not provide 656*4882a593Smuzhiyun a 2D scaled blit primitive, the implementation would 657*4882a593Smuzhiyun have to make use of the 3D texture engine to emulate a 658*4882a593Smuzhiyun scaled blit. This is recommended, since other modern 659*4882a593Smuzhiyun graphics cards also do not provide a simple 2D scaled 660*4882a593Smuzhiyun blit operation and an example of the more difficult 661*4882a593Smuzhiyun implementation pathway would be helpful to others. 662*4882a593Smuzhiyun </para> 663*4882a593Smuzhiyun </listitem> 664*4882a593Smuzhiyun <listitem> 665*4882a593Smuzhiyun <para> 666*4882a593Smuzhiyun Until XCopyAreaScaled is widely supported, applications 667*4882a593Smuzhiyun that require scaling will have to fall back to another 668*4882a593Smuzhiyun scaling method. We suggest OpenGL as the first fall-back 669*4882a593Smuzhiyun method because it is widely available and supported by 670*4882a593Smuzhiyun DMX. 671*4882a593Smuzhiyun </para> 672*4882a593Smuzhiyun <para> 673*4882a593Smuzhiyun A project centered around OpenGL-based scaling would 674*4882a593Smuzhiyun implement this scaling in VNC as an example. This work 675*4882a593Smuzhiyun would include re-writing the <command>vncviewer</command> 676*4882a593Smuzhiyun rendering engine to cache a master copy of the desktop 677*4882a593Smuzhiyun image for all operations. 678*4882a593Smuzhiyun </para> 679*4882a593Smuzhiyun </listitem> 680*4882a593Smuzhiyun <listitem> 681*4882a593Smuzhiyun <para> 682*4882a593Smuzhiyun Since OpenGL is not implemented everywhere, and may not 683*4882a593Smuzhiyun provide hardware-assisted performance in every 684*4882a593Smuzhiyun implementation, an application that requires scaling 685*4882a593Smuzhiyun should also fall back to using the X Video Extension. 686*4882a593Smuzhiyun </para> 687*4882a593Smuzhiyun <para> 688*4882a593Smuzhiyun This project would add support for the X Video Extension 689*4882a593Smuzhiyun to DMX and would add support to VNC to take advantage of 690*4882a593Smuzhiyun this extension without introducing artifacts. This 691*4882a593Smuzhiyun would require modifying the <command>vncviewer</command> rendering 692*4882a593Smuzhiyun engine to cache a master copy of the desktop image for 693*4882a593Smuzhiyun all operations. This project should also add support 694*4882a593Smuzhiyun for the RGB format to at least one XFree86 driver (e.g., 695*4882a593Smuzhiyun ATI Radeon). 696*4882a593Smuzhiyun </para> 697*4882a593Smuzhiyun <para> 698*4882a593Smuzhiyun The X Video Extension is one of the few popular 699*4882a593Smuzhiyun extensions that DMX does not support. We recommend 700*4882a593Smuzhiyun implementing the X Video Extension even if scaling is 701*4882a593Smuzhiyun the specific goal of that work. 702*4882a593Smuzhiyun </para> 703*4882a593Smuzhiyun </listitem> 704*4882a593Smuzhiyun </orderedlist> 705*4882a593Smuzhiyun </para> 706*4882a593Smuzhiyun <para> 707*4882a593Smuzhiyun We do <emphasis>not</emphasis> recommend implementation of the 708*4882a593Smuzhiyun XCreateScaledWindow extension because of the complexity 709*4882a593Smuzhiyun involved. We do <emphasis>not</emphasis> recommend implementation of the 710*4882a593Smuzhiyun XPutImageScaled extension because it requires the same amount 711*4882a593Smuzhiyun of work as the XCopyAreaScaled extension, but provides less 712*4882a593Smuzhiyun functionality. Further, server-side scaling with a large 713*4882a593Smuzhiyun frame buffer is <emphasis>not</emphasis> recommended because of the 714*4882a593Smuzhiyun performance implications. 715*4882a593Smuzhiyun </para> 716*4882a593Smuzhiyun <para> 717*4882a593Smuzhiyun The back-end scaling, especially with disconnect/reconnect 718*4882a593Smuzhiyun support should be explored in the future after 719*4882a593Smuzhiyun disconnect/reconnect is implemented, but not at the present 720*4882a593Smuzhiyun time. 721*4882a593Smuzhiyun </para> 722*4882a593Smuzhiyun </sect1> 723*4882a593Smuzhiyun 724*4882a593Smuzhiyun </article> 725*4882a593Smuzhiyun <!-- Local Variables: --> 726*4882a593Smuzhiyun <!-- fill-column: 72 --> 727*4882a593Smuzhiyun <!-- End: --> 728