Installing VMware Tools On Debian Lenny 5.0.2 With Gnome Desktop On ESX Server 3.5 Update 4

From time to time, installing VMware Tools on a Linux guest will cause you some grief. While there are lots of howto's, usually they're for VMware Workstation. Here's one that works in Debian/Lenny 5.0.2 on ESX Server 3.5 Update 4.

Important: before making changes on the guest, create a snapshot on your ESX Server. This way you can revert the changes if you get something wrong.

1. Install required Debian packages

On the Debian guest, open a root terminal:

Applications -> Accessories -> Root Terminal

Note: if you've installed the openssh server, you can connect using a Putty/SSH client if you prefer.

Before running the VMware Tools installation script, we need to install a few Debian packages:

apt-get install binutils gcc-4.1-base make linux-headers-$(uname -r)

To allow the vmware-user daemon to start after the install and prevent it from failing thereafter, do the following:

ln -s /usr/lib/libexpat.so.1 /usr/lib/libexpat.so.0

2. Copy and extract the VMware Tools installation files to a local directory

On your Virtual Infrastructure Client, from the menu load the VMware Tools virtual CD:

Inventory -> Virtual Machine -> Install/Upgrade VMware Tools

 (You should see a CD-ROM icon appear on your Debian desktop. Just ignore it.)

Create and/or navigate to a directory where you'd like to store the installation files (e.g., /var/installs), then extract the files:

mkdir /var/installs
cd /var/installs
tar zxvf /media/cdrom/VMwareTools-3.5.0-153875.tar.gz

Unmount the CD-ROM by canceling the VMware Tools Install/Upgrade on the VI-Client:

Inventory -> Virtual Machine -> End VMware Tools Install

Ignore the mount error on the Debian desktop by clicking 'Close' on the window that pops up.

3. Install VMware Tools

Important: if you are running a Putty/SSH session connected to the guest, at this point you must run the next three commands from a terminal session inside the guest, otherwise you'll be disconnected by the installation script after it reinitializes the network. In Debian/Lenny, Applications -> Accessories -> Root Terminal.

The VMware Tools Install detects and uses the 'CC' environment variable so it knows which compiler to use. We must specify gcc-4.1:

export CC=/usr/bin/gcc-4.1

(If you make a mistake, you'll be prompted during the install for the right file; no problem, just enter /usr/bin/gcc-4.1.)

Switch to the installation directory and run the installation script:

cd /var/installs/vmware-tools-distrib
./vmware-install.pl

Press [enter] on all of the default answers (even the startup display resolution) until the install finishes.

Note: on Debian/Lenny, if you select 1024x768 (the default), it will still boot up in 800x600, which IMO is better on ESX because the console window might not fit a 1024x768 desktop. You can always change the resolution later by going to System -> Preferences -> Screen Resolution.

4. Fixing the mouse

At this point, everything should be configured properly except for the mouse (although the network might be down, type ifup eth0 to bring it up if needed before rebooting). You'll find the mouse to be slow and jittery, and will hesitate going in and out of the guest window. Even after you reboot, it will stay this way, so lets fix it.

Edit the /etc/X11/xorg.conf file:

gedit /etc/X11/xorg.conf &

Comment-out the InputDevice section:

#Section "InputDevice"
#  Driver "vmmouse"
# Identifier "VMware Mouse"
#  Option "Buttons" "5"
#  Option "Device" "/dev/input/mice"
#  Option "Protocol" "IMPS/2"
#  Option "ZAxisMapping" "4 5"
#  Option "Emulate3Buttons" "true"
#EndSection

Copy and paste should now be working between the host and the guest. Copy/paste the following into the xorg.conf file: 

Section "InputDevice"
  Driver     "vmmouse"
  Identifier "VMware Mouse"
  Option     "CorePointer"
  Option     "Device"          "/dev/input/mice"
  Option     "Protocol"        "ps/2"
  Option     "Emulate3Buttons" "true"
EndSection

Now save the file and quit, then reboot the system:

reboot

That's it!

Note: our updated InputDevice section also works in Debian/Etch.

If you continue to have mouse problems:

Make sure the correct vmmouse driver is being used (on my first run, I ran the install differently than outlined here and the mouse driver didn't copy over):

ls -l /usr/lib/vmware-tools/configurator/XOrg/7.3/vmmouse_drv.so
ls -l /usr/lib/xorg/modules/input/vmmouse_drv.so

If the files are different, do this:

cd /usr/lib/xorg/modules/input
mv vmmouse_drv.so vmmouse_drv.so.orig
cp -p /usr/lib/vmware-tools/configurator/XOrg/7.3/vmmouse_drv.so .

You might need to fiddle with the InputDevice section in xorg.conf if problems persist. Google for other solutions.

TIP:

To simulate 1024x768 resolution in a 800x600 window, go to:

System -> Preferences -> Appearance -> Fonts (tab) -> Details...

Change Resolution (dots per inch) to 75 or 76.

Then adjust the icon size:

Applications -> System Tools -> File Browser
Edit -> Preferences

Change Icon View Defaults/Default zoom level to 75%.