slow request time on ISPConfig server

[torusturtle]

Hi

I'm using ISPConfig on a Xen virtual server. I installed it on Ubuntu 6.06 LTS with the perfect setup instructions.

Unfortunatelly is the initial response time very slow.
For example when loggin in with ssh it takes several seconds until user and password prompt is displayed.

Other virtual servers on the same main server responde instantly.

The some problem occurs with http, ftp or pop requests. I'm getting "waiting for the server" for several seconds. As soon as the connection is established the page/file/mail is loaded very fast.

Here are some outputs from my server:

uptime
14:23:09 up 130 days, 13:40, 1 user, load average: 0.06, 0.06, 0.04

top
top - 14:24:02 up 130 days, 13:41, 1 user, load average: 0.30, 0.12, 0.06
Tasks: 90 total, 2 running, 88 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3% us, 0.0% sy, 0.0% ni, 99.7% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 261056k total, 211808k used, 49248k free, 8576k buffers
Swap: 262136k total, 97656k used, 164480k free, 89980k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20738 root 16 0 7724 2504 2048 R 0.3 1.0 0:00.05 sshd
1 root 16 0 1472 292 260 S 0.0 0.1 0:08.39 init
2 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:01.24 ksoftirqd/0
4 root RT 0 0 0 0 S 0.0 0.0 0:00.04 watchdog/0
5 root 10 -5 0 0 0 S 0.0 0.0 0:00.86 events/0
6 root 10 -5 0 0 0 S 0.0 0.0 0:00.01 khelper
7 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kthread
8 root 10 -5 0 0 0 S 0.0 0.0 0:00.01 xenwatch
9 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 xenbus
23 root 10 -5 0 0 0 S 0.0 0.0 0:00.20 kblockd/0
64 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/0
63 root 15 0 0 0 0 S 0.0 0.0 1:56.65 kswapd0
582 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kseriod
691 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 kpsmoused
1612 root 15 0 0 0 0 S 0.0 0.0 2:26.62 kjournald
1758 root 12 -4 2112 52 48 S 0.0 0.0 0:00.25 udevd
2914 root 25 0 1740 48 44 S 0.0 0.0 0:00.00 courierlogger
2915 root 16 0 1844 144 116 S 0.0 0.1 0:00.00 authdaemond.pla
2920 root 15 0 2040 356 292 S 0.0 0.1 0:14.33 authdaemond.pla
2921 root 15 0 2040 356 292 S 0.0 0.1 0:14.28 authdaemond.pla
2922 root 16 0 2040 356 292 S 0.0 0.1 0:14.46 authdaemond.pla
2923 root 15 0 2040 356 292 S 0.0 0.1 0:13.78 authdaemond.pla
2924 root 16 0 2040 356 292 S 0.0 0.1 0:13.85 authdaemond.pla
2935 root 16 0 1832 188 144 S 0.0 0.1 0:00.30 couriertcpd
2937 root 18 0 1740 308 256 S 0.0 0.1 0:00.20 courierlogger
2955 root 15 0 1836 84 80 S 0.0 0.0 0:00.01 couriertcpd
2957 root 25 0 1744 88 84 S 0.0 0.0 0:00.00 courierlogger
2970 root 15 0 1836 192 144 S 0.0 0.1 0:17.51 couriertcpd
2974 root 18 0 1740 308 256 S 0.0 0.1 0:23.14 courierlogger
2990 root 19 0 1836 84 80 S 0.0 0.0 0:00.01 couriertcpd
2992 root 24 0 1740 88 84 S 0.0 0.0 0:00.00 courierlogger
3224 root 16 0 6140 1224 876 S 0.0 0.5 0:00.20 saslauthd
3226 root 16 0 6140 1292 944 S 0.0 0.5 0:00.24 saslauthd
3227 root 16 0 6140 1356 996 S 0.0 0.5 0:00.23 saslauthd
3228 root 17 0 6140 1292 944 S 0.0 0.5 0:00.21 saslauthd
3229 root 16 0 6140 1224 876 S 0.0 0.5 0:00.22 saslauthd
3243 root 16 0 4768 300 212 S 0.0 0.1 0:05.02 sshd
3764 root 17 0 1564 92 88 S 0.0 0.0 0:00.00 getty
25244 root 24 0 1888 76 72 S 0.0 0.0 0:00.01 dd
25246 klog 23 0 1476 32 28 S 0.0 0.0 0:00.00 klogd
27130 root 16 0 2124 744 616 S 0.0 0.3 0:00.46 cron
23324 root 15 0 0 0 0 S 0.0 0.0 0:05.98 pdflush
13613 root 16 0 12760 1736 1252 S 0.0 0.7 0:00.13 ispconfig_httpd
13614 root 16 0 2428 984 848 S 0.0 0.4 0:06.48 ispconfig_wconf
13621 admispco 15 0 13180 5148 1888 S 0.0 2.0 0:59.66 ispconfig_httpd
13720 root 16 0 59844 7764 5912 S 0.0 3.0 0:01.63 apache2
13859 bind 21 0 55644 20m 1508 S 0.0 8.1 3:32.34 named
13915 admispco 16 0 2620 1116 860 S 0.0 0.4 0:00.47 freshclam
13952 admispco 15 0 13176 5340 1912 S 0.0 2.0 0:48.15 ispconfig_httpd
19434 root 15 0 0 0 0 S 0.0 0.0 0:04.80 pdflush
26912 root 16 0 9052 800 604 S 0.0 0.3 0:14.72 nscd
26913 root 16 0 9052 800 604 S 0.0 0.3 0:00.03 nscd
26914 root 16 0 9052 800 604 S 0.0 0.3 0:02.50 nscd
26915 root 16 0 9052 800 604 S 0.0 0.3 0:02.80 nscd
26916 root 16 0 9052 800 604 S 0.0 0.3 0:02.27 nscd
26917 root 16 0 9052 800 604 S 0.0 0.3 0:02.50 nscd
26918 root 16 0 9052 800 604 S 0.0 0.3 0:02.94 nscd
26919 root 16 0 9052 800 604 S 0.0 0.3 0:02.92 nscd
20045 root 18 0 1604 408 360 S 0.0 0.2 0:00.53 cronolog
19502 root 25 0 2460 960 956 S 0.0 0.4 0:00.04 mysqld_safe
19563 mysql 16 0 252m 15m 3148 S 0.0 6.0 7:46.37 mysqld
19564 root 25 0 1556 440 428 S 0.0 0.2 0:00.01 logger
25574 www-data 15 0 62044 14m 10m S 0.0 5.8 0:00.80 apache2


netstat -tap (this one takes very long and i aborded it at the end)
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
tcp 0 0 *:mysql *:*
LISTEN 19563/mysqld
tcp 0 0 *:81 *:*
LISTEN 13613/ispconfig_htt
tcp 0 0 *:ftp *:*
LISTEN 21208/proftpd: (acc
tcp 0 0 pollux.mydomain.c:domain *:*
LISTEN 13859/named
tcp 0 0 82.150.62.164:domain *:*
LISTEN 13859/named
tcp 0 0 82.150.62.163:domain *:*
LISTEN 13859/named
tcp 0 0 82.150.62.162:domain *:*
LISTEN 13859/named
tcp 0 0 82.150.62.83:domain *:*
LISTEN 13859/named
tcp 0 0 82.150.62.82:domain *:*
LISTEN 13859/named
tcp 0 0 82.150.62.81:domain *:*
LISTEN 13859/named
tcp 0 0 82.150.62.80:domain *:*
LISTEN 13859/named
tcp 0 0 pollux.mydomain.c:domain *:*
LISTEN 13859/named
tcp 0 0 localhost.locald:domain *:*
LISTEN 13859/named
tcp 0 0 *:smtp *:*
LISTEN 26677/master
tcp 0 0 localhost.localdoma:953 *:*
LISTEN 13859/named
tcp 0 0 82.150.62.80:ftp 84-76-105-178.dcli:1623
ESTABLISHED21210/proftpd: web1
tcp6 0 0 *:imaps *:*
LISTEN 2955/couriertcpd
tcp6 0 0 *:pop3s *:*
LISTEN 2990/couriertcpd
tcp6 0 0 *:pop3 *:*
LISTEN 2970/couriertcpd
tcp6 0 0 *:imap2 *:*
LISTEN 2935/couriertcpd
tcp6 0 0 *:www *:*
LISTEN 13720/apache2
tcp6 0 0 *:ssh *:*
LISTEN 3243/sshd
tcp6 0 0 *:smtp *:*
LISTEN 26677/master
tcp6 0 0 ip6-localhost:953 *:*
LISTEN 13859/named
tcp6 0 0 *:https *:*
LISTEN 13720/apache2
tcp6 0 0 ::ffff:82.150.62.81:www www.someweb.com:47954
TIME_WAIT -
tcp6 0 0 ::ffff:82.150.62.81:www www.someweb.com:47857
TIME_WAIT -
tcp6 0 0 ::ffff:82.150.62.81:www www.someweb.com:47889
TIME_WAIT -
tcp6 0 0 ::ffff:82.150.62.81:www www.someweb.com:47921
TIME_WAIT -
tcp6 0 0 ::ffff:82.150.62.81:www www.someweb.com:47409
TIME_WAIT -
tcp6 0 0 ::ffff:82.150.62.81:www www.someweb.com:47767
TIME_WAIT -
tcp6 0 0 ::ffff:82.150.62.81:www www.someweb.com:47415
TIME_WAIT -
tcp6 0 0 ::ffff:82.150.62.81:www www.someweb.com:47959
TIME_WAIT -


Many thanks for your help!

[chuckl]

Slow netstat response is often DNS related, and offhand, I'd say you've got enough 'named' daemons running there for a root name server.

[edge]

chuckl is correct.

Have a look here:
http://www.howtoforge.com/forums/showthread.php?t=5114

[torusturtle]

Slow netstat response is often DNS related, and offhand, I'd say you've got enough 'named' daemons running there for a root name server.

I can't follow you.
Why is the login with ssh slow as well?
It has nothing to do with DNS as I'm using the IP and not domain name, hasn't it?

I use an external DNS service. Named service is only installed because ISPConfig wouldn't install if it wasn't.

The DNS record is the following:

mydomain.ch Host (A) Default 82.150.62.76
mydomain.ch Mail (MX) Default 10 mail.mydomain.ch
mydomain.ch Text (TXT) Default "v=spf1 ?all"
*.mydomain.ch Alias (CNAME) Default mydomain.ch
mail.mydomain.ch Host (A) Default 82.150.62.76

chuckl is correct.

Have a look here:
http://www.howtoforge.com/forums/showthread.php?t=5114

I tried that. Unfortunately it didn't help. :-(

[chuckl]

You're right, my post was not the soul of clarity, brevity is not always a good idea.
Try again. Netstat running very slowly is often a sign of slow DNS resolution - just an observation, not necessarily a root cause.
SSH by default does reverse DNS on the client that is connecting, which could easily cause the delay you are describing. Might be able to cut it down by adding an entry for the client compuer in the server hosts file, and checking that hosts.conf has an order hosts,bind entry.
I'd also check resolv.conf for the order in which the nameservers are listed, as they are queried in the order given. i.e. it is no use having fast resolution locally, if you are querying a remote nameserver.
Running on a VM, the networking setup - bridged, NAT, shared etc could also have an effect, as could network connection speed - 10, 100 etc. I'm assuming in all of this that your client computer is on the local LAN? Even with that, I have a noticeable difference in connection setup between a client on a 10baseT port, and one on a 100 port.
I have also seen comments that recommend that if you have multiple IP's on a single interface, connections to the 'base' i.e. etho:0 IP are quicker, but I've never attempted to verify it.
In your output above, a number of connections were in a TIME WAIT state, which can also clog up connection resolution.
You could simply be being slowed up by the mem swap system. Is the server busy, or idling most of the time?
The comment about the named daemons was simply that you have named listening on every IP on the box, all on the same subnet. Is the name server actually doing anything?
Final thought, is this a command line SSH connection or a GUI browse connection. I have noticed that doing a 'connect to server' from an Ubuntu desktop box to a server running in a VM can take an amazing length of time to set everything up, and ask for a password.
Hope some of that helps.

Rgds,

[torusturtle]

chuckl you're my hero!

Because I set up the server at home before I brought it to the ISP I still had my home router as top entry in /etc/resolv.conf.
Deleted it and the Server is much faster know!

FTP connection is still slow though.

[chuckl]

Glad that something worked for you!. You didn't necessarily have to delete it, just move it down the list. Sort of 'name server of last resort'.
I forgot to mention that it's a good idea to have a hosts file entry for the server on your client machine (if you don't already have one), can save a DNS lookup as well.
On the ftp side, we had a brief look at that a while back, proftpd was agonisingly slow to connect. There was some tentative thought that it was due to it trying to connect via IPv6 before switching to IP4.
Don't remember how we resolved it, if at all, so maybe the problem just went away.
Someone else here may have a better suggestion.

[torusturtle]

[...], can save a DNS lookup as well.[...]
how can this be achieved?

[chuckl]

If the client machine is a Linux box then in /etc/hosts or a Windows machine \Windows\system32\drivers\etc\hosts add a line with the IP address and name of the server e.g.

82.150.62.80 myserver.mydomain.com

thats it. Of course, if you are connecting using the IP address, then no change.

[torusturtle]

I found out the the slowest sites have several avatars with external links. So the server has to resolve all those links.

I guess that can't be saved in /etc/hosts?

Any ideas how to speed up those lookups?

[chuckl]

You can put any ip address/host combination you want in the hosts file. If those external sites have fixed IP addresses then simply add their IP address and FQDN (full hostname/sitename) to the hosts file.
Obviously, you don't want a 10000 line hostfile, or it will take longer than a DNS lookup.
The getting of the avatars could easily take longer than a DNS lookup however, and if they are accessed frequently, they should be cached on your local DNS server.
Can you not store them locally, or consider caching them on your server?

[torusturtle]

Do you mean by caching saving in /etc/hosts or is there an other way?
I tried to install dnsmasq as it has DNS caching but it doesn't work with bind and I can't find out how to cache with bind9.

I have eAccelerator installed. But that doesn't seam to help as loading the page takes around 4 seconds.

[chuckl]

Bind is a caching name server by design, you don't have to do anything, that is what it does, it caches DNS data locally on the server.
If you are fetching a lot of data from external sites, you have no control over the speed of that operation, so consider some form of external link/entity caching. I've never used anything like that, so I have no idea where to look. eAccellerator may well do it, I don't know.

If you are concerned about load times, go here:

http://www.websiteoptimization.com/services/analyze/

enter youw site url and check what the analyser says.