All services killed after making change in ISPConfig interface

[cbj4074]

I've seen this happen a couple of times as of late. Basically, I make a change to the ISPConfig settings and most services on the box are terminated immediately. Terminated services include Apache, Postfix, Amavis, Dovecot, syslogd... even SSHd seems to have restarted (my terminal session was disconnected, but I was able to reconnect immediately).

The specific nature of the change made in ISPConfig was SSL-related. I was installing an SSL certificate for a site... pretty routine. The only thing unusual was that the site in question was not "Enabled" at the time.

The only thing in Apache's error log is

[Fri Apr 27 06:50:03 2012] [notice] caught SIGTERM, shutting down

There is nothing at all in /var/log/mail.log, and logging stopped after this "crash" (syslogd tanked), so new events are not being registered either.

EDIT: I noticed that about 10 minutes before this incident, the following was logged:

Apr 27 06:37:26 localhost amavis[4085]: (04085-11) (!!)file(1) utility (/usr/bin/file) FAILED: run_command: can't fork: Cannot allocate memory at /usr/sbin/amavisd-new line 3077, line 4946.

Seems like the server's memory supply was exhausted. What else could cause this kind of across-the-board service termination? Maybe ISPConfig is just a red-herring in this case, and the change simply consumed enough memory to cause the crash.

Where else should I be looking for clues as to what, exactly, happened?

Also, is there any easy means by which to bring all services back online? Or should I just restart the box?

Thanks for any help!

[till]

Your problem is that services are out of memory. The problem occured already before you did the changes in ispconfig as you found in the mail log, its just that your ssl change required a apache restart and there ws not enough memory anymore to start it again. The memory was not consumed by ispconfig as ispconfig is not a service and the shell script consumes not much ram.

I recommend that you restart the server.

[cbj4074]

All is well after a reboot. Thanks, Till.

Amavis is quite a memory-hog. This is the third time in as many weeks that Amavis has chewed-up all available RAM (over 500MB, on average) and caused this server to tank.

As there seems to be no means by which to limit Amavis's memory consumption directly, might you have a recommendation for achieving the desired result by alternative means?

Not to digress, but Amavis's memory usage skyrocketed one time because an end-user had sent a group email in which the recipients' email addresses contained angle-brackets, e.g.:

Apr  5 09:29:33 localhost postfix/smtpd[24176]: warning: Illegal address syntax from unknown[XXX.XXX.XXX.XXX] in RCPT command: <'user@example.com'>

The message was re-queued repeatedly, eating-up more RAM each time, until memory was exhausted and the server crashed.

I would really like to prevent Amavis from being a constant risk to the rest of the server.

[till]

I never had any problems with amavis on a system yet and its very widely used, but all the systes that we manage have at least 2GB RAM. Replacing amavis should not be nescessary.

How much memory does your server has?
Do you use any custom spamassassin rulesets?

You can also limit the amavis memory usage by defining the number of amavis threads running and the max number of emails feeded to amavis by postfix.

[cbj4074]

Thanks for sharing your insights, Till.

I like Amavis, too, and I have no desire to replace it.

The server has only 1GB of physical memory, which, I realize, is hardly sufficient for most environments, but this is a development server on which we would prefer not to spend more money than absolutely necessary.

We do use custom SpamAssassin rule-sets; it makes sense that this would contribute to higher memory usage.

I have changed the maximum number of concurrent Amavisd-new processes from the default (2) to 1.

For anyone else who is interested in making the same change, this is how it's done:

/etc/amavis/conf.d/50-user

$max_servers = 1;

/etc/postfix/master.cf

# The number here MUST match the amavis configuration value.
amavis unix - - - - 1 smtp

All of this said, I'm still not sure how to prevent the amavis issue that I described in my second post. Any ideas there, Till?

Thanks again!

[till]

All of this said, I'm still not sure how to prevent the amavis issue that I described in my second post. Any ideas there, Till?

Maybe you should post that on a amvis mailinglist as it might be a bug in amavis email address handling.