Go Back   HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials > Linux Forums > Server Operation

Do you like HowtoForge? Please consider supporting us by becoming a subscriber.
Reply
 
Thread Tools Display Modes
  #1  
Old 27th April 2012, 16:13
cbj4074 cbj4074 is offline
Senior Member
 
Join Date: Nov 2010
Posts: 368
Thanks: 25
Thanked 47 Times in 42 Posts
Default All services killed after making change in ISPConfig interface

I've seen this happen a couple of times as of late. Basically, I make a change to the ISPConfig settings and most services on the box are terminated immediately. Terminated services include Apache, Postfix, Amavis, Dovecot, syslogd... even SSHd seems to have restarted (my terminal session was disconnected, but I was able to reconnect immediately).

The specific nature of the change made in ISPConfig was SSL-related. I was installing an SSL certificate for a site... pretty routine. The only thing unusual was that the site in question was not "Enabled" at the time.

The only thing in Apache's error log is

Code:
[Fri Apr 27 06:50:03 2012] [notice] caught SIGTERM, shutting down
There is nothing at all in /var/log/mail.log, and logging stopped after this "crash" (syslogd tanked), so new events are not being registered either.

EDIT: I noticed that about 10 minutes before this incident, the following was logged:

Code:
Apr 27 06:37:26 localhost amavis[4085]: (04085-11) (!!)file(1) utility (/usr/bin/file) FAILED: run_command: can't fork: Cannot allocate memory at /usr/sbin/amavisd-new line 3077, line 4946.
Seems like the server's memory supply was exhausted. What else could cause this kind of across-the-board service termination? Maybe ISPConfig is just a red-herring in this case, and the change simply consumed enough memory to cause the crash.

Where else should I be looking for clues as to what, exactly, happened?

Also, is there any easy means by which to bring all services back online? Or should I just restart the box?

Thanks for any help!

Last edited by cbj4074; 27th April 2012 at 16:20.
Reply With Quote
Sponsored Links
  #2  
Old 27th April 2012, 17:13
till till is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 34,586
Thanks: 792
Thanked 4,983 Times in 3,903 Posts
Default

Your problem is that services are out of memory. The problem occured already before you did the changes in ispconfig as you found in the mail log, its just that your ssl change required a apache restart and there ws not enough memory anymore to start it again. The memory was not consumed by ispconfig as ispconfig is not a service and the shell script consumes not much ram.

I recommend that you restart the server.
__________________
Till Brehm
--
Get ISPConfig support and the ISPConfig 3 manual from ispconfig.org.
Reply With Quote
The Following User Says Thank You to till For This Useful Post:
cbj4074 (27th April 2012)
  #3  
Old 27th April 2012, 18:59
cbj4074 cbj4074 is offline
Senior Member
 
Join Date: Nov 2010
Posts: 368
Thanks: 25
Thanked 47 Times in 42 Posts
Default

All is well after a reboot. Thanks, Till.

Amavis is quite a memory-hog. This is the third time in as many weeks that Amavis has chewed-up all available RAM (over 500MB, on average) and caused this server to tank.

As there seems to be no means by which to limit Amavis's memory consumption directly, might you have a recommendation for achieving the desired result by alternative means?

Not to digress, but Amavis's memory usage skyrocketed one time because an end-user had sent a group email in which the recipients' email addresses contained angle-brackets, e.g.:

Code:
Apr  5 09:29:33 localhost postfix/smtpd[24176]: warning: Illegal address syntax from unknown[XXX.XXX.XXX.XXX] in RCPT command: <'user@example.com'>
The message was re-queued repeatedly, eating-up more RAM each time, until memory was exhausted and the server crashed.

I would really like to prevent Amavis from being a constant risk to the rest of the server.
Reply With Quote
  #4  
Old 27th April 2012, 19:16
till till is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 34,586
Thanks: 792
Thanked 4,983 Times in 3,903 Posts
Default

I never had any problems with amavis on a system yet and its very widely used, but all the systes that we manage have at least 2GB RAM. Replacing amavis should not be nescessary.

How much memory does your server has?
Do you use any custom spamassassin rulesets?

You can also limit the amavis memory usage by defining the number of amavis threads running and the max number of emails feeded to amavis by postfix.
__________________
Till Brehm
--
Get ISPConfig support and the ISPConfig 3 manual from ispconfig.org.
Reply With Quote
  #5  
Old 1st May 2012, 20:31
cbj4074 cbj4074 is offline
Senior Member
 
Join Date: Nov 2010
Posts: 368
Thanks: 25
Thanked 47 Times in 42 Posts
Default

Thanks for sharing your insights, Till.

I like Amavis, too, and I have no desire to replace it.

The server has only 1GB of physical memory, which, I realize, is hardly sufficient for most environments, but this is a development server on which we would prefer not to spend more money than absolutely necessary.

We do use custom SpamAssassin rule-sets; it makes sense that this would contribute to higher memory usage.

I have changed the maximum number of concurrent Amavisd-new processes from the default (2) to 1.

For anyone else who is interested in making the same change, this is how it's done:

/etc/amavis/conf.d/50-user
Code:
$max_servers = 1;
/etc/postfix/master.cf
Code:
# The number here MUST match the amavis configuration value.
amavis unix - - - - 1 smtp
All of this said, I'm still not sure how to prevent the amavis issue that I described in my second post. Any ideas there, Till?

Thanks again!
Reply With Quote
  #6  
Old 2nd May 2012, 12:58
till till is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 34,586
Thanks: 792
Thanked 4,983 Times in 3,903 Posts
 
Default

Quote:
All of this said, I'm still not sure how to prevent the amavis issue that I described in my second post. Any ideas there, Till?
Maybe you should post that on a amvis mailinglist as it might be a bug in amavis email address handling.
__________________
Till Brehm
--
Get ISPConfig support and the ISPConfig 3 manual from ispconfig.org.
Reply With Quote
Reply

Bookmarks

Tags
apache, crash, dovecot, postfix, ssh

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Version 1.3 of the ISPConfig 3 Manual is finally available! falko General 44 2nd December 2011 12:04
ISPConfig 3, change password from within webmail flyingscubakev Installation/Configuration 3 18th January 2011 16:17
Moving the ISPconfig web interface and subdomain cozened Tips/Tricks/Mods 5 28th November 2009 16:03
How do I change the ISPConfig Website IP, not the server IP? mcaramb Installation/Configuration 1 9th June 2009 12:24
SP-Server Setup - Ubuntu 5.10 "Breezy Badger" - Page 6 (changes) LuisC-SM HOWTO-Related Questions 0 21st April 2006 15:16


All times are GMT +2. The time now is 09:23.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.