Apache won't start: "Too many files error"?

[bpmee]

Hi All,



I suddenly have a strange Apache error. I'm on Fedora Core 5 test server using ISPConfig 2.3.1-dev.

Apache (httpd) went down about 2 hours ago. I tried the typical /etc/rc.d/init.d/httpd restart command, and noticed that apache reported "failure" for both stopping and starting httpd.

I had a look at my error log: /var/log/httpd/error_log and noticed the following error:

piped_log_spawn: unable to setup child process '/root/ispconfig/cronolog --symlink=/var/log/httpd/ispconfig_access_log /var/log/httpd/ispconfig_access_log_%Y_%m_%d': Too many open files
Unable to open logs
Unable to open logs
Unable to open logs
Unable to open logs
Unable to open logs
Unable to open logs
Unable to open logs
Unable to open logs
Unable to open logs
Unable to open logs
Unable to open logs
Unable to open logs
Unable to open logs
Unable to open logs
Unable to open logs
Unable to open logs
Unable to open logs
Unable to open logs
piped_log_spawn: unable to setup child process '/root/ispconfig/cronolog --symlink=/var/log/httpd/ispconfig_access_log /var/log/httpd/ispconfig_access_log_%Y_%m_%d': Too many open files
Unable to open logs

I haven't changed httpd.conf!

Also, please note that ISPconfig's apache loads fine and ISPConfig is accessible from the web.

My server is under a heavy load at the moment, but I have never had issues with Apache in the past when putting the server under load...



Any ideas? Thanks.........

[falko]

Do you use a cronolog line in ech vhost, or just once in the main Apache configuration? If you have many vhosts and use cronolog once per vhost, you have many open access logs at the same time.

[bpmee]

Hi Falko,

Thanks for your reply!

You're correct, I do have a lot of vhosts in my configuration.

I check vhosts_ispconfig.conf and noticed no extra entries for cronolog.

The only place I found the cronolog entry was in my httpd.conf file, under Part 3, Vhosts...

Here is part of my http.conf server config section, I dropped some commented sections out:

User apache
Group apache

### Section 2: 'Main' server configuration
#
#
#
#
ServerAdmin root@localhost

#
#
#ServerName www.example.com:80

#
# ServerName directive.
#

UseCanonicalName Off

#
#
DocumentRoot /var/www/html

#
# Each directory to which Apache has access can be configured with respect
#
<Directory />
    Options Includes Indexes FollowSymLinks
    AllowOverride None
</Directory>

#
# Note that from this point forward you must specifically allow
#
# This should be changed to whatever you set DocumentRoot to.
#
<Directory /var/www/html>

#
# Possible values for the Options directive are "None", "All",
# or any combination of:
#   Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
#
# Note that "MultiViews" must be named *explicitly* --- "Options All"
# doesn't give it to you.
#
# The Options directive is both complicated and important.  Please see
# http://httpd.apache.org/docs/2.2/mod/core.html#options
# for more information.
#
    Options Includes Indexes FollowSymLinks

#
# AllowOverride controls what directives may be placed in .htaccess files.
# It can be "All", "None", or any combination of the keywords:
#   Options FileInfo AuthConfig Limit
#
    AllowOverride None

#
# Controls who can get stuff from this server.
#
    Order allow,deny
    Allow from all

</Directory>

#
# UserDir: The name of the directory that is appended onto a user's home
# directory if a ~user request is received.
#
# See also: http://httpd.apache.org/docs/misc/FAQ.html#forbidden
#
<IfModule mod_userdir.c>
    #
    # UserDir is disabled by default since it can confirm the presence
    # of a username on the system (depending on home directory
    # permissions).
    #
    UserDir disable

    #
    # To enable requests to /~user/ to serve the user's public_html
    # directory, remove the "UserDir disable" line above, and uncomment
    # the following line instead:
    # 
    #UserDir public_html

</IfModule>

#
# Control access to UserDir directories.  The following is an example
# for a site where these directories are restricted to read-only.
#
#<Directory /home/*/public_html>
#    AllowOverride FileInfo AuthConfig Limit
#    Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec
#    <Limit GET POST OPTIONS>
#        Order allow,deny
#        Allow from all
#    </Limit>
#    <LimitExcept GET POST OPTIONS>
#        Order deny,allow
#        Deny from all
#    </LimitExcept>
#</Directory>

#
# DirectoryIndex: sets the file that Apache will serve if a directory
# is requested.
#
# The index.html.var file (a type-map) is used to deliver content-
# negotiated documents.  The MultiViews Option can be used for the 
# same purpose, but it is much slower.
#
DirectoryIndex index.html index.htm index.shtml index.cgi index.php index.php3 index.pl

#
# AccessFileName: The name of the file to look for in each directory
# for additional configuration directives.  See also the AllowOverride
# directive.
#
#AccessFileName .htaccess

#
# The following lines prevent .htaccess and .htpasswd files from being 
# viewed by Web clients. 
#
<Files ~ "^\.ht">
    Order allow,deny
    Deny from all
</Files>

#
# TypesConfig describes where the mime.types file (or equivalent) is
# to be found.
#
TypesConfig /etc/mime.types

#
# DefaultType is the default MIME type the server will use for a document
#
DefaultType text/plain

#
# The mod_mime_magic module allows the server to use various hints from the
# contents of the file itself to determine its type.  The MIMEMagicFile
# directive tells the module where the hint definitions are located.
#
<IfModule mod_mime_magic.c>
#   MIMEMagicFile /usr/share/magic.mime
    MIMEMagicFile conf/magic
</IfModule>

#
# HostnameLookups: Log the names of clients or just their IP addresses
#
HostnameLookups Off

#
# EnableMMAP: Control whether memory-mapping is used to deliver
# http://httpd.apache.org/docs/2.2/mod/core.html#enablemmap
#
#EnableMMAP off

#
# EnableSendfile: Control whether the sendfile kernel support is 
# used to deliver files (assuming that the OS supports it). 
# The default is on; turn this off if you serve from NFS-mounted 
# filesystems.  Please see
# http://httpd.apache.org/docs/2.2/mod/core.html#enablesendfile
#
#EnableSendfile off

#
# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a <VirtualHost>
# container, error messages relating to that virtual host will be
# logged here.  If you *do* define an error logfile for a <VirtualHost>
# container, that host's errors will be logged there and not here.
#
ErrorLog logs/error_log

#
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
#
LogLevel warn

#
# The following directives define some format nicknames for use with
# a CustomLog directive (see below).
#
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

# "combinedio" includes actual counts of actual bytes received (%I) and sent (%O); this
# requires the mod_logio module to be loaded.
#LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio

#
# The location and format of the access logfile (Common Logfile Format).
# If you do not define any access logfiles within a <VirtualHost>
# container, they will be logged here.  Contrariwise, if you *do*
# define per-<VirtualHost> access logfiles, transactions will be
# logged therein and *not* in this file.
#
#CustomLog logs/access_log common

#
# If you would like to have separate agent and referer logfiles, uncomment
# the following directives.
#
#CustomLog logs/referer_log referer
#CustomLog logs/agent_log agent

#
# For a single logfile with access, agent, and referer information
# (Combined Logfile Format), use the following directive:
#
CustomLog logs/access_log combined

Perhaps VHOSTS is misconfigured, here is my httpd.conf vhosts section:

### Section 3: Virtual Hosts
#
# VirtualHost: If you want to maintain multiple domains/hostnames on your
# machine you can setup VirtualHost containers for them. Most configurations
# use only name-based virtual hosts so the server doesn't need to worry about
# IP addresses. This is indicated by the asterisks in the directives below.
#
# Please see the documentation at 
# <URL:http://httpd.apache.org/docs/2.2/vhosts/>
# for further details before you try to setup virtual hosts.
#
# You may use the command line option '-S' to verify your virtual host
# configuration.

#
# Use name-based virtual hosting.
#
#NameVirtualHost *:80
#
# NOTE: NameVirtualHost cannot be used without a port specifier 
# (e.g. :80) if mod_ssl is being used, due to the nature of the
# SSL protocol.
#

#
# VirtualHost example:
# Almost any Apache directive may go into a VirtualHost container.
# The first VirtualHost section is used for requests without a known
# server name.
#
#<VirtualHost *:80>
#    ServerAdmin webmaster@dummy-host.example.com
#    DocumentRoot /www/docs/dummy-host.example.com
#    ServerName dummy-host.example.com
#    ErrorLog logs/dummy-host.example.com-error_log
#    CustomLog logs/dummy-host.example.com-access_log common
#</VirtualHost>

<Directory /var/www/sharedip>
    Options +Includes +Indexes
    AllowOverride None
    AllowOverride Indexes AuthConfig Limit FileInfo
    Order allow,deny
    Allow from all
    <Files ~ "^\.ht">
    Deny from all
    </Files>
</Directory>

###############ispconfig_log###############
LogFormat "%v||||%b||||%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined_ispconfig
CustomLog "|/root/ispconfig/cronolog --symlink=/var/log/httpd/ispconfig_access_log /var/log/httpd/ispconfig_access_log_%Y_%m_%d" combined_ispconfig

<Directory /home/www/*/web>
    Options +Includes +Indexes -ExecCGI
    AllowOverride None
    AllowOverride Indexes AuthConfig Limit FileInfo
    Order allow,deny
    Allow from all
    <Files ~ "^\.ht">
    Deny from all
    </Files>
</Directory>

<Directory /home/www/*/user/*/web>
    Options +Includes -Indexes -ExecCGI
    AllowOverride None
    AllowOverride Indexes AuthConfig Limit FileInfo
    Order allow,deny
    Allow from all
    <Files ~ "^\.ht">
    Deny from all
    </Files>
</Directory>

<Directory /home/www/*/cgi-bin>
    Options ExecCGI -Indexes
    AllowOverride None
    AllowOverride Indexes AuthConfig Limit FileInfo
    Order allow,deny
    Allow from all
    <Files ~ "^\.ht">
    Deny from all
    </Files>
</Directory>

Include /etc/httpd/conf/vhosts/Vhosts_ispconfig.conf

Does the above indicate a cronolog for EACH vhost, or one cronolog for all as you suggested?

Thanks!

[bpmee]

Hi Falko, others,

I found a solution to this problem, but I am not sure if it is sustainable or healthy for the server.

It is here: http://nixforums.org/about55655.html

Basically, HTTPD is telling me that the server has reached its processing limits.

When I check the processing limits by doing:

ulimit -n
1024

I get 1024.

According to this post, if I increase my process limits, for example doubling to 2048:

ulimit -n 2048

the problem should be fixed.

I tested this solution and it appeared to work. HTTPD restarted normally and websites were functional.

To make sure HTTPD would restart in the future correctly, I added:

ulimit -n 2048
echo Ulimit set to 2048

to my HTTPD file in /etc/rc.d/init.d/httpd

My new question: Is this healthy for the server? Will I be putting exessive load on my box? Could this start a run-away server?

Thanks!

[falko]

I don't see any problems with the new setting.