Segmentation fault

Discussion in 'ISPConfig 3 Priority Support' started by agentmoller001, Jul 13, 2017.

  1. I keep getting segmentation faults in the ISPConfig Cron Log. I had the data center check the server and this was the response.
    Here are the two lines that keep showing up in the ISPConfig Cron Log:
    Code:
    Thu Jul 13 09:41:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17:  6995 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Thu Jul 13 09:41:01 PDT 2017
    Thu Jul 13 09:41:01 PDT 2017
    Thu Jul 13 09:41:01 PDT 2017 finished.
    Thu Jul 13 09:42:02 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17:  7042 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Thu Jul 13 09:42:02 PDT 2017
    Thu Jul 13 09:42:02 PDT 2017
    Thu Jul 13 09:42:02 PDT 2017 finished.
    The OS is Debian Jessie in an OpenVZ container on dedicated server I rent. ISPConfig 3 version 3.1.5. No pending updates/upgrades available. No fails in the openvz beancounter.
    Code:
    Version: 2.5
    
           uid  resource                     held              maxheld              barrier                limit              failcnt
    
        30101:  kmemsize                205803536            224780288  9223372036854775807  9223372036854775807                    0
    
                lockedpages                     0                    0  9223372036854775807  9223372036854775807                    0
    
                privvmpages               2544266              2669855  9223372036854775807  9223372036854775807                    0
    
                shmpages                   111906               112578  9223372036854775807  9223372036854775807                    0
    
                dummy                           0                    0                    0                    0                    0
    
                numproc                       332                  381  9223372036854775807  9223372036854775807                    0
    
                physpages                 2054561              2059473                    0              6291456                    0
    
                vmguarpages                     0                    0  9223372036854775807  9223372036854775807                    0
    
                oomguarpages              1482392              1482392  9223372036854775807  9223372036854775807                    0
    
                numtcpsock                     85                  128     1801439850948198     1801439850948198                    0
    
                numflock                      424                  430  9223372036854775807  9223372036854775807                    0
    
                numpty                          2                   13  9223372036854775807  9223372036854775807                    0
    
                numsiginfo                      0                   45  9223372036854775807  9223372036854775807                    0
    
                tcpsndbuf                 1563560              2534848  4611686018427387903  9223372036854775807                    0
    
                tcprcvbuf                 1494840              2267728  4611686018427387903  9223372036854775807                    0
    
                othersockbuf               899368              2370328  4611686018427387903  9223372036854775807                    0
    
                dgramrcvbuf                     0                27744  9223372036854775807  9223372036854775807                    0
    
                numothersock                  421                  447     1801439850948198     1801439850948198                    0
    
                dcachesize               77102998             77231262  9223372036854775807  9223372036854775807                    0
    
                numfile                      4007                 4270  9223372036854775807  9223372036854775807                    0
    
                dummy                           0                    0                    0                    0                    0
    
                dummy                           0                    0                    0                    0                    0
    
                dummy                           0                    0                    0                    0                    0
    
                numiptent                     150                  150  9223372036854775807  9223372036854775807                    0
    Memory not exhausted
    Code:
    # free -m
                 total       used       free     shared    buffers     cached
    Mem:         24576       8122      16453        437          0       2107
    -/+ buffers/cache:       6014      18561
    Swap:        12288          0      12288
     
    Last edited: Jul 13, 2017
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    Please try to turn on debug loglevel in ISPConfig so we might be able to see which program is failing here. The problem can be related to the hardware like flaky RAM or a mainboard issues as well, or it is an issue in the PHP interpreter itself or third party program which gets called by ISPConfig, but that's less likely.
     
  3. Here is the output of a section with a segmentation fault (11:09:01) in the ISPConfig Cron Log with debug logging enabled. I don't believe there to be any helpful information in the log for this particular situation. What do you recommend I try next?
    Code:
    Fri Jul 14 11:06:01 PDT 2017
    Fri Jul 14 11:06:01 PDT 2017
    Fri Jul 14 11:06:01 PDT 2017 14.07.2017-11:06 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    Fri Jul 14 11:06:01 PDT 2017 14.07.2017-11:06 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    Fri Jul 14 11:06:01 PDT 2017 finished.
    Fri Jul 14 11:07:01 PDT 2017
    Fri Jul 14 11:07:01 PDT 2017
    Fri Jul 14 11:07:01 PDT 2017 14.07.2017-11:07 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    Fri Jul 14 11:07:01 PDT 2017 14.07.2017-11:07 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    Fri Jul 14 11:07:01 PDT 2017 finished.
    Fri Jul 14 11:08:01 PDT 2017
    Fri Jul 14 11:08:01 PDT 2017
    Fri Jul 14 11:08:01 PDT 2017 14.07.2017-11:08 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    Fri Jul 14 11:08:01 PDT 2017 14.07.2017-11:08 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    Fri Jul 14 11:08:01 PDT 2017 finished.
    Fri Jul 14 11:09:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 10114 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Fri Jul 14 11:09:01 PDT 2017 /usr/local/ispconfig/server/server.sh: line 22: 10115 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php
    Fri Jul 14 11:10:01 PDT 2017
    Fri Jul 14 11:10:01 PDT 2017
    Fri Jul 14 11:10:01 PDT 2017 c1aps1 does not exceed quota qize: 104857600 > 4453884
    Fri Jul 14 11:10:01 PDT 2017 c1aps10 does not exceed quota qize: 104857600 > 5035314
    Fri Jul 14 11:10:01 PDT 2017 14.07.2017-11:10 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    Fri Jul 14 11:10:01 PDT 2017 14.07.2017-11:10 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    Fri Jul 14 11:10:01 PDT 2017 finished.
    
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    So it seems as if the segfault does not happen on every run. can you see at which minutes it always happens, this can be used to narrow down by looking at the cron plugin schedules to find out which actions get run at that exact time.
     
  5. Here are the entries from 8:00 AM to current time 12:45 PM today. It seems to be completely random.
    Code:
    [email protected]:/var/log/ispconfig# cat cron.log | grep "Segmentation fault"
    Fri Jul 14 08:05:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 10324 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Fri Jul 14 08:08:01 PDT 2017 /usr/local/ispconfig/server/server.sh: line 22: 11063 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php
    Fri Jul 14 08:24:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 13653 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Fri Jul 14 08:40:01 PDT 2017 /usr/local/ispconfig/server/server.sh: line 22: 16183 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php
    Fri Jul 14 08:50:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 17848 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Fri Jul 14 08:53:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 18614 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Fri Jul 14 08:59:01 PDT 2017 /usr/local/ispconfig/server/server.sh: line 22: 19528 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php
    Fri Jul 14 09:09:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 21246 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Fri Jul 14 09:15:01 PDT 2017 /usr/local/ispconfig/server/server.sh: line 22: 22315 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php
    Fri Jul 14 09:26:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 24602 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Fri Jul 14 09:30:01 PDT 2017 /usr/local/ispconfig/server/server.sh: line 22: 24788 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php
    Fri Jul 14 09:35:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 25611 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Fri Jul 14 09:36:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 26146 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Fri Jul 14 10:14:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 32638 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Fri Jul 14 10:18:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17:   965 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Fri Jul 14 10:32:01 PDT 2017 /usr/local/ispconfig/server/server.sh: line 22:  3746 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php
    Fri Jul 14 10:53:02 PDT 2017 /usr/local/ispconfig/server/server.sh: line 22:  7399 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php
    Fri Jul 14 11:01:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17:  9059 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Fri Jul 14 11:09:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 10114 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Fri Jul 14 11:09:01 PDT 2017 /usr/local/ispconfig/server/server.sh: line 22: 10115 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php
    Fri Jul 14 11:19:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 12179 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Fri Jul 14 11:45:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 18161 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Fri Jul 14 11:56:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 20515 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Fri Jul 14 12:24:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 26108 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Fri Jul 14 12:25:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 26159 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Fri Jul 14 12:28:02 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 26942 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php
    Fri Jul 14 12:36:01 PDT 2017 /usr/local/ispconfig/server/server.sh: line 22: 28466 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    Did you look at the syslog file and grep for segfault, to see if other services fail as well from time to time?
     
  7. Yes, there is nothing.
    Code:
    [email protected]:/var/log# cat syslog.1 | grep "Segmentation fault"
    [email protected]:/var/log# cat syslog.1 | grep "Seg fault"
    [email protected]:/var/log# cat syslog.1 | grep "segfault"
    [email protected]:/var/log# cat syslog.1 | grep "Segmentation"
    [email protected]:/var/log# cat syslog.1 | grep "Segfault"
    
    [email protected]:/var/log# cat syslog | grep "Segmentation fault"
    [email protected]:/var/log# cat syslog | grep "Seg fault"
    [email protected]:/var/log# cat syslog | grep "segfault"
    [email protected]:/var/log# cat syslog | grep "Segmentation"
    [email protected]:/var/log# cat syslog | grep "Segfault"
    
    I also ran each one of the crons manually every minute for ten minutes ( https://www.faqforge.com/linux/debugging-ispconfig-3-server-actions-in-case-of-a-failure/ ) No segmentation fault error displayed during that time.
     
  8. Should I just not worry about it? Or any other ideas?
     
  9. till

    till Super Moderator Staff Member ISPConfig Developer

    It might be best to ignore it for now, maybe you find something later, as long as you don't see any faults in the actual system config or services.
     

Share This Page