server.php segfault

Discussion in 'ISPConfig 3 Priority Support' started by atle, Apr 4, 2021.

  1. atle

    atle Member HowtoForge Supporter

    Somthing segfaults when server.php is run, and server.php never ends to run, the lockfile is there and the process is running.
    Excerp from log file:

    Code:
    Apr  4 11:38:01 ic CRON[18409]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Apr  4 11:38:01 ic CRON[18408]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Apr  4 11:38:14 ic kernel: [13906.912572] php[18392]: segfault at 7fffca8d9f80 ip 000055b77ce7c241 sp 00007fffca8d9f80 error 6 in php7.3[55b77cc82000+261000]
    Apr  4 11:38:14 ic kernel: [13907.128543] Code: ff 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 f5 41 54 49 89 fc 53 48 81 ec f8 00 00 00 48 8b 45 18 <48> 89 8d e0 fe ff ff 4c 89 85 f0 fe ff ff 44 89 8d 18 ff ff ff 48
    Processes running
    [​IMG]

    Any tips how to trouble shoot this?
     
  2. atle

    atle Member HowtoForge Supporter

    ... and this could also be helpful, seems to be line 22
    Code:
    Sun 04 Apr 2021 11:45:01 AM CEST 04.04.2021-11:45 - WARNING - There is already a lockfile set, but no process running with this pid (18982). Continuing.
    Sun 04 Apr 2021 11:46:01 AM CEST 04.04.2021-11:46 - WARNING - There is already an instance of server.php running with pid 19035.
    Sun 04 Apr 2021 11:46:10 AM CEST /usr/local/ispconfig/server/server.sh: line 22: 19035 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php
    Sun 04 Apr 2021 11:47:01 AM CEST 04.04.2021-11:47 - WARNING - There is already a lockfile set, but no process running with this pid (19035). Continuing.
    Sun 04 Apr 2021 11:48:02 AM CEST 04.04.2021-11:48 - WARNING - There is already an instance of server.php running with pid 19249.
    Sun 04 Apr 2021 11:48:17 AM CEST /usr/local/ispconfig/server/server.sh: line 22: 19249 Segmentation fault      /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php
    Sun 04 Apr 2021 11:49:01 AM CEST 04.04.2021-11:49 - WARNING - There is already a lockfile set, but no process running with this pid (19249). Continuing.
    
    
     
  3. atle

    atle Member HowtoForge Supporter

    Well, not so helpful, the line referes to the call

    Code:
    cd /usr/local/ispconfig/server
    /usr/bin/php -q \
        -d disable_classes= \
        -d disable_functions= \
        -d open_basedir= \
        /usr/local/ispconfig/server/server.php
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    Which PHP Version do you use?
    Are the PHP packages the ones that ship with the OS or sury, or did you build PHP yourself?
    Do you use any custom PHP modules e.g. ones manually compiled with pear?
    Any other segfaults in the systems syslog file? A segfault can also be hardware related like faulty RAM, motherboard issues etc.
    Do you have enough RAM?
     
  5. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Also can come from filesystem corruption, you might run fsck and force reinstall all php related packages
     
  6. atle

    atle Member HowtoForge Supporter

    Its Debian 10, php 7.3.
    auto-install script used, hence sury used.
    No special php modules

    Code:
    [email protected]:/var/log# php -m
    [PHP Modules]
    calendar
    Core
    ctype
    curl
    date
    dom
    exif
    fileinfo
    filter
    ftp
    gd
    gettext
    hash
    iconv
    imagick
    imap
    intl
    ionCube Loader
    json
    libxml
    mbstring
    mysqli
    mysqlnd
    openssl
    pcntl
    pcre
    PDO
    pdo_mysql
    pdo_sqlite
    Phar
    posix
    pspell
    readline
    recode
    Reflection
    session
    shmop
    SimpleXML
    soap
    sockets
    sodium
    SPL
    sqlite3
    standard
    sysvmsg
    sysvsem
    sysvshm
    tidy
    tokenizer
    wddx
    xml
    xmlreader
    xmlrpc
    xmlwriter
    xsl
    Zend OPcache
    zip
    zlib
    
    [Zend Modules]
    Zend OPcache
    the ionCube PHP Loader + ionCube24
    No other segfaults in syslog.
    RAM is 4GB for a non-production server, with a couple of sites, should be no worries.
    Its a VPS, hence no MB etc. issues.
     
  7. atle

    atle Member HowtoForge Supporter

    There are 4 items in the queue. How can I remove them, to exclude these jobs to be the cause.
     
  8. atle

    atle Member HowtoForge Supporter

    I found out how to increment the queue counter, so there is no queue items pending now.
    No segfaults either when server.sh is run.
    Will now test different queue jobs to see what happens. It segfaults for some kind of queue job.
     
  9. atle

    atle Member HowtoForge Supporter

    Its probably a mail_forwarding job that caused the segfault.
     
  10. atle

    atle Member HowtoForge Supporter

    Yes, adding mail domain alias causes the segfault.
     
  11. atle

    atle Member HowtoForge Supporter

    I enabled debug logging, added a mail domain alias, and here is where it hangs (and segfaults I assume)

    Code:
    05.04.2021-16:23 - DEBUG - Unable to register function 'process' from plugin 'software_update_plugin' for event 'software_update_inst_insert'
    05.04.2021-16:23 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    05.04.2021-16:23 - DEBUG - Found 1 changes, starting update process.
    05.04.2021-16:23 - DEBUG - Calling function 'user_settings_update' from plugin 'rspamd_plugin' raised by event 'mail_forwarding_insert'.
    q.sh: line 24: 17680 Segmentation fault      /usr/bin/php -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php
    
    Anyone?
     
  12. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    There is a bug in 3.2.3 creating a loop writing spamfilter settings for a domain alias, make sure you have updated to the current version (3.2.4). I didn't see that causing segfaults, and don't know that it would without an outside problem (hardware or whatever), but maybe you issue will clear with that?
     
  13. atle

    atle Member HowtoForge Supporter

    Oki, I am in 323. Will update and see. Since it is a multi server environment, should I update I any particular order?

    And, I did test this on a slave mail server, and there was no issues in this server. Both servers are identical, except one is master and the other mail only.

    Code:
    Mon 05 Apr 2021 04:41:01 PM CEST 05.04.2021-16:41 - DEBUG - Unable to register function 'process' from plugin 'software_update_plugin' for event 'software_update_inst_insert'
    Mon 05 Apr 2021 04:41:01 PM CEST 05.04.2021-16:41 - DEBUG - Found 1 changes, starting update process.
    Mon 05 Apr 2021 04:41:01 PM CEST 05.04.2021-16:41 - DEBUG - Replicated from master: REPLACE INTO `mail_forwarding` (`forwarding_id`,`sys_userid`,`sys_groupid`,`sys_perm_user`,`sys_perm_group`,`sys_perm_other`,`server_id`,`source`,`destination`,`type`,`active`,`allow_send_as`,`greylisting`) VALUES ('6','1','30','riud','riud','','5','@foo.com','@bar,com','aliasdomain','y','n','n')
    Mon 05 Apr 2021 04:41:01 PM CEST 05.04.2021-16:41 - DEBUG - Calling function 'user_settings_update' from plugin 'rspamd_plugin' raised by event 'mail_forwarding_insert'.
    Mon 05 Apr 2021 04:41:01 PM CEST 05.04.2021-16:41 - DEBUG - Processed datalog_id 3530
    Mon 05 Apr 2021 04:41:01 PM CEST 05.04.2021-16:41 - DEBUG - Calling function 'restartRspamd' from module 'mail_module'.
    Mon 05 Apr 2021 04:41:01 PM CEST 05.04.2021-16:41 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    Mon 05 Apr 2021 04:41:01 PM CEST finished server.php.
    Mon 05 Apr 2021 04:42:01 PM CEST 05.04.2021-16:42 - DEBUG - Unable to register function 'process' from plugin 'software_update_plugin' for event 'software_update_inst_insert'
    Mon 05 Apr 2021 04:42:01 PM CEST 05.04.2021-16:42 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    Mon 05 Apr 2021 04:42:02 PM CEST finished server.php.
    Mon 05 Apr 2021 04:43:01 PM CEST 05.04.2021-16:43 - DEBUG - Unable to register function 'process' from plugin 'software_update_plugin' for event 'software_update_inst_insert'
    Mon 05 Apr 2021 04:43:01 PM CEST 05.04.2021-16:43 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    Mon 05 Apr 2021 04:43:01 PM CEST finished server.php.
     
  14. atle

    atle Member HowtoForge Supporter

    Yes, upgrade to 324 did solve it. Thanks Jesse.
     
  15. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    I update the master first, then all slave servers, and for any version updates always answer yes to reconfigure permissions in the master db. There is a setting to put servers in a maintenance mode first; I don't know exactly what that does (disables changes in the ui?), but I pretty much never (remember to) use that.
    good deal.
     
    atle likes this.

Share This Page