Server/ISPC upgrade gone wrong

Discussion in 'ISPConfig 3 Priority Support' started by Nilpo, Mar 4, 2017.

  1. Nilpo

    Nilpo Member HowtoForge Supporter

    I am running Perfect Server on CentOS 6. Everything was working fine until I decided to do upgrades last night. I first upgraded packages (yum upgrade) and then upgraded ISPConfig. Now I am having nothing but problems and I wish I had left well enough alone.
    Code:
    Linux myhost.com 2.6.32-042stab116.2 #1 SMP Fri Jun 24 15:33:57 MSK 2016 x86_64 x86_64 x86_64 GNU/Linux
    To begin, I am getting 500 errors every other page load for sites on my server. I cannot seem to track down a reason. The only thing I see is that php-cgi is now causing OOM errors, but I don't know how to diagnose that any further. I'm seeing dozens of these over and over in /var/log/messages.
    Code:
    Mar  3 18:15:08 myhost kernel: [14805396.443762] Out of memory in UB 474: OOM killed process 11586 (php-cgi) score 0 vm:203876kB, rss:84512kB, swap:0kB
    Mar  3 18:15:10 myhost kernel: [14805398.387054] Out of memory in UB 474: OOM killed process 12109 (php-cgi) score 0 vm:208044kB, rss:89948kB, swap:0kB
    Mar  3 18:15:12 myhost kernel: [14805400.516331] Out of memory in UB 474: OOM killed process 12060 (php-cgi) score 0 vm:202756kB, rss:85224kB, swap:0kB
    Mar  3 18:15:15 myhost kernel: [14805403.339489] Out of memory in UB 474: OOM killed process 11973 (php-cgi) score 0 vm:205844kB, rss:88396kB, swap:0kB
    Mar  3 18:15:16 myhost kernel: [14805404.700555] Out of memory in UB 474: OOM killed process 12422 (php-cgi) score 0 vm:208920kB, rss:92184kB, swap:0kB
    Mar  3 18:15:19 myhost kernel: [14805407.487566] Out of memory in UB 474: OOM killed process 11494 (php-cgi) score 0 vm:199392kB, rss:82116kB, swap:0kB
    Mar  3 18:15:21 myhost kernel: [14805409.106790] Out of memory in UB 474: OOM killed process 12561 (php-cgi) score 0 vm:206024kB, rss:88268kB, swap:0kB
    The system PHP was updated as was Apache. However, some of the affected sites are running on alternate PHP versions, not the system version.

    Also, mail delivery has stopped. All messages are being deferred as shown in /var/log/maillog
    Code:
    Mar  3 18:21:00 myhost postfix/smtp[13157]: 0C8563E00E9A: to=<admin@myhost.com>, relay=127.0.0.1[127.0.0.1]:10026, delay=1183, delays=1183/0.02/0/0, dsn=4.4.2, status=deferred (lost connection with 127.0.0.1[127.0.0.1] while receiving the initial server greeting)
    Mar  3 18:21:00 myhost postfix/error[14144]: 01D5E3E00E9F: to=<user@domain.com>, relay=none, delay=498, delays=498/0.03/0/0.01, dsn=4.4.2, status=deferred (delivery temporarily suspended: lost connection with 127.0.0.1[127.0.0.1] while receiving the initial server greeting)
    Mar  3 18:21:00 myhost postfix/error[14143]: B003E3E00EA3: to=<user@domain.com>, relay=none, delay=493, delays=493/0.03/0/0.01, dsn=4.4.2, status=deferred (delivery temporarily suspended: lost connection with 127.0.0.1[127.0.0.1] while receiving the initial server greeting)
    Mar  3 18:21:00 myhost postfix/error[14145]: 085043E00EA5: to=<user@domain.com>, relay=none, delay=491, delays=491/0.03/0/0.01, dsn=4.4.2, status=deferred (delivery temporarily suspended: lost connection with 127.0.0.1[127.0.0.1] while receiving the initial server greeting)
    Mar  3 18:21:00 myhost postfix/error[14146]: 818543E00EA8: to=<user@domain.com>, relay=none, delay=490, delays=490/0.04/0/0, dsn=4.4.2, status=deferred (delivery temporarily suspended: lost connection with 127.0.0.1[127.0.0.1] while receiving the initial server greeting)
    I'm lost as to where to go from here. I've double checked main.cf and all looks good. All services are running. I am running CSF, but I am not seeing any corresponding firewall rules being triggered.
     
  2. Nilpo

    Nilpo Member HowtoForge Supporter

    Here is the output from 'postconf -n'
    Code:
    alias_database = hash:/etc/aliases
    alias_maps = hash:/etc/aliases
    body_checks = regexp:/etc/postfix/body_checks
    broken_sasl_auth_clients = yes
    command_directory = /usr/sbin
    config_directory = /etc/postfix
    daemon_directory = /usr/libexec/postfix
    data_directory = /var/lib/postfix
    debug_peer_level = 2
    header_checks = regexp:/etc/postfix/header_checks
    html_directory = no
    inet_interfaces = all
    inet_protocols = all
    mail_owner = postfix
    mailbox_size_limit = 0
    mailq_path = /usr/bin/mailq.postfix
    manpage_directory = /usr/share/man
    maximal_backoff_time = 15m
    maximal_queue_lifetime = 1h
    message_size_limit = 0
    milter_command_timeout = 30s
    milter_connect_timeout = 30s
    milter_content_timeout = 30s
    milter_default_action = tempfail
    milter_protocol = 2
    mime_header_checks = regexp:/etc/postfix/mime_header_checks
    minimal_backoff_time = 5m
    myhostname = myhost.com
    mynetworks = 127.0.0.0/8 [::1]/128
    nested_header_checks = regexp:/etc/postfix/nested_header_checks
    newaliases_path = /usr/bin/newaliases.postfix
    non_smtpd_milters = $smtpd_milters
    proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps $virtual_alias_domains $sender_bcc_maps $virtual_mailbox_maps $virtual_mailbox_domains $relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps $recipient_canonical_maps $relocated_maps $transport_maps $mynetworks $smtpd_sender_login_maps
    queue_directory = /var/spool/postfix
    queue_run_delay = 5m
    readme_directory = /usr/share/doc/postfix-2.6.6/README_FILES
    receive_override_options = no_address_mappings
    relay_domains = mysql:/etc/postfix/mysql-virtual_relaydomains.cf
    relay_recipient_maps = mysql:/etc/postfix/mysql-virtual_relayrecipientmaps.cf
    relayhost =
    sample_directory = /usr/share/doc/postfix-2.6.6/samples
    sender_bcc_maps = proxy:mysql:/etc/postfix/mysql-virtual_outgoing_bcc.cf
    sendmail_path = /usr/sbin/sendmail.postfix
    setgid_group = postdrop
    smtp_bind_address = 123.45.67.89
    smtp_tls_exclude_ciphers = RC4, aNULL
    smtp_tls_protocols = !SSLv2,!SSLv3
    smtp_tls_security_level = may
    smtpd_client_message_rate_limit = 100
    smtpd_client_restrictions = check_client_access mysql:/etc/postfix/mysql-virtual_client.cf
    smtpd_helo_required = yes
    smtpd_helo_restrictions = permit_sasl_authenticated, permit_mynetworks, check_helo_access regexp:/etc/postfix/helo_access, reject_invalid_hostname, reject_non_fqdn_hostname, check_helo_access regexp:/etc/postfix/blacklist_helo
    smtpd_milters = inet:127.0.0.1:8891
    smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination, check_recipient_access mysql:/etc/postfix/mysql-virtual_recipient.cf
    smtpd_restriction_classes = greylisting
    smtpd_sasl_auth_enable = yes
    smtpd_sasl_authenticated_header = yes
    smtpd_sasl_path = private/auth
    smtpd_sasl_type = dovecot
    smtpd_sender_login_maps = proxy:mysql:/etc/postfix/mysql-virtual_sender_login_maps.cf
    smtpd_sender_restrictions = check_sender_access regexp:/etc/postfix/tag_as_originating.re , permit_mynetworks, permit_sasl_authenticated, check_sender_access mysql:/etc/postfix/mysql-virtual_sender.cf, check_sender_access regexp:/etc/postfix/tag_as_foreign.re
    smtpd_tls_cert_file = /etc/postfix/smtpd.cert
    smtpd_tls_exclude_ciphers = RC4, aNULL
    smtpd_tls_key_file = /etc/postfix/smtpd.key
    smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3
    smtpd_tls_protocols = !SSLv2,!SSLv3
    smtpd_tls_security_level = may
    smtpd_use_tls = yes
    transport_maps = hash:/var/lib/mailman/data/transport-mailman, proxy:mysql:/etc/postfix/mysql-virtual_transports.cf
    unknown_local_recipient_reject_code = 550
    virtual_alias_domains =
    virtual_alias_maps = hash:/etc/mailman/virtual-mailman, proxy:mysql:/etc/postfix/mysql-virtual_forwardings.cf, proxy:mysql:/etc/postfix/mysql-virtual_email2email.cf
    virtual_gid_maps = mysql:/etc/postfix/mysql-virtual_gids.cf
    virtual_mailbox_base = /var/vmail
    virtual_mailbox_domains = proxy:mysql:/etc/postfix/mysql-virtual_domains.cf
    virtual_mailbox_maps = proxy:mysql:/etc/postfix/mysql-virtual_mailboxes.cf
    virtual_transport = dovecot
    virtual_uid_maps = mysql:/etc/postfix/mysql-virtual_uids.cf
     
  3. Nilpo

    Nilpo Member HowtoForge Supporter

    Ok, I think I'm on to the mail problem. I found this in /var/log/maillog
    Code:
    Mar  3 19:16:45 myhost amavis[18601]: (!)DENIED ACCESS from IP 123.45.67.89, policy bank 'ORIGINATING'
    
    Mar  3 19:16:45 myhost amavis[18627]: (!)DENIED ACCESS from IP 123.45.67.89, policy bank ''
    
    That is my server IP address. What's odd is that I have the amavis line commented out in main.cf right now because I can never get it working right. It shouldn't even doing anything!
     
  4. Nilpo

    Nilpo Member HowtoForge Supporter

    Ok, I was able to get Amavis fully disabled by commenting out the single lines in /etc/postfix/tag_as_foreign.re and /etc/postfix/tag_as_originating.re.

    Near as I can tell, Amavis doesn't like connections coming from my interface address instead of loopback. I'm not sure how to fix that, but mail is working fine with it disabled. Since amavis has never worked, I guess that's ok.

    I'm still stuck on the memory problem.
     
  5. till

    till Super Moderator Staff Member ISPConfig Developer

    Did you try to compile the alternative versions again. It might be that they are incompatible with seom shared libs of the system which were replaced by the update.
     
  6. Nilpo

    Nilpo Member HowtoForge Supporter

    I did not. I'll give that a try, Till. Thanks.
     
  7. Nilpo

    Nilpo Member HowtoForge Supporter

    Just to bring this thread to a close:
    I decided not to fix the installed versions of PHP. They were outdated (5.4, 5.5) and did not have caching extensions installed. Instead, I built and installed the latest versions of 5.6, 7.0, and 7.1 with memache, memcached, etc and removed the older versions. System PHP is still outdated 5.3, but I'm not adding any more repositories. I don't want to deal with anymore possible conflicts during routine upgrades.
     
    till likes this.

Share This Page