Issues updating from Ubuntu 18.04 to Ubuntu 20.04

Discussion in 'General' started by Bob White, Aug 22, 2021.

  1. Bob White

    Bob White Member

    I'm currently running the perfect server under Ubuntu 18.04 LTS with ISPConfig 3.2.2. All has been running well for several years. Last night, I tried updating to Ubuntu 20.04 LTS and things didn't go well. Fortunately, I have a couple of spare hard drives, so I cloned the working drive onto them, swapped out the working drive with one of the new clones, and tried updating the clone.
    The first issue was when it was trying to update the myphpadmin database. Got errors saying it couldn't connect to the database. Eventually, I figured out that mysqld wasn't running. I connected, started mysqld, and that seemed to get past that issue.
    I've been updating Roundcube as new versions came out. The update process was, I think, trying to update Roundcube to 1.3.6 and failing because I'm running version 1.4.11. I finally told it to abort that update, and it moved on, only to pitch fits at the end of the update when it tried to install it again. As a result, I'm not sure the update completed successfully. Is there a better way around this?
    The end result was a bootable system, but trying to connect to Roundcube resulted in a white screen.
    I think I need to try it again, going through the 20.04 server configuration to make sure there's not something missing.
    Would I be better off to wait until 22.04 comes out next year, make a new, clean installation, and copy the vmail folders from the 18.04 configuration to the new one?
    Thanks,
    Bob
     
  2. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Ubuntu 18.04 dist upgrade to 20.04 assumes the OS is plain 18.04. If you have installed roundcube from outside Ubuntu 18.04 there is no guarantee the upgrade goes successfully. Same for all othere software. Examine the Ubuntu upgrade instructions and preconditions, I'm sure they are documented somewhere.
    When you have successfully done the upgrade to 20.04, follow the ISPConfig Perfect server guider for 20.04 to install the necessary packages and do the configurations. Then do
    Code:
    ispconfig_update.sh --force
    to force ISPConfig to configure for applications properly.
    If your testing shows dist-upgrade does not work, another way is to install a new empty ISConfig 3.2 on Ubuntu 20.04 and use ISPConfig Migration Tool to copy the data from old server to this new. You need two servers running at the same time.
     
    Last edited: Aug 23, 2021
    ahrasis likes this.
  3. Bob White

    Bob White Member

    I initially installed Roundcube when I first built the server, but have been updating it from outside since then. I guess that's the problem there. I may try backing up the Roundcube database, uninstalling it via apt, doing the upgrade to 20.04, and then putting Roundcube back.
    The only problem I can see with having an empty ISPConfig 3.2 on a new 20.04 server and migrating is the hardware involved - I'd be installing it on different hardware than what I would want to run it on long-term. It is an option, and one I can try if all else fails.
    Thanks,
    Bob
     
  4. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    Basically I think RC won't affect much of your server upgrade as it is merely a website. So long you have your database and mail server are backup properly, you shouldn't be worried in upgrading to Ubuntu 20.04. You can surely rebuild RC after you finished with the upgrade.
     
  5. Bob White

    Bob White Member

    Well, I'm almost there. I've upgraded the server and gotten almost everything working again. Except for SMTP authentication and ISPConfig.
    I can receive email and it shows up in my Roundcube inbox. When I try to send email, I get a SMTP Error 535: Authentication failed. I send email out through Google's SMTP server, so it may be failing there. I believe I made that setting in ISPConfig, but when I go to ISPConfig, I just get a blank screen, no login prompt - nothing.

    Is there something that didn't get put in the right place for the ISPConfig web page to come up? The two ispconfig files are in the Apache sites-enabled folder. There's nothing in the /var/www/html folder, which is the root web folder.

    Here is master.cf:
    Code:
    #
    # Postfix master process configuration file.  For details on the format
    # of the file, see the master(5) manual page (command: "man 5 master" or
    # on-line: http://www.postfix.org/master.5.html).
    #
    # Do not forget to execute "postfix reload" after editing this file.
    #
    # ==========================================================================
    # service type  private unpriv  chroot  wakeup  maxproc command + args
    #               (yes)   (yes)   (no)    (never) (100)
    # ==========================================================================
    smtp      inet  n       -       y       -       -       smtpd
    #smtp      inet  n       -       y       -       1       postscreen
    #smtpd     pass  -       -       y       -       -       smtpd
    #dnsblog   unix  -       -       y       -       0       dnsblog
    #tlsproxy  unix  -       -       y       -       0       tlsproxy
    submission inet n       -       y       -       -       smtpd
      -o syslog_name=postfix/submission
      -o smtpd_tls_security_level=encrypt
      -o smtpd_sasl_auth_enable=yes
      -o smtpd_client_restrictions=permit_sasl_authenticated,reject
    #  -o smtpd_reject_unlisted_recipient=no
    #  -o smtpd_client_restrictions=$mua_client_restrictions
    #  -o smtpd_helo_restrictions=$mua_helo_restrictions
    #  -o smtpd_sender_restrictions=$mua_sender_restrictions
    #  -o smtpd_recipient_restrictions=
    #  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
    #  -o milter_macro_daemon_name=ORIGINATING
    smtps     inet  n       -       y       -       -       smtpd
      -o syslog_name=postfix/smtps
      -o smtpd_tls_wrappermode=yes
      -o smtpd_sasl_auth_enable=yes
      -o smtpd_client_restrictions=permit_sasl_authenticated,reject
    #  -o smtpd_reject_unlisted_recipient=no
    #  -o smtpd_client_restrictions=$mua_client_restrictions
    #  -o smtpd_helo_restrictions=$mua_helo_restrictions
    #  -o smtpd_sender_restrictions=$mua_sender_restrictions
    #  -o smtpd_recipient_restrictions=
    #  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
    #  -o milter_macro_daemon_name=ORIGINATING
    #628       inet  n       -       y       -       -       qmqpd
    pickup    unix  n       -       y       60      1       pickup
    cleanup   unix  n       -       y       -       0       cleanup
    qmgr      unix  n       -       n       300     1       qmgr
    #qmgr     unix  n       -       n       300     1       oqmgr
    tlsmgr    unix  -       -       y       1000?   1       tlsmgr
    rewrite   unix  -       -       y       -       -       trivial-rewrite
    bounce    unix  -       -       y       -       0       bounce
    defer     unix  -       -       y       -       0       bounce
    trace     unix  -       -       y       -       0       bounce
    verify    unix  -       -       y       -       1       verify
    flush     unix  n       -       y       1000?   0       flush
    proxymap  unix  -       -       n       -       -       proxymap
    proxywrite unix -       -       n       -       1       proxymap
    smtp      unix  -       -       y       -       -       smtp
    relay     unix  -       -       y       -       -       smtp
    #       -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
    showq     unix  n       -       y       -       -       showq
    error     unix  -       -       y       -       -       error
    retry     unix  -       -       y       -       -       error
    discard   unix  -       -       y       -       -       discard
    local     unix  -       n       n       -       -       local
    virtual   unix  -       n       n       -       -       virtual
    lmtp      unix  -       -       y       -       -       lmtp
    anvil     unix  -       -       y       -       1       anvil
    scache    unix  -       -       y       -       1       scache
    #
    # ====================================================================
    # Interfaces to non-Postfix software. Be sure to examine the manual
    # pages of the non-Postfix software to find out what options it wants.
    #
    # Many of the following services use the Postfix pipe(8) delivery
    # agent.  See the pipe(8) man page for information about ${recipient}
    # and other message envelope options.
    # ====================================================================
    #
    # maildrop. See the Postfix MAILDROP_README file for details.
    # Also specify in main.cf: maildrop_destination_recipient_limit=1
    #
    maildrop  unix  -       n       n       -       -       pipe
      flags=DRhu user=vmail argv=/usr/bin/maildrop -d vmail ${extension} ${recipient} ${user} ${nexthop} ${sender}
    #
    # ====================================================================
    #
    # Recent Cyrus versions can use the existing "lmtp" master.cf entry.
    #
    # Specify in cyrus.conf:
    #   lmtp    cmd="lmtpd -a" listen="localhost:lmtp" proto=tcp4
    #
    # Specify in main.cf one or more of the following:
    #  mailbox_transport = lmtp:inet:localhost
    #  virtual_transport = lmtp:inet:localhost
    #
    # ====================================================================
    #
    # Cyrus 2.1.5 (Amos Gouaux)
    # Also specify in main.cf: cyrus_destination_recipient_limit=1
    #
    And here is main.cf

    Code:
    # See /usr/share/postfix/main.cf.dist for a commented, more complete version
    
    
    # Debian specific:  Specifying a file name will cause the first
    # line of that file to be used as the name.  The Debian default
    # is /etc/mailname.
    myorigin = /etc/mailname
    
    smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
    biff = no
    
    # appending .domain is the MUA's job.
    append_dot_mydomain = no
    
    # Uncomment the next line to generate "delayed mail" warnings
    #delay_warning_time = 4h
    
    readme_directory = /usr/share/doc/postfix
    
    # TLS parameters
    smtpd_tls_cert_file = /etc/postfix/smtpd.cert
    smtpd_tls_key_file = /etc/postfix/smtpd.key
    smtpd_use_tls = yes
    smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
    smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
    
    # See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
    # information on enabling SSL in the smtp client.
    
    smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
    myhostname = fileserver.thecovey.net
    alias_maps = hash:/etc/aliases, hash:/var/lib/mailman/data/aliases
    alias_database = hash:/etc/aliases, hash:/var/lib/mailman/data/aliases
    mydestination = fileserver.thecovey.net, localhost, localhost.localdomain
    mynetworks = 127.0.0.0/8, 192.168.1.0/24
    mailbox_size_limit = 0
    recipient_delimiter = +
    inet_interfaces = all
    inet_protocols = ipv4
    html_directory = /usr/share/doc/postfix/html
    virtual_alias_domains =
    virtual_alias_maps = hash:/var/lib/mailman/data/virtual-mailman, proxy:mysql:/etc/postfix/mysql-virtual_forwardings.cf, proxy:mysql:/etc/postfix/mysql-virtual_email2email.cf
    virtual_mailbox_domains = proxy:mysql:/etc/postfix/mysql-virtual_domains.cf
    virtual_mailbox_maps = proxy:mysql:/etc/postfix/mysql-virtual_mailboxes.cf
    virtual_mailbox_base = /var/vmail
    virtual_uid_maps = mysql:/etc/postfix/mysql-virtual_uids.cf
    virtual_gid_maps = mysql:/etc/postfix/mysql-virtual_gids.cf
    sender_bcc_maps = proxy:mysql:/etc/postfix/mysql-virtual_outgoing_bcc.cf
    smtpd_sasl_auth_enable = yes
    smtpd_sasl_authenticated_header = yes
    smtpd_restriction_classes = greylisting
    greylisting = check_policy_service inet:127.0.0.1:10023
    smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination, reject_rbl_client zen.spamhaus.org, check_recipient_access mysql:/etc/postfix/mysql-virtual_recipient.cf, check_recipient_access mysql:/etc/postfix/mysql-virtual_policy_greylist.cf
    smtpd_tls_security_level = may
    transport_maps = hash:/var/lib/mailman/data/transport-mailman, proxy:mysql:/etc/postfix/mysql-virtual_transports.cf
    relay_domains = mysql:/etc/postfix/mysql-virtual_relaydomains.cf
    relay_recipient_maps = mysql:/etc/postfix/mysql-virtual_relayrecipientmaps.cf
    smtpd_sender_login_maps = proxy:mysql:/etc/postfix/mysql-virtual_sender_login_maps.cf
    proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps $virtual_alias_domains $sender_bcc_maps $virtual_mailbox_maps $virtual_mailbox_domains $relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps $recipient_canonical_maps $relocated_maps $transport_maps $mynetworks $smtpd_sender_login_maps
    smtpd_helo_required = yes
    smtpd_helo_restrictions = permit_sasl_authenticated, permit_mynetworks, check_helo_access regexp:/etc/postfix/helo_access, reject_invalid_hostname, reject_non_fqdn_hostname, reject_invalid_helo_hostname, reject_unknown_helo_hostname, check_helo_access regexp:/etc/postfix/blacklist_helo
    smtpd_sender_restrictions = check_sender_access regexp:/etc/postfix/tag_as_originating.re , permit_mynetworks, permit_sasl_authenticated, check_sender_access mysql:/etc/postfix/mysql-virtual_sender.cf, check_sender_access regexp:/etc/postfix/tag_as_foreign.re
    smtpd_client_restrictions = check_client_access mysql:/etc/postfix/mysql-virtual_client.cf
    smtpd_client_message_rate_limit = 100
    maildrop_destination_concurrency_limit = 1
    maildrop_destination_recipient_limit = 1
    virtual_transport = dovecot
    
    header_checks = pcre:/etc/postfix/header_checks
    mime_header_checks = pcre:/etc/postfix/mime_header_checks
    nested_header_checks = pcre:/etc/postfix/nested_header_checks
    body_checks = pcre:/etc/postfix/body_checks
    
    owner_request_special = no
    smtp_tls_security_level = may
    smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3
    smtpd_tls_protocols = !SSLv2,!SSLv3
    smtp_tls_protocols = !SSLv2,!SSLv3
    smtpd_tls_exclude_ciphers = RC4, aNULL
    smtp_tls_exclude_ciphers = RC4, aNULL
    dovecot_destination_recipient_limit = 1
    smtpd_sasl_type = dovecot
    smtpd_sasl_path = private/auth
    content_filter = amavis:[127.0.0.1]:10024
    receive_override_options = no_address_mappings
    smtp_generic_maps = hash:/etc/postfix/generic
    
    home_mailbox = Maildir/
    
    # Postfix configuration to relay e-mail through ISP
    
    relayhost = [smtp.gmail.com]:587
    smtp_sasl_auth_enable = yes
    smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
    smtp_sasl_security_options =
    # Enable STARTTLS encryption
    smtp_use_tls = yes
    
    # where to find CA certificates
    smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
    
    # Postfix configuration as SMTP server
    
    #smtpd_sasl_local_domain =
    #smtpd_sasl_security_options =
    broken_sasl_auth_clients = yes
    smtpd_sasl_auth_enable = yes
    
    message_size_limit = 0
    
     
  6. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    You need to force update ISPConfig reconfiguring all services after upgrading Ubuntu.
     
  7. Bob White

    Bob White Member

    I did that - ispconfig_update.sh --force. Seemed to complete without errors after I installed curl and a couple of PHP packages.
     
  8. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Have you done

     
    ahrasis likes this.
  9. Bob White

    Bob White Member

    I figured it out. The roundcube configuration wasn't passing the username and password when it tried to connect. Once I updated config.inc.php to include this:
    $config['smtp_user'] = '%u';
    $config['smtp_pass'] = '%p';​
    I was able to send email.
     
    ahrasis likes this.
  10. Bob White

    Bob White Member

    ISPConfig's web interface still doesn't come up, however.
     
  11. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    Is your web server up? Are other websites on that server working properly? If you try to access it locally, have you tried to access it externally? Have you try accessing via ip address instead?
     
  12. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    Did you read #8 and did you run the update with reconfigure services AFTERWARDS?
     
  13. Bob White

    Bob White Member

    I executed the "ispconfig_update --force" afterwards, if that's what you mean.
    Bob
     
  14. Bob White

    Bob White Member

    Here's the last few lines of the apache error log. Seems to have some problems with certificate stapling, whatever that is. I don't have any other websites - I use this as an email consolidator and file server for our home network. I tried connecting via the IP address, got a warning about the certificate, and then a blank screen when I proceeded. The last line of the log says the root of the web server doesn't have an index file in it, which I mentioned earlier. I have anonymized the lines since I'm posting it publicly.

    Code:
    [Mon Sep 06 10:08:11.565016 2021] [ssl:error] [pid 61805] AH02604: Unable to configure certificate ********.thecovey.net:8080:0 for stapling
    [Mon Sep 06 10:08:11.567077 2021] [ssl:warn] [pid 61805] AH01906: ********.thecovey.net:8081:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
    [Mon Sep 06 10:08:11.567354 2021] [ssl:error] [pid 61805] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: emailAddress=*******@triad.rr.com,CN=*******.thecovey.net,OU=IT Department,O=Home,L=City,ST=State,C=US / issuer: <redacted>,C=US / serial: 948493D8D6B0057E / notbefore: Nov 19 03:48:43 2016 GMT / notafter: Nov 17 03:48:43 2026 GMT]
    [Mon Sep 06 10:08:11.567416 2021] [ssl:error] [pid 61805] AH02604: Unable to configure certificate ********.thecovey.net:8081:0 for stapling
    [Mon Sep 06 10:08:11.567697 2021] [:error] [pid 61805] python_init: Python version mismatch, expected '2.7.17', found '2.7.18'.
    [Mon Sep 06 10:08:11.567896 2021] [:error] [pid 61805] python_init: Python executable found '/usr/bin/python'.
    [Mon Sep 06 10:08:11.567914 2021] [:error] [pid 61805] python_init: Python path being used '/usr/lib/python2.7:/usr/lib/python2.7/plat-x86_64-linux-gnu:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-old:/usr/lib/python2.7/lib-dynload'.
    [Mon Sep 06 10:08:11.567416 2021] [ssl:error] [pid 61805] AH02604: Unable to configure certificate ********.thecovey.net:8081:0 for stapling
    [Mon Sep 06 10:08:11.567697 2021] [:error] [pid 61805] python_init: Python version mismatch, expected '2.7.17', found '2.7.18'.
    [Mon Sep 06 10:08:11.567896 2021] [:error] [pid 61805] python_init: Python executable found '/usr/bin/python'.
    [Mon Sep 06 10:08:11.567914 2021] [:error] [pid 61805] python_init: Python path being used '/usr/lib/python2.7:/usr/lib/python2.7/plat-x86_64-linux-gnu:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-old:/usr/lib/python2.7/lib-dynload'.
    [Mon Sep 06 10:08:11.567966 2021] [:notice] [pid 61805] mod_python: Creating 8 session mutexes based on 150 max processes and 0 max threads.
    [Mon Sep 06 10:08:11.567986 2021] [:notice] [pid 61805] mod_python: using mutex_directory /tmp
    [Mon Sep 06 10:08:11.605602 2021] [mpm_prefork:notice] [pid 61805] AH00163: Apache/2.4.41 (Ubuntu) mod_fcgid/2.3.9 OpenSSL/1.1.1f mod_python/3.3.1 Python/2.7.18 configured -- resuming normal operations
    [Mon Sep 06 10:08:11.605683 2021] [core:notice] [pid 61805] AH00094: Command line: '/usr/sbin/apache2'
    [Mon Sep 06 10:10:02.303655 2021] [autoindex:error] [pid 61813] [client ::1:37218] AH01276: Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm,index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm,standard_index.html) found, and server-generated directory index forbidden by Options directive
    
     
  15. Bob White

    Bob White Member

    I've made some progress with email. Whatever was supposed to populate dh.pem created an empty file. I found a thread where Till mentioned how to populate it using Openssl. When I did that, Dovecot is much happier and sends and receives email happily. Still no web interface to ISPConfig, however.
     
  16. Chris_UK

    Chris_UK Active Member HowtoForge Supporter

    Did you opt to have ISPConfig installed on the server (the panel)
     
  17. Bob White

    Bob White Member

    The server has been running ISPConfig for years. I was running version 3.2.2 on Ubuntu 18.04, and was upgrading the server to Ubuntu 20.04. I didn't reinstall it, no. I could access it on port 8080 prior to the OS upgrade.
     
  18. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    last chance: did you follow #8?
     
  19. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    I think he did not though we all know that following every steps in Ubuntu 20.04 tutorial is necessary as well reinstalling ISPConfig and reconfiguring all services thereafter.
     
  20. Bob White

    Bob White Member

    I was rapidly coming to the conclusion that even though I had an up-to-date version of ISPConfig installed before the update, that it needed to be reinstalled after the update.
    I appreciate all the help and the work behind the Perfect Server and all the other instructions you guys have built. It would be nice if the upgrade process was a little more documented. It's the "we all know" part that isn't necessarily communicated. When I went through the 20.04 tutorial, I found that everything that had been done by the 18.04 tutorial was already done and there were just a few odds and ends of things that needed to be done. My assumption - and it was a bad one, and my fault - was that ISPConfig was still properly configured. It isn't, and I'll fix that.
    Thanks,
    Bob
     

Share This Page