Sieve scripts & ISPConfig upgrade

Discussion in 'Installation/Configuration' started by sonik14, Mar 31, 2022.

  1. sonik14

    sonik14 New Member

    Hello,
    I recently upgraded to the latest ISPConfig version through the `ispconfig_update.sh` script. This reset my `sieve_filter.master` to the default. I was using plugin `vacation-seconds`. However, all the mailboxes still have in their script the `vacation-seconds` plugin. This should have been updated/rewriten automatically by ISPConfig but it isn't.
    The only solution I have found for now is to manually save each mailbox again. This indeed overwrites the `/var/vmail/domain/usermail/.ispconfig.sieve`. However I have many users. Can this be done automatically somehow? Re-sync option is not doing anything about this.
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    This means you missed making your changes in an update-safe way by putting the template file in /usr/local/ispconfig/server/conf-custom/

    No, this should not be rewritten automatically on update. neither websites nor email accounts shall get altered when an update is installed.

    If resync is not being able to alter that, then there is no automatic way in doing this.
     
    sonik14 likes this.
  3. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Just a thought, you might be able to loop through all mail accounts in the api, read the old script and add a newline or comment to it and write it back, I think that would trigger a rewrite. Indeed, reworking things so a resync would rewrite those would be handy at times.
     
  4. sonik14

    sonik14 New Member

    So I have another issue too after the upgrade. When I create a new email through ISPConfig, everything seems fine, but this email cannot receive ANY email from another domain. The emails do not stay in queue and also the sender does not receive any bounce for failure, so they fail to be delivered silently. Old emails still receive properly even if they trigger errors when processing mail filters because vacation-seconds does not exist.

    Even if i remove ALL the below custom configuration I had done, the issue persists.

    The log I get in mail.info
    Before the Upgrade these were the custom changes I had made in the configuration files. Some of them were reset after the upgrade.

     
    Last edited: Apr 6, 2022
  5. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    I would try undoing your amavis changes first. This:
    Is address verification (via 127.0.0.1:10025) failing because of recipient addr restrictions; normally these are empty for port 10025, but I'm not clear on the full mail delivery path (loop?) you have created with your changes.

    An alternative might be to just remove recipient verification entirely via more custom postfix config; it is only there to catch some corner cases in mail forwarding and probably not needed on most systems.
     
    Last edited: Apr 6, 2022
  6. sonik14

    sonik14 New Member

    I have tried removing all the custom conf of amavis but it still doesn't work (only for new emails created). Before the upgrade everything was working fine. The amavis conf was needed because gmail was rejecting the automatic replies generated, because it was not validating DKIM, because DKIM runs via amavis and these emails (pickup-sieve) were skipping amavis completely and I found this solution that worked for me.

    So for now I don't know what may be messed by my previous changes after the upgrade, but I can't spot it even if I revert everything.

    ---

    EDIT:

    This works. Thank you, but I would really like if I could spot the cause.
    What makes it fail is:
    Code:
    check_recipient_access proxy:mysql:/etc/postfix/mysql-verify_recipients.cf
    so it works if I use
    Code:
    smtpd_recipient_restrictions = permit_mynetworks, reject_unknown_recipient_domain, reject_unlisted_recipient, permit_sasl_authenticated, reject_non_fqdn_recipient, reject_unauth_destination, check_recipient_access proxy:mysql:/etc/postfix/mysql-virtual_recipient.cf, check_recipient_access mysql:/etc/postfix/mysql-virtual_policy_greylist.cf, check_policy_service unix:private/quota-status
    
     
    Last edited: Apr 7, 2022
  7. sonik14

    sonik14 New Member

    I am trying now to configure in the safe-way. However I don't understand why `/etc/dovecot/dovecot.conf` does not contain
    Code:
    !include conf.d/*.conf
    So it does not make sense to edit any file in `conf.d`. So how do I add vacation-seconds sieve plugin to be upgrade safe?
     
  8. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Correct, all configuration has been in the single dovecot.conf file for quite some years, though more recently a second file is included for local customizations. I've not tested your exact settings, but try putting them in /usr/local/ispconfig/server/conf-custom/install/dovecot_custom.conf.master and either copy to /etc/dovecot/conf.d/99-ispconfig-custom-config.conf or have the installer do that by rerunning and letting it reconfigure services. Most (though I think not 100%) things needing changed can be done via that custom file which is nicer because you don't have to merge your copy with upstream changes.
     
    sonik14 likes this.
  9. sonik14

    sonik14 New Member

    This works smoothly. Thanks.
    Summary:
    My current custom files are:
    • /usr/local/ispconfig/server/conf-custom/install/dovecot_custom.conf.master
    • /usr/local/ispconfig/server/conf-custom/sieve_filter.master
    • /usr/local/ispconfig/server/conf-custom/install/mm_cfg.py.master
    Dovecot is configured automatically and sieve custom filters are applied ONLY to new mailboxes or if an existing manually modifies filters. Re-sync does not synchronize sieve files, so I cannot cascade any change for old users.

    ---

    So the only question left is: Why this recipient restriction in postfix/main does not allow receiving emails only for new mailboxes after the upgrade?
    Code:
    check_recipient_access proxy:mysql:/etc/postfix/mysql-verify_recipients.cf
    with this error
     
    Last edited: Apr 8, 2022
  10. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Have you created proxymap for that table?
     
  11. sonik14

    sonik14 New Member

    No but I haven't also added this configuration line in postfix. ISPConfig adds it while reconfiguring services.
     
  12. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Ispconfig also adds that to proxy_read_maps when reconfiguring services; sounds like you have additional custom config that needs updated.
     
    Last edited: Apr 7, 2022
  13. sonik14

    sonik14 New Member

    Any tip on how I could spot these, because there is nothing written in our Issue log?

    The only custom change I had was amavis and debian 11bug (https://www.howtoforge.com/community/threads/bullseye-for-ispconfig.87450/page-2#post-427169) which is now fixed.

    The only custom conf I have found is the following related to mailman but I don't know if it could affect it and it was also working fine before.

     
    Last edited: Apr 7, 2022
  14. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    For changes to the upstream templates, you could compare with diff if you have a copy of the old and new, or use gitlab to navigate changes to a file (eg. click on History to see commits ... there should be ways to compare by tags, etc.).
     
  15. sonik14

    sonik14 New Member

    What about this query in `mysql-verify_recipients.cf` that causes the issue?
    Code:
    query = SELECT 'reject_unverified_recipient' FROM mail_domain WHERE domain = '%s' AND active = 'y' AND server_id = 1
    
    The column reject_unverified_recipient does not exist in my database table.

    That's the structure:
    Code:
    # Name Type Collation Attributes Null Default Comments Extra
    
    UNSIGNED No None
    AUTO_INCREMENT
    2  sys_userid  int(11)
    UNSIGNED No 0
    
    3  sys_groupid  int(11)
    UNSIGNED No 0
    
    4  sys_perm_user  varchar(5) utf8_general_ci
    No
    
    
    5  sys_perm_group  varchar(5) utf8_general_ci
    No
    
    
    6  sys_perm_other  varchar(5) utf8_general_ci
    No
    
    
    7  server_id ]  int(11)
    UNSIGNED No 0
    
    8  domain  varchar(255) utf8_general_ci
    No
    
    
    9  dkim  enum('n', 'y') utf8_general_ci
    No n
    
    10  dkim_selector  varchar(63) utf8_general_ci
    No default
    
    11  dkim_private  mediumtext utf8_general_ci
    Yes NULL
    
    12  dkim_public  mediumtext utf8_general_ci
    Yes NULL
    
    13  relay_host  varchar(255) utf8_general_ci
    No
    
    
    14  relay_user  varchar(255) utf8_general_ci
    No
    
    
    15  relay_pass  varchar(255) utf8_general_ci
    No
    
    
    16  active  enum('n', 'y') utf8_general_ci
    No n
    
     
  16. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    It is a literal string, not a table column.
     
    sonik14 likes this.
  17. sonik14

    sonik14 New Member

    I am posting the diff between my old main.cf and the main.cf (same for master) after the upgrade and the automatic reconfiguration:

    With the old conf it works properly.

    Specifically I need to comment out these two lines from main.cf so new mailboxes can receive emails.
    Code:
    address_verify_virtual_transport = smtp:[127.0.0.1]:10025
    address_verify_transport_maps = static:smtp:[127.0.0.1]:10025
    
    Apparently I also need to add to master.cf these lines (by adding them in /usr/local/ispconfig/server/conf-custom/install/postfix_custom.conf.master they don't have any effect)
    Code:
    submission     inet     n    -    y    -    -    smtpd
      -o syslog_name=postfix/submission
      -o smtpd_tls_security_level=encrypt
      -o smtpd_tls_wrappermode=no
      -o smtpd_sasl_auth_enable=yes
      -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
      -o smtpd_recipient_restrictions=permit_mynetworks,permit_sasl_authenticated,reject
      -o smtpd_sasl_type=dovecot
      -o smtpd_sasl_path=private/auth
    
    smtps     inet  n       -       y       -       -       smtpd
      -o syslog_name=postfix/smtps
      -o smtpd_tls_wrappermode=yes
      -o smtpd_sasl_auth_enable=yes
      -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
      -o smtpd_recipient_restrictions=permit_mynetworks,permit_sasl_authenticated,reject
      -o smtpd_sasl_type=dovecot
      -o smtpd_sasl_path=private/auth
    
     

    Attached Files:

    Last edited: Apr 8, 2022

Share This Page