Email cleanup

Discussion in 'Feature Requests' started by liane, Nov 7, 2018.

  1. liane

    liane Member HowtoForge Supporter

    Hi,
    My users are increasingly using IMAP clients to manage their mails, and they do it badly, putting mails in trash (that may have different names with various clients, ex "trash", "corbeille"...), spam (that also have a plethora of names) and of course leave them there to fill space with junk.
    I think it could be a big management help to have settings, perhaps on a user or domain basis, to help clean up all this mess, something like:
    delete mails older than x days in folders "folder list"
    That could also be used to cleanup the "cur" dir (not the "new" dir of course) for users that think my server is named Gmail and they can use teras of space...
    Or do others have a smart solution to manage this problem?
     
  2. Jesse Norell

    Jesse Norell Well-Known Member

    Mailbox quotas of course prevent the "teras of space" issue. We often clear old mail out of I think Trash, Junk, Spam, Deleted, Deleted Messages. I believe there's also an imap folder flag to mark folders as "trash" that you could probably use to clear oddly named folders (when an imap client supports/sets that .. I think roundcube might). Often mail clients can be set to do some cleanup. Some users can be taught to do it, some can't. So a "smart" solution might be: enable quotas so things don't get too out of hand, make the quotas somewhat generous (measured in gb, not mb like 20 years ago), do what automatic cleanup you can, and implement a policy that it's the user's responsibility to manage the space themselves, along with providing some info/tips on how, and easily accessible means of seeing how much space of their limit they have used and where, and an appropriate level of customer service. (Eg. if you run your gmail account out of space, and go complaining to Google, what is the likely response? I've not tried, but I'd guess a brief explanation and then point you to info on how you can take care of the problem.)
     
  3. Jesse Norell

    Jesse Norell Well-Known Member

    In a few minutes of checking, dovecot probably doesn't support special use mailbox flags on your ISPConfig server if you haven't set it up yourself (see https://wiki.dovecot.org/MailboxSettings). Might be worth an feature request to do that, it wouldn't be that hard to configure (eg. see /etc/dovecot/conf.d/15-mailboxes.conf on a debian box). Your original feature request of listing mailbox names to automatically clean up is probably worth implementing, though I sure would not clear out old mail from INBOX/cur/ (or any other non-trash/non-spam folder) - sometimes very important messages are kept there a long time, for the very reason that they are important and not to be deleted.
     
  4. liane

    liane Member HowtoForge Supporter

    Thanks for your reply.
    Of course, quotas are enabled so that users can not fill all the available space (that does not prevent them from trying), they simply fill their mailbox, then complain...
    As you said, we can teach some... but others (many) no, hence my need for a tool to ease this burden and I really can not believe that I'm the only one to have this type of users.
    You are also right about not systematically applying this treatment to all mailboxes, hence the ability to select by mail / domain.
    I am far from being comfortable with dovecot, but your suggestion seems interesting to me, I will look on this side, thank you
     
  5. Jesse Norell

    Jesse Norell Well-Known Member

    I have not setup special_use flags myself, though it looks straight forward; but for a basic cleanup this is what I have (English named folders only):
    Code:
    [email protected]:~# cat /usr/local/sbin/mailbox-maintenance
    #!/bin/sh
    
    doveadm expunge -A \( mailbox Trash or mailbox SPAM or mailbox Junk or mailbox Deleted or mailbox Deleted\ Items \) savedbefore 30d
    
    [email protected]:~# cat /etc/cron.d/mailbox-maintenance
    SHELL=/bin/bash
    PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
    
    # expire old messages from Trash, SPAM, etc.
    15 1 * * *  root  /usr/local/sbin/mailbox-maintenance
    
    Remember to make /usr/local/sbin/mailbox-maintenance executable.
     
    till likes this.
  6. liane

    liane Member HowtoForge Supporter

    Hey, looks like a great start for what I'd like.
    Now, adding to that any localized variant of spam/trash names (the special_user flags of dovecot seems perfect for that).
    For the mail/domain filter, at worse, I can use a specific doveadm line for each that needs it, since I'm the only one creating the domains, that will be easy.
    Well, I still think that could be a feature of the ispconfig panel, but you guided me to a great solution, thanks!
     
  7. Jesse Norell

    Jesse Norell Well-Known Member

    Agreed, it would be a good (and fairly simple) feature to add. Probably make a default "folders to clean up" list in main config, and a max time in days, and possibly let individual email domains override those settings.
     
    liane likes this.
  8. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    if a user can not purge it's trash-folder, he will simply run into quota-problems. i don't like the idea, to automaticly empty certain mail-folders.
     
  9. liane

    liane Member HowtoForge Supporter

    Florian, he will run into quota problems, and then complain because he just don't know/care how to fix the problem. When you have a bunch of these, you start to feel the pain.
    I never suggested to systematically and automatically clean any folders, that must be optional, and only for specific cases, and well, what is the problem in cleaning the trash folder, it is the user who put the mails there, right?
    That said, it seems to be not that easy, for some reason, probably because I don't understand how /etc/dovecot/conf.d/15-mailboxes.conf is supposed to work, I'm not able to make "doveadm expunge" work properly, mails in ".Corbeille" are ignored and even .Trash is not cleaned in all mailboxes
     
  10. till

    till Super Moderator Staff Member ISPConfig Developer

    You are using ISPConfig? Then you are editing a wrong file, the whole dovecot configuration is in /etc/dovecot/dovecot.conf
     
  11. liane

    liane Member HowtoForge Supporter

    Oh, that's why anything I tried in 15-mailboxes.conf had no effect :/
    Yes, I'm using ISPConfig of course, so how can I declare in this file that ".Corbeille" is to be treated like ".Trash" with the "doveadm expunge" command? do I just paste in this file what was in the 15-mailboxes.conf file? That would be handy since I could then use the autoexpunge parameter (and if dovecot has this parameter, maybe it is that I'm not the only one needing it ;)
    or perhaps I'm not using the right command at all?
     
  12. Jesse Norell

    Jesse Norell Well-Known Member

    just add that to the list of folder named to be expunged (without the leading period). See the example I put above.
     
  13. liane

    liane Member HowtoForge Supporter

    Tried that, but unfortunately, it doesn't work (with an existing [email protected] of course)
    Code:
    doveadm expunge -u [email protected] mailbox Corbeille savedbefore 2w
    and I still have old mails in folder .Corbeille
    Code:
    ls -lht .Corbeille/cur/ | tail -2
    -rw------- 1 vmail vmail  25K Mar  5  2018 1520258137.M375465P28131.mail1,S=25034,W=25457:2,S
    -rw------- 1 vmail vmail 3.0K Mar  5  2018 1520258127.M151103P28125.mail1,S=3053,W=3162:2,S
    
     
  14. Jesse Norell

    Jesse Norell Well-Known Member

    Is the folder name correct? Eg. maybe it's INBOX/Corbeille ?

    Has that mail been in that same folder for that long, or was it recently moved from another folder?

    What does 'doveadm user -u [email protected]' show?
     
  15. liane

    liane Member HowtoForge Supporter

    I think it's the correct name
    Code:
    # /var/vmail/domain/user/Maildir# ls -lha
    total 1.1M
    drwx------ 12 vmail vmail 4.0K Nov 14 19:05 .
    drwx------  4 vmail vmail 4.0K Nov 14 19:04 ..
    drwx------  5 vmail vmail 4.0K Mar  4  2018 .Brouillons
    drwx------  5 vmail vmail 4.0K Nov 14 17:44 .Corbeille
    drwxr-xr-x  2 vmail vmail 224K Nov 14 19:04 cur
    -rw-------  1 vmail vmail  13K Nov 14 17:15 dovecot.index
    -rw-------  1 vmail vmail 668K Nov 14 19:06 dovecot.index.cache
    -rw-------  1 vmail vmail  19K Nov 14 19:05 dovecot.index.log
    -rw-------  1 vmail vmail  33K Nov 14 13:51 dovecot.index.log.2
    -rw-------  1 vmail vmail   13 Dec 29  2017 dovecot-keywords
    -rw-------  1 vmail vmail  45K Nov 14 19:00 dovecot-uidlist
    -rw-------  1 vmail vmail    8 Jul  9 13:09 dovecot-uidvalidity
    -r--r--r--  1 vmail vmail    0 Feb 20  2017 dovecot-uidvalidity.58aaa814
    drwx------  5 vmail vmail 4.0K Feb 20  2017 .Drafts
    drwx------  5 vmail vmail 4.0K Nov  8 13:24 .Envoy&AOk-s
    drwx------  5 vmail vmail 4.0K Feb 20  2017 .Junk
    drwxr-xr-x  2 vmail vmail  12K Nov 14 19:00 new
    drwx------  5 vmail vmail 4.0K Mar  3  2018 .Sent
    -rwxr--r--  1 vmail vmail   23 Nov  8 13:28 subscriptions
    drwx------  2 vmail vmail 4.0K Nov 14 19:00 tmp
    drwx------  5 vmail vmail 4.0K Nov  8 10:43 .Trash
    
    # /var/vmail/domain/user/Maildir# ls -lh .Corbeille/
    total 28K
    drwx------ 2 vmail vmail 4.0K Jul 17 10:42 cur
    -rw------- 1 vmail vmail 2.2K Nov 14 17:31 dovecot.index.cache
    -rw------- 1 vmail vmail 5.5K Nov  8 14:13 dovecot.index.log
    -rw------- 1 vmail vmail 2.9K Jul 17 10:42 dovecot-uidlist
    -rw------- 1 vmail vmail    0 Mar  5  2018 maildirfolder
    drwx------ 2 vmail vmail 4.0K Mar  5  2018 new
    drwx------ 2 vmail vmail 4.0K Jul 17 10:42 tmp
    
    It was there from the start, it is my mailbox so I'm pretty sure it wasn't moved recently, I might be wrong, but isn't the mail date that is used?
    Code:
    # doveadm user -u [email protected]
    userdb: [email protected]
      user      : [email protected]
      home      : /var/vmail/domain/user
      mail      : maildir:/var/vmail/domain/user/Maildir
      uid       : 5000
      gid       : 5000
      quota_rule: *:storage=524288000B
      sieve     : /var/vmail/domain/user/.sieve
    
     
    Last edited: Nov 14, 2018 at 7:15 PM
  16. Jesse Norell

    Jesse Norell Well-Known Member

    That all looks good.
    'savedbefore' appears to use the ctime of the file itself; you can also use 'before' (from "internal" date) and 'sentbefore' (from Date: header); see doveadm-search-query man page for more options.

    I'd try playing around with searches and figure out what's not matching; eg. "doveadm search -u [email protected] ALL" vs. "doveadm search -u [email protected] mailbox INBOX" vs. "doveadm search -u [email protected] mailbox Corbeille" vs. "doveadm search -u [email protected] mailbox Corbeille savedbefore 1h". My guess is something reset ctime on your files; for that matter, "ls -lc /var/vmail/domain/user/Maildir/.Corbeille/???/" should show you what ctimes are.
     
    liane likes this.
  17. liane

    liane Member HowtoForge Supporter

    those all list all files in .Corbeille folder
    mmm, good guess, all those files show a date of Nov 8 13:28 ?!?!?!
    So, given that what imports me is the mail date, your suggestion to use "before" gives exactly the result I want ("sentbefore" looks ok, but seems slow)
    Thanks a lot for your help!
     

Share This Page