ISPCONFIG3 - and fetch/getmail and imapsync (sad story)

Discussion in 'ISPConfig 3 Priority Support' started by craig baker, Mar 28, 2014.

  1. craig baker

    craig baker Member HowtoForge Supporter

    ok - just noticed a sadness. I've been guided to imapsync to migrate mail from one host to another - works just great!

    However I wanted to use fetchmail/getmail to keep the box up to date - and it absolutely 100% is duplicating existing messages that imapsync has already downloaded.

    it happens whether you configure getmail to be pop or imap
    (and btw - what about the case where the host wants and oddball or encrypted pop or imap setting? dont see the settings anywhere?

    now I can just cron imapsync every 20 minutes and forget about getmail but thats not the ispconfig way is it?

    any idea why getmail is duplicating?

    also - no note anywhere I could find - how often is getmail run when you have it active on a mailbox?

    I see in the /etc/getmail folder lines like these:
    -rw------- 1 getmail getmail 4507056 Mar 28 08:15 [email protected]
    -rw------- 1 getmail getmail 12816 Mar 28 08:15 [email protected]

    now presumably these are what getmail uses to check that its not already gotten a message. but imapsync delivers right to the Maildir - so
    it seems getmail cannot really (ever) tell its already got that message sitting in the Maildir.

    any way to have it check the Maildir for the message rather than check this local file?
    or anyway to (ahem) create this oldmail file above FROM the Maildir contents so getmail wont get messages that are already there?
    or do I have to forget about getmail completely?

    I saw that dovecot now has a deduplicate option - but alas the current dovecot available via centos repo is 2.0.x and that option is not available till 2.2 it seems.

    HOWEVER - that obviously wont help me in the long run as getmail looks like it will simply keep duplicating until it has processed all 135,000 messages!

    any ideas on how to solve?
    ps I LOVE Ispconfig3. get tired of reading that Till? <grin>
    Last edited: Mar 28, 2014
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    getmail does not know that you downloaded the emails with imapysync, so it downloads them again until it has downloaded all messages. There must be some kind of file or database were getmail tracks which was the last email it has downloaded, so there must be a way to trick it so that it assumes all mails were downloaded already. But I dont know which file this is, so you might want to ask this on the getmail user list:
  3. craig baker

    craig baker Member HowtoForge Supporter

    imapsync woes

    yes I had already determined this is exactly the problem getmail simply has no idea that imapsync was invoked so it retrieves all the emails as well and dupes everything.

    I had already looked over getmails docs and there seems to be no solution - there is a way to throttle down getmail so it gets only so many messages per session - this customer has 150k messages on the old server and I had hobbled my server by downloading them all.

    you can use max_messages_per_session to do this.

    but question occurs how often does ispconfig3 fire getmail up? if it fires it up too frequently the throttling wont help!

    One other possiblity would be to have getmail deliver directly to the Maildir mailbox (what imapsync is doing after all!).

    this is possible per their docs:


    The Maildir destination delivers to a qmail-style maildir. The maildir must already exist, and must contain all of the subdirectories required by the maildir format. getmail will not create the maildir if it does not exist. If you're not familiar with the maildir format, the requirements in a nutshell are: it must be a directory containing three writable subdirectories cur, new, and tmp, and they must all reside on the same filesystem.

    The Maildir destination takes one required parameter:

    path (string) — the path to the maildir, ending in slash (/). This value will be expanded for leading ~ or ~USER and environment variables in the form $VARNAME or ${VARNAME}. You might want to deliver messages to a maildir named Maildir in your home directory; you could do this with a configuration like this:

    type = Maildir
    path = ~/Maildir/

    in /usr/local/ispconfig/server/conf ispconfig's getmail doc contains:

    type = MDA_external
    path = /usr/sbin/sendmail
    arguments = ("-i", "-bm", "{DESTINATION}")
    unixfrom = true

    so - I could try putting Maildir but then I have to have the general homedir maildir address.
    but if course we cant use the ~ because we are not actually using user home directories. ispconfig puts the mail


    hmm how do we tell getmail to translate 'username' into the Maildir path?
    Last edited: Mar 31, 2014
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    If you want to set this custom setup just for one account, then edit the file in /etc/getmail of that account directly, instead of using the default template. to protect the file from being overwritten by ispconfig, make it immutable with "chattr +i /etc/getmail/youraccountfile".
    Last edited: Mar 31, 2014
  5. craig baker

    craig baker Member HowtoForge Supporter


    in the /etc/getmail dir ? which is now empty? whats the syntax of the fileename so it knows its the right user account? got an example anywhere?
    thanks :)
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    Just created one on my testserver for mailserver "mail.mydomain.tld" and user "myuser":

    ls -la /etc/getmail/
    total 20
    drwx------   2 getmail root  4096 Apr  5 00:14 .
    drwxr-xr-x 109 root    root 12288 Apr  4 23:31 ..
    -r--------   1 getmail root   328 Apr  5 00:14 mail_domain_tld_myuser.conf
    the file has this content:

    # message_log = /var/log/getmail.log
    message_log_syslog = true
    delete = true
    read_all = false
    type = SimpleIMAPRetriever
    server = mail.domain.tld
    username = myuser
    password = mypassword
    type = MDA_external
    path = /usr/sbin/sendmail
    arguments = ("-i", "-bm", "[email protected]")
    But basically it would be even better if you use a different custom filename when you create such a config file manually so it wont collide with ispconfig, you can use any name for the file, it just has to end with .conf and must be owned by the user "getmail". Getmail will pick up the new config automatically.
  7. craig baker

    craig baker Member HowtoForge Supporter

    yep looks good one last question -

    how often does getmail poll for new mail?
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    Every 5 minutes, see getmail cronjob:

    crontab -u getmail -l
  9. craig baker

    craig baker Member HowtoForge Supporter

    excellent - now ONE more question LOL

    ok but ran into BIG issue - when I use getmail as normal (not delivering directly to Maildirs) - because I want to use spamassassin for example - the MTA system stamps all my 4 year old imap mails - with the current date!

    this is because we are sending it to sendmail as it if was infact newly received mail! and sendmail stamps the date accordingly (and wrongly)

    any setting to have it use an existing received-upon date instead?

    now Imap delivers the mail right to the Maildirs - and uses the correct (received) date in each email so dates all appear properly

    Last edited: Apr 5, 2014
  10. till

    till Super Moderator Staff Member ISPConfig Developer

    The getmail function in ispconfig is made to fetch emails every 5 minutes from a external account and send them trough the spamfildetr, so it has to send the emails trough sendmail. The functions is not made to copy over large old mailboxes, thats why it does not suits your special case that well.

Share This Page