Dovecot with FTS Lucene

Discussion in 'Tips/Tricks/Mods' started by Loveless, Jul 6, 2017.

  1. Loveless

    Loveless Member

    If you're running dovecot for IMAP, I can highly recommend installing dovecot lucene instead of solr or just dovecot's plain Full Text Search. CLucene yields extremely fast IMAP search results on huge mailfolders, using webmail packages like Squirrelmail or Rainloop.
    Here's how I did it on a couple of Ubuntu dovecot servers:

    I installed Peter van der Does' dovecot repository: https://launchpad.net/~pdoes/ archive/ubuntu/dovecot
    Then, as root, I simply did
    ~# apt-get update && apt-get install dovecot-lucene -y​
    which upgrades to a more recent dovecot-core and installs a very recent C++ CLucene library and the fts_lucene plugin module. After that you may want to apt-mark hold dovecot to not autoupdate, or even remove the pdoes repo, depending on your wishes.

    I then changed my /etc/dovecot/dovecot.conf to activate and make proper use of lucene;
    mail_plugins = $mail_plugins fts fts_lucene
    plugin {
    fts = lucene
    fts_lucene = [email protected] normalize no_snowball
    fts_autoindex=yes
    fts_autoindex_max_recent_msgs=80
    fts_index_timeout=90
    }​

    And next I'll post my full non-default dovecot configs. Note some performance improvements for dovecot that I added, and I removed the quota plugin (I don't bother my users with that, I think it's a retarded subject for mail, and you should auto purge trash and junk folders putting this under /etc/cron.monthly :

    #!/bin/sh
    doveadm expunge -A mailbox Junk savedbefore 60d
    doveadm expunge -A mailbox Trash savedbefore 660d
    doveadm force-resync -A '*'
    exit 0

    Code:
    :~# doveconf -n
    # 2.2.31 (65cde28): /etc/dovecot/dovecot.conf
    # Pigeonhole version 0.4.19 (e5c7051)
    # OS: Linux 4.4.0-78-generic x86_64 Ubuntu 16.04.2 LTS
    auth_cache_size = 24 M
    auth_cache_ttl = 18 hours
    auth_mechanisms = plain login
    disable_plaintext_auth = no
    listen = *,[::]
    log_path = /var/log/dovecot.log
    log_timestamp = "%Y-%m-%d %H:%M:%S "
    login_greeting = Holy Moly
    mail_fsync = never
    mail_max_userip_connections = 24
    mail_plugins = " fts fts_lucene"
    mail_privileged_group = vmail
    mail_temp_dir = /ramm/dovecot
    mailbox_idle_check_interval = 59 secs
    mailbox_list_index = yes
    maildir_very_dirty_syncs = yes
    managesieve_notify_capability = mailto
    managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext
    passdb {
      args = /etc/dovecot/dovecot-sql.conf
      driver = sql
    }
    plugin {
      fts = lucene
      fts_autoindex = yes
      fts_autoindex_max_recent_msgs = 80
      fts_index_timeout = 90
      fts_lucene = [email protected] normalize no_snowball
      sieve = /var/vmail/%d/%n/.sieve
      sieve_max_redirects = 25
    }
    protocols = imap pop3
    service auth {
      unix_listener /var/spool/postfix/private/auth {
      group = postfix
      mode = 0660
      user = postfix
      }
      unix_listener auth-userdb {
      group = vmail
      mode = 0600
      user = vmail
      }
      user = root
    }
    service imap-login {
      process_min_avail = 4
      service_count = 64
      vsz_limit = 128 M
    }
    service lmtp {
      unix_listener /var/spool/postfix/private/dovecot-lmtp {
      group = postfix
      mode = 0600
      user = postfix
      }
    }
    service pop3-login {
      service_count = 0
    }
    ssl_cert = </etc/postfix/smtpd.cert
    ssl_key =  # hidden
    ssl_protocols = !SSLv2 !SSLv3
    userdb {
      driver = prefetch
    }
    userdb {
      args = /etc/dovecot/dovecot-sql.conf
      driver = sql
    }
    protocol pop3 {
      pop3_uidl_format = %08Xu%08Xv
    }
    protocol lda {
      mail_fsync = optimized
      mail_plugins = sieve
      postmaster_address = # hidden
    }
    protocol lmtp {
      mail_fsync = optimized
      mail_plugins = sieve
      postmaster_address = # hidden
    }
    
    I use multitail in another terminal to witness if all the changes go as planned:
    ~# multitail /var/log/dovecot.log /var/log/syslog /var/log/mail.log​

    Make sure ISPconfig's database has the right entries in them (you'll see errors when Searching IMAP in your dovecot log if you don't do this). Database `dbispconfig` needs to have two new columns:

    alter table `mail_user` add `disableindexer-worker` enum('n','y') NOT NULL DEFAULT 'n';
    alter table `mail_user` add `disablequota-status` enum('y','n') NOT NULL DEFAULT 'y';​

    If you disable quota status, like I do, you can also remove the CONCAT storage queries from /etc/dovecot/dovecot-sql.conf, so they change into this:

    # password-query with prefetch
    password_query = SELECT email as user, password, maildir as userdb_home, CONCAT( maildir_format, ':', maildir, '/', IF(maildir_format='maildir','Maildir',maildir_format)) as userdb_mail, uid as userdb_uid, gid as userdb_gid, CONCAT(maildir, '/.sieve') as userdb_sieve FROM mail_user WHERE (login = '%u' OR email = '%u') AND `disable%Ls` = 'n' AND server_id = '1'

    user_query = SELECT email as user, maildir as home, CONCAT( maildir_format, ':', maildir, '/', IF(maildir_format='maildir','Maildir',maildir_format)) as mail, uid, gid, CONCAT(maildir, '/.sieve') as sieve FROM mail_user WHERE (login = '%u' OR email = '%u') AND `disable%Ls` = 'n' AND server_id = '1'
    Restart dovecot, now with Lucene
    ~# service dovecot restart​
    check if it runs OK:
    ~# service dovecot status​

    Then, last but most importantly, force dovecot to reindex all mailfolders, as root do;

    # doveadm -v index -u '*' '*'​
    (the -v is verbose, so you see it actually notices the changed indexing config)
    For assurance, also sync the caches:

    # doveadm force-resync -A '*'​

    That should do it. Now try your superfast search on some string on [ All folders ] in roundcube or whatever your IMAP server client is.
     
    Last edited: Jul 6, 2017
    cwispy, Jesse Norell and till like this.

Share This Page