HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials - Why did Squirrelmail, Dovecot/IMAP & Outbound mail break?

HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials (http://www.howtoforge.com/forums/index.php)

- Technical (http://www.howtoforge.com/forums/forumdisplay.php?f=8)

- - Why did Squirrelmail, Dovecot/IMAP & Outbound mail break? (http://www.howtoforge.com/forums/showthread.php?t=40474)

Why did Squirrelmail, Dovecot/IMAP & Outbound mail break?

Help! I need some seasoned advice please.

We're running the OldStable version of Debian Etch from August 2008. We've been using squirrelmail connecting through Dovecot's Imap and Pop3 servers since then to provide either SSH or TLS/SSL connections to postfix mail via squirrelmail on our server. Although the SSL capability is installed, we're really not using it -- choosing to use SSH with strong passwords instead.

This configuration has given us NO problems since we started... until today. For unexplained reasons this morning the IMAP interface suddenly began refusing or failing connections to everyone trying to connect through SquirrelMail (or that's the way it looks from the outside). It also fails to send out any emails. I've tried rebooting the server but that made almost no difference.

The problem was first reported by a user. I then verified it. What I was seeing BEFORE the reboot when I tried to login was an error from SquirrelMail that said:

Error connecting to IMAP server: myserver.com.
11074 :

but what I'm seeing since the server reboot is:

Error connecting to IMAP server: myserver.com.
11087 :

The IMAP server connect problem seems to be isolated to SquirrelMail. At least I ran 2 tests and found I CAN connect to the IMAP server using both Microsoft Outlook 2003 and Outlook Express and can see the contents of all folders on the server. So the IMAP problem only shows up in SquirrelMail. But it DOES prevent ANY users from loggin in through SquirrelMail

However, the inability to send mail OUT from the server shows up everywhere. Mail sent internally between accounts on the server -- either within a single domain or between domains -- and even from remote users connected to the server through Outlook or Outlook express gets delivered fine. But email addressed to anyone outside the server to any domain -- whether Yahoo or or MSN or Google or whereever is all bouncing back with a "relay request denied" error.

For instance, I sent an email from one of my server accounts to my yahoo inbox and it bounced back.

Other things I've tried are:

checked port status with

nmap -qT

both port 443 and 993 are reported as open -- one with imap and the other with imaps.

I restarted inetd. It had no effect.

I restarted postfix. It had no effect.

I restarted the whole server. The IMAP login error code when attempting to login via SquirrelMail changed 11074 to 11087. That's all. All other behaviors remain the same.

I also confirmed the SquirrelMail login failure problem occurs in IE 7, IE 8 and Firefox from 3 different machines in multiple geo-locations and networks AND both with and without the user's local firewall running. So the issue is definitely ON the server and seems to be isolated to squirrelmail even though no changes have been made to squirrelmail or any of its components in months.

When I checked the mail.err log, I found the following series of seemingly useless error messages:

Code:

Oct 25 08:40:33 axx012503 dovecot: POP3(sarah): unlink(/var/mail/sarah.lock) failed: Permission denied

Oct 25 09:31:48 axx012503 dovecot: POP3(mymailname): UIDs broken with partial sync in mbox file /var/mail/mymailname

Oct 25 10:02:28 axx012503 dovecot: POP3(mymailname): UIDs broken with partial sync in mbox file /var/mail/mymailname

Oct 26 01:01:06 axx012503 postfix/sendmail[30523]: fatal: root(0): queue file write error

Oct 26 15:06:53 axx012503 dovecot: POP3(sarah): unlink(/var/mail/sarah.lock) failed: Permission denied

Oct 27 01:01:04 axx012503 postfix/sendmail[6715]: fatal: root(0): queue file write error

Oct 27 08:56:15 axx012503 dovecot: POP3(bigdork): unlink(/var/mail/bigdork.lock) failed: Permission denied

Oct 27 08:56:16 axx012503 dovecot: POP3(bigdork): file_dotlock_delete() failed with mbox file /var/mail/bigdork: No such file or directory

Oct 27 14:57:56 axx012503 dovecot: POP3(mymailname): UIDs broken with partial sync in mbox file /var/mail/mymailname

That's about it. I've been racking my brains on this all day and I'm no closer to a resolution now than I was when the problem was reported 12 hours ago. At this point, any hints, helpful suggestions or questions would be appreciated! :rolleyes:

Thanks!

Are you using Maildir or mbox? What's the output of

Code:

ls -la /var/mail/

?
Are there any errors in your mail log?
Can you check if your server has been blacklisted? http://mxtoolbox.com/blacklists.aspx

Quote:

Originally Posted by falko (Post 208806)

Are you using Maildir or mbox? What's the output of

ls -la /var/mail/ ?

Are there any errors in your mail log?

Can you check if your server has been blacklisted? http://mxtoolbox.com/blacklists.aspx

First, thanks for the reply, falko. I appreciate your efforts to help.

Second, here's the result of that ls -la /var/mail/ command

Code:

drwxrwsr-x   2 root     mail         4096 Oct 28 08:31 .

drwxr-xr-x  15 root     root         4096 Oct 13 06:59 ..

-rw-------   1 mail     mail          104 Jun 16 07:00 .bash_history

-rw-------   1 mail     mail           35 Jun 16 06:59 .lesshst

-rw-rw----   1 alyianna mail          538 Jun 19 16:30 alyianna

-rw-rw----   1 board    mail        24928 Oct 13 15:14 board

-rw-rw----   1 doorprod mail            0 Jun 16 07:16 doorproductions

-rw-rw----   1 daemon   mail         9434 Dec 15  2008 daemon

-rw-rw----   1 dave     mail      3425224 Feb 25  2009 dave

-rw-rw----   1 devion   mail        26275 Oct 27 03:30 devion

-rw-rw----   1 eagle    mail      6044885 Oct 27 18:59 eagle

-rw-rw----   1 sarah    mail       418274 Oct 28 08:30 sarah

-rw-rw----   1 sarahtv  mail          538 Nov  7  2008 sarahtv

-rw-rw----   1 eric     mail       205715 Mar  4  2009 eric

-rw-rw----   1 martinpr mail          538 Sep 10  2008 martinpress

-rw-rw----   1 events   mail     121615204 Oct 27 18:29 events

-rw-rw----   1 winterex mail         6140 Oct 28 08:31 winterexchange

-rw-rw----   1 mike     mail          538 Sep  9  2008 mike

-rw-rw----   1 mikelinh mail          538 Jun 15 09:05 mikelinhart

-rw-rw----   1 info     mail          869 Oct  1 17:43 info

-rw-rw----   1 jody0309 mail        67007 Apr 15  2009 jody0309

-rw-rw----   1 legna725 mail        86253 Oct 27 18:59 legna7259

-rw-rw----   1 lovingfn mail        18110 Oct 17 18:52 lovingfn

-rw-rw----   1 mail     mail     44542363 Oct 28 08:22 mail

-rw-rw----   1 masterr  mail          538 Oct 28 07:08 masterr

-rw-rw----   1 pianoman mail        20110 Oct  7 10:07 pianomancollette

-rw-rw----   1 pianoman mail      1015593 Oct 28 08:30 pianomanshop

-rw-rw----   1 mistyblu mail        22925 Mar  4  2009 mistyblue

-rw-rw----   1 mornings mail            0 Jun 16 07:10 morningstarentjobs

-rw-rw----   1 nobody   mail     63284373 Oct 28 07:17 nobody

-rw-rw----   1 noohpyt0 mail        38444 Oct 17 18:52 noohpyt0348

-rw-rw----   1 noom2567 mail        40084 Sep 23 21:43 noom2567

-rw-rw----   1 oatdrol3 mail        81547 Oct 27 18:59 oatdrol3665

-rw-rw----   1 ojoc4957 mail         8674 Mar  4  2009 ojoc4957

-rw-rw----   1 penmark  mail          538 Oct 28 08:31 penmark

-rw-rw----   1 blackcon mail          538 Sep 22  2008 blackconsultants

-rw-rw----   1 blacklab mail          538 Sep 22  2008 blacklabs

-rw-rw----   1 rednib00 mail        10481 Apr 26  2009 rednib0006

-rw-rw----   1 markgold mail          538 Oct 19 07:13 markgoldstein

-rw-rw----   1 sandy    mail       265579 Oct 27 20:10 sandy

-rw-rw----   1 savether mail          538 Sep 15  2008 savethehorse

-rw-rw----   1 shannon0 mail       111760 Apr  5  2009 shannon0309

-rw-rw----   1 candylan mail            0 Jun 16 07:12 candyland

-rw-rw----   1 tgblack  mail          538 Oct 28 08:30 tgblack

-rw-rw----   1 theinfor mail         3486 Oct 28 08:30 ourinformalchateau

-rw-rw----   1 theweste mail         3902 Sep  9  2008 theeasternmotel

-rw-rw----   1 uucp     mail        21696 May 16 12:49 uucp

-rw-rw----   1 webcandy mail         3674 Oct 28 08:31 webcandy

-rw-rw----   1 webwoods mail         3480 Oct 28 08:31 webwoodscraft

-rw-rw----   1 whipping mail          538 Oct 28 06:49 whippingboy

-rw-rw----   1 wwphost  mail       849129 Oct 28 07:17 wwphost

-rw-rw----   1 www-data mail     238754025 Oct 28 08:06 www-data

-rw-rw----   1 yesac387 mail        19819 Mar 17  2009 yesac3876

I also checked the mxtoolbox site. It shows the server's primary IP address has been blacklisted by the Barracuda BRBL spam engine. The secondary IP address is not listed there. Barracuda is the ONLY blacklist that identifies any problems with the server and the confusing part is that when I checked the reputations of every one of the individual domains hosted on the server, not ONE Of them shows any issue at BRBL! ??!!?

How's that again? How is it possible for the server's IP address to be listed as a spam source if all domains on the server are innocent? As is common these days, all domains on the server do share the same IP address.

As you saw, I checked the mail error log and did find some issues there - although I couldn't make heads nor tails out of what they were telling me. I have not checked the mail log yet. I'll do that and see what I find. I admit though that I'm not sure exactly what to look for there.

For the record, I watch emails and email bounces on this server pretty close and I DO see some email bounces I can't explain which purport to be bouncing emails from sites I KNOW aren't sending those emails. I control all domains on the server. I KNOW what emails are sent out by those domains and the email loads aren't heavy, Frankly, the number of weird bounces I've seen hasn't been large enough to be a big concern to me or produce a full scale investigation into how those emails are happening to begin with. I'm very aggressive about trying to keep spammers and click-phishers out of the two sets of forums hosted on our server... manually reviewing and approving all join requests, and running queries every day that are designed to identify and remove the dozens of bots that try to register in those forums daily.

In short, I'm using every standard technique I know of to prevent server hacking -- ssh-secured logins, hard-to-guess usernames, strong and hard to guess 12 - 25 character passwords, limited telnet/putty access, limited email accounts, etc. But I'll admit I'm NOT using IPTables. I couldn't see the benefit to that. What the heck can an outsider do with a port that's not being used by the server? Or conversely, if the port is being used for outgoing smtp mail, I don't see why it's a security concern? I don't mean to seem stupid or ignorant here. But I don't get it. What am I missing?

Still, I want to STOP the blasted spam as much as anyone else does. So, I'll gladly listen to suggestions on how to further tighten security on my server... and how to chase down, isolate and kill the source of those unexplained outbound emails. I'm NOT averse to fighting the spam wars. I'm just not sure how to isolate and kill the mysterious sources of "how-the-heck-did-that-happen?" spam that seems to occur on many servers despite the best efforts of the admins to stop it. Like most server admins, I do have my limits. I can't spend my whole life to fighting spam either.

Thanks!

One more piece to the puzzle...

I just realized there is another possibly-relevant piece to this puzzle.

There is a single domain on this server for which I host a mailman mailing list. When I first moved that site and its list to my server a couple of years ago it was receiving a huge load of junk mail every hour of evey day. Worse yet the inbound email address most of the spam was being sent to was the same one the mailman list had long been hooked to.

So, when I installed mailman, I configured it to only accept mail from known list members and configured things so the server would bounce all mail that wasn't accepted by mailman back to the sender. Later I realized that this so-called junk mail "back-scatter" was an issue that caused some Black lists to rate the server poorly. So, I changed that bounce solution to accept all such mail and deliver it to >null instead.

As far as I know, it's still operating that way on that mailman domain. That approach shouldn't cause a black-list problem, should it?

Thanks.

Quote:

Originally Posted by websissy (Post 208821)

How's that again? How is it possible for the server's IP address to be listed as a spam source if all domains on the server are innocent? As is common these days, all domains on the server do share the same IP address.

Some blacklists list whole subnets if one or more IPs from that subnet send out spam. But usually blacklists write something about their policy on their web sites, so maybe you can find out if this is the case.