I've been getting the same problem and also message 451 from yahoo, which I haven't solved yet either. Mail mail works fine when sending from a yahoo mail, but not to a yahoo email address. I found the following article which may explain what's happening.
Yahoo have listed a few things to check.
* Remove email addresses that bounce
* Examine your retry policies
* Pay attention to the responses from our SMTP servers
* Don't send unsolicited email
* Provide a method of unsubscribing
* Ensure your mail servers are not open relays.
Not sure if this is related, but if I manage to solve it I'll post back.