Using Google Apps for your domain's email? Well you definately need to set up some SPF records.
It seems yahoo is extremely strict when it comes to checking for SPF records on a domain, and bounces anything with missing records. Many other email providers like hotmail simply SPAM your message straight away.
Google provide the following resource for this problem: http://www.google.com/support/a/bin/answer.py?hl=en&answer=33786
But, they recommend using:
v=spf1 include:aspmx.googlemail.com ~all.
That's all fine and well...but what about the official website? Oddly, they recommend using - instead of ~ (http://www.openspf.org/FAQ/Common_mistakes):
v=spf1 include:aspmx.googlemail.com -all
But seeing as it's google hosting my mail, I've been using ~ successfully for some time now, with no more bounces.
Testing whether you're configured correctly
How can I configure my server?
If you have access to all your DNS records for your domain you can add it yourself (for example through WHM or root plesk panel) but on most shared hosts just fire off an email to the support team who will add the record for you.
On Plesk & Cpanel you can add either a SPF record or a TXT record through your DNS editor, making it easy to do this yourself.