Hotmail and my SPF Nightmare

29Mar07

Do you know what a SPF record is?

No?

Neither did I until Microsoft decided to class me as a spammer, and if you read on you might just save yourself from loosing several days of your life trying to implement one.

Me Sir, a ’spammer’?

Anyway, to understand what I’m rambling on about we need a bit of background, and why this ‘SPF Record’ is getting me so wound up.

I, unlike the majority of people with common sense, use Hotmail as my primary email provider, and have done since I first starting using the Internet. In fact I had my Hotmail address before it became part of the Microsoft empire. One thing that annoys me however, is that I am now having to put up with more and more spam, despite efforts to curtail it.

We all know the stress of sorting through spam, and thank the people who work on solutions to filter out or just stop that crap coming through. However, I am sure you will understand my annoyance when I found out that thanks to the configuration of my (dv) dedicated-virtual server I have in fact been branded a ’spammer’ by Microsoft, and as a result they appear to be black holing any mail sent to a Hotmail account from my (dv).

Before I go any further I would just like to clarify that this is in fact nothing to do with the (dv) server as a product or Media Temple, but rather the way in which a virtual server environment works. I have found dozens of references via Google of people complaining of the same problems, and interestingly most seem to refer to people running VPS environments using Plesk.

As with all things, when something goes wrong, you have to learn how it works to be able to fix it, and thus I have been learning some of the ins and outs of running mail servers and the DNS system.

Disclaimer: At this point I would just like to say I only have a (very) basic idea about how either work, so don’t take anything I say as gospel, but rather use it as a loose guide and reference to where you may find further help.

Where is my mail going!?

After getting in touch with the guys at (mt) I decided that I needed to find where my bloody mail was going. I wasn’t getting a bounceback mail from the Hotmail server, and Thunderbird told me that the mail was delivered. Thankfully due to the fact that the (dv) allows you to delve into the OS to see what’s going off, I thought I would interview the SMTP log and see what was going off. The SMTP server in Plesk’s case is called “Qmail’ and the logs are located at # /usr/local/psa/var/log/maillog and can be read in a number of ways. In this case I found the easiest way to track what was going off was to use the tail -f command which spurts out the log information for events as they are happening, and this is what I got when I tried to send an email to my Hotmail account:


Mar 22 17:32:23 as qmail: 1174584743.517414 delivery 437: success: 65.54.244.168_accepted_message./Remote_host_said:_250_ <4602beef.1080905@helloian.com>_Queued_mail_for_delivery/

So it would seem that the Hotmail server is accepting the mail, queuing it, but never actually delivering it, due to their spam filtering technology. A quick search on Google showed that plenty people seemed to have experienced the same problem. Interestingly most were using Plesk, and virtually all of them were using Qmail as their SMTP server. Clicking the seemingly never ending list of results, I realised that not one had any comments regarding a working solution, but the acronym SPF kept popping up a lot, so I decided it was worth a look.

The Sender Policy Framework

The Sender Policy Framework allows a domain owner to specify which machines are allowed to send email on its behalf. This kind of mechanism is unfortunately not present in the Simple Mail Transfer Protocol, a fact that allows spammers to send e-mail from forged addresses relatively easily, as there is no inbuilt validation when an email is sent and then received.

Fortunately the remedy is relatively straight forward to implement. The SPF record is applied as a TXT type entry in the domain’s DNS record, and it’s as simple as that. Now, when you send an email, the receiving mail server can use this SPF record to verify that the origin of the email is legitimate. To help illustrate what is happening, below is a MIME header from an email I sent between two accounts on my (dv).

Return-Path: 
Delivered-To: 3-sayhello@helloian.com
Received: (qmail 32062 invoked from network);
29 Mar 2007 17:59:58 +0100
Received: from 85-211-13-70.dyn.gotadsl.co.uk 

(HELO ?192.168.1.5?) (85.211.13.70)
  by distillate-hosting.net with (DHE-RSA-AES256-SHA encrypted)
  SMTP; 29 Mar 2007 17:59:58 +0100
Message-ID: <460bf1ee.4020508@distillate.co.uk>
Date: Thu, 29 Mar 2007 18:05:50 +0100
From: Ian Halliday 
User-Agent: Thunderbird 1.5.0.10 (Windows/20070221)
MIME-Version: 1.0

The confusion arises when the receiving machine reads the email is claiming to be from the domain ‘distillate.co.uk’ but has been sent via the server ‘distillate-hosting.net’. As far as the machine is concerned, there is no link between the claimed sender and the machine it originated from. There is no way to tell if this information is legitimate or not.

The reason that my initial searches on Google seemed to show that it was mostly VPS users with multiple domains that were suffering from this problem is that by its very nature, a VPS server running by multiple domains will send mail from the mail server of any given domain (in my case distillate.co.uk) through the SMTP server of the host VPS platform (distillate-hosting.net in my case). Unfortunately emails sent using this setup look very similar to ’spam’ messages, and the Hotmail spam filter (known as ‘SmartScreen’) is quick to step in and black hole the email, meaning it never reaches its destination, despite the Hotmail server notifying the sender that the email has been received and delivered.

Fortunately, this is where the SPF record steps in to clear matters up. The SPF record tells the receiving machine that the server ‘distillate-hosting.net’ sends mail on behalf of the mail exchanger for the domain ‘distillate.co.uk’ and this is written as:

v=spf1 mx ip4:XXX.XXX.XXX.XXX mx:mail.YYYYYY.YYY ?all

Where:

  • v=spf1 Denotes the following as a SPF record.
  • mx States that the Mail Exchanger sends outbound mail for server as stated in the next segment<. i="">
  • ip4:XXX.XXX.XXX.XXX Is the IPv4 formatted IP address of the (dv) server.
  • mx:mail.YYYYYY.YYY States that the Mail Exchanger of the domain specified (YYYYYY.YYY) sends mail through the IP previously specified.
  • ?all States that any IP’s that fail to meet any of the listed ‘mechanisms’ will return “neutral”, thus will be treated as if a record does not exist.

To clarify, the SPF record for my domain distillate.co.uk is entered in the DNS zone file as:

v=spf1 mx ip4:216.70.127.122 mx:mail.distillate-hosting.net ?all

The Open SPF website explains the above is more detail, and offers a tool to help you set up your SPF record. Microsoft also have a similar tool available which after being referred to by Hotmail technical support, turned out to be more of a hindrance than a help. The Microsoft tool, and many other references recommend that a PTR mechanism is included in the SPF record. The PTR record allows reverse lookup of an IP address; that is identify the domain of an IP address. The reverse lookup is used to verify that the domain name and IP address in the email MIME header actually correlate and have not been faked. Whilst this sounds like a good idea, actually processing a reverse look up takes a considerable amount of time and it is not generally a method employed by large email providers like Hotmail. In fact Hotmail refused my initial SPF record as it included this PTR mechanism. To quote Hotmail technical support:

The specification for SPF records (RFC 4408) discourages use of “ptr” for performance and reliability reasons. This is especially important for Windows Live Mail, Hotmail and other large ISPs as a result of the very high volume of mail we receive each day. We highly recommend you remove the “ptr” mechanism from your SPF record and, if necessary, replace it with other SPF mechanisms that do not require a reverse DNS lookup, such as “a”, “mx”, “ip4″ and “include.”

Troubleshooting

The very nature of the DNS system made this problem a very frustrating one to tackle, as you don’t see instant results from your implementation, but of course have to wait anywhere up to 48 hours for the information to propagate throughout the internet. You can however use some of the tools on the Open SPF website to check your record is configured properly. Once you have confirmed that your record is set up correctly you can also send a blank email to check-auth@verifier.port25.com which will test your SPF record, and email you back the results.

I also found dnstuff.com invaluable in testing my DNS set-up. Whilst it doesn’t check the functionality of your SPF record (it only checks that you have one), then DNS Report tool on dnsstuff.com gives you feedback on all aspects of your DNS configuration and can be an excellent tool for troubleshooting.

SPF Works!

Finally I can email Hotmail users without worrying if it will go through, and if you are running a (dv) or similar setup then I strongly suggest you use a SPF record, even if you are having no problems at the moment. One way of making life even easier for yourself in the future if you use Plesk would be to use your Plesk server as the nameserver for all domains residing on it, and set up a SPF record in the main server DNS page, accessible from the main server configuration page. By doing this all new domains will automatically have the correct SPF record setup for them. If you are only running a few domains, just make the changes in (mt)’s account center and continue to use the (mt) nameservers.

If the above doesn’t work for you, get in touch with your hosting provider and make sure you have run all the tests I mentioned. Unfortunately in the end there is no substitute for really understanding what is going wrong, so I suggest you familiarise yourself with how the DNS system works. Wikipedia has an excellent article and Media Temple’s Knowledgebase has a more concise article available, either of which should put you on the right track.

Update:

Well I may have spoken slightly too soon regarding everything being fine. It turns out that my emails are still not guaranteed to go straight through to any given Hotmail inbox, but rather the Hotmail spam filter will take a while to learn that my domains are trustworthy and that the SPF records check out. At the moment some emails go through okay, some go to the Junk folder.

I have been informed by Microsoft that over time (approximately a month) more of my emails should go straight through to the inbox. If anyone gets an email that lands in their junk mail (by subscribing to comment updates for example) you would be doing us both a great favour by checking ‘this is not junk’, which will ensure all mail from my server reaches your inbox in future, and that I will look better in the eyes of the Hotmail spam filter.


12 Responses to “Hotmail and my SPF Nightmare”


  1. Great post - I’ve just started setting up a VPS through WiredTree, and I’m giving myself a crash course in DNS and name servers too. Me no likey.

    Anyway, I’ve been experimenting with using Google Mail for all my domains through Google Apps for Your Domain. I’m thinking this might circumvent the problem entirely since everything is run through Google’s mail servers. My nameservers will be hit first for the MX records, so I’m not entirely sure about that, though. Thoughts?

  2. Adam, I can appreciate your pain having to look into the DNS system. Fortunately it’s actually relatively simple once you get a feel for the order of things.

    An interesting approach using Google Apps for your domain, which I am going to look into, although it’s far from ideal. Clients (and myself) want to be able to use Mail/Thunderbird/Outlook for their email.

    I’ll take a look and get back to you on that.

  3. Ian, you can actually set a preference in gmail to enable POP, and then have the best of both worlds - gmail’s interface for webmail, and thunderbird (or outlook or mail) for when you’re in the office.

    I’ll admit, it seems almost too good to be true, especially since it’s free, but I haven’t found the catch yet. Although, I guess one catch could be that you’re at the mercy of whatever changes google decides to make in the service.

  4. Good Read -

    I feel your pain - I too am having the “We highly recommend you remove the ???????ptr??????? mechanism from your SPF record and, if necessary, replace it with other SPF mechanisms that do not require a reverse DNS lookup, such as ???????a???????, ???????mx???????, ???????ip4??????? and ???????include.???????” blues, and I am not using your setup - running Santronic’s WINServer on Windows XP Pro.

    Woke up one morning two weeks ago, mail from my site is evidently falling out of the end a wire somewhere before it reaches any hotmail account.

    I found the MS reply to be totally unacceptable and incomplete - they do not state they will reject mail if the ptr setting is found in the spf record, in fact they do not state any solutions or reason whatsoever for mail that, according to all log file data at my site, indicates hotmail has accepted/queued for delivery. And, as in your case and all others I have read about, the hotmail servers to not bounce the msgs, nor evidently do they return proper true or accurate smtp repsonse codes.

    Only the ‘250 queued…’ is returned to the ssending server, which to me would seem that they are intercepting and witholding proper, comforming communications which ARE desired to be received by users of hotmail.

    I in fact, am one of those users! I can not even get mail to the external hotmail account I maintain for standby and outside exchanges from my own bonafide non-spamming site.

    I have complained both as a hotmail user and and as system administrator.

    I could very easily just remove the ptr setting from the spf record, but it was there for a purpose, one of which was to insure that mail sent from our system was verifiable to as many other agents as possible WITHOUT the hassles of falsing and other non-delivery maladies, while simultaneously providing assurance of the proper authority of which systems, clients, and or IPs were allowed to use our mailing system.

    I am not exactly the sharpest knife in the drawer, but I have some basic knowledge and experience with the internet, and this latest fiasco with hotmail does not seem to be exactly proper for such a large high-profile organization.

    Que Sera - you’ll notice I did not use any of my hotmail accounts for this record, I likely wil post a notice on my sites informing visitors and members that we can not maintain reliable communciations with Windows Live or hotmail accounts - this hurts, as we have many clients who use hotmail accounts for messaging with our sites/services.

    My 2 cents worth, glad you at least got some satisfaction.


    Dan

  5. Thank you so much for posting this! I had such a large problem with this SPF deal for the longest time. After reading through this and having you point me to that website I got it all working. Thanks man! Your a life saver.

  6. There is a rumour that Hotmail will still misbehave unless you have submitted your SPF record to Windows Live Mail/MSN Hotmail by sending an email listing your domain names (one per line) to senderid@microsoft.com

    You will also find that Yahoo misbehaves unless you have a DomainKey properly defined :(

  7. Thank you so much for this article. I have been “battling” with my VPS/Plesk at 1and1 with hotmail for over a week now. I thought the SPF was a solution, but I haven’t been able to make it work.

    When you sign up to a 1and1 VPS, they automatically add a 1and1 domain (lets call it 1and1domain.com). My domain is also added to the VPS (lets name it MyDomain.com… how original). So the VPS has two domains on one IP (…. XXX.XXX.XXX.XXX).

    When I send email via mail() under MyDomain.com, this is what I get in the raw source:
    Received: from 1and1domain.com (MyDomain.com [XXX.XXX.XXX.XXX])

    So far, I have the same issue. But, the email server for the 1and1 domain is not mail.1and1domain.com, but mx01.1and1.com .

    Therefore, could you tell me if this SPF file seems ok:
    v=spf1 a mx a:1and1domain.com mx:mx01.1and1.com ip4:XXX.XXX.XXX.XXX ?all

    On a slightly different note… I cannot add a DNS TXT Record to the 1and1 DNS control panel! They’re telling me that I need to get another nameserver to add the TXT (SPF)! If I simply point to ns.MyDomain.com and have the following in Plesk, will this be enough?

    MyDomain.com. NS ns.MyDomain.com.
    mail.MyDomain.com. A XXX.XXX.XXX.XXX
    ns.MyDomain.com. A XXX.XXX.XXX.XXX
    MyDomain.com. A XXX.XXX.XXX.XXX
    webmail.MyDomain.com. A XXX.XXX.XXX.XXX
    ftp.MyDomain.com. CNAME MyDomain.com.
    www.MyDomain.com. CNAME MyDomain.com.
    MyDomain.com. MX (10) mail.MyDomain.com.
    XXX.XXX.XXX.XXX / 24 PTR MyDomain.com.
    MyDomain.com. TXT v=spf1 a mx a:1and1domain.com mx:mx01.1and1.com ip4:XXX.XXX.XXX.XXX ?all

    Or do I need to sign up to an external Nameserver (DNS) service?
    Thanks again.

  8. Sign the petition calling Microsoft to remove its SmartScreen technology from hotmail.
    http://www.petitiononline.com/notsmart/

  9. hello there,

    i do have also a vps with mt. all my emails are going to the spam box of hotmail account despite my spf. i was wondering if it’s not related to the helo name in greeting. the helo name uses the server name, in my case: as.3uropa.com. i don’t use ptr so my spf looks like yours, pointing the mx to the different domain names hosted on as.3uropa.com

    hotmail policy is a nightmare. if you have any clue on what’s going on with me. the irony of that is when using dnstuff report on any of my vps domain names i have better results than one of my domains hosted on mt grid server (which has at least 3 warnings) when any email from that domain name isn’t put in hotmail spam box.

    i was thinking about committing suicide with an ethernet cable but i go wireless

  10. Thanks for this. Probably the best walk through of SPF’s I’ve read. And I’ve read a ton. Having a similar problem with versions of Outlook. I’m creating a send-to-friend PHP script using authentication and SMTP, have the SPF set up and the code checked, and I still can’t get past Outlook’s client spam filter. Mime header looks much like what you’ve included above. I can get email through on Entourage, mail.app, gmail, yahoo, and even past Spam Assassin on Exchange.

    Anyone have any experience or advice on working with Outlook?

  1. 1 » Hotmail and my SPF Nightmare Pingback on Apr 20th, 2007 at 4:47 am
  2. 2 My daily readings 05/13/2007 « Strange Kite Pingback on May 13th, 2007 at 11:35 am

Leave a Reply

Please ensure comments are written in good English and are properly structured. This is for the benefit for all visitors, to make information easily accessible. Please avoid the use of mobile phone style short hand and writing completely in lower case.

Following these guidlines will help everyone using the site and also speed up the moderation process.

Thank you.