Solving SMTP “554 Permanent scan failure” non-delivery errors

Recently some users started getting non-delivery reports (NDRs) from our Exchange server with messages like the following:

554 Permanent scan failure. Email Session ID: {4C209A7E-0-3A6A8C0-1FFFF}

Looking at the SMTP logs (C:Program FilesMicrosoftExchange ServerTransportRolesLogsProtocolLogSmtpSend) showed the offending sessions looking something like this:

* attempting to connect
+
< 220 mail.example.com ESMTP
> EHLO smtp.example.com
< 250-mail.example.com
< 250-8BITMIME
< 250-SIZE 10485760
< 250 STARTTLS
* 1600 sending message
> MAIL FROM:<Jo.User@example.com> SIZE=1061757
< 250 sender <Jo.User@example.com> ok
> RCPT TO:<A.N.Other@example.com>
< 250 recipient <A.N.Other@example.com> ok
> DATA
< 354 go ahead
< 554 Permanent scan failure. Email Session ID: {4C209928-2-3A6A8C0-1FFFF}
> QUIT
- Remote

Very odd as I couldn’t find any information about this “Permanent scan failure”.

Then looking at the SMTP message headers of messages that did get through (Outlook hides them away in the message properties as “Internet headers”) on the receiving side offered up a clue:

Received: from mail.example.com (4.3.2.1) by blah.example.com
(5.6.7.8) with Microsoft SMTP Server (TLS) id 8.1.340.0; Thu, 18 Mar
2010 10:17:49 -0400
Received: from smtp.example.com (HELO smtp.example.com) ([1.2.3.4]) by
mail.example.com with SMTP; 18 Mar 2010 10:19:34 -0400
X-CheckPoint: {4BA23672-7-3A6A8C0-7B6}
Message-ID: <CAC3AC395FD04CB1BB7DCEC764E7816E@example.com>
From: fred <fred@example.com>
To: <johndoe@example.com>
...

The CheckPoint firewall was injecting an ID into the headers which had exactly the same format as the “Session ID” from the ones that were failing with “Permanent scan failure”.

So unbeknownst to us CheckPoint was proxying/filtering the SMTP traffic and barfing on it for some reason. When we switched off the SMTP checks that CheckPoint was doing (they weren’t required as we have another mail gateways in place) the problem went away.

Backing up an Exchange Mailbox to a PST file

I’ve never trusted Exchange Server backup 100% ever since Exchange 2000, following a service pack, refused to restore backups from the non-service packed version (yes honestly).

So I’ve always had a 2-pronged approach to backup, do the usual monolithic backup using NTBackup, but also have mailboxes individually backed up as plain old PST files, which Outlook can easily mount to make it easier to do partial restores. In the past I’ve used ExMerge to do this, but it’s a becoming a bit neglected and is very clunky to script: it’s a Windows app (rather than a console app) that’s driven by an INI file.

Anyway, whilst doing some integration work against Exchange for a client I came across Dmitry Streblechenko’s superb Redemption Data Objects library. It is a really easy to use COM wrapper around Extended MAPI – a super-charged version of Collaboration Data Objects (CDO).

Honestly I really can’t understand why Microsoft didn’t ship a library like this (or improve CDO) rather than expecting you to write gnarly C++/COM/MAPI code to do what this library allows you to do easily from .NET code or a script. The Exchange API goalposts move from one release to the next: “Use the M: drive! No, use WebDAV! No, use ExOLEDB!… No, use Web Services!” with the only constant being good old MAPI.

Anyway – here’s part of our PST backup script – it relies on the Redemption Data Objects (RDO) COM DLL being registered. The free developer version pops up a prompt once when you RegSvr32 it, the royalty-free redistributable version is a incredibly reasonable $199.99. RDO relies on MAPI being installed, so grab it here if it’s not present on your system.

/* BackupPst.js */

// e.g. copyMailboxToPst(
           "EXCH01",
           "FredB",
           "c:\backups\fredb.pst",
           "FredB backup")

function copyMailboxToPst(serverName, userName, pstFile, pstName)
{
  var session = new ActiveXObject("Redemption.RDOSession");
  session.LogonExchangeMailbox(userName, serverName);
  WScript.Echo("Logged on to " + session.ExchangeMailboxServerName);

  var mailbox =  session.Stores.DefaultStore;
  var pstStore = session.Stores.AddPSTStore(pstFile, 1, pstName);

  try
  {
    WScript.Echo("Opened " + mailbox.IPMRootFolder.FolderPath);
  }
  catch(err)
  {
    WScript.Echo("Error opening mailbox '" + userName
       + "' (access denied?). " + err.description);
    return;
  }

  foreach(mailbox.IPMRootFolder.Folders, function(folder)
  {
    WScript.Echo(" * " + folder.Name);
    folder.CopyTo(pstStore.IPMRootFolder);
  });

  pstStore.Remove();
}

// Utility to allow enumeration of COM collections
function foreach(collection, fn)
{
  for(var e = new Enumerator(collection); !e.atEnd(); e.moveNext())
  {
    if(fn(e.item()))
      break;
  }
}

This could be further improved for incremental backups by using RDO’s newly introduced wrappers to the “Incremental Change Synchronization” API where you can use the same syncing technology that Outlook’s cached Exchange mode uses!

Exchange 2007 – Custom IP Block List Provider Error Message

We use Microsoft Exchange 2007 at InfoBasis for email, and for my sins I have to manage it. It seems a little enterprisey unwieldy at times for an organisation as small as ours but the Outlook calendaring integration is a must for us and I’ve mostly worked out how it works and how to keep it running happily (thanks to Brian Reid of C7 Solutions who helped us implement it).

We use the built-in anti-spam filtering – specifically the IP Block Listing, aka “DNS blacklist” (or DNSBL) to block emails from known spam sources. I won’t explain how to set it up here as there are plenty of other resources that cover this. Because occasionally we’d get a false positive (eg. from a legitimate AOL or Yahoo account) I was investigating how to write Transport/Edge providers to provide some sort of dynamic spam white list functionality which didn’t seem to be provided out of the box. In so doing I discovered that most of the Exchange 2007 code base seems to have been re-written in .NET (which may explain why some feel it was released unfinished/too early). So I pointed Lutz Roeder’s .NET Reflector (which reverse-engineers .NET DLLs) at a few of the Exchange Server DLLs in Program FilesMicrosoftExchange ServerBin and had a snoop around. Thankfully, I eventually stumbled on the Update-Safelist PowerShell cmdlet which does white listing based on Outlook’s Safe Senders – so I didn’t have to write any code after all.

Anyway, on my travels I came across ConnectionFilteringAgent.BlockListDnsCallback (in the Microsoft.Exchange.Transport.Agent.ConnectionFiltering namespace) in Microsoft.Exchange.Transport.Agent.Hygiene.dll. This method formats the message that gets returned to the senders mail server should they be listed by a DNSBL. This was of interest because in Exchange 2003 you could use format codes in the error message (e.g. “The IP address %0 was rejected by black list %2”) to help legitimate email senders work out why their email was blocked (as documented in KB 823866). But, I couldn’t find any documentation on how to use similar format codes in Exchange 2007 – but here in BlockListDnsCallback was this line of code:

message = string.Format((string) provider.RejectionResponse,
   this.queryData.hostIP,
   this.queryData.provider.Name,
   this.queryData.provider.LookupDomain);

Which means you can use the following sort of custom error message in Exchange Management Console > Organization Configuration > Hub Transport > Anti-spam > IP Block List Providers > Edit > Error Messages:

Your IP address {0} has been found on '{1}' (looked up on {2})

Hope that helps!