Solving SMTP “554 Permanent scan failure” non-delivery errors

Recently some users started getting non-delivery reports (NDRs) from our Exchange server with messages like the following:

554 Permanent scan failure. Email Session ID: {4C209A7E-0-3A6A8C0-1FFFF}

Looking at the SMTP logs (C:Program FilesMicrosoftExchange ServerTransportRolesLogsProtocolLogSmtpSend) showed the offending sessions looking something like this:

* attempting to connect
+
< 220 mail.example.com ESMTP
> EHLO smtp.example.com
< 250-mail.example.com
< 250-8BITMIME
< 250-SIZE 10485760
< 250 STARTTLS
* 1600 sending message
> MAIL FROM:<Jo.User@example.com> SIZE=1061757
< 250 sender <Jo.User@example.com> ok
> RCPT TO:<A.N.Other@example.com>
< 250 recipient <A.N.Other@example.com> ok
> DATA
< 354 go ahead
< 554 Permanent scan failure. Email Session ID: {4C209928-2-3A6A8C0-1FFFF}
> QUIT
- Remote

Very odd as I couldn’t find any information about this “Permanent scan failure”.

Then looking at the SMTP message headers of messages that did get through (Outlook hides them away in the message properties as “Internet headers”) on the receiving side offered up a clue:

Received: from mail.example.com (4.3.2.1) by blah.example.com
(5.6.7.8) with Microsoft SMTP Server (TLS) id 8.1.340.0; Thu, 18 Mar
2010 10:17:49 -0400
Received: from smtp.example.com (HELO smtp.example.com) ([1.2.3.4]) by
mail.example.com with SMTP; 18 Mar 2010 10:19:34 -0400
X-CheckPoint: {4BA23672-7-3A6A8C0-7B6}
Message-ID: <CAC3AC395FD04CB1BB7DCEC764E7816E@example.com>
From: fred <fred@example.com>
To: <johndoe@example.com>
...

The CheckPoint firewall was injecting an ID into the headers which had exactly the same format as the “Session ID” from the ones that were failing with “Permanent scan failure”.

So unbeknownst to us CheckPoint was proxying/filtering the SMTP traffic and barfing on it for some reason.  When we switched off the SMTP checks that CheckPoint was doing (they weren’t required as we have another mail gateways in place) the problem went away.