Lost in Reception

(In case it’s not amazingly obvious, I do have a commercial interest in the topic of this post!)

I am thrilled to be able to say that BoxSentry has finally released the third and most ambitious form of our approach to preventing email loss caused by [anti-]spam filters: TrustCloud.

The website covers what it is and does pretty well so I won’t repeat it here. Sadly, as with RealMail and LogiQ, only the administrators or developers of email security systems can deploy it; an end-user deployable version remains a wishlist item for the time being.

Six months ago an oversight by McAfee rendered thousands of PCs inoperative. Over the weekend a near-identical oversight by Sourcefire caused an immense number of legitimate email messages to be misclassified as malware.

I commented on the problems that arose when older versions of McAfee’s VirusScan accepted signature updates that required a newer engine version to operate correctly. In addition to pointing out that what happened was not specific to McAfee, I wrote:

Fail safe! Deal with version skew. Bricking a device because its software was out of date is not a reasonable failure mode.

My thinking behind this was that signature matching engines of any sort should (a) know their “version” and (b) have a means for an automatically-delivered signature update to state what engine version it requires. I took for granted that any engine for which these two things were true would, of course, refuse to deploy a signature update which required a newer engine version than was present. This turns out to be an incorrect assumption.

Over the weekend, ClamAV users running an older engine had a problem with this new signature misclassifying legitimate messages as malware:

d41d8cd98f00b204e9800998ecf8427e:0:Exploit.PDF-9669

Setting aside any other considerations for the moment, this is a rather special signature:

$ md5sum </dev/null
d41d8cd98f00b204e9800998ecf8427e  -
$

which suggests that someone at Sourcefire was putting cleverness ahead of simplicity; a rule that a particular element of a PDF file must never be empty should probably be implemented directly in the PDF parser, rather than delegated to a signature match particularly when – as appears to be the case here – the code that can use the signature is rather new.

In any event, ClamAV does have an engine-version specification and checking mechanism, as I had suggested for McAfee’s case half a year ago, but it’s only advisory!

WARNING: Your ClamAV installation is OUTDATED!
WARNING: Local version: 0.92.1 Recommended version: 0.95.3
DON’T PANIC! Read http://www.clamav.net/support/faq
…
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Current functionality level = 26, recommended = 44

I suspect that this is tied up with the view that a little protection is better than none, an engine that knows to simply ignore signatures that it can’t parse and, in the early days anyway, a tradition of making signatures as specific as possible, so the false-positive risk remained negligible. The nett result of otherwise-sensible choices is a serious disruption to the very services that ClamAV is deployed to protect.

So, further to my suggestions of six months ago, add a requirement that engines should never accept signature updates that state a requirement for a newer engine than is present.

The requirement to do something sensible when such a situation is detected (prominently tell the user/admin, etc.) remains an important additional requirement for anyone using a signature-based system.

A final thought: I’m sure it won’t help, but maybe signature and update numbers containing “66″ should be avoided outright; it is perhaps tempting fate, just a little bit :-)

An interesting post by Laura Atkins on a trend towards raising the bar for bulk senders.

It’s [almost] surprising that this has taken so long to occur.

I haven’t previously noticed Building Web Reputation Systems, a quote from Schneier about a Sims Mafia alerted me to it:

“Hi! I see from your hub that you’re new to the area. Give me all your Simoleans or my friends and I will make it impossible to rent a house.”

Needless to say, the assumptions that the article as a whole works from are too simple to build a useful system from, but the Sims story illustrates on important point; the errors that beginners make in this area are causing actual harm to people and businesses. Getting workable solutions into the real world is becoming increasingly important and urgent.

Interesting article by J. D. Falk on the difficulties of bulk email and an argument that it may be a bad idea even for legitimate senders.

A breakdown of different failure modes for DKIM and their relative prevalence, as seen by Cisco’s email infrastructure: Common Errors Causing DKIM Verification Failures

Much that lines up with my own experience from John Levine: How do you test spam filters?

We’ve decided that certain filtering methodologies are simply unacceptable, such as, rejection notification by bounce rather than SMTP reject.

Quite.

Terry Zink has just posted a sensible Best practices for sending outbound mail which shares a fair bit with MAAWG’s Sender BCP Document.

This is a difficult approach to take a position on. I applaud any activity to make it easier to separate good from bad, but:

• There really are too many other pressures on rDNS naming, not least the large already deployed base of billions of names, to hold out much hope for the rollout of an rDNS naming standard across more than a tiny fraction of the world’s ISPs.
• Penalising legitimate senders for the non-malicious [non-]actions of their service providers is an unwarranted harm (penalising them for malicious actions and negligent inaction by their service providers is also harmful, but difficult to avoid).

The establishing of norms for white hats to make clear their expectations about their own infrastructure is a good idea, but using these norms by a communication partner requires two additional pieces of data:

• An assessment of the integrity of the person/organisation in control of the rDNS: as soon as rDNS records meeting this norm become a means for receivers to prioritise legitimate senders, those address blocks whose rDNS is controlled by abusers will suddenly have rDNS records which meet this norm. Note that this is equivalent to the need for reputation data for authentication systems (SPF, DKIM, …).
• An assessment of the ability of the person/organisation in control of the rDNS to keep their data up to date. That the records meet the suggested patterns only tells an assessor about what was happening when the records were created, not about what’s happened since. This is likely to be a rather more difficult task to get right than, say, keeping SPF/DKIM records up to date if only because of the sheer number of records involved.

On the whole, using rDNS to aid in spam identification provides a pretty marginal gain, at too great a cost (in FUSSP terms it “Requires immediate total cooperation from everybody at once”), with too much need for additional information and with too great a false positive risk (there are whole countries where rDNS control is unavailable to access/hosting customers of ISPs) to be particularly useful.

Of course, making rDNS informative for occasional reference by administrators (as distinct from high-frequency automated assessment by software) – and keeping it up to date – remains sound practice.

Various things:

• Keep your AV software up to date, not just your signatures.
• Corollary: make certain you’re aware of the presence of out-of-date protection in your environment.
• Fail safe! Deal with version skew. Bricking a device because its software was out of date is not a reasonable failure mode.
• There is nothing in what happened that is specific to McAfee. Similar risks apply to all protective systems.
• As important as host-based protective software is, it actually introduces an entire class of risks that need to be understood.
• Human error is often at least as dangerous as malice. Provide defence in depth not only to limit the damage done by intruders, but also that caused by accidents.

Further thoughts:

• This is about false positives in in-host malware identification rather than in-network spam identification, but there is some overlap. In case you missed it, details are here. The issue appears to be that signature update “DAT 5644″ is not intended for use on EOL’d engine version 5100, but the latter is accepting the update anyway and then misclassifying legitimate software including vital HP/Compaq drivers, which are being quarantined, which is blue-screening many thousands of affected machines.
• Brandon Wiley’s curious blue idea suggests that this problem is going to get worse over time because, serious as the risks of automatically deploying the latest and greatest software are, the risks associated with not minimising exposure are greater, and becoming more so. Worse, the update distribution mechanisms required to deploy protection as quickly as possible provide a potential vector for intruders. A single exploit in a major AV vendor’s release process could, in a matter of hours, produce a botnet hundreds of times larger than anything seen to date. (Think this sort of exploit can’t happen? We could always hope that banks turn out to be less adept at managing risk than computer security companies are.) This almost argues for different schedules for machines at different levels sensitivity and, paradoxically, for updates on more sensitive machines (those which should be behind several layers of protection already) to be applied less rapidly. It similarly argues for intentionally using a mix of systems – and minority platforms in particular – in an organisation’s most sensitive environments.
• There are now [at least] two commercial pressures that lead to the “device bricking” scenario: (a) the well-known “jail-breaking” of iPhones leading to updates failing in a way which renders the device unusable and (b) the “out-of-date” software automatically applying updates which has the same effect. Even if a vendor is willing to punish customers who knowingly work against them (similar to the DRM-peddler’s conceit that a device owner is nothing more than a source of profit), customers who fail, for one reason or another, to keep their software up to date really should be accommodated, at least to the point where failures are graceful. This is not an appeal to altruism but to self-interest on the vendor’s part: right now, thousands of CSOs, IT managers and the like are wondering whether they’ll ever purchase McAfee products again. At least some will decide not to. The simple, pragmatic act of having signature updates contain enough metadata to indicate which engine version they require and having engines check this, rather than blindly applying updates, would have prevented this.
• The “fail safe” choice is more difficult than my comment suggests but, in any event, an auto-lobotomy is not a reasonable failure mode (any more than auto-vaporising is). The problem is that dealing with the incompatibility by ceasing to apply updates means that a machine is being unexpectedly transitioned from “up to date protection” to “out of date protection” while still in use, connected to the Internet, etc. There is a policy choice here: for corporate-owned machines where security responsibility is not with the user of the machine, a protective shutdown and lockout which only an authorised administrator can undo is a better response. For machines where the user is also responsible for security (e.g. owns the machine), a system-tray alert that protection is now outdated – and will require an upgrade to resolve – would be a more appropriate response.
• Host-based protective software (at least where it has the ability to prevent, rather than simply to alert) introduces self-DoS risks which affect product selection and configuration choices. This is not to say that host-based software shouldn’t be used (quite the contrary, I believe that it is a must), just that it is worth recognising that host- and network- based aren’t just “the same thing in two different places”; rather they have a different set of risks and benefits.