Lost in Reception

Six months ago an oversight by McAfee rendered thousands of PCs inoperative. Over the weekend a near-identical oversight by Sourcefire caused an immense number of legitimate email messages to be misclassified as malware.

I commented on the problems that arose when older versions of McAfee’s VirusScan accepted signature updates that required a newer engine version to operate correctly. In addition to pointing out that what happened was not specific to McAfee, I wrote:

Fail safe! Deal with version skew. Bricking a device because its software was out of date is not a reasonable failure mode.

My thinking behind this was that signature matching engines of any sort should (a) know their “version” and (b) have a means for an automatically-delivered signature update to state what engine version it requires. I took for granted that any engine for which these two things were true would, of course, refuse to deploy a signature update which required a newer engine version than was present. This turns out to be an incorrect assumption.

Over the weekend, ClamAV users running an older engine had a problem with this new signature misclassifying legitimate messages as malware:

d41d8cd98f00b204e9800998ecf8427e:0:Exploit.PDF-9669

Setting aside any other considerations for the moment, this is a rather special signature:

$ md5sum </dev/null
d41d8cd98f00b204e9800998ecf8427e  -
$

which suggests that someone at Sourcefire was putting cleverness ahead of simplicity; a rule that a particular element of a PDF file must never be empty should probably be implemented directly in the PDF parser, rather than delegated to a signature match particularly when – as appears to be the case here – the code that can use the signature is rather new.

In any event, ClamAV does have an engine-version specification and checking mechanism, as I had suggested for McAfee’s case half a year ago, but it’s only advisory!

WARNING: Your ClamAV installation is OUTDATED!
WARNING: Local version: 0.92.1 Recommended version: 0.95.3
DON’T PANIC! Read http://www.clamav.net/support/faq
…
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Current functionality level = 26, recommended = 44

I suspect that this is tied up with the view that a little protection is better than none, an engine that knows to simply ignore signatures that it can’t parse and, in the early days anyway, a tradition of making signatures as specific as possible, so the false-positive risk remained negligible. The nett result of otherwise-sensible choices is a serious disruption to the very services that ClamAV is deployed to protect.

So, further to my suggestions of six months ago, add a requirement that engines should never accept signature updates that state a requirement for a newer engine than is present.

The requirement to do something sensible when such a situation is detected (prominently tell the user/admin, etc.) remains an important additional requirement for anyone using a signature-based system.

A final thought: I’m sure it won’t help, but maybe signature and update numbers containing “66″ should be avoided outright; it is perhaps tempting fate, just a little bit :-)

An interesting post by Laura Atkins on a trend towards raising the bar for bulk senders.

It’s [almost] surprising that this has taken so long to occur.

I haven’t previously noticed Building Web Reputation Systems, a quote from Schneier about a Sims Mafia alerted me to it:

“Hi! I see from your hub that you’re new to the area. Give me all your Simoleans or my friends and I will make it impossible to rent a house.”

Needless to say, the assumptions that the article as a whole works from are too simple to build a useful system from, but the Sims story illustrates on important point; the errors that beginners make in this area are causing actual harm to people and businesses. Getting workable solutions into the real world is becoming increasingly important and urgent.

Interesting article by J. D. Falk on the difficulties of bulk email and an argument that it may be a bad idea even for legitimate senders.

A breakdown of different failure modes for DKIM and their relative prevalence, as seen by Cisco’s email infrastructure: Common Errors Causing DKIM Verification Failures

Much that lines up with my own experience from John Levine: How do you test spam filters?

We’ve decided that certain filtering methodologies are simply unacceptable, such as, rejection notification by bounce rather than SMTP reject.

Quite.

Terry Zink has just posted a sensible Best practices for sending outbound mail which shares a fair bit with MAAWG’s Sender BCP Document.

This is a difficult approach to take a position on. I applaud any activity to make it easier to separate good from bad, but:

• There really are too many other pressures on rDNS naming, not least the large already deployed base of billions of names, to hold out much hope for the rollout of an rDNS naming standard across more than a tiny fraction of the world’s ISPs.
• Penalising legitimate senders for the non-malicious [non-]actions of their service providers is an unwarranted harm (penalising them for malicious actions and negligent inaction by their service providers is also harmful, but difficult to avoid).

The establishing of norms for white hats to make clear their expectations about their own infrastructure is a good idea, but using these norms by a communication partner requires two additional pieces of data:

• An assessment of the integrity of the person/organisation in control of the rDNS: as soon as rDNS records meeting this norm become a means for receivers to prioritise legitimate senders, those address blocks whose rDNS is controlled by abusers will suddenly have rDNS records which meet this norm. Note that this is equivalent to the need for reputation data for authentication systems (SPF, DKIM, …).
• An assessment of the ability of the person/organisation in control of the rDNS to keep their data up to date. That the records meet the suggested patterns only tells an assessor about what was happening when the records were created, not about what’s happened since. This is likely to be a rather more difficult task to get right than, say, keeping SPF/DKIM records up to date if only because of the sheer number of records involved.

On the whole, using rDNS to aid in spam identification provides a pretty marginal gain, at too great a cost (in FUSSP terms it “Requires immediate total cooperation from everybody at once”), with too much need for additional information and with too great a false positive risk (there are whole countries where rDNS control is unavailable to access/hosting customers of ISPs) to be particularly useful.

Of course, making rDNS informative for occasional reference by administrators (as distinct from high-frequency automated assessment by software) – and keeping it up to date – remains sound practice.

Various things:

• Keep your AV software up to date, not just your signatures.
• Corollary: make certain you’re aware of the presence of out-of-date protection in your environment.
• Fail safe! Deal with version skew. Bricking a device because its software was out of date is not a reasonable failure mode.
• There is nothing in what happened that is specific to McAfee. Similar risks apply to all protective systems.
• As important as host-based protective software is, it actually introduces an entire class of risks that need to be understood.
• Human error is often at least as dangerous as malice. Provide defence in depth not only to limit the damage done by intruders, but also that caused by accidents.

Further thoughts:

• This is about false positives in in-host malware identification rather than in-network spam identification, but there is some overlap. In case you missed it, details are here. The issue appears to be that signature update “DAT 5644″ is not intended for use on EOL’d engine version 5100, but the latter is accepting the update anyway and then misclassifying legitimate software including vital HP/Compaq drivers, which are being quarantined, which is blue-screening many thousands of affected machines.
• Brandon Wiley’s curious blue idea suggests that this problem is going to get worse over time because, serious as the risks of automatically deploying the latest and greatest software are, the risks associated with not minimising exposure are greater, and becoming more so. Worse, the update distribution mechanisms required to deploy protection as quickly as possible provide a potential vector for intruders. A single exploit in a major AV vendor’s release process could, in a matter of hours, produce a botnet hundreds of times larger than anything seen to date. (Think this sort of exploit can’t happen? We could always hope that banks turn out to be less adept at managing risk than computer security companies are.) This almost argues for different schedules for machines at different levels sensitivity and, paradoxically, for updates on more sensitive machines (those which should be behind several layers of protection already) to be applied less rapidly. It similarly argues for intentionally using a mix of systems – and minority platforms in particular – in an organisation’s most sensitive environments.
• There are now [at least] two commercial pressures that lead to the “device bricking” scenario: (a) the well-known “jail-breaking” of iPhones leading to updates failing in a way which renders the device unusable and (b) the “out-of-date” software automatically applying updates which has the same effect. Even if a vendor is willing to punish customers who knowingly work against them (similar to the DRM-peddler’s conceit that a device owner is nothing more than a source of profit), customers who fail, for one reason or another, to keep their software up to date really should be accommodated, at least to the point where failures are graceful. This is not an appeal to altruism but to self-interest on the vendor’s part: right now, thousands of CSOs, IT managers and the like are wondering whether they’ll ever purchase McAfee products again. At least some will decide not to. The simple, pragmatic act of having signature updates contain enough metadata to indicate which engine version they require and having engines check this, rather than blindly applying updates, would have prevented this.
• The “fail safe” choice is more difficult than my comment suggests but, in any event, an auto-lobotomy is not a reasonable failure mode (any more than auto-vaporising is). The problem is that dealing with the incompatibility by ceasing to apply updates means that a machine is being unexpectedly transitioned from “up to date protection” to “out of date protection” while still in use, connected to the Internet, etc. There is a policy choice here: for corporate-owned machines where security responsibility is not with the user of the machine, a protective shutdown and lockout which only an authorised administrator can undo is a better response. For machines where the user is also responsible for security (e.g. owns the machine), a system-tray alert that protection is now outdated – and will require an upgrade to resolve – would be a more appropriate response.
• Host-based protective software (at least where it has the ability to prevent, rather than simply to alert) introduces self-DoS risks which affect product selection and configuration choices. This is not to say that host-based software shouldn’t be used (quite the contrary, I believe that it is a must), just that it is worth recognising that host- and network- based aren’t just “the same thing in two different places”; rather they have a different set of risks and benefits.

In addition to the false-positive risks in email that interest me generally and the real-world analogies that I noted a couple of months back, it appears that some behaviour by legitimate software developers is indistinguishable from that of malware developers.

Since its inception, the closed-source software industry has benefited from – even depended upon – the fact that object code is much harder to work with than source code, so keeping source code secret and delivering object code to customers has been an effective way to help keep a developer’s secrets secret. Increasingly, tools for reverse-engineering are diminishing this advantage so those developers whose businesses are particularly dependent upon keeping their implementation details secret are turning to code-obfuscation tools to make analysis of their delivered code even more difficult. Naturally, malware authors are using the same tools to the same end.

Igor Muttik opens a recent post by suggesting that anti-malware technology is digging an elephant trap for us all because of the above; he moves on to suggest that legitimate software authors might consider (a) not obfuscating their code in the first place or (b) signing their code if they do obfuscate it.

This closely parallels the forces at work in dealing with email abuse:

• It is unwise to behave like a bad guy (conceal click-counters under links in email, send frequently to people who haven’t asked you to; or, in the physical world, wear a balaclava into a bank, …).
• An important mitigation of the false positive risk is to increase the ability of assessors to identify good actors which, when errors in both directions are common, is not simply the inverse of the ability to identify bad actors.

Part of the solution being pursued by anti-malware vendors (who might now prefer to be termed security vendors) is a transition to whitelisting, much as is beginning to happen on a large scale for email.

Perhaps this is obvious. We depend upon our ability to recognise trustworthy professionals (doctors, accountants, engineers) through the licensing procedures of their professional bodies, upon police officers’ ability to recognise trustworthy participants in high-risk activities (notably the driving of powered vehicles on public streets) through vehicle registration and driver licensing procedures and upon our own ability to recognise trustworthy borrowers through credit-rating agencies. Frequently the means of dealing with negligent or malicious behaviour in public, at least when life and limb are not at stake, is to provide the means for anyone to recognise individuals who are trustworthy. Perhaps applying such ideas to email and to software loading/execution is merely going back to the future.

Further thoughts:

• The term “elephant trap” is an interesting one. I’ve never given any thought to the trapping of elephants but I imagine that a large, disguised pit placed where elephants might be expected, or caused, to stampede would suffice. Whether the resulting trapped elephants, with multiple broken bones, would be of value as beasts of burden is a speculation left open to the reader. Wikipedia has an entirely different take on this: apparently it’s the name of a situation in chess.
• Legitimate developers using obfuscation – or even those merely depending upon the difficulty of working with object code -, malware developers and DRM advocates all share a similar mindset: that it is reasonable to turn a computer against its owner contrary to that owner’s desires (albeit with their consent, for suitably convoluted uses of the term “consent“), a situation described in common law as conversion when consent is absent. When your customers are your opponents, life gets difficult. It bothers me that my living has tended to derive from sources that [believe that they] [might] have this dependency.

(Relevant disclosure: as usual, I am writing on my own behalf but there is a considerable overlap between my view and the views and initiatives of BoxSentry.)