Lost in Reception

A breakdown of different failure modes for DKIM and their relative prevalence, as seen by Cisco’s email infrastructure: Common Errors Causing DKIM Verification Failures

Much that lines up with my own experience from John Levine: How do you test spam filters?

We’ve decided that certain filtering methodologies are simply unacceptable, such as, rejection notification by bounce rather than SMTP reject.

Quite.

Terry Zink has just posted a sensible Best practices for sending outbound mail which shares a fair bit with MAAWG’s Sender BCP Document.

This is a difficult approach to take a position on. I applaud any activity to make it easier to separate good from bad, but:

• There really are too many other pressures on rDNS naming, not least the large already deployed base of billions of names, to hold out much hope for the rollout of an rDNS naming standard across more than a tiny fraction of the world’s ISPs.
• Penalising legitimate senders for the non-malicious [non-]actions of their service providers is an unwarranted harm (penalising them for malicious actions and negligent inaction by their service providers is also harmful, but difficult to avoid).

The establishing of norms for white hats to make clear their expectations about their own infrastructure is a good idea, but using these norms by a communication partner requires two additional pieces of data:

• An assessment of the integrity of the person/organisation in control of the rDNS: as soon as rDNS records meeting this norm become a means for receivers to prioritise legitimate senders, those address blocks whose rDNS is controlled by abusers will suddenly have rDNS records which meet this norm. Note that this is equivalent to the need for reputation data for authentication systems (SPF, DKIM, …).
• An assessment of the ability of the person/organisation in control of the rDNS to keep their data up to date. That the records meet the suggested patterns only tells an assessor about what was happening when the records were created, not about what’s happened since. This is likely to be a rather more difficult task to get right than, say, keeping SPF/DKIM records up to date if only because of the sheer number of records involved.

On the whole, using rDNS to aid in spam identification provides a pretty marginal gain, at too great a cost (in FUSSP terms it “Requires immediate total cooperation from everybody at once”), with too much need for additional information and with too great a false positive risk (there are whole countries where rDNS control is unavailable to access/hosting customers of ISPs) to be particularly useful.

Of course, making rDNS informative for occasional reference by administrators (as distinct from high-frequency automated assessment by software) – and keeping it up to date – remains sound practice.

Various things:

• Keep your AV software up to date, not just your signatures.
• Corollary: make certain you’re aware of the presence of out-of-date protection in your environment.
• Fail safe! Deal with version skew. Bricking a device because its software was out of date is not a reasonable failure mode.
• There is nothing in what happened that is specific to McAfee. Similar risks apply to all protective systems.
• As important as host-based protective software is, it actually introduces an entire class of risks that need to be understood.
• Human error is often at least as dangerous as malice. Provide defence in depth not only to limit the damage done by intruders, but also that caused by accidents.

Further thoughts:

• This is about false positives in in-host malware identification rather than in-network spam identification, but there is some overlap. In case you missed it, details are here. The issue appears to be that signature update “DAT 5644″ is not intended for use on EOL’d engine version 5100, but the latter is accepting the update anyway and then misclassifying legitimate software including vital HP/Compaq drivers, which are being quarantined, which is blue-screening many thousands of affected machines.
• Brandon Wiley’s curious blue idea suggests that this problem is going to get worse over time because, serious as the risks of automatically deploying the latest and greatest software are, the risks associated with not minimising exposure are greater, and becoming more so. Worse, the update distribution mechanisms required to deploy protection as quickly as possible provide a potential vector for intruders. A single exploit in a major AV vendor’s release process could, in a matter of hours, produce a botnet hundreds of times larger than anything seen to date. (Think this sort of exploit can’t happen? We could always hope that banks turn out to be less adept at managing risk than computer security companies are.) This almost argues for different schedules for machines at different levels sensitivity and, paradoxically, for updates on more sensitive machines (those which should be behind several layers of protection already) to be applied less rapidly. It similarly argues for intentionally using a mix of systems – and minority platforms in particular – in an organisation’s most sensitive environments.
• There are now [at least] two commercial pressures that lead to the “device bricking” scenario: (a) the well-known “jail-breaking” of iPhones leading to updates failing in a way which renders the device unusable and (b) the “out-of-date” software automatically applying updates which has the same effect. Even if a vendor is willing to punish customers who knowingly work against them (similar to the DRM-peddler’s conceit that a device owner is nothing more than a source of profit), customers who fail, for one reason or another, to keep their software up to date really should be accommodated, at least to the point where failures are graceful. This is not an appeal to altruism but to self-interest on the vendor’s part: right now, thousands of CSOs, IT managers and the like are wondering whether they’ll ever purchase McAfee products again. At least some will decide not to. The simple, pragmatic act of having signature updates contain enough metadata to indicate which engine version they require and having engines check this, rather than blindly applying updates, would have prevented this.
• The “fail safe” choice is more difficult than my comment suggests but, in any event, an auto-lobotomy is not a reasonable failure mode (any more than auto-vaporising is). The problem is that dealing with the incompatibility by ceasing to apply updates means that a machine is being unexpectedly transitioned from “up to date protection” to “out of date protection” while still in use, connected to the Internet, etc. There is a policy choice here: for corporate-owned machines where security responsibility is not with the user of the machine, a protective shutdown and lockout which only an authorised administrator can undo is a better response. For machines where the user is also responsible for security (e.g. owns the machine), a system-tray alert that protection is now outdated – and will require an upgrade to resolve – would be a more appropriate response.
• Host-based protective software (at least where it has the ability to prevent, rather than simply to alert) introduces self-DoS risks which affect product selection and configuration choices. This is not to say that host-based software shouldn’t be used (quite the contrary, I believe that it is a must), just that it is worth recognising that host- and network- based aren’t just “the same thing in two different places”; rather they have a different set of risks and benefits.

In addition to the false-positive risks in email that interest me generally and the real-world analogies that I noted a couple of months back, it appears that some behaviour by legitimate software developers is indistinguishable from that of malware developers.

Since its inception, the closed-source software industry has benefited from – even depended upon – the fact that object code is much harder to work with than source code, so keeping source code secret and delivering object code to customers has been an effective way to help keep a developer’s secrets secret. Increasingly, tools for reverse-engineering are diminishing this advantage so those developers whose businesses are particularly dependent upon keeping their implementation details secret are turning to code-obfuscation tools to make analysis of their delivered code even more difficult. Naturally, malware authors are using the same tools to the same end.

Igor Muttik opens a recent post by suggesting that anti-malware technology is digging an elephant trap for us all because of the above; he moves on to suggest that legitimate software authors might consider (a) not obfuscating their code in the first place or (b) signing their code if they do obfuscate it.

This closely parallels the forces at work in dealing with email abuse:

• It is unwise to behave like a bad guy (conceal click-counters under links in email, send frequently to people who haven’t asked you to; or, in the physical world, wear a balaclava into a bank, …).
• An important mitigation of the false positive risk is to increase the ability of assessors to identify good actors which, when errors in both directions are common, is not simply the inverse of the ability to identify bad actors.

Part of the solution being pursued by anti-malware vendors (who might now prefer to be termed security vendors) is a transition to whitelisting, much as is beginning to happen on a large scale for email.

Perhaps this is obvious. We depend upon our ability to recognise trustworthy professionals (doctors, accountants, engineers) through the licensing procedures of their professional bodies, upon police officers’ ability to recognise trustworthy participants in high-risk activities (notably the driving of powered vehicles on public streets) through vehicle registration and driver licensing procedures and upon our own ability to recognise trustworthy borrowers through credit-rating agencies. Frequently the means of dealing with negligent or malicious behaviour in public, at least when life and limb are not at stake, is to provide the means for anyone to recognise individuals who are trustworthy. Perhaps applying such ideas to email and to software loading/execution is merely going back to the future.

Further thoughts:

• The term “elephant trap” is an interesting one. I’ve never given any thought to the trapping of elephants but I imagine that a large, disguised pit placed where elephants might be expected, or caused, to stampede would suffice. Whether the resulting trapped elephants, with multiple broken bones, would be of value as beasts of burden is a speculation left open to the reader. Wikipedia has an entirely different take on this: apparently it’s the name of a situation in chess.
• Legitimate developers using obfuscation – or even those merely depending upon the difficulty of working with object code -, malware developers and DRM advocates all share a similar mindset: that it is reasonable to turn a computer against its owner contrary to that owner’s desires (albeit with their consent, for suitably convoluted uses of the term “consent“), a situation described in common law as conversion when consent is absent. When your customers are your opponents, life gets difficult. It bothers me that my living has tended to derive from sources that [believe that they] [might] have this dependency.

(Relevant disclosure: as usual, I am writing on my own behalf but there is a considerable overlap between my view and the views and initiatives of BoxSentry.)

Terry writes a reasoned post on the consequence of the use by users of “This is Spam” buttons for no-longer-wanted newsletters, but then makes on odd assertion:

The one drawback of this feature is that the sender is blissfully unaware that their messages are not getting through to the end user.  However, that’s a topic to be dealt with by the email-deliverability folks, I’m in the business of impeding mail flow!

I have a longer post coming on deciding-at-SMTP-receipt time what to do with a message. Some of the arguments are finely balanced but this one is a slam dunk: if the recipient has indicated that they don’t wish to receive messages from a specified sender and that sender sends to that recipient again, then the message should be refused at RCPT TO (i.e. before the message even crosses the wire). This makes clear to the sender that this address is not to be sent to again and does so in a way which reduces resource consumption in the anti-spam system; it’s an actual free lunch!

Oh, and Terry, if you’re really in the business of impeding mail flow, you could save yourself a lot of trouble by simply turning off your servers :-) I’d suggest that you’re actually in the business of facilitating the flow of legitimate mail and that impeding the flow of abusive mail is merely a means to that end.

I wrote:

I gather that VB intends to publish both the statistically sound false-positive rates and the industry-standard “nice” rates. I’m hoping that they do so.

Well, my wish was granted! (you’ll need to register)

For each product they stated both “FP rate” and “FP of total mail corpus”. Once various special cases (see below) have been removed, FP rates of 1-3% were observed. Like any other actual measurement, these numbers are not necessarily representative of anything – and these may have been worsened by VB’s particular experiment setup – however they are at the upper end of what BoxSentry is seeing each time we perform an analysis.

Before commenting on the specifics, I’d like to address what I see as an error in VB’s report. Anyone who has completed high-school physics will have dealt with measurement and the need to discard erroneous measurements before reaching conclusions. The most common reason for choosing to do this is the observation of results that lie so far outside the norm that they are clearly incorrect. (Scenario: you’re analysing data on [human] child birth, one of your subjects has a recorded birth weight of 6125kg. This is clearly an error and should be removed from your data before calculating average birth weights.) This occurred with one of VB’s subjects. The report says:

a false positive rate of over 25% of all ham messages is
almost certainly a sign of product misconfiguration

While I feel a certain degree of schadenfreude at this product’s results, I am inclined to agree with this assessment and therefore feel that VB’s results would have been more sound had this product been removed from the results entirely. The decision to keep the rules constant for any complete run of the tests is a good idea, but not the extent of producing obviously incorrect results.

I’d also be inclined to discount the ClamAV and SpamAssassin results because, in both cases, the spam-catch rate was too low to be useful: the false negatives (spam reaching the inbox) were a considerable multiple of the true negatives (legitimate email reaching the inbox).

This leaves just three products. Despite VB’s careful disclaimers, their results are largely what I’d expected, albeit at the high end of the range. Let us hope that these results aren’t enough to discourage vendor participation in future trials.

The loss in confidence in email as a communication channel is beginning to spread to other channels as well.

To date, most concern – my own included – about mail loss through spam-filter false-positives has been about the harm done to people and organisations by this loss, and the decrease in confidence that these losses cause in email as a communication channel. It seems that the amount of email/SMS scamming going on world-wide is starting to cause people to question all communication claiming to be from an authority figure. While I welcome any decrease in the gullibility of populations at large – it is this very gullibility that has given spammers free reign to begin with – organisations that have grown up in an environment where citizens uncritically trust communication appearing to be from an authority may find their ability to function critically impaired in populations where widespread scamming causes a rapid decrease in acceptance of claims of authority and where those organisations don’t have an immediately obvious way of proving to citizens that they are who they say they are.

An interesting article by Dan Blacharski on problems in Korea notes exactly this. Upon being contacted by a police officer advising cancellation of a payment to a fraudster, the now-suspicious victim replied:

“Dirty swindler! If you’re a policeman, I’m your grandfather!”

The ultimate solution:

[government authorities] must establish a safe protocol for communicating with citizens when it is necessary to ensure legitimacy

which is much easier said than done. Finally, this nugget:

Some Korean police departments are sending a written summons before making a phone call.

which doesn’t really address the problem; convincing people to trust people who assert their identity twice will simply cause scammers to assert their identity twice too.

Virus Bulletin is gearing up to perform regular tests of anti-spam systems as an addition to their existing coverage of anti-virus systems.
They’ve performed their first trial and published anonymised results that are interesting, to say the least. They report false positive rates for the submitted systems falling between 0.04% and 0.4%. Unfortunately:

Following industry standards, the false positive rate, or ‘FP rate’, is the ratio of the number of false positives relative to the total number of emails.

Correctly determining what numbers to use when calculating a false positive rate requires a rudimentary knowledge of statistics. Early in spam’s history (when it was <1% of all email), getting the denominator wrong (“all messages” rather than “all legitimate messages”) made very little difference. Now that spam is closer to 95% of all email – and still climbing – this small error makes an enormous difference. At some stage vendors chose to report false-positive rates calculated over total message volume, rather than legitimate message volume, because it made the figures a little better. Sadly, there’s no convenient time for a vendor to undo that practice, even when the difference now makes the published figures more or less meaningless.

According to VB:

During this period, the filters saw a total of 20,764 emails, 877 of which were classified as ham by VB’s employees (the recipients)

Changing the denominator from 20 764 to 877 multiplies the result by 20 764 / 877 ~= 23.676. This is a pretty large correction; the “correct” figures for VB’s trial run are therefore 0.95% – 9.5%.

But, does it matter? Numbers like 0.04% look so small that they can be disregarded, but once it’s clear that this number means the loss of 1 in 100 legitimate messages, a potential customer is likely to see the number as being rather more important. Pity the vendor with the 0.4% result; 1 legitimate message in 10 will go astray!

I gather that VB intends to publish both the statistically sound false-positive rates and the industry-standard “nice” rates. I’m hoping that they do so.