Spam costs society at least twice the $20 billion/year that the authors estimate, and probably a great more than that. Their paper is well argued and mostly well researched, however they do make one peculiar and wholly unsupported assertion
False positives … are … so rare that we ignore them in this estimate
which falls down on at least two fronts:
- false-positive errors (legitimate messages not reaching inboxes) occur at a rate of the same order as that at which false-negative errors (spam reaching inboxes) do, and
- false-positive errors inflict more variable, and typically far higher, costs than false-negative errors do.
As part of my work with TrustSphere I’ve been involved in the analysis of email security systems from most vendors and the review of those systems’ assessments of around half a billion messages received by a variety of customer types located in multiple regions and have observed that false-positive errors occur at similar rates in all cases. The consequences of a single false-positive error range from “no impact” through time wasted searching spam folders/quarantines all the way to compliance failures, lost sales, missed deadlines/appointments, loss of trust in an organisation and its infrastructure, angry customers, supply chain disruptions, etc.
Inconveniently, TrustSphere has not yet published its findings in a peer-reviewed journal. Things to look forward to…
- Their assertion really was unsupported: they just say it one time, offer no supporting data and make no further mention of it. This struck me as an odd choice in an otherwise pretty solid paper.
- It occurred to me to wonder how “good” the Journal of Economic Perspectives actually is. It turns out that the Australian Research Council (a statutory body) has actually published a ranking of 20 000 peer-reviewed journals! The Journal receives the highest ranking: A* “Typically an A* journal would be one of the best in its field or subfield…”. It’s worth noting that there is another similarly-named publication – the International Journal of Economic Perspectives – which receives the lowest ranking: C “Tier C includes quality, peer reviewed, journals that do not meet the criteria of the higher tiers.” Sadly, the publication of a ranking ruffled sufficient feathers that no updates have been or will be published. (thanks)
(I know; it’s been a long time between posts… also, I obviously do have a commercial interest in the topic of this post.)
Enterprises that want to ensure delivery of important business messages should consider the TrustSphere LogiQ App – don’t let those important emails get lost in your junk folder!
One of my bugbears when talking to customers about message loss through spam filter false-positive errors is that most email security vendors understate their false-positive rates by about an order of magnitude. Terry has noticed this too:
The industry cheats quite a bit with their SLAs, the language is deliberately ambiguous. If a company claims a 1 in 25,000 false positive SLA, what that means is that they permit 1 false positive per 25,000 messages. This means that if the spam/ham ratio is 10:1, then in 25,000 messages there will be 2272 hams and 22,728 spam messages. If one of the good messages is flagged as spam, then the good mail FP rate is 1/2272 = 0.04%, which is actually quite high. Yet by saying that you permit 1 in 25,000 messages, and messages is not defined but assumed to be both spam + non-spam, vendors have permitted themselves a lot of leeway when calculating how accurate their product is against good mail… by a factor of 10.
Most of the loss of legitimate email through filtering arises because the mindset is about blocking bad messages. BoxSentry has been making the case for several years that combining this approach with the ability to recognise most legitimate messages will improve the accuracy of filtering and, in many cases, reduce the resource cost of doing so. I’ve just noticed a three year old white paper by F5 making a similar argument about security in general.
Murphy and Salchow describe one way of looking at combining “positive” (everything not permitted is prohibited) and “negative” (everything not prohibited is permitted) approaches to security and make an efficiency argument for choosing a combined approach rather than using one or the other exclusively. The argument makes some sense, but overlooks the fact that in environments where a great deal of what distinguishes good from bad behaviour is unknowable, even in principle, the quantitative efficiency argument quickly grows into a qualitative argument about what is and isn’t possible. That is, the “more efficient” combined model is being compared with either of the single-approach models in which the cost of providing equivalent protection would be infinite.
I agree with Murphy and Salchow’s argument that the approaches should be combined, but I believe that the argument for doing so is considerably stronger than what they offer.
(Stated another way, they end up understating the impact because they fail to acknowledge that the number of unknown behaviours – good or bad – is usually effectively infinite, meaning that their graph on page 4 is incorrect. Of course there are situations in which all possible behaviours can reliably be finitely enumerated – for example there are only 65536 TCP port numbers that a client can be attempting to connect to – and those are situations where classifying all possible behaviours as either bad or possibly good is straightforward, but these ceased being important places to look long before the paper was written in 2007.)
Opting out of email filtering may now be an even less viable option than it was a month ago.
In the decision, reiterated in the appeal, the court held that to have standing under CAN SPAM you have to show actual damages from the spam, and have to show that you tried to filter the spam out. Actual damages make some sense, but requiring filtering is just wrong, since it flips the point of CAN SPAM on its head–the only reason we have to filter is that people send us the spam that CAN SPAM presumably is intended to deter.
From the Gordon/Virtumundo ruling:
Courts must of course be careful to distinguish the ordinary costs and burdens associated with operating an Internet access service from actual harm. We expect a legitimate service provider to secure adequate bandwidth and storage capacity and take reasonable precautions, such as implementing spam filters, as part of its normal operations. Courts should take an especially hard look at the cited harm if it suspects at the outset that a plaintiff is not operating a bona fide Internet access service, as is the case here.
I’d suggest that John has this wrong, that across multiple areas of law it is taken for granted that before a citizen seeks to avail him/herself of a court’s assistance in resolving a problem that he/she has taken all reasonable (in some cases “all possible”) steps to prevent, avoid and/or resolve the problem him/herself and that this perhaps applies no less to the delivery of unsolicited email. Examples include
- the obligation of a trademark holder to vigorously defend a trademark independently of a particular court action,
- the requirement for those seeking equitable relief to have clean hands
- the requirement for those who are wronged to act swiftly.
What this implies for the running of filters is interesting (Levine again):
Haselton also explained why he doesn’t use filters: all the ones he’s tried have blocked an unacceptable amount of wanted mail, particularly since unlike most people in the US he gets a lot of mail from India and China, which spam filters tend to block. I suspect that his experience says as much about his limited ability to manage his mail system as it does about the inherent failings of filters, but he has a legitimate business reason not to filter.
My experience with BoxSentry suggests that spam filters really are typically worse at dealing with email emanating from Asia, for a number of reasons (dictionaries of bad words are generally better developed for English-speaking recipients, DNSBL maintainers tend to be in North America or Europe and so tend to correct erroneous listings in these areas more rapidly than those for senders in Asia, traffic-analysis systems tend to have more collectors deployed in Europe and North America than in Asia, etc.), however an individual who has taken upon himself the operation of an email security system might reasonably be expected to take on the burden of compensating for all of that too, which may have been John’s point.
More broadly this suggests that opting out of the use of filtering to begin with, which is what I did until joining BoxSentry a few years ago, has as an additional consequence the opting out of equitable relief as well. Those who would choose not to use a filter are vanishingly small in number, but the situation for those who choose to do so has now become just a little more unpleasant.
Rather than rehash my entire article, some fine-tuning:
Bricking a device because its software was out of date is not a reasonable failure mode.
…nor indeed, bricking a device because of a broken process for developing and testing signature updates. Stated another way, “wrong version” isn’t the only possible cause of problems in this area; automated distribution of signatures/software multiplies the impact of all kinds of bug. David Cawley drew my attention to BitDefender’s problem and suggested a possible cause: an automated process for recognising files created by malware coupled with no smoke-testing for new signature releases. In a previous life I created a signature distribution system and, amongst other things, included a step to automatically load a copy of the about-to-be-released signature set onto an instance of the target system and verified that it didn’t crash as a result. This was not quite the same scenario as that for a host-based anti-malware system (the test amounted to “does the signature set load correctly”), but the idea is the same: if you’re going to automatically distribute signatures, or software, then verifying that it will run in its likely deployment environment(s) without disrupting service is a must. Even if that’s not entirely feasible, or not feasible for all possible platform configurations, sanity checking proposed blocked files against NIST’s RDS, and indeed, your own software (apparently BitDefender was also quarantining itself) would appear to be a bare minimum level of diligence.
There is nothing in what happened that is specific to McAfee. Similar risks apply to all protective systems.
[The set of problems around automated distribution of software, signatures, etc.] almost argues for different [update] schedules for machines at different levels sensitivity and, paradoxically, for updates on more sensitive machines (those which should be behind several layers of protection already) to be applied less rapidly.
My guess is that errors of this type are going to be made again and again. Practising defence in depth, and including security vendor errors amongst the threats to defend against, remain vitally important.
(In case it’s not amazingly obvious, I do have a commercial interest in the topic of this post!)
The website covers what it is and does pretty well so I won’t repeat it here. Sadly, as with RealMail and LogiQ, only the administrators or developers of email security systems can deploy it; an end-user deployable version remains a wishlist item for the time being.
Six months ago an oversight by McAfee rendered thousands of PCs inoperative. Over the weekend a near-identical oversight by Sourcefire caused an immense number of legitimate email messages to be misclassified as malware.
I commented on the problems that arose when older versions of McAfee’s VirusScan accepted signature updates that required a newer engine version to operate correctly. In addition to pointing out that what happened was not specific to McAfee, I wrote:
Fail safe! Deal with version skew. Bricking a device because its software was out of date is not a reasonable failure mode.
My thinking behind this was that signature matching engines of any sort should (a) know their “version” and (b) have a means for an automatically-delivered signature update to state what engine version it requires. I took for granted that any engine for which these two things were true would, of course, refuse to deploy a signature update which required a newer engine version than was present. This turns out to be an incorrect assumption.
Over the weekend, ClamAV users running an older engine had a problem with this new signature misclassifying legitimate messages as malware:
Setting aside any other considerations for the moment, this is a rather special signature:
$ md5sum </dev/null
which suggests that someone at Sourcefire was putting cleverness ahead of simplicity; a rule that a particular element of a PDF file must never be empty should probably be implemented directly in the PDF parser, rather than delegated to a signature match particularly when – as appears to be the case here – the code that can use the signature is rather new.
In any event, ClamAV does have an engine-version specification and checking mechanism, as I had suggested for McAfee’s case half a year ago, but it’s only advisory!
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Local version: 0.92.1 Recommended version: 0.95.3
DON’T PANIC! Read http://www.clamav.net/support/faq
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Current functionality level = 26, recommended = 44
I suspect that this is tied up with the view that a little protection is better than none, an engine that knows to simply ignore signatures that it can’t parse and, in the early days anyway, a tradition of making signatures as specific as possible, so the false-positive risk remained negligible. The nett result of otherwise-sensible choices is a serious disruption to the very services that ClamAV is deployed to protect.
So, further to my suggestions of six months ago, add a requirement that engines should never accept signature updates that state a requirement for a newer engine than is present.
The requirement to do something sensible when such a situation is detected (prominently tell the user/admin, etc.) remains an important additional requirement for anyone using a signature-based system.
A final thought: I’m sure it won’t help, but maybe signature and update numbers containing “66” should be avoided outright; it is perhaps tempting fate, just a little bit :-)
An interesting post by Laura Atkins on a trend towards raising the bar for bulk senders.
It’s [almost] surprising that this has taken so long to occur.
“Hi! I see from your hub that you’re new to the area. Give me all your Simoleans or my friends and I will make it impossible to rent a house.”
Needless to say, the assumptions that the article as a whole works from are too simple to build a useful system from, but the Sims story illustrates on important point; the errors that beginners make in this area are causing actual harm to people and businesses. Getting workable solutions into the real world is becoming increasingly important and urgent.