Lost in Reception

The loss in confidence in email as a communication channel is beginning to spread to other channels as well.

To date, most concern – my own included – about mail loss through spam-filter false-positives has been about the harm done to people and organisations by this loss, and the decrease in confidence that these losses cause in email as a communication channel. It seems that the amount of email/SMS scamming going on world-wide is starting to cause people to question all communication claiming to be from an authority figure. While I welcome any decrease in the gullibility of populations at large – it is this very gullibility that has given spammers free reign to begin with – organisations that have grown up in an environment where citizens uncritically trust communication appearing to be from an authority may find their ability to function critically impaired in populations where widespread scamming causes a rapid decrease in acceptance of claims of authority and where those organisations don’t have an immediately obvious way of proving to citizens that they are who they say they are.

An interesting article by Dan Blacharski on problems in Korea notes exactly this. Upon being contacted by a police officer advising cancellation of a payment to a fraudster, the now-suspicious victim replied:

“Dirty swindler! If you’re a policeman, I’m your grandfather!”

The ultimate solution:

[government authorities] must establish a safe protocol for communicating with citizens when it is necessary to ensure legitimacy

which is much easier said than done. Finally, this nugget:

Some Korean police departments are sending a written summons before making a phone call.

which doesn’t really address the problem; convincing people to trust people who assert their identity twice will simply cause scammers to assert their identity twice too.

Virus Bulletin is gearing up to perform regular tests of anti-spam systems as an addition to their existing coverage of anti-virus systems.
They’ve performed their first trial and published anonymised results that are interesting, to say the least. They report false positive rates for the submitted systems falling between 0.04% and 0.4%. Unfortunately:

Following industry standards, the false positive rate, or ‘FP rate’, is the ratio of the number of false positives relative to the total number of emails.

Correctly determining what numbers to use when calculating a false positive rate requires a rudimentary knowledge of statistics. Early in spam’s history (when it was <1% of all email), getting the denominator wrong (“all messages” rather than “all legitimate messages”) made very little difference. Now that spam is closer to 95% of all email – and still climbing – this small error makes an enormous difference. At some stage vendors chose to report false-positive rates calculated over total message volume, rather than legitimate message volume, because it made the figures a little better. Sadly, there’s no convenient time for a vendor to undo that practice, even when the difference now makes the published figures more or less meaningless.

According to VB:

During this period, the filters saw a total of 20,764 emails, 877 of which were classified as ham by VB’s employees (the recipients)

Changing the denominator from 20 764 to 877 multiplies the result by 20 764 / 877 ~= 23.676. This is a pretty large correction; the “correct” figures for VB’s trial run are therefore 0.95% – 9.5%.

But, does it matter? Numbers like 0.04% look so small that they can be disregarded, but once it’s clear that this number means the loss of 1 in 100 legitimate messages, a potential customer is likely to see the number as being rather more important. Pity the vendor with the 0.4% result; 1 legitimate message in 10 will go astray!

I gather that VB intends to publish both the statistically sound false-positive rates and the industry-standard “nice” rates. I’m hoping that they do so.