Lost in Reception

Terry writes a reasoned post on the consequence of the use by users of “This is Spam” buttons for no-longer-wanted newsletters, but then makes on odd assertion:

The one drawback of this feature is that the sender is blissfully unaware that their messages are not getting through to the end user.  However, that’s a topic to be dealt with by the email-deliverability folks, I’m in the business of impeding mail flow!

I have a longer post coming on deciding-at-SMTP-receipt time what to do with a message. Some of the arguments are finely balanced but this one is a slam dunk: if the recipient has indicated that they don’t wish to receive messages from a specified sender and that sender sends to that recipient again, then the message should be refused at RCPT TO (i.e. before the message even crosses the wire). This makes clear to the sender that this address is not to be sent to again and does so in a way which reduces resource consumption in the anti-spam system; it’s an actual free lunch!

Oh, and Terry, if you’re really in the business of impeding mail flow, you could save yourself a lot of trouble by simply turning off your servers :-) I’d suggest that you’re actually in the business of facilitating the flow of legitimate mail and that impeding the flow of abusive mail is merely a means to that end.

I wrote:

I gather that VB intends to publish both the statistically sound false-positive rates and the industry-standard “nice” rates. I’m hoping that they do so.

Well, my wish was granted! (you’ll need to register)

For each product they stated both “FP rate” and “FP of total mail corpus”. Once various special cases (see below) have been removed, FP rates of 1-3% were observed. Like any other actual measurement, these numbers are not necessarily representative of anything – and these may have been worsened by VB’s particular experiment setup – however they are at the upper end of what BoxSentry is seeing each time we perform an analysis.

Before commenting on the specifics, I’d like to address what I see as an error in VB’s report. Anyone who has completed high-school physics will have dealt with measurement and the need to discard erroneous measurements before reaching conclusions. The most common reason for choosing to do this is the observation of results that lie so far outside the norm that they are clearly incorrect. (Scenario: you’re analysing data on [human] child birth, one of your subjects has a recorded birth weight of 6125kg. This is clearly an error and should be removed from your data before calculating average birth weights.) This occurred with one of VB’s subjects. The report says:

a false positive rate of over 25% of all ham messages is
almost certainly a sign of product misconfiguration

While I feel a certain degree of schadenfreude at this product’s results, I am inclined to agree with this assessment and therefore feel that VB’s results would have been more sound had this product been removed from the results entirely. The decision to keep the rules constant for any complete run of the tests is a good idea, but not the extent of producing obviously incorrect results.

I’d also be inclined to discount the ClamAV and SpamAssassin results because, in both cases, the spam-catch rate was too low to be useful: the false negatives (spam reaching the inbox) were a considerable multiple of the true negatives (legitimate email reaching the inbox).

This leaves just three products. Despite VB’s careful disclaimers, their results are largely what I’d expected, albeit at the high end of the range. Let us hope that these results aren’t enough to discourage vendor participation in future trials.