Lost in Reception

One of my bugbears when talking to customers about message loss through spam filter false-positive errors is that most email security vendors understate their false-positive rates by about an order of magnitude. Terry has noticed this too:

The industry cheats quite a bit with their SLAs, the language is deliberately ambiguous.  If a company claims a 1 in 25,000 false positive SLA, what that means is that they permit 1 false positive per 25,000 messages.  This means that if the spam/ham ratio is 10:1, then in 25,000 messages there will be 2272 hams and 22,728 spam messages.  If one of the good messages is flagged as spam, then the good mail FP rate is 1/2272 = 0.04%, which is actually quite high.  Yet by saying that you permit 1 in 25,000 messages, and messages is not defined but assumed to be both spam + non-spam, vendors have permitted themselves a lot of leeway when calculating how accurate their product is against good mail… by a factor of 10.