Centuries-old techniques developed by Thomas Bayes find modern applications because they are simple and effective.
Centuries-old techniques developed by Thomas Bayes find modern applications because they are simple and effective. (1) Bayesian techniques are integrated into automated categorization of unwanted spam, the formation of medical diagnoses, virus detection, and several other uses that advance compatible business objectives.
Bayes Theorem uses Conditional Probability to calculate the probability of A given B, provided that the probability of B given A, the probability of A and the probability of B are all known.
For example, suppose an automated program could determine that a particular phrase is present in 70% of spam and 50% of non-spam emails, and that an email is 90% likely to be spam, and suppose that A means "the email is spam", B means "the email is not spam", and C means "the phrase is present". Then P(A), the probability of A, is 90%. P(B) is 10%. P(C|A), the probability of C given that A is true, is 70%. P(C|B) is 50%. P(C) = (0.70 * 0.90) + (0.50 * 0.10) = 0.68 = 68%. Some useful probabilities for classifying the email would then be P(A|C), the probability that the email is spam given that the phrase is present, and P(B|C), the probability that the email is not spam given that the phrase is present. Using conditional probability, P(A|C) = P(A) * P(C|A) / P(C). P(A|C) = 0.90 * 0.70 / 0.68 ~= 0.92647 ~= 93%. P(B|C) = P(B) * P(C|B) / P(C) = 0.10 * 0.50 / 0.68 ~= 0.07353 ~= 7%. Therefore, the probability that the email is spam, based on that one data point, is 93%, and the probability that it is not spam is 7%.
To improve the accuracy of this technique, a computer program could analyse thousands of data points in less than a tenth of a second, which is approximately how long it takes to download an email message. The program could test for phrase D, phrase E, phrase F, etc., and use data about each one to modify the overall confidence that the email is spam or not spam. In hand-waving mathematical terms, if C, D and E are found to be true, the program can automatically determine P(A|C,D,E) (the probability that the email is spam given that C, D, and E are true) using an extended version of Bayes Theorem, applied to a bayesian network. By using good tokens (i.e. by asking the right questions, using all available information), this technique can be up to 99.5% accurate with 0.3% false positives.
A related algorithm is state-based, so that rather than using a directed acyclic graph of conditional probabilities (a bayesian network), one could automatically produce a set of known states, along with transition matrices (stochastic matrices) showing the probability of moving from a given state to another (transition probabilities). These stochastic matrices fall naturally from statistics coming from a large enough set. CRM114 uses this technique to improve spam detection accuracy beyond naive bayesian techniques, and suggests innovative approaches to potentially bring 99.999% reliability.
The applications of statistical techniques such as described above go beyond spam detection. For instance, one could imagine the same or similar techniques being used in the medical field to classify DNA, in industry to properly direct calls that come through automated call systems, in elections to predict how a particular message may affect polls, etc.
To learn more about how statistical methods can be used to improve your business, please contact us.
Works Cited
- J J O'Connor and E F Robertson. "Thomas Bayes." From The MacTutor History of Mathematics archive.
- William S. Yerazunis. "The Spam Filtering Plateau at 99.9% Accuracy and How to Get Past It." From CRM114 - the Controllable Regex Mutilator.
- Eric W. Weisstein. "Bayes' Theorem." From MathWorld--A Wolfram Web Resource.
- Eric W. Weisstein. "Conditional Probability." From MathWorld--A Wolfram Web Resource.
- Paul Graham. "Better Bayesian Filtering." From Paul Graham.
