Monday, March 22, 2004

Benford's Law


This is not suppose to be a mathematical discourse, but what follows is truely amazing fact. I recenly read a Sharlock Holmes book on cases involving probabilities. One simple conclusion is that 'randomness' does not imply uniformity. Meaning that if you are asked to select 10 random numbers between 1 and 50, by nature of human mind, one will always end up selecting uniformly distributed numbers. Think, isn't that true? By simple logical thinking you would not like to select some number which has already been selected because of associated small probability, and consequently end up selecting unformly separated series. So this is the way to distinguish true random numbers from imaginary random numbers. Of course, result will be probabilisticlly true, since after all, with a finite probability any given set of "random" numbers can be achieved in output of true random numbers. Overall, though, each number is equally likely and in large enough sample all qill occur uniformly.



However, and here comes the twist, if you note down first digit of any number you come across in a day - in newspaper, TV, talk, bus number, phone, label - just anything, and write it down and count the occurance of 1's,2's...9's etc., you will find - loo and behold - non uniform distribution, a direct contracdition to what I just said! Before I give explanation, lets credit this to its original founder BENFORD. He postulated that



Probability that first digit is 's' in any randomly selected number = ln(1 + 1/s)



That is digit 1 is about 2 times as likely as digit 9. Same holds if you go to a supermarket and note down first digit of prices of all items. Now the reason: Proportion changes slower initially and faster later! That is if the inflation rate is 5% constant and price of some item is 1 Rs today, than if you calculate price for next 100 years, you will find similar thing happening. It means that price digit will change from 1 to 2 in 14 years, from 2 to 3 in 8 years, ..., from 8 to 9 in 2 years, from 9 to 10(hence 1) in 1 year.



Another reason is discretization of things. That is you go to buy a packet of sweets. You would see that available packings are 50g, 100g, 250g, 500g, 1kg, 2kg, 5kg... did you find something strange? That different in packing size initially is 50 g, but we don't have packings of 1.05kg??? Hence again, small denominations are more fine tuned. Thats the nature's way of keeping things simple. I will finish with last "natural" example. Go to a bottomhill, and measure the diameters of all stones lying there. You will realise that smaller stones are more fine tuned with small difference in diameter, while the difference becomes large for large stones. Otherwise there will be too many diameters to keep track of, if the 1 meter boulder incremented by 1mm to 2 meter boulder!

Breaking the Bias – Lessons from Bayesian Statistical Perspective

Equitable and fair institutions are the foundation of modern democracies. Bias, as referring to “inclination or prejudice against one perso...