Today for a while, when a plane passed overhead (they do often where I am), the bulb on my desk lamp dimmed. I, of course, assumed the two events were related.  The fact is, planes passed over every couple of minutes, and the light only dimmed every half hour, and I’ve just now found it’s because I was kicking the cable under the table without knowing it.  They’re unconnected…

That’s what psychologists call an illusory correlation – the false connection of two things, based on data.   (it’s also a tongue-twister).

Sod’s law (Murphy’s Law) is a example – we tend to connect negative events, and ignore positive (or neutral) ones.  How often have you been driving along, only to be confronted at the top of a hill and round a bend with a truck that’s halfway across the road?  “Always happens at the top of a hill and round a bend, typical!” you’ll think.  Obviously, 99% of the time it doesn’t, but we’ll remember the times it does.

So why is this important?  Well, it usually isn’t, because we muddle along anyway.  It can get odd when unexplained events (lights in the sky) are connected with unconfirmed causes (UFOs from outer space).  Or when “there’s no smoke without fire”, which has probably convicted a fair number of innocent people. 

My interest is because at my company, Cognitive Match (of which Favy is now a part) we’re focussed on ways of making REAL connections in observed data.  And equally I guess uncovering the “illusory” ones…

Of course, there will be a long tail of this stuff which is never seen by more than the person who created it, and at the top end there will be a small percentage that are viewed a lot.  Sure enough, a wildcard type search (searching for “*”, if that’s valid) turns up the top video with 101 million views (a music video, like a lot in the top results of that list)…  Wikipedia’s got some notes on the “heavy tail” distribution, which I’m guessing is what this is.

The important take out is that very quickly (by which I mean already) there’s too much content on Youtube for one person to make sense of.  And therefore ways of pre-selecting, filtering and locating stuff of interest – like in every area online now – are needed, beyond just search…

- how happy and productive you are
- your social status
- your social group

Fundamentally this is just an extension of any form of data mining – take a large amount of data, and try and make some determinations from it.  The examples based on social group and status can be fairly easily explained – by where you spend your time (the types of shop, street, district), and other mobile phones that yours tends to hang out with.  Happiness and productivity was a correlation they discovered when they combined location and call data with questionnaires.

The same group are doing some interesting work with other areas that use mobile data – such as “social serendipity” – trying to match users that happen to be in similar locations, and that have similar profiles or interests.  People have tried to release products into that space for as long as I can remember, but no-one’s yet cracked it, so it will be interesting to see if this research helps.

A lot of reality mining to date has been to do with mobiles (like BlueTooth MyBlogLog), but obviously anything that can sense us and feed data about us will add to this: cars, PCs, toasters… The more, to my mind, the merrier.

While we’re pushing out big numbers, here are some more to goggle at… They’re not sourced (some of them are estimated, some might be wildly out – but all were spotted on fairly reputable sites).

• 1.4 billion internet users
• 50 billion videos viewed online in February
• 3.3 billion searches on Baidu per month
• 500 million videos on YouTube
• 4.1 billion photos on Facebook
• 2 billion images on Flickr
• 533 million results for “insurance” search
• 10 million articles on Wikipedia
• 3 billion songs sold on iTunes
• 100 million MySpace members
• 10 million songs scrobbled on last.fm a day

Pretty overwhelming, huh.