The volume of data is [...] ]]> The UK Government is running a competition offering participants up to £20,000 if they can create new uses for existing free sources of government data by combining the data into new and useful information (a mash-up). You can find a link to the data here.

The volume of data is immense. I started trawling the UK census from 2001, burrowing down to the small village I live in. Of the 4700 or so residents, there are three buddhists but disappointingly no Jedi (there’s an urban myth that Jedi is put down by so many people who have no specific religious persuasion). Apparently there are no 1-room properties, so no studio flats and no basement flats (no-one lives below ground level). 2.4% of houses have no central heating and only 5.5% of the population are not in good health.

I’m not sure what this indicates other than don’t try selling central heating or life insurance policies where I live! But being less facetious, the power in this data will be in deriving new value. I can see two immediate uses/methods.

Firstly, most of this data is useful to people looking for new places to live; schools information, services information, crime levels and so on. So, develop a website and put a postcode in to see how your prospective area rates.

Second, trawl the data and find the ends of the spectrum – the good and bad, best and worst of each metric. For example, which area has the highest population density? Which has the worst crime? This information could be great for business planning; don’t set up a locksmiths in an area with the lowest crime rate and so on.

Of course the hardest part will not be to correlate different data sources but in bringing together a consistent view of the information. Some data is accessed via APIs, some by XML, some in Excel format. What “common” point of reference can be used? Postcode? Address?

Whatever, the availability of more and more data content will be absolutely invaluable, but for me, I’d like to see more real time information to be mashed up. For instance, at the airport a live XML feed of flight arrivals/departures that I can read in the taxi when I’m running late (I don’t want to have to log onto their website); same for the train; a feed showing my nearest tube station or restaurant as I travel; a feed of the waiting time at all the Disnet rides, so I can pick the shortest queue without having to find a status board; I’m sure there are many many more.

Rich data is a wonderful thing; bring it on!

This strikes me as an unbelievably stupid plan. In the first place, every UK citizen will be baulking at this incredible intrusion into civil liberties. Second, it is [...] ]]> The BBC reported today that the UK government is planning a tracking database for all phone calls made and emails sent in the UK.

This strikes me as an unbelievably stupid plan. In the first place, every UK citizen will be baulking at this incredible intrusion into civil liberties. Second, it is highly unlikely that the government could ever deliver such a database based on their previous track record with developing and deploying large scale IT projects – think of the UK tax system and new NHS IT system (which will apparently be 4 years behind schedule). Will this database contain all the content of the calls or just a list of who called/emailed who?

Assuming the former, let’s do some “back of the fag packet” calculations…

According to the Mobile Operators Association, there were 70 million mobile subscribers at the end of 2006, with an average 100 minutes per user per month and 12 text messages per week. That’s 84,000,000,000 minutes and 43,680,000,000 text messages in a 12 month period, the timescale the database is expected to hold data for.

Using the following forum posting referring to Skype, a conservative estimate of bandwidth is 30kb/s or about 225KB per minute. A text message is a maximum of 160 characters. This means the government database would need a 12 month capacity of only 6.36TB to store the text messages (assuming no database or filesystem overhead) but a whopping 17.6PB of storage to hold the voice calls. Let’s assume those calls are made consistently over the course of a year, then the system would need to ingest about 600MB/s of data.

Now, I would imagine even the UK government wouldn’t be stupid enough to attempt to store the content of all those calls (I didn’t even attempt to calculate the email traffic). If they don’t, let’s face it, you have to question what the point is, if the content isn’t being recorded as anyone with any nefarious intent will simply use anonymous pay-as-you-go SIM cards and bypass all the tracking. Still, at an efficiency of say, 40%, the 44PB potentially needed could set up the lucky EMC or HDS salesperson for life! I wonder if any of them have done the same calculation as me….?