webofevil | (no subject)

China and Iran are criticised for how much the state controls information. How different is it in this country?

It's less different than we’d like to think. For example, a group of computer programmers was trying to get hold of Hansard, the parliamentary record. They asked for it politely, they didn’t get access to it. They ended up scraping it off the web. The parliamentary officials couldn't stand this; they thought they should have a right to control who had access to this information. These computer programmers had a huge battle to get access to this supposedly public data so we could see how our MPs voted and when our MPs had attended debates.

Interview with Heather Brooke, Guardian

I really haven’t the faintest idea what she’s talking about, and by the sound of it neither does she. The entirety of Hansard is on the web so that absolutely anyone at any time can access it—or “scrape it off”, as she puts it—and search it for exactly the information that she’s describing. Whatever battle these “computer programmers” faced either involved some whole other information that isn’t a matter of public record or resulted from massive confusion on their part.

Another option is that the programmers demanded expensive hard copies stretching back years, which no-one was going to be arsed to provide them with, and didn’t know that this information was freely available elsewhere. Still, that’s where an investigative reporter might be expected to shine.

I actually admire Heather Brooke for the part she played in unearthing the expenses scandal but holy shit, if this is indicative of the competence of her investigations, maybe all of that needn’t have taken five years after all.

Flat | Top-Level Comments Only

What she's saying is they weren't given access to the raw data in some easily machine-readable format. Instead, they had to do a lot of programming to convert from the final Hansard HTML back into a proper database (this is what she means by "screenscraping"). I know the TheyWorkForYou people and this is accurate.

I'm pretty sure they didn't demand hard copies.

Obviously this isn't exactly Chinese levels of secrecy, because Hansard is publically accessible, but it takes a lot more work to figure out how your MP voted from Hansard than from TheyWorkForYou, to the extent that most people wouldn't bother.

I think the reason the Hansard IT people were reluctant to release the information to TWFY is because it would show up the poor quality of their programming and website design, rather than due to pressure from above.

Edited 2010-04-19 21:56 (UTC)

Exactly, MySociety went through so much crap getting that set up, and for all the wrong reasons. There're still MPs that want their sites shut down.