Fuzzier Logic

“Peer review does not guarantee quality”

I am still catching up on my podcast backlog after my 2 week holiday in August. The excellent ‘More or Less’ provided the gem of a quote in the title during a discussion about meta-analyses.

Professor Stephen Senn was explaining why careless mathematics can distort the results of a meta-analysis (things like including a prior meta-analysis amongst your data sets can lead to double-counting – see this paper). The presenter, Tim Harford, suggested that surely this is a problem easily fixed. A reader spots an error in a published meta-analysis, contacts the journal and a correction ensues. A suggestion that was quickly knocked back by Prof Senn. The problem, as he sees it, is that we have no culture of correction; that peer reviewed results are considered irreproachable.

Doesn’t peer review offer some guarantee of quality?, suggests Harford. “Peer review is of minimal value” is the response to this, “…checkability is what really guarantees quality”. Senn goes on to suggest that scientists sign an undertaking to provide raw original data to anyone who requests it.

This was the clearest argument I’ve heard, not against peer review, but for the availability of raw data, and for post-publication quality control on a grand scale.

This multi-eyes approach to quality checking, post-publication, is familiar from somewhere…

The same edition of the show had a section on data visualisation, and bought the ‘Napoleon’s March’ graphic to my attention. I had not previously been aware of this ‘infographic’, produced in the mid-19th century.

Tags: podcast, publishing, science, statistics

Nightmare awakenings

Yesterday, a Cold War script written for broadcast in the event of a nuclear attack was published. The script was penned in the 70s by the BBC and the government, and was released by the National Archives.

Most mornings I am woken by my radio alarm clock, which I turn off by hitting the radio on, this radio is tuned to Five Live, since my alarm goes off on the half hour I like to lie and listen to the news before getting up (if I haven’t been rudely awakened by my son before this that is). Yesterday morning, Five Live was dealing with the release of this nuclear warning script by broadcasting it, in full, voiced by a very grave sounding announcer. They were doing this at the precise time my alarm went off.

Made for quite a sudden awakening, I can tell you.

Tags: BBC, personal, Radio, trivia