Fuzzier Logic

“Peer review does not guarantee quality”

I am still catching up on my podcast backlog after my 2 week holiday in August. The excellent ‘More or Less’ provided the gem of a quote in the title during a discussion about meta-analyses.

Professor Stephen Senn was explaining why careless mathematics can distort the results of a meta-analysis (things like including a prior meta-analysis amongst your data sets can lead to double-counting – see this paper). The presenter, Tim Harford, suggested that surely this is a problem easily fixed. A reader spots an error in a published meta-analysis, contacts the journal and a correction ensues. A suggestion that was quickly knocked back by Prof Senn. The problem, as he sees it, is that we have no culture of correction; that peer reviewed results are considered irreproachable.

Doesn’t peer review offer some guarantee of quality?, suggests Harford. “Peer review is of minimal value” is the response to this, “…checkability is what really guarantees quality”. Senn goes on to suggest that scientists sign an undertaking to provide raw original data to anyone who requests it.

This was the clearest argument I’ve heard, not against peer review, but for the availability of raw data, and for post-publication quality control on a grand scale.

This multi-eyes approach to quality checking, post-publication, is familiar from somewhere…

The same edition of the show had a section on data visualisation, and bought the ‘Napoleon’s March’ graphic to my attention. I had not previously been aware of this ‘infographic’, produced in the mid-19th century.

Tags: podcast, publishing, science, statistics

Randomness, statistics and understanding

So here I am, sitting in a statistics workshop, having finished all the exercises ahead of time, musing on how much easier all this stuff is once you understand where it all comes from. This made me think that I have found this workshop more understandable and simpler to tackle because I have pretty much finished reading a marvellous little book called ‘The Drunkard’s Walk’ by Leonard Mlodinow.

Mlodinow aims to educate the reader about randomness and statistics, by way of history and illustrative example, and he succeeds admirably. The book is a walk through mathematics from the Greeks and Romans, by way of the renaissance, to Einstein and the modern day. Each important advance toward the modern day study of statistics is illustrated with excellent examples and anecdotes, many of them personal to the author. The Monty Hall problem, the anomoly of Jeanne Calment, who reverse-mortgaged her apartment to a 47 year old lawyer when she was 90, only to outlive him (and he died aged 77), even the author’s own (false) positive AIDS test makes for an intriguing case study, and illustrates the importance of understanding prior probabilities when reporting the results of a test.

The setting of all this stuff in context has really helped my brain with the basic concepts, and even without this current course, I feel like I’ve got a much better grip on statistics in general. A remarkable claim for a popular science book. I look forward to the remaining 30 or so pages.

Tags: book, randomness, Review, statistics