This was something that troubled me for a while: why people’s first-hand experiences are so often put aside, especially in science, as not sufficient to prove something?
In other words, why shouldn’t anecdotes be taken as evidence?
When I started to read about the how science really works (see my previous post on this, Getting science), I was a bit uneasy about why first-hand knowledge could be so casually cast aside.
When you hear people talk about their experiences, those events weight a lot on their beliefs: you know someone who got over cancer using prayer; someone saw a UFO or believes she was abducted by aliens; you believe that homeopathy is true because it works for you…
The list is endless, we can always include our experiences, large and small, into our world of personal belief, and they often weight heavily on that belief, overshadowing and trivializing all other evidence to the contrary.
Why then should we be wary of anecdotes?
1- For one, we have to recognise that we attach a lot of meaning to them because they are experiences, meaning that they happened to us or someone we trust enough to be judged as our own experience.
Those experiences hide the bigger picture though: we attach more meaning to something that happened close to us than everything else that happened to everybody else: we see what’s close to us and this blinds us to the contrary experiences that happened to others.
2- A strong experience, one that has a big impact on our life, will re-enforce our belief, making us more confident about us being right: if you tell me that smoking cuts 10 years of your life but I know of someone who smoked all her life and is 110 year old, then the anecdote, that you can live long and smoke, will certainly have more strength on my appreciation of what smoking does.
In the same class of examples, most people have experienced that nothing happened to them after they drove while being drunk: they did not crash or felt they drove dangerously, re-enforcing the belief that they do not pose a risk.
Yet another example, and one that is being much exploited, is that of winning the lottery: million of people play, re-enforced in they belief that they will win it because someone invariably does.
3- We are extremely bad at grasping the meaning of numbers. Statistics and evaluating numbers are extremely unnatural to our brain. We did not evolve to easily deal with that sort of complex mathematical concepts and it requires real effort to get into the mindset necessary for objectively being able to assess statistical relevance.
That is tightly tied in with the concept above that something that happens to contradict objective evidence is usually given more credence because it may be closer to us, and relies on actual experiences, a thing that we, human, tend to value above ‘meaningless’ numbers.
4- An anecdote can re-enforces an existing belief: if you look for anecdotes that seem to validate your existing belief, these anecdotes will make your belief stronger, appearing to give credence to your world-view.
In other words, we remember the hits and forget the misses.
We are often selective about what we actually accept as evidence for or against something in view of our existing belief about the subject.
Usually, we will accept anecdotes for our belief as being additional proof and anecdotes against our belief as being simply wrong, insufficient evidence or untrustworthy.
We tend to be uncritical when an anecdote confirms our belief, and become critical when it denies it.
5- A variation of this is when scientists publish positive findings but almost never publish articles to say they’ve failed. Yet failure teaches us as much as success: if one scientist finds that a particular homeopathic remedy cures cold and publishes it but 10 others did similar research that showed no effect and do not publish their findings, guess which will get the most attention?
6- The main argument against anecdotes though is a simple one to grasp: imagine that I would want to promote cigarettes as something inoffensive for your health.
We know the scientific evidence against smoking is great, yet I could easily find a thousand, a million or 100 millions people for whom cigarette had no real adverse effect.
Those numbers are really large and could not be ignored. If I was to make a petition with all those people they would make a very persuasive argument that cigarette is actually harmless.
Yet, we know that isn’t true. All those witnesses to the contrary would just be anecdotes, they may be valid experiences in themselves but they cannot be used to reach a definitive conclusion.
To have a clear picture of a subject, you have to study the whole population involved.
Anecdotes have a lot of persuasive power, but they may amount to no evidence at all when you look at the whole picture.
What we know about cigarette is that 1 smoker 2 will die because of it. That leaves a lot of people for whom cigarette will have no life-threatening effect (apart from yellow teeth, parched skin, stinking clothes and very bad breath).
Someone who is very religious may believe in the strength of prayer as a way to heal. He may even have first-hand evidence that it works: he was sick, prayed, and got better.
His experience tells him that prayer works.
What may actually have happened is less certain.
A lot of diseases are not 100% fatal, and among those that are, there is usually a varying percentage of remissions that are not explained. They are not explained because we don’t know everything about every particular individual cases: the disease could have been misdiagnosed, it could be of an unidentified strain that is slightly different from the usually fatal one; it could have been cured simply because the body was strong enough to fight it, either through a change of lifestyle or particular chemical reactions; etc.
To really know what happen you would need to know and make sense of all possible relevant information and be able to have total control over the course of the evolution of the disease.
Of course, this is almost always impossible, which mean that there is enough room for unforeseen, unexpected things to happen, like a ‘miraculous’ remission.
Because there is no sufficiently informed reason to explain the facts, it’s easy to label the remission as being a result of prayer, or anything else that the sick person was believing would help him.
As a parenthesis, the strange thing is that miraculous remission is viewed as a result of faith in a belief while the contrary, un-expected death, is never viewed as a consequence of that same belief: praying people who survive believe they were saved by their faith yet praying people who die never blame their faith for it.
Reeks of double-standard to me.
Strangely enough, the fact that there is no evidence to make a conclusive prognosis becomes evidence in favour of the belief, and it becomes an anecdote that re-enforces the believer.
Overall though, these anecdotes are not sufficient: there is absolutely no hard evidence -beyond the anecdotes themselves- to prove that prayer or most alternative remedies have any positive impact, yet many will be influenced by the anecdote only and believe it to be true and sufficient to validate the belief.
In science, there is also the issue of reproducibility: an experience has to be reproducible to have confirmatory power.
Anecdotes are usually unexplained and impossible to reproduce, making them hard to confirm.
In most fields of inquiry proving something usually involves lots of experiments and the statistical analysis of their results.
In statistics, results are not always consistent: some will prove your theory, others won’t, but what’s important is how many will prove it compared to those that do not.
Wide ranges of results are often expected, yet no-one would focus on just the few that seem to disprove the experiment unless they show real significance beyond the calculated margins of error.
Anecdotes are like single uncontrolled experiments whose result we would extrapolate to a conclusion that we consider definitive.
That would be wrong: no-one would -and should- consider basing a conclusion on a single isolated fact or even a collection of isolated facts.
Anything that can only be analysed through statistics will contain in its studied population odd results. There are lots of reasons for those odd results, most of them are to do with the difficulty to control everything about an experiment: your equipment introduces errors, the observer introduces errors, some experiments may have been improperly prepared; there are so many things that can go wrong and are beyond our total control.
Unless the odd results indicate a clear and identifiable trend that means something, they are just part of the process, part of the background noise that we expect to have: they’re just a manifestation of chaos.
When someone tells you with forceful conviction that something is true because it happened to them don’t denigrate their experience: what they feel may seem true to them, but their interpretation and expansion of that experience as sufficient proof that they are right actually -very likely- may be wrong.
If the only African you have ever seen was an albinos, would you be right to conclude that all Africans are white?
If you take a homeopathic remedy to cure your cold and you eventually get better is that sufficient proof that homeopathy works or it is just proof that you don’t know enough about your condition and about how diseases work to actually make a better judgement?
I’m shooting at homeopathy here, but I could include most remedies, whether they are proven or not: because you get something to feel better and you end-up feeling better isn’t proof that what you took actually has any relevant effect at all.
Without adequate statistical evidence to prove effectiveness of a particular medicine your experience is just an anecdote, an uncontrolled single experiment.
