Many times a day, we are bombarded with the results of some statistical survey or study or some such, usually completely bereft of any analysis of what those results imply. In some cases, this is because the info is presented in the form of an advertisement, and the advertiser wants you to arrive at your own conclusions; he carefully words the ad to cause you to arrive at the conclusion that favors his product, regardless of whether it's actually the right conclusion. That way, he can't be sued for false advertising; he didn't tell you the wrong idea, you arrived at it of your own free will.
More often, I expect, the source presenting the statistics has no more ability to draw correct conclusions than the unsuspecting citizen. After all, the conclusions seem obvious to him as well.
Here's one of my favorite statistics: More people die in hospitals than in any other place. Does anyone doubt the validity of that statistic? I doubt it; it sounds perfectly probable to me. Does anyone conclude that hospitals are terribly dangerous places to be, and should be avoided at all costs? Of course not -- that would be drawing an incorrect conclusion from a valid study. But it only seems to take a slightly less obvious example to get people to draw all the wrong conclusions.
A common statistic: People who drop out of school don't make as much money as people who complete their high school education. I have no reason to doubt this statistic; however, it is often used as a reason to do everything in our power to make sure that kids stay in school -- including, unfortunately for them and everyone else, making the curriculum so easy a turnip could pass. But keeping those kids in school accomplishes nothing. The reason that students who drop out don't do as well is because they are losers, and compelling them to stay in school won't change that, these same students still won't do well whether they get a diploma or not. The only difference keeping them in school makes is the loss of meaning of the diploma, which used to be a sign of a non-loser, but means nothing anymore.
A brief primer on critical thinking as applied to statistics:
Let's assume that a perfectly valid, unimpeachable study has proven that whenever A occurs, B also occurs. The typical layman will immediately conclude that A causes B. However, from a strictly logical viewpoint, there are at least three perfectly plausible conclusions:
Another example: There are lots of studies that relate guns to crime. So, do we conclude that guns cause crime? There sure seem to be a lot of people that feel that way; just get rid of the guns, the crime will go away. I hate to be the bearer of bad news, but I've got a sneaking suspicion that crime causes guns. Or, perhaps, our worthless justice system causes both crime and guns. In either case, you'll notice, doing away with the guns won't do a thing for the crime; the criminals will just find other ways to rob and kill people.
While there may be several possible conclusions from a valid study, there are far more reasons to suspect that many studies being done are far from valid. Was the study done correctly? Did it sample enough individuals to provide reasonable confidence in the conclusions? Did it really prove a significant correlation?
The medical profession is guilty of making bad assumptions along these lines on almost a daily basis. They must be forgiven to a certain extent, since they are not able to hold to the strict analytical guidelines required of most of the sciences; if they get some data indicating that a certain substance is likely to kill you, they are not about to set up a double-blind study and kill off a statistically significant number of citizens to prove beyond doubt that, yes, the stuff will kill you.
In fact, the medical profession considers it their moral responsibility to jump to conclusions. If they get even one indication that a particular activity may be hazardous to your health, it would be unethical for them to keep that a secret; such information should be dispersed immediately, so that any individuals who wish to may take appropriate actions to change their ways.
But, only too often, that early indication and the advice based upon it take on the status of gospel, and society's entire way of eating, or exercising, or living in general is changed forever -- and then later indications contradict the earlier study. For decades, we were advised to eat selections from the four major food groups -- but if you watched closely, the four major food groups changed several times! One of those food groups when I was young said to eat generous helpings of red meat daily; now the stuff is practically considered poison. I wish I had kept one of those posters. Now, they have finally ditched the four food groups altogether and gone to a food "pyramid", but I have no confidence it will be any more permanent; the medical profession's credibility is dangerously low.
Often, reading the actual report on a study will clarify many of the shortcomings of the methodology used -- statisticians generally try to be honest that way -- but nobody points them out in the 30-second TV ad. They just tell you that the study says that the product is good stuff.
Some studies make glaring errors. If A and B both happen together, it might be just coincidence -- and the typical study doesn't even attempt to address this issue. If A happens 98% of the time all by itself, and B happens 96% of the time all by itself, then of course most of the time A happens, B also happens. So what?
One good reason to doubt studies is because the source is biased. Does anyone really believe studies done by tobacco companies that show that tobacco isn't harmful? Of course not -- at least, nobody that isn't addicted to tobacco. But the fact is, a lot of studies are every bit as biased and every bit as deserving of being ignored.
Take statistics on drug abuse. I mean any statistics on drug abuse, at least in the US. Drug abuse is illegal, how could you possibly expect to gather accurate data on it? The drug abusers aren't gonna give honest answers. Those who oppose drug abuse aren't gonna give honest answers. Think about this: could you gather enough data to get an accurate picture of the amount of drug abuse going on in your own neighborhood? Frankly, many of you might have trouble coming up with good numbers for drug abuse in your own house. The idea that any national statistics are worth the paper they're printed on is simply ludicrous.
Here's another of my favorites: Some significant percentage of people who try marijuana go on to harder drugs. If we assume this is true, think about the reasons that might explain it. Of course, whenever you hear it, you're expected to draw the obvious conclusion: there's something about marijuana that makes you wanna take harder drugs. This, of course, is bull, and there are a remarkable number of Americans that know it from personal experience, including Bill Clinton. So, let's try another conclusion: Marijuana use is illegal, and harder drugs are also illegal, so any person who tries marijuana is already a person who doesn't care about the law. If a person doesn't care about the law, he might as well try hard drugs as well -- in other words, he's the type of guy who's likely to try hard drugs even if marijuana didn't exist. This is an example of where you're expected to conclude that A (marijuana use) causes B (hard drug use), when the real relationship is that both A and B are caused by C (user doesn't care about the law). In this case, the reason the statistic shows a relationship between marijuana and hard drugs is the fact that marijuana is illegal; if it were legalized, a lot more people might try it (those who do care about the law), but no more people would try harder drugs, and the statistical tie-in would go away. Ironically, this particular statistic is often used to support laws against marijuana use.
But let's try another reason: Marijuana use is not only illegal, but we have heard for decades from our government how harmful it is. The problem is, it's not harmful; actual scientific tests haven't even been able to show that it causes lung cancer! Almost every unbiased source agrees it's less harmful to the human body than either tobacco or alcohol, and it also appears to be less harmful to society than either. So, when a person tries it for himself, he finds out that the government has been lying to him. Why should he now believe this same government when they tell him that crack cocaine, heroin, LSD, and other drugs are harmful? Their credibility is zero. There are people around him using these hard drugs, and they seem to be having a good time. Sure, perhaps a high percentage of those who try marijuana will go on to harder drugs -- and it's the government's fault!
Of course, there is no good reason to believe the statistic anyway. Just where do you suppose they came up with it? Did they ask people on the street "Did you ever smoke marijuana? Did it cause you to wanna shoot some heroin?" Not likely. More likely, they talked to people that ended up in the rehab clinics and hospitals from overdoses of really dangerous drugs, and they ask them "How did you get started?" and the dopeheads answer, "Well, when I was a little kid I did some pot..." and, of course, these airheads conclude that what they are looking at is the results of everybody who has ever smoked a joint. Why? Because everyone they ask who didn't end up in the emergency room answers "Me? No, I never smoked any weed." The stuff is illegal, you cannot expect honest answers.
Believe it or not, a fairly good reason to doubt many studies is that the researchers set out to answer a question. This is considered good science; the researcher is supposed to form a hypothesis ("Does A cause B?") and then formulate a procedure for determining the correctness of that hypothesis. The problem is, he may be all wet, completely off track, barking up the wrong tree, way out in left field. B is actually caused by Q, which hasn't even occurred to the researcher, which means he doesn't even have any questions on his survey about Q, he hasn't measured the incidence of Q, he has failed to even notice Q when he walked past it in the lab. Now, if the researcher had started with an open-ended question ("What causes B?"), perhaps someone along the line would have pointed out the possibility that Q had something to do with it. But, believe it or not, starting studies with open-ended questions is not considered good practice. So, the researcher focuses on A, and since a lot of the people he's asking to fill out his survey seem to think it makes sense that A causes B for the same reasons that the researcher himself came up with that hypothesis, they happily answer the questions regarding A the way the researcher expects them to -- and you end up being told that "Studies confirm that A causes B."
Try this example: The researcher formulates the hypothesis that people turn to crime because of a bad upbringing. So, he waddles on down to the prison and starts interviewing convicts: "What was your childhood like?" The convicts, of course, happily tell all kinds of horror stories about how their dads got drunk every night and beat them. The researcher comes away with ample evidence to show, beyond a doubt, that a lousy homelife during the formative years is the real cause of crime in society today. Did he bother to find out whether the criminals really had all that bad a childhood? Did he bother to find out if other people with similar childhoods came out just fine? What would have happened if he had just gone in there and asked "Why did you rob that liquor store?" There are all kinds of conclusions he might have come to, including that crime was caused by high unemployment, alcoholism, the minimum wage being too low, whatever. He might even have arrived at the correct conclusion: the criminals think they can get away with it.
My recommendation to anyone and everyone who lives in this modern age: learn to think critically. In the case of statistics being thrown at you, think about a couple of things:
Return to Kirby Palm's opinions page.
Return to Kirby Palm's home page.
Of course, if you have questions or comments, you are welcome to send e-mail to me at palmk@nettally.com.