A life that doesn’t quite align with the norm often comes with a dose of skepticism and push-back against what is generally accepted. This isn’t always a bad thing. There have been far too many instances of government, health organizations, companies, and individuals getting things wrong even when they should have been trustworthy.
I’m not exception to this. I am a huge skeptic of the USDA’s food pyramid and guidelines for healthy eating, for instance. I’m wary of the production methods for and potential food-supply repercussions of GMO foods. I don’t accept the safety of many of the chemicals used in large-scale agriculture, I’m not blind to the problem of animals raised for meat in confined feeding operations, and I would always prefer to find a root cause and/or a natural treatment rather than pop a pill that treats symptoms despite believing my doctor is well-trained and probably wants to do right by me as her patient.
However, I do think that skepticism can go too far. I often see people so skeptical of anything that government, Big Agra, or Big Pharma does, that they won’t accept even the most scientifically backed and logically sound defenses of certain practices, regulations, products, etc. I know these people are generally good people trying to do their best with the information they believe they know, but I also often see them employing logical fallacies or unsound criteria when they come to their conclusions.
I’d like to talk to you for a few minutes about coming to sound conclusions based on good logic and quality evidence through trustworthy sources. Why? Well, if you’re trying to convince someone to go organic with their garden and you sound like a nut-job, or if your life decisions are endangering other people’s lives or well-being, you’re going to lose credibility. Perhaps your conclusions were right even though your reasoning was terrible, but now your neighbor or friend won’t listen to the next person who comes along with credible information because you made something into a conspiracy theory.
Evaluating Scientific Evidence
I mentioned that I don’t trust the USDA’s guidelines for healthy eating. Why? Research.
I’m not talking about iffy websites that lack credibility, obscure and unrepeatable studies, or books that lack absolutely any backing from other sources.
I’m talking about reading books by scientific journalists that survey dozens of studies, the history of how the health recommendations came to be, etc. I’m talking about reading up on biology and how our bodies utilize different nutrients. I’m talking evaluating studies and whether the conclusions in the headlines reflect the reality of the study’s meaning and application.
How does one research well, then? Specifically, how does one evaluate evidence presented to them?
I’m going to lean heavily on this article and give you a quick breakdown of types and quality of evidence.
- Sources that lack quality altogether–The evidence is shaky at best and only gets things right by accident, and that only rarely.
- Anecdotes–Stories used to support a belief. Anecdotes have their place to some extent, but don’t necessarily establish real cause-and-effect or a universal experience. These are usually just emotional stories to support someone’s foregone conclusion if not accompanied or supported by other quality evidence.
- A false authority–Some experts are reliable, and others aren’t. There are a good handful of ways to help identify real experts in the article linked above.
- Media and press releases–Sometimes the conclusions of a study or of research are released, but before peer review is done or with misleading conclusions presented. You know, “A glass of wine is equivalent to an hour at the gym.” Check the real evidence behind what your friend posted on Facebook.
- Meetings and presentations–Even in scientific conventions, these presentations are often for new ideas that still are lacking in research or peer review.
- Animal and cell studies–What happens in an animal or to a small group of cultured cells may not apply to what happens in an actual human body. These studies are good indicators for further research and such, but don’t always definitively prove anything.
- Case reports–These can give a “heads up” for further study or give an example of a real-life happening, usually in medicine, but generally only focus on a very small sample and rarely actually prove anything on their own.
- Cross-sectional studies–These give very limited information and are not controlled for confounding factors. They are usually simply used to determine how much of a population has a given disease at a specific time.
- Case-controlled studies–People with a disease are compared to those without. This is getting into quality evidence, but there are limitations.
- Cohort (retrospective) studies–A sample is followed prospectively and subsequent evaluations are made. Usually thousands of people are followed over a long period of time to determine disease outcomes based on certain factors. These are often used to prove that a certain correlation is not a cause.
- Small randomized controlled trials (or double-blind trials)–The small size can make the results questionable, but randomized controlled trials provide some of the most reliable evidence. The outcome should repeatable in larger groups to accept the conclusion of a small randomized controlled trials.
- Large randomized control trials–Some of the best evidence available comes through these. The studies are double-blind, the people are places into the groups randomly, and the results tend to be definitive and repeatable.
- Systematic reviews and meta analysis–These survey all of the existing research on a topic, assessing each item for its quality and weighing the conclusions of each appropriately. A meta analysis pools available data for a large analysis.
It’s also appropriate to say something about journal publications. Not all journals are created equal, and the peer review offered by the high-quality scientific journals far outstrip that of the lower-quality, lesser-known journals. Saying a scientist, doctor, researcher, etc. has published in a journal does not automatically make the article or its author reliable.
Accepting Quality Evidence
Too often people will dismiss a study based on faulty criteria.
For example, a common one is the source of funding for a study. That can be quite valid, but not always. For example, a person will say, “The government funded it so I don’t trust it.” This is usually because the person has already decided that the government has an agenda in the area being studied and therefore anything funded by the government is to be completely dismissed.
However, it’s not that simple. For instance, is the study high quality (large randomized controlled trial, for example)? Were the methods used in the study sound? Were procedures followed correctly? Does the data of the study match the presented conclusions? Is the study repeatable? Has it been repeated with the same results in other high-quality studies? If the answer is yes to all of these, then the source of funding detracts little, if at all, from the conclusions. If the answer is no, especially to multiple questions, then the study starts to become questionable. Simply following the money doesn’t tell you everything about the trustworthiness of a study. It can tell you something, but not everything.
If a soda company funds a study and claims the study shows that soda is totally healthy, it’s probably a bad study. Why? Well, there is plenty of quality evidence proving that soda is unhealthy. The methods used or other aspects of the study are likely questionable. It becomes safe to say that, based on these other factors, the soda company likely created a study designed to come to a desired conclusion to sell their product. On the other hand, if a soda company funds a high-quality study and comes to the same conclusions that other high-quality studies have come to, the company is probably not going to trumpet the study to the media, but now the fact that the soda company funded the study doesn’t mean you can’t trust it.
A logical fallacy is a flaw in reasoning or in the presentation of an argument. Instead of trying to explain the fallacies to you, I’d encourage you to browse through this beautiful gem of a website, https://yourlogicalfallacyis.com/
Sometimes a quick internet search is all it takes to turn up the flaws in someone’s newest health trend or the latest conspiracy theory. You find a source that’s fairly credible — a website that cites quality sources such as studies or presents clear logical arguments, for instance — and voila! you now know better. Google certainly has its place in debunking the latest idea on alkalizing water or the cure for cancer that Big Pharma is hiding from the population.
I hope this gave you some greater insight on fact-checking and thinking twice about your conclusions. Please, be skeptical in the sense that you don’t automatically accept everything you hear. Please, be careful in how you present your arguments for or against something. Please, don’t make our population dumber than it already can be these days.