Kristin Conlin; Mario Rautner; and Mike Caulfield
Addressing confirmation bias and cognitive bias through the lens of a journalist or an investigator provides a framework for understanding how humans seek information and how we process that information to create understanding. The following content is from the Tactical Tech Collective’s Exposing the Invisible Tool Kit.
Investigations are only as strong as the evidence you collect for them. Without evidence to support your work (in school, this could be a paper, journal entry, larger project or presentation, or simply a position you take in a class discussion), your claims or statements have little substance and will not convince your audience.
It is therefore important for investigators to understand the type of evidence they can gather, the reliability and strength of the sources from which evidence is collected and when the combination of evidence is strong enough for the investigation to be turned into a story, a report, a claim, a campaign or made public in any other way, if that’s your final aim.
Developing a hypothesis
A hypothesis is “a supposition or proposed explanation made on the basis of limited evidence as a starting point for further investigation”. Using a hypothesis will not only give your investigation focus, it will also help guide you through the evidence that you need to collect and to assess information you’ve already gathered. For more clarity, we can use the term “working hypothesis” since it implies a work in progress, which evolves and may change based on the evidence you find (or do not find).
In his co-authored book “Story-Based Inquiry”, Professor Mark Lee Hunter, a founding member of the Global Investigative Journalism Network (GIJN) argues that using hypotheses is the core of the investigative method and that developing a hypothesis gives you something to verify, which means you can assess your evidence against it. This in turn will help you structure and plan your investigation. Such structured research will make it much more likely that you will apply techniques that will lead you to important evidence.
It is also imperative to recognize that your starting hypothesis is not set in stone. It can, and should, change as you discover and interpret new evidence. At times, a starting hypothesis might even look irrational, speculative or ungrounded but as you find more proof, you should be able to adjust and expand it. Not doing so means you risk falling into the trap of wrong theories or conspiracy theories – although even supporters of such theories are able to support their hypotheses with some form of evidence or interpretations of it. This can create mis– or dis– information which will be discussed later.
Hypothetical example
You may have received an anonymous tip-off that a tannery (a leather processing plant) is dumping toxic waste into a river in your neighborhood, killing fish. First, you quickly check a few facts, including applicable regulations for effluent releases and the location of various factories in relation to the river on satellite. You also talk to some residents and one of them reports seeing dead fish in the river recently. At this point you do not know if there is any truth to the information you received, but because there are a number of companies located in the vicinity of the river, you develop an initial hypothesis:
- “Someone is illegally releasing chemicals into the river.”
This example is used throughout this chapter.
Pause a moment and identify an idea you have or a lived experience, then create a hypothesis from that experience.
What is proof and what if there is no proof?
Types of evidence and searching for proof
Once you have adopted a working hypothesis, it will guide you through your investigation and evidence collection (research). There are many different types of evidence (source types and content within those sources) that you might come across in the course of your investigation. Evidence may be direct or indirect, obtained through field research (primary source acquisition) or desktop work (research and analysis of the sources you collect), and it can be physical or digital. Most evidence falls into more than one of these categories. These differentiations matter when it to comes to interpreting your evidence and ascertaining its strength and reliability. Often the type of evidence is linked to the type of investigation you are carrying out. For instance, field investigation tends to result in physical evidence, such as environmental samples, or in video and photographic evidence.
Direct vs indirect evidence
Direct evidence (primary sources) usually speaks straight to your hypothesis and establishes facts.[1] Examples for this include video, audio and photographic evidence. Eyewitness accounts of events, official documents and records are also considered direct evidence.
Indirect evidence (secondary sources), on the other hand, is usually evidence that supports an assertion but is not strong enough and mostly second-hand and therefore not able to establish facts to the same degree. Examples of this include statements by people who did not observe events directly, statements by spokespersons for others and generally any kind of evidence that requires you to make inferences. An inference is an opinion or conclusion you form based on evidence that does not secondary sourcesdirectly speak to that conclusion. We make inferences every day: for example, when you see a parcel in front of your door you presume it is for you without looking at the direct evidence, the label on the package. This is only strong enough to support a hypothesis if there is an accumulation of indirect evidence and when no other logical conclusion is possible. Most of the time when you come across indirect evidence it will be necessary to carry out further investigations and to verify it with other sources whenever possible.
Example
It is important to keep in mind that even direct evidence may only confirm a specific component of your hypothesis rather than the whole story.
In our example of water pollution, a report from a laboratory that shows toxic synthetic chemicals in a river only establishes the fact of the river having been polluted. It does not provide any information about who caused the pollution, nor does it necessarily show the exact spot on the river where the pollution originally occurred. Examples of indirect evidence with this case study may be an observation by an eyewitness of discoloured water or the discovery of dead fish. None of these examples establishes or disproves whether there is pollution in the river since discolouration and fish mortality can occur from natural causes and a company spokesperson is not able to provide direct evidence.
It is important to keep in mind that what is indirect evidence for one statement or hypothesis, can be direct evidence for another. If a company spokesperson says that “our company does not discharge toxic chemicals into the river,” this can be considered only indirect evidence of the company in fact not causing pollution. It is, however, direct evidence for establishing that the company denies having caused the river pollution.
Human vs physical vs digital evidence
Evidence can originate from a wide range of sources which can be broadly categorized into:
- human
- physical
- digital
Evidence from human sources can come from anyone who is broadly associated with or touched by your investigations (research), even if just on the periphery. For instance, this can include eyewitnesses, victims, government officials, scientists, spokespeople, journalists or experts. Ultimately, digital and physical evidence are always linked and influenced on some level by human evidence and sources, since it is always originally developed, created or analyzed by humans. We’ll discuss this more in the next section on assessing evidence, but for now it is worth keeping in mind that videos and photos are taken by people, that environmental samples are analyzed by machines and computers controlled and operated by people and that documents are written and edited by people. People are subjective and have points of views or positions that may affect the evidence they produce.
Physical evidence broadly applies to paper documents, books, magazines, contracts, invoices, environmental samples, some photos and videos and more. Even though our world is increasingly becoming digital, physical evidence is important and is generally more trustworthy than most of its digital counterparts since it cannot be manipulated as easily. Since the digital and physical worlds are increasingly integrating, the boundaries between physical and digital evidence are also blurring. Only twenty years ago, videos and photos would have been considered physical evidence but recent photographic and video evidence is now strictly speaking digital, as it originates from digital cameras. This is one reason why assessing the origin and strength of evidence and the sources that provide it has become so important. A similar trend occurs for instance with documents that are increasingly delivered digitally. Only a few years ago a Freedom of Information request might have yielded hundreds of pages of physical documents. Now it is much more likely that the government’s response to your request includes PDF documents and digital records. In many ways, the conversion to digital media and the associated ease with which information can be altered poses an important challenge for investigators.
Because of the blurring of these boundaries and the rapid move towards digital document provision and storage, we are using a narrow and somewhat fluid definition of digital evidence in the context of this guide. Here, digital evidence only applies to information that is found on the internet or in a digital file sharing system. This includes online articles, reports and studies, digital databases, videos and photos. Photos and videos straight from the cameras and in their raw formats are considered physical evidence here.
Note however that with some effort nearly everything in the digital world can be manipulated including images and video. The metadata of photographs that you have not taken yourself can be very easily altered and should not be relied on as evidence, unless you can verify and confirm it from its original source(s).
Evidence from desk and field research
Evidence from desktop research includes anything you can uncover with your computer, such as researching scientific or newspaper articles, information from databases, even some video, audio and photographic evidence. Desktop evidence also includes anything that is emailed to you via Freedom of Information requests or through online conversations.
Field research requires access to primary sources which may or may not always be possible. It can be gathered through a range of activities such as interviews and evidence collection in libraries and archives.
Analyzing evidence
In order to understand to what extent the evidence you’ve gathered might constitute proof, it is necessary to analyze its strength. Every piece of information or evidence comes with risks and may raise further questions. Therefore, the ability of investigators to question the reliability and strength of the evidence gathered becomes vital. This skepticism towards one’s own opinions and biases is a fundamental characteristic of honest/fair investigators.
Strength & accuracy of sources
When interpreting evidence, it is crucial to understand the reliability and motivations of the source of the information. This means that the same piece of information can be considered strong or weak depending on its origin and the context in which it was provided. It is up to the investigator to interpret the evidence in combination with its origin and source to ascertain its strength.
Some evidence has already undergone at least some sort of review process that makes it generally stronger and more trustworthy.
- Scientific articles for instance have generally gone through peer reviews, although one should be wary of pre-prints that have been published before completion of reviews. Special attention needs to be given to the funding sources of such articles as well, since many companies commission research that may turn favorable to their interests.
- Reports by respectable media organizations, even if they exist entirely in the digital world, are often reviewed internally by fact checkers. That does not mean you shouldn’t double-check their sources and data, as their information may not always be correct. Checking their affiliation and codes of conduct is also a good way of establishing some trust. For instance, commercial and non-profit media organizations that are part of the International Fact Checking Network or the Global Investigative Journalism Network (GIJN) may have systems in place that result in more accuracy.
- The less corrupt and more transparent a government, the more the information they release can be trusted. However, overall national, state and local government information alone should rarely be considered strong enough to contradict your hypothesis on its own.
It is a well known phenomenon that humans often tend to believe what they want to believe and – whether we like it or not – investigators are not immune to such cognitive bias.
Note
Even scientists who try to use replicable methodologies for experiments that can be repeated in order to obtain the same results are not objective in all cases. This potential lack of objectivity can influence the outcome of their work. Author Angela Saini wrote that “Scientists who imagine that bias lies in others, not themselves, fail to recognize that to live in the world today is to be drip-fed assumptions and prejudices that guide our thoughts and actions.”
When bias exists for scientists, who generally apply the most objective methods to their work, then this is surely the case for investigators, who generally deal with subjects that do not follow a strict scientific methodology. Understanding the role that investigators themselves play in the outcome of their investigations is just as important. Not doing so carries the risk of including inaccurate information, or misleading readers to incorrect conclusions. This is why, when in doubt, it’s better to presume that you do not have enough proof for your hypothesis.
Cognitive bias
Cognitive bias is a broad term that refers to the way our judgement and decision making is influenced by the context and framing of information and our personal beliefs. Cognitive bias was first defined through psychological research in the 1970s but people have been aware of the phenomenon for hundreds of years.
Cognitive bias can have serious consequences.
An awareness of bias is particularly important for citizen investigators, and citizens in general, because they often work alone or in a small group, frequently have personal or emotional attachments to the outcome of their investigation and can lack review processes that are common within professional non-profit organizations and the media.
- One of the most relevant cognitive biases for investigators is confirmation bias, in which we favor ideas that confirm our existing beliefs and what we think we know. As a result, people might favor specific scientific evidence or other data over contradictory evidence and also come to broad conclusions aligned with one’s bias. Investigators, journalists, and researchers are not immune from this bias.
- Another important bias to be aware of is the sunk-cost effect. This is defined as a tendency to continue an endeavor or pursuit once an investment in it has been made. This investment can be monetary but it can also be time and effort. For investigators this is relevant, for instance, if evidence that disproves a hypothesis could easily be ignored especially if it is uncovered after a lot of effort has already been spent on the investigation.
There are many other forms of cognitive bias that can easily influence our ability to carry out investigations objectively.
- Availability bias: when people give more credence to solutions and information they remember first, which is often the most recent. This can be a fallacy in investigations, in particular for those that stretch over long periods of time.
- Anchoring bias: is nearly the opposite of availability bias. Here people are more likely to believe the initial piece of information that therefore serves as an anchor against which future information and evidence is compared.
The good news is that awareness of one’s biases, the application of specific methods and tools to avoid it and training can all reduce the extent to which cognitive bias affects our decision making and thinking. Some of the techniques that help reduce the risk of bias include:
- Seeking second opinions of evidence interpretation by colleagues and other investigators
- Reviewing your conclusion after some time has passed and
- Making an effort to disprove conclusions and evidence interpretation
Mitigating information risks and evidence verification
There are numerous actions you can take once you realize that your evidence may not provide enough proof to support your hypothesis. For instance, a rule that is often applied in journalism requires at least three sources that are independent from each other to confirm the same information before it can be published.
Analyzing competing hypotheses
One of the best ways to exclude at last some of your bias and tendency to want to support what you believe is to try to gather evidence that disproves your hypothesis. Testing for alternative explanations of your assertion throughout your investigation will help you to be more objective. A technical term for this is an Analysis of Competing Hypothesis (ACH) which is a technique often used by intelligence services. In this process the investigator first generates possible hypotheses, then creates a matrix which is filled with evidence that is then tested for logic and consistency. The hypotheses are then evaluated and ranked.
There may be investigations in which you collect evidence that contradicts or even disproves your hypothesis. It is important to be objective and open-minded when you analyze your evidence. This is not always easy, especially if you are personally invested in your hypothesis.
The only ethical and acceptable action to take when you uncover facts and strong evidence that contradicts your working hypothesis is to accept that your initial hypothesis is wrong. Then, you either make a new/alternative one that is able to account for the new facts as objectively as possible, or accept that your current investigation will not be able to prove what you thought it would, and end your investigation. Oftentimes, this situation is more complex than just a prove-disprove end result, so it is worth spending some time evaluating your initial questions and investigation process including your methodology and tools, the evidence you have collected and any possible alternatives you might have at this point.
To avoid confirmation bias:
- Avoid asking questions that imply a certain answer. If you ask, “Did the Holocaust happen?,” for example, it is implied that the Holocaust was faked. If you want information on the Holocaust, sometimes it’s better just to start with a simple noun search, e.g. “Holocaust,” and read summaries that show how we know what happened.
- Avoid using terms that imply a certain answer. As an example, if you query, “Women 72 cents on the dollar” you’ll likely get articles that tell you women make 72 cents on the dollar. But if you search for “Women 80 cents on the dollar” you’ll get articles that say women make 80 cents on the dollar. Searching for general articles on the “wage gap” might be a better choice.
- Avoid culturally loaded terms. As an example, the term “black-on-white crime” is term used by white supremacist groups, but is not a term generally used by sociologists, nor do statistics support this claim. As such, if you put that term into the Google search bar, you are going to get some sites that will carry the perspective of white supremacist sites, and be lousy sources of serious sociological analysis.
* If you don’t know if a term or phrase is culturally loaded, pay special attention to and practice the following two steps:- Think carefully about what constitutes an authoritative source before you search. Use that criteria to assess your search results. Then, once you acquire search results, use lateral reading techniques to assess the sources you chose to pursue/click on.
- Scan results for better terms. Maybe your first question about whether the holocaust happened turned up a lousy result set in general but did pop up a Wikipedia article on Holocaust denialism. Use that term (in this case, “denialism”) to make a better search for what you actually want to know.
- Robyn's noted in a 2001 publication that we must constantly reiterate [that primary sources], are the subjective interpretations of another person’s observation of an event or activity. Not surprisingly, therefore, many professional historians have written that it is their duty to approach primary sources with a healthy skepticism in the research process. Robyns, Marcus C. “The Archivist as Educator: Integrating Critical Thinking Skills into Historical Research Methods Instruction.” American Archivist 64, no. 2 (Fall/Winter 2001): 363–384. https://doi.org/10.17723/aarc.64.2.q4742x2324j10457. ↵
A cognitive bias towards confirmation of the hypothesis under study. Seeking sources that confirm a hypothesis rather than gathering information to learn more about a given topic.
systematic error in thinking that occurs when people are processing and interpreting information in the world around them and affects the decisions and judgements that they make (see: https://www.verywellmind.com/about-us.)
A wide range of information sources and investigation results that that in totality court to support or contradict an investigation hypothesis. Direct evidence establishes the main fact immediately while indirect evidence establishes facts from which the main fact can be inferred.
Something that is supposed; an assumption made to account for known facts, conjecture.
material that contains firsthand accounts of events and that was created contemporaneous to those events or later recalled by an eyewitness. From Society of American Archivists.
A work that is not based on direct observation of or evidence directly associated with the subject, but instead relies on sources of information, such as a primary source. Secondary sources can also be understood as a work commenting on another work (primary sources), such as reviews, criticism, and commentaries.
Information that describes properties of a file, be it image, document, sound recording, map etc. For example the contents of an image are the visible elements in it, while the date the image was taken, the location and device it was taken on, are called metadata.
Continue an endeavor or pursuit once an investment in it has been made.
