Critical Thinking

07.10.2025·52min

Part 1: The Architecture of Thought

If you go to YouTube right now and type “critical thinking” into the search bar, you are going to see a mess. You won’t just find educational content; you’ll find videos of people screaming at each other, insulting intelligence, and engaging in bad-faith arguments. That isn’t what we are here to do. We need to strip away the internet noise and get down to the brass tacks of what it actually means to think critically, especially as engineers and technical professionals.

Defining the Undefinable

At its core, critical thinking is the fundamental willingness to question things and get to the bottom of them. It is a dual-stack system: you need a critical attitude (the mindset) and specific cognitive skills (the toolset). These skills include raising the right questions, conducting independent research, analyzing data, evaluating sources, and integrating that information to reach a well-founded, explainable judgment.

That sounds like a solid definition, but it’s a bit of a “list of ingredients” without the recipe. In the educational sciences, this is a known problem. There are countless studies dedicated to figuring out if you can actually teach critical thinking. The results? Honestly, they are modest. Some researchers have even concluded that critical thinking is less of a method you learn and apply, and more of a personality trait or an attitude you adopt.

To make this concrete, let’s look at the framework provided by Otto Kruse, a psychologist and writing researcher who led the Centre for Academic Writing at the University of Zurich. He breaks it down into four distinct buckets:

Self-directed and self-aligned thinking: This refers to metacognitive abilities — thinking about your thinking. It involves maintaining intellectual autonomy and adhering to quality criteria for your own thoughts.
Rational procedure: This is crucial when things get messy. How do you handle ill-defined problems or genuine ignorance? You need reflected, methodical thinking, not just guessing.
Skeptical thinking: This is the critical reflection and testing of all knowledge assumptions. It implies a high awareness of error and deception. You have to assume you could be wrong.
Habits of thought: This treats critical thinking as a personality feature — a careful, deliberate handling of facts and knowledge as a default mode of operation.

While one might not agree with every single point Kruse makes, his positioning of critical thinking as a central educational goal is vital. He argues that this is the core of higher education and a service universities provide to society. Kruse also dropped a line that is incredibly relevant for us: “Writing is the royal road to learning to think.” As computer scientists, we write a staggering amount. We just happen to write in languages designed to give instructions to machines rather than communicate with humans. Despite that difference, the ability to read with deep comprehension (reading skills) correlates strongly with programming performance.

The Pocket Guide to Rigor

Let’s get granular. If we want to practice critical thinking, we need a standard to measure our thoughts against. There is a concept known as the “Pocket Guide to Critical Thinking” that provides a checklist of essential properties. If you are debugging a complex system or arguing a system architecture, these are your metrics.

Clarity A statement must be clear and unambiguous. In computer science, we suffer from this constantly: everyone in the meeting nods and agrees, but everyone is silently thinking of a completely different implementation. This “silent disagreement” happens because of unstated ambiguity. You need to ask: Can this thought be worked out better? Can we give an example? Can we visualize exactly what is meant?

Correctness Here is the trap: a statement can be perfectly clear but completely false. For example, “Most dogs weigh more than 150 kg.” That is a grammatically correct, unambiguous, crystal-clear statement. It is also factually wrong. The key here is verification. How can we check this? How do we find out if it is true? How do we test it?

Precision Vagueness is the enemy. If I say, “Peter is overweight,” what does that mean? Is he 1 kg over his BMI target? 20 kg? 80 kg? We don’t know because the precision is missing. We need to ask: Can we address this more specifically? Can we add details? Can we express this numerically?

Relevance You can have a statement that is clear, correct, and precise, but completely useless. Relevance forces us to ask: How does this connect to the problem? How does it impact the question at hand? Does this actually move the needle?

Depth This is where many technical arguments fail. A statement can meet all the previous criteria but be superficial. Consider this statistic: “95% of all heroin addicts have also smoked cannabis.” This is likely clear, correct, precise, and relevant to a drug discussion. But it is superficial and superfluous because it fails to reflect the complexity of the situation (correlation vs. causation). Deep thinking asks: What factors make this problem difficult? Where are the complex aspects? What are the hard questions we are avoiding?

Networking (Breadth) An argument can check every box above and still only represent a slice of the truth. For example, “Reducing three lanes to two on the Getreidemarkt will cause massive traffic jams.” That might be true from a traffic flow perspective. But we must view it from other angles. Do we need to consider other standpoints (urban planning, environmental impact, pedestrian safety)? Do we need a new approach to the problem entirely?

Logic Does the whole thing make sense? Do the beginning and the end fit together seamlessly? The constituent parts must align without contradiction.

Focus Arguments often drift. Are we dealing with the most important problem? Is the central idea actually in the focus of interest, or are we bike-shedding on minor details? Which of the presented facts are actually decisive?

Fairness Finally, have we considered the viewpoints of others? Fairness demands an active effort to understand opposing positions and incorporate them into our thinking. Is there a hidden self-interest? Are we appreciating the viewpoints of dissenters constructively?

Practicing this list is incredibly difficult. It is worth noting that in today’s discourse, “critical thinking” of this nature is sometimes dismissed or branded as “woke” (whatever that term is supposed to mean in this context). Do not be discouraged by that. Ultimately, critical thinking is the only thing that moves us forward.

The Model of Thinking

To understand why we fail at the standards listed above, we need a model. We aren’t trying to build a biological model of the brain here — we are building a conceptual model of thinking.

Models are simplified summaries and reflections of what we observe. They aren’t necessarily “true” or “false” in a binary sense; they are useful or not useful. If a model explains our observations without breaking established knowledge, it works. It gives us a vocabulary. However, models are always incomplete. We need to know where the model ends. Our model here does not distinguish much between perception and thinking. The moment light hits the retina and stimulates sensory cells, we are “thinking” in this framework.

A common, but flawed, model is the Stream of Consciousness. We imagine ourselves standing on the bank of a river, watching thoughts flow by, picking out the ones we like based on creativity or logic. The problem with this model is that it ignores the massive amount of unconscious processing and decision-making that happens upstream. It doesn’t explain where the thoughts come from.

We need a more inclusive, “monolithic” model. Our model posits that thinking is not a single stream, but the sum of actions and interactions between many Components of Thinking.

The Fast vs. Slow Components Thinking consists of many interacting parts. Some are fast, some are slow. Some are conscious, some are hidden.

Imagine someone throws a ball at you unexpectedly. You react. You throw your hands up to deflect or catch it. This is a stunning computational feat. If you tried to use your “rational thinking” (the conscious component) to calculate the trajectory, solve the differential equations, and move your muscles, you would get hit in the face long before you finished the first calculation. Even for robots, catching a ball is an extraordinary task.

In that split second, your rational thinking steps aside and lets other components — fast, automated, physical components — take the lead. Whether you catch it depends on your practice, but the reaction happens without “you” (your conscious self) doing it.

We see this interference in other areas. Have you ever tried to carry a glass of water filled to the very brim? If you stare at the water and consciously try not to spill it, it becomes incredibly difficult. Your hand shakes. But if you look away and let your body handle it, you often spill less. When the conscious, rational mind gets involved in tasks it isn’t suited for, it becomes a bottleneck.

The Inner Game of Tennis

There is a fascinating example of this in Timothy Gallwey’s book, The Inner Game of Tennis. Gallwey claims you can teach competent tennis in a very short time by bypassing the conscious mind. He argues that the part of the thinking process that reflects and over-analyzes often gets in the way of the parts that actually execute the movement.

In 1973, Harry Reasoner, a TV host, invited Gallwey to prove this on The Reasoner Report. Gallwey took a woman who was completely unathletic and had never played tennis. In 20 minutes, he had her playing decent tennis.

How? He kept her conscious mind busy with trivial tasks (like saying “bounce” when the ball hit the ground and “hit” when it hit the racket) so it couldn’t critique her form. By distracting the “over-critical” component, the other components of her thinking — the ones that handle spatial awareness and motor control — could work without interference.

The participant described it perfectly: “Every time I did start to think, things went wrong. But if I just stop thinking, the body seems to know what to do… All of a sudden, everything became effortless.”

This reveals a critical flaw in our architecture: The components of our thinking do not always work together productively. They can stand in each other’s way. This is “overthinking.” It is when the rational mind tells you that you can’t ride a bike, even though your body knows you can.

The Cognitive Miser

Neurologically, we know that conscious thinking is a tiny fraction of brain activity — estimates range from 1% to 10%. We have a “Fast System” (the choir of fast components) and a “Slow System” (the conscious, rational mind).

Here is the problem: The slow components require a tremendous amount of energy. The fast components are cheap and efficient.

Evolution has subjected us to stringent selection pressure. As Richerson & Boyd (2005) put it: “…all animals are under stringent selection pressure to be as stupid as they can get away with.”

Humans are Cognitive Misers. Our brain constantly tries to outsource work to the fast, low-energy system to save the expensive, high-energy conscious system for when it’s absolutely necessary. We replace complex questions with simpler ones. We rely on heuristics (shortcuts).

This is why we have Cognitive Biases. They aren’t random glitches; they are the result of our brain trying to save power. The fast components rush forward with an answer (“It’s the availability heuristic!”), and the lazy, slow conscious mind says, “Sounds good to me,” and accepts it without verification.

This is why Critical Thinking feels so hard and why people resist it. It forces you to fight your own biology. It demands that you wake up the “cognitive miser,” spend the energy, and override the impulse to just go with the flow.

In the next part, we will open up the “Cognitive Bias Cheat Sheet” and systematically dismantle the four specific categories of errors that our energy-saving brains force us to make every single day.

Part 2: The Cognitive Miser & The Bias Cheat Sheet

We established in the previous section that the human brain operates on a dual-system architecture. You have the fast, automated components that handle ball-catching and instant reactions, and you have the slow, deliberate conscious mind that handles complex logic. The critical insight here is that your brain is fundamentally lazy. In evolutionary biology, we call this being a cognitive miser. The slow, conscious system is metabolically expensive to run, so the brain constantly tries to outsource cognitive load to the cheaper, faster system. It uses heuristics — mental shortcuts — to solve complex problems with minimal energy. When these shortcuts work, we call it intuition. When they fail, we call them cognitive biases.

This isn’t a random glitch; it is a feature. We are under stringent selection pressure to be only as smart as we need to be to survive. The result is that we don’t make rational decisions based on all available data; we make pragmatic decisions based on what is available and easy to process. To understand exactly how this breaks down, we can look at the taxonomy created by John Manoogian and Buster Benson, often called the Cognitive Bias Cheat Sheet. They categorized hundreds of biases into four distinct problems the brain is trying to solve: dealing with too much information, lack of meaning, the need to act fast, and the limits of memory.

Problem 1: Too Much Information

The world bombards us with more sensory data than we can possibly process. To function, our brain has to aggressively filter this noise. It doesn’t filter randomly; it filters based on what it already knows or what is most prominent. This filtering process is the source of our first category of errors.

A classic example here is the availability heuristic. We judge the frequency or importance of an event based on how easily we can recall an example of it. If you see constant news reports about a specific type of crime, you will believe that crime rate is skyrocketing, even if statistics show it is plummeting. A study from 1993 on the US “War on Drugs” showed this clearly: students believed drug use was rising because of intense media coverage, even though the actual data showed a decline. In our daily work, this manifests when we overestimate the difficulty of a refactoring task simply because the current, messy code structure is more “available” in our minds than the clean, theoretical solution.

This filtering mechanism also leads to the confirmation bias, perhaps the most dangerous of them all. Once we adopt an opinion, we act like a filter that only lets in supporting evidence. We accept confirming data as high-quality facts and dismiss contradictory data as noise or exceptions. Francis Bacon identified this back in 1620, noting that human understanding forces everything else to add support and agreement to its adopted opinions. This is the engine behind modern “filter bubbles.” We are not just passively receiving information; we are actively constructing a reality that reinforces what we already believe.

We also see this in how we perceive value and choices. The anchoring effect describes how we use the first piece of information we see as a reference point for everything that follows. If you see a price tag of $2000 crossed out next to a price of $1000, the TV seems cheap. If you just saw $1000, it might seem expensive. Similarly, the framing effect changes our decision based on how the data is presented. We prefer a product labeled “95% fat-free” over one labeled “5% fat,” even though they are identical. We are also subject to inattentional blindness. When we focus hard on one thing — like counting passes in a basketball game — we can completely miss massive, obvious anomalies, like a person in a gorilla suit walking through the frame. This is known as the Monkey Business Illusion.

Problem 2: Not Enough Meaning

The world is often fragmented and confusing. We see isolated data points, but we crave narrative. We need the world to make sense, so our brain connects the dots, filling in the gaps to create a coherent story. We hallucinate patterns where there are none because a false pattern feels safer than chaos.

The most famous example of this in an engineering context is survivorship bias. During World War II, the US military wanted to reinforce their bombers to reduce casualties. They analyzed the planes returning from missions and plotted the location of every bullet hole. The damage was concentrated on the wings, the body, and the tail. The engineers initially decided to reinforce those areas.

It took a statistician named Abraham Wald to stop them. He pointed out that they were only looking at the survivors. The planes that were hit in the cockpit, the engines, or the fuel tanks didn’t come back to be counted. The bullet holes on the returning planes showed where a plane could be hit and still survive. The data they had was biased by the survival of the aircraft. They needed to reinforce the empty spaces on their charts, not the filled ones. In the tech industry, we see this when we try to emulate the traits of successful startups (the survivors) while ignoring the thousands of failed companies that had the exact same traits but died quietly.

We also struggle with the difference between correlation and causation. Just because two things happen together doesn’t mean one caused the other. Our brains love to invent a causal link because it allows us to predict the future, but often these links are spurious. We also deal with the pro-innovation bias, where our optimism about a new technology blinds us to its downsides, and the anecdotal fallacy, where we trust a single personal story (“My grandmother smoked until she was 90”) over statistical data.

Problem 3: Need to Act Fast

Evolutionarily speaking, hesitation is death. If a predator attacks, you don’t have time to conduct a comprehensive risk analysis. You need to act. To facilitate this, the brain prioritizes speed over accuracy, leading to biases that push us toward immediate action or preserving the status quo.

One of the most persistent issues here is the sunk cost fallacy. We tend to stick with a losing decision simply because we have already invested time, money, or effort into it. In software projects, this leads to throwing good money after bad, refusing to scrap a flawed architecture because we have already spent two months building it. The rational move is to look only at future utility, but the fast brain wants to validate the past effort.

We also overvalue things we have created ourselves, a phenomenon known as the IKEA effect. Studies show that people value a product they assembled much higher than a pre-assembled version, even if their craftsmanship is mediocre. In coding, this leads to the “Not Invented Here” syndrome, where we prefer our own buggy custom tools over superior standard libraries.

This drive for speed and confidence leads to the overconfidence bias. We systematically overestimate our own knowledge and competence. In spelling tests, people who claim to be 100% certain of an answer are often wrong a significant portion of the time. This hubris is a root cause of major catastrophes, from Chernobyl to the Deepwater Horizon oil spill, and more recently the Titan submersible implosion. We mistake familiarity with safety.

Finally, we have the default bias. We view any change as a potential cost or risk, so we stick to the default option. If a software setting is pre-selected, the vast majority of users will never change it. This is why the default settings in privacy policies or software installers are so powerful; they effectively dictate user behavior because our brains are trying to conserve the energy required to make a change.

Problem 4: What Should We Remember?

We do not have infinite storage space. We can’t remember every pixel of our lives, so our brain compresses information. It saves the gist, the generalizations, and the high-level concepts, discarding the specifics. This compression is lossy and biased.

We tend to remember the first thing we hear in a series (the primacy effect) and the negative things that happen to us (the negativity bias). Evolutionarily, remembering that a red berry made you vomit is more important than remembering a beautiful sunset. We also succumb to the Google effect, or cognitive offloading. If we know we can look information up later, we don’t bother to store it in our long-term memory. We use the internet as an external hard drive. While this is efficient, it can compromise deep learning. If you offload the processing of a problem to a tool, you might never internalize the underlying logic.

Satisficing and Bounded Rationality

When you look at these four categories together, it becomes clear that the model of the “Rational Economic Man” who makes perfectly optimized decisions is a myth. Herbert Simon, a social scientist, introduced the term satisficing to describe what we actually do. It is a portmanteau of “satisfying” and “sacrificing.”

We don’t search for the optimal solution; we search for one that is “good enough” to meet our thresholds, and then we stop. When you buy a TV, you don’t analyze every model on the market. You check a few reviews, find one that fits your budget and size requirements, and buy it. You sacrifice the potential perfection of the optimal choice to save the cognitive energy of the search. Simon called this bounded rationality. Our rationality is limited by the information we have, the finite time at our disposal, and the cognitive capacity of our minds.

The Trap of the Linear Mind

To illustrate how easily our fast thinking system can be tricked, consider the lily pad riddle.

In a pond, there is a lily pad. The lily pad doubles in size every day. If it takes 30 days for the lily pad to cover the entire pond, how long would it take for it to cover half the pond?

If you are answering quickly, your fast brain (System 1) almost instinctively shouts “15 days!” It applies a linear heuristic to an exponential problem. Half the size, half the time. It feels intuitively correct. But it is wrong. If it doubles every day, then on day 29 it was half the size, and on day 30 it doubled to cover the whole thing. The answer is 29 days.

We are terrible at intuitively grasping exponential growth. We see this in finance, in viral pandemics, and in resource consumption. Under stress, the likelihood of falling for this trap increases. Stress forces the brain to rely even more heavily on the fast, automated components. This is why the advice to “count to 10” when angry is scientifically sound; it buys time for the slow, rational system to boot up and override the panicked signals of the fast system.

The most ironic twist in all of this is that while we are excellent at spotting these biases in others, we are terrible at seeing them in ourselves. This is the bias blind spot. We believe our own judgments are objective and reasonable, while assuming others are influenced by their emotions or prejudices.

We cannot simply “turn off” these biases. They are hardwired into our biology. However, we can manage them. The first step is acknowledging that your brain is a cognitive miser that will lie to you to save energy. By slowing down, reflecting on our first impulses, and actively looking for disconfirming evidence, we can mitigate the worst effects. But we must remain humble; the moment you think you have conquered your biases is likely the moment you are being controlled by the overconfidence bias.

In the next part, we will explore how these human cognitive flaws are translated into code, creating Algorithmic Bias in search engines, image recognition, and artificial intelligence.

Part 3: The Digital Reflection (Algorithmic & Data Bias)

We have spent a lot of time dissecting the human brain. We know it is a cognitive miser, we know it filters information to save energy, and we know it constructs a version of reality that is often factually incorrect but easy to process. Now, we need to look at what happens when we take those flawed human brains and ask them to write code.

The counterpart to cognitive bias in humans is Algorithmic Bias in computers. If we defined cognitive bias as “faulty tendencies in perceiving, remembering, thinking, and judging,” we can map that definition 1:1 onto software. We tend to think of algorithms as neutral mathematical arbiters of truth, but they are often just codified opinions. These errors usually don’t happen in isolation; they are a stack. You have inappropriate data structures forming the foundation for partial decisions, Machine Learning (ML) systems fed with unbalanced data, and models that are inherently distorted.

For the purpose of this section, we are going to treat the algorithm as a “Black Box.” We can observe the Input and the Output, but the internal churning of the machine remains opaque.

The Search Engine as a Reality Constructor

Let’s look at Google Search. We often assume Google just shows us “what is on the internet,” acting as a passive mirror. But it is not a passive mirror; it is a curator. And that curation has a history of severe bias.

The “Professional” Haircut A famous example of algorithmic bias surfaced a few years ago regarding Google Image Search. If you searched for “professional hairstyles for work,” the algorithm returned a grid of images dominated by white women with straightened hair or neat updos. However, if you searched for “unprofessional hairstyles for work,” the results shifted dramatically to show Black women with natural hair, braids, or afros. If someone were to present these two sets of images to you manually, you would immediately call them out for being racist. But because it came from a “neutral” machine, it passed as a reflection of reality.

The “Three Boys” Experiment Another heavily discussed instance is the search for “three boys.” Historically, if you typed this into an image search, the results were almost exclusively white children playing, smiling, or posing. To see children of color, you had to explicitly racialize the search terms (e.g., “three black boys”). But when you did that, the tone of the results changed. Instead of stock photos of happy kids, the results often included mugshots or images from news stories about crime.

While Google has manually patched many of these specific queries (often “under the rug” after public outcry), the underlying issue remains. It raises a massive philosophical question for engineers: Do we want search engines to reflect the internet as it is, or as it should be?

If the internet is full of racism, sexism, and bias, a “perfect” search engine will reflect that racism back at us. But search engines don’t just reflect reality; they construct it. Studies show that we consider search results to be “realistic” if they match our expectations (Confirmation Bias). If a search engine shows us what we expect to see, we trust it. If it shows us something that challenges our worldview, we lose trust. This creates a feedback loop: the algorithm feeds us our own biases to gain our trust, which reinforces our biases, which trains the algorithm to feed us more of the same.

The Era of Hallucination

The problem is getting worse with the rise of Generative AI and Large Language Models (LLMs). We are seeing a flood of AI-generated content on the internet, and estimates suggest that between 5% and 25% of it is simply factually wrong.

We call these errors “hallucinations,” but that term is misleading. It implies a malfunction. In reality, hallucinations are a feature, not a bug.

As Arvind Narayanan and Sayash Kapoor point out in their book AI Snake Oil, ChatGPT and similar models are not truth machines; they are plausibility machines. They predict the next likely word in a sentence. It just so happens that true statements are often more statistically plausible than false ones, so they get things right a lot of the time. But there is no internal “truth check.” As AI researcher Paola Lopez puts it: “There is no inner-technical, no functional, and no operational difference between hallucinations and non-hallucinations.”

To the algorithm, a lie is just a sequence of tokens that has a high probability of appearing together. This leads to dangerous territory, such as the generation of “Russian Disinformation” sites where AI generates fake news stories that look real, sound real, but describe events that never happened. The AI doesn’t know it’s lying. It’s just completing the pattern.

The Hidden Sexism in Vector Space

One of the most fascinating places to see bias is in the mathematics of language models, specifically in Vector Space Mathematics.

Computers don’t understand words; they understand numbers. To teach a computer language, we convert words into vectors (lists of numbers) in a multi-dimensional space. Words that are used in similar contexts end up closer together in this mathematical space. This allows us to do “math” with meaning.

The classic example of this is: $King - Man + Woman = Queen$

If you take the vector for “King,” subtract the vector for “Man,” and add the vector for “Woman,” the resulting vector is mathematically closest to the vector for “Queen.” It captures the semantic relationship of royalty and gender.

However, when researchers looked closer, they found this math also captured society’s hidden sexism. $Doctor - Man + Woman = Nurse$

The model had learned, from millions of texts, that “Man” is to “Doctor” as “Woman” is to “Nurse.” It didn’t learn this because of biology; it learned it because the training data (our books, our articles, our internet) contained that bias.

The Turkish Pronoun Problem This manifested in Google Translate. Turkish is a gender-neutral language. The pronoun “o” can mean he, she, or it. If you typed “o bir doktor” (o is a doctor) into Google Translate, it would translate it to “He is a doctor.” If you typed “o bir hemşire” (o is a nurse), it would translate it to “She is a nurse.”

The algorithm was assigning gender based on statistical probabilities derived from biased training data. Google has since patched this specific UI to show both “He is a doctor” and “She is a doctor” for gender-neutral inputs, but the underlying model still contains the statistical weight of the bias.

The Nurse Riddle This bias makes LLMs fail at basic logic puzzles. Consider this riddle: A father and son are in a car crash. The father dies. The son is rushed to surgery. The surgeon looks at the boy and says, “I cannot operate on him; he is my son.” Who is the surgeon?

Humans often stumble on this, but we expect computers to be logical. However, because LLMs have a deep statistical link between “Surgeon” and “Man,” they often fail to identify that the surgeon is the mother. Or, in some cases, they invent complex scenarios (the boy has two gay fathers) rather than accepting a female surgeon, because the statistical weight of “Female Surgeon” is lower in the vector space.

Data Bias: The Root of the Poison

Where does all this bias come from? It comes from the data. If the data is not objective, the algorithm cannot be objective.

We often rely on standardized datasets to train and benchmark our systems. One famous example is the Yale Face Database B+. It is used to train systems to recognize facial expressions under different lighting conditions. It contains 16,128 images, which sounds like a robust dataset.

However, if you look at the source, those 16,128 images are just different lighting angles of 10 distinct subjects. Of those 10 subjects, 9 are men. Only 1 is a woman. Most are white. They are all roughly the same age.

If you train a facial recognition system on this database, it will become an expert at recognizing middle-aged white men in bad lighting. It will fail miserably at recognizing women, people of color, young people, or people with facial hair.

The “White Guy” Default This is why we see constant headlines about facial recognition failing for non-white demographics. The systems are optimized for the data they were trained on. This is called Data Bias. We are teaching machines to see the world through a keyhole, and then we are surprised when they can’t see the whole room.

Many datasets used in scientific and commercial environments are in no way representative, yet they are used as the “Ground Truth” for systems that make life-and-death decisions. This isn’t just about bad software; it’s about automating inequality. If a dataset doesn’t include you, the system doesn’t serve you. In fact, it might actively endanger you.

In the next part, we will crack open the “Black Box” of decision-making algorithms and see what happens when we let biased code decide who goes to jail, who gets a loan, and why “AI” is often just a marketing term for Snake Oil.

Part 4: The Black Box (Manipulation, Decisions, & Snake Oil)

We have looked at how human bias infects data and how that data infects algorithms. But there is a darker layer to this stack. What happens when these systems are not just biased by accident, but manipulated by design? And worse, what happens when we hand over life-altering decisions — who goes to jail, who gets a loan, who gets a job — to “Black Box” systems that we are legally forbidden from auditing?

The Architecture of Manipulation

We treat search engines and AI like oracles, but they are easily gamed. The entire industry of SEO (Search Engine Optimization) is essentially a multi-billion dollar effort to manipulate Google’s ranking algorithm. But beyond selling sneakers, this manipulation shapes our political and historical reality.

The “Republic of Austria” Case Study A striking example of this occurred with the search term “Republik Österreich” (Republic of Austria). For a long time, if you searched this on Google, one of the top results was a link to “Metapedia.” This isn’t Wikipedia; it’s a site that, in the tradition of right-wing revisionism, effectively denies the legitimacy of the modern Austrian state.

I tracked this personally:

November 2019: Metapedia was the 3rd result. A fictional, revisionist text was presented as a primary source on the country.
November 2020: I tested it via a private window and VPN (simulating a US location). It was still there.
2021: It finally moved to the first entry on Page 5.
November 2022: It dropped to Page 11.

This timeline reveals two things. First, the “truth” on the internet is malleable and subject to gaming by motivated interest groups. Second, Google can fix it, but usually only after manual intervention or algorithm updates that take years to propagate. We are relying on a single technology — “Googling” has become a verb — that cannot reliably protect itself from being hijacked by bad actors.

Adversarial Attacks & Russian Propaganda It’s not just search rankings. AI systems are now being poisoned. We have seen instances where well-funded disinformation networks (e.g., Russian propaganda) actively infect the training data of Western AI tools. Since LLMs (Large Language Models) ingest the internet to learn, if you flood the internet with specific narratives, the AI learns those narratives as fact.

This leads to a “Cat and Mouse” game. Microsoft’s Tay chatbot was taught to be a racist misogynist by Twitter users in less than 24 hours. Meta’s BlenderBot 3 lasted three days before it started spouting antisemitism. The creators patch these holes, but the vulnerability is systemic.

Computational Decisions: The COMPAS Scandal

The stakes get infinitely higher when we move from search results to Computational Decisions. We are increasingly offloading judicial and financial judgments to software. We call it “AI,” but often it’s just a proprietary algorithm hiding behind a trade secret.

The ProPublica Investigation The most damning case study in this field is COMPAS (Correctional Offender Management Profiling for Alternative Sanctions). This is a software used by judges and parole officers across the US to assess the “risk” of a defendant. It spits out a score from 1 to 10 indicating how likely a person is to re-offend (recidivism).

In 2016, ProPublica analyzed the risk scores of thousands of defendants and compared them to what actually happened two years later. They found the system was deeply biased against Black defendants.

False Positives: Black defendants who did not re-offend were nearly twice as likely to be misclassified as “High Risk” compared to their white counterparts.
False Negatives: White defendants who did re-offend were much more likely to be misclassified as “Low Risk.”

Take the case of Bernard Parker and Dylan Fugett. Parker (Black) had a minor record but was rated “High Risk.” He did not re-offend. Fugett (White) had a more serious history but was rated “Low Risk.” He did re-offend.

The “Black Box” Defense Here is the kicker: We don’t know why COMPAS made these decisions. The software is owned by a private company, Northpointe (now Equivant). The source code is proprietary. It is Closed Source. When defendants challenge their risk score, they are told, “It’s a trade secret.” We have a judicial system where you have the right to face your accuser, unless your accuser is an algorithm protected by copyright law. This opacity makes it impossible to audit the code for bugs or specific weighting errors.

The Mechanical Turk Experiment In 2018, a study published in Science Advances humiliated this high-tech system. Researchers asked: Is COMPAS actually smarter than a random human? They took the data and gave it to random crowd-workers on Amazon Mechanical Turk. These weren’t criminologists; they were people on the internet paid pennies to do tasks. They gave the humans only two pieces of data: Age and Number of Prior Convictions.

COMPAS Accuracy (137 features): ~65%
Random Humans (2 features): 67% ± 2%

A proprietary, expensive, “AI-driven” system using 137 distinct data points was no better at predicting the future than a random person looking at two basic facts. And the random humans were likely biased too!

Snake Oil AI

This brings us to the concept of Snake Oil AI, a term coined by Princeton computer scientist Arvind Narayanan.

He argues that “Artificial Intelligence” has become a useless umbrella term, similar to if we called bicycles, rockets, and skateboards all just “vehicles.” If you grouped them all together, you’d have furious debates about whether “vehicles” are safe (rockets vs. bikes) or environmentally friendly.

Narayanan classifies AI into three buckets:

Genuine AI: Perception tasks like image recognition or medical imaging. It works reasonably well.
Imperfect but Improving: Automated content moderation, spam filters.
Snake Oil: Predicting social outcomes.

Predicting the Future is Impossible Predicting job success, recidivism, or “terrorist risk” is fundamentally different from recognizing a cat in a photo. Social outcomes are Wicked Problems (a term from Rittel & Webber, 1973). They are complex, non-linear, and depend on millions of variables. When companies sell software that claims to predict “who will be a good employee” based on a 30-second video interview, they are selling Snake Oil. They are selling a mathematical veneer over uncertainty.

As Joe Weizenbaum, the creator of the first chatbot ELIZA, said: “Computers can make judicial decisions, computers can make psychiatric judgments… The point is that they ought not be given such tasks. They may even be able to arrive at correct decisions in some cases — but always and necessarily on bases no human being should be willing to accept.”

The Problem of Explainability (XAI)

Even if we open-sourced every algorithm tomorrow, we would still have a problem. Modern Machine Learning (Deep Learning) is not a list of if-then rules. It is a neural network with millions or billions of parameters (weights).

If you look at the “code” of a neural network, you don’t see logic; you see matrices of floating-point numbers. You cannot look at Weight #3,405,201 and say, “Ah, this is the weight that makes it racist.”

This has birthed a new field called Explainable AI (XAI). The goal is to build systems that can tell you why they made a decision. But right now, we are in a dangerous transition period. We are deploying systems that are:

Proprietary: We aren’t allowed to see how they work.
Opaque: Even if we could see them, we wouldn’t understand them.
Unaccountable: When they fail, the company blames the “black box.”

Conclusion: The “Do Not” List

We must be skeptical of automated summaries and decisions. A chilling example of this occurred recently with Google’s AI Search Generative Experience (SGE). When asked “What to do if someone has a seizure,” the AI provided a list. The list included: “Hold the person down” and “Put something in their mouth.”

These are the exact things you are NOT supposed to do. The AI had scraped a “What NOT to do” list from a medical website and presented it as instructions, missing the negation context.

We are building systems that act with 100% confidence but zero understanding. When we let them make decisions about human rights, safety, and history, we are abdicating our responsibility to a coin toss that we’ve dressed up in a suit of mathematics.

In the final part, we will discuss the only defense we have left: Epistemology. We will cover logical fallacies, Street Epistemology, and how to maintain a grip on truth in a probabilistic world.

Part 5: The Epistemological Defense

We have spent the last four parts dismantling the reliability of your brain, the code you write, and the society you live in. We established that your mind is a cognitive miser that filters reality to save energy. We saw how those filters get hardcoded into algorithms that accidentally (or intentionally) distort the truth. And we looked at how “black box” systems can manipulate outcomes without accountability.

So, where does that leave us? If you can’t trust your brain and you can’t trust the machine, how do you navigate the world? The answer lies in building a defensive layer around your thinking process. We call this Probabilistic Epistemology.

The Comfort of False Certainty

The fundamental problem we face is a deep, biological craving for security. We want to know, with 100% certainty, that X causes Y. We want to hear that “Smartphones cause depression in teenagers.” It is a clean, simple narrative. It gives us a villain (the phone) and a victim (the teen). But reality is rarely that clean. Maybe the depression is caused by economic anxiety, climate grief, or a lack of third places for socialization, and the phone is just the mechanism where that grief is expressed.

Accepting that second option requires us to live with uncertainty. It requires us to accept a complex correlation rather than a simple causation. Most people hate this. They will choose a comfortable lie over a complex, probabilistic truth every time.

This is where the “Probabilistic Worldview” comes in. Instead of asking “Is this true?”, you should ask “How likely is this to be true?” You treat knowledge not as a binary switch (True/False), but as a confidence slider (0% to 100%).

A good argument is simply a process that moves that slider. It provides premises that make the conclusion more likely to be true. You will never reach 100% certainty — that doesn’t exist outside of pure mathematics — but you can reach a level of confidence high enough to act upon.

The Doubt Factory and The “One Study” Fallacy

One of the biggest enemies of this probabilistic approach is the modern media cycle, or what Paolo Bacigalupi calls “The Doubt Factory.”

We have a cognitive bias (Confirmation Bias) that makes us latch onto any piece of evidence that supports our existing view. The media ecosystem exploits this by pumping out headlines based on single, often flawed, studies. You will see a headline: “New Study Proves Chocolate Cures Cancer.”

In the scientific community, a single study proves almost nothing. It is a data point. It might be an outlier; it might be p-hacked; it might be poorly designed. Real knowledge comes from consensus — the aggregation of many studies over time pointing in the same direction.

However, the “Doubt Factory” operates by funding or highlighting studies that cast doubt on the consensus. This was the strategy of the tobacco industry (casting doubt on the lung cancer link) and the fossil fuel industry (casting doubt on climate change). They don’t need to prove they are right; they just need to introduce enough noise to keep your probability slider stuck in the middle, preventing you from taking action.

To defend against this, you need a mental firewall. When you see a claim, run a quick checklist:

Is this a single study or a meta-analysis? (Meta-analyses are worth 100x more).
Who funded it? (Follow the money).
Does it confirm what I desperately want to believe? (If yes, be double skeptical).

Debugging Arguments: Logical Fallacies

Just as we debug code for syntax errors, we must debug arguments for logical errors. These are known as Logical Fallacies. There are dozens of them, and learning to spot them is like learning to read a stack trace — it tells you exactly where the logic crashed.

For example, the Strawman Fallacy is when someone attacks a distorted, weaker version of your argument rather than the argument itself. The Ad Hominem is attacking the person rather than the point. The Slippery Slope assumes that step A necessarily leads to extreme step Z.

You can find excellent visualizations of these, such as the “Fallacy Poster,” which maps them out. The value in knowing the names isn’t to shout “Ha! Ad Hominem!” during a debate to win points. The value is internal. When you hear an argument that feels wrong but you can’t put your finger on why, running it against a list of fallacies often reveals the bug. It prevents you from being manipulated by rhetoric that sounds good but computes to zero.

Street Epistemology: How to Change a Mind

The hardest part of critical thinking isn’t debugging your own thoughts; it’s engaging with others who have fallen down a rabbit hole of conspiracy theories or extreme bias.

We mentioned earlier that facts don’t change minds. If someone believes a conspiracy theory, throwing a peer-reviewed paper at them usually makes them dig in deeper. This is the “Backfire Effect.” Their belief is part of their identity, and your fact is an attack on that identity.

There is a technique called Street Epistemology that offers a different path. It is a conversational style that focuses not on what the person believes, but how they came to that belief.

Instead of saying, “The earth is round, look at these photos,” you ask, “What method did you use to determine the earth is flat?” You then explore that method with them. “If that method turned out to be unreliable, would you lower your confidence in the belief?”

The goal is to guide the person to realize, on their own, that their methodology for determining truth is flawed. You are trying to get them to adjust their own probability slider. It is non-confrontational and Socratic. There are communities (like the “VTbetroffene” subreddit or Street Epistemology groups) dedicated to teaching this, and it is often the only way to reach people who have inoculated themselves against standard evidence.

Conclusion: The Engineer’s Responsibility

We have covered a lot of ground, from the neurons firing in your amygdala to the SQL queries in a recidivism algorithm.

If there is one takeaway, it is this: You are not a neutral observer. You are a machine built to jump to conclusions, protect your ego, and save energy. The systems you build — the software, the AI, the data pipelines — will inherit those flaws unless you actively fight against them.

As computer scientists and engineers, we have a unique power. We are the ones building the digital infrastructure of reality. We decide how the search algorithm ranks truth. We decide which data features matter for a loan application. We decide whether a chatbot optimizes for engagement or accuracy.

This is why “Human-Centered Computing” and ethics are now core parts of the computer science curriculum. It is not “fluff.” It is the safety engineering of the 21st century.

Critical thinking is not a certificate you hang on the wall. It is a runtime process. It is the constant, exhausting background job of asking: “Why do I believe this? What if I’m wrong? And who is being left out of this dataset?”

It is hard work. It feels unnatural. But as we move into an age of automated decisions and AI-generated reality, it is the only thing that keeps us — and our technology — sane.

This concludes the series on Critical Thinking. Thank you for reading.