Belief architecture: why people defend falsehoods and how ideology shapes thinking

What a belief is and how it differs from knowledge

In everyday speech, "I think," "I reckon," "I know," and "I believe" sound almost the same. To analyze the mechanisms, it helps to separate these levels, because each reacts differently to new data.

An assumption is a temporary, low-stakes hypothesis. "The bus is probably late because of traffic." If it turns out to be roadworks, nobody feels humiliated. An assumption is easy to let go of because it rests on nothing and holds nothing up.

An opinion is a judgment that mixes taste, values, and partial data. "This film drags," "this reform is unfair." Arguments about opinions often turn out to be arguments about criteria: one person rates the film by pacing, another by depth of character. Facts matter here, but they do not settle everything.

A belief is a relatively stable representation of how the world works, what is possible in it, and what is right. It differs from an opinion less by content than by its place in the system: other conclusions, decisions, and actions rest on it. "People mostly act out of self-interest," "natural is safer than artificial," "our country has always fought defensively" — these are not isolated statements but supports for a whole set of other thoughts.

Knowledge, in the epistemic sense, is a belief that is not merely true but has reliable grounds and has survived testing. Philosophers have argued about the exact definition for centuries, but for practice a working distinction is enough: knowledge differs from belief in the quality of its support and its openness to revision. The boundary moves. What one generation of doctors considered knowledge, the next recognizes as error; what began as a bold hypothesis becomes a textbook model.

Beliefs form a web

The philosopher Willard Quine described our beliefs as a "web of belief": the most fundamental ones at the center, the specific and easily replaced ones at the periphery. When experience contradicts some belief, we can adjust different parts of the web, and we usually choose the periphery so as not to touch the center. The metaphor does not literally describe the brain, but it explains an everyday observation well: some views we change in a minute, while others do not budge for years, even though the evidence against them is no weaker.

The networked nature of beliefs has an important consequence. When you refute one node, you inevitably touch its neighbors. A person who admits that the medicine they gave their child did not work must also reconsider their trust in their doctor, in their source of advice, and in their own decision. The cost of updating is set not by the weight of a single fact but by the number of connections it drags along.

What beliefs do

A belief performs several jobs at once. It explains: it gives a causal picture of events. It predicts: it lets us expect certain consequences. It orients: it suggests how to act. It soothes: it removes uncertainty. And it binds: it makes a person part of a group that shares the same picture. Only the first two functions depend directly on truth. The other three can work perfectly well on a false belief — which is exactly why false beliefs can be so durable.

Hence a practical conclusion: the argument "you believe, but I know" often masks an argument about the quality of grounds and the cost of being wrong. It is more useful to ask what the claim rests on and what would change if it turned out false. This move is unpacked in the piece on layers of claims and evidence: separate what is claimed from what is shown, and what is shown from what is proven.

How beliefs form: prediction, experience, and emotion

Intuitively, it feels as if eyes and ears work like a camera and a microphone: they record the world, and then the mind draws conclusions. Contemporary cognitive science describes an almost opposite picture.

The brain as a prediction machine

The idea goes back to Hermann von Helmholtz, who in the nineteenth century spoke of "unconscious inferences" in perception. Today it is developed by predictive processing models, associated with Karl Friston, Andy Clark, and Anil Seth. In simplified form: the brain constantly builds a prediction of which signal is about to arrive from the senses and compares the prediction with the actual signal. The mismatch — the prediction error — is used to update the model. Anil Seth calls perception a "controlled hallucination": we see our own prediction, which reality keeps correcting.

It is important not to overstate the status of these models. They are an influential theoretical framework, not a definitively proven account of how the brain works, and many details are contested. But one idea from it is especially useful for thinking about beliefs: the weight of a prediction error depends on how much the system trusts the signal and how confident it is in the prediction. If the expectation is very confident and the signal is noisy or ambiguous, the brain is more likely to "fill in" the signal to match the expectation than to rebuild the expectation to match the signal.

The usual illustration is the typos we fail to notice in our own writing: the prediction overrides what is actually on the page. The same logic applies at the level of meaning: a person convinced that a colleague is hostile will read a neutral email as cold. They are not consciously lying to themselves — they genuinely see it that way.

From observation to belief

A single observation rarely becomes a belief on its own. The path is usually longer. First the event is noticed — and we notice far from everything, only what attention picks out. Then it is interpreted, that is, given a cause. Then the interpretation is generalized: "it's always like this," "people like that do things like that." Then the generalization is repeated many times — in conversations, in inner speech, in the choice of what to read. With each repetition it becomes more familiar, and the familiar feels obvious.

Here an asymmetry appears that matters for everything that follows. The first interpretation forms quickly and with almost no resistance, because on empty ground it has nothing to compete with. Every subsequent piece of evidence is already read through it. That is why early beliefs — absorbed in the family, at school, in one's first professional environment — are especially tenacious: all later conclusions rest on them, and revising the foundation costs far more than adjusting the superstructure.

Culture as a set of ready-made explanations

Most of our beliefs are not derived from personal experience at all. We did not check for ourselves that germs cause disease — we accepted it from people and institutions we trust. This is a necessity: no one can verify everything independently. But it follows that the question "what should I believe?" often comes down in practice to "whom should I trust?" Culture, upbringing, and surroundings set not only the content of beliefs but also the rules: which explanations count as acceptable, whose voice carries more weight, what counts as a good argument.

Emotion as a marker of importance

Emotionally charged experience is remembered more firmly than neutral experience. A message that frightens, outrages, gives hope, or confirms that you belong among "us" has a better chance of entering your long-term model of the world than a dry number with no stakes. Emotion tags information as relevant to survival, so to speak. That is why the same statistic, presented in two different stories, leaves a different trace. It helps to get into the habit of noticing not only what a story says but also what it leaves out — a skill explored in the essay see the hole, not the donut.

Why the brain doesn't always seek the truth

It is convenient to think of thinking as a truth-seeking tool that occasionally breaks. It is more accurate to think of it as a system solving many tasks at once: understand the situation, conserve effort, preserve relationships and status, reduce anxiety, coordinate action with the group. Accuracy is one of these tasks — important, but not always the main one.

Heuristics: savings that sometimes cost a lot

In the 1970s, Daniel Kahneman and Amos Tversky described a set of heuristics — quick rules people use to make judgments under uncertainty. We estimate the probability of an event by how easily examples come to mind (availability), by how much a case resembles a typical image (representativeness), and we shift numerical estimates toward the first number we heard (anchoring). In Thinking, Fast and Slow, Kahneman popularized the distinction between two modes of thinking: fast, automatic, intuitive (System 1) and slow, effortful (System 2). These are convenient characters for explanation, not two regions of the brain.

One of his phrases is especially apt for the topic of beliefs: "what you see is all there is." Fast thinking builds the most coherent story from the available data and ignores what it cannot see. The coherence of a story subjectively feels like its truth — even though a story built on two random facts can be perfectly coherent.

Heuristics are not errors in themselves: in most everyday situations they produce a good-enough result at a low cost. Problems begin when a quick simplification replaces checking in places where mistakes are expensive: in medicine, finance, the courts, or when choosing whom to trust with your children or your vote.

Accuracy versus a desired conclusion

In 1990, Ziva Kunda published a review that became a classic: "The Case for Motivated Reasoning." Her central point was that reasoning can have different goals. When the goal is accuracy, people spend more effort, consider more options, and test conclusions more strictly. When the goal is a particular conclusion that is for some reason desirable, reasoning moves toward it.

The key detail in Kunda's work is a constraint. People cannot believe just anything because they want to. They try to build a justification that would persuade a dispassionate observer. Motivation works more subtly: it determines which memories are retrieved, which inference rules are applied, and where the search stops. An unpleasant conclusion is met with the question "must I believe this?" — and a reason to say "no" is found. A pleasant one is met with "can I believe this?" — and a reason to say "yes" is found. This asymmetry was later described well by psychologists Peter Ditto and David Lopez: we apply a stricter standard of proof to inconvenient information.

An important consequence follows: from the inside, motivated reasoning is indistinguishable from honest analysis. The person really does think, check, and doubt — the doubt is just unevenly distributed.

Classic demonstrations

In 1960, Peter Wason asked participants to guess the rule behind the number triple 2-4-6. They could propose their own triples and be told whether they fit. Many participants came up with a hypothesis like "each number is two more than the last" and tested it only with confirming examples: 8-10-12, 20-22-24. After many "yes" answers, they confidently announced the wrong rule. The real rule was broader — any ascending sequence — and the only way to find it was to try triples that would refute one's own hypothesis. Wason's later four-card selection task showed a similar picture; much depends on how it is framed, but the tendency to seek confirmation more readily than disconfirmation replicates across many variants.

In 1979, Charles Lord, Lee Ross, and Mark Lepper had supporters and opponents of the death penalty read descriptions of two studies — one apparently supporting a deterrent effect of capital punishment, the other refuting it. Both groups found more methodological flaws in the study that contradicted their position and, by their own account, came away more confident in their original views. The same mixed data pushed people in opposite directions. Later work showed that this "attitude polarization" effect does not always replicate and varies in strength, but biased assimilation of evidence is a robust finding.

A map of common biases

The table below is not a list of "vices" but a map of places where thinking predictably drifts from accuracy. It is worth remembering that many effects depend heavily on experimental conditions and are often exaggerated in popular writing.

Why people defend false information

When new information contradicts a belief, a conflict arises. There are several ways out, and changing one's view is far from the cheapest.

Cognitive dissonance

In 1957, Leon Festinger formulated the theory of cognitive dissonance: when two elements of a person's knowledge are incompatible — "I consider myself honest" and "I just lied" — an unpleasant tension arises that the person tries to reduce. There are several ways to do it: change the behavior, change the belief, add a justifying explanation, discredit the source of the unwelcome information, or narrow its scope.

The best-known experiment in this line was run by Festinger and James Carlsmith in 1959. Participants performed a boring task and were then asked to tell the next participant that it had been interesting. Some were paid one dollar for this, others twenty. Later, those who received a dollar rated the task as more enjoyable. Festinger's interpretation: twenty dollars was sufficient external justification for the lie, but one dollar was not, so the low-paid participants had to "pull" their own opinion toward what they had said. The experiment spawned decades of research and debate about alternative explanations, but the central idea — people tend to bring their beliefs in line with actions they have already taken — proved fruitful.

Even earlier, in When Prophecy Fails (1956), Festinger and his co-authors described a small group expecting the end of the world on a specific date. When the prophecy failed, some members did not abandon their faith but began recruiting new followers more actively. The authors explained it this way: people who had invested too much in the belief — quit jobs, cut ties, declared their faith publicly — cannot simply admit a mistake, and bringing in others helps reduce dissonance. A caveat is in order: the study was based on covert observation of a single group, and modern historians point to inaccuracies and to the observers' own interference. As an illustration of the mechanism it is valuable; as rigorous proof it is not.

Self-justification and the pyramid of choice

In Mistakes Were Made (But Not by Me), Carol Tavris and Elliot Aronson showed how dissonance works in ordinary life — among judges, doctors, politicians, and spouses. Their central metaphor is the pyramid of choice. Two people stand at the top, holding nearly identical views on some question. Each makes a small choice one way or the other. To justify their choice, each slightly strengthens the arguments for it and weakens the ones against. Step by step they descend opposite slopes of the pyramid and, after a while, end up at the bottom, sincerely unable to understand how the other could think differently.

An uncomfortable detail: the higher a person's self-esteem in some domain, the more painful the thought "I was wrong here," and the more effort goes into not admitting the mistake.

Defending identity

If a position has become part of the answer to "who am I?", refuting it sounds like "you are not who you think you are." The argument stops being an exchange of data and becomes a matter of saving face. Legal scholar Dan Kahan and colleagues described this as identity-protective cognition: when a question becomes a symbol of group membership — climate, guns, vaccines — it is in some sense more advantageous for people to share their group's position than to be accurate. Misjudging climate science costs one individual almost nothing personally; diverging from one's inner circle can cost a great deal.

This explains why admitting a mistake requires psychological safety: the ability to remain a decent person and a member of the group after changing one's view. Where that possibility is absent, defensiveness is, in its own way, a rational choice.

The emotional and meaning-making function of belief

Some beliefs provide not so much an explanation as an anchor: a sense of control, of a just world, of hope, a simple answer to chaos. Giving them up leaves not only the thought "I was wrong" but a void: the world becomes incomprehensible, dangerous, or unfair again. A person who believes illness can be prevented by eating the right way is defending not just a medical claim but the feeling that their fate is in their own hands.

That is why debunking that only destroys often fails. Research on the continued influence effect — a classic version was described by Hollyn Johnson and Colleen Seifert in 1994 — shows that corrected information keeps shaping reasoning if the retracted explanation leaves a gap behind. Participants read reports about a warehouse fire that first mentioned carelessly stored paints, then retracted that detail. Many kept referring to the paints when explaining the smoke and explosions. The influence weakened noticeably when the retracted cause was replaced with another plausible one. The brain needs not a hole but an alternative story.

The backfire effect: an important caveat

A popular idea holds that correcting a misconception strengthens it. It gained fame after a 2010 paper by Brendan Nyhan and Jason Reifler, in which, under some conditions, correcting a false political claim was indeed accompanied by stronger belief in it among a particular group of participants. However, later research — including a large-scale test by Thomas Wood and Ethan Porter published in 2019 — found such an effect only rarely. Nyhan himself later wrote that the backfire effect is the exception rather than the rule.

The Debunking Handbook (2020), by Stephan Lewandowsky, John Cook, Ullrich Ecker, and a large group of researchers, draws a cautious conclusion: do not refrain from correcting facts out of fear of strengthening the myth. Correction usually reduces belief in a false claim, though often not completely and not permanently. Attitudes are another story: a person may accept a corrected fact and still not change how they feel about the politician or party that spread it.

So the honest formulation is this: a correction may fail, especially if it touches identity, but it rarely makes things worse. The problem is more often not "backfire" but that the correction is weaker, rarer, and duller than the original lie.

Rationalization

Finally, the quietest mechanism is rationalization. A person first arrives at a conclusion, often intuitively or under the pressure of wanting it, and then finds arguments for it. In his 2001 paper "The Emotional Dog and Its Rational Tail," Jonathan Haidt proposed a model of moral judgment in which intuition comes first and reasoning more often acts as its lawyer than its judge. He described a phenomenon he called moral dumbfounding: participants confidently condemned an act that harmed no one, and when the experimenter refuted each of their arguments, they did not change their verdict but said they "just know it's wrong." Haidt's model is contested, and he does not claim that reasoning is powerless — but it explains well why a refuted argument is so often replaced by a new one while the conclusion stays the same.

A simple question helps catch rationalization in yourself: "What fact would make me change my mind?" If there is no answer, the position is probably not resting on facts. The same test is central to Julia Galef's book on the scout mindset: the scout wants to know how things are; the soldier wants to defend a position.

Group thinking: conformity, authority, and "us"

Beliefs are rarely formed alone. Most of what we believe we learned from others, discuss with others, and test against others. That makes the social environment not a backdrop but one of the main builders of our picture of the world.

Asch's experiments

In the early 1950s, Solomon Asch asked participants to compare line lengths — a task so easy that people working alone almost never erred. But in the experimental group, the other "participants" were confederates who, on some trials, unanimously gave an obviously wrong answer. In the original series, a noticeable share of the real participants' answers — about a third on critical trials — matched the mistaken majority. At the same time, many participants never yielded once, and most answered correctly on most trials, which Asch himself considered an equally important result.

A crucial detail: when a participant had even one ally giving the correct answer, conformity dropped sharply. One honest voice in a group is sometimes enough to make it easier for others to say what they see.

Asch's results depend on culture and era; replications in different countries produced effects of different strength. But the central idea holds up: the social situation changes what a person is willing to say, and sometimes what they are willing to see.

Milgram's experiments and their reassessment

In the early 1960s, Stanley Milgram studied obedience to authority. Participants in the role of "teacher" were instructed by the experimenter to punish a "learner" with increasingly strong electric shocks (in reality no current was delivered, and the learner was an actor). In the baseline version, about two thirds of participants went all the way to the top of the scale, even though many were visibly distressed, protested, and hesitated. The result was shocking because psychiatrists and laypeople alike had predicted beforehand that only a handful would do so.

These studies must be read with care. First, they raised serious ethical questions and could not be run in their original form today. Second, researcher Gina Perry's work with Milgram's archives showed that the procedure was not followed as strictly as described and that some participants doubted the shocks were real. Third, Alex Haslam and Stephen Reicher proposed a reinterpretation: participants obeyed not blindly but to the extent that they identified with the experimenter's scientific goal; the direct order "you have no other choice" was actually more likely to trigger refusal. Milgram varied the conditions, and obedience rates differed greatly across variants.

So the cautious conclusion is this: situation, authority, and investment in a shared goal can lead ordinary people to do things they would not have expected of themselves. But this is not proof that "anyone will do anything on command," nor a simple model of blind obedience.

Polarization and groupthink

More everyday mechanisms work without laboratories. Group polarization, described as early as the 1960s, means that after discussion among like-minded people, the group's position usually shifts toward a more extreme version of its initial view. Partly because people hear more arguments in one direction, partly because being "a bit more radical than average" brings status. Legal scholar Cass Sunstein has written extensively about how this works in courts, corporate boards, and online communities.

In the 1970s, Irving Janis described groupthink — a state of a cohesive group in which the drive for consensus suppresses critical evaluation; his material was foreign-policy failures such as the Bay of Pigs invasion. Its signs include an illusion of invulnerability, pressure on dissenters, and self-censorship. Rigorous tests do not always support the model, but as a set of warning signs it is useful.

There is also a quiet mechanism — pluralistic ignorance: everyone privately has doubts but assumes others are certain, and so stays silent. A group can sustain for years a belief that most of its members, individually, only half share.

Us and them

Dividing the world into "us" and "them" simplifies moral arithmetic. Our side's mistake is explained by circumstances; theirs, by character. Evidence from our source is accepted with trust; from theirs, with suspicion. Admitting a mistake in front of the group is often harder than admitting it to oneself: the stake is not only epistemic but membership.

Hence the paradox of identical information. Two people who receive the same facts reach opposite conclusions because they live in different networks of trust, with different norms about what counts as a good argument, and with different costs of switching sides. Their dispute looks like a dispute about facts, but in reality it is a dispute about whom to believe.

Ideology: when beliefs become a system

The word "ideology" often sounds like an insult or a synonym for propaganda. Here it is used in a broader and more neutral sense: an ideology is a coherent system of ideas, values, and explanations through which a person or group reads events. Political doctrines are just one special case.

Ideology is not only about politics

Ideological patterns arise wherever there is a shared picture of the world and a shared identity. A corporate culture with its "this is how we do things here, and it's right." A scientific school that rejects a competing theory for decades. A tech community convinced that any social problem can be solved with the right tool. Parenting forums with rigid camps on feeding or child-rearing. Fitness and diet movements with their own sacred cows and heresies. Professional cultures where newcomers absorb not only knowledge but also ideas about who counts as a "real" specialist.

In all these cases, the same elements are at work:

• a picture of the world and a set of acceptable causes for events;
• values and a notion of fairness;
• an image of a threat or an adversary;
• a desired future;
• the place of "me" and "us" in this story.

Hard core and protective belt

The philosopher of science Imre Lakatos described scientific research programs as a structure made of a "hard core" — basic tenets not open to revision within the program — and a "protective belt" of auxiliary hypotheses. When an experiment contradicts the theory, the belt is modified while the core stays. Lakatos was talking about science and did not consider this a vice: it lets a program develop without collapsing at the first anomaly. But he also distinguished progressive programs, which use new hypotheses to predict new facts, from degenerating ones, which merely explain failures after the fact.

This framework describes ideological systems remarkably well. If one fact is refuted, the system can be saved with an auxiliary explanation: "the data were rigged," "that's an exception," "that's about something else," "the true version of the idea has never been tried." A detail changes; the frame survives. An ideology's resilience comes from redundant connections: each tenet is propped up by several others, and knocking out one is not enough.

Ideology as a filter

That is why ideology works as a cognitive filter. The same event gets different names in different systems: "defense" or "aggression," "care" or "control," "reform" or "destruction." This is not necessarily lying: word choice often genuinely reflects different values. But the filter also acts on the perception of facts: what to treat as important, which source as reliable, which coincidence as chance and which as evidence.

Usefulness and the danger threshold

Ideology can be useful. It reduces uncertainty, coordinates collective action, transmits values across generations, and gives people a common language. Without a shared system of ideas, neither civil rights movements nor scientific communities nor cohesive teams would be possible.

The danger threshold is unfalsifiability. Karl Popper proposed distinguishing scientific theories from unscientific ones by whether they specify which observation could refute them. A theory that explains any outcome forbids nothing — and therefore tells us nothing about the world. When an ideology builds every refutation into its own logic in advance — "criticism proves the enemies fear the truth," "doubt is a sign of weak faith" — its epistemic function gives way to the function of belonging. Staying inside the picture becomes more important than refining the map.

A useful self-check for any system of views, including your own: what observation would force you to admit that part of it is wrong, and how does it treat dissenters from within — does it argue with them or expel them?

How false beliefs spread

One person with a false belief is a problem of one head. A false belief shared by millions is a problem of the environment. At the level of the environment, several amplifiers are at work.

Repetition

The illusory truth effect was first described by Lynn Hasher, David Goldstein, and Thomas Toppino in 1977: statements that participants had seen before were rated as truer than new ones. The mechanism is probably linked to processing fluency: the familiar is easier to process, and ease of processing is mistaken for a sign of truth. A 2015 study by Lisa Fazio and colleagues showed that the effect works even when people know the correct answer — although that knowledge weakens it noticeably. The practical implication is unpleasant: repeating a lie many times, including in order to refute it, makes it more familiar. That is why the authors of The Debunking Handbook advise leading with the fact rather than the myth and mentioning the myth only once, with an explicit warning.

Emotion and novelty

In 2018, Soroush Vosoughi, Deb Roy, and Sinan Aral published an analysis in Science of how fact-checked news spread on Twitter over several years. In their sample, false stories spread farther and faster than true ones. The authors linked this to the novelty of false stories and the emotions they evoked — surprise and disgust — and noted that people, not bots, played the main role. The study covers one platform and a particular sample, so its conclusions should be extended to the whole information space with caution. But the logic is clear: a lie is not constrained by reality and can be as surprising and outrageous as it needs to be in order to spread.

Trust in the source

For most people, checking the source is effectively the main check: we cannot evaluate an epidemiological study ourselves, but we can decide whether we trust the person retelling it. That is why the battle over beliefs is often fought not over facts but over authority: undermining trust in a source is easier than refuting its data.

Bubbles and echo chambers

In 2020, the philosopher C. Thi Nguyen proposed a useful distinction. An epistemic bubble is an environment in which a person simply does not hear other voices: they are filtered out, absent from the feed, absent from one's social circle. A bubble is relatively easy to burst — it is enough to show the missing perspectives. An echo chamber is an environment in which other voices are heard but discredited in advance: "they're all bought," "you can't trust them." An echo chamber is more resilient, because encountering an outside view does not destroy it — it confirms it.

Recommendation algorithms optimize for engagement, not accuracy, and so tend to show what already gets a reaction. But the data on the strength of the "filter bubble" are mixed: many studies show that people's own choices — whom to follow, what to click, whom to befriend — shape the homogeneity of their information environment as much as algorithms do, if not more. Technology speeds up and scales old mechanisms; it did not invent them.

Conspiracy thinking as a special case

In a 2017 review, psychologists Karen Douglas, Robbie Sutton, and Aleksandra Cichocka identified three groups of motives that make conspiracy theories appealing. Epistemic: the desire to understand and find meaning, especially when an event is big and the official explanation seems insufficient. Existential: the need to feel safe and in control. Social: the desire to maintain a positive image of oneself and one's group — if "we" suffered a setback, it is easier to believe someone else's malicious intent is to blame.

Hence the typical features of conspiracy constructions:

• a big event seems to demand a big cause — chance and coincidence look unconvincing;
• someone's design is assumed behind events, even where there is none;
• absence of evidence is read as evidence of a cover-up;
• refutation is absorbed into the theory: "they're trying to silence us."

This architecture is almost unfalsifiable: every new fact either confirms the theory or "proves" that opponents are hiding the truth. It is an extreme form of the general tendency of ideological systems to protect their frame. The reverse is also worth remembering: real conspiracies have occurred in history, and suspicion of power is not a vice in itself. What distinguishes conspiracy thinking is not the topic but the way it handles evidence — whether it is willing to name what could refute the suspicion.

Disinformation as a tool

Deliberate disinformation uses the same channels, and its goal is often not to convince you of a specific lie but to erode trust in all sources at once, so that people choose by allegiance. Defense starts not with "believing the opposite" but with daily checking habits and the ability to see gaps in a story. A separate new risk zone is AI-generated retellings and summaries: they sound confident and smooth, and fluency, as we have seen, is easily mistaken for truth. How to read such texts is covered in the piece on AI summaries.

Why intelligence doesn't protect against delusion

A common myth says a smart person is automatically better at separating truth from falsehood. The empirical picture is subtler and less comfortable.

The mind as a lawyer

High intelligence helps build complex models, find arguments, and spot weak points in reasoning. But these abilities have no built-in direction. They can be turned toward testing one's own position or toward defending it: picking exceptions, redefining terms, attacking the method of an inconvenient study instead of its substance. The more sophisticated the mind, the more sophisticated the defense — and the more convincing it looks from the inside.

Psychologist Keith Stanovich spent many years studying so-called myside bias — the tendency to evaluate evidence and generate arguments in favor of one's own side. In his work, this tendency turned out to be only weakly related to general cognitive ability: people with high intelligence displayed it no less than others. In a 2012 study by Richard West, Russell Meserve, and Stanovich, higher cognitive ability also did not reduce the "bias blind spot" — the tendency to see biases in others but not in oneself.

Work by Dan Kahan and colleagues on "motivated numeracy" provided a vivid illustration: when a data table concerned a neutral topic, more numerate participants answered correctly more often, but when the same numbers concerned a politically charged topic, the gap between holders of different positions was larger among them. Replications have produced more modest and mixed results, but the general line is consistent with other data: the ability to reason does not guarantee the desire to reason honestly.

Expertise doesn't transfer

Competence in one field does not automatically extend to others: a physicist can be naive about epidemiology, a doctor about statistics. What does transfer easily is the feeling of confidence — into areas where a person has neither data nor tools. Without intellectual humility — the understanding that "I may be wrong here because this isn't my field" — intelligence raises confidence faster than accuracy.

What actually helps

Research on intellectual humility and on what psychologist Jonathan Baron called actively open-minded thinking points elsewhere: what matters is not the power of the mind but the style of using it. Willingness to look for arguments against one's own position, to acknowledge uncertainty, to change one's mind when the arguments demand it — these traits are more strongly associated with accurate judgments than raw reasoning ability. Philip Tetlock's work on forecasters showed something similar: the best forecasts came not from the most erudite experts with a grand theory, but from people who updated their estimates often, thought in probabilities, and readily admitted mistakes.

Critical thinking is a separate skill, not a side effect of intelligence: the habit of separating the layers of a claim, looking for disconfirmation, checking sources, noticing one's own emotion. It can be trained, including with tools like ThinkLens, but no tool replaces honesty with oneself.

Hence the paradox: "I'm smart, so I'm right" is a poor criterion. A better one is the willingness to name in advance the conditions under which the position would be revised.

How to change beliefs — your own and other people's

If beliefs rest on dissonance, identity, emotion, and the group, then changing them must take all these supports into account, not just the facts.

Your own beliefs: practical steps

The hardest thing is changing your own views, because from the inside they feel not like beliefs but like reality. What helps are techniques that move thinking outside your head and make it checkable.

Separate fact, interpretation, and assumption in writing. "My colleague didn't reply to my email" is a fact. "He's ignoring me" is an interpretation. "He's unhappy with my work" is an assumption. In your head these three layers merge into a single feeling; on paper you can see where the data end.

Name the conditions for revision in advance. Before an argument or a decision, state which observable result would change your confidence. If the answer is "none," you are dealing not with a conclusion but with a belief worth examining separately.

Consider the opposite. In 1984, Lord, Lepper, and Elizabeth Preston tested what helps reduce biased assimilation of evidence. Asking people to "be objective and unbiased" barely helped. A specific instruction worked much better: ask yourself whether you would have rated the study the same way if it had produced the opposite result. An abstract intention is weaker than a concrete procedure.

Look for the strongest version of the other position. Before arguing, state your opponent's position in a way they themselves would agree with. This technique is unpacked in the piece on steelmanning and hidden assumptions. It protects against the temptation to defeat a caricature.

Check the source, date, method, and baseline. Not just the headline, but where the number came from, what it is compared with, and who ran the study. Many false conclusions rest on a missing denominator.

Speak in probabilities. "Probably," "likely not," "I don't know" are not weakness but precision. Written-down confidence estimates can later be checked against reality to see how well calibrated you are.

Keep your identity small. The fewer views have become part of your answer to "who am I," the cheaper they are to change. You can be a person who values accuracy rather than a person who holds position X. Then changing a position confirms your identity instead of destroying it.

Seek outside criticism. Ask someone who disagrees with you to take apart your argument. Run a "premortem," as psychologist Gary Klein proposed: imagine the decision has already failed and describe why. Your own blind spot is best seen through someone else's eyes.

Other people's beliefs: what works better than debunking

Direct confrontation — "here are the facts, you're wrong" — often triggers identity defense. Some approaches take this into account better.

An alternative explanation instead of a void. As research on the continued influence effect shows, a correction works better when it offers a coherent replacement. The Debunking Handbook recommends a structure sometimes called the "truth sandwich": lead with the accurate fact, mention the myth once with a warning that it is false, explain why it is misleading, and end with the accurate fact again.

Inoculation instead of cure. Psychological inoculation theory, going back to William McGuire's work in the 1960s, proposes exposing people in advance to a weakened version of a manipulative technique. Research by Sander van der Linden, Jon Roozenbeek, and colleagues showed that short games and videos explaining common disinformation techniques improved people's ability to recognize manipulation in a number of experiments. The effects are usually moderate and fade over time, but this kind of prevention does not require arguing about a specific belief and so touches identity less.

Questions instead of statements. Approaches close to William Miller and Stephen Rollnick's motivational interviewing rest on respect for the other person's autonomy. Instead of proving a point, you help the person explore the gap between their own values and their position: "You value honesty — how does that square with this source having been wrong several times already?" A conclusion a person reaches on their own does not need defending.

Personal conversation and shared experience. In 2016, David Broockman and Joshua Kalla published a study of so-called deep canvassing: volunteers went door to door and held unhurried conversations in which they listened more and invited people to recall their own experiences rather than arguing. In their experiment, even a short conversation of this kind produced a noticeable reduction in prejudice that lasted for months. It is one study with a specific topic and conditions, but it fits the general logic well: what changes attitudes is not pressure but the feeling of having been heard.

Lowering the cost of losing face. Admitting a mistake is easier when it does not mean admitting you are a fool. The framing "given the information you had, that conclusion made sense" helps separate the person from the error. Someone who feels safe finds it easier to doubt.

The limits of influence

Even the best methods guarantee nothing. Some belief systems are held together by social sanctions — the threat of losing family, friends, or a job — more strongly than by any arguments. In such cases a person can doubt privately while publicly holding the position for years. Respecting this reality is also part of an honest conversation: the goal is not "to persuade at any cost" but to create conditions in which updating becomes possible. For more on where legitimate influence ends and manipulation begins, see the guide on whether a person can be "programmed".

Is objective thinking possible?

Fully "pure" perception, free of models, expectations, and values, is unavailable to humans — that follows from how perception works. But it does not follow that all views are equally valid and there is no truth. The conclusion is rather the opposite: since an individual is inevitably biased, objectivity has to be built not inside the head but outside it — through procedures.

Science is precisely such a set of procedures. Measurement instead of impression. Independent replication instead of trust in the author. Preregistration of hypotheses, so the question cannot be fitted to the answer. Blind and double-blind trials, so the researcher's expectations do not affect the result. Open data, so others can recompute. Peer review, so other eyes look for errors. None of these procedures makes a scientist impartial; together, they make the system the scientist works in more impartial.

For an individual, the equivalent is personal procedures: write down predictions and compare them with outcomes, explicitly separate values from facts, invite criticism, name the conditions for revision. You cannot get rid of all biases. You can reduce their weight where the stakes are high.

The line between belief and knowledge runs not through the feeling of certainty but through the quality of the grounds and readiness for correction. A person is probably incapable of being aware of every factor in their own decision — any decision has too many hidden inputs, from fatigue to whatever they read that morning. That is why the skill of revision matters more than the illusion of final correctness.

How the full cycle works

Put the sections together and you get a closed loop in which each stage feeds the next.

It all starts with information and experience — but not all information, only what enters the field of attention. Attention is already selective: we notice what we expect, what is emotionally significant, and what is common in our environment. Then what is noticed gets interpreted. The interpretation passes through the brain's predictions, heuristics, the emotion of the moment, and the explanations available in the culture. The resulting story feels like perception, not like a conclusion.

A repeated interpretation hardens into a belief — a stable node in the model of the world. The more other conclusions, decisions, and actions rest on it, the more expensive it is to revise. If the belief is shared by a group, if it has become part of the answer to "who am I" and "who are we," the node turns into an element of identity. Now doubting it means doubting yourself and your people.

When contradictory information appears, defense kicks in: dissonance is resolved by reinterpretation, inconvenient data are checked more strictly, the source is discredited, and a new argument takes the place of the refuted one. From the inside, this feels like sober analysis. The group supports the defense, and the echo chamber explains in advance why outside voices cannot be trusted.

And here the loop closes: defense changes what new information a person is even willing to see. They subscribe to confirming sources, avoid unpleasant conversations, remember the hits and forget the misses. The next turn begins with an even more filtered experience, and the belief grows stronger still — not because there is more evidence, but because inconvenient evidence has stopped getting through.

The loop can be broken at any stage: at the input — broaden your sources; at interpretation — notice the emotion before the conclusion; at consolidation — name the conditions for revision; at the level of identity — separate yourself from the position; during defense — check whether the standard is the same for convenient and inconvenient data; in the group — be that one dissenting voice that, as Asch showed, makes honesty easier for everyone else. The donut metaphor is useful again: look not only at the argument presented but also at the alternatives missing from the story.

FAQ

Research and reading behind this guide

• Leon Festinger, A Theory of Cognitive Dissonance (1957); Festinger and James Carlsmith, the forced-compliance experiment (1959); Festinger, Henry Riecken, and Stanley Schachter, When Prophecy Fails (1956).
• Peter Wason, the 2-4-6 task (1960) and the four-card selection task (1966).
• Solomon Asch, conformity studies (early 1950s).
• Stanley Milgram, Obedience to Authority (1974); Gina Perry's critical archival analysis; the reinterpretation by Alex Haslam and Stephen Reicher.
• Daniel Kahneman and Amos Tversky, work on heuristics (1974); Kahneman, Thinking, Fast and Slow (2011).
• Ziva Kunda, "The Case for Motivated Reasoning" (1990).
• Charles Lord, Lee Ross, and Mark Lepper on biased assimilation of evidence (1979); Lord, Lepper, and Preston on considering the opposite (1984).
• Jonathan Haidt, "The Emotional Dog and Its Rational Tail" (2001) and The Righteous Mind (2012).
• Brendan Nyhan and Jason Reifler, "When Corrections Fail" (2010); Thomas Wood and Ethan Porter, "The Elusive Backfire Effect" (2019).
• Stephan Lewandowsky, John Cook, Ullrich Ecker, and co-authors, The Debunking Handbook (2020).
• Carol Tavris and Elliot Aronson, Mistakes Were Made (But Not by Me).
• Lynn Hasher and co-authors on the illusory truth effect (1977); Lisa Fazio and co-authors (2015).
• Soroush Vosoughi, Deb Roy, and Sinan Aral on the spread of false news (2018).
• Karen Douglas, Robbie Sutton, and Aleksandra Cichocka on the psychology of conspiracy theories (2017); C. Thi Nguyen on echo chambers and epistemic bubbles (2020).
• Dan Kahan and co-authors on identity-protective cognition and motivated numeracy; Keith Stanovich on myside bias.
• Imre Lakatos on the structure of research programs; Karl Popper on falsifiability.
• Andy Clark, Anil Seth, and Karl Friston on predictive processing.

Popular retellings inevitably simplify. Many effects in this field have turned out weaker on replication than in the original studies, or depend on conditions. When the stakes are high, go back to reviews and primary sources, and limit your conclusions to what the sample and method allow.

Conclusion

Belief architecture explains two truths about people at once: we are capable of subtle reality-testing — and capable of defending a mistake for years. The difference often lies not in intelligence but in which task thinking is solving right now: refining the map or preserving the house built on it. Ideology, the group, and emotion make that house warm and lived-in. Critical thinking reminds us that the house stands on the territory, not in place of it, and that sometimes the foundation has to be relaid.

As a rule, people do not set out to deceive themselves. They hold on to a model of the world that gives them stability, connection, and meaning. The trouble starts when defending the structure becomes more important than what the structure was built for — understanding reality.

The practical takeaway is simple to state and hard to practice: treat your settled views as hypotheses with a history, not as character traits. Name in advance what would make you change your mind. Then facts stop being insults — and become information again.