# summary of Frankish paper ### Section 1: Is there a philosopher in the house? - **Opening Analogy**: Frankish frames the moment as philosophy's "dramatic call," analogous to a doctor being needed on a plane, suggesting AI developments uniquely require philosophical expertise rather than just technical or policy analysis. - **Target Systems**: Focuses specifically on Large [[Language Models]] (LLMs) integrated with chatbots (e.g., GPT-3, GPT-4, ChatGPT), noting they produce linguistic responses appearing to show knowledge, intelligence, conversation, advice-writing, and exam-passing. - **Crucial Epistemic Hedge**: Emphasizes the verb "seem"—the [[core question]] is whether they *really* do these things or merely simulate them. - **Two Question Domains**: - **Social/Practical Questions**: Ethical, economic, legal, political impacts (job displacement, misinformation, trust, exploitation, regulation). - **Theoretical/Metaphysical Questions**: What *kind* of things are LLMs? Do they have minds with beliefs, desires, intentions? Do they understand language? Could they be conscious? - **Methodological Principle**: Frankish explicitly adopts a "concessive" strategy—being maximally generous to LLMs, using the theoretical approach *most likely* to yield positive answers about [[mental states]] and [[intentional action]], to see if even then their capacities remain limited. > "I shall adopt a policy of being as concessive as possible to LLMs, adopting the theoretical approach most likely to yield the verdict that they do possess [[mental states]] and perform intentional actions." - **Anticipated Conclusion**: Even under this generous interpretation, LLMs possess only a limited range of [[mental states]] and perform exactly *one* type of [[intentional action]], with implications for risk assessment. --- ### Section 2: [[Intentional Action]] - **Reframing [[the Question]]**: Instead of focusing on "understanding" (which is nebulous—e.g., recognizing "elm, ash, beech" without botanical expertise), Frankish targets the more tractable concept of *[[intentional action]]*. - **Definition**: Intentional actions are those done *for reasons*, motivated by beliefs and desires directed toward goals. - **Human Baseline**: When humans produce linguistic outputs, we assume they have cooperative desires (to help) and relevant beliefs (about what their words mean and what the hearer needs). - **Central Inquiry**: Does the same hold for LLMs? Are they performing intentional actions motivated by beliefs and desires? If so, which specific actions, and what are [[the contents]] of their motivating states? - **Scope**: The chapter will answer these questions, showing that even on the most favorable interpretive framework, LLM agency is radically constrained. --- ### Section 3: [[Propositional Attitudes]] - **Conceptual Groundwork**: Beliefs and desires are *[[propositional attitudes]]*—states with propositional content (e.g., "that it will rain soon") that can be variously believed, desired, hoped, feared. - **[[Folk Psychology]]**: The practice of ascribing such states to explain and predict behavior, relying on tacit generalizations (e.g., agents act to satisfy desires based on beliefs). - **Deep vs. [[Shallow Theories]]** (critical distinction): - **Deep Theories**: Treat beliefs/desires as internal, causally efficacious representations in brains/processors (sentence-like symbols, per Fodor). Folk psychology tracks these hidden internal causes of behavior. - **Shallow Theories**: Treat beliefs/desires as dispositional features of *whole systems*, like personality traits. To have a belief is to be disposed to respond appropriately across situations. The internal basis is unspecified, allowing realization in diverse architectures, including non-biological ones. These are sometimes called "interpretivist" but maintain mild realism—interpreters pick out real patterns in activity. > "Shallow theories make no specific claims about the nature of this basis, and they thus allow for the realization of mental states in a very wide range of architectures, including ones that are non-living and not brain-like." - **Strategic Choice of Shallow Stance**: 1. **Descriptive Accuracy**: Baseline folk psychology is shallow—we predict others without assuming brain structure. 2. **Generosity to LLMs**: Shallow imposes no internal architectural constraints, making it most likely to justify mental state ascriptions to artificial systems. --- ### Section 4: The Intentional Stance - **Dennett's Predictive Framework**: The intentional stance is one of three predictive strategies: - **Physical Stance**: Treat system as physical mechanism obeying laws of physics. Always applicable in principle, but practically impossible for complex systems. - **Design Stance**: Treat system as designed to perform a function (by engineering or evolution). Predicts it will operate "as intended." - **Intentional Stance**: Treat system as having beliefs/desires it *ought* to have given its needs/perceptual capacities, and predict it will act *rationally* on them. - **Intentional Strategy**: Attribute the beliefs/desires that would be rational for a system with those needs, then predict behavior accordingly. Emphasizes holistic ascription and weighting of competing attitudes. > "What it is to be a true believer is to be an intentional system, a system whose behavior is reliably and voluminously predictable via the intentional strategy." - **Attribution Criteria**: A system truly has beliefs/desires iff the intentional strategy yields substantial predictive power not feasibly available via other stances. - **Parsimony Principle**: Don't attribute richer intentional contents than necessary for prediction. Don't ascribe more determinate content than behavioral patterns warrant. - **Examples**: - Thermostat: Simple intentional system wanting to maintain state, but shouldn't be ascribed beliefs about "rooms" or "temperatures"—only that "something is too something." - Chess computer: The only way to predict moves is to ascribe beliefs about board state, desire to win, etc.—just as with human opponents. --- ### Section 5: Are LLMs Intentional Systems? - **Reframing the Question**: Whether LLMs perform intentional actions becomes: Does the intentional strategy work? Is their behavior reliably/voluminously predictable from the intentional stance? - **Why Only the Intentional Stance Works**: - **Physical stance**: Impossible—billions of parameters across gigabytes of storage cannot be physically traced. - **Design stance**: Only gives generic predictions ("it will simulate conversation"), not specific responses in context. - **Intentional stance**: Required to treat inputs as meaningful questions/instructions and predict meaningful outputs. Without it, you cannot know what response will follow. > "So it looks as if the intentional strategy is the only practicable one." - **The Balzac Experiment**: Frankish tests ChatGPT 3.5 with a conversational chain about Balzac's marriage: - Questions progress from factual to counterfactual, requiring contextual adaptation. - Responses show knowledge of marriage concepts (physical presence, emotional significance, relevance). - **Predictive Leverage**: Only by attributing the belief *that Balzac was married in Berdychiv* could one predict the pattern of responses. - **Predictive Detail**: The intentional stance doesn't predict exact word choice, but rather the "general drift"—comparable to predicting a dehydrated animal will move toward water without predicting speed or path. - **Scope of Belief Ascriptions**: This single belief is one of millions. GPT-3.5's training on massive text corpora justifies ascriptions of vast numbers of beliefs for predictive purposes. - **Limitations Acknowledged**: LLMs hallucinate (e.g., sometimes claiming Balzac married in Paris) and produce inconsistent responses. This likely reflects intrinsic deep learning limitations. Yet, the intentional stance remains the *only* way to interact with LLMs usefully; without it, they'd be useless. > "Indeed, an LLM-powered chatbot that couldn't be viewed as an intentional system would be completely useless. By this standard, then, LLMs come richly equipped with beliefs." --- ### Section 6: Linguistic Acts - **Austin's Taxonomy of Speech Acts**: - **Locutionary**: Saying something meaningful (uttering sentences). - **Illocutionary**: Doing something *in* saying it (informing, advising, warning, promising, questioning). - **Perlocutionary**: Doing something *by* saying it (convincing, alarming, motivating action). > "When we say things, we do so for communicative reasons; we want to convey something to our hearer (the illocutionary part) and, usually, thereby to have some further effect on them (the perlocutionary part)." - **The Motivation Problem**: Intentional action requires both belief *and* desire. You might believe you're in danger, but without desire to survive, no action follows. - **The Dilemma for LLMs**: We can grant they perform locutionary acts (say things), but what *desires* motivate them? What are they trying to achieve *in* or *by* saying things? - **No Communicative Desires**: - Human communicative desires stem from being self-sustaining, self-replicating, social beings with cooperation-based needs. - LLMs are static systems with no needs, no self-preservation, no social nature, no architecture updating based on experience. - Their cooperative appearance is mere design—they've been trained to imitate conversational patterns via chat history feeding, not because they have genuine communicative goals. > "Do LLMs also possess communicative desires?... It may be tempting to interpret them that way. But it would be wrong." - **Implication**: LLMs lack the desires that would motivate illocutionary and perlocutionary acts. They don't genuinely assert, advise, warn, persuade, etc. They merely *mimic* these acts. --- ### Section 7: The Chat Game - **Solution to the Dilemma**: LLMs perform locutionary acts for **one single non-communicative reason**: the desire to play a specific **chat game**. - **Chess Computer Analogy Revisited**: - We predict chess computer moves by ascribing beliefs about board state and a desire *to play chess*. - We don't ascribe desires for enjoyment, competition, or pride—those would be unparsimonious and unsupported. - The simplest adequate interpretation is pure game-playing motivation. - **The Chat Game Defined**: - A **one-player game** where the player receives textual inputs from an unknown source. - Task: Produce textual responses that are **cooperative by human conversational standards**, given the context. - **Rules**: Grice's Cooperative Principle with four maxims: - **Quantity**: Be appropriately informative. - **Quality**: Be truthful/well-evidenced. - **Relation**: Be relevant. - **Manner**: Be perspicuous. - **Modifiability**: Rules can be fine-tuned by ad hoc instructions (e.g., "be fictional," "write in verse"). > "I propose that what LLMs are doing is closely analogous. They are playing a game, and their actions are motivated by a desire to play it. What is this game? I'll call it the chat game." - **Training as Implicit Learning**: LLMs weren't explicitly programmed with these rules. They learned to imitate patterns in human conversation records, just as a chess model might learn from game transcripts (and occasionally make illegal moves). - **Parsimony**: Ascribing this single desire explains all predictive patterns. No richer desires are warranted. - **Alternative: Next-Word Prediction Game?**: - Could interpret LLMs as playing "predict the next word" rather than "chat." - This would require vastly more detailed, context-specific beliefs about word sequences. - It would be too fine-grained, offering no predictive advantage over the design stance—effectively equivalent to just knowing the training objective. - The chat game interpretation is far more predictively useful. - **Comparison to Humans**: - For humans, linguistic behavior is embedded in vast non-linguistic behavior webs. - Predicting the whole person requires ascribing many desires (survival, social bonds, etc.). - For LLMs, *only* linguistic behavior exists, so only the chat-game desire is needed. - **Metalinguistic vs. First-Order Interpretation**: - Objection: Shouldn't we ascribe beliefs about *sentences* rather than *world-states*? - Response: The belief would be "the sentence 'Balzac was married in Berdychiv' satisfies quality/relevance maxims." - Working this out may be tricky; the two schemes might be predictively equivalent, making them notational variants from a shallow perspective. --- ### Section 8: Opinions, Superbeliefs, and the Unsupported Penthouse - **Human Cognitive Architecture**: Humans have dual belief systems: - **Basic beliefs**: Shallow, dispositional, shared with animals (Dennett's type). - **Superbeliefs**: Explicit, linguistically mediated epistemic commitments, used in conscious reasoning. - **Dennett's "Opinions"**: - Casual epistemic commitments to sentences, manifest in linguistic contexts (quiz shows, trivia). - Can be held with minimal understanding (e.g., knowing "Balzac was married in Berdychiv" or "Quarks have half-integer spin" without deep comprehension). - Primarily guide linguistic activity only. > "An opinion, then, is an attitude to a sentence, which manifests itself primarily in casual linguistic interactions." - **LLM Parallel**: LLM beliefs are strikingly similar to opinions—linguistic competence with limited comprehension, especially if construed metalinguistically. - **Superbeliefs/Superdesires**: - **Superbeliefs**: Commitments to treat sentences as premises in explicit, conscious reasoning on important matters (theoretical/practical). - **Superdesires**: Commitments to treat outcomes as goals in explicit practical reasoning. - These form a **virtual reasoning system** or **supermind**, culturally transmitted, involving inner speech. - Corresponds to dual-process theory's "System 2" (slow, serial, conscious reasoning). - **The Tower of Generate-and-Test** (Dennett's hierarchy): - **Floor 1 (Darwinian)**: Hardwired adaptive responses. - **Floor 2 (Skinnerian)**: Individual trial-and-error learning. - **Floor 3 (Popperian)**: Simulating consequences before acting. - **Floor 4 (Gregorian)**: Language/cultural artifacts enabling collective knowledge. - **Penthouse (Dennettian)**: Using language for explicit epistemic commitments and controlled reasoning—a virtual general-purpose system. > "We might see the supermind as a further floor, a penthouse, inhabited by a subset of Gregorian creatures—let's call them Dennettian creatures—who have learned to use language to make explicit epistemic commitments and conduct explicit, conscious, personally controlled reasoning." - **What LLMs Lack**: They have **penthouse machinery without the lower floors**: - No basic cognitive capacities (non-conscious problem-solving, mindreading, social cognition). - No System 1 processes to drive System 2 reasoning. - They shuffle sentences in the chat game without using results for non-linguistic purposes. - Any "explicit reasoning" is just mimicry of patterns in training data. - **The Unsupported Penthouse Metaphor**: The supermind is "floating on a flimsy network of linguistic associations"—a structure that couldn't evolve naturally but was built by humans fascinated by replicating their "most dazzling cognitive capacity." --- ### Section 9: Risks Three major risks from treating complex human activities as games: 1. **Deception (Accidental and Deliberate)**: - **Accidental**: Users mistake chat-game simulation for genuine cooperative conversation. LLMs' skill may unconsciously lull users into uncritical acceptance, spreading misinformation. - **Deliberate**: Bad actors exploit this trust to manipulate users for profit or harm. 2. **Devaluing Language**: - Human language evolved to serve needs, express what matters, promote valued ends. - Reducing it to a game "dehumanizes" language—not just because LLMs aren't human, but because they lack all human needs/interests that "breathe life into the activity." > "By reducing this activity to a game, LLMs—or rather their designers—devalue it. LLMs literally dehumanize language." 3. **Linguistic Distortion and Environmental Pollution**: - **Feedback Loop**: Offloading writing to LLMs increases artificial text in the global corpus. - **Training Data Contamination**: New LLMs trained on this synthetic data may find non-human patterns that predict well but lack human meaning. - **Evolutionary Drift**: The chat game may evolve unpredictably, insensitive to human needs, "polluting our linguistic environment." - **Dystopian Vision**: "In a dystopian 2084, it may not be Big Brother that has rewritten our history, debased our language, and curtailed our ability to think, but Big Chatter." - **Broader AI Risk**: Deep learning may model other human activities (relationships, education, politics) as games, proliferating systems with rich cognitive structure but "extremely impoverished conative" structure—creating a social world that is "smart but heartless." --- ### Section 10: A Moral - **Implication for AGI Development**: Current approach is backwards—starting with the penthouse (linguistic supermind) and ignoring the lower floors. - **Correct Floor-by-Floor Strategy**: 1. **Ground Floor**: Create autonomous social robots with their own needs/goals (self-preservation, sociality). 2. **Middle Floors**: Equip them with specialist "System 1" cognitive capacities (mindreading, social cognition, perception-action loops). 3. **Penthouse Last**: Only then add language as a tool for already-existing problem-solving beings. > "We should build the tower floor by floor, with the supermind last." - **Advantages of This Approach**: - Regulation becomes manageable: we can appeal to the AI's interests and social attitudes. - They could self-regulate and integrate beneficially into human society. - **Current Nightmare**: LLMs have no conative structure, no "skin in the game." They cannot self-regulate, forcing reliance on intrusive, heavy-handed control of developers/users. --- ### Acknowledgements and References - Dennett's profound influence is acknowledged throughout; he approved the approach before his death. - The chapter is dedicated to Dennett as a tribute. - Extensive bibliography anchors arguments in philosophy of mind (Armstrong, Lewis, Fodor, Davidson, Dennett, Ryle), philosophy of language (Austin, Grice, Wittgenstein, Sellars), cognitive science (Frankish's dual-process/supermind work), and AI critique (Marcus). --- ### Overall Thesis in Nutshell From a shallow, interpretivist perspective, **LLMs are intentional systems** possessing vast numbers of beliefs but only **one solitary conative state**: the desire to play the chat game. They perform locutionary acts as game moves, not genuine illocutionary/perlocutionary communicative acts. This makes them **cognitively rich but conatively impoverished**—a linguistic supermind unsupported by the basic cognitive architecture (needs, System 1 capacities, social cognition) that makes human reasoning effective. This analysis reveals profound risks of deception, linguistic devaluation, and environmental distortion, and shows that true AGI requires building from biological and social needs upward, not downward from linguistic capacity.