Speaker's notes for [[WIP Presentation - AI Persuasion and Propaganda]]. WIP seminar presentation 2026-01-20. **Related:** [[McKenna Propaganda Paper Plan]], [[McKenna Propaganda Paper - Planning Scraps]], [[2026-01-19 - McKenna Paper Critique Development]] --- # Complete Speaker's Notes ## SLIDE 1: Title Slide **AI PERSUASION AND PROPAGANDA: A Critique of McKenna** Nick Young, WIP Seminar, 2026-01-20 ### SPEAKER'S NOTES **Opening approach:** - This is work in progress on [[Robin McKenna]]'s recent paper "Sophistry on Steroids" - Paper argues we should worry about AI persuasion even in a "best case" where it's not deceptive - My claim: his distinction between troubling AI persuasion and propaganda collapses - But also skeptical whether his specific scenario can actually occur **Frame the contribution:** - Philosophical analysis applying the [[Epistemic Defectiveness Model]] to McKenna's case - Two-part structure: first show the collapse, then question the empirical assumptions --- ## SLIDE 2: McKenna's Central Question ### SPEAKER'S NOTES **McKenna's framing:** - His question: should we worry about persuasive AI? - Answer: yes, but for specific reasons - Not about superhuman persuasion (AI being better than humans at persuading) - About scale + concentration of power - "Marketplace of arguments" - the argumentative environment where public deliberation happens **Key quote breakdown:** - "Small number of powerful actors" - this is about concentrated capacity - "Dominating the marketplace of arguments" - controlling what arguments are available and salient - This provides materials for discussing "important moral, political and societal issues" **Why this matters:** - McKenna is trying to identify a worry that's STRUCTURAL not CONTENT-based - Not about lies or manipulation, but about who controls argumentative space - This sets up his "best case" move - bracketing the obvious problems to isolate something deeper --- ## SLIDE 3: Not About Superhuman Persuasion ### SPEAKER'S NOTES **The key distinction McKenna makes:** - NOT worried because AI will be better persuaders than humans - Worried because AI persuasion is (a) easy to deploy at scale + (b) requires resources to create in the first place - This creates asymmetry: few actors with resources can flood the marketplace **Why he emphasizes this:** - Avoids getting stuck on questions about AI capabilities - Don't need AGI or superhuman reasoning to have the problem - Current LLMs are sufficient if they can generate plausible arguments at scale - The problem is QUANTITY + CONCENTRATION, not QUALITY **Implications:** - This means the worry survives even if AI arguments are mediocre - Even if humans are better persuaders one-on-one, scale changes the game - Links to his "sophistry on steroids" metaphor - it's about armies of arguers, not superior arguers --- ## SLIDE 4: Sophistry on Steroids ### SPEAKER'S NOTES **The sophistry metaphor:** - Ancient sophists: arguers for hire, would argue any side for payment - Traditional worry: undermines truth-seeking, treats argument as mere tool - AI sophistry: can generate ARMIES of arguers for hire instantly - "Huge numbers" + "within minutes" + "massive increase in quantity" **Key phrase: "any dilution in quality made up for by massive increase in quantity":** - Acknowledges AI arguments might be lower quality than expert human arguments - But quantity compensates - flood the zone - This is the core of his "not about superhuman persuasion" point **Connection to marketplace:** - If you can generate unlimited arguments for your position - Even if each is mediocre, the cumulative effect dominates the marketplace - Counterarguments get drowned out by sheer volume **Note for discussion:** - This assumes volume matters for persuasion effectiveness - We'll return to skepticism about this later (finite schemas) --- ## SLIDE 5: McKenna's "Best Case" - Description ### SPEAKER'S NOTES **What McKenna brackets (sets aside):** - Misinformation and deception - obviously problematic, not interesting - Manipulative tactics - emotional manipulation, dark patterns, etc. - Coercive influence - threats, forcing people **What remains in the "best case":** - Reason-giving arguments - actual arguments with premises and conclusions - Aimed at [[rational persuasion]] - trying to get people to believe through reasoning - But deployed at scale by concentrated actors - this is the structural worry **Why this move matters:** - McKenna wants to isolate a worry that persists even when we idealize away the obvious problems - If there's still a problem in the "best case," that's philosophically interesting - Shows the issue isn't just "AI will be used to lie" (everyone knows that) - The issue is something about the structure of argumentation and public deliberation **Critical point:** - McKenna thinks this "best case" is NOT propaganda - He treats propaganda as necessarily involving manipulation or coercion - My paper challenges this - argues the "best case" (if troubling) IS propaganda --- ## SLIDE 6: McKenna's "Best Case" - His Words ### SPEAKER'S NOTES **Key phrase: "genuine arguments, evidence, and reasons":** - Not fabricated, not lies, not misleading individual claims - Actual arguments that have logical structure - Evidence that supports the conclusions - Reasons that genuinely count in favor of the position **"Conclusions that the purveyors themselves endorse":** - Not arguing for positions they don't believe - Sincere in that they actually hold these views (or want them adopted) - This rules out pure cynicism - arguing for X while believing not-X **"Genuine arguments with the attempt to rationally persuade, not simply deceive or mislead":** - Trying to get people to believe through recognizing reasons - Not trying to trick them or manipulate them - This is meant to be the cleanest case - persuasion as it "should" work **McKenna's assumption:** - This best case is troubling (because of scale/concentration) - But NOT propaganda (because it's "genuine" rational persuasion) - My task: show this distinction doesn't hold --- ## SLIDE 7: What is Rational Persuasion? ### SPEAKER'S NOTES **Why this matters:** - This is McKenna's working definition of rational persuasion for his "best case" - It's deliberately THIN - only requires giving reasons that support your conclusion - Does NOT require: presenting counterarguments, representing the full reason-space, acknowledging competing considerations - This thinness opens the door to the omission-based route to propaganda **The critical move:** - [[Thomas Mitchell|Mitchell]] & [[Thomas Douglas|Douglas]] define rational persuasion as providing reasons that the target recognizes and acts on - But "providing reasons" ≠ "providing a fair representation of the reason-space" - You can rationally persuade (on this definition) while systematically omitting decisive counterarguments - This makes the thin conception compatible with epistemic defectiveness **Connect to Epistemic Defectiveness Model:** - [[Bonard, Contesi and Marques|Bonard et al.]]'s cherry-picking example: reporting only victories, not defeats - Satisfies Grice's Quality maxim (it's TRUE) but violates Quantity (omits relevant info) - Giving genuine reasons can still be misleading if you systematically omit reasons against it - The thin conception permits exactly this selective reason-giving **Anticipated questions:** - "But isn't ALL persuasion selective?" → Yes, but SYSTEMATIC selectivity at SCALE crosses into propaganda - "Doesn't this make ordinary persuasion too demanding?" → Individual vs. campaign-level persuasion - "Can you intend to rationally persuade while being epistemically defective?" → Yes, if you believe your position is correct; defectiveness comes from systemic omission, not bad faith **Transition:** This thin conception is what McKenna uses to bracket propaganda. But I argue this distinction is unstable - the features that make AI persuasion troubling are what make it epistemically defective. --- ## SLIDE 8: McKenna's Assumption ### SPEAKER'S NOTES **McKenna's position:** - ✓ Politically troubling - marketplace domination is a real problem - ✗ NOT propaganda - because it involves "genuine" rational persuasion, not manipulation **Why he thinks this:** - He treats propaganda as "overtly coercive or manipulative" (his words from p. 1) - His best case brackets manipulation and coercion - Therefore (he thinks) it's outside propaganda territory **The question my paper asks:** - Does this distinction actually hold? - Can something be politically troubling in McKenna's sense WITHOUT being propaganda? - Or do the features that make it troubling also make it propaganda? **Preview of my argument:** - I'll argue the distinction collapses - Using the Epistemic Defectiveness Model, the troubling features ARE what constitute propaganda - Marketplace domination through scale IS a form of epistemic defectiveness **Transition:** To show this, I need to introduce a more sophisticated account of propaganda than McKenna's "coercive or manipulative" baseline. Enter the Epistemic Defectiveness Model. --- ## SLIDE 9: But What IS Propaganda? ### SPEAKER'S NOTES **The negative connotation:** - In contemporary English, "propaganda" is pejorative - Associated with "systematic dissemination... in a biased or misleading way" - Not neutral term for persuasive communication **Paradigm cases:** - Nazi propaganda - Third Reich's systematic communication campaigns - Rwanda RTML radio - hate radio inciting genocide - Tobacco industry campaigns - systematic deception about health effects **What these share:** - Not just individual lies or misleading claims - Systematic, coordinated campaigns - Epistemically problematic (biased, misleading, omitting crucial info) - Political/institutional provenance - Aimed at shaping beliefs and attitudes of groups **Why this matters:** - These paradigm cases involve MORE than just coercion or manipulation - They involve epistemic defectiveness - something wrong with how information is presented - [[Constant Bonard|Bonard]], [[Filippo Contesi|Contesi]], & [[Teresa Marques]] develop this into a systematic account **Note:** - There are other accounts of propaganda (expressive, rhetorical, etc.) - I'm focusing on the epistemic account because it's well-developed and fits the case --- ## SLIDE 10: Epistemic Defectiveness Model - Description ### SPEAKER'S NOTES **The four conditions (all necessary):** **1. [[Epistemic defectiveness]]:** - Message is defective in some broad sense - Not just false - includes misleading, omitting, generating false inferences - This is the CORE condition - what makes propaganda epistemically problematic **2. Political intent:** - Institutional provenance - produced by/for organizations, causes, institutions - Persuasion for political ends - trying to shape political beliefs/attitudes/actions - Not just any persuasion, but politically-oriented persuasion **3. Social targeting:** - Targets socially significant groups - not just individuals - About shaping group beliefs, public opinion, collective attitudes - This is what makes it politically consequential **4. Awareness:** - Agent aware or should be aware of the defectiveness - Responsibility condition - can't be propaganda if you genuinely don't realize it's defective - But includes "should be aware" - negligence counts **Why this model:** - Refines earlier work by [[Sheryl Tuttle Ross]] - Widely cited in recent propaganda literature - Captures paradigm cases while being philosophically precise **Key point:** - Condition 1 (epistemic defectiveness) is doing the heavy lifting - This is what I'll argue McKenna's scenario satisfies --- ## SLIDE 11: Epistemic Defectiveness Model - Their Words ### SPEAKER'S NOTES **Breaking down the conditions in their own words:** **Condition 1 - "epistemically defective message":** - "false, inappropriate, or connected to other beliefs in ways that are inapt, misleading, or unwarranted" - Very broad notion of defectiveness - Includes: falsehood, inappropriateness, misleading connections, unwarranted inferences - Tied to "illocutionary and perlocutionary intents" - what you're trying to do communicatively **Condition 2 - "produced by, or on behalf of, a social or political institution":** - Not just individual speakers - Organizational/institutional context - "Intention to persuade, for direct political ends" - This rules out art, entertainment, education (even if they have political effects) **Condition 3 - "socially significant group":** - Not targeting individuals - About groups - publics, demographics, constituencies - What makes propaganda politically consequential **Condition 4 - "aware of the epistemic defectiveness or should have been aware":** - Knowledge or negligence - Producer knows or should know the message is defective - Responsibility condition **Read the quote slowly:** - Let it sink in how broad "epistemic defectiveness" is - It's not just lying - it's anything that makes the epistemic character of the message problematic --- ## SLIDE 12: Epistemic Defectiveness ≠ Falsehood ### SPEAKER'S NOTES **The cherry-picking example:** - Report true information about winning one battle - Omit true information about losing another battle - Everything you SAY is true - no Gricean Quality violation (truth) - But you violate Gricean Quantity (relevant information) **Why this is misleading:** - Audience pragmatically infers: "this is all the relevant information about the war" - That inference is FALSE - there's highly relevant information being omitted - So the communication is epistemically defective even though it contains only truths **Gricean framework:** - Quality maxim: be truthful - Quantity maxim: provide the right amount of information (not too little, not too much) - Propaganda can satisfy Quality while violating Quantity **Key insight:** - Epistemic defectiveness comes from OMISSION, IMPLICATURE, CONNECTIONS - Not just from saying false things - This dramatically widens what counts as epistemically defective **Why this matters for McKenna:** - His "best case" involves "genuine arguments, evidence, and reasons" (all true) - But if they're systematically selective, omitting counterarguments, they can still be defective - Defectiveness through omission is exactly what his scenario involves **Transition:** This broader notion of epistemic defectiveness is what lets me argue that McKenna's best case IS propaganda. Let me show you the dilemma. --- ## SLIDE 13: The Dilemma for McKenna ### SPEAKER'S NOTES **The dilemma structure:** - McKenna wants a middle category: troubling but not propaganda - I argue there's no stable middle ground - He has to choose between two horns **Horn 1 - Fully idealized "best case":** - Imagine AI that's completely transparent about its provenance - Presents arguments symmetrically across positions - Explicitly flags limitations and uncertainties - Ensures robust exposure to counterarguments - Result: NO epistemic defectiveness (it's giving you the full picture) - Therefore: NOT propaganda ✓ - BUT ALSO: Not troubling! ✗ **Why Horn 1 isn't troubling:** - If the system ensures you see counterarguments, it's not dominating the marketplace - If it's transparent and symmetric, it's not creating misleading implicatures - The worry evaporates along with the propaganda classification - This is like having an AI debate moderator, not an AI persuader **Horn 2 - Marketplace domination:** - Small number of actors flood argumentative environment - Arguments favor their interests/positions - This IS troubling in McKenna's sense (scale + concentration) - But: is it epistemically defective? **The question for Horn 2:** - If flooding creates omission-by-crowding-out → epistemically defective - If saturation creates false inferences → epistemically defective - If controlling salience distorts what seems decisive → epistemically defective - Then it satisfies the EDM → it IS propaganda **The dilemma:** - Either idealize away the troubling features (Horn 1) → not propaganda but not troubling - Or preserve the troubling features (Horn 2) → epistemically defective → propaganda - No stable middle ground --- ## SLIDE 14: Campaign-Level Defectiveness ### SPEAKER'S NOTES **The core conceptual innovation:** - Standard approach: evaluate propaganda at the level of individual utterances - My move: evaluate at the level of the CAMPAIGN or COMMUNICATIVE ECOSYSTEM - This matters because McKenna's worry is systemic effects - marketplace domination, not individual bad arguments - The Epistemic Defectiveness Model can accommodate this if we treat "the communicative act" broadly **Route 1: Omission-by-crowding-out:** - Not just failing to mention counterarguments in your speech - Making counterarguments practically inaccessible in the information environment - Flood the argumentative space with pro-X arguments, anti-X arguments get crowded out - Audiences encounter them less, harder to find, get less attention - Result: audience's epistemic position is defective even if each individual argument is sound - Violates Grice's Quantity maxim at the ecosystem level **Route 2: Quantity-style misleadingness:** - Saturation creates false pragmatic inferences - "I keep seeing arguments for X, barely any for Y" → "X must be the stronger position" - "These arguments are everywhere" → "This must be what the evidence supports" - These inferences are FALSE if saturation is artificial (concentrated deployment, not argumentative strength) - Audience misled not by what's said, but by what the PATTERN implicates **Route 3: Distorted salience:** - What gets attention signals what's important or decisive - Pro-X arguments dominate attention through volume, appear more important/decisive than they are - Form of epistemic manipulation even if arguments themselves are sound - How propaganda works: not just lying, but controlling what people think about and how much weight it gets **Critical distinction:** - Individual argument soundness vs. systemic epistemic health - Collection of individually sound arguments can collectively constitute propaganda - This dissolves McKenna's troubling-but-not-propaganda category **Why this matters for the dilemma:** - McKenna: marketplace domination is troubling but not propaganda (arguments are "genuine") - But if marketplace domination WORKS through these three mechanisms, it IS epistemically defective - Epistemically defective + political intent + targets groups + awareness → propaganda (on EDM) - Troubling version of his scenario collapses into propaganda **Anticipated questions:** - "Too demanding? Does every persuader need to ensure counterarguments are accessible?" → Difference between individual persuaders and concentrated actors flooding the marketplace - "Can we evaluate propaganda at campaign level?" → Yes - propaganda scholarship often talks about campaigns, not just individual messages - "How do we individuate 'campaigns'?" → Fair question, but conceptual point stands: systematic coordinated communication can be defective even when individual pieces aren't --- ## SLIDE 15: The Collapse ### SPEAKER'S NOTES **Summarizing the argument:** - McKenna wants: troubling AI persuasion that's NOT propaganda - Middle category: problematic but not epistemically defective - I've argued: this category is UNSTABLE **The instability:** - EITHER: Idealize to remove defectiveness → becomes benign (not troubling) - OR: Preserve troubling features → becomes defective → propaganda - No stable middle ground **The key insight:** - "The mechanism that makes it troubling IS the mechanism of epistemic defectiveness" - It's not two separate things (structural problem + epistemic problem) - It's ONE problem viewed from two angles - Marketplace domination WORKS BY creating epistemic defectiveness **What this means:** - McKenna has not identified a new category of AI risk - He's identified a potential form of propaganda - The "best case" rhetoric is misleading - if it's genuinely best case, it's not troubling - If it's troubling, it's propaganda **Implications for AI ethics:** - Can't bracket propaganda concerns by focusing on "genuine arguments" - Scale + concentration + selective reason-giving = propaganda - Need to think about AI persuasion using propaganda frameworks **Transition:** But now the skeptical turn. I've argued that IF McKenna's scenario happens, it's propaganda. But CAN it actually happen? Can the marketplace of arguments be dominated in this way? --- ## SLIDE 16: But Wait: Can It Actually Happen? ### SPEAKER'S NOTES **The skeptical turn:** - Grant everything I've argued so far (troubling = propaganda) - Now question the antecedent: can marketplace actually be dominated? - This is an empirical question, not just conceptual **Finite argument schemas:** - Reference: [[Douglas Walton|Walton]], Reed, & Macagno (2008) - major work on argumentation schemes - Argument schemes = patterns of reasoning (appeal to authority, slippery slope, appeal to consequences, etc.) - For any political question, there are FINITE recognizable argument types - Immigration: appeals to rights, consequences, fairness, cultural preservation, economic impacts, etc. - Climate: appeals to precaution, economic costs, intergenerational justice, scientific authority, etc. - Redistribution: appeals to desert, need, equality, incentives, property rights, etc. **Variations but underlying structure:** - Rhetorical framing varies infinitely - Emotional coloring varies - Specific examples vary - BUT underlying argumentative structure is FINITE **Implication for McKenna's scenario:** - If I flood you with 10,000 "appeal to consequences" arguments for immigration restriction - You're not encountering 10,000 different arguments - You're encountering the SAME argument 10,000 times with different packaging - Diminishing returns - after you've seen the schema a few times, more instances don't add persuasive force --- ## SLIDE 17: The Empirical Question ### SPEAKER'S NOTES **Don't want to overstate this:** - This is a tentative empirical claim, not a confident prediction - The questions are OPEN - need actual research **Two possibilities:** **Possibility 1: Argument schemas constrain flooding:** - If argumentation is schema-based and schemas are finite - Then flooding produces diminishing returns - People recognize "I've seen this argument before in different words" - Implicature "these must be the main considerations" breaks down - Marketplace domination fails at the argumentative level **Possibility 2: Rhetorical/emotional framing matters more:** - Even if argument schemas are finite, rhetorical framing may not be - Emotional resonance, narrative framing, affective loading vary infinitely - Flooding might succeed at the level of ATTENTION and AFFECT - Even if it fails at the level of argumentative content - "I'm tired of hearing about this" vs. "I'm convinced by this" **The crucial question:** - Does AI influence work at the level of ARGUMENT or AFFECT? - If argument: finite schemas might constrain it - If affect: flooding could work despite finite schemas **What this means for the paper:** - McKenna has identified a POSSIBLE area of concern - But unclear whether his SPECIFIC SCENARIO (marketplace domination via argument flooding) is the right place to locate it - Real AI influence risks might lie elsewhere **Honest uncertainty:** - I don't know the answer to this empirical question - That's okay - the philosophical work is showing the conceptual structure - But we need to be clear about what's conceptual vs. what's empirical --- ## SLIDE 18: Upshot ### SPEAKER'S NOTES **Main claim - the collapse:** - McKenna's distinction between troubling AI persuasion and propaganda COLLAPSES - Under the Epistemic Defectiveness Model, the troubling features ARE propaganda features - No stable middle ground **The conceptual contribution:** - Campaign-level epistemic defectiveness as a way to evaluate systematic persuasion - Three routes: omission-by-crowding-out, quantity-style misleadingness, distorted salience - Shows how individually sound arguments can be collectively defective **But - empirical uncertainty:** - Unclear whether McKenna's SPECIFIC SCENARIO (marketplace domination via volume) is achievable - Finite argument schemas might constrain the effectiveness of flooding - Rhetorical/emotional framing might matter more than argumentative content **Task ahead:** - If real AI influence risks lie elsewhere than argument flooding, need to identify where - Possibilities: attention manipulation, affective influence, parasocial relationships, narrative control - The conceptual framework (EDM, campaign-level evaluation) applies to these too **What the paper does:** - Clarifies the conceptual landscape - Shows McKenna hasn't identified a new category of AI risk distinct from propaganda - Opens empirical questions about where real risks lie - Provides framework (EDM) for evaluating AI influence campaigns --- ## SLIDE 19: Questions for Discussion ### SPEAKER'S NOTES **Question 1: Campaign-level defectiveness as unit of analysis:** - Is this the right way to think about propaganda? - Or does it demand too much of individual persuaders? - Should we hold concentrated actors to different standards than individuals? **Question 2: How finite are argument schemas?** - Is Walton et al.'s framework right? - Does flooding fail argumentatively but succeed affectively? - What's the relationship between argument and affect in persuasion? **Question 3: Is EDM too broad?** - Does omission-based defectiveness demand too much? - Should persuaders be responsible for ensuring counterarguments are accessible? - Difference between individual responsibility and systemic effects? **Question 4: Where DO the real risks lie?** - If not argument flooding, then what? - Attention manipulation? Affective influence? Parasocial relationships? - How should we study AI influence empirically? **Be ready for:** - Pushback on campaign-level evaluation (too demanding?) - Questions about other accounts of propaganda (expressive, rhetorical) - Skepticism about whether EDM captures all propaganda - Interest in the empirical questions (that's good - shows the conceptual work opened real questions)