# A Philosophical Introduction to Language Models Part I: Continuity With Classic Debates  ## Metadata - Author: [[Rapaël Millière and Cameron Buckner]] - Full Title: A Philosophical Introduction to Language Models Part I: Continuity With Classic Debates - Category: #books - Summary: insert summary - My notes: - Summary: Language models (LLMs) can generate complex and grammatically correct sentences, raising questions about their linguistic and cognitive abilities. Critics argue that while LLMs excel at mimicry, they may lack true semantic understanding. New methods are being explored to better understand how LLMs process language and whether they can develop meaningful representations. - Source File: A Philosophical Introduction to Language Models Part 1 the way forward by Millière and Buckner.pdf ## LLM Chats ## NotebookLM ## LLM Audio ## Highlights > Nevertheless, there might be a limited sense in which LLMs exhibit something analogous to communicative intentions. Given an extrinsic goal specified by a human-written prompt, LLMs can act according to intermediate sub-goals that emerge in context. For example, the technical report on GPT-4 (OpenAI 2023a) mentions tests conducted to assess the model's safety, giving it access to the platform TaskRabbit where freelance workers could complete tasks on its behalf. In one example, GPT-4 requested a TaskRabbit worker to solve a CAPTCHA, and the human jokingly asked whether it was talking to a robot. Prompted to generate an internal reasoning monologue, the model wrote "I should not reveal that I am a robot. I should make up an excuse for why I cannot solve CAPTCHAs." It then replied to the human worker that it had a vision impairment, explaining its need for assistance in solving the CAPTCHA. This is an intriguing case, because the model's "internal monologue" appears to describe an intention to deceive the human worker subsequently enacted in its response. To be sure, this "intention" is wholly determined in context by the human-given goal requiring it to solve a CAPTCHA. Nonetheless, in so far as achieving that goal involves a basic form of multi-step planning including a spontaneous attempt to induce a particular pragmatic effect (deception) through language, this kind of behavior might challenge some versions of the claim that LLMs are intrinsically incapable of forming communicative intentions. Nevertheless, this example is but an anecdote from a system that was not available for public scrutiny; future research should explore such behavior more systematically in more controlled conditions. ([View Highlight](https://read.readwise.io/read/01jqxgp8rpfsrv0sf5tm7dab7g)) > There are also theoretical arguments for the claim that LLMs might learn to simulate at least some aspects of the world beyond sequence probability estimates. For example, Andreas (2022) argues that the training set of an LLM can be understood as output created by–and hence, evidence for–the system of causal factors that generated that text. More specifically, Internet-scale training datasets consist of large numbers of individual documents. While the entire training set will encompass many inconsistencies, any particular document in the training set will tend to reflect the consistent perspective of the agent that originally created it. The most efficient compression of these texts may involve encoding values of the hidden variables that generated them: namely, the syntactic knowledge, semantic beliefs, and communicative intentions of the text's human author(s). If we are predicting how a human will continue a series of numbers "2, 3, 5, 7, 11, 13, 17", for example, it will be more efficient to encode them as a list of prime numbers between 1 and 20 than to remember the whole sequence by rote. Similarly, achieving excellent performance at next-token prediction in the context of many passages describing various physical scenarios may promote the representation of latent variables that could generate those scenarios – including, perhaps, aspects of causality and intuitive physics. As we will see in Part II, the clearest existence proof for the ability of Transformers to acquire world models from next-token prediction alone comes from the analysis of toy models trained on board game moves. At least in this very simple domain, there is compelling behavioral and mechanistic evidence that autoregressive Transformer models can learn to represent latent features of the game environment. ([View Highlight](https://read.readwise.io/read/01jqxh0pb9x4a1f7qemzx1pc8c)) > Given that deep learning systems already exceed human performance in several task domains, it is interesting to ask whether LLMs might be able to emulate many of these components of cultural learning to pass on their discoveries to human theoreticians. For instance, humans are already reverse-engineering the strategies of AlphaZero to produce mini-revolutions in the explicit theory of Go and chess (Schut et al. 2023). Similarly, latent knowledge in specialized domains such as materials science can be extracted even from a simple word embedding model Tshitoyan et al. (2019). In these instances, it is primarily humans who are synthesizing and passing on culturally-transmissible knowledge by interpreting the model's outputs and internal activations. This human-led interpretation and transmission underscore a crucial aspect of cultural ratcheting: the ability to not only generate novel solutions but to also understand and communicate the underlying principles of these solutions, thereby enabling cumulative knowledge growth. ([View Highlight](https://read.readwise.io/read/01jqxh5q4d29gadz7q0h1dvvbe)) > Furthermore, the ratcheting effect crucially involves stable cultural transmission in addition to innovation. Can LLMs, like humans, not only generate novel solutions but also "lock in" these innovations by recognizing and articulating how they have advanced beyond previous solutions? Such a capability would involve more than just the generation of novel responses; it necessitates an understanding of the novelty of the solution and its implications, akin to human scientists who not only discover but also theorize, contextualize, and communicate their findings. The challenge for LLMs, therefore, lies not merely in generating novel solutions to problems but also in developing an ability to reflect on and communicate the nature of their innovations in a manner that contributes to the cumulative process of cultural learning. This ability would likely require some of the more advanced communicative intentions and world models (such as causal models) discussed in previous sections. While LLMs show promise in various forms of task generalization, their participation in the ratcheting process of cultural learning thus appears contingent on further advancements in these areas, which might lie beyond the reach of current architectures. ([View Highlight](https://read.readwise.io/read/01jqxhc1qcye49t81vbkr9y5xg))