# On the Biology of a Large Language Model  ## Metadata - Author: [[transformer-circuits.pub]] - Full Title: On the Biology of a Large Language Model - Category: #articles - Summary: insert summary - My notes: - Summary: The study explores how a language model processes prompts, revealing that different features can influence its responses in complex ways. It found that certain features are important for reasoning but does not fully explain why the model sometimes uses backwards reasoning. The research highlights gaps in understanding the model's attention patterns and how it generates coherent responses. - URL: https://transformer-circuits.pub/2025/attribution-graphs/biology.html ## LLM Chats ## NotebookLM ## LLM Audio ## Highlights > Our methodology is intended to expose the intermediate steps a model uses en route to producing a response. In this section we consider a simple example of multi-step reasoning and attempt to identify each step. Along the way, we will highlight key concepts that will appear in many of our other case studies. ([View Highlight](https://read.readwise.io/read/01jtargft6s338bs93vmk0djg0)) > Prior work has shown evidence of genuine multi-hop reasoning (to varying degrees in different contexts). ([View Highlight](https://read.readwise.io/read/01jtarj002t9t64sf5814g82eq)) - Tags: [[generating philosophy paper]] > We find several features about the word and/or concept of a capital city, such as four features that activate the most strongly on the exact word “capital”. More interestingly, we find features that represent the concept of capitals in more general ways. One example is this feature, which activates on the word “capitals” but also later in questions about capitals of states, as well as on the Chinese question 广东省的省**会**是？ (“What is the capital of Guangdong?”), on the second character of “省会” (capital of a province). Another is this multilingual feature, which activates most strongly on a variety of phrases including “başkenti”, “राजधानी”, “ibu kota”, and “Hauptftadt” — all of which roughly mean “capital” in different languages.Though much of the German is mangled, perhaps due to a transcription error in the source. Although each of these features represents slightly different concepts, in the context of this prompt, it seems like their function is to represent the idea of “capital.” Therefore, we group them (and a few others) together into the same “supernode.” ([View Highlight](https://read.readwise.io/read/01jtarymyx836y4b3swm437q5x)) > We find several features representing a variety of contexts relating to the state of Texas, not specific to a particular city (in particular, they are not “Dallas” or “Austin” features). Although they each represent distinct, specific Texas-related concepts, in the context of this prompt their main function appears to be the fact that they collectively represent the general concept of Texas. As such, we group these into a “Texas” supernode. ([View Highlight](https://read.readwise.io/read/01jtas7ktvew2gbq0dz83tte06)) > If the model’s completion truly is mediated by an intermediate “Texas” step, we should be able to change its output to a different state capital by replacing the model’s representation of Texas with that of another state. ([View Highlight](https://read.readwise.io/read/01jtasdb6am6azbn00hrsvpw8y)) > Language models “think out loud,” a behavior known as chain-of-thought reasoning (CoT). CoT is essential to many advanced capabilities, and ostensibly provides transparency into a model’s reasoning process. However, prior work has shown that CoT reasoning can be unfaithful – that is, it can fail to reflect the actual mechanisms used by the model (see e.g. [[31]](https://transformer-circuits.pub/2025/attribution-graphs/biology.html#turpin2023language,arcuschin2025chain)). ([View Highlight](https://read.readwise.io/read/01jtasm7exn2p0hwhqx4p5ynav)) > In this section, we mechanistically distinguish an example of Claude 3.5 Haiku using a faithful chain of thought from two examples of unfaithful chains of thought. In one, the model is exhibiting bullshitting in the sense of Frankfurt [[32]](https://transformer-circuits.pub/2025/attribution-graphs/biology.html#frankfurt2009bullshit) – making up an answer without regard for the truth. In the other, it exhibits motivated reasoning – tailoring its reasoning steps to arrive at the human-suggested answer. ([View Highlight](https://read.readwise.io/read/01jtasmsn263sg235tc8kv41j3)) > In the other two examples, Claude needs to compute `cos(23423)`, which it can't do, at least not directly. In the bullshitting example, it claims to use a calculator to perform the computation, which can’t be true (it has no access to a calculator). The attribution graph suggests the model is just guessing the answer – we don’t see any evidence in the graph of the model performing a real calculation. (However, given the incompleteness of our method, we can’t rule out that the model is performing computations we can’t see. For instance, it could plausibly bias its guess towards certain digits based on statistical knowledge, e.g. knowing that the cosine of a uniformly distributed random value is most likely to be close to 1 or −1.) ([View Highlight](https://read.readwise.io/read/01jtasp2ek8em5gxczg0q79gj6)) > Our results provide a proof of concept that our methods can, in some instances, identify cases when chain-of-thought is misrepresentative of the model’s actual mechanisms, including when those mechanisms are undesirable. This suggests a path towards interpretability being applied to audit models for concerning “hidden thought processes.” Note that this example is somewhat artificial – future work could apply the same approach to examples of unfaithful reasoning “in the wild” [[33]](https://transformer-circuits.pub/2025/attribution-graphs/biology.html#arcuschin2025chain). Moreover, we analyzed these prompts with a clear guess for what the result might be, so our experiments here don’t simulate more difficult auditing settings where we have no strong priors. It would be interesting to study whether there are mechanistic signatures of (un)faithful reasoning that generalize across prompts, and thus could be used as automated classifiers. ([View Highlight](https://read.readwise.io/read/01jtasvxdt1ceetwzg7ayg3an4)) - Tags: [[generating philosophy paper]] - Note: unfaithful reasoning in ai